Unfortunately when I first started this project, I didn't have the necessary equipment to enable replicas across all indexes (each index usually being a month or quarter of Reddit data). Over the years, there have been multiple node failures, crashes, power outages, etc. that have affected the health of the cluster.

The good news is that we now have the necessary equipment to start indexing all data to a new cluster with redundant nodes / storage arrays to keep the overall health of the cluster strong.

Over the next two weeks (starting late Monday evening or Tuesday), I will begin the process of moving over all data to a new cluster (version 8.31 for the Elasticsearch users out there). I anticipate the entire process will take at a minimum five days and at a maximum two weeks (Probably one week is a decent target).

Once this is done, all historical Reddit data will be made available along with improvements in how we process removal requests. We had another power outage this evening that caused more issues which is exasperated by the lack of redundancy.

I will update on the progress and let everyone know when the entire dataset is available. I will also enable aggregations since the new hardware should be able to support the increased load.

If you have any questions, let me know -- I also post updates on Twitter so feel free to interact with me there as well.

I hope everyone has a safe and fun holiday! May you and your family stay healthy and happy.

Thanks to everyone for your support including the mods here that will often ping me via text when there are major issues. :)

Thanks!

Edit I just wanted to mention that until we are able to bring the new cluster online, older data will be unreliable with gaps until we switch over to the new cluster. So for the time being, if you use the API, please note that some data will be unavailable. Thank you!