As many of you have noticed, the API has been returning a lot more 5xx errors that usual lately. Part of the reason is that certain users are running extremely expensive aggregations on 10 terabytes of data in the cluster and causing the cluster to destabilize. These aggregations may be innocent or it could be an attempt to purposely overload the API.

For the time being, I am disabling aggregations (the aggs parameter) until I can figure out which aggregations are causing the cluster to destabilize. This won't be a permanent change, but unfortunately some aggregations are consuming massive amounts of CPU time and causing the cluster to fall behind which causes the increase in 5xx errors.

If you use aggregations for research, please let me know which aggregations you use in this thread and I'll be happy to test them to see which ones are causing issues.

We are going to be adding additional nodes to the cluster and upgrading the entire cluster to a more recent version of Elasticsearch.

What we will probably do is segment the data in the cluster so that the most recent year's worth of data reside on their own indexes and historical data will go to other nodes where complex aggregations won't take down the entire cluster.

I apologize for this aggravating issue. The most important thing right now is to keep the API up and healthy during the election so that people can still do searches, etc.

The API will currently be down for about half an hour as I work to fix these issues so that the API becomes more stable.

Thank you for your patience!