I have created a new script that will auto repair shards that fall out of the cluster. This script checks shard health every 5 minutes and if it detects missing shards, it will automatically rejoin them.

Shards can drop out of the cluster for a number of reasons but 99% of the time it is due to a timeout where Elasticsearch basically marks the shard as unavailable.

This should help keep the cluster more consistent and when a shard does decide to sleep on the job it will be forced to rejoin the cluster. This should keep any shard failures under six minutes at most.

They don't fail often but when they do, it screws up aggregation counts and causes some objects to go missing while it is offline.

As always, you can pass "metadata=true" to ensure all shards reported for your query.