Backoff Advanced Search migrations if cluster is saturated

Problem

We can inadvertently put too much pressure (or sometimes not enough) on our Elasticsearch cluster during Advanced Search migrations. We don't currently have any insight into the health of our Elasticsearch cluster during migrations.

Solution

Create a Search::ClusterHealthCheck.healthy? and a corresponding Search::ClusterHealthCheck.unhealthy? interface that returns true or false depending on our criteria. I think a good place to start would be to simply return unhealthy if the average heap size for data nodes is above a certain threshold.

If Search::ClusterHealthCheck reports that the cluster is unhealthy we should do nothing and re-queue the Advanced Search migration for a specified back-off interval.