Draft: Use elastic retry setting for search requests

What does this MR do and why?

Follow-up from Add application setting for elasticsearch retry... (!169903 - merged) which introduced an application setting called elasticsearch_retry_on_failure. This MR uses that setting to retry search requests when they fail. This only applies to search requests, not index, maintanence, deletes, etc.

Screenshots or screen recordings

Future work

All search requests currently go through the proxy client which is part of the elasticsearch gem and monkey-patched to suit our needs. We have introduced an execute_search method but it's not currently being used. If that method starts being used for searches, we need to also add retry logic there. It's non-trivial so I excluded that from this MR.

References

Please include cross links to any resources that are relevant to this MR This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

• !169903 (merged)
• #486935
• https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/current/advanced-config.html

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

1. Run migrations
2. Run a rails console with ELASTIC_CLIENT_DEBUG=true
3. Update the setting to be non-zero: settings = ApplicationSetting.current; settings.elasticsearch_retry_on_failure = 5; settings.save!
4. Set the elasticsearch url to something that doesn't exist so that requests fail: settings = ApplicationSetting.current; settings.elasticsearch_url = "http://localhost:9300"; settings.save!
5. Run a search: ::SearchService.new(User.first, {scope: 'projects', search: '*'}).search_objects
   1. See that the request fails and retries 5 times
6. Do the same for a non-search query, e.g. Gitlab::Elastic::Helper.default.client.ping
   1. See that the request fails and does not retry
7. Update the setting to be 0: settings = ApplicationSetting.current; settings.elasticsearch_retry_on_failure = 0; settings.save!
8. Run a search: ::SearchService.new(User.first, {scope: 'projects', search: '*'}).search_objects
   1. See that the request fails and does not retry
9. Test with a correct elasticsearch URL and see that everything works as normal

Related to #486935