Add health-checks settings keys for Sidekiq

What does this MR do?

We will soon be serving Sidekiq metrics from a separate server process. During this transition, we temporarily plan to instantiate SidekiqExporter twice: once where it is used now (in the worker), and again in the new server process. This is because we want to keep health-checks being served from workers, and only serve metrics from the new server process.

To that end, we need to make a clearer distinction between health-checks and metrics endpoint config, which are currently both part of SidekiqExporter and its settings bucket. This will be a breaking change, so we are taking small incremental steps to remain backwards-compatible.

This MR starts by splitting up settings keys between metrics and health-checks for Sidekiq, so that those servers can be configured independently. For this to remain backwards compatible, we default the new settings keys for health checks to the existing exporter settings. In the application MR, we will start using the former to instantiate the in-worker server.

Related issues

• gitlab#345802 (closed)
• gitlab MR: gitlab!74875 (merged)

Test plan

Verify that new settings apply during Chef run

By default, address and port settings for health_checks_* settings should default to whatever is set for exporter. So without any changes to config, run a gitlab-ctl reconfigure.

Verify new settings emitted to gitlab.yml with address and port set to what is set for the exporter:

root@local:/# grep -A 10 "sidekiq_exporter" /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml
    sidekiq_exporter:
      enabled: true
      log_enabled: false
      address: localhost
      port: 8082

    sidekiq_health_checks:
      enabled: true
      log_enabled: false
      address: localhost
      port: 8082

✅

Now add the following line to gitlab.rb and reconfigure:

sidekiq['health_checks_listen_port'] = 8092

this produces a diff during the Chef run:

* templatesymlink[Create a gitlab.yml and create a symlink to Rails root] action create
    * template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create
      - update content in file /var/opt/gitlab/gitlab-rails/etc/gitlab.yml from a8a81b to 0bcf49
      --- /var/opt/gitlab/gitlab-rails/etc/gitlab.yml	2021-11-18 11:52:59.879265411 +0000
      +++ /var/opt/gitlab/gitlab-rails/etc/.chef-gitlab20211118-5307-oc4ilh.yml	2021-11-18 11:59:59.382344459 +0000
      @@ -738,7 +738,7 @@
             enabled: true
             log_enabled: false
             address: localhost
      -      port: 8082
      +      port: 8092

Convince ourselves it's been written out:

grep -A 10 "sidekiq_exporter" /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml
    sidekiq_exporter:
      enabled: true
      log_enabled: false
      address: localhost
      port: 8082

    sidekiq_health_checks:
      enabled: true
      log_enabled: false
      address: localhost
      port: 8092

✅

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

• Merge Request Title, and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
• trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• [-] Documentation created/updated (I think it could be confusing to document these settings at this point, since we do not yet have the capability to spawn a separate server)
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for the GitLab Chart opened: gitlab-org/charts/gitlab!2272 (merged)