Expose concurrent worker metrics to prom

What does this MR do and why?

corrective action from gitlab-com/gl-infra/production#18146 (closed) and addresses #437646 (closed)

This MR adds a new sidekiq worker metric to prometheus.

I added it to the ConcurrencyLimit::ResumeWorker so it will only run for workers that have this middleware enabled. The ConcurrencyLimit::ResumeWorker runs every minute and will report the metric whether there are jobs in the queue or not.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

N/A

How to set up and validate locally

patch for testing

diff --git a/ee/app/workers/zoekt/indexer_worker.rb b/ee/app/workers/zoekt/indexer_worker.rb
index 70bbeacd7d16..8fc3b42713eb 100644
--- a/ee/app/workers/zoekt/indexer_worker.rb
+++ b/ee/app/workers/zoekt/indexer_worker.rb
@@ -20,9 +20,10 @@ class IndexerWorker
     sidekiq_options retry: 2
     idempotent!
     pause_control :zoekt
-    concurrency_limit -> { 30 if Feature.enabled?(:zoekt_limit_indexing_concurrency) }
+    concurrency_limit -> { 1 }
 
     def perform(project_id, options = {})
+      sleep(30)
       return unless ::Gitlab::CurrentSettings.zoekt_indexing_enabled?
       return unless ::License.feature_available?(:zoekt_code_search)
 

1. enable sidekiq metrics in gdk

 gdk config set gitlab.rails_background_jobs.sidekiq_exporter_enabled true
 gdk reconfigure

2. apply the patch above to manually set this worker very low, and add a sleep timer to cause the worker queue to back up
3. in rails console, queue the worker for every project in your database

    Project.all.each { |p| Zoekt::IndexerWorker.perform_async(p.id) }

4. load the metrics page for sidekiq metrics: http://gdk.test:3807/metrics
5. ensure the values are updated for the metric

# HELP sidekiq_concurrency_limit_max_concurrent_jobs Multiprocess metric
# TYPE sidekiq_concurrency_limit_max_concurrent_jobs gauge
sidekiq_concurrency_limit_max_concurrent_jobs{worker="ElasticCommitIndexerWorker",pid="sidekiq_0"} 30
sidekiq_concurrency_limit_max_concurrent_jobs{worker="PackageMetadata::GlobalAdvisoryScanWorker",pid="sidekiq_0"} 10
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Search::Zoekt::DeleteProjectWorker",pid="sidekiq_0"} 100
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Security::ScanExecutionPolicies::CreatePipelineWorker",pid="sidekiq_0"} 10000
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Security::ScanResultPolicies::AddApproversToRulesWorker",pid="sidekiq_0"} 200
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Zoekt::IndexerWorker",pid="sidekiq_0"} 1
# HELP sidekiq_concurrency_limit_queue_size Multiprocess metric
# TYPE sidekiq_concurrency_limit_queue_size gauge
sidekiq_concurrency_limit_queue_size{worker="ElasticCommitIndexerWorker"} 0
sidekiq_concurrency_limit_queue_size{worker="PackageMetadata::GlobalAdvisoryScanWorker"} 0
sidekiq_concurrency_limit_queue_size{worker="Search::Zoekt::DeleteProjectWorker"} 0
sidekiq_concurrency_limit_queue_size{worker="Security::ScanExecutionPolicies::CreatePipelineWorker"} 0
sidekiq_concurrency_limit_queue_size{worker="Security::ScanResultPolicies::AddApproversToRulesWorker"} 0
sidekiq_concurrency_limit_queue_size{worker="Zoekt::IndexerWorker"} 44