Add deferred_count field on deferring Sidekiq jobs

Resolves gitlab-com/gl-infra/scalability#2382 (closed)

What does this MR do and why?

Context: The DeferJobs Sidekiq middleware can be used to defer jobs from a worker during an incident. More info in this runbooks doc https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/sidekiq/deferring-jobs.md.

This MR adds deferred_count field on deferring Sidekiq jobs.

This adds more observability information on how many times a job has been deferred.

How to set up and validate locally

1. Pull branch and restart Sidekiq gdk restart rails-background-jobs
2. Enable feature flag to defer the jobs

Feature.enable(:"defer_sidekiq_jobs_Chaos::SleepWorker")

3. Perform a job

Chaos::SleepWorker.perform_async(1)

4. Check gdk tail rails-background-jobs for the deferred_count field:

2023-06-13_16:17:34.81290 rails-background-jobs : {"severity":"INFO","time":"2023-06-13T16:17:34.812Z","retry":3,"queue":"default","backtrace":true,"version":0,"queue_namespace":"chaos","args":["1"],"class":"Chaos::SleepWorker","jid":"77dadb2743af5cc847b8c499","created_at":"2023-06-13T16:17:34.666Z","correlation_id":"d3fca0355ce8c0cb1f9fd20542f0b119","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:2f0bc834677c58fe9b274ea1acbbcd384d61ad88243f7ae8b6c38429900bba4f","size_limiter":"validated","enqueued_at":"2023-06-13T16:17:34.690Z","job_size_bytes":3,"pid":39417,"message":"Chaos::SleepWorker JID-77dadb2743af5cc847b8c499: deferred: 0.111993 sec","job_status":"deferred","scheduling_latency_s":0.009848,"redis_calls":10,"redis_duration_s":0.000516,"redis_read_bytes":1429,"redis_write_bytes":1186,"redis_cache_calls":7,"redis_cache_duration_s":0.000347,"redis_cache_read_bytes":1426,"redis_cache_write_bytes":504,"redis_queues_calls":3,"redis_queues_duration_s":0.000169,"redis_queues_read_bytes":3,"redis_queues_write_bytes":682,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.004499,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.111993,"completed_at":"2023-06-13T16:17:34.812Z","load_balancing_strategy":"primary","job_deferred_by":"feature_flag","deferred_count":1,"db_duration_s":0.0}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.