Skip to content

Use database replica for active jobs

Sylvester Chin requested to merge sc1-delay-active-jobs into master

What does this MR do and why?

This MR changes the way the application handles ActiveJobs in Sidekiq by enabling database load-balancing with the :delayed data consistency behaviour.

Currently, active jobs always use the primary db instance (:always data consistency).

See #403257 (closed)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

  1. Open 2 terminals. 1 for gdk to send jobs. 1 to tail the logs

  2. Fire some mail jobs

Notify.note_issue_email(User.first.id, Note.first.id).deliver_later

The tailed logs will show updated "load_balancing_strategy":"primary" and "worker_data_consistency":"always"

> gdk tail rails-background-jobs
{"severity":"INFO","time":"2023-03-31T06:15:40.494Z","retry":3,"queue":"mailers","backtrace":true,"class":"ActionMailer::MailDeliveryJob","args":["[FILTERED]"],"jid":"dd07dc2a744fda2eaa14c388","created_at":"2023-03-31T06:15:34.705Z","correlation_id":"cc60d4156a62d4be811b3d189540d557","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:mailers:4cc5d8fdfbef6675c3df0a4b81502683499eb921276291645e0228fdcfc9b634","size_limiter":"validated","enqueued_at":"2023-03-31T06:15:34.712Z","job_size_bytes":367,"pid":89342,"message":"ActionMailer::MailDeliveryJob JID-dd07dc2a744fda2eaa14c388: done: 5.725739 sec","job_status":"done","scheduling_latency_s":0.056292,"redis_calls":7,"redis_duration_s":0.001193,"redis_read_bytes":608,"redis_write_bytes":453,"redis_cache_calls":5,"redis_cache_duration_s":0.000988,"redis_cache_read_bytes":606,"redis_cache_write_bytes":267,"redis_queues_calls":2,"redis_queues_duration_s":0.000205,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":39,"db_write_count":3,"db_cached_count":12,"db_replica_count":0,"db_primary_count":39,"db_main_count":39,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":12,"db_main_cached_count":12,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.071,"db_main_duration_s":0.071,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":4.71612,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":5.725739,"completed_at":"2023-03-31T06:15:40.494Z","load_balancing_strategy":"primary","db_duration_s":0.095896}
  1. Enable the feature flag and try again
::Feature.enable(:use_replica_for_active_jobs)
Notify.note_issue_email(User.first.id, Note.first.id).deliver_later

The tailed logs will show updated "load_balancing_strategy":"primary_no_wal" and "worker_data_consistency":"delayed"

{"severity":"INFO","time":"2023-03-31T06:16:40.026Z","retry":3,"queue":"mailers","backtrace":true,"class":"ActionMailer::MailDeliveryJob","args":["[FILTERED]"],"jid":"fb2bea6793e7a3dbd4aee1cb","created_at":"2023-03-31T06:16:39.857Z","correlation_id":"16af0f1595686d9032df8df63a79f280","worker_data_consistency":"delayed","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:mailers:cc6adfc2b53472e681dd6d33726a0face0f7d8cae61e15331a18dec587544c60","size_limiter":"validated","enqueued_at":"2023-03-31T06:16:39.875Z","job_size_bytes":367,"pid":89342,"message":"ActionMailer::MailDeliveryJob JID-fb2bea6793e7a3dbd4aee1cb: done: 0.15003 sec","job_status":"done","scheduling_latency_s":0.001138,"redis_calls":15,"redis_duration_s":0.001589,"redis_read_bytes":1835,"redis_write_bytes":1121,"redis_cache_calls":13,"redis_cache_duration_s":0.001447,"redis_cache_read_bytes":1833,"redis_cache_write_bytes":935,"redis_queues_calls":2,"redis_queues_duration_s":0.000142,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":41,"db_write_count":3,"db_cached_count":12,"db_replica_count":0,"db_primary_count":41,"db_main_count":41,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":12,"db_main_cached_count":12,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.039,"db_main_duration_s":0.039,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.112288,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.15003,"completed_at":"2023-03-31T06:16:40.026Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.015941}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports