Follow-up: Allow Idempotent Sidekiq jobs to use read-only database replicas
This is the follow-up item of #322452 (closed)
In #322452 (closed) we allowed Sidekiq jobs to use read-only database replicas.
Since the deduplication should always take into account the latest binary replication pointer into account, not the first one, we decided to make these two mutually exclusive, and the deduplication will not work with sticky
or delayed
.
We should improve this, and allow using sticky and delayed on idempotent jobs as well.
Proposed solution:
- Replication strategy will store the latest WAL locations in Redis in case that the job is dropped due to the deduplication strategy.
- We store this WAL location only if the job is configured to utilize LB capabilities (
data_consistency != :always
) and the write-ahead location has changed. - On job execution, the replication strategy will update job's hash with
dedup_wal_locations
containing the latest location from the Redis cache (in case it exists). - SidekiqServerMiddleware will use an updated
dedup_wal_locations
location when checking if the replica has caught up.
We decided to split !66280 (comment 650854524) and to merge some of the changes that are by design less-risky:
Rollout plan:
Step1: Preparation work:
-
Merge Change order of sidekiq LoadBalancing middleware -
Merge Support multiple wal locations for each database
Step2:
-
Merge Allow idempotent jobs to use load balancing - behind the ff
Step3: Follow up work:
-
Rollout of preserve_latest_wal_locations_for_idempotent_jobs feature flag -
Remove legacy wal locations
from SidekiqServerMiddleware -
Merge Create Rubocop rule to set include_shceduling: true
when enabling LB for idempotnent jobs -
Replace hardcoded database config name :main when we merge unification strategy #336566 (closed)
Edited by Nikola Milojevic