Skip to content

Update PipelineProcessWorker deduplication strategy to `until_executed`

Leaminn Ma requested to merge update-pipeline-process-worker into master

What does this MR do and why?

Updates the PipelineProcessWorker's deduplication strategy from :until_executing to :until_executed, if_deduplicated: :reschedule_once, ttl: 1.minute.

The purpose of this change is to:

  • Provide more clarity on pipeline processing.
  • Improve performance by reducing the number of jobs that run and then are immediately dropped from not obtaining the lease in AtomicProcessService.execute.

Because we are making this change behind a Feature Flag, we took the approach of saving the deduplication settings in the Sidekiq job itself. When this Feature Flag is removed, those modifications will also be removed.

Feature Flag: ci_pipeline_process_worker_dedup_until_executed

How to set up and validate locally

  1. Enable the FF ci_pipeline_process_worker_dedup_until_executed.
  2. Run any pipeline and observe that it runs just like before enabling the FF.
  3. To see the deduplication strategy that's in effect:
  • Run a pipeline with this config example (gitlab-ci.yml). This config has a high likelihood that multiple worker jobs will get deduplicated.
  • Tail the Sidekiq logs (filtered to only show the logs of interest): tail -1 -f log/sidekiq.log | grep -E -i '\"message\"\:\"PipelineProcessWorker.*dedup.*' -oh
  • Click the "Play all manual jobs" button.
  • Observe that when the FF is on, the deduplication strategy for the PipelineProcessWorker is outputted as such:

"message":"PipelineProcessWorker JID-d66c2be751819f0fb3ce83e8: deduplicated: dropped until executed","deduplication.type":"dropped until executed","deduplication.options.if_deduplicated":"reschedule_once"}

  • When the FF is off:

"message":"PipelineProcessWorker JID-07ad11aae2a62a25469f5c34: deduplicated: dropped until executing","deduplication.type":"dropped until executing"}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #388539 (closed)

Edited by Leaminn Ma

Merge request reports