Skip to content

Update BulkImports::EntityWorker deduplication strategy

George Koltsov requested to merge georgekoltsov/update-bulk-import-entity-worker-deduplicate-strategy into master

What does this MR do and why?

Mentions #352984 (closed)

This MR updates BulkImports::EntityWorker's deduplication strategy from until_executed to until_executing. The main reason is sometimes the migration can get stalled in started state, without progressing further. EntityWorker enqueues many PipelineWorkers, which when finished, enqueues EntityWorker again, so that a new batch of pipeline workers can be process, from the next stage.

Migration stall can happen intermittently, not always, due to timings, in the following scneario:

  1. Entity worker schedules 3 pipeline workers
  2. First pipeline worker finishes and enqueues entity worker
  3. While entity worker is executing (and will exit early, since current 'stage' is running), the other 2 pipeline workers enqueue entity worker with the same arg 2 more times
  4. Both of the enqueued Entity worker jobs will be deduplicated and entity worker will not be enqueued anymore
  5. As a result, migration is stalled in started state

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by George Koltsov

Merge request reports