Fix Enqueuer capacity conflict with deduplication
Problem
In !83091 (merged), we updated the Enqueuer from deduplicate :until_executing
to deduplicate :until_executed
.
The EnqueuerWorker
does not take any args, so with this new deduplication setting, we have in essence nullified our ability to have more than one import happening at a time. Our feature flags controlling capacity no longer have an affect, and anytime the Enqueuer re-enqueues to reach capacity, those jobs are instantly deduplicated
We can see this happening in the logs.
Solution
We need a way to have multiple jobs in the queue but ensure that the same container repository does not get queued twice. We don't know which container repository we will be using until the worker starts, so we need an alternative method to make the jobs not look like duplicates.
- Use an exclusive lease on the
#perform
method - Pass a random unique arg to the worker (perhaps a UUID?) so that they are not deduplicated. This essentially overrides the deduplication, but that is what we want here. We want to control the capacity within the worker itself.