Enqueuer worker: scope the lease key with the repository id
🔥 Problem
The Enqueuer worker use an exclusive lease key to make sure that no two executions handle the same image repository.
The problem is that the lease key is currently a fixed one.
We're observing a pretty high amount of lease key taken
messages: 170K+ messages in the last 24 hours. That's 2.5 times the amount of successful executions that start migrations.
In addition to that, the exclusive lease key can be left behind during shutdowns/restarts.
Using a fixed key, these conditions will pause the migration for 30 minutes (the lease key duration).
The core issue is that migration on repo A has nothing to do with the migration on repo B. If there is an issue with the lease key while processing repo A, we shouldn't impact repo B execution.
🚒 Solution
- Scope the lease key with the repo id so that we still guarantee that no 2 executions process the same repo.
Edited by David Fernandez