Add exclusive lock for UpdateRepositoryStorageWorker
What does this MR do and why?
Contributes to https://gitlab.com/gitlab-org/gitlab/-/issues/429049
Problem
- It's possible to run multiple migration workers for the same project/snippet/group simultaneously
- The worker can be killed and rescheduled by Sidekiq interrupt signal. It will leave the migration in inconsistent state.
Solution
Use an exclusive lock in storage migration workers. The exclusive lease key includes a project/snippet/group id to prevent simultaneous updates. The key value is a Sidekiq worker jid to track the owner of the update.
This setup should handle following situations:
- Worker tries to migrate a repository under existing migration (result: job is marked as failed)
- Worker started a migration but was interrupted and rescheduled. (result: job is marked as failed, lock is released)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.