Skip to content

Add exclusive lock on project in ElasticCommitIndexerWorker

What does this MR do?

Indexing the same project twice at the same time causes conflict errors in Elasticsearch and this usually leads to failures which can lead to a project never being successfully indexed correctly. The simplest solution is to use a lock in the worker and just fail immediately if it's already running and let Sideikiq retries requeue the project if it does have more updates that need indexing. When it next runs it will check if there are new commits that need indexing.

We use a 1 hour TTL on the lock since it can take a long time to index a new project.

NOTE: We do de-duplicate this job in-queue as well but this de-duplication key is cleared (rightly so) when the job starts so long running jobs will still end up running duplicates in parallel and that's why we need this extra lock.

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Closes #32648 (closed)

Edited by Dylan Griffith

Merge request reports

Loading