Skip to content

Support auto-batching background migrations

Patrick Bair requested to merge pb-track-background-job-executions into master

What does this MR do?

For details about the motivation/plan behind this change, see: #300915 (closed)

In order to limit the scope of the initial work, this MR implements simplified scheduling and processing of the migration jobs. We will use a single sidekiq-cron job that runs the next migration job inline (calling the added Scheduler class). This is done by starting with the oldest active migration, and processing all jobs once, until there are no remaining jobs. In future iterations we can add support for multiple crons to parallelize the active migrations, and build on retry logic, etc. This should be much easier to manage since the full migration is tracked in the database.

Running the job directly in the cron job gives us more control over single-threaded execution and batching. We can create and run jobs on an as-needed basis, which allows us to immediately see the effects if we alter the migration, such as by changing batch size or pausing execution. We also have much less risk of losing asynchronous jobs, and being unsure if we should continue scheduling additional work.

For example usage, the subsequent MR adds the migration helper methods: !55872 (merged)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Mayra Cabrera

Merge request reports