Fix batched background migration to backfill missing search data
We had to pause the batched background migration to backfill missing search data for #361219 (closed).
The reason is that because of batching_scope: -> (relation) { Issue }
, each batched job instead of breaking down the work in sub-batches, starts iterating issues
table from the start, until eventually gets terminated with Sidekiq::Shutdown
. The related Slack discussion is here - https://gitlab.slack.com/archives/CNZ8E900G/p1657161519642019?thread_ts=1656578448.257079&cid=CNZ8E900G.
Specifying batching_scope
is needed when we have additional filters and not iterating by id only. There is some work in progress to improve BatchedMigrationJob
's API, and we should add more docs/examples to make it easier to understand and use.
In order to complete this migration we need new MR that will:
- Fix
lib/gitlab/background_migration/backfill_imported_issue_search_data.rb
- Make
db/post_migrate/20220621040800_backfill_imported_issue_search_data.rb
a no-op - Reschedule the migration by deleting the old one and enqueuing a new migration
Here is an example MR of rescheduling a batched migration - !87496 (diffs).