Skip to content

Backfill `type_new` column on integrations

What does this MR do?

Adds a background migration to backfill the new integrations.type_new column based on the legacy class name in type.

Extends the background migration RSpec matchers so we can verify the arguments for batched migration classes too.

Issue: #333507 (closed), part of the epic &2504 (closed) to rename "services" to "integrations", and specifically the child epic &6177 (closed).

The type_new column was added in !66541 (merged).

Migration output

$ rails db:migrate:up VERSION=20210727113447
== 20210727113447 BackfillIntegrationsTypeNew: migrating ======================
unknown OID 28: failed to recognize type of 'relfrozenxid'. It will be treated as String.
unknown OID 1034: failed to recognize type of 'relacl'. It will be treated as String.
unknown OID 194: failed to recognize type of 'relpartbound'. It will be treated as String.
== 20210727113447 BackfillIntegrationsTypeNew: migrated (0.0627s) =============

$ rails db:migrate:down VERSION=20210727113447
== 20210727113447 BackfillIntegrationsTypeNew: reverting ======================
== 20210727113447 BackfillIntegrationsTypeNew: reverted (0.0131s) =============

(I'm curious about those unknown OID errors, something weird with my DB? 🤔)

The Sidekiq job for Database::BatchedBackgroundMigrationWorker gets triggered every minute but seems to only actually run every 10 minutes, with gdk tail rails-background-jobs | grep cronjob:database_batched_background_migration you might see some deduplicated jobs first before it executes and you can see the changes in integrations:

2021-08-02_17:42:22.38781 rails-background-jobs : {"severity":"INFO","time":"2021-08-02T17:42:22.387Z","queue":"cronjob:database_batched_background_migration","args":[],"class":"Database::BatchedBackgroundMigrationWorker","retry":0,"backtrace":true,"version":0,"queue_namespace":"cronjob","jid":"fa2945f4476539b1a9e38219","created_at":"2021-08-02T17:42:22.386Z","meta.caller_id":"Cronjob","meta.feature_category":"database","correlation_id":"1b4813d9fe09197a81783b4abe3bfc85","idempotency_key":"resque:gitlab:duplicate:cronjob:database_batched_background_migration:592d9619e1997b640b70ce6a22f6713bc7793bb7a4e342b7380d90b691fcd6ae","duplicate-of":"9c51154e3ba659fba7a706d1","job_size_bytes":2,"pid":1129573,"job_status":"deduplicated","message":"Database::BatchedBackgroundMigrationWorker JID-fa2945f4476539b1a9e38219: deduplicated: dropped until executing","deduplication.type":"dropped until executing"}

Background Migration Details

  • 1169469 records to update on gitlab.com
  • batch size = 1000
  • 1169469 / 1000 = 1170 batches

Estimated time per batch: Between ~90ms to ~2s for UPDATE query with 1000 items

2 mins delay per batch (safe for the given total time per batch)

1170 batches * 2 min per batch = 39 hours to run all the scheduled jobs

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Related to #333507 (closed)

Edited by Luke Duncalfe

Merge request reports

Loading