Backfill LfsObjectsProject records of forks
What does this MR do?
LinkLfsObjects
background migration jobs were disabled as it's causing some statement timeouts on staging.
This re-schedules the background migration jobs but this time it'll not result to a statement timeout.
Related MRs:
Migration
== 20200217091401 RescheduleLinkLfsObjects: migrating =========================
== 20200217091401 RescheduleLinkLfsObjects: migrated (0.2321s) ================
Ran this test on #database-lab: https://gitlab.com/snippets/1942110. Created 1k forks of a source project with 100k LFS objects. The last 2 queries in that snippet are for querying the forks
in Gitlab::BackgroundMigration::LinkLfsObjects
and the insert query.
Given the query plans of those queries a background migration job can finish in around 22 seconds. Since we have 14,094 forks to be migrated as of this writing, that means we will have 15 background migration jobs enqueued with 2 minute delay. That said, this means the migration will be done in ~30 minutes.
This is based on the test data that I created. The formula I used is:
((18 + ((13 * 100) * 1000)) / 1000) / 60
Where:
-
18
- inms
, time spent on querying theforks
(1k per background migration job) -
13
- inms
, time spent on inserting 1kLfsObjectsProject
records -
100
- number of insert batches (100k records per project, 1k batch per insert) -
1000
- number of forks to be migrated per background migration job -
1000
-ms
inseconds
-
60
-seconds
inminutes
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team