Skip to content

Move migration statistics to a file per migration

Simon Tomlinson requested to merge per-directory-instrumentation-testing into master

What does this MR do and why?

Describe in detail what your merge request does and why.

Moves the migration-stats.json file from a single file with a list of per-migration statistics to a file per-migration, so that each migration's data is encapsulated in a single directory.

By making all the data for a migration self-contained within the migration's directory, it will be easier for the testing pipeline notifier to treat batches from background migrations the same way that it treats individual non-background migrations.

This must not be merged until per-migration migration-stats.json (gitlab-org/database-team/gitlab-com-database-testing!89 - merged), which makes the pipeline forward-compatible with the new format.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

migration-stats.json is under the migration directory:

tree tmp/migration-testing
tmp/migration-testing
└── up
    ├── AddCreatorIdToDeployTokens
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── AddDatabaseGrafanaConfigToApplicationSettings
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── AddIndexToDeployTokensOnCreatorId
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── AddIndexesForPrimaryEmailSecondCleanupMigration
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── AddMigrationPlanIndexToContainerRepositories
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── AddTextLimitToDatabaseGrafanaApplicationSettings
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── AddTmpIndexToSupportLeakyRegexCleanup
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── AddUserForeignKeyToDeployTokens
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── ChangeSearchRateLimitDefaultValues
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── ChangeSearchRateLimitValues
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── CleanupAfterFixingIssueWhenAdminChangedPrimaryEmail
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── CleanupDraftDataFromFaultyRegex
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── DropTemporaryIndexesForPrimaryEmailMigrationSecondCleanup
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── FinalizeProjectNamespacesBackfill
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── PrepareIndexForCiJobArtifactsUnlockedWithExpireAt
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    └── metadata.json

16 directories, 61 files

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Set up at least one pending migration (perhaps you have one already, or can roll back the latest one)
  2. rm -r tmp/migration-testing
  3. Run bundle exec rake gitlab:db:migration_testing:up
  4. See the per-migration migration-stats.json files in tmp/migration-testing/up

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Simon Tomlinson

Merge request reports