Skip to content

Adds extra stats to batched background migration sampling

Leonardo da Rosa requested to merge database-53-estimate-length-of-capped-batched-background-migrations into master

What does this MR do and why?

Describe in detail what your merge request does and why.

Solves partially: 53 - Estimate length of capped batched background migrations

This first MR adds extra stats under meta section to migration-stats.json when sampling Batched Background migrations.

With theses new metrics, we can estimate how much time the migration will take in production environment - TBD in another MR in https://gitlab.com/gitlab-org/database-team/gitlab-com-database-testing

migration-stats.json:

{
  "version": null,
  "name": "batch_1",
  "walltime": 1.7080299999797717,
  "success": true,
  "total_database_size_change": 8192,
  "query_statistics": [],
  "meta": {
    "max_batch_size": 1,
    "total_tuple_count": 5000000,
    "interval": 180
  }
}
With max_batch_size: 25_000 and 3 minutes cool down:
# Total tuple count
5_000_000

# Max batch size
25_000

# Number of batches
5_000_000 / 25_000
=> 200

# Interval
(180 / 60).minutes
=> 3 minutes

# Time to complete (minutes)
200 * 3
=> 600

# Time to complete (hours)
200 * 3 / 60
=> 10
With max_batch_size: 2_000 and 1 minute cool down:
# Total tuple count
5_000_000

# Max batch size
2_000

# Number of batches
5_000_000 / 2_000
=> 2500

# Interval
(60 / 60).minutes
=> 1 minutes

# Time to complete (minutes)
2500 * 1
=> 2500

# Time to complete (hours)
2500 * 1 / 60
=> 41

# Time to complete (days)
2500 * 1 / 60 / 24.0
=> 1.7

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. First, add a new batched background migration: bundle exec rails g post_deployment_migration my_post_deployment_migration
  2. Check the examples in https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#example
  3. Run the test sampling: bundle exec rake gitlab:db:migration_testing:sample_batched_background_migrations:main | tee tmp/migration-testing/main/background_migrations/full-batched-migration-output.log || true
  4. Check the batch-details.json file in tmp/migration-testing/main/background_migrations/YourMigration/

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Leonardo da Rosa

Merge request reports