Prepare indexes for bigint column conversions
What does this MR do?
Prepare indexes for asynchronous creation for tables that need to have columns converted from int
to bigint
. We need to do this for tables which have large indexes on GitLab.com, so deployments can complete in a reasonable time. The indexes should be created as part of the reindexing process that runs every weekend. Async index creation was added in !66478 (merged)
For example, with ci_job_artifacts
we have 5 indexes that contain the id
or job_id
columns, both of which are being converted from int
to bigint
. The migration in this MR schedules those 5 indexes to be created asynchronously, before the migration to swap the columns is merged. Once the swap migration runs, the add_concurrent_index
will be a no-op since the indexes will already exist. On self-managed installations, the indexes will be created as part of the swap migration as normal.
The related MRs for the swaps are: !65601 (merged), !66625 (merged), !66088 (merged)
How to setup and validate locally
Note the migration only inserts a record into postgres_async_indexes
for each index, so runtime is not a concern.
rails db:migrate:up VERSION=20210804151444
== 20210804151444 PrepareIndexesForCiJobArtifactBigintConversion: migrating ===
-- index_exists?(:ci_job_artifacts, :id_convert_to_bigint, {:unique=>true, :name=>:index_ci_job_artifact_on_id_convert_to_bigint, :algorithm=>:concurrently})
-> 0.0039s
-- add_index_options(:ci_job_artifacts, :id_convert_to_bigint, {:unique=>true, :name=>:index_ci_job_artifact_on_id_convert_to_bigint, :algorithm=>:concurrently})
-> 0.0000s
-- index_exists?(:ci_job_artifacts, [:project_id, :id_convert_to_bigint], {:where=>"file_type = 18", :name=>:index_ci_job_artifacts_for_terraform_reports_bigint, :algorithm=>:concurrently})
-> 0.0030s
-- add_index_options(:ci_job_artifacts, [:project_id, :id_convert_to_bigint], {:where=>"file_type = 18", :name=>:index_ci_job_artifacts_for_terraform_reports_bigint, :algorithm=>:concurrently})
-> 0.0000s
-- index_exists?(:ci_job_artifacts, :id_convert_to_bigint, {:where=>"file_type = 18", :name=>:index_ci_job_artifacts_id_for_terraform_reports_bigint, :algorithm=>:concurrently})
-> 0.0028s
-- add_index_options(:ci_job_artifacts, :id_convert_to_bigint, {:where=>"file_type = 18", :name=>:index_ci_job_artifacts_id_for_terraform_reports_bigint, :algorithm=>:concurrently})
-> 0.0000s
-- index_exists?(:ci_job_artifacts, [:expire_at, :job_id_convert_to_bigint], {:name=>:index_ci_job_artifacts_on_expire_at_and_job_id_bigint, :algorithm=>:concurrently})
-> 0.0026s
-- add_index_options(:ci_job_artifacts, [:expire_at, :job_id_convert_to_bigint], {:name=>:index_ci_job_artifacts_on_expire_at_and_job_id_bigint, :algorithm=>:concurrently})
-> 0.0000s
-- index_exists?(:ci_job_artifacts, [:job_id_convert_to_bigint, :file_type], {:unique=>true, :name=>:index_ci_job_artifacts_on_job_id_and_file_type_bigint, :algorithm=>:concurrently})
-> 0.0027s
-- add_index_options(:ci_job_artifacts, [:job_id_convert_to_bigint, :file_type], {:unique=>true, :name=>:index_ci_job_artifacts_on_job_id_and_file_type_bigint, :algorithm=>:concurrently})
-> 0.0000s
== 20210804151444 PrepareIndexesForCiJobArtifactBigintConversion: migrated (0.0531s)
rails db:migrate:down VERSION=20210804151444
== 20210804151444 PrepareIndexesForCiJobArtifactBigintConversion: reverting ===
== 20210804151444 PrepareIndexesForCiJobArtifactBigintConversion: reverted (0.0183s)
rails db:migrate:up VERSION=20210804153307
== 20210804153307 PrepareIndexesForTaggingBigintConversion: migrating =========
-- index_exists?(:taggings, :id_convert_to_bigint, {:unique=>true, :name=>:index_taggings_on_id_convert_to_bigint, :algorithm=>:concurrently})
-> 0.0032s
-- add_index_options(:taggings, :id_convert_to_bigint, {:unique=>true, :name=>:index_taggings_on_id_convert_to_bigint, :algorithm=>:concurrently})
-> 0.0000s
-- index_exists?(:taggings, [:taggable_id_convert_to_bigint, :taggable_type], {:name=>:i_taggings_on_taggable_id_convert_to_bigint_and_taggable_type, :algorithm=>:concurrently})
-> 0.0020s
-- add_index_options(:taggings, [:taggable_id_convert_to_bigint, :taggable_type], {:name=>:i_taggings_on_taggable_id_convert_to_bigint_and_taggable_type, :algorithm=>:concurrently})
-> 0.0000s
-- index_exists?(:taggings, [:taggable_id_convert_to_bigint, :taggable_type, :context], {:name=>:i_taggings_on_taggable_bigint_and_taggable_type_and_context, :algorithm=>:concurrently})
-> 0.0019s
-- add_index_options(:taggings, [:taggable_id_convert_to_bigint, :taggable_type, :context], {:name=>:i_taggings_on_taggable_bigint_and_taggable_type_and_context, :algorithm=>:concurrently})
-> 0.0000s
-- index_exists?(:taggings, [:tag_id, :taggable_id_convert_to_bigint, :taggable_type, :context, :tagger_id, :tagger_type], {:unique=>true, :name=>:taggings_idx_tmp, :algorithm=>:concurrently})
-> 0.0018s
-- add_index_options(:taggings, [:tag_id, :taggable_id_convert_to_bigint, :taggable_type, :context, :tagger_id, :tagger_type], {:unique=>true, :name=>:taggings_idx_tmp, :algorithm=>:concurrently})
-> 0.0000s
== 20210804153307 PrepareIndexesForTaggingBigintConversion: migrated (0.0388s)
rails db:migrate:down VERSION=20210804153307
== 20210804153307 PrepareIndexesForTaggingBigintConversion: reverting =========
== 20210804153307 PrepareIndexesForTaggingBigintConversion: reverted (0.0175s)
rails db:migrate:up VERSION=20210804154407
== 20210804154407 PrepareIndexesForCiStageBigintConversion: migrating =========
-- index_exists?(:ci_stages, :id_convert_to_bigint, {:unique=>true, :name=>:index_ci_stages_on_id_convert_to_bigint, :algorithm=>:concurrently})
-> 0.0032s
-- add_index_options(:ci_stages, :id_convert_to_bigint, {:unique=>true, :name=>:index_ci_stages_on_id_convert_to_bigint, :algorithm=>:concurrently})
-> 0.0001s
-- index_exists?(:ci_stages, [:pipeline_id, :id_convert_to_bigint], {:where=>"status in (0, 1, 2, 8, 9, 10)", :name=>:index_ci_stages_on_pipeline_id_and_id_convert_to_bigint, :algorithm=>:concurrently})
-> 0.0022s
-- add_index_options(:ci_stages, [:pipeline_id, :id_convert_to_bigint], {:where=>"status in (0, 1, 2, 8, 9, 10)", :name=>:index_ci_stages_on_pipeline_id_and_id_convert_to_bigint, :algorithm=>:concurrently})
-> 0.0000s
== 20210804154407 PrepareIndexesForCiStageBigintConversion: migrated (0.0262s)
rails db:migrate:down VERSION=20210804154407
== 20210804154407 PrepareIndexesForCiStageBigintConversion: reverting =========
== 20210804154407 PrepareIndexesForCiStageBigintConversion: reverted (0.0140s)
Does this MR meet the acceptance criteria?
Conformity
-
I have included changelog trailers, or none are needed. (Does this MR need a changelog?) -
I have added/updated documentation, or it's not needed. (Is documentation required?) -
I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?) -
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?) -
I have self-reviewed this MR per code review guidelines. -
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines) -
I have followed the style guides. -
This change is backwards compatible across updates, or this does not apply.
Availability and Testing
-
I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.) -
I have tested this MR in all supported browsers, or it's not needed. -
I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.