Finalize conversion to bigint for push_event_payloads (take 2) (!65795) · Merge requests · GitLab.org / GitLab

Krasimir Angelov requested to merge 288005-finalize-push-events-payloads-bigint-conversion-take-2 into master Jul 09, 2021

What does this MR do?

This MR is same as !64577 (merged), which change had to be reverted with !65112 (merged) in order to prevent problems with self-managed on 14.1.

Now we want to do this in 14.2.

On a high level, the operation takes the following steps:

Ensure the migration of event_id to event_id_convert_to_bigint is completed.
Copy indexes and FKs
Swap columns

Cleanup (removing the old int columns and the triggers) will be done once events table conversion is also finalized. This leaves us with a trigger that copy values from bigint column to integer column, but this works fine, as long we do not hit the limit for integer.

Related to #288005 (closed).

Before merging

Announce in #g_delivery channeel on Slack that there is a post-deployment migration which will take approx. 132 minutes (probably less on production) introduced with this MR.

Database migrations

⏱ Timing

From !64577 (merged):

Creating unique index on "push_event_payloads" ("event_id_convert_to_bigint") takes 69.322 min on DB Lab - https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/4670/commands/16704.
Validating the FK takes 49.528 min on DB Lab - https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/4670/commands/16711.

This is approximately matching what the migrations testing pipeline reports - 132 min (7917.6 s).

The above times are from DB Lab, from the production logs when !65112 (merged) was deployed to production we can see that the index creation took ~18 minutes, and the FK validation took ~14 minutes.

⬆ Up

$ bundle exec rails db:migrate:up VERSION=20210709024048
== 20210709024048 FinalizePushEventPayloadsBigintConversion2: migrating =======
-- transaction_open?()
   -> 0.0000s
-- index_exists?("push_event_payloads", :event_id_convert_to_bigint, {:unique=>true, :name=>"index_push_event_payloads_on_event_id_convert_to_bigint", :algorithm=>:concurrently})
   -> 0.0024s
-- execute("SET statement_timeout TO 0")
   -> 0.0008s
-- add_index("push_event_payloads", :event_id_convert_to_bigint, {:unique=>true, :name=>"index_push_event_payloads_on_event_id_convert_to_bigint", :algorithm=>:concurrently})
   -> 0.0147s
-- execute("RESET ALL")
   -> 0.0007s
-- transaction_open?()
   -> 0.0000s
-- foreign_keys("push_event_payloads")
   -> 0.0042s
-- execute("ALTER TABLE push_event_payloads\nADD CONSTRAINT fk_a5e47ac4c5\nFOREIGN KEY (event_id_convert_to_bigint)\nREFERENCES events (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0084s
-- execute("ALTER TABLE push_event_payloads VALIDATE CONSTRAINT fk_a5e47ac4c5;")
   -> 0.0116s
-- execute("LOCK TABLE push_event_payloads, events IN ACCESS EXCLUSIVE MODE")
   -> 0.0008s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name(:event_id)
   -> 0.0000s
-- quote_column_name("event_id_tmp")
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id\" TO \"event_id_tmp\"")
   -> 0.0008s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name(:event_id_convert_to_bigint)
   -> 0.0000s
-- quote_column_name(:event_id)
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id_convert_to_bigint\" TO \"event_id\"")
   -> 0.0011s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name("event_id_tmp")
   -> 0.0000s
-- quote_column_name(:event_id_convert_to_bigint)
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id_tmp\" TO \"event_id_convert_to_bigint\"")
   -> 0.0007s
-- quote_table_name("trigger_07c94931164e")
   -> 0.0000s
-- execute("ALTER FUNCTION \"trigger_07c94931164e\" RESET ALL")
   -> 0.0008s
-- change_column_default("push_event_payloads", :event_id, nil)
   -> 0.0043s
-- change_column_default("push_event_payloads", :event_id_convert_to_bigint, 0)
   -> 0.0032s
-- execute("ALTER TABLE push_event_payloads DROP CONSTRAINT push_event_payloads_pkey")
   -> 0.0009s
-- rename_index("push_event_payloads", "index_push_event_payloads_on_event_id_convert_to_bigint", "push_event_payloads_pkey")
   -> 0.0008s
-- execute("ALTER TABLE push_event_payloads ADD CONSTRAINT push_event_payloads_pkey PRIMARY KEY USING INDEX push_event_payloads_pkey")
   -> 0.0012s
-- remove_foreign_key("push_event_payloads", {:name=>"fk_36c74129da"})
   -> 0.0039s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name("fk_a5e47ac4c5")
   -> 0.0000s
-- quote_column_name("fk_36c74129da")
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\"\nRENAME CONSTRAINT \"fk_a5e47ac4c5\" TO \"fk_36c74129da\"\n")
   -> 0.0011s
== 20210709024048 FinalizePushEventPayloadsBigintConversion2: migrated (0.0990s)

⬇ Down

$ bundle exec rails db:migrate:down VERSION=20210709024048
== 20210709024048 FinalizePushEventPayloadsBigintConversion2: reverting =======
-- transaction_open?()
   -> 0.0000s
-- index_exists?("push_event_payloads", :event_id_convert_to_bigint, {:unique=>true, :name=>"index_push_event_payloads_on_event_id_convert_to_bigint", :algorithm=>:concurrently})
   -> 0.0018s
-- execute("SET statement_timeout TO 0")
   -> 0.0006s
-- add_index("push_event_payloads", :event_id_convert_to_bigint, {:unique=>true, :name=>"index_push_event_payloads_on_event_id_convert_to_bigint", :algorithm=>:concurrently})
   -> 0.0038s
-- execute("RESET ALL")
   -> 0.0006s
-- transaction_open?()
   -> 0.0000s
-- foreign_keys("push_event_payloads")
   -> 0.0037s
-- execute("ALTER TABLE push_event_payloads\nADD CONSTRAINT fk_a5e47ac4c5\nFOREIGN KEY (event_id_convert_to_bigint)\nREFERENCES events (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0025s
-- execute("ALTER TABLE push_event_payloads VALIDATE CONSTRAINT fk_a5e47ac4c5;")
   -> 0.0032s
-- execute("LOCK TABLE push_event_payloads, events IN ACCESS EXCLUSIVE MODE")
   -> 0.0007s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name(:event_id)
   -> 0.0000s
-- quote_column_name("event_id_tmp")
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id\" TO \"event_id_tmp\"")
   -> 0.0009s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name(:event_id_convert_to_bigint)
   -> 0.0000s
-- quote_column_name(:event_id)
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id_convert_to_bigint\" TO \"event_id\"")
   -> 0.0008s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name("event_id_tmp")
   -> 0.0000s
-- quote_column_name(:event_id_convert_to_bigint)
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id_tmp\" TO \"event_id_convert_to_bigint\"")
   -> 0.0007s
-- quote_table_name("trigger_07c94931164e")
   -> 0.0000s
-- execute("ALTER FUNCTION \"trigger_07c94931164e\" RESET ALL")
   -> 0.0009s
-- change_column_default("push_event_payloads", :event_id, nil)
   -> 0.0021s
-- change_column_default("push_event_payloads", :event_id_convert_to_bigint, 0)
   -> 0.0029s
-- execute("ALTER TABLE push_event_payloads DROP CONSTRAINT push_event_payloads_pkey")
   -> 0.0014s
-- rename_index("push_event_payloads", "index_push_event_payloads_on_event_id_convert_to_bigint", "push_event_payloads_pkey")
   -> 0.0009s
-- execute("ALTER TABLE push_event_payloads ADD CONSTRAINT push_event_payloads_pkey PRIMARY KEY USING INDEX push_event_payloads_pkey")
   -> 0.0009s
-- remove_foreign_key("push_event_payloads", {:name=>"fk_36c74129da"})
   -> 0.0033s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name("fk_a5e47ac4c5")
   -> 0.0000s
-- quote_column_name("fk_36c74129da")
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\"\nRENAME CONSTRAINT \"fk_a5e47ac4c5\" TO \"fk_36c74129da\"\n")
   -> 0.0007s
== 20210709024048 FinalizePushEventPayloadsBigintConversion2: reverted (0.0528s)

Database schema changes

 CREATE TABLE push_event_payloads (
     commit_count bigint NOT NULL,
-    event_id integer NOT NULL,
+    event_id_convert_to_bigint integer DEFAULT 0 NOT NULL,
     action smallint NOT NULL,
     ref_type smallint NOT NULL,
     commit_from bytea,
@@ -17446,7 +17446,7 @@ CREATE TABLE push_event_payloads (
     ref text,
     commit_title character varying(70),
     ref_count integer,
-    event_id_convert_to_bigint bigint DEFAULT 0 NOT NULL
+    event_id bigint NOT NULL
 );

Does this MR meet the acceptance criteria?

Conformity

I have included changelog trailers, or none are needed. (Does this MR need a changelog?)
I have added/updated documentation, or it's not needed. (Is documentation required?)
I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
I have self-reviewed this MR per code review guidelines.
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
I have followed the style guides.
This change is backwards compatible across updates, or this does not apply.

Availability and Testing

I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
I have tested this MR in all supported browsers, or it's not needed.
I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

Label as security and @ mention @gitlab-com/gl-security/appsec
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
Security reports checked/validated by a reviewer from the AppSec team

Related to #288005 (closed)

Edited Jul 27, 2021 by Krasimir Angelov

Finalize conversion to bigint for push_event_payloads (take 2)