Finalize conversion to bigint for push_event_payloads (!64577) · Merge requests · GitLab.org / GitLab

Krasimir Angelov requested to merge 288005-finalize-push-events-payloads-bigint-conversion into master Jun 22, 2021

What does this MR do?

This MR, first if its type, finalize the conversion to bigint for push_event_payloads table. Once we have this done we can finalize the conversion for the events table, which is referenced from push_event_payloads.

On a high level, the operation takes the following steps:

Ensure the migration of event_id to event_id_convert_to_bigint is completed.
Copy indexes and FKs
Swap columns

Cleanup (removing the old int columns and the triggers) will be done once events table conversion is also finalized. This leaves us with a trigger that copy values from bigint column to integer column, but this works fine, as long we do not hit the limit for integer.

Related to #288005 (closed).

Before merging

Announce in #g_delivery channeel on Slack that there is a post-deployment migration which will take approx. 132 minutes (probably less on production) introduced with this MR.

Database migrations

⏱ Timing

Creating unique index on "push_event_payloads" ("event_id_convert_to_bigint") takes 69.322 min on DB Lab - https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/4670/commands/16704.
Validating the FK takes 49.528 min on DB Lab - https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/4670/commands/16711.

This is approximately matching what the migrations testing pipeline reports - 132 min (7917.6 s).

⬆ Up

$ bundle exec rails db:migrate:up VERSION=20210622041846
== 20210622041846 FinalizePushEventPayloadsBigintConversion: migrating ========
-- transaction_open?()
   -> 0.0000s
-- index_exists?("push_event_payloads", :event_id_convert_to_bigint, {:unique=>true, :name=>"index_push_event_payloads_on_event_id_convert_to_bigint", :algorithm=>:concurrently})
   -> 0.0015s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- add_index("push_event_payloads", :event_id_convert_to_bigint, {:unique=>true, :name=>"index_push_event_payloads_on_event_id_convert_to_bigint", :algorithm=>:concurrently})
   -> 0.0033s
-- execute("RESET ALL")
   -> 0.0006s
-- transaction_open?()
   -> 0.0000s
-- foreign_keys("push_event_payloads")
   -> 0.0032s
-- execute("ALTER TABLE push_event_payloads\nADD CONSTRAINT fk_a5e47ac4c5\nFOREIGN KEY (event_id_convert_to_bigint)\nREFERENCES events (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0018s
-- execute("ALTER TABLE push_event_payloads VALIDATE CONSTRAINT fk_a5e47ac4c5;")
   -> 0.0037s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name(:event_id)
   -> 0.0000s
-- quote_column_name("event_id_tmp")
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id\" TO \"event_id_tmp\"")
   -> 0.0008s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name(:event_id_convert_to_bigint)
   -> 0.0000s
-- quote_column_name(:event_id)
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id_convert_to_bigint\" TO \"event_id\"")
   -> 0.0008s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name("event_id_tmp")
   -> 0.0000s
-- quote_column_name(:event_id_convert_to_bigint)
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id_tmp\" TO \"event_id_convert_to_bigint\"")
   -> 0.0007s
-- change_column_default("push_event_payloads", :event_id, nil)
   -> 0.0022s
-- change_column_default("push_event_payloads", :event_id_convert_to_bigint, 0)
   -> 0.0020s
-- execute("ALTER TABLE push_event_payloads DROP CONSTRAINT push_event_payloads_pkey")
   -> 0.0008s
-- rename_index("push_event_payloads", "index_push_event_payloads_on_event_id_convert_to_bigint", "push_event_payloads_pkey")
   -> 0.0008s
-- execute("ALTER TABLE push_event_payloads ADD CONSTRAINT push_event_payloads_pkey PRIMARY KEY USING INDEX push_event_payloads_pkey")
   -> 0.0007s
-- remove_foreign_key("push_event_payloads", {:name=>"fk_36c74129da"})
   -> 0.0033s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name("fk_a5e47ac4c5")
   -> 0.0000s
-- quote_column_name("fk_36c74129da")
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\"\nRENAME CONSTRAINT \"fk_a5e47ac4c5\" TO \"fk_36c74129da\"\n")
   -> 0.0006s
== 20210622041846 FinalizePushEventPayloadsBigintConversion: migrated (0.0544s)

⬇ Down

$ bundle exec rails db:migrate:down VERSION=20210622041846
== 20210622041846 FinalizePushEventPayloadsBigintConversion: reverting ========
-- transaction_open?()
   -> 0.0000s
-- index_exists?("push_event_payloads", :event_id_convert_to_bigint, {:unique=>true, :name=>"index_push_event_payloads_on_event_id_convert_to_bigint", :algorithm=>:concurrently})
   -> 0.0026s
-- transaction_open?()
   -> 0.0000s
-- foreign_keys("push_event_payloads")
   -> 0.0036s
-- execute("SET statement_timeout TO 0")
   -> 0.0006s
-- execute("ALTER TABLE push_event_payloads VALIDATE CONSTRAINT fk_a5e47ac4c5;")
   -> 0.0007s
-- execute("RESET ALL")
   -> 0.0006s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name(:event_id)
   -> 0.0000s
-- quote_column_name("event_id_tmp")
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id\" TO \"event_id_tmp\"")
   -> 0.0008s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name(:event_id_convert_to_bigint)
   -> 0.0000s
-- quote_column_name(:event_id)
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id_convert_to_bigint\" TO \"event_id\"")
   -> 0.0006s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name("event_id_tmp")
   -> 0.0000s
-- quote_column_name(:event_id_convert_to_bigint)
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\" RENAME COLUMN \"event_id_tmp\" TO \"event_id_convert_to_bigint\"")
   -> 0.0007s
-- change_column_default("push_event_payloads", :event_id, nil)
   -> 0.0026s
-- change_column_default("push_event_payloads", :event_id_convert_to_bigint, 0)
   -> 0.0032s
-- execute("ALTER TABLE push_event_payloads DROP CONSTRAINT push_event_payloads_pkey")
   -> 0.0012s
-- rename_index("push_event_payloads", "index_push_event_payloads_on_event_id_convert_to_bigint", "push_event_payloads_pkey")
   -> 0.0011s
-- execute("ALTER TABLE push_event_payloads ADD CONSTRAINT push_event_payloads_pkey PRIMARY KEY USING INDEX push_event_payloads_pkey")
   -> 0.0015s
-- remove_foreign_key("push_event_payloads", {:name=>"fk_36c74129da"})
   -> 0.0049s
-- quote_table_name("push_event_payloads")
   -> 0.0000s
-- quote_column_name("fk_a5e47ac4c5")
   -> 0.0000s
-- quote_column_name("fk_36c74129da")
   -> 0.0000s
-- execute("ALTER TABLE \"push_event_payloads\"\nRENAME CONSTRAINT \"fk_a5e47ac4c5\" TO \"fk_36c74129da\"\n")
   -> 0.0009s
== 20210622041846 FinalizePushEventPayloadsBigintConversion: reverted (0.0395s)

Database schema changes

Before this MR

gitlabhq_development=# \d push_event_payloads
                         Table "public.push_event_payloads"
           Column           |         Type          | Collation | Nullable | Default 
----------------------------+-----------------------+-----------+----------+---------
 commit_count               | bigint                |           | not null | 
 event_id                   | integer               |           | not null | 
 action                     | smallint              |           | not null | 
 ref_type                   | smallint              |           | not null | 
 commit_from                | bytea                 |           |          | 
 commit_to                  | bytea                 |           |          | 
 ref                        | text                  |           |          | 
 commit_title               | character varying(70) |           |          | 
 ref_count                  | integer               |           |          | 
 event_id_convert_to_bigint | bigint                |           | not null | 0
Indexes:
    "push_event_payloads_pkey" PRIMARY KEY, btree (event_id)
Foreign-key constraints:
    "fk_36c74129da" FOREIGN KEY (event_id) REFERENCES events(id) ON DELETE CASCADE
Triggers:
    trigger_07c94931164e BEFORE INSERT OR UPDATE ON push_event_payloads FOR EACH ROW EXECUTE PROCEDURE trigger_07c94931164e()

After this MR

gitlabhq_development=# \d push_event_payloads
                         Table "public.push_event_payloads"
           Column           |         Type          | Collation | Nullable | Default 
----------------------------+-----------------------+-----------+----------+---------
 commit_count               | bigint                |           | not null | 
 event_id_convert_to_bigint | integer               |           | not null | 0
 action                     | smallint              |           | not null | 
 ref_type                   | smallint              |           | not null | 
 commit_from                | bytea                 |           |          | 
 commit_to                  | bytea                 |           |          | 
 ref                        | text                  |           |          | 
 commit_title               | character varying(70) |           |          | 
 ref_count                  | integer               |           |          | 
 event_id                   | bigint                |           | not null | 
Indexes:
    "push_event_payloads_pkey" PRIMARY KEY, btree (event_id)
Foreign-key constraints:
    "fk_36c74129da" FOREIGN KEY (event_id) REFERENCES events(id) ON DELETE CASCADE
Triggers:
    trigger_07c94931164e BEFORE INSERT OR UPDATE ON push_event_payloads FOR EACH ROW EXECUTE FUNCTION trigger_07c94931164e()

Does this MR meet the acceptance criteria?

Conformity

I have included changelog trailers, or none are needed. (Does this MR need a changelog?)
I have added/updated documentation, or it's not needed. (Is documentation required?)
I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
I have self-reviewed this MR per code review guidelines.
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
I have followed the style guides.
This change is backwards compatible across updates, or this does not apply.

Availability and Testing

I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
I have tested this MR in all supported browsers, or it's not needed.
I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

Label as security and @ mention @gitlab-com/gl-security/appsec
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
Security reports checked/validated by a reviewer from the AppSec team

Related to #288005 (closed)

Edited Jun 28, 2021 by Heinrich Lee Yu

Finalize conversion to bigint for push_event_payloads