Skip to content

Remove records without group from webhooks

What does this MR do?

After adding a NOT VALID foreign key from web_hooks to groups to prevent invalid records with the foreign key !57735 (merged).

This is the data migration to fix the existing records before we can validate the foreign key.

Count number of invalid records

It looks like we don't have any invalid records in the database (this is a query to check the number of invalid records, we are not executing this query in the migration).

https://postgres.ai/console/gitlab/gitlab-production-tunnel/sessions/3052/commands/10061

SELECT "web_hooks"."id"
FROM "web_hooks"
WHERE "web_hooks"."type" = 'GroupHook'
  AND "web_hooks"."group_id" IS NOT NULL
  AND "web_hooks"."group_id" NOT IN
    (SELECT "namespaces"."id"
     FROM "namespaces");

Queries

each_batch generates two queries to get the start id and end id for the relation (the second query is the same with an OFFSET of the batch size (1_000 in this case).

SELECT "web_hooks"."id"
FROM "web_hooks"
WHERE "web_hooks"."type" = 'GroupHook'
ORDER BY "web_hooks"."id" ASC
LIMIT 1
Time: 669.741 ms
  - planning: 0.175 ms
  - execution: 669.566 ms
    - I/O read: 639.779 ms
    - I/O write: N/A

https://postgres.ai/console/gitlab/gitlab-production-tunnel/sessions/3099/commands/10245

and then we have the DELETE statement, I guess this will use a BETWEEN in production:

DELETE
FROM "web_hooks"
WHERE "web_hooks"."type" = 'GroupHook'
  AND "web_hooks"."id" BETWEEN 548001 AND 549000
  AND "web_hooks"."group_id" NOT IN
    (SELECT "namespaces"."id"
     FROM "namespaces")
Time: 15.457 ms
  - planning: 3.207 ms
  - execution: 12.250 ms
    - I/O read: 6.112 ms
    - I/O write: N/A

https://postgres.ai/console/gitlab/gitlab-production-tunnel/sessions/3099/commands/10249

Migration

$ rails db:migrate
== 20210330091751 RemoveRecordsWithoutGroupFromWebhooksTable: migrating =======
== 20210330091751 RemoveRecordsWithoutGroupFromWebhooksTable: migrated (0.0230s)
$ rails db:migrate:down VERSION=20210330091751
== 20210330091751 RemoveRecordsWithoutGroupFromWebhooksTable: reverting =======
== 20210330091751 RemoveRecordsWithoutGroupFromWebhooksTable: reverted (0.0000s)

Related to #202423 (closed)

Edited by Arturo Herrero

Merge request reports

Loading