Drop foreign keys prior to dropping individual detached partitions
What does this MR do and why?
This MR modifies the detached partition dropper so that it drops the foreign keys of the detached table individually, and using the lock_retries
strategy to reduce the impact of the locking.
This will reduce the locks taken when the partition is actually dropped, since each foreign key drop requires an access exclusive lock on both involved tables.
The incorrect behavior being fixed here was pointed out in #337155 (comment 701918411), and the overarching issue for partition pruning is #332199 (closed)
Screenshots or screen recordings
These are strongly recommended to assist reviewers and reduce the time to merge your change.
How to set up and validate locally
To validate this locally, we'll need to set up some old partitions that can be dropped.
- Change the line
partitioned_by :created_at, strategy: :monthly, retain_for: 3.months
in theWebHookLog
model toretain_for: 6.months
so that we generate some extra partitions. - Enable the relevant feature flags:
Feature.enable(:partition_pruning)
,Feature.enable(:drop_detached_partitions)
- Close your console, kill spring if it's running, and
gdk stop rails
. - Run
bin/rails db:reset
. This will wipe your local database, but it's necessary to convince the partitioning code to create partitions for the past. - Run
gdk psql -c '\d+ web_hook_logs'
to view the partitions of the table. You should see partitions back to2021-04-01
(6 months ago). - Change the
retain_for
line inWebHookLog
back to3.months
- In a new console, sync the partitions of the
web_hook_logs
table:Gitlab::Database::Partitioning::PartitionManager.new(WebHookLog).sync_partitions
- Verify that you only see 3 months of past partitions attached (plus the default 5 into the future) when you run
gdk psql -c '\d+ web_hook_logs'
.
We now have 3 tables that used to be partitions of the web_hook_logs table, and we want to test that they're dropped correctly.
- First, update their drop time to be in the past. In a rails console:
Postgresql::DetachedPartition.update_all(drop_after: 1.day.ago)
(By default the drop time is next week) - Finally, in a rails console, trigger the dropping for these partitions:
Gitlab::Database::Partitioning::DetachedPartitionDropper.new.perform
. You'll see the 3 tablesgitlab_partitions_dynamic.web_hook_logs_202104
,_202105
, and_202106
dropped, with their foreign keys to theweb_hooks
table individually dropped before the tables are dropped.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.