Rake task gitlab:db:lock_writes should include table partitions
What does this MR do and why?
Addressing issue: #388308 (closed)
Background:
- Since we introduced the support for multiple databases which is the current setup on GitLab.com (production) and (staging), we have added a rake task to lock the writes on the
gitlab_main
tables on the databaseci
, andgitlab_ci
tables on themain
database. With this MR, we are also including the attached partitions of the tables to into the locking mechanism. With this we also make sure that partitions are also not written to directly from the application. Because that's not supposed to happen anyway.
Some other small issues to address in this issue.
- Renaming
let
blocks and changinginclude_examples
toit_behaves_like
. Because using the samelet
with parameterized variables causes a caching problem in thelet
. With this it returns the wrong variable. See the 2nd warning here. - Not using the same
instance
object as the default one for tests. BecauseTablesLocker
callslock_writes
andunlock_writes
in each run. But we need to test against a particular table only. Eitherlock
orunlock
.
How to set up and validate locally
Steps to validate locally
-
Make sure you are running in decomposed database mode
-
Create test partitions on both
main
andci
export PSQL_STATEMENT="CREATE TABLE gitlab_partitions_dynamic.test_partition PARTITION OF security_findings FOR VALUES IN (0)"
export MAIN_DATABASE=$(cat config/database.yml|yq .development.main.database)
export CI_DATABASE=$(cat config/database.yml|yq .development.ci.database)
gdk psql -d $MAIN_DATABASE -c $PSQL_STATEMENT
gdk psql -d $CI_DATABASE -c $PSQL_STATEMENT
- Run this rake task, to lock the writes on the
gitlab_ci
tables on themain
, andgitlab_main
on theci
database
VERBOSE=1 ./bin/rake gitlab:db:lock_writes
- Try to write/delete from the partitions on both
main
andci
PSQL_STATEMENT="DELETE FROM gitlab_partitions_dynamic.test_partition"
It should not fail on main
gdk psql -d $MAIN_DATABASE -c $PSQL_STATEMENT
But it should fail on ci
gdk psql -d $CI_DATABASE -c $PSQL_STATEMENT
- Clean up
- Unlock the writes on all the tables
VERBOSE=1 ./bin/rake gitlab:db:unlock_writes
- Drop the partitions
PSQL_STATEMENT="DROP TABLE gitlab_partitions_dynamic.test_partition"
gdk psql -d $MAIN_DATABASE -c $PSQL_STATEMENT
gdk psql -d $CI_DATABASE -c $PSQL_STATEMENT
Related Followup
In this related follow up issue: #391638 (closed) we are going to include tests in a similar fashion for the detached partitions
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #388308 (closed)
Edited by Omar Qunsul