Add sharding key for events table
Sharding keys need to be set for the tables:
- events
This involves choosing one of the following, based on the intended behaviour of the table:
-
The table is not cell-local
- Set
gitlab_schema
togitlab_main_clusterwide
.
- Set
-
The table is cell-local and requires a sharding key
- Set
gitlab_schema
togitlab_main_cell
- Add a
sharding_key
ordesired_sharding_key
configuration. If the configuration is known but the chosen key doesn't yet meet not-null and foreign key requirements, you can add an exception toallowed_to_be_missing_not_null
orallowed_to_be_missing_foreign_key
to get the pipeline passing. Please link to a follow-up issue in a code comment next to the exception. - You may also need to set
allow_cross_joins
,allow_cross_transactions
andallow_cross_foreign_keys
if changing the schema causes pipeline failures. Seedb/docs/epics.yml
for an example.
- Set
-
The table is cell-local and does not require a sharding key
- Set
gitlab_schema
togitlab_main_cell
and - Set
exempt_from_sharding
totrue
.
- Set
Documentation
- Choosing either the gitlab_main_cell or gitlab_main_clusterwide schema
- Defining a sharding key for all cell-local tables
- Defining a desired_sharding_key to automatically backfill a sharding_key
Proposal
Based on #462801 (comment 2015672699).
-
%17.4:
-
Remove index_events_on_group_id_partial
as it is a redundant toindex_events_on_group_id_and_id
andindex_events_for_group_activity
. (!164709 (merged)) -
Add column personal_namespace_id bigint
:-
GitLab.com (!165127 (merged)) -
Self-managed (!165591 (merged))
-
-
Create index on (personal_namespace_id) WHERE personal_namespace_id IS NOT NULL
:-
GitLab.com -
async (!165127 (merged)) - [-]
sync(not needed, done in !165591 (merged))
-
-
Self-managed (!165591 (merged))
-
-
Add FK (personal_namespace_id) REFERENCES namespaces(id) ON DELETE CASCADE
:-
GitLab.com -
Self-managed (!165591 (merged))
-
-
-
%17.5 (required stop):
-
Update app to populate personal_namespace_id
when noproject_id
orgroup_id
is present. Create NOT VALID check constraint to ensure that at least one ofproject_id
,group_id
, andpersonal_namespace_id
is not null (!164979 (merged))
-
-
%17.6:
-
Back-fill records where project_id is null and group_id is null
. (!166563 (merged))
-
-
%17.9:
-
Finalize the backfill background migration -
Validate the check_events_sharding_key_is_not_null
constraint. -
Define composite sharding key on project_id
,group_id
, andpersonal_namespace_id
.
-
Edited by Krasimir Angelov