Skip to content

Allow configuring desired_sharding_key in db/docs

Dylan Griffith requested to merge 429164-desired-sharding-key into master

What does this MR do and why?

Allow configuring desired_sharding_key in db/docs

As part of our Cells effort we are going to need to backfill a sharding_key to many tables. The sharding_key can be any foreign key that references projects, namespaces or organizations. You can read more about this at https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-sharding-key-for-all-cell-local-tables .The enforcement of having a sharding_key for new tables was already implemented in !136682 (merged) .

Additionally we have recently shown examples of how we plan to backfill a sharding_key to a table that is missing it in !136800 (merged) .

This MR extends this work by allowing a very minimal way to declare in the db/docs file a desired_sharding_key a strategy to backfill a new column that will eventually become the sharding_key. An example has been given in this MR which looks like:

desired_sharding_key:
  project_id:
    references: projects
    backfill_via:
      parent:
        foreign_key: scanner_id
        table: vulnerability_scanners
        sharding_key: project_id
        belongs_to: scanner

To understand best how this YML data will be used you can map it onto the merge request we created manually in !136800 (merged) . The idea will be to automatically create this. The content of the YML specifies the parent table and it's sharding_key to backfill from in the batched background migration. It also specifies a belongs_to relation which will be added to the model to automatically populate the sharding_key in the before_save.

The idea behind this is to save developers the effort of creating MRs which add the new column and writing a batched background migration to backfill this column and all the followup work related to that. Instead we will be building automation that uses this data in the YML file to automatically create all the merge requests for creating the column and backfilling it.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #429164 (closed)

Merge request reports