Skip to content

Add basic migration support for ClickHouse

What does this MR do and why?

NOTE: This MR looks very large, but more than half of the changes are renames or new fixtures.

Currently, the ClickHouse tables are setup by ClickHouseTestRunner#ensure_schema, by running db/click_house/main/*.sql files against the database.

This MR adds basic support for ClickHouse migrations:

  • first commit: support classes inspired by the Active Record equivalent (https://github.com/rails/rails/blob/main/activerecord/lib/active_record/migration.rb and the clickhouse-activerecord gem). I've chosen to not rely on the equivalent ActiveRecord classes since:
    • the amount of code saved is not significant.
    • we'd need to catch any ActiveRecord exceptions and raise equivalent ClickHouse exceptions, otherwise the exception class might mislead users.
    • we would be exposed to potential bugs due to subtle changes in the base ActiveRecord classes.
  • second commit: Rake targets (rake gitlab:clickhouse:migrate and rake gitlab:clickhouse:rollback, including support for VERSION, VERBOSE, and SCOPE environment variables).
  • third commit: moves the existing db/click_house/main/*.sql files to db/click_house/migrate/*.rb files.

Closes #428124 (closed)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Command Before After
rake gitlab:clickhouse:migrate image image
rake gitlab:clickhouse:rollback VERSION=20230808140217 image image

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  • Setup a development ClickHouse server, see https://docs.gitlab.com/ee/development/database/clickhouse/clickhouse_within_gitlab.html

  • Test several migration commands, such as:

    • rake gitlab:clickhouse:migrate
    • rake gitlab:clickhouse:rollback

    You can also pass specific VERSION values.

    At any point, you can run SELECT * FROM schema_migrations FINAL on the ClickHouse client (clickhouse client -d gitlab_clickhouse_development) to see the internal state of the migrations.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Pedro Pombeiro

Merge request reports

Loading