Clickhouse - use exclusive lease for migrations and data ingestion
In Clickhouse - basic migrations support (#428124 - closed) we introduced the basic migrations support
We should also ensure that:
-
only one process can execute the migrations in a given time -
Ideally, data ingestion workers are paused during the migration execution. Before starting migration: -
Pause ClickHouse workers based on presence of ClickHouse::MigrationSupport::Migrator::LEASE_KEY
in Redis. -
Create RegisterClickHouseWorker
middleware that registers ClickHouse workers in aclick_house:workers:active_workers
Redis list with an appropriate TTL score while the worker is active. -
Create ClickHouseWorker
concern that enablesClickHouseRegisterWorker
. -
Ensure that Redis list is empty before starting migration, i.e. ZRANGEBYSCORE click_house:workers:active_workers {now} +inf
returns empty.
-
We can try to achieve this by using already existing mechanism of exclusive locks
Edited by Pedro Pombeiro