Periodical events sync worker for ClickHouse
What does this MR do and why?
This MR implements a worker that periodically syncs events
data to ClickHouse. The data syncing would be SaaS only for now.
How does it work:
- Job is enqueued by cron.
- Check if the job is enabled (CH configured and FF is on).
- Acquire distributed lock using Redis.
- Load the cursor from CH so we can continue the syncing from a particular
events.id
value. (ClickHouse::SyncCursor
class) - Build an enumerator that yields
events
usingEachBatch
. - The enumerator is passed to a Gzip CSV writer.
- When reaching the threshold (5k rows), the CSV file is closed and uploaded to CH.
- Start a new loop and a new csv file.
- When time is up or no more data is left stop the processing and update the cursor with the last inserted
events.id
.
PG queries
The worker will mostly read data from replicas.
- Batching query: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/21760/commands/70634
- Load batch query: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/21760/commands/70635 (seeing 300-1500ms execution times from PG.ai)
How to set up and validate locally
- Ensure that ClickHouse is set up: https://docs.gitlab.com/ee/development/database/clickhouse/clickhouse_within_gitlab.html#configure-your-rails-application
- Enable the feature flag:
Feature.enable(:event_sync_worker_for_click_house)
- Invoke the worker:
ClickHouse::EventsSyncWorker.new.perform
- Verify that the events table on CH is populated:
ClickHouse::Client.select("select * from events", :main)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #414937 (closed)
Edited by Adam Hegyi