Data migration strategy for moving `audit_events` from PG to Clickhouse
Problem Statement
Sync Audit Events from PG to ClickHouse.
Implementation Plan
- Write a cron job worker which enqueues all the partitions of audit events into a worker.
- Write a worker that takes input from a partition of the audit event table and syncs its data into ClickHouse.
- The syncing should happen by making a CSV and should use batching.
- The worker should close the CSV and upload at a certain time limit.
- And when the job re-runs it should run from where the previous worker left.
Edited by Harsimar Sandhu