Implement custom events data for ClickHouse sync storage
Summary
We have DB table sync strategy to ClickHouse. However sometimes we don't really need to sync entire table and we can store custom pre-calculated data for further usage. So we need a way to save data to CH with buffer. Also we need a way to cleanup data from CH on demand.
Suggestion
Implement "ClickHouse::DataBuffer" which buffers custom data that should be written to CH. Then add a worker which runs every X minutes and dumps buffered data to CH storage. Classes should be reusable for any CH table.
TODO: Add suggestion for data cleanup mechanism.
Example:
WriteBuffer.save(:code_suggestion_requested, {timestamp: 123, user_id: 456}) # saves to buffer. doesn't actually write to CH. params are table name + map of columns and values
Worker.perform # dump whatever is in the buffer to CH.
We have lib/click_house/write_buffer.rb as a starting point.
Scope
Scope of this issue is only "write" part to clickhouse. Clickhouse cleanup should be handled in #457702
Current state
- We have a code suggestions worker. We can make it reusable as described in this thread
- We have CodeSuggestionsUsage model which can serve as an example of custom ClickHouse events model.
- We have AiTracking module which is specific to CodeSuggestions and should be generalized to work for other features too.
Edited by Pavel Shutsin