Add interface to create analytics stack for project
What does this MR do and why?
- Adds an
initialize_analytics_stack
method toEE::Project
that sets up the connection between a GitLab project, Jitsu, Clickhouse and Cube as defined in the groupproduct analytics issue: Discussion: Project -> Jitsu Mapping (gitlab-org/analytics-section/product-analytics/general-discussion#2 - closed) - Adds a bunch of application settings required to perform those actions, including a password for jitsu which is stored as an encrypted variable.
- Pushes the process itself in to a background worker.
Database Review
Migrate
main: == 20220818125332 AddJitsuTrackingColumnsToApplicationSettings: migrating =====
main: -- add_column(:application_settings, :jitsu_host, :text)
main: -> 0.0028s
main: -- add_column(:application_settings, :jitsu_project_xid, :text)
main: -> 0.0005s
main: -- add_column(:application_settings, :clickhouse_connection_string, :text)
main: -> 0.0004s
main: -- add_column(:application_settings, :jitsu_administrator_email, :text)
main: -> 0.0004s
main: -- add_column(:application_settings, :encrypted_jitsu_administrator_password, :binary)
main: -> 0.0004s
main: -- add_column(:application_settings, :encrypted_jitsu_administrator_password_iv, :binary)
main: -> 0.0004s
main: == 20220818125332 AddJitsuTrackingColumnsToApplicationSettings: migrated (0.0057s)
main: == 20220818125703 AddJitsuTrackingColumnsToApplicationSettingsTextLimits: migrating
main: -- transaction_open?()
main: -> 0.0000s
main: -- current_schema()
main: -> 0.0011s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nADD CONSTRAINT check_dea8792229\nCHECK ( char_length(jitsu_host) <= 255 )\nNOT VALID;\n")
main: -> 0.0010s
main: -- current_schema()
main: -> 0.0002s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0001s
main: -- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT check_dea8792229;")
main: -> 0.0006s
main: -- execute("RESET statement_timeout")
main: -> 0.0002s
main: -- transaction_open?()
main: -> 0.0000s
main: -- current_schema()
main: -> 0.0001s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nADD CONSTRAINT check_fc732c181e\nCHECK ( char_length(jitsu_project_xid) <= 255 )\nNOT VALID;\n")
main: -> 0.0005s
main: -- current_schema()
main: -> 0.0001s
main: -- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT check_fc732c181e;")
main: -> 0.0005s
main: -- transaction_open?()
main: -> 0.0000s
main: -- current_schema()
main: -> 0.0001s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nADD CONSTRAINT check_d4865d70f3\nCHECK ( char_length(clickhouse_connection_string) <= 255 )\nNOT VALID;\n")
main: -> 0.0005s
main: -- current_schema()
main: -> 0.0001s
main: -- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT check_d4865d70f3;")
main: -> 0.0005s
main: -- transaction_open?()
main: -> 0.0000s
main: -- current_schema()
main: -> 0.0001s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nADD CONSTRAINT check_ec3ca9aa8d\nCHECK ( char_length(jitsu_administrator_email) <= 255 )\nNOT VALID;\n")
main: -> 0.0005s
main: -- current_schema()
main: -> 0.0001s
main: -- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT check_ec3ca9aa8d;")
main: -> 0.0005s
main: == 20220818125703 AddJitsuTrackingColumnsToApplicationSettingsTextLimits: migrated (0.0220s)
Rollback
main: == 20220818125703 AddJitsuTrackingColumnsToApplicationSettingsTextLimits: reverting
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nDROP CONSTRAINT IF EXISTS check_dea8792229\n")
main: -> 0.0010s
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nDROP CONSTRAINT IF EXISTS check_fc732c181e\n")
main: -> 0.0006s
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nDROP CONSTRAINT IF EXISTS check_d4865d70f3\n")
main: -> 0.0005s
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nDROP CONSTRAINT IF EXISTS check_ec3ca9aa8d\n")
main: -> 0.0005s
main: == 20220818125703 AddJitsuTrackingColumnsToApplicationSettingsTextLimits: reverted (0.0136s)
main: == 20220818125332 AddJitsuTrackingColumnsToApplicationSettings: reverting =====
main: -- remove_column(:application_settings, :encrypted_jitsu_administrator_password_iv, :binary)
main: -> 0.0021s
main: -- remove_column(:application_settings, :encrypted_jitsu_administrator_password, :binary)
main: -> 0.0005s
main: -- remove_column(:application_settings, :jitsu_administrator_email, :text)
main: -> 0.0003s
main: -- remove_column(:application_settings, :clickhouse_connection_string, :text)
main: -> 0.0003s
main: -- remove_column(:application_settings, :jitsu_project_xid, :text)
main: -> 0.0003s
main: -- remove_column(:application_settings, :jitsu_host, :text)
main: -> 0.0003s
main: == 20220818125332 AddJitsuTrackingColumnsToApplicationSettings: reverted (0.0057s)
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
How to set up and validate locally
-
Set up the devkit and start it running.
docker-compose up
-
In this branch, run
rails c
to open the rails console -
Enable the feature flag
Feature.enable(:jitsu_connection_proof_of_concept)
-
Set the required application settings:
The Jitsu attributes are set up when first configuring your jitsu instance.
ApplicationSetting.current.update(jitsu_project_xid: 'XXXXXXXXXX', jitsu_host: 'http://localhost:8000', clickhouse_connection_string: 'http://test:test@clickhouse:8123', jitsu_administrator_email: 'XXXXXXX', jitsu_administrator_password: 'XXXXXXX') reload!
-
Find a project on the console:
a = Project.find(PROJECT_ID)
-
Run
ProductAnalytics::InitializeStackService.new(container: a).execute
-
Ensure that a sidekiq job was enqueued.
-
In the Jitsu configurator, ensure that a new API key (with the project ID) has been created, along with a new associated connection to a clickhouse destination.
👇
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.