Consolidation of tables for streaming audit events to various external destinations
Problem Statement
For streaming audit events to external destinations, currently we have following tables for storing destinations and related data in form of event filters or headers:
- Custom http destinations:
- For Group level audit events:
audit_events_external_audit_event_destinations
audit_events_streaming_headers
audit_events_streaming_event_type_filters
- For instance level audit events:
audit_events_instance_external_audit_event_destinations
instance_audit_events_streaming_headers
audit_events_streaming_instance_event_type_filters
- For Group level audit events:
- GCP config:
- For group level:
audit_events_google_cloud_logging_configurations
- For instance level:
audit_events_instance_google_cloud_logging_configurations
- For group level:
- AWS config:
- For group level:
audit_events_amazon_s3_configurations
- For group level:
- Event type filters:
- For each type of config we have 2 tables, 1 for group level and 1 for instance level, for storing audit event filters.
- Namespace filters:
- For each type of config we have 2 tables, 1 for group level and 1 for instance level, for storing namespace filters.
- Streaming headers for custom http destinations:
- There are 2 tables, one for group level headers and another for instance level headers.
And these tables will keep on increasing as we keep on introducing new integrations for streaming audit events. And even a minor change required in all these destinations require too many MRs and code changes.
Most of the code and structure for these tables, their models, graphql apis are quite similar and it is a repetitive task every time we have to add a new type of streaming destination.
The major difference between different kind of destinations is some kind of config that needs to stored for each of them and the way data will be streamed as each one will have different kind of urls and api signatures.
Listing down schemas of several tables here:
-
Group level custom http destinations:
CREATE TABLE audit_events_amazon_s3_configurations (
id bigint NOT NULL,
created_at timestamp with time zone NOT NULL,
updated_at timestamp with time zone NOT NULL,
namespace_id bigint NOT NULL,
access_key_xid text NOT NULL,
name text NOT NULL,
bucket_name text NOT NULL,
aws_region text NOT NULL,
encrypted_secret_access_key bytea NOT NULL,
encrypted_secret_access_key_iv bytea NOT NULL,
CONSTRAINT check_3a41f4ea06 CHECK ((
_char_length
_(bucket_name) <= 63)),
CONSTRAINT check_72b5aaa71b CHECK ((
_char_length
_(aws_region) <= 50)),
CONSTRAINT check_90505816db CHECK ((
_char_length
_(name) <= 72)),
CONSTRAINT check_ec46f06e01 CHECK ((
_char_length
_(access_key_xid) <= 128))
);
-
Group level gcp config:
CREATE TABLE audit_events_google_cloud_logging_configurations (
id bigint NOT NULL,
namespace_id bigint NOT NULL,
created_at timestamp with time zone NOT NULL,
updated_at timestamp with time zone NOT NULL,
google_project_id_name text NOT NULL,
client_email text NOT NULL,
log_id_name text DEFAULT 'audit_events'::text,
encrypted_private_key bytea NOT NULL,
encrypted_private_key_iv bytea NOT NULL,
name text NOT NULL,
CONSTRAINT check_0ef835c61e CHECK ((
_char_length
_(client_email) <= 254)),
CONSTRAINT check_55783c7c19 CHECK ((
_char_length
_(google_project_id_name) <= 30)),
CONSTRAINT check_898a76b005 CHECK ((
_char_length
_(log_id_name) <= 511)),
CONSTRAINT check_cdf6883cd6 CHECK ((
_char_length
_(name) <= 72))
);
-
Group level aws config:
CREATE TABLE audit_events_amazon_s3_configurations (
id bigint NOT NULL,
created_at timestamp with time zone NOT NULL,
updated_at timestamp with time zone NOT NULL,
namespace_id bigint NOT NULL,
access_key_xid text NOT NULL,
name text NOT NULL,
bucket_name text NOT NULL,
aws_region text NOT NULL,
encrypted_secret_access_key bytea NOT NULL,
encrypted_secret_access_key_iv bytea NOT NULL,
CONSTRAINT check_3a41f4ea06 CHECK ((
_char_length
_(bucket_name) <= 63)),
CONSTRAINT check_72b5aaa71b CHECK ((
_char_length
_(aws_region) <= 50)),
CONSTRAINT check_90505816db CHECK ((
_char_length
_(name) <= 72)),
CONSTRAINT check_ec46f06e01 CHECK ((
_char_length
_(access_key_xid) <= 128))
);
-
Instance level custom http:
CREATE TABLE audit_events_instance_external_audit_event_destinations (
id bigint NOT NULL,
created_at timestamp with time zone NOT NULL,
updated_at timestamp with time zone NOT NULL,
destination_url text NOT NULL,
encrypted_verification_token bytea NOT NULL,
encrypted_verification_token_iv bytea NOT NULL,
name text NOT NULL,
CONSTRAINT check_433fbb3305 CHECK ((
_char_length
_(name) <= 72)),
CONSTRAINT check_4dc67167ce CHECK ((
_char_length
_(destination_url) <= 255))
);
-
Group level event type filters for custom http destinations, similar is there for instance:
CREATE TABLE audit_events_streaming_event_type_filters ( id bigint NOT NULL, created_at timestamp with time zone NOT NULL, updated_at timestamp with time zone NOT NULL, external_audit_event_destination_id bigint NOT NULL, audit_event_type text NOT NULL, CONSTRAINT check_d20c8e5a51 CHECK ((char_length(audit_event_type) <= 255)) );
-
Group level namespace filters, similar for instance without association to group:
CREATE TABLE audit_events_streaming_group_namespace_filters ( id bigint NOT NULL, created_at timestamp with time zone NOT NULL, updated_at timestamp with time zone NOT NULL, external_streaming_destination_id bigint NOT NULL, namespace_id bigint NOT NULL );
-
Streaming headers for group level custom http destinations:
CREATE TABLE audit_events_streaming_headers ( id bigint NOT NULL, created_at timestamp with time zone NOT NULL, updated_at timestamp with time zone NOT NULL, external_audit_event_destination_id bigint NOT NULL, key text NOT NULL, value text NOT NULL, active boolean DEFAULT true NOT NULL, CONSTRAINT check_53c3152034 CHECK ((char_length(key) <= 255)), CONSTRAINT check_ac213cca22 CHECK ((char_length(value) <= 255)) );
Proposal
We are trying to propose a consolidated way of storing all the configs for different type of destinations in a generic fashion, rather than duplicating most of the code every time we are adding a new type of streaming destination, which will make it easier and faster to add new streaming destination integrations. So following is the list of proposed tables:
-
For storing configurations of all type of destinations for top-level groups we are going to create a table
audit_events_group_external_streaming_destinations
which will have- a jsonb field named config, which will store all the config of the destination, config can be
destination_url
for custom http destinations, bucket name and other information for aws config and similarly for other kind of destinations. We are not going to create headers table and will be storing headers information in the config object for custom http destinations. - a string field named type, which could be http, aws, gcp and so on. This will help us determine to which type of destination this config belongs too.
create_table :audit_events_group_external_streaming_destinations do |t| t.timestamps_with_timezone null: false t.text :name, null: false, limit: 72 t.text :type, null: false, limit: 20 t.references :group, null: false, index: { name: NAMESPACE_INDEX_NAME }, foreign_key: { to_table: :namespaces, on_delete: :cascade } t.jsonb :config, null: false t.binary :encrypted_secret_token, null: false t.binary :encrypted_secret_token_iv, null: false end
- a jsonb field named config, which will store all the config of the destination, config can be
-
Similar table will be created for instance level destinations of any type, only without a reference to a group.
create_table :audit_events_instance_external_streaming_destinations do |t| t.timestamps_with_timezone null: false t.text :name, null: false, limit: 72 t.text :type, null: false, limit: 20 t.jsonb :config, null: false t.binary :encrypted_secret_token, null: false t.binary :encrypted_secret_token_iv, null: false end
-
A single group level event type filters which will have a 1:many association with group level external destination table
create_table :audit_events_streaming_group_audit_event_type_filters do |t| t.timestamps_with_timezone null: false t.references :external_streaming_destination, null: false, index: false, foreign_key: { to_table: 'audit_events_group_external_streaming_destinations', on_delete: :cascade } t.text :audit_event_type, null: false, limit: 255 t.index [:external_streaming_destination_id, :audit_event_type], unique: true, name: UNIQ_INDEX_NAME end
-
A single instance level event type filters which will have a 1:many association with instance level external destination table
create_table :audit_events_streaming_instance_audit_event_type_filters do |t| t.timestamps_with_timezone null: false t.references :external_streaming_destination, null: false, index: false, foreign_key: { to_table: 'audit_events_instance_external_streaming_destinations', on_delete: :cascade } t.text :audit_event_type, null: false, limit: 255 t.index [:external_streaming_destination_id, :audit_event_type], unique: true, name: UNIQ_INDEX_NAME end
-
A single group level namespace filters table which will have a 1:many association with group level external destination table
create_table :audit_events_streaming_group_namespace_filters do |t| t.timestamps_with_timezone null: false t.references :external_streaming_destination, null: false, index: { name: DESTINATION_INDEX_NAME }, foreign_key: { to_table: 'audit_events_group_external_streaming_destinations', on_delete: :cascade } t.references :namespace, null: false, index: { name: NAMESPACE_INDEX_NAME, unique: true }, foreign_key: { on_delete: :cascade } end
-
A single instance level namespace filters table which will have a 1:many association with instance level external destination table
create_table :audit_events_streaming_instance_namespace_filters do |t| t.timestamps_with_timezone null: false t.references :external_streaming_destination, null: false, index: { name: DESTINATION_INDEX_NAME }, foreign_key: { to_table: 'audit_events_instance_external_streaming_destinations', on_delete: :cascade } t.references :namespace, null: false, index: { name: NAMESPACE_INDEX_NAME, unique: true }, foreign_key: { on_delete: :cascade } end
Notes:
- We could have used polymorphic association for instance and group level event type and namespace filters or could have created a common config table for group and instance and associate it with their respective destination tables. But there is a problem with that, as per https://docs.gitlab.com/ee/development/database/multiple_databases.html#gitlab-schema, namespace and application settings will be stored in different schemas and it might raise a problem for us.
- Now we will require only 2 tables each for storing destination config, event type and namespace filters.
- With this new schema, the number of tables will not grow with newer integrations and will remain constant, so this brings no. of tables from O(n) to O(1), which will reduce our efforts and delivery time.
- You can refer to POC MR !140362 (closed) for all information regarding this.
- As discussed in #427187 (comment 1701485204), we can keep an overall maximum limit on the no. of destinations a namespace or instance can have irrespective of the type of destinations.
Models:
We have created a POC MR !140362 (closed) for showing which models we would have to create.
GraphQL APIs:
- We can create two kind of apis, one for groups and another for instance.
- So in total we would require following apis:
- 4 CRUD APIs for group level destinations.
- 4 CRUD APIs for instance level destinations.
- 4 CRUD APIs for group level event type filters.
- 4 CRUD APIs for instance level event type filters.
- 4 CRUD APIs for group level namespace filters.
- 4 CRUD APIs for instance level namespace filters.
- Similar to the tables and models, this structure will bring down the total no. of APIs that we need to create for each type of destination. Again bringing down API efforts from O(n) to O(1).
- We can deprecate old APIs in 17.0.
- Check file changes for
ee/app/graphql/mutations/audit_events/group/external_streaming_destinations/create.rb
in !140362 (diffs) for a sample create API for destination, delete, list are going to somewhat similar to what we have currently and the apis for event type filters and namespace filters are also going to be similar to the existing ones.
WHY?
- The most important aspect is to reduce time of development and integration of different streaming destinations in future. Currently we are spending atleast 2-3 milestones for introducing a new streaming destination on group level and instance level. All the refactoring discussed here may take 2-3 milestones but it will pave the path for integrating new types of streaming destinations with less than 1 milestone, saving us time in long term. We can do similar efforts on the frontend too and reduce the overall development cycle.
- With less no. of code changes, the possibility of something going wrong will also decrease.
Efforts and timeline
For efforts you can checkout POC MR for this !140362 (closed), which also give you a rough estimate of the time it is going to take.
Listing down approximate time required for this, including review:
- For creating group and instance level destination config tables and models - 2 weeks.
- Creating event type filters and namespace filters tables and their models, this is dependent on step 1 - 2 weeks
- APIs for group and instance level destinations, this is dependent on step 1 - 2 weeks
- APIs for event filters and namespace filters, this is dependent on step 2 - 2 weeks
- Enabling streaming of audit events to new destinations - 1 week
- Miscellaneous things such as concerns for destination validation, streaming strategies - 1 week
Approximately this whole process should take 3 milestones, some work can be done in parallel.
Efforts on adding new type of destinations
- We would have to add the new destination type to the enum for supported destination types.
- Add validator for the destination and add streaming destination strategy.
- This would require 1-2 weeks of efforts.