Skip to content

Helper method for streaming audit event destinations to create legacy destinations

Andrew Jung requested to merge 503242/add_helpers_to_create_audit_event_destination_models into master

What does this MR do and why?

This MR adds a helper module for our audit events destinations graphql create APIs:

- When a legacy audit event destination is created -> it creates a stream destination model and associates them via their stream_destination_id and legacy_destination_ref columns

  • When a new streaming model is created, then a legacy audit event destination model is created in the same fashion.
  • This is all under the feature flag: audit_events_external_destination_streamer_consolidation_refactor via http://localhost:3000/rails/features/ or rails console.
  • ❗ Once the stream destinations tables (new) are all migrated and fully in use, all of this code will be removed along with the legacy API routes.

Going to separate Create / Updates / Delete into separate MRs

References

Please include cross links to any resources that are relevant to this MR. This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

MR acceptance checklist

Models:

  • HTTP: AuditEvents::InstanceExternalAuditEventDestination, AuditEvents::ExternalAuditEventDestination
  • AWS: AuditEvents::Instance::AmazonS3Configuration, AuditEvents::AmazonS3Configuration
  • GCP: AuditEvents::Instance::GoogleCloudLoggingConfiguration, AuditEvents:GoogleCloudLoggingConfiguration

New models:

AuditEvents::Group::ExternalStreamingDestination & AuditEvents::Instance::ExternalStreamingDestination which have a category column of http, gcp, or aws; and a json column called config that stores category specific attributes.

FYI: There is no UI component to create the 'new' streaming models yet, as such, we can only use the GraphQL API.

How to set up and validate locally

To test:

  • Checkout this branch locally, update migrations, and open gdk -- Ensure you have an Ultimate/EE license

  • Enable the feature flag audit_events_external_destination_streamer_consolidation_refactor

  • For legacy -> stream destination: you can use the UI following the documentation here to do so:

  • To test the reverse flow, open http://localhost:3000/-/graphql-explorer on your gdk instance.

EX:

mutation CreateInstanceAuditEventStreamingDestination {
  instanceAuditEventStreamingDestinationsCreate(
    input: {
      config: {
        googleProjectIdName: "google-project-id-name",
        clientEmail: "clientEmail@example.com",
        logIdName: "gcp-log-id-name"
      }, 
      category: "gcp", 
      name: "Test Local Destination API GCP 12345"
      secretToken:"SecretToken12345!" # required
    }
  ) {
    externalAuditEventDestination {
      id
      name
      category
      config
      secretToken
    }
    errors
  }
}
mutation CreateGroupAuditEventStreamingDestination {
  groupAuditEventStreamingDestinationsCreate(
    input: {
      groupPath:"twitter",
      config: {
        googleProjectIdName: "google-project-id-name",
        clientEmail: "clientEmail@example.com",
        logIdName: "gcp-log-id-name"
      }, 
      category: "gcp", 
      name: "Test Local Destination API GCP 12345"
      secretToken:"SecretToken12345!" # required
    }
  ) {
    externalAuditEventDestination {
      id
      name
      group {
        fullName
        fullPath
      }
      category
      config
      secretToken
    }
    errors
  }
}

Configs are different for each category, and they can be found from their json schemas here:

Example demo where I create a new streaming model through API -> it shows up as a legacy destination as well: Screen_Recording_2024-12-04_at_2.00.12_PM

Example demo where I create a legacy instance destination for HTTP and it shows up as a stream destination via API: Screen_Recording_2024-12-04_at_2.05.40_PM

Edited by Andrew Jung

Merge request reports