Move creation of security policy project to background worker

Why are we doing this work

When a security policy is created for the first time, we create the security policy project by calling the securityPolicyProjectCreate mutation. Since this mutation creates the project in the web request itself, if the policy project is created for a large group, some database queries might timeout.

Relevant links

Feedback Issue (Internal)

Non-functional requirements

Documentation:
Feature flag:
Performance:
Testing:

Implementation plan

backend Create a new graphql mutation to create the security policy project and the policy YAML asynchronously in a single mutation. The mutation should invoke a new worker (Security::CreateSecurityPolicyProjectWorker)
backend Create a graphql subscription (Subscriptions::SecurityPolicyProjectCreated)
backend Create a graphql trigger (security_policy_project_created) and call the trigger from the new worker (Security::CreateSecurityPolicyProjectWorker). The trigger should return the IID of the MR created in the policy project.
frontend update create_scan_execution_policy.mutation.graphql to a new, generic name
frontend create a new subscription under queries called security_policy_project_created.subscription.graphql
frontend update the save policy utils method to use the new mutation and query
frontend update all editor_components to not create an assigned security policy project and use the new method
- I am not sure how the subscription will work not being in the Vue component; we may have to move the method out of the utils file, which would be fine since it is only one API call instead of three, but we could reuse the method still by potentially moving it to editor_layout or a named/scoped slot or, as a last resort, a mixin

Partial frontend patch

diff --git a/ee/app/assets/javascripts/security_orchestration/components/policy_editor/constants.js b/ee/app/assets/javascripts/security_orchestration/components/policy_editor/constants.js
index c85bec142800..ddf1f8c33cc7 100644
--- a/ee/app/assets/javascripts/security_orchestration/components/policy_editor/constants.js
+++ b/ee/app/assets/javascripts/security_orchestration/components/policy_editor/constants.js
@@ -43,8 +43,6 @@ export const DELETE_MODAL_CONFIG = {
 
 export const MATCH_ON_INCLUSION_LICENSE = 'match_on_inclusion_license';
 
-export const DEFAULT_MR_TITLE = s__('SecurityOrchestration|Update scan policies');
-
 export const POLICY_RUN_TIME_MESSAGE = s__(
   'SecurityOrchestration|Policy changes may take some time to be applied.',
 );
diff --git a/ee/app/assets/javascripts/security_orchestration/components/policy_editor/pipeline_execution/editor_component.vue b/ee/app/assets/javascripts/security_orchestration/components/policy_editor/pipeline_execution/editor_component.vue
index 85414e618ee2..d1247cea2d0d 100644
--- a/ee/app/assets/javascripts/security_orchestration/components/policy_editor/pipeline_execution/editor_component.vue
+++ b/ee/app/assets/javascripts/security_orchestration/components/policy_editor/pipeline_execution/editor_component.vue
@@ -10,7 +10,7 @@ import {
   PARSING_ERROR_MESSAGE,
   SECURITY_POLICY_ACTIONS,
 } from '../constants';
-import { assignSecurityPolicyProject, modifyPolicy } from '../utils';
+import { modifyPolicy } from '../utils';
 import EditorLayout from '../editor_layout.vue';
 import DimDisableContainer from '../dim_disable_container.vue';
 import ActionSection from './action/action_section.vue';
@@ -97,13 +97,6 @@ export default {
     changeEditorMode(mode) {
       this.mode = mode;
     },
-    async getSecurityPolicyProject() {
-      if (!this.newlyCreatedPolicyProject && !this.assignedPolicyProject.fullPath) {
-        this.newlyCreatedPolicyProject = await assignSecurityPolicyProject(this.namespacePath);
-      }
-
-      return this.newlyCreatedPolicyProject || this.assignedPolicyProject;
-    },
     handleError(error) {
       if (error.message.toLowerCase().includes('graphql')) {
         this.$emit('error', GRAPHQL_ERROR_MESSAGE);
@@ -122,7 +115,7 @@ export default {
       this.setLoadingFlag(action, true);
 
       try {
-        const assignedPolicyProject = await this.getSecurityPolicyProject();
+        const { assignedPolicyProject } = this;
         const mergeRequest = await modifyPolicy({
           action,
           assignedPolicyProject,
diff --git a/ee/app/assets/javascripts/security_orchestration/components/policy_editor/scan_execution/editor_component.vue b/ee/app/assets/javascripts/security_orchestration/components/policy_editor/scan_execution/editor_component.vue
index a4336cd4e55d..3088cf787993 100644
--- a/ee/app/assets/javascripts/security_orchestration/components/policy_editor/scan_execution/editor_component.vue
+++ b/ee/app/assets/javascripts/security_orchestration/components/policy_editor/scan_execution/editor_component.vue
@@ -16,7 +16,7 @@ import {
 } from '../constants';
 import EditorLayout from '../editor_layout.vue';
 import DimDisableContainer from '../dim_disable_container.vue';
-import { assignSecurityPolicyProject, modifyPolicy } from '../utils';
+import { modifyPolicy } from '../utils';
 import RuleSection from './rule/rule_section.vue';
 import ScanAction from './action/scan_action.vue';
 import ActionSection from './action/action_section.vue';
@@ -174,13 +174,6 @@ export default {
       this.policy[property] = value;
       this.updateYamlEditorValue(this.policy);
     },
-    async getSecurityPolicyProject() {
-      if (!this.newlyCreatedPolicyProject && !this.assignedPolicyProject.fullPath) {
-        this.newlyCreatedPolicyProject = await assignSecurityPolicyProject(this.namespacePath);
-      }
-
-      return this.newlyCreatedPolicyProject || this.assignedPolicyProject;
-    },
     async handleModifyPolicy(act) {
       const action =
         act ||
@@ -192,7 +185,7 @@ export default {
       this.setLoadingFlag(action, true);
 
       try {
-        const assignedPolicyProject = await this.getSecurityPolicyProject();
+        const { assignedPolicyProject } = this;
         const mergeRequest = await modifyPolicy({
           action,
           assignedPolicyProject,
diff --git a/ee/app/assets/javascripts/security_orchestration/components/policy_editor/scan_result/editor_component.vue b/ee/app/assets/javascripts/security_orchestration/components/policy_editor/scan_result/editor_component.vue
index 0347971a1396..d8028ff293ad 100644
--- a/ee/app/assets/javascripts/security_orchestration/components/policy_editor/scan_result/editor_component.vue
+++ b/ee/app/assets/javascripts/security_orchestration/components/policy_editor/scan_result/editor_component.vue
@@ -20,7 +20,7 @@ import {
   MAX_ALLOWED_RULES_LENGTH,
 } from '../constants';
 import EditorLayout from '../editor_layout.vue';
-import { assignSecurityPolicyProject, modifyPolicy } from '../utils';
+import { modifyPolicy } from '../utils';
 import DimDisableContainer from '../dim_disable_container.vue';
 import ScanFilterSelector from '../scan_filter_selector.vue';
 import SettingsSection from './settings/settings_section.vue';
@@ -386,13 +386,6 @@ export default {
     handleParsingError() {
       this.hasParsingError = true;
     },
-    async getSecurityPolicyProject() {
-      if (!this.newlyCreatedPolicyProject && !this.assignedPolicyProject.fullPath) {
-        this.newlyCreatedPolicyProject = await assignSecurityPolicyProject(this.namespacePath);
-      }
-
-      return this.newlyCreatedPolicyProject || this.assignedPolicyProject;
-    },
     async handleModifyPolicy(act) {
       const action = act || this.policyActionName;
 
@@ -400,7 +393,7 @@ export default {
       this.setLoadingFlag(action, true);
 
       try {
-        const assignedPolicyProject = await this.getSecurityPolicyProject();
+        const { assignedPolicyProject } = this;
         const mergeRequest = await modifyPolicy({
           action,
           assignedPolicyProject,
diff --git a/ee/app/assets/javascripts/security_orchestration/components/policy_editor/utils.js b/ee/app/assets/javascripts/security_orchestration/components/policy_editor/utils.js
index b2774d53bb49..0be9c277fcfa 100644
--- a/ee/app/assets/javascripts/security_orchestration/components/policy_editor/utils.js
+++ b/ee/app/assets/javascripts/security_orchestration/components/policy_editor/utils.js
@@ -1,15 +1,13 @@
 import { intersection, uniqBy, uniqueId } from 'lodash';
 import { isValidCron } from 'cron-validator';
 import { sprintf, s__ } from '~/locale';
-import createPolicyProject from 'ee/security_orchestration/graphql/mutations/create_policy_project.mutation.graphql';
-import createScanExecutionPolicy from 'ee/security_orchestration/graphql/mutations/create_scan_execution_policy.mutation.graphql';
+import createPolicy from 'ee/security_orchestration/graphql/mutations/create_policy.mutation.graphql';
+import securityPolicyProjectCreated from 'ee/security_orchestration/graphql/queries/security_policy_project_created.subscription.graphql';
 import { gqClient } from 'ee/security_orchestration/utils';
-import createMergeRequestMutation from '~/graphql_shared/mutations/create_merge_request.mutation.graphql';
 
 import {
   BRANCHES_KEY,
   BRANCH_TYPE_KEY,
-  DEFAULT_MR_TITLE,
   PRIMARY_POLICY_KEYS,
   RULE_MODE_SCANNERS,
   SECURITY_POLICY_ACTIONS,
@@ -30,34 +28,6 @@ const checkForErrors = ({ errors, validationErrors }) => {
   }
 };
 
-/**
- * Creates a merge request for the changes to the policy file
- * @param {Object} payload contains the path to the parent project, the branch to merge on the project, and the branch to merge into
- * @returns {Object} contains the id of the merge request and any errors
- */
-const createMergeRequest = async ({ projectPath, sourceBranch, targetBranch }) => {
-  const input = {
-    projectPath,
-    sourceBranch,
-    targetBranch,
-    title: DEFAULT_MR_TITLE,
-  };
-
-  const {
-    data: {
-      mergeRequestCreate: {
-        mergeRequest: { iid: id },
-        errors,
-      },
-    },
-  } = await gqClient.mutate({
-    mutation: createMergeRequestMutation,
-    variables: { input },
-  });
-
-  return { id, errors };
-};
-
 /**
  * Creates a new security policy on the security policy project's policy file
  * @param {Object} payload contains the path to the project and the policy yaml value
@@ -74,13 +44,21 @@ const updatePolicy = async ({
       scanExecutionPolicyCommit: { branch, errors, validationErrors },
     },
   } = await gqClient.mutate({
-    mutation: createScanExecutionPolicy,
+    mutation: createPolicy,
     variables: {
       mode: action,
       name,
       fullPath: namespacePath,
       policyYaml: yamlEditorValue,
     },
+    subscribeToMore: {
+      document: securityPolicyProjectCreated,
+      variables() {
+        return {
+          fullPath: namespacePath,
+        };
+      },
+    },
   });
 
   return { branch, errors, validationErrors };
@@ -98,18 +76,12 @@ export const modifyPolicy = async ({
   namespacePath,
   yamlEditorValue,
 }) => {
-  const newPolicyCommitBranch = await updatePolicy({
+  const mergeRequest = await updatePolicy({
     action,
     name,
     namespacePath,
     yamlEditorValue,
-  });
-
-  checkForErrors(newPolicyCommitBranch);
-
-  const mergeRequest = await createMergeRequest({
     projectPath: assignedPolicyProject.fullPath,
-    sourceBranch: newPolicyCommitBranch.branch,
     targetBranch: assignedPolicyProject.branch,
   });
 
@@ -118,28 +90,6 @@ export const modifyPolicy = async ({
   return mergeRequest;
 };
 
-/**
- * Creates a new security policy project and assigns it to the current project
- * @param {String} fullPath
- * @returns {Object} contains the new security policy project and any errors
- */
-export const assignSecurityPolicyProject = async (fullPath) => {
-  const {
-    data: {
-      securityPolicyProjectCreate: { project, errors },
-    },
-  } = await gqClient.mutate({
-    mutation: createPolicyProject,
-    variables: {
-      fullPath,
-    },
-  });
-
-  checkForErrors({ errors });
-
-  return { ...project, branch: project?.branch?.rootRef, errors };
-};
-
 /**
  * Converts scanner strings to title case
  * @param {Array} scanners (e.g. 'container_scanning', `dast`, etcetera)
diff --git a/ee/app/assets/javascripts/security_orchestration/graphql/mutations/create_policy_project.mutation.graphql b/ee/app/assets/javascripts/security_orchestration/graphql/mutations/create_policy_project.mutation.graphql
deleted file mode 100644
index c84de9c315f4..000000000000
--- a/ee/app/assets/javascripts/security_orchestration/graphql/mutations/create_policy_project.mutation.graphql
+++ /dev/null
@@ -1,12 +0,0 @@
-mutation createPolicyProject($fullPath: String!) {
-  securityPolicyProjectCreate(input: { fullPath: $fullPath }) {
-    project {
-      fullPath
-      id
-      branch: repository {
-        rootRef
-      }
-    }
-    errors
-  }
-}
diff --git a/ee/app/assets/javascripts/security_orchestration/graphql/mutations/create_scan_execution_policy.mutation.graphql b/ee/app/assets/javascripts/security_orchestration/graphql/mutations/create_scan_execution_policy.mutation.graphql
deleted file mode 100644
index 727f94cb8c39..000000000000
--- a/ee/app/assets/javascripts/security_orchestration/graphql/mutations/create_scan_execution_policy.mutation.graphql
+++ /dev/null
@@ -1,18 +0,0 @@
-mutation updatePolicy(
-  $fullPath: String!
-  $mode: MutationOperationMode = APPEND
-  $name: String!
-  $policyYaml: String!
-) {
-  scanExecutionPolicyCommit(
-    input: { name: $name, fullPath: $fullPath, operationMode: $mode, policyYaml: $policyYaml }
-  ) {
-    branch
-    errors
-    validationErrors {
-      field
-      message
-      title
-    }
-  }
-}

Verification steps

Edited Sep 13, 2024 by Alexander Turinske