Allow scan execution policies to create pipelines

Why are we doing this work

As someone accountable for an organization's security posture, I want to be confident that security scans are running on all of my repositories, even if those repositories do not have CI/CD configured.

Today, scan execution policies only ensure that the security scan configuration is present in pipelines that run. Something still needs to ensure that those pipelines are triggered. To ensure that those pipelines are triggered, new pipelines should be created if a scan execution policy applies to the repository. This should happen even if Auto DevOps is disabled and no .gitlab-ci.yml is present.

Relevant links

Epic: &6880 (closed)

Draft MR: !121087 (merged)

Non-functional requirements

Documentation:
Feature flag:
Performance:
Testing:

Implementation plan

Overview

Use the Gitlab::Ci::Config::SecurityOrchestrationPolicies::Processor class to merge the security scan configuration if a scan execution policy applies to the repository.

Steps

Add a new feature flag :scan_execution_policy_pipelines
Add a new SecurityPolicy Source class in ee/lib/gitlab/ci/project_config/

# frozen_string_literal: true

module Gitlab
  module Ci
    class ProjectConfig
      class SecurityPolicy < Gitlab::Ci::ProjectConfig::Source
        def content
          return {} unless custom_content.present?

          custom_content
        end
        strong_memoize_attr :content

        def source
          :security_policy_source
        end
      end
    end
  end
end

Add the new source as a valid config source in app/models/concerns/enums/ci/pipeline.rb

===================================================================
diff --git a/app/models/concerns/enums/ci/pipeline.rb b/app/models/concerns/enums/ci/pipeline.rb
--- a/app/models/concerns/enums/ci/pipeline.rb	(revision 962cd79163143412164d556430d6823abee57e41)
+++ b/app/models/concerns/enums/ci/pipeline.rb	(date 1681231651125)
@@ -82,7 +82,8 @@
           external_project_source: 5,
           bridge_source: 6,
           parameter_source: 7,
-          compliance_source: 8
+          compliance_source: 8,
+          security_policy_source: 9
         }
       end
     end

Add a unit test to ensure that ::Gitlab::Ci::Config::SecurityOrchestrationPolicies::Processor accepts empty config files

context 'when the config is empty' do
      let(:config) { {} }

      it 'does not include scan-policies stage' do
        expect(subject[:stages]).to eq(%w[.pre build test deploy .post dast])
      end

      it 'extends config with additional jobs' do
        expect(subject.keys).to include(expected_jobs)
        expect(subject.values).to include(expected_configuration)
      end
    end

In lib/gitlab/ci/pipeline/chain/config/content, change the logic of perform! method.

If security policies and the the :scan_execution_policy_pipelines feature flag are enabled: merge the security policies with the pipeline config if it exists, or an empty config if it doesn't.

Since we are updating every pipeline configuration, maybe it's worth adding a fallback mechanism into ::Gitlab::Ci::Config::SecurityOrchestrationPolicies::Processor to return the original configuration if an error occurs during the merge. Otherwise, we couldn't prevent a pipeline from running.

===================================================================
diff --git a/lib/gitlab/ci/pipeline/chain/config/content.rb b/lib/gitlab/ci/pipeline/chain/config/content.rb
--- a/lib/gitlab/ci/pipeline/chain/config/content.rb	(revision 3750856bcbde0399f001df8bdcc086843261a126)
+++ b/lib/gitlab/ci/pipeline/chain/config/content.rb	(date 1680729782454)
@@ -10,14 +10,14 @@
             include ::Gitlab::Utils::StrongMemoize
 
             def perform!
               if pipeline_config&.exists?
-                @pipeline.build_pipeline_config(content: pipeline_config.content)
-                @command.config_content = pipeline_config.content
-                @pipeline.config_source = pipeline_config.source
-                @command.pipeline_config = pipeline_config
+                config = pipeline_config
+                config = merge_security_policies(config) if security_policies_enabled?
+                build_pipeline_config(config)
+              elsif security_policies_enabled?
+                security_policies_config = {}
+                security_policies_config = merge_security_policies(security_policies_config)
+                build_pipeline_config(security_policies_config)
               else
                 error('Missing CI config file')
               end
@@ -29,6 +29,24 @@
 
             private
 
+            def build_pipeline_config(pipeline_config)
+              @pipeline.build_pipeline_config(content: pipeline_config.content)
+              @command.config_content = pipeline_config.content
+              @pipeline.config_source = pipeline_config.source
+              @command.pipeline_config = pipeline_config
+            end
+
+            def security_policies_enabled?
+              Feature.enabled?(:scan_execution_policy_pipeline, project) && project&.feature_available?(:security_orchestration_policies)
+            end
+
+            def merge_security_policies(config)
+              ::Gitlab::Ci::Config::SecurityOrchestrationPolicies::Processor.new(config,
+                project,
+                @pipeline.ref,
+                pipeline_config.source).perform
+            end
+
             def pipeline_config
               strong_memoize(:pipeline_config) do
                 ::Gitlab::Ci::ProjectConfig.new(

Remove process_security_orchestration_policy_includes from ee/lib/ee/gitlab/ci/config_ee.rb

Verification steps

When new changes are pushed to a repository, new pipelines should be created if a scan execution policy applies to the repository. This should happen even if Auto DevOps is disabled and no .gitlab-ci.yml is present.

Enable the feature flag

rails c

Feature.enabled?(:scan_execution_policy_pipelines)

Auto DevOps disabled and .gitlab-ci.yml not present

Create a new project with a readme file.
Go to /-/settings/ci_cd
Click in Expand in the Auto DevOps section
Disable the option Default to Auto DevOps pipeline
Create a new Scan execution policy
Update the readme file using the web ide and push the changes to the main branch
Go to '-/pipelines' page
Check if a pipeline was created
Check if the pipeline contains the jobs defined by the Scan Execution policy

.gitlab-ci.yml not present and Auto DevOps enabled

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

Edited May 29, 2023 by Marcos Rocha