Skip to content

Adds a new E2E test of Compliance pipelines

Mark Lapierre requested to merge ml-add-compliance-pipeline-e2e-test into master

Description of the test

Test design: gitlab-org/quality/testcases#4016 (closed)
Test case: https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/413715 (I'll let the automation complete it)

  • New API-only E2E test. Frontend tests provide coverage for the UI.
  • Change group runner resource to use the group runners API, not the user-specific API.
  • Adds Resource::Project#wait_for_pipeline to allow waiting for and fetching a pipeline with specific attributes.
  • Remove some redundancy from Pipeline::Resource method names.

Sample results

Passing result in Allure reports

Test execution logs 
[Jun 01 2023 04:24:51 UTC (QA Tests)] INFO  -- Starting test: Govern Compliance pipeline when a compliance framework has a compliance pipeline runs that pipeline in a different project that has the compliance framework assigned
[Jun 01 2023 04:24:51 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-5' via api in 0.14 seconds
[Jun 01 2023 04:24:52 UTC (QA Tests)] DEBUG -- <QA::Resource::Group> Attribute :full_path has both API response `gitlab-qa-sandbox-group-5/compliance-pipeline-xp060bnd` and a block. API response will be picked. Block will be ignored.
[Jun 01 2023 04:24:52 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-5/compliance-pipeline-xp060bnd' via api in 0.77 seconds
[Jun 01 2023 04:24:52 UTC (QA Tests)] INFO  -- Runner api_get exception caught and handled: No value was computed for id of QA::Resource::GroupRunner.
[Jun 01 2023 04:24:52 UTC (QA Tests)] INFO  -- Executing: `docker pull registry.gitlab.com/gitlab-org/gitlab-runner:alpine`
[Jun 01 2023 04:24:55 UTC (QA Tests)] DEBUG -- Command output:
alpine: Pulling from gitlab-org/gitlab-runner
0ce1dd7918a4: Pulling fs layer
b7ebeea25943: Pulling fs layer
1c14bcbcaa52: Pulling fs layer
0ce1dd7918a4: Download complete
1c14bcbcaa52: Download complete
0ce1dd7918a4: Pull complete
b7ebeea25943: Verifying Checksum
b7ebeea25943: Download complete
b7ebeea25943: Pull complete
1c14bcbcaa52: Pull complete
Digest: sha256:11d0dc3681ef475e848ec857babb29c4cd6a31d39cfa26e3fba41ffd5536d2d5
Status: Downloaded newer image for registry.gitlab.com/gitlab-org/gitlab-runner:alpine
registry.gitlab.com/gitlab-org/gitlab-runner:alpine
[Jun 01 2023 04:24:55 UTC (QA Tests)] INFO  -- Executing: `docker network inspect test`
[Jun 01 2023 04:24:55 UTC (QA Tests)] DEBUG -- Command output:
[
    {
        "Name": "test",
        "Id": "bb5c71cd956ecbaf2b6b0cc95a16c929fe672212fcb083b0b2d40c3e128a718e",
        "Created": "2023-06-01T04:16:54.607084049Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "b870e69db7cc5da28a28470e24fc1f0c4f6b41346753b5acc6874338fc2cbc78": {
                "Name": "gitlab-ee-fad77215",
                "EndpointID": "6f38e5cd68284481d7e7df0fdb6cb28e8d7e2ad74043ebe004c44ee74084350a",
                "MacAddress": "02:42:ac:13:00:02",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            },
            "c7fd063ac27fcfe275fc7a040b6eb35dad880ad1ce30c14df91604a2f0c260c7": {
                "Name": "gitlab-ee-qa-0e9f4f6c",
                "EndpointID": "d3d0eea06668b2807e37b6dec6b55e7436d0d8b5f5942b71038ac13b87c4b4c7",
                "MacAddress": "02:42:ac:13:00:03",
                "IPv4Address": "172.19.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]
[Jun 01 2023 04:24:55 UTC (QA Tests)] INFO  -- Executing: `docker run -d --rm --network test --name runner-for-compliance-pipeline-xp060bnd  --privileged registry.gitlab.com/gitlab-org/gitlab-runner:alpine   && docker exec --detach runner-for-compliance-pipeline-xp060bnd sh -c "printf 'concurrent = 1\ncheck_interval = 0\n\n[session_server]\n  session_timeout = 1800' > /etc/gitlab-runner/config.toml && gitlab-runner register   --non-interactive --name runner-for-compliance-pipeline-xp060bnd --url http://gitlab-ee-fad77215.test --registration-token **** --tag-list runner-for-compliance-pipeline-xp060bnd --executor shell && gitlab-runner run" `
[Jun 01 2023 04:24:58 UTC (QA Tests)] DEBUG -- Command output:
775bf90393456e462536934733bc45cc1c7b0b7c59d2bbbf2a325a53d9fda213
[Jun 01 2023 04:24:58 UTC (QA Tests)] INFO  -- Executing: `docker logs runner-for-compliance-pipeline-xp060bnd`
[Jun 01 2023 04:24:58 UTC (QA Tests)] INFO  -- Executing: `docker network inspect test`
[Jun 01 2023 04:24:58 UTC (QA Tests)] DEBUG -- Command output:
[
    {
        "Name": "test",
        "Id": "bb5c71cd956ecbaf2b6b0cc95a16c929fe672212fcb083b0b2d40c3e128a718e",
        "Created": "2023-06-01T04:16:54.607084049Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "775bf90393456e462536934733bc45cc1c7b0b7c59d2bbbf2a325a53d9fda213": {
                "Name": "runner-for-compliance-pipeline-xp060bnd",
                "EndpointID": "d2782fcf1306d371f8a60ebbde32d805eecbd87f47d0f00c4fa72ba1d97fa3bf",
                "MacAddress": "02:42:ac:13:00:04",
                "IPv4Address": "172.19.0.4/16",
                "IPv6Address": ""
            },
            "b870e69db7cc5da28a28470e24fc1f0c4f6b41346753b5acc6874338fc2cbc78": {
                "Name": "gitlab-ee-fad77215",
                "EndpointID": "6f38e5cd68284481d7e7df0fdb6cb28e8d7e2ad74043ebe004c44ee74084350a",
                "MacAddress": "02:42:ac:13:00:02",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            },
            "c7fd063ac27fcfe275fc7a040b6eb35dad880ad1ce30c14df91604a2f0c260c7": {
                "Name": "gitlab-ee-qa-0e9f4f6c",
                "EndpointID": "d3d0eea06668b2807e37b6dec6b55e7436d0d8b5f5942b71038ac13b87c4b4c7",
                "MacAddress": "02:42:ac:13:00:03",
                "IPv4Address": "172.19.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]
[Jun 01 2023 04:24:59 UTC (QA Tests)] DEBUG -- Retrying action with: max_duration: 60; sleep_interval: 1; raise_on_failure: true; retry_on_exception: false
[Jun 01 2023 04:25:00 UTC (QA Tests)] DEBUG -- ended retry
[Jun 01 2023 04:25:00 UTC (QA Tests)] DEBUG -- Details of the runner fetched using tag_list: {:id=>1, :description=>"runner-for-compliance-pipeline-xp060bnd", :ip_address=>"172.19.0.4", :active=>true, :paused=>false, :is_shared=>false, :runner_type=>"group_type", :name=>"gitlab-runner", :online=>true, :status=>"online"}
[Jun 01 2023 04:25:00 UTC (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::GroupRunner with name 'runner-for-compliance-pipeline-xp060bnd' via api in 7.99 seconds
[Jun 01 2023 04:25:00 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/compliance-pipeline-xp060bnd/pipeline-project-7c84fdefedfbf3b2' via api in 0.47 seconds
[Jun 01 2023 04:25:02 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Repository::Commit via api in 1.41 seconds
[Jun 01 2023 04:25:03 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-5/compliance-pipeline-xp060bnd/compliant-project-43fc5d2aa286a690' via api in 1.45 seconds
[Jun 01 2023 04:25:04 UTC (QA Tests)] INFO  -- ==> Built a QA::EE::Resource::ComplianceFramework with name 'Praesentium est delectus necessitatibus.' via api in 0.11 seconds
[Jun 01 2023 04:25:05 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Repository::Commit via api in 0.45 seconds
[Jun 01 2023 04:25:06 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::Repository::Commit via api in 0.44 seconds
[Jun 01 2023 04:25:06 UTC (QA Tests)] INFO  -- ==> Built a QA::Resource::MergeRequest with iid '1' via api in 1.63 seconds
[Jun 01 2023 04:25:06 UTC (QA Tests)] INFO  -- Running eventually matcher with 'be_truthy' operator with: '{:sleep_interval=>0.5}' arguments
[Jun 01 2023 04:25:06 UTC (QA Tests)] DEBUG -- evaluating expectation, attempt: 1
[Jun 01 2023 04:25:07 UTC (QA Tests)] DEBUG -- Retrying action with: max_duration: 60; sleep_interval: 1; raise_on_failure: true; retry_on_exception: false
[Jun 01 2023 04:25:14 UTC (QA Tests)] DEBUG -- ended retry

How to set up and validate locally

From the qa directory:

bundle install
export WEBDRIVER_HEADLESS=false # If you'd like to watch the test in action
export QA_GITLAB_URL="http://gdk.test:3000" # Only needed if GDK is not running on http://127.0.0.1:3000
bundle exec rspec qa/specs/features/ee/api/10_govern/compliance_pipeline.rb

Checklist

  • Confirm the test has a testcase: tag linking to an existing test case in the test case project.
  • [-] Note if the test is intended to run in specific scenarios. If a scenario is new, add a link to the MR that adds the new scenario.
  • Follow the end-to-end tests style guide and best practices.
  • Use the appropriate RSpec metadata tag(s).
  • Most resources will be cleaned up via the general cleanup task. Check that is successful, or ensure resources are cleaned up in the test:
    • [-] New resources have api_get_path and api_delete_path implemented if possible.
    • If any resource cannot be deleted in the general delete task, make sure it is ignored.
    • If any resource cannot be deleted in the general delete task, remove it in the test (e.g., in an after block).
  • Ensure that no transient bugs are hidden accidentally due to the usage of waits and reloads.
  • Verify the tags to ensure it runs on the desired test environments.
  • [-] If this MR has a dependency on another MR, such as a GitLab QA MR, specify the order in which the MRs should be merged.
  • [-] (If applicable) Create a follow-up issue to document the special setup necessary to run the test: ISSUE_LINK
  • [-] If the test requires an admin's personal access token, ensure that the test passes on your local environment with and without the GITLAB_QA_ADMIN_ACCESS_TOKEN provided.
Edited by Mark Lapierre

Merge request reports