Skip to content

Fix same stage needs behavior when chaining with failure or manual

Furkan Ayhan requested to merge 358110-same-stage-needs-chain into master

What does this MR do and why?

When retrying a failed job or playing a manual job, we mark the subsequent jobs as "created". However, we could not handle this for chains of needs. In this MR, we fix this behavior. The details of this bug and the fix is in the "Screenshots or screen recordings" section below.

These changes are behind the feature flag ci_requeue_with_dag_object_hierarchy (#373148 (closed)) .

Related to #358110 (closed)

Screenshots or screen recordings

Example 1

build:
  script: exit $(($RANDOM % 2))

test:
  script: exit 0
  needs: [build]

deploy:
  script: exit 0
  needs: [test]

Before enabling the feature flag:

  • The "deploy" job stays in the "skipped" state.
Click to expand

2

After enabling the feature flag:

Click to expand

3

Example 2

job1:
  script: exit 0
  when: manual

job2:
  script: exit 0
  needs: [job1]

job3:
  script: exit 0
  when: manual
  needs: [job2]

job4:
  script: exit 0
  needs: [job3]

Before enabling the feature flag:

  • The "job3" and "job4" jobs stay in the "skipped" state.
Click to expand

1

After enabling the feature flag:

Click to expand

4

Database

Before

-- Example pipeline: https://gitlab.com/gitlab-org/gitlab/-/pipelines/634207789
-- Example job: "graphql-schema-dump", https://gitlab.com/gitlab-org/gitlab/-/jobs/2995760625
-- cold: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/11996/commands/42549

SELECT "ci_builds".*
FROM "ci_builds"
WHERE
  "ci_builds"."type" IN ('Ci::Processable', 'Ci::Build', 'Ci::Bridge')
  AND "ci_builds"."commit_id" = 634207789
  AND "ci_builds"."status" IN ('success') -- this is actually skipped but I made it success for test purposes
  AND (
    stage_idx > 1
    OR
    "ci_builds"."scheduling_type" = 1
    AND
    (EXISTS (SELECT 1 FROM "ci_build_needs" WHERE (ci_builds.id = ci_build_needs.build_id) AND "ci_build_needs"."name" = 'graphql-schema-dump'))
  )
ORDER BY "ci_builds"."stage_idx" ASC;

After

❗ This query is not the actual query. Please see: !98315 (merged) ❗

-- Example pipeline: https://gitlab.com/gitlab-org/gitlab/-/pipelines/634207789
-- Example job: "graphql-schema-dump", https://gitlab.com/gitlab-org/gitlab/-/jobs/2995760625
-- cold: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/11996/commands/42552

SELECT "ci_builds".*
FROM (
  (
    WITH RECURSIVE "base_and_descendants" AS (
      (
        SELECT "ci_builds".*
        FROM "ci_builds"
        WHERE
          "ci_builds"."type" IN ('Ci::Processable', 'Ci::Build', 'Ci::Bridge')
          AND
          "ci_builds"."id" = 2995760625
      )
      UNION
      (
        SELECT "ci_builds".*
        FROM "ci_builds", "base_and_descendants", "ci_build_needs"
        WHERE
          "ci_builds"."type" IN ('Ci::Processable', 'Ci::Build', 'Ci::Bridge')
          AND
          "ci_build_needs"."build_id" = "ci_builds"."id"
          AND
          "ci_build_needs"."name" = "base_and_descendants"."name"
          AND
          "ci_builds"."commit_id" = 634207789
      )
    )
    SELECT "ci_builds".*
    FROM "base_and_descendants" AS "ci_builds"
    WHERE
      "ci_builds"."id" NOT IN (
        SELECT "ci_builds"."id"
        FROM "ci_builds"
        WHERE
          "ci_builds"."type" IN ('Ci::Processable', 'Ci::Build', 'Ci::Bridge')
          AND
          "ci_builds"."id" = 2995760625
      )
  )
  UNION
  (
    SELECT "ci_builds".*
    FROM "ci_builds"
    WHERE
      "ci_builds"."type" IN ('Ci::Processable', 'Ci::Build', 'Ci::Bridge')
      AND
      "ci_builds"."commit_id" = 634207789
      AND
      (stage_idx > 1)
  )
) ci_builds
WHERE
  "ci_builds"."type" IN ('Ci::Processable', 'Ci::Build', 'Ci::Bridge')
  AND
  ("ci_builds"."status" IN ('success')) -- this is actually skipped but I made it success for test purposes
ORDER BY "ci_builds"."stage_idx" ASC;

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Furkan Ayhan

Merge request reports