Skip to content

Limit the number of active jobs early in the pipeline creation

Fabio Pitino requested to merge enforce-activity-limit-before-populate-2 into master

What does this MR do and why?

Related to https://gitlab.com/gitlab-org/gitlab/-/issues/356310

In this MR we are moving the enforcement of ci_active_jobs limit earlier in the Ci::CreatePipelineService chain, before the Chain::Populate which assigns all the seeded jobs, stages and related objects to the pipeline. By moving failing the pipeline earlier we would only persist an empty pipeline rather than the whole pipeline with jobs.

This ensures that:

  1. We persist much less data in case of failure.
  2. We don't run any side-effects (hooks, workers, etc.) related to failing jobs.

In order to do implement this change we added a new step early in the chain called Chain::Limit::ActiveJobs that is activated via a feature flag <code data-sourcepos="11:150-11:175">ci_limit_active_jobs_early</code>. The old steps Chain::Limit::JobActivity, later in the chain, will be deactivated via the same feature flag.

Screenshots or screen recordings

When running the pipeline via the Run Pipeline button in the UI, the pipeline is triggered with save_on_errors: false so we should see only a flash message with the error and no pipeline persisted.

image

When running a pipeline after a git push (or changes via Pipeline Editor or WebIDE) we should see a failed empty pipeline since the pipeline is created with save_on_errors: true.

image

How to set up and validate locally

  1. Enable the feature flag
Feature.enable(:ci_limit_active_jobs_early)
  1. Set ci_active_jobs limit to a low value:
Plan.default.actual_limits.update!(ci_active_jobs: 10)
  1. Trigger a pipeline with 6 jobs having them all with script: sleep 60 to simulate some work.
job:
  parallel: 6
  script: sleep 60
  1. Ensure the pipeline is created successfully.
  2. Trigger the same pipeline again, bringing the total count of active jobs to 12.
  3. Ensure the empty pipeline is dropped with the expected error message.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Fabio Pitino

Merge request reports