Skip to content

Throttle pipelines creation rate

Marius Bobin requested to merge mb-ci-throttle-pipeline-creation into master

What does this MR do and why?

Related to https://gitlab.com/gitlab-org/gitlab/-/issues/352232

Since we already have a limit for how many active pipelines a project could have, I think it would be a good idea to add a limit on how many pipelines could be created by a user for a commit in a short period of time. This should stop rogue pipelines that the other two limits fail to catch.

This limits the number of pipelines created for the same project/commit/user tuple to 20 per minute. This value is not definitive yet.

Rollout plan

  • enable ci_track_throttle_pipelines_creation for a week and monitor the logs
  • if there isn't any entry in the log file, proceed to disable ci_throttle_pipelines_creation_dry_run and start enforcing the limit
  • if there are log entries, analyze them and reconsider the limit.
  • as a follow-up, move the limit into an application_settings column

Screenshots or screen recordings

image

image

How to set up and validate locally

  1. Run database migrations: bin/rails db:migrate
  2. Update the limit to a lower value: ApplicationSetting.current.update!(pipelines_create_limit: 2)
  3. Enable the ci_throttle_pipelines_creation feature flag:
    • Feature.enable(:ci_throttle_pipelines_creation)
  4. Disable the ci_throttle_pipelines_creation_dry_run feature flag to enforce the limit:
    • Feature.disable(:ci_throttle_pipelines_creation_dry_run)
  5. Open 3 tabs on the new pipelines in the same project: namespace/project/-/pipelines/new
  6. Press the Run pipeline button in each tab

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Marius Bobin

Merge request reports