Skip to content

Create application limits for scheduled pipeline

Release notes

Scheduled pipeline that run very frequently can impact an instance's performance. Previously any user could schedule unlimited amount of scheduled pipelines that runs every 5 min In GitLab 14.0, we are limiting the time interval for schedule pipeline to be 1hr minimum

  • The limitation will be enforced for free tier (SaaS) only.
  • Existing configuration will not be changed.
  • There would be no cap for paid users.

Problem to solve

Without application limits on schedule pipelines they can be scheduled to run at a frequency that causes harmful load on GitLab.com's shared runners that can potentially become a DOS attack that must be disabled.

Further details

As part of gitlab-com/gl-infra/production#3800 (closed) we've seen some cron patterns with a high frequency which was causing a load on our GitLab.com shared runner fleet, causing a GitLab to DOS itself because of these harmful cron patterns. During the incident we ended up disabling this high-frequency cron expressions. It was discussed during the incident that as a corrective action it would be good to disallow such expressions so no one can create a harmful schedule pipeline.

Proposal

Set a rate limit on pipeline schedules as part of Application Limits (self-managed instances is out of scope).
The implementation is for SaaS only, the default for GitLab.com will be:

  • 10 schedules per project daily for Free tier users only Already implemented
  • Time interval cant go below 1hr.
  • No cap for paid.

We should automatically adjust any new scheduler using a corn job that is less than 1hr. In addition in the Schedule pipeline, we should add a generic message that will inform the user that the configuration is invalid, the banner will contain a link to the section in the documentation that explains the limitation of a scheduled pipeline.

Potential enhancement - (only if it doesn't complicate the MVC)

Display the banner for paid users when:

  • Paid user tries to create a cron job that is less than 5min. The existing experience today is that we automatically adjust the pipeline to run every 5min without notifying the user.

User experience goal

  • Add a notification bar when setting up an unsupported cron job for schedule pipelines

Instance admins can set Application Limits for schedule pipeline feature via the Rails console, as it happens today for other application limits.

Documentation

We should document this limitation at docs https://docs.gitlab.com/ee/user/gitlab_com/#gitlab-cicd

Permissions and Security

Only instance admins can see/edit the Application Limits for pipeline schedules.

Availability & Testing

TBD, we'll need to better define how the product is going to respond to pipeline build frequencies that exceed our threshold before we can complete this section.

Links / references

Edited by Dov Hershkovitch