Skip to content

Fail builds which are waiting for resource for too long

Problem

In some cases, pipeline jobs could be stuck at waiting_for_resource status, for example, when .enqueue failed even after the build successfully retained a resource.

We should drop waiting_for_resource builds after some period. We can use StuckCiJobsWorker in such case.

Proposal

  • Add a configuration for the settings for timeout similar to the job timeout setting Resource_group_settings
  • Add Resouce groups section to Settings>CI/CD. Located after Variables and before 'Pipeline triggers' sections
  • Setting name should be named Resource timeout
  • If waiting_for_resource exceeds the time configured the job should be terminated.
  • default timeout shall be 1 hour.

Technical proposal

  • Use StuckCiJobsWorker to cancel long waiting jobs.
  • Add resource_timeout (integer, unit: minute, minimum: 60(an hour), maximum: 10080(a week)) column into project_ci_cd_settings.
  • The value can be configured in UI, likely setting page.
  • In StuckCiJobsWorker, we search the target rows with the following query.
SELECT * FROM ci_builds
LEFT JOINS project_ci_cd_settings ON project_ci_cd_settings.project_id = ci_builds.project_id
WHERE status = 'waiting_for_resouce' AND
      waiting_for_resource_at + (project_ci_cd_settings.resource_timeout * interval '1 minute') > now()

and drops the builds with resource_waiting_timeout as failure_reasons.

  • Since we depend on cron worker and the interval is 0 * * * * (every hour), the timed out jobs won't be dropped at the exact timing but when the worker is fired.

Tasks

  • Make resource_timeout config for resource group
  • Extend StuckCiJobsWorker to drop long waiting builds after the configured period.
  • Publish the feature (Documentation and remove a feature flag)

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

Edited by 🤖 GitLab Bot 🤖