Skip to content

Compare live vs actual CI minutes consumption

Fabio Pitino requested to merge compare-live-and-persisted-minutes-consumption-with into master

What does this MR do?

Related to https://gitlab.com/gitlab-org/gitlab/-/issues/20856

Background

This MR builds on top of !59263 (merged) where we added the ability to track the consumption of CI minutes for running builds, called "live consumption". This tracking is not entirely accurate since we would miss some timeframe between the builds being assigned to the runner and the time between last Runner ping about the build and the time the build completes. It's however considered good enough approach for live tracking since we traded accuracy for performance. To balance out with the missed accuracy we are giving 1000 minutes of grace periods to all namespaces before we start dropping builds.

About this MR

When a build completes we run Ci::Minutes::UpdateBuildMinutesService which is responsible for persisting the actual CI minutes consumption. At this point we can compare the actual consumption with the "live consumption" from !59263 (merged) which is used to determine whether to drop running builds.

The comparison is recorded as a Prometheus histogram in order to understand the accuracy of the live consumption.

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Fabio Pitino

Merge request reports