Add metric to track state of new packages
What does this MR do and why?
Describe in detail what your merge request does and why.
Content
Add a metric called delivery_auto_deploy_new_package_state
which
will track if there are packages newer than the one on gstg-cny
.
The metric will track packages in the running
, success
or failed
states, i.e. packages that are being built, have been successfully
built or have failed building.
The metric name (delivery_auto_deploy_new_package_state
) isn't the best, so I'm open to suggestions.
The metric has a single label (pkg_state
), which can have 3 possible values (running
, success
, failed
) to track packages that are building, have been successfully built and have failed building. If there are one or more new packages (newer than what's on gstg-cny) being built, delivery_auto_deploy_new_package_state{pkg_state='running'}
will be set to 1. If there are no new packages being built delivery_auto_deploy_new_package_state{pkg_state='running'}
will be set to 0. Similarly for the success
and failed
states.
The metric is changed on the following events:
-
When a new package starts being built,
delivery_auto_deploy_new_package_state{pkg_state="running"}
is set to 1. -
When a package completes building successfully,
delivery_auto_deploy_new_package_state{pkg_state="success"}
is set to 1.- And we check to see if
delivery_auto_deploy_new_package_state{pkg_state="running"}
anddelivery_auto_deploy_new_package_state{pkg_state="failed"}
should be set to 0.pkg_state="running"
should be set to 0 if there are no new packages being built.pkg_state="failed"
should be set to 0 if there are no new packages that have failed building. This is done by checking for packager pipelines indev.gitlab.org
that are for versions newer than what's ongstg-cny
.
- And we check to see if
-
When package building fails,
delivery_auto_deploy_new_package_state{pkg_state="failed"}
is set to 1.- And we check to see if
delivery_auto_deploy_new_package_state{pkg_state="running"}
should be set to 0.
- And we check to see if
-
When a package is deployed to
gstg-cny
, we check to see if all 3 package states (success
,failed
,running
) should be set to 0.
This metric can be used in the following Prometheus query to get the intersection of time when gstg-cny
was idle and new packages were in different states.
(max by (pkg_state) (delivery_auto_deploy_new_package_state) > bool 0) * on() group_left max(delivery_auto_deploy_environment_state{target_env="gstg",target_stage="cny",env_state="ready"})
gitlab-com/gl-infra/delivery#20337 (closed)
Author Check-list
-
Has documentation been updated?