[Feature flag] Rollout of `ci_runner_builds_queue_on_replicas`
What
Rollout and remove the :ci_runner_builds_queue_on_replicas
feature flag.
Owners
- Team: CI
- Most appropriate slack channel to reach out to:
#g_ci
- Best individual to reach out to:
@grzesiek
- MR: !56849 (merged)
Expectations
What are we expecting to happen?
We expect reduction of BIG query hitting primary database and ending up in slow query log:
- https://log.gprd.gitlab.net/goto/5b2712978229188d04cc91f976a01df2
- https://log.gprd.gitlab.net/goto/ee7664a73195d4f13d4c7b0f218a01f4
What might happen if this goes wrong?
We might see 500 on jobs/request
endpoint due to much more pressure on replicas
.
What can we monitor to detect problems with this?
- https://log.gprd.gitlab.net/goto/5b2712978229188d04cc91f976a01df2
- https://log.gprd.gitlab.net/goto/ee7664a73195d4f13d4c7b0f218a01f4
Roll Out Steps
/chatops run feature delete ci_runner_builds_queue_on_replicas
-
Enable on staging ( /chatops run feature set ci_runner_builds_queue_on_replicas true --staging
) -
Test on staging -
Coordinate a time to enable the flag with the SRE oncall and release managers - In
#production
mention@sre-oncall
and@release-managers
. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
- In
-
If it is possible to perform an incremental rollout, this should be preferred. Proposed increments are: 10%
,50%
,100%
. Proposed minimum time between increments is 15 minutes.-
10%
:/chatops run feature set ci_runner_builds_queue_on_replicas 10
-
50%
:/chatops run feature set ci_runner_builds_queue_on_replicas 50
-
100%
:/chatops run feature set ci_runner_builds_queue_on_replicas true
-
-
Announce on the issue that the flag has been enabled -
Remove feature flag and add changelog entry. Ensure that the feature flag definition YAML file has been removed in the same MR that is removing the feature flag from the code -
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel
Rollback Steps
-
This feature can be disabled by running the following Chatops command:
/chatops run feature delete ci_runner_builds_queue_on_replicas
Edited by Kamil Trzciński (Back 2025-01-01)