[Feature flag] Roll out `:cleanup_lfs_during_gc`
What
Remove the :cleanup_lfs_during_gc
feature flag ...
Owners
- Team: Create: Source Code
- Most appropriate slack channel to reach out to:
#g_create_source-code-be
- Best individual to reach out to:
@nick.thomas
Expectations
What are we expecting to happen?
Project housekeeping should now routinely remove unreferenced LFS objects
What might happen if this goes wrong?
The extra work during GitGarbageCollectWorker
execution could be unwelcome
If it erroneously removes LFS objects when it shouldn't, we could lose data
What can we monitor to detect problems with this?
The main concern I have is around performance. We're adding more work to an expensive worker, and this could cause problems.
We should watch:
- https://dashboards.gitlab.net/d/sidekiq-queue-detail/sidekiq-queue-detail?orgId=1&from=now-6h&to=now&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-queue=git_garbage_collect
- https://dashboards.gitlab.net/d/alerts-sat_disk_sus_read_iops/alerts-disk_sustained_read_iops-saturation-detail?orgId=1
- https://dashboards.gitlab.net/d/alerts-sat_disk_sus_write_iops/alerts-disk_sustained_write_iops-saturation-detail?orgId=1
- https://log.gprd.gitlab.net/goto/6618e5da43cfb62b9339db6f726a0fd2
Beta groups/projects
If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.
-
nick.thomas/www-gitlab-com
project -
nick.thomas/gitlab-ee
project -
gitlab-org
group -
gitlab-com
group
Roll Out Steps
-
Enable on staging -
Test on staging -
Ensure that documentation has been updated !40979 (merged) -
Enable on GitLab.com for individual groups/projects listed above and verify behaviour -
Coordinate a time to enable the flag with #production
and#g_delivery
on slack. -
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
-
Cross post chatops slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove feature flag and add changelog entry -
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel
Edited by Nick Thomas