Experiment with using AI in gitlab-housekeeper feature flag removal
Problem
Removing feature flags after they've been enabled is a manual task today. We implemented a partial automation with https://docs.gitlab.com/ee/development/feature_flags/#optionally-add-a-patch-file-for-automated-removal-of-feature-flags in !145617 (merged) .
But this requires developers to do the upfront work to create the patch file and then the gitlab-housekeeper can automatically remove the feature flag for you later.
Solution
A lot of the time removing a feature flag is a trivial and mechanical task. Automation through traditional computing approaches (eg. manipulating the Ruby AST) are unlikely to be effective because there are just too many edge cases that require some human judgement (especially when removing the feature flags from specs, but even normal code can have early returns or whole methods that can be removed).
So far some initial experimentation was done in !141962 and !142095 with limited success. We may still need to experiment with how we prompt the AI (eg. how we ask the question but also how we chunk up the code to send) and how we ask for the responses (eg. in a diff or whole segments of code) .
We should also collect a realistic set of validation diffs from previous feature flags that have been removed from gitlab-org/gitlab so that we can experiment more efficiently.
Example MRs
The following MRs could be used for validation as they serve as the kind of straight-forward feature flag removal MRs that we'd expect an LLM to be able to figure out:
- !139344 (closed)
- !96748 (merged)
- !146610 (closed)
- !146738 (merged) => might be a slightly trickier case
- !146112 (merged)
- !146355 (merged)
- !146609 (merged)
- !146492 (merged)
- !146287 (merged)
- !146320 (merged)