Gitlab housekeeper initial implementation with finalize BBM
What does this MR do and why?
This is an initial implementation of the problem and solutions described in this blueprint !134487 (merged) and it was extracted from a POC implementation in !134485 (closed) which contained other work we don't want to include in the first iteration.
This is a gem which can be run locally or in CI to do static and dynamic analysis of the codebase and using a list of predefined "keeps" it will automatically create merge requests for things that developers would have otherwise needed to remember to do themselves.
It is analogous to a mix of rubocop -a
and GitLab Dependency Bot.
The word "keep" is used to describe a specific rule to apply to the code to match a required change and actually edit the code. The word "keep" was chosen as it sounds like "cop" and is very similar to implementing a rubocop rule as well as code to autocorrect the rule.
Today we have a complete implementation of the OverdueFinalizeBackgroundMigration
keep which finds all batched background migrations added before the last required stop upgrade point and then checks in Postgres.ai to confirm that this migration is complete on GitLab.com and then generates a migration to finalize the batched background migration. It has generated many migration MRs already.
How the code is organized
The code is organized in a very similar way to RuboCop in that we have a overall gem called gitlab-housekeeper
that contains the generic logic of looping over all keeps
(analogous to Cops) which are rules for how to detect changes that can be made to the code and then actually how to correct them.
This MR includes the following high level pieces:
-
gems/gitlab-housekeeper
which is the generic gem. It should be responsible for runningkeeps
which will return aChange
object. The housekeeper then uses this change object to create a git branch, push the branch, create a merge request- The
gitlab-housekeeper
command which just calls theRunner
- The
Runner
which contains the high level operation of thegitlab-housekeeper
gem. It looks at the options provided and runs the correct keeps. It then takes theChange
results from the keeps and creates merge requests - The
Git
client is just a wrapper around shelling out to git - The
GitlabClient
is just a thin wrapper aroundHTTParty
to make HTTP requests to create merge requests - The
Shell
class is just a wrapper around shelling out. Mainly this is useful for stubbing in specs for all the other code.
- The
- The
keeps/
directory which is analogous to ourrubocop
directory. It contains implementations of specific keeps and also ahelpers
directory that we expect to be used in multiple keeps-
keeps/helpers/postgres_ai.rb
is a wrapper around thePG
client that we can use to encapsulate code that looks up things from the current state of our production database. For now it is used to check if a batched background migration is finished in production -
keeps/overdue_finalize_background_migration.rb
: This is our first keep we've implemented. It has been used successfully to generate at least 15 merged migrations already. This is a pretty complicated piece of code but at a high level it is looking fordb/docs/batched_background_migrations
files and trying to find ones that were added before the last required stop and have already finished on GitLab.com . This means it's safe to finalize them. It then generates a migration with the change necessary to finalize the BBM.
-
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.