Mechanism to add rspec meta to specs
What does this MR do and why?
Describe in detail what your merge request does and why.
As part of gitlab-org/quality/quality-engineering/team-tasks#1918 (closed) (OKR), we want to automate promotion to reliable bucket and quarantining reliable test that are no longer reliable.
In gitlab-org/gitlab!139742 (merged), we created json files as output of script that runs the reliable report. The json files contain specs that need to be quarantined or promoted to reliable.
This MR implements code that uses those json files and create MRs for quarantining unstable reliable specs and promoting specs to reliable.
Notes:
- This MR depends on !128 (merged). Once that MR is merged, the target branch of this MR should be changed to
main
. - The code in this MR does not decide which tests to quarantine or promote to reliable. It just takes a json file and acts on it.
- There is a limitation in the quarantining code. It currently does not detect if a spec is already in quarantine.
Credits
Thanks to @ddavison for his work on the confiner project. Some code in this MR takes inspiration from that project.
Screenshots or screen recordings
These are strongly recommended to assist reviewers and reduce the time to merge your change.
How to set up and validate locally
For example MRs already created as part of testing, please see: https://gitlab.com/sliaquat/gitlab/-/merge_requests?scope=all&state=all
For testing quarantining of unstable reliable specs:
QA_TEST_DRI_HANDLE='<your_gitlab_handle>' CI_SLACK_WEBHOOK_URL="<available_in_1p>" SLACK_QA_CHANNEL="qa-notifications-test-channel" QA_LOG_LEVEL='debug' exe/update-test-meta --project "sliaquat/gitlab" --token "<your_personal_access_token>" --unstable_specs_file "path/to/unstable_specs.json"
Notes:
-
QA_TEST_DRI_HANDLE
is for testing only to avoid pinging other team members. When not provided, the code will use the roulette to ping the appropriate stage SET. -
SLACK_QA_CHANNEL
set here is for testing only. When not provided, the message will be send totest-platform-maintainers
Slack channel. - In
--project
, provide a GitLab project where you would like to operate on and create MRs. It is best to use your own fork of the GitLab project to avoid creating unnecessary MRs. A point to note is that MRs will not have all the labels if your fork doesn't have them. - For
--unstable_specs_file
, you can use this file as an example: unstable_specs.json
For testing promoting stable specs to reliable:
QA_TEST_DRI_HANDLE='<your_gitlab_handle>' QA_LOG_LEVEL='debug' exe/update-test-meta --project "sliaquat/gitlab" --token "<your_personal_access_token>" --stable_specs_file "path/to/stable_specs.json"
Notes:
-
QA_TEST_DRI_HANDLE
is for testing only to avoid pinging other team members. When not provided, the code will use the roulette to ping the appropriate stage SET. - In
--project
, provide a GitLab project where you would like to operate on and create MRs. It is best to use your own fork of the GitLab project to avoid creating unnecessary MRs. A point to note is that MRs will not have all the labels if your fork doesn't have them. - In
--stable_specs_file
, you can use this file as an example: stable_specs.json
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.