Include all specs in metadata script
What does this MR do and why?
Fixes test_metadata_only
to include all tests except skip
and quarantine
(without an only tag) in the dry-run. It was previously not getting metadata for geo
, sanity_feature_flags
, transient
, and orchestrated
tests as well as any tests with the only
or except
tags. This way we can verify proper test case link usage and quarantine tags with the qa:metadata-lint job for all tests that are run in any environment/pipeline.
Further context: #381301 (comment 1157609775) Two orchestrated tests were missing testcase links(should have been caught by the qa:metadata-lint
job) which revealed a regression with search since when a testcase link isn't present, we fall back to using search to find the test case.
Fixing this also revealed duplicate test case usage which has been fixed in this MR.
How to set up and validate locally
- From the gitlab/qa folder, run
bundle exec bin/qa Test::Instance::All http://localhost:3000 --test-metadata-only
- From the gitlab/ folder, run
./scripts/qa/testcases-check qa/tmp/test-metadata.json
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.