Skip to content

Include all specs in metadata script

Désirée Chevalier requested to merge qa-include-all-specs-in-metadata-script into master

What does this MR do and why?

Fixes test_metadata_only to include all tests except skip and quarantine(without an only tag) in the dry-run. It was previously not getting metadata for geo, sanity_feature_flags, transient, and orchestrated tests as well as any tests with the only or except tags. This way we can verify proper test case link usage and quarantine tags with the qa:metadata-lint job for all tests that are run in any environment/pipeline.

Further context: #381301 (comment 1157609775) Two orchestrated tests were missing testcase links(should have been caught by the qa:metadata-lint job) which revealed a regression with search since when a testcase link isn't present, we fall back to using search to find the test case.

Fixing this also revealed duplicate test case usage which has been fixed in this MR.

How to set up and validate locally

  1. From the gitlab/qa folder, run bundle exec bin/qa Test::Instance::All http://localhost:3000 --test-metadata-only
  2. From the gitlab/ folder, run ./scripts/qa/testcases-check qa/tmp/test-metadata.json

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Désirée Chevalier

Merge request reports