E2E test: use rerun_file_path for E2E test metrics
What does this MR do and why?
The reliable report uses the test metric data from InfluxDB. When reviewing this report I noticed that some Create
tests had a stage unknown
.
This is because we derive the stage from the file path, but in this case in was a shared_example
not in a stage directory. We can use the rerun_file_path
as that is the file path that would re-run the spec. From what I can find, there's not much documentation around rspec metadata, but there is some info here about using rerun_file_path
How to set up and validate locally
This test uses a shared_example:
QA_RUN_TYPE=fake CI=true QA_EXPORT_TEST_METRICS=false QA_INFLUXDB_URL=foo.com QA_INFLUXDB_TOKEN=footoken CI_PIPELINE_CREATED_AT=2021-02-23T20:58:41Z QA_SAVE_TEST_METRICS=true QA_LOG_LEVEL=DEBUG QA_GITLAB_URL=http://192.168.4.31:3000 bundle exec rspec qa/specs/features/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb --format QA::Support::Formatters::TestMetricsFormatter
Before->After regular spec (no change)
[{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Create a new merge request creates a merge request with a milestone and label","file_path":"/browser_ui/3_create/merge_request/create_merge_request_spec.rb","status":"passed","smoke":"false","reliable":"false","quarantined":"false","retried":"false","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"code_review","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347762"},"fields":{"id":"./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb[1:1:2]","run_time":30231,"api_fabrication":11891,"ui_fabrication":9801,"total_fabrication":21692,"retry_attempts":0,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":""}},{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Create a new merge request creates a basic merge request","file_path":"/browser_ui/3_create/merge_request/create_merge_request_spec.rb","status":"passed","smoke":"true","reliable":"false","quarantined":"false","retried":"false","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"code_review","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347738"},"fields":{"id":"./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb[1:1:1]","run_time":13553,"api_fabrication":90,"ui_fabrication":8168,"total_fabrication":8258,"retry_attempts":0,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":""}}]
[{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Create a new merge request creates a basic merge request","file_path":"/browser_ui/3_create/merge_request/create_merge_request_spec.rb","status":"passed","smoke":"true","reliable":"false","quarantined":"false","retried":"false","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"code_review","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347738"},"fields":{"id":"./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb[1:1:1]","run_time":26752,"api_fabrication":11694,"ui_fabrication":8497,"total_fabrication":20191,"retry_attempts":0,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":""}},{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Create a new merge request creates a merge request with a milestone and label","file_path":"/browser_ui/3_create/merge_request/create_merge_request_spec.rb","status":"passed","smoke":"false","reliable":"false","quarantined":"false","retried":"false","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"code_review","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347762"},"fields":{"id":"./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb[1:1:2]","run_time":18543,"api_fabrication":89,"ui_fabrication":9784,"total_fabrication":9873,"retry_attempts":0,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":""}}]
Before->After, shared_example (before file_path
referred to shared_example
path and stage was null
):
[{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Codeowners when the project is in the root group and the code owner is a user behaves like code owner merge request is approved and merged","file_path":"/shared_examples/merge_with_code_owner_shared_examples.rb","status":"failed","smoke":"false","reliable":"false","quarantined":"false","retried":"true","job_name":null,"merge_request":"false","run_type":"fake","stage":null,"product_group":"source_code","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347805"},"fields":{"id":"./qa/specs/features/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb[1:1:1:2:1:1]","run_time":80160,"api_fabrication":30,"ui_fabrication":0,"total_fabrication":30,"retry_attempts":1,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":"Unable to find css \"[data-testid=\\\"approvals_number_field\\\"],[data-qa-selector=\\\"approvals_number_field\\\"]\""}},{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Codeowners when the project is in the root group and the code owner is the root group behaves like code owner merge request is approved and merged","file_path":"/shared_examples/merge_with_code_owner_shared_examples.rb","status":"failed","smoke":"false","reliable":"false","quarantined":"false","retried":"true","job_name":null,"merge_request":"false","run_type":"fake","stage":null,"product_group":"source_code","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347804"},"fields":{"id":"./qa/specs/features/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb[1:1:1:1:1:1]","run_time":39639,"api_fabrication":32,"ui_fabrication":0,"total_fabrication":32,"retry_attempts":1,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":"Unable to find css \"[data-testid=\\\"approvals_number_field\\\"],[data-qa-selector=\\\"approvals_number_field\\\"]\""}}]
[{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Codeowners when the project is in the root group and the code owner is the root group behaves like code owner merge request is approved and merged","file_path":"/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb","status":"failed","smoke":"false","reliable":"false","quarantined":"false","retried":"true","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"source_code","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347804"},"fields":{"id":"./qa/specs/features/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb[1:1:1:1:1:1]","run_time":80718,"api_fabrication":174,"ui_fabrication":0,"total_fabrication":174,"retry_attempts":1,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":"Unable to find css \"[data-testid=\\\"approvals_number_field\\\"],[data-qa-selector=\\\"approvals_number_field\\\"]\""}},{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Codeowners when the project is in the root group and the code owner is a user behaves like code owner merge request is approved and merged","file_path":"/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb","status":"failed","smoke":"false","reliable":"false","quarantined":"false","retried":"true","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"source_code","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347805"},"fields":{"id":"./qa/specs/features/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb[1:1:1:2:1:1]","run_time":39525,"api_fabrication":31,"ui_fabrication":0,"total_fabrication":31,"retry_attempts":1,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":"Unable to find css \"[data-testid=\\\"approvals_number_field\\\"],[data-qa-selector=\\\"approvals_number_field\\\"]\""}}]
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.