Skip to content

E2E test: use rerun_file_path for E2E test metrics

Jay McCure requested to merge jmc-metrics-file_path into master

What does this MR do and why?

The reliable report uses the test metric data from InfluxDB. When reviewing this report I noticed that some Create tests had a stage unknown.

This is because we derive the stage from the file path, but in this case in was a shared_example not in a stage directory. We can use the rerun_file_path as that is the file path that would re-run the spec. From what I can find, there's not much documentation around rspec metadata, but there is some info here about using rerun_file_path

How to set up and validate locally

This test uses a shared_example:

QA_RUN_TYPE=fake CI=true QA_EXPORT_TEST_METRICS=false QA_INFLUXDB_URL=foo.com QA_INFLUXDB_TOKEN=footoken CI_PIPELINE_CREATED_AT=2021-02-23T20:58:41Z  QA_SAVE_TEST_METRICS=true QA_LOG_LEVEL=DEBUG QA_GITLAB_URL=http://192.168.4.31:3000 bundle exec rspec qa/specs/features/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb --format QA::Support::Formatters::TestMetricsFormatter

Before->After regular spec (no change)

[{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Create a new merge request creates a merge request with a milestone and label","file_path":"/browser_ui/3_create/merge_request/create_merge_request_spec.rb","status":"passed","smoke":"false","reliable":"false","quarantined":"false","retried":"false","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"code_review","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347762"},"fields":{"id":"./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb[1:1:2]","run_time":30231,"api_fabrication":11891,"ui_fabrication":9801,"total_fabrication":21692,"retry_attempts":0,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":""}},{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Create a new merge request creates a basic merge request","file_path":"/browser_ui/3_create/merge_request/create_merge_request_spec.rb","status":"passed","smoke":"true","reliable":"false","quarantined":"false","retried":"false","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"code_review","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347738"},"fields":{"id":"./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb[1:1:1]","run_time":13553,"api_fabrication":90,"ui_fabrication":8168,"total_fabrication":8258,"retry_attempts":0,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":""}}]
[{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Create a new merge request creates a basic merge request","file_path":"/browser_ui/3_create/merge_request/create_merge_request_spec.rb","status":"passed","smoke":"true","reliable":"false","quarantined":"false","retried":"false","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"code_review","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347738"},"fields":{"id":"./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb[1:1:1]","run_time":26752,"api_fabrication":11694,"ui_fabrication":8497,"total_fabrication":20191,"retry_attempts":0,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":""}},{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Create a new merge request creates a merge request with a milestone and label","file_path":"/browser_ui/3_create/merge_request/create_merge_request_spec.rb","status":"passed","smoke":"false","reliable":"false","quarantined":"false","retried":"false","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"code_review","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347762"},"fields":{"id":"./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb[1:1:2]","run_time":18543,"api_fabrication":89,"ui_fabrication":9784,"total_fabrication":9873,"retry_attempts":0,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":""}}]

Before->After, shared_example (before file_path referred to shared_example path and stage was null):

[{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Codeowners when the project is in the root group and the code owner is a user behaves like code owner merge request is approved and merged","file_path":"/shared_examples/merge_with_code_owner_shared_examples.rb","status":"failed","smoke":"false","reliable":"false","quarantined":"false","retried":"true","job_name":null,"merge_request":"false","run_type":"fake","stage":null,"product_group":"source_code","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347805"},"fields":{"id":"./qa/specs/features/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb[1:1:1:2:1:1]","run_time":80160,"api_fabrication":30,"ui_fabrication":0,"total_fabrication":30,"retry_attempts":1,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":"Unable to find css \"[data-testid=\\\"approvals_number_field\\\"],[data-qa-selector=\\\"approvals_number_field\\\"]\""}},{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Codeowners when the project is in the root group and the code owner is the root group behaves like code owner merge request is approved and merged","file_path":"/shared_examples/merge_with_code_owner_shared_examples.rb","status":"failed","smoke":"false","reliable":"false","quarantined":"false","retried":"true","job_name":null,"merge_request":"false","run_type":"fake","stage":null,"product_group":"source_code","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347804"},"fields":{"id":"./qa/specs/features/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb[1:1:1:1:1:1]","run_time":39639,"api_fabrication":32,"ui_fabrication":0,"total_fabrication":32,"retry_attempts":1,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":"Unable to find css \"[data-testid=\\\"approvals_number_field\\\"],[data-qa-selector=\\\"approvals_number_field\\\"]\""}}]
[{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Codeowners when the project is in the root group and the code owner is the root group behaves like code owner merge request is approved and merged","file_path":"/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb","status":"failed","smoke":"false","reliable":"false","quarantined":"false","retried":"true","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"source_code","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347804"},"fields":{"id":"./qa/specs/features/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb[1:1:1:1:1:1]","run_time":80718,"api_fabrication":174,"ui_fabrication":0,"total_fabrication":174,"retry_attempts":1,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":"Unable to find css \"[data-testid=\\\"approvals_number_field\\\"],[data-qa-selector=\\\"approvals_number_field\\\"]\""}},{"name":"test-stats","time":"2021-02-24T06:58:41.000+10:00","tags":{"name":"Create Codeowners when the project is in the root group and the code owner is a user behaves like code owner merge request is approved and merged","file_path":"/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb","status":"failed","smoke":"false","reliable":"false","quarantined":"false","retried":"true","job_name":null,"merge_request":"false","run_type":"fake","stage":"create","product_group":"source_code","testcase":"https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347805"},"fields":{"id":"./qa/specs/features/ee/browser_ui/3_create/repository/merge_with_code_owner_in_root_group_spec.rb[1:1:1:2:1:1]","run_time":39525,"api_fabrication":31,"ui_fabrication":0,"total_fabrication":31,"retry_attempts":1,"job_url":null,"pipeline_url":null,"pipeline_id":null,"job_id":null,"merge_request_iid":null,"failure_exception":"Unable to find css \"[data-testid=\\\"approvals_number_field\\\"],[data-qa-selector=\\\"approvals_number_field\\\"]\""}}]

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jay McCure

Merge request reports

Loading