Skip to content

Add JSON RSpec formatter for e2e tests

Mark Lapierre requested to merge ml-add-json-rspec-formatter-with-extra-metadata into master

What does this MR do?

Adds a custom RSpec formatter that exposes status_issue, quarantine, and screenshot metadata in test results in a JSON format, as well as all the other data we currently get from the JUnit XML format report.

To be used when reporting e2e test results in issues

Part of https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/504

The formatter can be used with CLI options: --format QA::Support::JsonFormatter --out "tmp/rspec.json"

Examples:

A test with status_issue assigned:

{
  "id": "./qa/specs/features/api/1_manage/users_spec.rb[1:1:1]",
  "description": "GET /users",
  "full_description": "Manage Users API GET /users",
  "status": "passed",
  "file_path": "./qa/specs/features/api/1_manage/users_spec.rb",
  "line_number": 11,
  "run_time": 0.213950913,
  "pending_message": null,
  "status_issue": "https://gitlab.com/gitlab-org/quality/testcases/-/issues/460",
  "quarantine": null,
  "screenshot": null
}

A test in quarantine:

{
  "id": "./qa/specs/features/api/1_manage/users_spec.rb[1:1:2]",
  "description": "GET /users/:username with a valid username",
  "full_description": "Manage Users API GET /users/:username with a valid username",
  "status": "pending",
  "file_path": "./qa/specs/features/api/1_manage/users_spec.rb",
  "line_number": 18,
  "run_time": 0.040366781,
  "pending_message": "In quarantine : https://gitlab.com/gitlab-org/gitlab/-/issues/225303",
  "status_issue": null,
  "quarantine": {
    "issue": "https://gitlab.com/gitlab-org/gitlab/-/issues/225303",
    "type": "bug"
  },
  "screenshot": null
}

A failed test:

{
  "id": "./qa/specs/features/browser_ui/1_manage/login/log_in_spec.rb[1:1:2]",
  "description": "fails",
  "full_description": "Manage basic user login fails",
  "status": "failed",
  "file_path": "./qa/specs/features/browser_ui/1_manage/login/log_in_spec.rb",
  "line_number": 24,
  "run_time": 3.955657934,
  "pending_message": null,
  "status_issue": null,
  "quarantine": null,
  "screenshot": {
    "html": "/home/mark/dev/gitlab/gitlab/qa/tmp/qa-test-2020-08-06-11-35-00-69907fd3982518b0/manage_basic_user_login_fails.html",
    "image": "/home/mark/dev/gitlab/gitlab/qa/tmp/qa-test-2020-08-06-11-35-00-69907fd3982518b0/manage_basic_user_login_fails.png"
  },
  "exceptions": [
    {
      "class": "RuntimeError",
      "message": "This is what an exception message looks like",
      "backtrace": [
        "/home/mark/dev/gitlab/gitlab/qa/qa/specs/features/browser_ui/1_manage/login/log_in_spec.rb:31:in `block (3 levels)",
        ...
      ]
    }
  ]
}

A test with multiple failures:

{
      "id": "./qa/specs/features/api/1_manage/users_spec.rb[1:1:1]",
      "description": "GET /users",
      "full_description": "Manage Users API GET /users",
      "status": "failed",
      "file_path": "./qa/specs/features/api/1_manage/users_spec.rb",
      "line_number": 15,
      "run_time": 9.773683378,
      "pending_message": null,
      "status_issue": "https://gitlab.com/gitlab-org/quality/testcases/-/issues/460",
      "quarantine": null,
      "screenshot": null,
      "exceptions": [
        {
          "class": "RuntimeError",
          "message": "Error in test",
          "backtrace": [
            "/home/mark/dev/gitlab/gitlab/qa/qa/specs/features/api/1_manage/users_spec.rb:21:in `block (3 levels) in <module:QA>'",
            ...
          ]
        },
        {
          "class": "RuntimeError",
          "message": "Error in after",
          "backtrace": [
            "/home/mark/dev/gitlab/gitlab/qa/qa/specs/features/api/1_manage/users_spec.rb:12:in `block (3 levels) in <module:QA>'",
            ...
          ]
        },
        {
          "class": "Selenium::WebDriver::Error::WebDriverError",
          "message": "chrome not reachable\n  (Session info: chrome=84.0.4147.105)\n  (Driver info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee976b2c3e3b9f335cde7),platform=Linux 4.15.0-112-generic x86_64)",
          "backtrace": [
            "/home/mark/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/selenium-webdriver-3.142.6/lib/selenium/webdriver/remote/response.rb:72:in `assert_ok'",
            ...
          ]
        }
      ]
    },

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Mark Lapierre

Merge request reports