Add tests to the CI schema

What does this MR do and why?

For #355696 (closed)

The pipeline editor uses a JSON schema to validate the CI config on the frontend, as well as provide useful definitions/annotations and autocomplete features to the user. As the rules for writing a CI config gets updated, the JSON schema is also updated to reflect these changes.

With the size of the schema and number of rules, it becomes tedious to manually test these changes. This MR creates a base test helper for testing JSON schemas using ajv to provide test coverage increase our confidence in the code.

The test helper allows testing with JSON test files and YAML test files. The YAML input was added because the Pipeline Editor is written in YAML.

MR Breakdown

The main changes are in spec/frontend/__helpers__/matchers/to_validate_json_schema.js (extends expect with a new method to validate JSON schemas) and spec/frontend/editor/schema/ci/ci_schema_spec.js (specs for the CI schema).

Most of the new files are tests (mostly imported from schemastore, with minimal changes for linting) and adding yarn packages.

The JSON test files are tests used by schemastore when they hosted the CI schema before. These files have been deleted already on the main branch, but we've been given permission (recommendation) to copy the test files to GitLab.

As for the YAML tests, we've added new keywords to the CI json schema since we started hosting it on GitLab. These new definitions don't have JSON tests yet, so I used this as an opportunity to see if we can create tests in YAML format instead by adding tests in YAML. The plan is to start writing tests for the CI schema in YAML going forward.

Yarn packages

I added ajv and ajv-formats. ajv is what we can use to validate json schemas using json test files. This is also the same package used by schemastore to test its json schemas.

ajv-formats defines different formats (such as URI, date-time, etc) so they can be used in json schema definitions and validations.

How It Works

This is how the spec/frontend/editor/schema directory should look like now:

spec/frontend/editor/schema
├── base_test_setup.js
└── ci
    ├── ci_schema_spec.js
    ├── json_tests
    │   ├── negative_tests
    │   └── positive_tests
    └── yaml_tests
        ├── negative_tests
        └── positive_tests

Positive tests are json/yaml snippets that we expect to be valid. Negative tests are json/yaml snippets that we expect to be invalid (i.e. they intentionally use the schema incorrectly). This allows us the check that our json schema correctly validates different examples of input.

base_test_setup.js is our test helper and initializes ajv for us. The actual tests are in ci_schema_spec.js. What it does is that it loops through all the files in the test directories and checks with ajv if the json schema of our choosing validates them.

To help make writing these tests easier, I extended jest::expect with the new methods toValidateJsonSchema and toInvalidateJsonSchema. These methods will parse ajv's error messages to check if the file is valid or not.

Thus, we can write our tests this way:

import Ajv from 'ajv';
import AjvFormats from 'ajv-formats';
import CiSchema from '~/editor/schema/ci.json';
import CacheYaml from './yaml_tests/positive_tests/cache.yml';
import CacheNegativeYaml from './yaml_tests/negative_tests/cache.yml';

const ajv = new Ajv({
  strictTypes: false,
  strictTuples: false,
  allowMatchingProperties: true,
});

AjvFormats(ajv);
const schema = ajv.compile(CiSchema);

it(`schema validates correct use of the CACHE keyword`, () => {
  expect(CacheYaml).toValidateJsonSchema(schema);
  expect(CacheNegativeYaml).not.toInvalidateJsonSchema(schema);
});

With this setup, developers and contributors who want to update the CI schema can add or update the yaml/json test files and run the script to see if it works. No more manual testing in the editor! 😄

Screenshots or screen recordings

N/A

How to set up and validate locally

1. Install missing yarn packages.
2. Run the following command:

yarn jest --watch spec/frontend/editor/schema/ci/ci_schema_spec.js

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.