ci: run visual tests for changed components only
What does this MR do?
This changes the CI setup so that visual tests only run for the
components that were actually changed in an MR. This is done by
generating a list of changed files basenames and trying to find
.stories.js
files located in the same directory.
The full-blown visual
test job can be run manually in an MR and runs as
usual in the default branch.
How does it work?
When pushing changes, we get a list of changed files and iterate through it. We try to find any .stories.js
file living in the same directory and we run visual regression tests against them. If, for any given file, no .stories.js
file can be found, we err on the side of caution and run the full test suite.
Examples
Minimal test run
accordion/
├─ accordion.vue
├─ accordion_item.vue
├─ accordion.stories.js
In this case, changing accordion/accordion_item.vue
would result in accordion.stories.js
being included in the minimal test run.
Minimal test run (multiple sibling stories)
accordion/
├─ accordion.vue
├─ accordion_item.vue
├─ accordion.stories.js
├─ accordion_item.stories.js
Changing accordion/accordion_item.vue
would result in both accordion.stories.js
and accordion_item.stories.js
being run.
Full test run
accordion/
├─ accordion.vue
├─ accordion.stories.js
accordion_item/
├─ accordion_item.vue
Changing accordion_item/accordion_item.vue
would result in the full test suite being run because no .stories.js
file can be found next to it.
Note: If
accordion_item/
did contain a.stories.js
file, only it would be tested in a minimal test run.accordion.stories.js
would not be tested. The assumption is that, if a change toaccordion_item.vue
resulted in visual changes, then its visual test would break and you would have to runupdate_screenshots
which still runs against all stories, which in turn would also updateaccordion
's screenshots if they were affected by the change.
Benefit
The visual
CI job is infamously slow. In the last 100 successful pipelines, it took an average of 16 minutes to complete.
Considering that we rarely change more than a component at a time, running the full visual
test on every commit feels extremely expensive and time-consuming. This is especially true lately, as most MRs only add CSS utils that don't have any impact on the components themselves.
In cases where no component was touched in an MR, the visual_minimal
job can complete in around 2mn20s (~85% faster).
When a single component was changed, the job could complete in around 4mn40s (~70% faster).
Closes #1682 (closed)