New Results Handling and Automated Comparisons
Closes #50 (closed)
This MR facilitates the comparison of multiple test results and uploading them to a wiki page, with the goal of being able to compare results from multiple version of GitLab. To achieve this several challenges had to be tackled and, as such, this MR makes several large improvements to the tool along with facilitating the automated comparisons.
MR makes the following changes:
k6 comparisons pipeline
- Adds a new k6 comparisons pipeline that spins up our custom made GitLab docker images (via new
run-gitlab-docker
script) of multiple versions and runs our tests against each in parallel. After completion our new wiki script (report-results-wiki
) reports on these results (see below) - Our custom Images can be found over on the performance-images project. These are built each month after the new release.
- Parallel pipelines are limited to be hardcoded and requires us to bump the number each month for now until this is improved.
- README updated with correct results links
run-k6
updates:
- The script's underlying results output has been rebuilt to be based on JSON for allowing easier parsing by other scripts. Text table summary is still generated at the end as well so no changes appear on the surface but the underlying JSON results file is now much more useful as raw data.
- Environment, Options and Test files can be passed as just filenames and no longer needing the full path. Test paths can also be passed or even the name of a folder that contains tests.
- Loads in each of the above first to report any missing errors earlier
- Script also shows relative paths instead of full paths in output
- Script now also has a
exclude_tests
option that will exclude any tests from being run if their name matches the string given -
table_print
output has been refactored and moved to therun_k6
library to now return the table as a string variable instead of printing directly to STDOUT - Result file names have been unified onto one naming convention and now includes the environment version if available
- Test times are now all found in the new results json file
- Cleaned up options descriptions to match the above
New scripts \ libraries:
-
ci-report-results-wiki
- Replacement forupdate-wiki
that parses one or more k6 test results and uploads them to the given wiki page in Markdown format. When there's more than one result the script will add a comparison table at the top of the page. Example page - https://gitlab.com/gitlab-org/quality/performance/wikis/test_page2. -
ci-report-test-info-wiki
- Replacement forcollect-test-info
script, this script collects all test info and posts to the give wiki page directly instead of file output. -
ci-report-results-slack
- Refactor ofreport-results-slack
. Script now reports with the JSON results file as well as being able to post a snippet if one results file and a message if none or many. Also updated to usegtp_common
library where appropriate. -
run-gitlab-docker
- Convenience script for running a GitLab docker and waiting for the application to be ready. Designed to be used with our new Performance Images - New library
gpt_common
- A common library designed to be shared across all scripts as appropriate. Contains one functions for now namedmake_http_request
Updated scripts \ libraries:
-
import-project
- Updated to usegtp_common
library where appropriate
Misc Updates
- Updated main CI pipelines to use new scripts above and latest Docker images
- Update gemlist with latest gems
- Various small updates and improvements
Edited by Grant Young