Add Results Score
Closes #55 (closed)
Adds in an overall percentage score to signify at a high level how well an environment has performed against expected targets. This is calculated on RPS, namely what RPS was achieved versus the target and then distilled into a percentage.
Known issue endpoint tests are excluded from the calculation in this version for now as the aim here is to show how well a specific environment setup is performing via good endpoints (i.e. if a good endpoint is found to be notable slower than reference then that environment will have a reduced score).
MR also updates docs and several other small improvements. Will lead to a v1.1.1 release of GPT.
Good
* Environment: Docker
* Environment Version: 12.7.0 `680a5284470`
* Option: 20s_20rps
* Date: 2020-02-05
* Run Time: 9m 11.71s (Start: 17:29:09 UTC, End: 17:38:21 UTC)
* GPT Version: v1.1.1
█ Overall Results Score: 94.76%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|------|--------------------|-----------|----------------------|----------------|------------------
api_v4_groups_group | 20/s | 19.0/s (>16.00/s) | 51.45ms | 57.76ms (<500ms) | 100.00% (>95%) | Passed
api_v4_groups_projects | 20/s | 19.05/s (>16.00/s) | 51.27ms | 55.59ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_deploy_keys | 20/s | 19.1/s (>16.00/s) | 21.29ms | 23.70ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_languages | 20/s | 19.1/s (>16.00/s) | 21.21ms | 23.17ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_merge_requests | 20/s | 18.7/s (>12.80/s) | 141.49ms | 185.70ms (<1500ms) | 100.00% (>95%) | Passed¹
api_v4_projects_merge_requests_merge_request | 20/s | 19.1/s (>16.00/s) | 51.40ms | 55.32ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_merge_requests_merge_request_changes | 20/s | 19.2/s (>16.00/s) | 41.86ms | 45.17ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_merge_requests_merge_request_commits | 20/s | 19.1/s (>16.00/s) | 54.40ms | 66.40ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_merge_requests_merge_request_discussions | 20/s | 5.95/s (>8.00/s) | 2341.61ms | 2880.16ms (<4000ms) | 100.00% (>95%) | FAILED¹²
api_v4_projects_pagination_keyset | 20/s | 19.0/s (>16.00/s) | 62.87ms | 70.27ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_pagination_offset | 20/s | 19.0/s (>16.00/s) | 63.54ms | 71.43ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_project | 20/s | 19.05/s (>16.00/s) | 57.96ms | 65.78ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_project_pipelines | 20/s | 19.15/s (>16.00/s) | 33.01ms | 35.19ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_project_search_blobs | 20/s | 18.85/s (>16.00/s) | 102.80ms | 115.86ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_project_services | 20/s | 19.1/s (>16.00/s) | 20.04ms | 21.99ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_repository_branches | 20/s | 1.6/s (>0.80/s) | 6445.03ms | 9275.59ms (<15000ms) | 100.00% (>95%) | Passed¹
api_v4_projects_repository_branches_branch | 20/s | 16.05/s (>16.00/s) | 172.55ms | 270.00ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_repository_commits | 20/s | 19.1/s (>16.00/s) | 51.84ms | 56.14ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_repository_commits_sha | 20/s | 19.1/s (>16.00/s) | 45.07ms | 48.81ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_repository_commits_sha_diff | 20/s | 19.1/s (>16.00/s) | 42.56ms | 46.29ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_repository_commits_sha_signature | 20/s | 19.15/s (>16.00/s) | 37.91ms | 40.56ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_repository_files_file | 20/s | 19.15/s (>16.00/s) | 42.33ms | 45.33ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_repository_files_file_raw | 20/s | 19.1/s (>16.00/s) | 46.73ms | 49.23ms (<500ms) | 100.00% (>95%) | Passed
api_v4_projects_repository_tree | 20/s | 19.15/s (>16.00/s) | 39.21ms | 60.19ms (<500ms) | 100.00% (>95%) | Passed
api_v4_user | 20/s | 19.1/s (>16.00/s) | 19.10ms | 22.46ms (<500ms) | 100.00% (>95%) | Passed
api_v4_users | 20/s | 19.1/s (>16.00/s) | 22.45ms | 24.86ms (<500ms) | 100.00% (>95%) | Passed
Bad
* Environment: Docker
* Environment Version: 12.7.0 `680a5284470`
* Option: 60s_100rps
* Date: 2020-02-05
* Run Time: 2m 4.84s (Start: 17:39:31 UTC, End: 17:41:36 UTC)
* GPT Version: v1.1.0
█ Overall Results Score: 58.58%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
-----------------------|-------|--------------------|-----------|--------------------|----------------|-----------------
api_v4_groups_group | 100/s | 59.62/s (>80.00/s) | 1515.92ms | 1668.36ms (<500ms) | 100.00% (>95%) | FAILED²
api_v4_groups_projects | 100/s | 57.55/s (>80.00/s) | 1571.39ms | 1748.19ms (<500ms) | 100.00% (>95%) | FAILED²
Edited by Grant Young