Improved Web Tests
Closes #142 (closed) #175 (closed)
Related #73 (closed)
MR updates Web Tests to be more representative by calling more of the relevant endpoints a user's browser would for a specific page. Goal of this is to increase most popular Web Controllers coverage while mimicking a user (i.e. load a web page's main URLs together rather targeting controllers and their actions directly). Dynamic or authenticated URLs aren't covered by this MR. In addition, rendering is specifically not covered under this tool (as k6
isn't a browser) and falls under SiteSpeed.
Task required solving several notable problems, specifically how best to handle multiple URL calls in one test to be checked correctly and have enough information to debug each.
MR update notes:
- All Web Tests have been updated to new format based on goals above.
- Test output has been improved to have more useful information utilising more percentiles and k6 tags to show more details about each URL called in tests where there's more than one.
-
k6
is updated to0.26.0
as this is compatible and improvesbatch()
functionality for the new tests - Other small improvements such as updating the Bundler version and syntax to follow new deprecations.
Most popular Web Controller endpoints for the last 24 hours are available to see on our Kibana server. MR aims to investigate and cover all of following based on the criteria above and for areas we have data in (more tests will be added along with API if data is still to be generated, e.g. CI Jobs).
Current progress list:
-
Projects::GitHttpController
(1,671,118) -
Projects::MergeRequests::ContentController
(901,124) -
Projects::PipelinesController
(331,956) -
Projects::CommitsController
(300,211) -
Projects::MergeRequestsController
(280,959) -
Projects::CommitController
(172,599) -
ProjectsController
(144,375) -
Projects::BlobController
(96,827) -
Projects::MergeRequests::DiffsController
(78,064) -
Projects::BranchesController
(28,570) -
Projects::TreeController
(26,889) -
GroupsController
(26,235) -
UsersController
(24,740)
Example output of the new web test format with more details on multiple urls:
█ Web - User
data_received..................: 11 MB 186 kB/s
data_sent......................: 120 kB 2.0 kB/s
group_duration.................: avg=1884.46ms min=200.88ms med=1998.48ms max=2107.73ms p(50)=1998.48ms p(75)=2003.40ms p(90)=2009.62ms p(95)=2025.28ms count=577
http_req_blocked...............: avg=3.93ms min=0.00ms med=0.01ms max=114.81ms p(50)=0.01ms p(75)=0.01ms p(90)=0.01ms p(95)=0.02ms count=1154
http_req_connecting............: avg=3.93ms min=0.00ms med=0.00ms max=114.71ms p(50)=0.00ms p(75)=0.00ms p(90)=0.00ms p(95)=0.00ms count=1154
http_req_duration..............: avg=220.49ms min=143.62ms med=216.41ms max=391.58ms p(50)=216.41ms p(75)=284.58ms p(90)=290.70ms p(95)=307.68ms count=1154
http_req_receiving.............: avg=56.05ms min=0.03ms med=1.30ms max=153.65ms p(50)=1.30ms p(75)=112.59ms p(90)=113.37ms p(95)=113.73ms count=1154
http_req_sending...............: avg=0.02ms min=0.01ms med=0.02ms max=0.15ms p(50)=0.02ms p(75)=0.02ms p(90)=0.03ms p(95)=0.04ms count=1154
http_req_tls_handshaking.......: avg=0.00ms min=0.00ms med=0.00ms max=0.00ms p(50)=0.00ms p(75)=0.00ms p(90)=0.00ms p(95)=0.00ms count=1154
http_req_waiting...............: avg=164.42ms min=143.55ms med=166.14ms max=279.16ms p(50)=166.14ms p(75)=172.46ms p(90)=181.29ms p(95)=200.37ms count=1154
✓ { endpoint:calender.json }...: avg=151.93ms min=143.55ms med=149.15ms max=255.43ms p(50)=149.15ms p(75)=150.85ms p(90)=159.21ms p(95)=166.94ms count=577
✓ { endpoint:root }............: avg=176.91ms min=163.88ms med=171.91ms max=279.16ms p(50)=171.91ms p(75)=175.46ms p(90)=194.36ms p(95)=210.56ms count=577
✓ http_reqs......................: 1154 19.233265/s
✓ { endpoint:calender.json }...: 577 9.616633/s
✓ { endpoint:root }............: 577 9.616633/s
iteration_duration.............: avg=1881.22ms min=0.14ms med=1998.47ms max=2107.74ms p(50)=1998.47ms p(75)=2003.39ms p(90)=2009.63ms p(95)=2025.28ms count=578
iterations.....................: 577 9.616633/s
✓ successful_requests............: 100.00% ✓ 1154 ✗ 0
vus............................: 1 min=1 max=20
vus_max........................: 20 min=20 max=20
All k6 tests have finished after 61.3s!
Results summary:
* Environment: 10k
* Version: 12.7.0-pre `60ddd8fb177`
* Option: 60s_200rps
* Date: 2020-01-29
* Run Time: 1m 1.3s (Start: 15:04:53 UTC, End: 15:05:54 UTC)
NAME | RPS | RPS RESULT | REQ AVG | REQ P95 | REQ STATUS | RESULT
---------|------|--------------------|----------|----------|----------------|----------------
web_user | 20/s | 19.23/s (>16.00/s) | 164.42ms | 200.37ms | 100.00% (>95%) | Passed
Results files:
k6/results/10k_v12-7-0-pre_2020-01-29_150453/10k_v12-7-0-pre_2020-01-29_150453_results_output.log
k6/results/10k_v12-7-0-pre_2020-01-29_150453/10k_v12-7-0-pre_2020-01-29_150453_results.json