Add scenario to create a commit via API
What does this MR do?
/label Quality ~"Quality:performance"
Closes #315 (closed)
The MR adds scenario to create a commit via API, the addition is related to Q3 KR https://gitlab.com/gitlab-com/Product/-/issues/1345#note_391571094.
Scenario steps:
- Setup stage: Create a group
- Setup stage: Create project under the group
- Setup stage: Create
X
files for commit request tests during - Test Stage: Send create commit requests randomly using files from the step above
- Teardown stage: Delete a group to clean up the data
Wasn't able to find any metrics on commit sizes, so suggest to iterate if the data below is not realistic.
- How many files to create in the Setup stage?
- Started with 10, this number doesn't affect the response time. It affects how many concurrent requests could be sent. If RPS is very high and the number of files is low, more
commit create
requests will be sent using the same file at the same time which would lead to errorCould not update master. Please refresh and try again
. We might need to bump up this number if we see such error on some ref envs.
- Started with 10, this number doesn't affect the response time. It affects how many concurrent requests could be sent. If RPS is very high and the number of files is low, more
- Which file type should be used?
- Went with
.md
for now.
- Went with
- How many actions should be sent in each commit?
- Currently sending 3 update actions and 1 create with each test request == changing 3 files and creating a new one with each request
- Tried to double the actions, the TTFB was quite the same.
- What content should be sent?
- Currently sending 300 lines, 10_000 characters (* 4 actions) = total size 40 kb. k6 has built-in randomBytes but bytes and integers wouldn't be a realistic content so using a simple string.
- Tried to triple the content size, response time was on par with above.
Test result example against 5k:
* Environment: 5k
* Environment Version: 13.3.0-pre `9e404d35ecc`
* Option: 60s_100rps
* Date: 2020-08-19
* Run Time: 1m 15.93s (Start: 20:20:05 UTC, End: 20:21:21 UTC)
* GPT Version: v2.0.6
❯ Overall Results Score: 122.0%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
-------------------------|-----|------------------|----------|-------------------|----------------|-----------------
scenario_api_new_commits | 1/s | 1.22/s (>0.80/s) | 571.18ms | 619.02ms (<700ms) | 100.00% (>95%) | PASSED
Related issues
Checklist
-
Ensure test data for the new endpoint is representative. We recommend using large data ("edge cases") in performance tests. The impact of optimizations is more visible on the big numbers and large data may expose more performance bottlenecks. -
Fill in @endpoint
and@description
tags with the information regarding the new test. -
Create a bug if the endpoint response time is above the performance main target (500 ms)
.- Specify created issue link under
@issue
tag.
- Specify created issue link under
-
Consider specifying the additional @flags
:-
unsafe
- required for Unsafe tests. Ensure that data before and after the test is consistent. -
search
- labels Search related tests. -
dash_url
- labels Web tests with the check if the endpoint path has a dash \ redirect.
-
-
Web tests - Specify all called controller methods under the
@description
. - Consider creating MR to add the new test to GitLab Performance SiteSpeed.
- Specify all called controller methods under the
Follow-up items
-
Create a bug if the endpoint response time is above the main target(500 ms) across ref envs -
Bump up the number of files if we see errors on larger ref envs like 50k