Skip to content

Add scenario to create a commit via API

Nailia Iskhakova (OOO) requested to merge 315-create-commit-scenario into master

What does this MR do?

/label Quality ~"Quality:performance"

Closes #315 (closed)

The MR adds scenario to create a commit via API, the addition is related to Q3 KR https://gitlab.com/gitlab-com/Product/-/issues/1345#note_391571094.

Scenario steps:

  • Setup stage: Create a group
  • Setup stage: Create project under the group
  • Setup stage: Create X files for commit request tests during
  • Test Stage: Send create commit requests randomly using files from the step above
  • Teardown stage: Delete a group to clean up the data

Wasn't able to find any metrics on commit sizes, so suggest to iterate if the data below is not realistic.

  • How many files to create in the Setup stage?
    • Started with 10, this number doesn't affect the response time. It affects how many concurrent requests could be sent. If RPS is very high and the number of files is low, more commit create requests will be sent using the same file at the same time which would lead to error Could not update master. Please refresh and try again. We might need to bump up this number if we see such error on some ref envs.
  • Which file type should be used?
    • Went with .md for now.
  • How many actions should be sent in each commit?
    • Currently sending 3 update actions and 1 create with each test request == changing 3 files and creating a new one with each request
    • Tried to double the actions, the TTFB was quite the same.
  • What content should be sent?
    • Currently sending 300 lines, 10_000 characters (* 4 actions) = total size 40 kb. k6 has built-in randomBytes but bytes and integers wouldn't be a realistic content so using a simple string.
    • Tried to triple the content size, response time was on par with above.

Test result example against 5k:

* Environment:                5k
* Environment Version:        13.3.0-pre `9e404d35ecc`
* Option:                     60s_100rps
* Date:                       2020-08-19
* Run Time:                   1m 15.93s (Start: 20:20:05 UTC, End: 20:21:21 UTC)
* GPT Version:                v2.0.6

❯ Overall Results Score: 122.0%

NAME                     | RPS | RPS RESULT       | TTFB AVG | TTFB P90          | REQ STATUS     | RESULT          
-------------------------|-----|------------------|----------|-------------------|----------------|-----------------
scenario_api_new_commits | 1/s | 1.22/s (>0.80/s) | 571.18ms | 619.02ms (<700ms) | 100.00% (>95%) | PASSED

Related issues

#315 (closed)

Checklist

  • Ensure test data for the new endpoint is representative. We recommend using large data ("edge cases") in performance tests. The impact of optimizations is more visible on the big numbers and large data may expose more performance bottlenecks.
  • Fill in @endpoint and @description tags with the information regarding the new test.
  • Create a bug if the endpoint response time is above the performance main target (500 ms).
    • Specify created issue link under @issue tag.
  • Consider specifying the additional @flags:
    • unsafe - required for Unsafe tests. Ensure that data before and after the test is consistent.
    • search - labels Search related tests.
    • dash_url - labels Web tests with the check if the endpoint path has a dash \ redirect.
  • Web tests

Follow-up items

  • Create a bug if the endpoint response time is above the main target(500 ms) across ref envs
  • Bump up the number of files if we see errors on larger ref envs like 50k

Merge request reports