Skip to content

Further tweak for new commits API scenario

Nailia Iskhakova requested to merge commit-api-scenario-tweak into master

Related to #315 (closed)

The MR changes the scenario_api_new_commits.js scenario a bit:

  • Push commits with start_branch parameter to create commits_count branches in the setup stage. Previously all commits hit a single master branch that was also not very realistic.
  • Increase commits_count for higher RPS

On larger environment with higher RPS throughput we see errors like Could not update master. Please refresh and try again. or Branch diverged., example pipeline against 25k - https://gitlab.com/gitlab-org/quality/performance/-/pipelines/182101162. This appears to be related to the fact that with higher throughput like 4 RPS we have a larger amount of requests that are hitting the same files concurrently. Now we're mitigating this issue by creating several branches to push commits to. Though it's still possible to hit this error due to the fact that we're randomly pushing commits to branches and concurrency could still appear, but on the much lower scale.

Results against 25k with updated test:

█ Results summary
* Environment:                25k
* Environment Version:        13.4.0-pre `e8c7d7ed255`
* Option:                     60s_500rps
* Date:                       2020-08-26
* Run Time:                   1m 11.2s (Start: 16:07:03 UTC, End: 16:08:14 UTC)
* GPT Version:                v2.0.6
❯ Overall Results Score: 102.24%
NAME                     | RPS | RPS RESULT       | TTFB AVG | TTFB P90          | REQ STATUS    | RESULT         
-------------------------|-----|------------------|----------|-------------------|---------------|----------------
scenario_api_new_commits | 5/s | 5.13/s (>4.00/s) | 469.19ms | 490.92ms (<700ms) | 99.65% (>95%) | Passed

Follow-ups

  • Clarify with Gitaly team if this limit of concurrent updates is expected.

Merge request reports