Skip to content

Issues List API degrades under load

Summary

Performance testing is showing that Issues List API degrades under load on all reference architectures except for 10k:

NAME                     | RPS  | RPS RESULT         | TTFB AVG  | TTFB P90             | REQ STATUS     | RESULT
-------------------------|------|--------------------|-----------|----------------------|----------------|-------
api_v4_projects_issues   | 40/s | 36.5/s (>19.20/s)  | 846.76ms  | 1134.65ms (<2000ms)  | 100.00% (>99%) | Passed
█ Results summary
* Environment:                50k
* Environment Version:        12.9.0-pre `062dcdf9199`
* Option:                     60s_1000rps
* Date:                       2020-03-16
* Run Time:                   1h 3m 40.31s (Start: 09:20:29 UTC, End: 10:24:10 UTC)
* GPT Version:                v1.2.3
NAME                    | RPS    | RPS RESULT           | TTFB AVG  | TTFB P90             | REQ STATUS     | RESULT 
------------------------|--------|----------------------|-----------|----------------------|----------------|--------
api_v4_projects_issues  | 1000/s | 944.23/s (>720.00/s) | 844.44ms  | 1018.05ms (<2000ms)  | 100.00% (>95%) | Passed 

Detailed Stats:
  █ API - Issues List
     data_received..............: 2.7 GB  46 MB/s
     data_sent..................: 11 MB   176 kB/s
     group_duration.............: avg=938.11ms min=139.70ms med=925.06ms max=50024.40ms p(90)=1112.61ms p(95)=1182.55ms
     http_req_blocked...........: avg=0.02ms   min=0.00ms   med=0.00ms   max=16.52ms    p(90)=0.01ms    p(95)=0.01ms   
     http_req_connecting........: avg=0.01ms   min=0.00ms   med=0.00ms   max=16.42ms    p(90)=0.00ms    p(95)=0.00ms   
     http_req_duration..........: avg=845.49ms min=139.39ms med=829.70ms max=49924.66ms p(90)=1019.16ms p(95)=1087.39ms
     http_req_receiving.........: avg=1.02ms   min=0.08ms   med=0.98ms   max=15.40ms    p(90)=1.20ms    p(95)=1.32ms   
     http_req_sending...........: avg=0.03ms   min=0.01ms   med=0.02ms   max=5.24ms     p(90)=0.05ms    p(95)=0.06ms   
     http_req_tls_handshaking...: avg=0.00ms   min=0.00ms   med=0.00ms   max=0.00ms     p(90)=0.00ms    p(95)=0.00ms   
   ✓ http_req_waiting...........: avg=844.44ms min=139.18ms med=828.66ms max=49923.71ms p(90)=1018.05ms p(95)=1086.31ms
   ✓ http_reqs..................: 56654   944.231763/s
     iteration_duration.........: avg=938.11ms min=0.15ms   med=925.06ms max=50024.41ms p(90)=1112.62ms p(95)=1182.57ms
     iterations.................: 56654   944.231763/s
   ✓ successful_requests........: 100.00% ✓ 56654  ✗ 0     
     vus........................: 1       min=1    max=1000
     vus_max....................: 1000    min=1000 max=1000
  • 10k:
NAME RPS RPS RESULT TTFB AVG TTFB P90 REQ STATUS RESULT
api_v4_projects_issues 200/s 192.03/s (>144.00/s) 186.10ms 235.92ms (<500ms) 100.00% (>95%) Passed

Steps to reproduce

  1. Check out the Performance Tool
  2. Run the specific test with the run-k6 command. For example against the 50k environment you would run this following from the project root: ./run-k6 -e environments/50k.json -o 60s_1000rps.json -t api_v4_projects_issues.js. You will need an ACCESS_TOKEN for this endpoint as well.
  3. If you're seeking to run the test against your own environment the Tool's documentation has details on how to achieve this.

What is the current bug behavior?

The results above show that the Issues List API has a P90 of 1018.05ms. This was tested on our 50k Reference Architecture with a RPS target of 1000/s. It targeted our own gitlabhq project that has 7k issues.

What is the expected correct behavior?

As per our new incoming performance targets this endpoint is slightly above our third tier target of 1000ms. Task is to improve the endpoint's performance to meet our performance target (<500ms).

Edited by Nailia Iskhakova