Single Issue API performance regression degraded by 8000ms after upgrade or fresh installs
The Single Issue API has degraded substantially today and is now well over our new performance target of 500ms.
This looks to be new with the same endpoint performing well on yesterday's Nightly package compared to today's (10k used as an example but all environments are now showing similar behavior):
10k (Yesterday - 2020-05-21):
* Environment: 10k
* Environment Version: 13.0.0-pre `f45762cdbd6`
* Option: 60s_200rps
* Date: 2020-05-21
* Run Time: 59m 39.61s (Start: 01:23:16 UTC, End: 02:22:55 UTC)
* GPT Version: v1.3.1
❯ Overall Results Score: 97.36%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|-------|----------------------|------------|-----------------------|-----------------|-------
api_v4_projects_issues_issue | 200/s | 191.23/s (>160.00/s) | 189.71ms | 249.03ms (<3000ms) | 100.00% (>95%) | Passed
10k (Today - 2020-05-22):
* Environment: 10k
* Environment Version: 13.0.0-pre `1ded6098e8d`
* Option: 60s_200rps
* Date: 2020-05-22
* Run Time: 59m 33.52s (Start: 01:29:30 UTC, End: 02:29:04 UTC)
* GPT Version: v1.3.1
❯ Overall Results Score: 94.95%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | REQ STATUS | RESULT
---------------------------------------------------------|-------|----------------------|------------|-----------------------|-----------------|--------
api_v4_projects_issues_issue | 200/s | 19.52/s (>160.00/s) | 8630.76ms | 10337.89ms (<3000ms) | 100.00% (>95%) | FAILED³
As shown the endpoint today is responding in just over 10s compared to yesterday where it was around 200ms on the same environment. As part of the new performance targets this endpoint is unfortunately now well over our TTFB target.
As the endpoint was performing well yesterday the task here is to restore it's performance.
Note we do have another issue open against this endpoint but it looks to be different with that one being more sporadic and having a different metrics profile.