Projects list API gitaly performance
There is a strong correlation between poor performing requests to /api/:version/projects
and poor performance from Gitaly.
Query: json.path.keyword : "/api/v4/projects" and json.method.keyword : "GET" and json.duration_s < 15
At the 5.5s mark there is a significant contribution from Gitaly. At 4.5s Gitaly is 20% of the response, but by 5.5s it's about 70%.
The correlation holds as we zoom out, though I'm not sure how we end up with such long queries without timeouts.
We can illustrate the contribution from Gitaly compared to the total duration:
Average performance from Gitaly is much better. But, we care about the 95% performance because our error budget is affected by the extremities.
Contribution of queries where Gitaly is over/under 3 seconds (an arbitrary threshold) is: