Skip to content

Return 503 instead of 429 when Gitaly returns ResourceExhausted

Quang-Minh Nguyen requested to merge qmnguyen0711/convert-429-to-503-when-handle-resource-exhausted into master

What does this MR do and why?

In !119054 (merged), GitLab Rails propagates Gitaly's ResourceExhausted response codes back to users. It returns a friendly message and a 429 status code. This HTTP status turns out to be confusing. It's about the user sending too many requests, aka the user's responsibility to back off. Unfortunately, it's not the case here.

Gitaly returns ResourceExhausted response codes when it wants to push back traffic as it's at capacity. Gitaly applies different limiters, mostly concurrency limiting, not rate limiting. As a result, 429 does not reflect the system's actual behavior.

This commit switches the returned HTTP code to 503. This status code is appropriate for the described case. The Retry-After header stays intact, as it's a supported use case: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After

Screenshots or screen recordings

N/A

How to set up and validate locally

  • Setup Gitaly concurrency limiting. This limiter limits the concurrency of tag-finding operations to 1.
[[concurrency]]
rpc = "/gitaly.CommitService/FindCommits"
max_per_repo = 1
max_queue_size = 1
max_queue_wait = "1s"
  • Restart Gitaly
  • Find a project with a considerable amount of tags, or we can create a massive amount of them.
  • Access a project tag list via Web and API. The first request is expected to work as usual.
  • At the same time, issue multiple similar requests. They are expected to fail with 503 status

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports