Remove spawn token limiter
We have a SpawnToken setup to limit the maximum number of child processes that can be forked in parallel.
This was originally added in 2017 when contention on Go's syscall.ForkLock
caused performance issues in production. However, highly loaded self-managed environments may exceed the 10-process default limit and see extended delays in forking new processes, causing significant performance issues. Generally, the node is already under heavy load when this happens, but not always.
With Go 1.21, scheduled for August 2023, syscall.ForkLock
should become friendlier to parallel forking with https://github.com/golang/go/issues/23558 / https://go-review.googlesource.com/c/go/+/421441.
We performed a series of benchmarking to prove the effectiveness of the new mechanism.
Go 1.21 is now the minimum Go version for Gitaly. It was used broadly in production since January/February 2024 right before Go 1.20's end of life (MR). Let's look at spawn token metrics across the cluster in the last 6 months:
- Spawn token queue length:
- Spawn token timeouts:
Overall, the spawn token queue has been almost flat for months. We observed no significant spawn token congestion. The time window kinda matches the point of time Go 1.21 was rolled out. So, I believe that spawn tokens could be deemed redundant now. Let's remove that limiter to simplify the code base.