Match stream buffer size in Praefect and Gitaly with the client's
In https://gitlab.com/gitlab-org/gitaly/-/issues/4639#note_1181766622, we've identified a problem where Praefect's memory usage blows up as it buffers responses over sidechannels (fetches).
Praefect is proxying the response from Gitaly to the client. Praefect is faster at consuming the response from Gitaly than the clients are consuming from Praefect. This causes Praefect to buffer the response until the client can consume it. While the clients configure a 256KB buffer, Praefect holds a 16MB buffer. This causes Praefect's memory usage to blow up disproportionally.
We should configure the buffer in Praefect to match the size of the buffer used in the clients so Praefect doesn't end up unnecessarily buffer too much.
Yamux doesn't support configuring a different buffer size for each stream. The gRPC traffic is going over a single stream, which is why we need to be careful about the performance impact of limiting the buffer size. Limiting the buffer size will also limit throughput especially if the latency is higher between Praefect and Gitaly.
To roll this out safely, we should open two connections to each Gitaly from Praefect, one as we do now and one with a buffer size that matches the clients and use a feature flag to roll out the new configuration gradually while monitoring the performance.