Document that Gitaly and Praefect require ALPN for TLS in GitLab 17.4

What does this MR do and why?

One customer attempted to upgrade from v17.3.1 to v17.4.3, but ran into the error:

FAIL: 14:connections to all backends failing; last error: Unavailable: ipv4:<ip-address>:3305: socket closed

The customer had a NGINX load balancer in front of Praefect, and after the upgrade the GitLab Rails client was not able to talk to Praefect and Gitaly nodes. NGINX was configured to use ngx_stream_core_module with proxy_ssl enabled. This meant that NGINX negotiated TLS with the client and Praefect nodes instead of forwarding the traffic as-is.

GitLab 17.4 upgraded grpc-go from v1.65.0 to v1.67.1, which now enforces ALPN for TLS connections (https://github.com/grpc/grpc-go/releases/tag/v1.67.0):

In accordance with RFC 7540, clients and servers will now reject
TLS connections that don't support ALPN. This can be disabled by
setting the environment variable GRPC_ENFORCE_ALPN_ENABLED to false
(case insensitive). Please file a bug if you encounter any issues with
this behavior. The environment variable to revert this behavior will
be removed in an upcoming release

By setting GRPC_GO_LOG_VERBOSITY_LEVEL=99 and GRPC_GO_LOG_SEVERITY_LEVEL=info, we were able to see that Praefect was failing to accept the TLS connection due a missing selected ALPN property.

This commit documents how to set GRPC_ENFORCE_ALPN_ENABLED to work around this issue and reiterates that a TCP load balancer should be used in front of Praefect. We managed to fix the problem by using NGINX's built-in gRPC load balancer (https://blog.nginx.org/blog/nginx-1-13-10-grpc), but we cannot recommend this yet.

References

https://gitlab.com/gitlab-com/request-for-help/-/issues/2127

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before	After

How to set up and validate locally

See https://gitlab.com/gitlab-com/request-for-help/-/issues/2127.