Allow configuration of TCP timeouts in rails' database.yml
libpq
allows you to tune low level TCP settings via connection parameters (see https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-KEEPALIVES). This enables you to make your database connections more resilient to disconnects or other network issues, preventing long blocking calls.
We specifically encountered a situation where a set of wrong iptables
rules made some of our database replicas silently drop TCP packages, which caused ActiveRecord
connections to block indefinitely and brought down the whole site (see https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/8528). We established that for our specific case a tcp_user_timeout
alone would've allowed the requests to timeout properly, but the keepalives
parameters are also probably nice to have (see https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/8600#note_410182134).
Currently there's no way to specify these configuration parameters in gitlab-rails' database.yml
. We should add that capability.