Skip to content

Set appropriate timeouts for PrometheusClient

Matthias Käppler requested to merge 232468-prometheus-conn-timeout into master

What does this MR do?

In #232468 (closed) we found that currently, Gitlab::HTTP and by extension PrometheusClient do not set a default read or even connection timeout. The default is whatever Ruby's Net::HTTP uses, which is 60 seconds -- that is very long, and in fact a terrible default to use. HTTP services should be considered ready to respond within a few seconds at most, otherwise they should be considered unavailable and retried later.

This MR changes the connection and read timeouts for PrometheusClient only, since this seemed to be the smallest change to make first. It sets it to:

  • connect/open timeout: 5s
  • read timeout: 10s

which are both still generous, but for something not on a performance critical path seemed to be reasonable.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Since this is just configuration, I did not add unit tests.

It is easy to verify though:

  1. Set Prometheus address to somewhere it is not listening, e.g. Settings.prometheus['listen_address'] = '192.178.1.2:9090'
  2. Run Gitlab::PrometheusClient.new(Gitlab::Prometheus::Internal.uri, :allow_local_requests => true).query('abc')

Before: Times out after 60s

After: Times out after 5s

Closes #232468 (closed)

Edited by Matthias Käppler

Merge request reports