[Feature flag] Enable default HTTP timeouts

What

Verify and remove the :http_default_timeouts feature flag

Owners

• Team: groupmemory
• Most appropriate slack channel to reach out to: #g_memory
• Best individual to reach out to: @mkaeppler

Expectations

What are we expecting to happen?

The expected but somewhat uninformed case (due to a lack of data) is that nothing will happen, since timeouts are still quite generous.

The best case would be that Sidekiq job queuing time for webhook jobs will go down, since we would not spend 60s anymore waiting for an HTTP service to respond. Moreover, the number of HTTP request timeouts in these jobs should stay flat.

The worst case would be that the number of timeouts in these jobs spikes up; this means that we chose the defaults too aggressively and that we are now failing to invoke webhooks. Sidekiq job queue time therefore should go down.

What might happen if this goes wrong?

We might be timing out earlier, and hence, more often, when invoking HTTP services such as webhooks.

What can we monitor to detect problems with this?

Breakdown of timeout errors we are recording in logs: https://log.gprd.gitlab.net/goto/5143fe55b85761eabe3c6bbd4e593d54

I don't think Sentry is useful to monitor in this case since we're catching these sorts of timeouts and log them instead, but can't hurt to look for increases in something like https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+Net%3A%3AOpenTimeout

Roll Out Steps

• Enable on staging
• Test on staging
• [-] Ensure that documentation has been updated (this is not a documented feature and a changelog entry should suffice)
• Enable on GitLab.com for individual groups/projects listed above and verify behaviour
• Coordinate a time to enable the flag with #production and #g_delivery on slack.
• Announce on the issue an estimated time this will be enabled on GitLab.com
• Enable on GitLab.com by running chatops command in #production
• Cross post chatops slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
• Announce on the issue that the flag has been enabled
• Remove feature flag and add changelog entry
• After the flag removal is deployed, clean up the feature flag by running chatops command in #production channel