Horizontal Pod Autoscalers cannot connect to metrics server
HPAs need to be able to contact the metrics server to properly scale their pods. This connection is broken right now, so pods backed by HPAs are continuously created and destroyed.
Warning FailedGetResourceMetric 12m (x6 over 13m) horizontal-pod-autoscaler did not receive metrics for any ready pods
Warning FailedComputeMetricsReplicas 12m (x6 over 13m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first
error is: failed to get cpu utilization: did not receive metrics for any ready pods
The problem here may be that the metrics server is auth protected, see https://gitlab.com/gitlab-org/gl-openshift/gitlab-operator/-/blob/master/config/default/kustomization.yaml#L31. Removing this protection may cause the HPAs to work, but we need to investigate the consequences of running the metrics server without authentication.
In the meantime we are disabling HPAs, https://gitlab.com/gitlab-org/gl-openshift/gitlab-operator/-/merge_requests/107.
Edited by Dustin Collins