agent.gitlab.com/id annotation being set by each kubernetes agent in cluster
This bug report was opened on behalf of a GitLab customer.
Due to current limitations on this Agent by not being able to support a cluster-wide agent instance that works across all gitlab root namespaces, and according to the documentation stating to use multiple agents in this scenario, we've deployed an agent for each root namespace that requires integration with each of our clusters.
This resulted in having as many as 10-15 different Gitlab Agent for Kubernetes deployments in some of our clusters.
Unfortunately, these seem to be afflicted by a terrible bug when there's more than one agent in the cluster: every single agent instance will try to take ownership of the annotations of every single Receiver object and its associated secret, leading to a never ending fight to set the agent.gitlab.com/id annotation in both the Receiver and the secret. In some clusters we have hundreds of Receivers and the impact is multiplied by the number of agent instances running on the cluster.
The result of this is that our own Receiver controllers are working in overtime with reconciles to every Receiver happening almost every second, and on top of that the load of our Kubernetes API servers as well as their logging volume is impacted. Right now this represents an additional useless 500Gb logs/day for Receiver/Secret patch operations in the Kubernetes API which in itself is costly, for changes that don't really need to happen.