Skip to content

[Controller] Fix endless reconcile loop by filtering out Status update events

Mitchell Nielsen requested to merge 190-fix-endless-reconcile-loop into master

Summary

Uses the GenerationChangedPredicate to skip any events that have no change in the object's metadata.generation field, meaning we no longer see the reconcile loop repeat infinitely.

More information: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/predicate#GenerationChangedPredicate

This required updating the Status logic because if Webservice was not fully running by the time the loop finished, then a new reconcile loop would never trigger (since there was no change in spec) meaning that the CR would never reflect Running instead of Initializing).

Note that this does not address issue #190 (closed), which is specifically related to objects supposedly being "updated" when an update/patch is not expected.

Closes #295 (closed)

Testing

  1. Deploy this branch (TAG=190-fix-endless-reconcile-loop make build_operator)
  2. Apply the basic GitLab CR
  3. In one panel, run watch kubectl -n gitlab-system get gitlab/gitlab and see it says Initializing
  4. In a second panel, run watch kubectl -n gitlab-system get deployment/gitlab-webservice-default
  5. In a third panel, run kubectl logs -n gitlab-system deployment/gitlab-controller-manager -c manager -f and observe that the reconcile loop no longer loops indefinitely
    • The exception here is that the reconcile loop will be intentionally requeued when waiting for core services (i.e. StatefulSets) and Webservice to come up running fully - you'll see log entires communicating this.
  6. When the Webservice deployment is fully running, see the GitLab CR update to Running at this time, logs from the Controller should stop until you intervene (i.e. update the GitLab CR or update a resource like a Deployment).

References

Edited by Mitchell Nielsen

Merge request reports