Gitlab Managed cert-manager installation to Kubernetes silently failed to deploy all required resources
Summary
Gitlab Managed cert-manager installation to Kubernetes silently failed to deploy all required resources
Steps to reproduce
Not sure if reproducible, as circumstances were during a production cluster upgrade. Steps that led to the issue:
- had a Gitlab managed GKE kubernetes cluster in eu-west-6 that had cert-manager installed on it when it was created, August 7 2019
- intending to upgrade cert-manager, went to the gitlab cluster page and 'Uninstall'd it
- verified visually that cert-manager pods, configs and secrets were removed from the cluster
- went back to gitlab and 'Install'd cert-manager. Install button greyed out, then a little while later, came back as an 'Uninstall' button, suggesting installation was successful.
- verified visually that new (and slightly different) cert-manager pods, configs and secrets had been deployed to the cluster. Checked pod logs, all seemed ok initially
- a little later, a new service was deployed, and the ingress started reporting a warning 'clusterissuer resource letsencrypt-prod not found'
- on inspection via kubectl, indeed no ClusterIssuer resources were present on the cluster.
I was able to recover by again uninstalling, then installing cert-manager. Now kubectl indicates that a ClusterIssuer resource called letsencrypt-prod exists, and ingresses are being issued with letsencrypt certs.
What is the current bug behavior?
Clicking 'Install' on the cert-manager managed kubernetes app page seems to report success even though not all required resources have been created on the cluster.
What is the expected correct behavior?
Ideally, installation should not fail ;)
Alternatively, if installation of a kubernetes managed app does not complete, the failure should be reported back to the gitlab cluster management dashboard; possibly the failure should be rolled back.
Logs of gitlab managed application installation should be available to allow for troubleshooting and identification of problems.
Output of checks
This bug happens on gitlab.com