Switch from startupProbe to livenessProbe in Modsecurity
What does this MR do?
Related to #37127 (closed)
In this MR we are switching from startupProbe to livenessProbe for modsecurity sidecar. startupProbe is not yet supported in current clusters.
What will happen when audit.log is missing?
⋊> ~ kubectl get events -n gitlab-managed-apps
0s Warning Unhealthy pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Liveness probe failed: ls: /var/log/modsec/audit.log: No such file or directory
0s Warning Unhealthy pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Liveness probe failed: ls: /var/log/modsec/audit.log: No such file or directory
0s Warning Unhealthy pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Liveness probe failed: ls: /var/log/modsec/audit.log: No such file or directory
0s Normal Killing pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Container modsecurity-log failed liveness probe, will be restarted
0s Warning Unhealthy pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Liveness probe failed: ls: /var/log/modsec/audit.log: No such file or directory
0s Normal Pulling pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Pulling image "busybox"
0s Normal Pulled pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Successfully pulled image "busybox"
0s Normal Created pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Created container modsecurity-log
0s Normal Started pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Started container modsecurity-log
0s Normal Pulling pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Pulling image "busybox"
0s Normal Pulled pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Successfully pulled image "busybox"
0s Normal Created pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Created container modsecurity-log
0s Normal Started pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Started container modsecurity-log
0s Warning BackOff pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Back-off restarting failed container
0s Warning BackOff pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Back-off restarting failed container
0s Warning BackOff pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Back-off restarting failed container
0s Normal Pulling pod/ingress-nginx-ingress-controller-6cfd695967-6c8mg Pulling image "busybox"
61m Normal Killing pod/ingress-nginx-ingress-controller-657ddfc567-sz5pd Stopping container nginx-ingress-controller
61m Normal Killing pod/ingress-nginx-ingress-controller-657ddfc567-sz5pd Stopping container modsecurity-log
60m Warning Unhealthy pod/ingress-nginx-ingress-controller-657ddfc567-sz5pd Readiness probe failed: HTTP probe failed with statuscode: 500
60m Warning Unhealthy pod/ingress-nginx-ingress-controller-657ddfc567-sz5pd Liveness probe failed: HTTP probe failed with statuscode: 500
60m Warning Unhealthy pod/ingress-nginx-ingress-controller-657ddfc567-sz5pd Liveness probe failed: Get http://172.17.0.8:10254/healthz: dial tcp 172.17.0.8:10254: connect: connection refused
61m Normal Scheduled pod/ingress-nginx-ingress-controller-657ddfc567-ts6rp Successfully assigned gitlab-managed-apps/ingress-nginx-ingress-controller-657ddfc567-ts6rp to minikube
61m Normal Pulled pod/ingress-nginx-ingress-controller-657ddfc567-ts6rp Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0" already present on machine
61m Normal Created pod/ingress-nginx-ingress-controller-657ddfc567-ts6rp Created container nginx-ingress-controller
61m Normal Started pod/ingress-nginx-ingress-controller-657ddfc567-ts6rp Started container nginx-ingress-controller
61m Normal Pulling pod/ingress-nginx-ingress-controller-657ddfc567-ts6rp Pulling image "busybox"
60m Normal Pulled pod/ingress-nginx-ingress-controller-657ddfc567-ts6rp Successfully pulled image "busybox"
60m Normal Created pod/ingress-nginx-ingress-controller-657ddfc567-ts6rp Created container modsecurity-log
60m Normal Started pod/ingress-nginx-ingress-controller-657ddfc567-ts6rp Started container modsecurity-log
61m Normal SuccessfulCreate replicaset/ingress-nginx-ingress-controller-657ddfc567 Created pod: ingress-nginx-ingress-controller-657ddfc567-ts6rp
60m Normal CREATE configmap/ingress-nginx-ingress-controller ConfigMap gitlab-managed-apps/ingress-nginx-ingress-controller⋊> ~ kubectl get pods -n gitlab-managed-apps -w -o wide
ingress-nginx-ingress-controller-6cfd695967-6c8mg 2/2 Running 0 12s 172.17.0.8 minikube <none> <none>
ingress-nginx-ingress-controller-56d445f56f-zdbjp 2/2 Terminating 1 7m33s 172.17.0.7 minikube <none> <none>
ingress-nginx-ingress-controller-56d445f56f-zdbjp 1/2 Terminating 1 7m55s 172.17.0.7 minikube <none> <none>
ingress-nginx-ingress-controller-56d445f56f-zdbjp 0/2 Terminating 1 8m33s 172.17.0.7 minikube <none> <none>
ingress-nginx-ingress-controller-56d445f56f-zdbjp 0/2 Terminating 1 8m43s 172.17.0.7 minikube <none> <none>
ingress-nginx-ingress-controller-56d445f56f-zdbjp 0/2 Terminating 1 8m43s 172.17.0.7 minikube <none> <none>
ingress-nginx-ingress-controller-6cfd695967-6c8mg 1/2 Error 1 2m58s 172.17.0.8 minikube <none> <none>
ingress-nginx-ingress-controller-6cfd695967-6c8mg 1/2 Error 2 3m1s 172.17.0.8 minikube <none> <none>
ingress-nginx-ingress-controller-6cfd695967-6c8mg 1/2 CrashLoopBackOff 2 3m2s 172.17.0.8 minikube <none> <none>
ingress-nginx-ingress-controller-6cfd695967-6c8mg 2/2 Running 3 3m22s 172.17.0.8 minikube <none> <none>Installation log:
⋊> ~ kubectl describe pod/ingress-nginx-ingress-controller-65dc55d79b-pzzb8 -n gitlab-managed-apps
Name: ingress-nginx-ingress-controller-65dc55d79b-pzzb8
Namespace: gitlab-managed-apps
Priority: 0
Node: minikube/192.168.64.7
Start Time: Wed, 15 Apr 2020 13:53:02 +0200
Labels: app=nginx-ingress
component=controller
pod-template-hash=65dc55d79b
release=ingress
Annotations: prometheus.io/port: 10254
prometheus.io/scrape: true
Status: Running
IP: 172.17.0.9
Controlled By: ReplicaSet/ingress-nginx-ingress-controller-65dc55d79b
Containers:
nginx-ingress-controller:
Container ID: docker://e4f55e16855962bc7e7a97e7f46813485cc7cb1fa0fad9592aae54ea9e07e2cc
Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
Image ID: docker-pullable://quay.io/kubernetes-ingress-controller/nginx-ingress-controller@sha256:ca2eee26afd16dc052c7950f13df6c906be279de64d990e88383c9f123556e06
Ports: 80/TCP, 443/TCP
Host Ports: 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--default-backend-service=gitlab-managed-apps/ingress-nginx-ingress-default-backend
--election-id=ingress-controller-leader
--ingress-class=nginx
--configmap=gitlab-managed-apps/ingress-nginx-ingress-controller
State: Running
Started: Wed, 15 Apr 2020 13:53:03 +0200
Ready: True
Restart Count: 0
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-ingress-controller-65dc55d79b-pzzb8 (v1:metadata.name)
POD_NAMESPACE: gitlab-managed-apps (v1:metadata.namespace)
Mounts:
/etc/nginx/modsecurity/modsecurity.conf from modsecurity-template-volume (rw,path="modsecurity.conf")
/var/log/modsec from modsecurity-log-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-ingress-token-tkz6f (ro)
modsecurity-log:
Container ID: docker://ae0af4e86815f1ccd4e5a3465664329b5a11105571385bbf6070cc5de232be9f
Image: busybox
Image ID: docker-pullable://busybox@sha256:89b54451a47954c0422d873d438509dae87d478f1cb5d67fb130072f67ca5d25
Port: <none>
Host Port: <none>
Args:
/bin/sh
-c
tail -f /var/log/modsec/audit.log
State: Running
Started: Wed, 15 Apr 2020 13:53:06 +0200
Ready: True
Restart Count: 0
Liveness: exec [ls /var/log/modsec/audit.log] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/log/modsec from modsecurity-log-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-ingress-token-tkz6f (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
modsecurity-template-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: ingress-nginx-ingress-controller
Optional: false
modsecurity-log-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
ingress-nginx-ingress-token-tkz6f:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-ingress-token-tkz6f
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned gitlab-managed-apps/ingress-nginx-ingress-controller-65dc55d79b-pzzb8 to minikube
Normal Pulled 23m kubelet, minikube Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0" already present on machine
Normal Created 23m kubelet, minikube Created container nginx-ingress-controller
Normal Started 23m kubelet, minikube Started container nginx-ingress-controller
Normal Pulling 23m kubelet, minikube Pulling image "busybox"
Normal Pulled 23m kubelet, minikube Successfully pulled image "busybox"
Normal Created 23m kubelet, minikube Created container modsecurity-log
Normal Started 23m kubelet, minikube Started container modsecurity-log
⋊> ~ kubectl logs install-ingress -f -n gitlab-managed-apps
+ helm init --upgrade
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.
Tiller (the Helm server-side component) has been updated to gcr.io/kubernetes-helm/tiller:v2.16.3 .
+ seq 1 30
+ helm version --tls --tls-ca-cert /data/helm/ingress/config/ca.pem --tls-cert /data/helm/ingress/config/cert.pem --tls-key /data/helm/ingress/config/key.pem
Client: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
+ s=0
+ break
+ exit 0
+ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.
+ helm upgrade ingress stable/nginx-ingress --install --atomic --cleanup-on-fail --reset-values --tls --tls-ca-cert /data/helm/ingress/config/ca.pem --tls-cert /data/helm/ingress/config/cert.pem --tls-key /data/helm/ingress/config/key.pem --version 1.29.7 --set 'rbac.create=true,rbac.enabled=true' --namespace gitlab-managed-apps -f /data/helm/ingress/config/values.yaml
Release "ingress" does not exist. Installing it now.
NAME: ingress
E0415 11:53:16.179290 1 portforward.go:372] error copying from remote stream to local connection: readfrom tcp4 127.0.0.1:35675->127.0.0.1:59214: write tcp4 127.0.0.1:35675->127.0.0.1:59214: write: broken pipe
LAST DEPLOYED: Wed Apr 15 11:53:01 2020
NAMESPACE: gitlab-managed-apps
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
ingress-nginx-ingress 14s
==> v1/ClusterRoleBinding
NAME AGE
ingress-nginx-ingress 14s
==> v1/ConfigMap
NAME AGE
ingress-nginx-ingress-controller 14s
==> v1/Deployment
NAME AGE
ingress-nginx-ingress-controller 14s
ingress-nginx-ingress-default-backend 14s
==> v1/Pod(related)
NAME AGE
ingress-nginx-ingress-controller-65dc55d79b-pzzb8 14s
ingress-nginx-ingress-default-backend-7789656965-7m2zg 14s
==> v1/Role
NAME AGE
ingress-nginx-ingress 14s
==> v1/RoleBinding
NAME AGE
ingress-nginx-ingress 14s
==> v1/Service
NAME AGE
ingress-nginx-ingress-controller 14s
ingress-nginx-ingress-default-backend 14s
==> v1/ServiceAccount
NAME AGE
ingress-nginx-ingress 14s
ingress-nginx-ingress-backend 14s
NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace gitlab-managed-apps get services -o wide -w ingress-nginx-ingress-controller'
An example Ingress that makes use of the controller:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name: example
namespace: foo
spec:
rules:
- host: www.example.com
http:
paths:
- backend:
serviceName: exampleService
servicePort: 80
path: /
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- www.example.com
secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tls⋊> ~ kubectl get events -n gitlab-managed-apps -w
0s Normal Scheduled pod/install-ingress Successfully assigned gitlab-managed-apps/install-ingress to minikube
0s Normal Pulling pod/install-ingress Pulling image "registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.16.3-kube-1.13.12"
0s Normal Pulled pod/install-ingress Successfully pulled image "registry.gitlab.com/gitlab-org/cluster-integration/helm-install-image/releases/2.16.3-kube-1.13.12"
0s Normal Created pod/install-ingress Created container helm
0s Normal Started pod/install-ingress Started container helm
0s Normal ScalingReplicaSet deployment/ingress-nginx-ingress-controller Scaled up replica set ingress-nginx-ingress-controller-65dc55d79b to 1
0s Normal SuccessfulCreate replicaset/ingress-nginx-ingress-controller-65dc55d79b Created pod: ingress-nginx-ingress-controller-65dc55d79b-pzzb8
0s Normal ScalingReplicaSet deployment/ingress-nginx-ingress-default-backend Scaled up replica set ingress-nginx-ingress-default-backend-7789656965 to 1
0s Normal Scheduled pod/ingress-nginx-ingress-controller-65dc55d79b-pzzb8 Successfully assigned gitlab-managed-apps/ingress-nginx-ingress-controller-65dc55d79b-pzzb8 to minikube
0s Normal SuccessfulCreate replicaset/ingress-nginx-ingress-default-backend-7789656965 Created pod: ingress-nginx-ingress-default-backend-7789656965-7m2zg
0s Normal Scheduled pod/ingress-nginx-ingress-default-backend-7789656965-7m2zg Successfully assigned gitlab-managed-apps/ingress-nginx-ingress-default-backend-7789656965-7m2zg to minikube
0s Normal Pulled pod/ingress-nginx-ingress-default-backend-7789656965-7m2zg Container image "k8s.gcr.io/defaultbackend-amd64:1.5" already present on machine
0s Normal Pulled pod/ingress-nginx-ingress-controller-65dc55d79b-pzzb8 Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0" already present on machine
0s Normal Created pod/ingress-nginx-ingress-default-backend-7789656965-7m2zg Created container nginx-ingress-default-backend
0s Normal Created pod/ingress-nginx-ingress-controller-65dc55d79b-pzzb8 Created container nginx-ingress-controller
0s Normal Started pod/ingress-nginx-ingress-controller-65dc55d79b-pzzb8 Started container nginx-ingress-controller
0s Normal Started pod/ingress-nginx-ingress-default-backend-7789656965-7m2zg Started container nginx-ingress-default-backend
0s Normal Pulling pod/ingress-nginx-ingress-controller-65dc55d79b-pzzb8 Pulling image "busybox"
0s Normal CREATE configmap/ingress-nginx-ingress-controller ConfigMap gitlab-managed-apps/ingress-nginx-ingress-controller
0s Normal Pulled pod/ingress-nginx-ingress-controller-65dc55d79b-pzzb8 Successfully pulled image "busybox"
0s Normal Created pod/ingress-nginx-ingress-controller-65dc55d79b-pzzb8 Created container modsecurity-log
0s Normal Started pod/ingress-nginx-ingress-controller-65dc55d79b-pzzb8 Started container modsecurity-log⋊> ~ kubectl get pods -n gitlab-managed-apps -w -o wide
install-ingress 0/1 Pending 0 0s <none> <none> <none> <none>
install-ingress 0/1 Pending 0 0s <none> minikube <none> <none>
install-ingress 0/1 ContainerCreating 0 0s <none> minikube <none> <none>
install-ingress 1/1 Running 0 3s 172.17.0.8 minikube <none> <none>
ingress-nginx-ingress-controller-65dc55d79b-pzzb8 0/2 Pending 0 0s <none> <none> <none> <none>
ingress-nginx-ingress-controller-65dc55d79b-pzzb8 0/2 Pending 0 0s <none> minikube <none> <none>
ingress-nginx-ingress-default-backend-7789656965-7m2zg 0/1 Pending 0 0s <none> <none> <none> <none>
ingress-nginx-ingress-default-backend-7789656965-7m2zg 0/1 Pending 0 0s <none> minikube <none> <none>
ingress-nginx-ingress-controller-65dc55d79b-pzzb8 0/2 ContainerCreating 0 0s <none> minikube <none> <none>
ingress-nginx-ingress-default-backend-7789656965-7m2zg 0/1 ContainerCreating 0 0s <none> minikube <none> <none>
ingress-nginx-ingress-default-backend-7789656965-7m2zg 0/1 Running 0 1s 172.17.0.10 minikube <none> <none>
ingress-nginx-ingress-default-backend-7789656965-7m2zg 1/1 Running 0 4s 172.17.0.10 minikube <none> <none>
ingress-nginx-ingress-controller-65dc55d79b-pzzb8 1/2 Running 0 4s 172.17.0.9 minikube <none> <none>
ingress-nginx-ingress-controller-65dc55d79b-pzzb8 2/2 Running 0 14s 172.17.0.9 minikube <none> <none>
install-ingress 0/1 Completed 0 23s 172.17.0.8 minikube <none> <none>
install-ingress 0/1 Terminating 0 30s 172.17.0.8 minikube <none> <none>
install-ingress 0/1 Terminating 0 30s 172.17.0.8 minikube <none> <none>⋊> ~ kubectl get deployments -n gitlab-managed-apps -w -o wide
ingress-nginx-ingress-controller 0/1 0 0 0s nginx-ingress-controller,modsecurity-log quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0,busybox app=nginx-ingress,release=ingress
ingress-nginx-ingress-default-backend 0/1 0 0 0s nginx-ingress-default-backend k8s.gcr.io/defaultbackend-amd64:1.5 app=nginx-ingress,release=ingress
ingress-nginx-ingress-controller 0/1 0 0 0s nginx-ingress-controller,modsecurity-log quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0,busybox app=nginx-ingress,release=ingress
ingress-nginx-ingress-default-backend 0/1 0 0 0s nginx-ingress-default-backend k8s.gcr.io/defaultbackend-amd64:1.5 app=nginx-ingress,release=ingress
ingress-nginx-ingress-controller 0/1 0 0 0s nginx-ingress-controller,modsecurity-log quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0,busybox app=nginx-ingress,release=ingress
ingress-nginx-ingress-default-backend 0/1 0 0 0s nginx-ingress-default-backend k8s.gcr.io/defaultbackend-amd64:1.5 app=nginx-ingress,release=ingress
ingress-nginx-ingress-default-backend 0/1 1 0 0s nginx-ingress-default-backend k8s.gcr.io/defaultbackend-amd64:1.5 app=nginx-ingress,release=ingress
ingress-nginx-ingress-controller 0/1 1 0 0s nginx-ingress-controller,modsecurity-log quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0,busybox app=nginx-ingress,release=ingress
ingress-nginx-ingress-default-backend 1/1 1 1 4s nginx-ingress-default-backend k8s.gcr.io/defaultbackend-amd64:1.5 app=nginx-ingress,release=ingress
ingress-nginx-ingress-controller 1/1 1 1 14s nginx-ingress-controller,modsecurity-log quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0,busybox app=nginx-ingress,release=ingress⋊> ~ kubectl get services -n gitlab-managed-apps -w -o wide
ingress-nginx-ingress-default-backend ClusterIP 10.104.29.94 <none> 80/TCP 0s app=nginx-ingress,component=default-backend,release=ingress
ingress-nginx-ingress-controller LoadBalancer 10.106.145.10 <pending> 80:31957/TCP,443:32354/TCP 0s app=nginx-ingress,component=controller,release=ingress
ingress-nginx-ingress-controller LoadBalancer 10.106.145.10 10.106.145.10 80:31957/TCP,443:32354/TCP 2s app=nginx-ingress,component=controller,release=ingressDoes this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. - [-] Tested in all supported browsers
- [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec - [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Alan (Maciej) Paruszewski