Change args on modsecurity-log container to ignore file inaccessiblity (!29909) · Merge requests · GitLab.org / GitLab

Alan (Maciej) Paruszewski requested to merge 37127-change-command-on-modsecurity-log-container into master Apr 20, 2020

What does this MR do?

While testing if fix provided in !28942 (merged) I have noticed that on managed kubernetes environments (on GCP Kubernetes Engine) that pods are restarted twice before livenessProbe starts veryfining if pod is healthy. This is happening because command from args is used when creating container (ContainerCreating), this phase will fail if there is no file /var/log/modsec/audit.log yet created, so pod is restarted, and usually after second restart we already have logs for modsecurity available and container can start successfully and Pod can be in Running state. We do not want to rely on args command but on LivenessProbe instead, this is why we are moving to use tail -F instead of tail -f command:

$ tldr tail

tail

Display the last part of a file.

...

- Keep reading file until `Ctrl + C`:
    tail -f file

- Keep reading file until `Ctrl + C`, even if the file is rotated:
    tail -F file

Before

$ kubectl get pod -n gitlab-managed-apps ingress-nginx-ingress-controller-64ddfc58fc-mzw9v
NAME                                                READY   STATUS    RESTARTS   AGE
ingress-nginx-ingress-controller-64ddfc58fc-mzw9v   2/2     Running   2          57s

$ kubectl describe pod -n gitlab-managed-apps ingress-nginx-ingress-controller-64ddfc58fc-mzw9v

Containers:
  ...
  modsecurity-log:
    Container ID:  docker://0d5c92369b14dcf97a9fa08de8bbaf85910979bf02d1953293df2f6ab37c446f
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:a2490cec4484ee6c1068ba3a05f89934010c85242f736280b35343483b2264b6
    Port:          <none>
    Host Port:     <none>
    Args:
      /bin/sh
      -c
      tail -f /var/log/modsec/audit.log
    State:          Running
      Started:      Mon, 20 Apr 2020 12:46:18 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 20 Apr 2020 12:45:57 +0200
      Finished:     Mon, 20 Apr 2020 12:45:57 +0200
    Ready:          True
    Restart Count:  2
    Liveness:       exec [ls /var/log/modsec/audit.log] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/log/modsec from modsecurity-log-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-ingress-token-96n85 (ro)
...
Events:
  Type     Reason     Age                From                                                          Message
  ----     ------     ----               ----                                                          -------
  Normal   Scheduled  83s                default-scheduler                                             Successfully assigned gitlab-managed-apps/ingress-nginx-ingress-controller-64ddfc58fc-mzw9v to gke-mparuszewski-staging-default-pool-dd5ecbb5
-9fpq
  Normal   Pulled     82s                kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-9fpq  Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0" already present on machine
  Normal   Created    82s                kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-9fpq  Created container nginx-ingress-controller
  Normal   Started    82s                kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-9fpq  Started container nginx-ingress-controller
  Warning  BackOff    75s (x3 over 80s)  kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-9fpq  Back-off restarting failed container
  Normal   Pulling    60s (x3 over 82s)  kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-9fpq  Pulling image "busybox"
  Normal   Pulled     60s (x3 over 82s)  kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-9fpq  Successfully pulled image "busybox"
  Normal   Created    60s (x3 over 82s)  kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-9fpq  Created container modsecurity-log
  Normal   Started    60s (x3 over 81s)  kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-9fpq  Started container modsecurity-log

$ kubectl logs ingress-nginx-ingress-controller-64ddfc58fc-mzw9v -n gitlab-managed-apps -c modsecurity-log --previous
tail: can't open '/var/log/modsec/audit.log': No such file or directory
tail: no files

After

$ kubectl get pod -n gitlab-managed-apps  ingress-nginx-ingress-controller-65bdf6b6d5-2svh4
NAME                                                READY   STATUS    RESTARTS   AGE
ingress-nginx-ingress-controller-65bdf6b6d5-2svh4   2/2     Running   0          21m

$ kubectl describe pod -n gitlab-managed-apps  ingress-nginx-ingress-controller-65bdf6b6d5-2svh4
...
Containers:
  ...
  modsecurity-log:
    Container ID:  docker://c6a8e26591efa14dde8368704a4101ccceab9e140a0ac0d3b7d1c601a0ea3c3f
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:a2490cec4484ee6c1068ba3a05f89934010c85242f736280b35343483b2264b6
    Port:          <none>
    Host Port:     <none>
    Args:
      /bin/sh
      -c
      tail -F /var/log/modsec/audit.log
    State:          Running
      Started:      Mon, 20 Apr 2020 12:23:08 +0200
    Ready:          True
    Restart Count:  0
    Liveness:       exec [ls /var/log/modsec/audit.log] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/log/modsec from modsecurity-log-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-ingress-token-96n85 (ro)
...
Events:
  Type    Reason     Age   From                                                          Message
  ----    ------     ----  ----                                                          -------
  Normal  Scheduled  20m   default-scheduler                                             Successfully assigned gitlab-managed-apps/ingress-nginx-ingress-controller-65bdf6b6d5-2svh4 to gke-mparuszewski-staging-default-pool-dd5ecbb5-n8bf
  Normal  Pulled     20m   kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-n8bf  Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0" already present on machine
  Normal  Created    20m   kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-n8bf  Created container nginx-ingress-controller
  Normal  Started    20m   kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-n8bf  Started container nginx-ingress-controller
  Normal  Pulling    20m   kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-n8bf  Pulling image "busybox"
  Normal  Pulled     20m   kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-n8bf  Successfully pulled image "busybox"
  Normal  Created    20m   kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-n8bf  Created container modsecurity-log
  Normal  Started    20m   kubelet, gke-mparuszewski-staging-default-pool-dd5ecbb5-n8bf  Started container modsecurity-log

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
[-] Tested in all supported browsers
[-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

[-] Label as security and @ mention @gitlab-com/gl-security/appsec
[-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
[-] Security reports checked/validated by a reviewer from the AppSec team

Change args on modsecurity-log container to ignore file inaccessiblity