Remove Kubernetes proxy write timeout
Fixes #255 (closed).
Fixes https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/671.
I have managed to reproduce the issue, although I'm getting a different error. This is probably because I don't have any load balancer in front of kas' Kubernetes proxy endpoint.
Setup:
- GDK
- agentk to kas via direct grpc (no nginx, no websockets)
- kubectl -> kas directly (no nginx)
- rancher desktop v1.5.1, Kubernetes v1.24.3
Without the fix:
date && kubectl logs --all-containers -f -n kube-system coredns-d76bd69b-g84qz -v 9 || date
Thu 25 Aug 2022 22:38:02 AEST
I0825 22:38:02.757075 52392 loader.go:374] Config loaded from file: /Users/mike/.kube/config
I0825 22:38:02.759703 52392 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.25.0 (darwin/arm64) kubernetes/a866cbe" 'http://gdk.test:8154/-/k8s-proxy/api/v1/namespaces/kube-system/pods/coredns-d76bd69b-g84qz'
I0825 22:38:02.761155 52392 round_trippers.go:495] HTTP Trace: DNS Lookup for gdk.test resolved to [{172.16.123.1 }]
I0825 22:38:02.761422 52392 round_trippers.go:510] HTTP Trace: Dial to tcp:172.16.123.1:8154 succeed
I0825 22:38:02.783655 52392 round_trippers.go:553] GET http://gdk.test:8154/-/k8s-proxy/api/v1/namespaces/kube-system/pods/coredns-d76bd69b-g84qz 200 OK in 23 milliseconds
I0825 22:38:02.783670 52392 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 0 ms TLSHandshake 0 ms ServerProcessing 22 ms Duration 23 ms
I0825 22:38:02.783675 52392 round_trippers.go:577] Response Headers:
I0825 22:38:02.783679 52392 round_trippers.go:580] X-Request-Id: 01GBAG4VR9RMG78K32H3S6EXAQ
I0825 22:38:02.783683 52392 round_trippers.go:580] Date: Thu, 25 Aug 2022 12:38:02 GMT
I0825 22:38:02.783687 52392 round_trippers.go:580] Cache-Control: no-cache, private
I0825 22:38:02.783690 52392 round_trippers.go:580] Content-Type: application/json
I0825 22:38:02.783693 52392 round_trippers.go:580] Via: 1.1 gitlab-agent/v0.0.0/00000000
I0825 22:38:02.783697 52392 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v0.0.0/00000000
I0825 22:38:02.783700 52392 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 0848073f-9750-4291-925c-57643e024a51
I0825 22:38:02.783704 52392 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 549b83f7-181d-45c6-90fe-f6df5c20e110
I0825 22:38:02.783707 52392 round_trippers.go:580] Audit-Id: 1429e569-f671-4e73-abf2-b270e52fcea9
I0825 22:38:02.785462 52392 request.go:1073] Response Body: ...
I0825 22:38:02.786834 52392 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.25.0 (darwin/arm64) kubernetes/a866cbe" 'http://gdk.test:8154/-/k8s-proxy/api/v1/namespaces/kube-system/pods/coredns-d76bd69b-g84qz/log?container=coredns&follow=true'
I0825 22:38:02.810774 52392 round_trippers.go:553] GET http://gdk.test:8154/-/k8s-proxy/api/v1/namespaces/kube-system/pods/coredns-d76bd69b-g84qz/log?container=coredns&follow=true 200 OK in 23 milliseconds
I0825 22:38:02.810804 52392 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 23 ms Duration 23 ms
I0825 22:38:02.810809 52392 round_trippers.go:577] Response Headers:
I0825 22:38:02.810814 52392 round_trippers.go:580] Audit-Id: 08821e73-6146-4f91-ada8-d7d1b9c6bd49
I0825 22:38:02.810818 52392 round_trippers.go:580] Cache-Control: no-cache, private
I0825 22:38:02.810822 52392 round_trippers.go:580] X-Request-Id: 01GBAG4VS2T2S1PQQC45E18XKR
I0825 22:38:02.810825 52392 round_trippers.go:580] Content-Type: text/plain
I0825 22:38:02.810829 52392 round_trippers.go:580] Date: Thu, 25 Aug 2022 12:38:02 GMT
I0825 22:38:02.810832 52392 round_trippers.go:580] Via: 1.1 gitlab-agent/v0.0.0/00000000
I0825 22:38:02.810836 52392 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v0.0.0/00000000
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
.:53
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[INFO] plugin/reload: Running configuration SHA512 = b941b080e5322f6519009bb49349462c7ddb6317425b0f6a83e5451175b720703949e3f3b454a24e77f3ffe57fd5e9c6130e528a5a1dd00d9000e4afd6c1108d
CoreDNS-1.9.1
linux/arm64, go1.17.8, 4b597f8
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
...
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
error: unexpected EOF
Thu 25 Aug 2022 22:38:30 AEST
With the fix - no error, log tailing is not getting interrupted so quickly. Note that log tailing is still limited by max connection age that kas sets. It is currently set to 30 minutes.
Edited by Mikhail Mazurskiy