Exclude frames with forwardStream()
Relates to More selective goroutine dumps (#618 - closed).
Follow up for Filter goroutine dumps with Kubernetes API prox... (!1915 - merged).
I was looking at the filtered dumps and noticed that we can filter them even more. Private API server can be blocking shutdown when it's proxying a request that is coming from a different kas.
Example event: https://new-sentry.gitlab.net/organizations/gitlab/issues/946651/events/f7ec0d291b784794bd511949260de422/
Dump: goroutines.txt
1 @ 0x442c8e 0x454ac5 0xbd23a5 0xbd2085 0xbd1798 0x169b4cc 0x94b352 0x169b64b 0x94b352 0xbba02e 0x94b352 0x169b751 0x94b352 0x16a945b 0x944645 0x94c587 0x94e19a 0x946ecb 0x47ad21
# 0xbd23a4 gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/internal/tunnel/tunserver.(*TunnelImpl).forwardStream+0x284 /tmp/build/internal/tunnel/tunserver/tunnel.go:186
# 0xbd2084 gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/internal/tunnel/tunserver.(*TunnelImpl).ForwardStream+0xa4 /tmp/build/internal/tunnel/tunserver/tunnel.go:79
# 0xbd1797 gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/internal/tunnel/tunserver.(*Router).routeToTunclient+0x6d7 /tmp/build/internal/tunnel/tunserver/router_to_tunclient.go:68
# 0x169b4cb gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/cmd/kas/kasapp.newPrivateAPIServerImpl.StreamServerErrorReporterInterceptor.func9+0x4b /tmp/build/internal/tool/grpctool/server_error_reporter.go:28
# 0x94b351 google.golang.org/grpc.getChainStreamHandler.func1+0xb1 /root/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1540
# 0x169b64a gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/cmd/kas/kasapp.newPrivateAPIServerImpl.StreamServerValidatingInterceptor.func8+0xaa /tmp/build/internal/tool/grpctool/server_validator.go:28
# 0x94b351 google.golang.org/grpc.getChainStreamHandler.func1+0xb1 /root/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1540
# 0xbba02d gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/internal/tool/grpctool.(*JWTAuther).StreamServerInterceptor+0x6d /tmp/build/internal/tool/grpctool/jwt_server_auth.go:61
# 0x94b351 google.golang.org/grpc.getChainStreamHandler.func1+0xb1 /root/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1540
# 0x169b750 gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/cmd/kas/kasapp.newPrivateAPIServerImpl.StreamRPCAPIInterceptor.func7+0xb0 /tmp/build/internal/module/modshared/rpc_api.go:43
# 0x94b351 google.golang.org/grpc.getChainStreamHandler.func1+0xb1 /root/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1540
# 0x16a945a gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/cmd/kas/kasapp.(*ConfiguredApp).constructObservabilityTools.(*ServerMetrics).StreamServerInterceptor.StreamServerInterceptor.func2+0x2ba /root/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35
# 0x944644 google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2+0x84 /root/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1531
# 0x94c586 google.golang.org/grpc.(*Server).processStreamingRPC+0x11e6 /root/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1695
# 0x94e199 google.golang.org/grpc.(*Server).handleStream+0xe39 /root/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1809
# 0x946eca google.golang.org/grpc.(*Server).serveStreams.func2.1+0x8a /root/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1029Edited by Mikhail Mazurskiy