Logging from zoekt-webserver processes forwarded to Elasticsearch
The zoekt-webserver
already supports log files with rotation . We should implement the same option in zoekt-dynamic-indexserver
.
We should also investigate if it's practical to make these JSON formatted log files and if not we might need a custom parser to deal with sending them in some structured way to Elasticsearch. I assume we already have such tools for many other systems we capture logs from that don't have JSON formatted log files (eg. Postgres).
Infra changes
-
Create new GCP pubsub topic: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5669 -
Configure Fluentd to forward logs to GCP pubsub: gitlab-com/gl-infra/k8s-workloads/tanka-deployments!733 (merged) - NOTE: this works on the default non-JSON log parser defined at https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/tanka-deployments/-/blob/7c610c1612af42586bbe278cab53d05f81df490b/lib/fluentd/index.containers.input.conf.libsonnet#L20
-
Create an index pattern in Elasticsearch: gitlab-com/runbooks!5734 (merged) -
Configure pubsubbeat to forward the logs to Elasticsearch: gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!2098 (merged) -
Configure Kibana index pattern: https://log.gprd.gitlab.net/app/management/kibana/indexPatterns/patterns/3f6d1a00-e8b6-11ed-a017-0d32180b1390#/ -
Configure Elasticsearch index template to ensure fields are correctly typed: gitlab-com/runbooks!5739 (merged)
Edited by Dylan Griffith