K8s 1.16 breaks default monitoring dashboard
Summary
(Summarize the bug encountered concisely)
K8s has changed the pod_name
and container_name
labels to pod
and container
respectively in 1.16 (https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.16.md#removed-metrics).
Our default metrics dashboard, and the pod metrics dashboard uses the pod_name
and container_name
labels extensively.
Steps to reproduce
(How one can reproduce the issue - this is very important)
- Make sure that your K8s cluster is using version 1.16 or later. On GKE you need to choose 1.16 for the master version when manually creating a cluster.
- You also need to follow these steps before you can do deployments with K8s 1.16.
- Check the default metrics dashboard (Operations > Metrics) in a project. All queries with
container_name
orpod_name
will return no data.
Example Project
(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)
(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)
What is the current bug behavior?
(What actually happens)
Charts do not populate with data when using K8s version 1.16 or later.
What is the expected correct behavior?
(What you should see instead)
Charts should populate with data.
Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)