gitlab shell: service monitor is not working for prometheus - connection refused
Summary
It seems to me that after merging service monitor MR !2376 (merged), service monitor for gitlab shell
(only) does not work, i.e. it cannot be accessed so that metrics could be seen.
Of note, all other service monitors, e.g. redis, postgress, pages, webservice etc. do work as expected. GitLab is running just fine and we can also use SSH for repositories. It is just that shell metrics are not exported/accessible.
# from toolbox container
curl -v http://gitlab-gitlab-pages-metrics:9235/metrics -> large prometheus file
curl -v http://gitlab-gitlab-shell:9122/metrics
>
* Failed to connect to gitlab-gitlab-shell port 9122: Connection refused
* Closing connection 0
curl: (7) Failed to connect to gitlab-gitlab-shell port 9122: Connection refused
Steps to reproduce
gitlab:
gitlab-shell:
metrics: # per default metrics are disabled
enabled: true
works and creates deployment with prometheus annotations. However the same thing with service monitor - even though it does create service monitor object - it wont permit prometheus to scrap gitlab shell metrics.
gitlab:
gitlab-shell:
metrics:
enabled: true
serviceMonitor:
enabled: true
Versions
- Chart: 6.2.1,
appVersion: 15.2.1
- Platform:
- Self-hosted: Rancher RKE 1
- Kubernetes: (
kubectl version
)- Server: v1.22.7
- Helm: (
helm version
)- Client: 3.9.1
Relevant logs
This is service monitor that gets created:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
annotations:
meta.helm.sh/release-name: gitlab
meta.helm.sh/release-namespace: gitlab
labels:
app: gitlab-shell
app.kubernetes.io/managed-by: Helm
chart: gitlab-shell-6.2.1
heritage: Helm
release: gitlab
name: gitlab-gitlab-shell
namespace: gitlab
spec:
endpoints:
- path: /metrics
port: http-metrics
namespaceSelector:
matchNames:
- gitlab
selector:
matchLabels:
app: gitlab-shell
release: gitlab
We use shell port 2224
so our k8s service
looks like (cleaned up a bit) this:
apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: gitlab
meta.helm.sh/release-namespace: gitlab
labels:
app: gitlab-shell
app.kubernetes.io/instance: gitlab
app.kubernetes.io/managed-by: Helm
chart: gitlab-shell-6.2.1
heritage: Helm
release: gitlab
name: gitlab-gitlab-shell
namespace: gitlab
spec:
.....
ports:
- name: http-metrics
port: 9122
protocol: TCP
targetPort: http-metrics
- name: ssh
port: 2224
protocol: TCP
targetPort: 2222
selector:
app: gitlab-shell
release: gitlab
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
Edited by John Malc