Skip to content

Advanced search: Add prometheus metrics

Terri Chu requested to merge 467252-expose-prom-metrics-for-indexed-projects into master

What does this MR do and why?

Related to #467252 (closed)

This MR adds two new prometheus metrics:

  1. a counter metric that counts total number of index repair operations where the indexing is kicked off. This can be useful to detect when we have index integrity issues
  2. a boolean setting gauge to tell if advanced search settings are enabled or disabled.

groupglobal search alerts currently only use queue size for increment and initial indexing. the alerts will also use the boolean setting gauge metric and only alert if indexing is not paused. indexing is often paused during migrations or change request/maintenance work. It will remove burden from team to remember which alerts to silence when doing change requests

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

N/A

How to set up and validate locally

I had to set up a single puma server to get the metrics to show up properly

  1. setup elasticsearch for gdk
  2. index the instance using gitlab:elastic:index rake task or Index the instance button in Admin UI
  3. open the metrics page for gdk: http://localhost:3000/-/metrics
  4. test the index repair service metric
    • delete a project from the projects index (I used flightjs with id 7) 
       curl --request DELETE \
 --url 'http://localhost:9200/gitlab-development-projects/_doc/project_7?routing=n_33'
    • delete code for the same project from the main index 
      curl --request POST \
  --url 'http://localhost:9200/gitlab-development/_delete_by_query?conflicts=proceed' \
  --header 'Content-Type: application/json' \
  --data '{
	"query": {
		"bool": {
			"must": [
				{
					"term": {
						"type": {
							"value": "blob"
						}
					}
				},
				{
					"term": {
						"project_id": {
							"value": 7
						}
					}
				}
			]
		}
	}
}'
    • run the repair service for that project 
       p = Project.find(7)
 Search::IndexRepairService.execute(p)
    • verify it shows up in: http://localhost:3000/-/metrics 
       # HELP search_advanced_index_repair_total Multiprocess metric
 # TYPE search_advanced_index_repair_total counter
 search_advanced_index_repair_total{document_type="blob"} 10
 search_advanced_index_repair_total{document_type="project"} 16
  5. test the cron metrics update cron worker 
# HELP search_advanced_boolean_settings Multiprocess metric
# TYPE search_advanced_boolean_settings gauge
search_advanced_boolean_settings{name="elasticsearch_indexing"} 1
search_advanced_boolean_settings{name="elasticsearch_indexing",pid="process_11922"} 0
search_advanced_boolean_settings{name="elasticsearch_indexing",pid="process_12512"} 0
search_advanced_boolean_settings{name="elasticsearch_indexing",pid="process_13074"} 0
search_advanced_boolean_settings{name="elasticsearch_indexing",pid="process_13617"} 0
search_advanced_boolean_settings{name="elasticsearch_indexing",pid="process_14159"} 0
search_advanced_boolean_settings{name="elasticsearch_indexing",pid="process_15496"} 0
search_advanced_boolean_settings{name="elasticsearch_indexing",pid="process_16108"} 0
search_advanced_boolean_settings{name="elasticsearch_indexing",pid="process_16674"} 0
search_advanced_boolean_settings{name="elasticsearch_indexing",pid="process_72460"} 1
search_advanced_boolean_settings{name="elasticsearch_indexing",pid="process_74349"} 0
search_advanced_boolean_settings{name="elasticsearch_pause_indexing"} 0
search_advanced_boolean_settings{name="elasticsearch_pause_indexing",pid="process_11922"} 0
search_advanced_boolean_settings{name="elasticsearch_pause_indexing",pid="process_12512"} 0
search_advanced_boolean_settings{name="elasticsearch_pause_indexing",pid="process_13074"} 0
search_advanced_boolean_settings{name="elasticsearch_pause_indexing",pid="process_13617"} 0
search_advanced_boolean_settings{name="elasticsearch_pause_indexing",pid="process_14159"} 0
search_advanced_boolean_settings{name="elasticsearch_pause_indexing",pid="process_15496"} 0
search_advanced_boolean_settings{name="elasticsearch_pause_indexing",pid="process_16108"} 0
search_advanced_boolean_settings{name="elasticsearch_pause_indexing",pid="process_16674"} 0
search_advanced_boolean_settings{name="elasticsearch_pause_indexing",pid="process_72460"} 0
search_advanced_boolean_settings{name="elasticsearch_pause_indexing",pid="process_74349"} 0
search_advanced_boolean_settings{name="elasticsearch_requeue_workers"} 0
search_advanced_boolean_settings{name="elasticsearch_requeue_workers",pid="process_11922"} 0
search_advanced_boolean_settings{name="elasticsearch_requeue_workers",pid="process_12512"} 0
search_advanced_boolean_settings{name="elasticsearch_requeue_workers",pid="process_13074"} 0
search_advanced_boolean_settings{name="elasticsearch_requeue_workers",pid="process_13617"} 0
search_advanced_boolean_settings{name="elasticsearch_requeue_workers",pid="process_14159"} 0
search_advanced_boolean_settings{name="elasticsearch_requeue_workers",pid="process_15496"} 0
search_advanced_boolean_settings{name="elasticsearch_requeue_workers",pid="process_16108"} 0
search_advanced_boolean_settings{name="elasticsearch_requeue_workers",pid="process_16674"} 0
search_advanced_boolean_settings{name="elasticsearch_requeue_workers",pid="process_72460"} 1
search_advanced_boolean_settings{name="elasticsearch_requeue_workers",pid="process_74349"} 0
search_advanced_boolean_settings{name="elasticsearch_search"} 1
search_advanced_boolean_settings{name="elasticsearch_search",pid="process_11922"} 0
search_advanced_boolean_settings{name="elasticsearch_search",pid="process_12512"} 0
search_advanced_boolean_settings{name="elasticsearch_search",pid="process_13074"} 0
search_advanced_boolean_settings{name="elasticsearch_search",pid="process_13617"} 0
search_advanced_boolean_settings{name="elasticsearch_search",pid="process_14159"} 0
search_advanced_boolean_settings{name="elasticsearch_search",pid="process_15496"} 0
search_advanced_boolean_settings{name="elasticsearch_search",pid="process_16108"} 0
search_advanced_boolean_settings{name="elasticsearch_search",pid="process_16674"} 0
search_advanced_boolean_settings{name="elasticsearch_search",pid="process_72460"} 1
search_advanced_boolean_settings{name="elasticsearch_search",pid="process_74349"} 0
Edited by Terri Chu

Merge request reports