Skip to content

ElasticSearch not working anymore after 9.0

They have started a new ES 5.1.2 cluster after upgrading to GitLab EE 9.0; but on indexing they are getting this error:


root@gitlab:/# gitlab-rake gitlab:elastic:index 
I, [2017-03-24T15:22:00.733802 #23713] INFO -- sentry: ** [Raven] Raven 2.0.2 r 
eady to catch errors 
rake aborted! 
Elasticsearch::Transport::Transport::Errors::BadRequest: [400] No handler found 
for uri [//gitlab-production] and method [PUT] 
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/elastic/helper.rb:29:in `cr 
eate_empty_index' 
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/elastic.rake:95:in `b 
lock (3 levels) in <top (required)>' 
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/elastic.rake:5:in `bl 
ock (3 levels) in <top (required)>' 
/opt/gitlab/embedded/bin/bundle:22:in `load' 
/opt/gitlab/embedded/bin/bundle:22:in `<main>' 
Tasks: TOP => gitlab:elastic:create_empty_index 
(See full trace by running task with --trace)

While on ElasticSearch they see this:

24/03/2017 16:21:39[2017-03-24T15:21:39,330][INFO ][o.e.n.Node ] [tEi2rUT] started 
24/03/2017 16:21:39[2017-03-24T15:21:39,419][INFO ][o.e.g.GatewayService ] [tEi2rUT] recovered [0] indices into cluster_state 
24/03/2017 16:21:46[2017-03-24T15:21:46,102][INFO ][o.e.c.m.MetaDataCreateIndexService] [tEi2rUT] [.monitoring-data-2] creating index, cause [auto(bulk api)], templates [.monitoring-data-2], shards [1]/[1], mappings [_default_, node, cluster_info, kibana] 
24/03/2017 16:21:46[2017-03-24T15:21:46,225][INFO ][o.e.c.m.MetaDataCreateIndexService] [tEi2rUT] [.monitoring-es-2-2017.03.24] creating index, cause [auto(bulk api)], templates [.monitoring-es-2], shards [1]/[1], mappings [shards, node, _default_, index_stats, index_recovery, cluster_state, cluster_stats, indices_stats, node_stats] 
24/03/2017 16:21:46[2017-03-24T15:21:46,395][INFO ][o.e.c.m.MetaDataMappingService] [tEi2rUT] [.monitoring-es-2-2017.03.24/dZGYsGG6TSSaIWBAY3PzPw] update_mapping [cluster_stats] 
24/03/2017 16:21:46[2017-03-24T15:21:46,479][INFO ][o.e.c.m.MetaDataMappingService] [tEi2rUT] [.monitoring-es-2-2017.03.24/dZGYsGG6TSSaIWBAY3PzPw] update_mapping [node_stats] 
24/03/2017 16:21:56[2017-03-24T15:21:56,600][INFO ][o.e.c.m.MetaDataMappingService] [tEi2rUT] [.monitoring-es-2-2017.03.24/dZGYsGG6TSSaIWBAY3PzPw] update_mapping [cluster_stats] 
24/03/2017 16:21:56[2017-03-24T15:21:56,624][INFO ][o.e.c.m.MetaDataMappingService] [tEi2rUT] [.monitoring-es-2-2017.03.24/dZGYsGG6TSSaIWBAY3PzPw] update_mapping [indices_stats] 
24/03/2017 16:21:56[2017-03-24T15:21:56,673][INFO ][o.e.c.m.MetaDataMappingService] [tEi2rUT] [.monitoring-es-2-2017.03.24/dZGYsGG6TSSaIWBAY3PzPw] update_mapping [index_stats] 
24/03/2017 16:22:01[2017-03-24T15:22:01,719][WARN ][o.e.d.r.a.a.i.RestTypesExistsAction] [HEAD /{index}/{type}] is deprecated! Use [HEAD /{index}/_mapping/{type}] instead

They have also restarted the server. Also, they're now on 9.0.1 with the same issue.

03291056_6h3b9

root@gitlab:/# curl http://10.2.41.240:9200/ 
{ 
"name" : "tEi2rUT", 
"cluster_name" : "docker-cluster", 
"cluster_uuid" : "39o6_7V0Sl-rGe4npKORtQ", 
"version" : { 
"number" : "5.1.2", 
"build_hash" : "c8c4c16", 
"build_date" : "2017-01-11T20:18:39.146Z", 
"build_snapshot" : false, 
"lucene_version" : "6.3.0" 
}, 
"tagline" : "You Know, for Search" 
} 
root@gitlab:/#

ZD ticket: https://gitlab.zendesk.com/agent/tickets/70595

cc// @nick.thomas @vsizov @stanhu