Add support for ES7 & Drop suport for ES5
To be merged after we've released gitlab-elasticsearch-indexer 2.0.0
What does this MR do?
- Remove the ability to run against v5 by taking out the
index.mapping.single_type
logic - Change our CI to test against
6.4.2
since we no longer want to support ES5 - Remove the standard token filter since it is not available in ES 7 and according to the docs it does nothing anyway
- Add support for ES7 by setting
include_type_name
when creating the index. This will ensure ES7 behaves like ES6 when creating mappings. See https://www.elastic.co/blog/moving-from-types-to-typeless-apis-in-elasticsearch-7-0 for more information. - Monkey patch the
elasticsearch-model
gem so that it theElasticsearch::Model::Response::Results#count
method handles the new object format response from ES7 servers
Equivalent changes for the gitlab-elasticsearch-indexer
at gitlab-elasticsearch-indexer!54 (merged)
NOTE: I ran CI that verifies we could test against Elasticsearch V7 https://gitlab.com/gitlab-org/gitlab/pipelines/109413089 . This commit was just to test and was reverted for now until we implement #196503 (comment 270297726)
NOTE: Another incompatible change that has happened in ES7 is that Elasticsearch server only counts the first 10,000 results in the total count by default which means that until we make some changes in GitLab we will not be able to paginate past 10,000 results. This seems like we can live with this limitation for now so I've extracted an issue to deal with this later in #196627 (closed)
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Manual QA
6.5.1
-
Enable ES -
Create a project with snippets, code, commits, issues, issue comments, merge request comments -
Perform searches for all resource types using *
search -
Perform boolean search tests like a | b
and(a | b) + c
-
Trigger a complete re-index using bundle exec rake gitlab:elastic:index
and again try all searches -
Start again but with only having Elasticsearch enabled for a single group and again try all searches -
Check the elasticsearch/logs/*.log
andgdk tail elasticsearch
for any errors or warnings
7.5.1
-
Enable ES -
Create a project with snippets, code, commits, issues, issue comments, merge request comments -
Perform searches for all resource types using *
search -
Perform boolean search tests like a | b
and(a | b) + c
-
Trigger a complete re-index using bundle exec rake gitlab:elastic:index
and again try all searches -
Start again but with only having Elasticsearch enabled for a single group and again try all searches -
Check the elasticsearch/logs/*.log
andgdk tail elasticsearch
for any errors or warnings