Improve Advanced Search counts queries
What does this MR do?
This MR changes ES queries that we use for counts (e.g. /search/count?group_id=&project_id=&repository_ref=&scope=blobs&search=update
) to make them more performant.
-
Move the Clause from MUST
toFILTER
this will save the "score" calculation. -
Remove Boosting for simple_query_string fields. This will also prevent calculation scores. -
Set the Size
0 , This will prevent fetching the documents.
hyperfine
Query benchmark withThis is a simple benchmark comparing old and new issues query
❯ hyperfine './old.sh' './new.sh'
Benchmark #1: ./old.sh
Time (mean ± σ): 17.0 ms ± 1.8 ms [User: 5.1 ms, System: 2.4 ms]
Range (min … max): 12.9 ms … 24.1 ms 122 runs
Benchmark #2: ./new.sh
Time (mean ± σ): 10.0 ms ± 1.0 ms [User: 5.1 ms, System: 2.3 ms]
Range (min … max): 6.8 ms … 13.7 ms 251 runs
Summary
'./new.sh' ran
1.70 ± 0.24 times faster than './old.sh'
scripts
❯ cat old.sh
#!/bin/sh
curl -XPOST -s -H "Content-Type: application/json" http://localhost:9200/gitlab-development-issues/_search --data-binary "@old.json" > /dev/null
❯ cat old.json
{"query":{"bool":{"must":[{"simple_query_string":{"_name":"issue:match:search_terms","fields":["iid^3","title^2","description"],"query":"update","lenient":true,"default_operator":"and"}}],"filter":[{"term":{"type":{"_name":"doc:is_a:issue","value":"issue"}}},{"bool":{"should":[{"bool":{"filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:any","value":0}}},{"terms":{"_name":"issue:authorized:project:issues:enabled_or_private","issues_access_level":[20,10]}}]}},{"bool":{"_name":"issue:authorized:project:visibility:10:issues:access_level","filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:visibility:10","value":10}}},{"terms":{"_name":"issue:authorized:project:visibility:10:issues:access_level:enabled_or_private","issues_access_level":[20,10]}}]}},{"bool":{"_name":"issue:authorized:project:visibility:20:issues:access_level","filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:visibility:20","value":20}}},{"terms":{"_name":"issue:authorized:project:visibility:20:issues:access_level:enabled_or_private","issues_access_level":[20,10]}}]}}]}}]}},"highlight":{"fields":{"iid":{},"title":{},"description":{}},"number_of_fragments":0,"pre_tags":["gitlabelasticsearch→"],"post_tags":["←gitlabelasticsearch"]}}
❯ cat new.sh
#!/bin/sh
curl -XPOST -s -H "Content-Type: application/json" http://localhost:9200/gitlab-development-issues/_search --data-binary "@new.json" > /dev/null
❯ cat new.json
{"query":{"bool":{"filter":[{"term":{"type":{"_name":"doc:is_a:issue","value":"issue"}}},{"simple_query_string":{"_name":"issue:match:search_terms","fields":["iid","title","description"],"query":"update","lenient":true,"default_operator":"and"}},{"bool":{"should":[{"bool":{"filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:any","value":0}}},{"terms":{"_name":"issue:authorized:project:issues:enabled_or_private","issues_access_level":[20,10]}}]}},{"bool":{"_name":"issue:authorized:project:visibility:10:issues:access_level","filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:visibility:10","value":10}}},{"terms":{"_name":"issue:authorized:project:visibility:10:issues:access_level:enabled_or_private","issues_access_level":[20,10]}}]}},{"bool":{"_name":"issue:authorized:project:visibility:20:issues:access_level","filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:visibility:20","value":20}}},{"terms":{"_name":"issue:authorized:project:visibility:20:issues:access_level:enabled_or_private","issues_access_level":[20,10]}}]}}]}}]}},"size":0}
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Related to #299099 (closed)
Edited by Dmitry Gruzd