Refactor search level filter in ES for code initially
What does this MR do and why?
Related to #477814 (closed)
The what
This MR introduces a refactor (behind a feature flag) for the search authorization filter generation for code search only. Other scopes will be refactored to use this in future MRs once this has been enabled in production for a bit.
The new filter is generated with two pieces:
- search level (global, group, or project) - filters by the IDs passed
- membership - filters by namespace traversal_ids and project_ids the user has access to via:
- direct membership to namespace or project
- shared membership to a project through a group
- shared membership to a group through a group
- direct membership to a namespace or project with a custom role that allows access
repository
access
The why
The original filters are sending through (in some cases) huge lists of projects to Elasticsearch. This causes issues on both database (timeouts) and Elasticsearch (query is too big/complex or hits all shards)
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
I tested the old way of grabbing project ids vs the new way in staging for global search. The difference is most apparent for global search but will be a nice improvement for group searches where users only have access to a few sub groups or projects
count project_ids before | count project_ids after |
---|---|
298 | 20 |
Database
query plans are provided in comments on where they are used for generation of traversal ids and projects ids at global, group and project level
How to set up and validate locally
It's too comprehensive to write out test cases for each permutation. This MR introduces new specs to test the filters with the supported memberships listed in the description. There are existing visibility specs which verify user access with different roles and feature access level settings.
If you'd like to test an individual user type/role/access level:
- setup gdk for elasticsearch
- index the instance in the Admin UI - Search Settings
- give a user access to
- a group
- a project (outside the group above)
- a second project with a role of guest with a custom role which allows
:repository
access - a group that has shared access to another group
- a group that has shared access to another project
- perform global, group, and project level searches for code
- verify the expected results come back