Skip to content

Include namespaces in project searches when query contains '/'

Brian Williams requested to merge bwill/frontend-full-path-search into master

What does this MR do and why?

Describe in detail what your merge request does and why.

Currently, most project searches only match against the project's path name, without the group name being included. For example, using the hamburger menu and searching for gitlab-org/gitlab won't return the gitlab repo in the results. This is because the project path is gitlab and namespaces (gitlab-org/) aren't included in the search. This behavior can be controlled using the search_namespaces parameter. If / is included in the search query, then the user is most likely trying to search for a path with a group name. In that case, namespaces should be included in the search.

Since these queries are issued from the REST API, they are scoped to project authorizations and also paginated, so the performance is decent. To minimize the risk of this causing performance issues in production, a feature flag is used.

SELECT
    "projects".*
FROM
    "projects"
    INNER JOIN "project_authorizations" ON "projects"."id" = "project_authorizations"."project_id"
    INNER JOIN "routes" ON "routes"."source_type" = 'Project'
    AND "routes"."source_id" = "projects"."id"
WHERE
    "project_authorizations"."user_id" = 8953999
    AND "projects"."pending_delete" = FALSE
    AND (
        (
            "routes"."path" ILIKE '%gitlab-org/%'
            OR "routes"."name" ILIKE '%gitlab-org/%'
        )
        OR "projects"."description" ILIKE '%gitlab-org/%'
    )
    AND "projects"."hidden" = FALSE limit 20;
 Limit  (cost=1066.67..13965.07 rows=2 width=764) (actual time=171.319..710.235 rows=20 loops=1)
   Buffers: shared hit=405 read=639 dirtied=18
   I/O Timings: read=1665.054 write=0.000
   ->  Gather  (cost=1066.67..13965.07 rows=2 width=764) (actual time=171.317..710.217 rows=20 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         Buffers: shared hit=405 read=639 dirtied=18
         I/O Timings: read=1665.054 write=0.000
         ->  Nested Loop  (cost=66.67..12964.87 rows=1 width=764) (actual time=176.331..625.943 rows=7 loops=3)
               Buffers: shared hit=405 read=639 dirtied=18
               I/O Timings: read=1665.054 write=0.000
               ->  Nested Loop  (cost=66.23..12169.86 rows=1463 width=58) (actual time=131.620..426.889 rows=33 loops=3)
                     Buffers: shared hit=245 read=397 dirtied=14
                     I/O Timings: read=1076.893 write=0.000
                     ->  Parallel Bitmap Heap Scan on public.project_authorizations  (cost=65.66..6303.79 rows=1645 width=4) (actual time=113.473..212.862 rows=33 loops=3)
                           Buffers: shared read=140 dirtied=1
                           I/O Timings: read=462.056 write=0.000
                           ->  Bitmap Index Scan using project_authorizations_pkey  (cost=0.00..64.68 rows=3947 width=0) (actual time=108.722..108.723 rows=8043 loops=1)
                                 Index Cond: (project_authorizations.user_id = 8953999)
                                 Buffers: shared read=40
                                 I/O Timings: read=106.079 write=0.000
                     ->  Index Scan using index_routes_on_source_type_and_source_id on public.routes  (cost=0.56..3.57 rows=1 width=54) (actual time=6.410..6.410 rows=1 loops=100)
                           Index Cond: (((routes.source_type)::text = 'Project'::text) AND (routes.source_id = project_authorizations.project_id))
                           Buffers: shared hit=245 read=257 dirtied=13
                           I/O Timings: read=614.837 write=0.000
               ->  Index Scan using projects_pkey on public.projects  (cost=0.44..0.53 rows=1 width=764) (actual time=5.962..5.962 rows=0 loops=100)
                     Index Cond: (projects.id = routes.source_id)
                     Filter: ((NOT projects.pending_delete) AND (NOT projects.hidden) AND (((routes.path)::text ~~* '%gitlab-org/%'::text) OR ((routes.name)::text ~~* '%gitlab-org/%'::text) OR (projects.description ~~* '%gitlab-org/%'::text)))
                     Rows Removed by Filter: 1
                     Buffers: shared hit=160 read=242 dirtied=4
                     I/O Timings: read=588.161 write=0.000

COLD

Time: 732.639 ms
  - planning: 22.165 ms
  - execution: 710.474 ms
    - I/O read: 1.665 s
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 405 (~3.20 MiB) from the buffer pool
  - reads: 639 (~5.00 MiB) from the OS file cache, including disk I/O
  - dirtied: 18 (~144.00 KiB)
  - writes: 0

WARM

Time: 28.765 ms
  - planning: 14.818 ms
  - execution: 13.947 ms
    - I/O read: 0.000 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 1044 (~8.20 MiB) from the buffer pool
  - reads: 0 from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

The cold-cache time exceeds the timing guidelines, but it performs about the same as the old query:

Click to expand
SELECT
    "projects".*
FROM
    "projects"
    INNER JOIN "project_authorizations" ON "projects"."id" = "project_authorizations"."project_id"
WHERE
    "project_authorizations"."user_id" = 8953999
    AND "projects"."pending_delete" = FALSE
    AND (
        "projects"."name" ILIKE '%gitlab-org/%'
        OR "projects"."path" ILIKE '%gitlab-org/%'
        OR "projects"."description" ILIKE '%gitlab-org/%'
    )
    AND "projects"."hidden" = FALSE limit 20;
 Limit  (cost=1066.10..12945.52 rows=3 width=764) (actual time=4.558..13.914 rows=20 loops=1)
   Buffers: shared hit=2039
   I/O Timings: read=0.000 write=0.000
   ->  Gather  (cost=1066.10..12945.52 rows=3 width=764) (actual time=4.556..13.904 rows=20 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         Buffers: shared hit=2039
         I/O Timings: read=0.000 write=0.000
         ->  Nested Loop  (cost=66.10..11945.22 rows=1 width=764) (actual time=1.536..3.396 rows=7 loops=3)
               Buffers: shared hit=2039
               I/O Timings: read=0.000 write=0.000
               ->  Parallel Bitmap Heap Scan on public.project_authorizations  (cost=65.66..6303.79 rows=1645 width=4) (actual time=0.853..1.216 rows=137 loops=3)
                     Buffers: shared hit=393
                     I/O Timings: read=0.000 write=0.000
                     ->  Bitmap Index Scan using project_authorizations_pkey  (cost=0.00..64.68 rows=3947 width=0) (actual time=1.694..1.695 rows=8043 loops=1)
                           Index Cond: (project_authorizations.user_id = 8953999)
                           Buffers: shared hit=40
                           I/O Timings: read=0.000 write=0.000
               ->  Index Scan using projects_pkey on public.projects  (cost=0.44..3.43 rows=1 width=764) (actual time=0.015..0.015 rows=0 loops=411)
                     Index Cond: (projects.id = project_authorizations.project_id)
                     Filter: ((NOT projects.pending_delete) AND (NOT projects.hidden) AND (((projects.name)::text ~~* '%gitlab-org/%'::text) OR ((projects.path)::text ~~* '%gitlab-org/%'::text) OR (projects.description ~~* '%gitlab-org/%'::text)))
                     Rows Removed by Filter: 1
                     Buffers: shared hit=1646
                     I/O Timings: read=0.000 write=0.000

COLD

Time: 959.031 ms
  - planning: 5.326 ms
  - execution: 953.705 ms
    - I/O read: 2.559 s
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 1353 (~10.60 MiB) from the buffer pool
  - reads: 1057 (~8.30 MiB) from the OS file cache, including disk I/O
  - dirtied: 47 (~376.00 KiB)
  - writes: 0

WARM

Time: 20.596 ms
  - planning: 6.509 ms
  - execution: 14.087 ms
    - I/O read: 0.000 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 2039 (~15.90 MiB) from the buffer pool
  - reads: 0 from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Location Feature flag disabled Feature flag enabled
Main menu Screenshot_2023-01-13_at_11.40.42_AM Screenshot_2023-01-13_at_11.49.15_AM
Import project members Screenshot_2023-01-13_at_11.59.13_AM Screenshot_2023-01-13_at_11.57.12_AM
Todo list project filter Screenshot_2023-01-13_at_12.02.47_PM Screenshot_2023-01-13_at_12.04.04_PM

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Log in to gdk as root
  2. Click the hamburger menu in the top left and search for gitlab-org/
  3. Receive no results
  4. Enable the :full_path_project_search feature flag using the rails console
  5. Reload the page
  6. Do the search again
  7. Receive results

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Brian Williams

Merge request reports

Loading