Draft: Access runners via `admin_runners` permission (!159657) · Merge requests · GitLab.org / GitLab

mo khan requested to merge mokhax/442851/get-runners-second-attempt into master Jul 16, 2024

What does this MR do and why?

This change allows users that belong to a custom role with the admin_runners permission enabled to be able to view the list of runners through via their custom role membership. It was previously merged in !156840 (merged) and reverted in !159621 (merged).

In the previous MR the service degradation was caused by the usage of the ::Preloaders::UserMemberRolesInProjectsPreloader and ::Preloaders::UserMemberRolesInProjectsPreloader preloaders. This is because these preloaders load all the permissions for the given user across all groups and projects. In this MR, I have attempted to optimize this by querying for groups/projects that are associated with custom roles that have the target permission (:admin_runners) enabled. This reduces the # of ::Group and ::Project records that are loaded into memory.

#442851 (closed)

Before:

SELECT ci_runners.* 
FROM (
  (
    SELECT ci_runners.*
    FROM ci_runners
    INNER JOIN ci_runner_projects runner_projects ON runner_projects.runner_id = ci_runners.id
    WHERE runner_projects.project_id IN (40966370, 43592462, 57633207, 57998909, 43723469, 43485681, 44025715, 45049685, 58558240, 58179498)
  )
  UNION
  (
    WITH cte_namespace_ids AS MATERIALIZED (
      SELECT ci_namespace_mirrors.namespace_id
      FROM (
        (
          SELECT ci_namespace_mirrors.*
          FROM ci_namespace_mirrors
          WHERE (((traversal_ids[1]),(traversal_ids[2]),(traversal_ids[3]),(traversal_ids[4]),(traversal_ids[5])) IN ((9970,11787569,60126506,60423513,69913952),(9970,11787569,60126506,60423513,70405406)))
        )
        UNION
        (
          SELECT ci_namespace_mirrors.*
          FROM ci_namespace_mirrors
          WHERE (((traversal_ids[1])) IN ((60357594),(70410142)))
        )
      ) ci_namespace_mirrors
    ),
    cte_project_ids AS MATERIALIZED (
      SELECT ci_project_mirrors.project_id
      FROM ci_project_mirrors
      WHERE (ci_project_mirrors.namespace_id IN (SELECT namespace_id FROM cte_namespace_ids))
    )
    SELECT ci_runners.*
    FROM ci_runners
    INNER JOIN ci_runner_projects ON ci_runner_projects.runner_id = ci_runners.id
    WHERE (ci_runner_projects.project_id IN (SELECT project_id FROM cte_project_ids))
  )
  UNION
  (
    WITH cte_namespace_ids AS MATERIALIZED (
      SELECT ci_namespace_mirrors.namespace_id
      FROM (
        (
          SELECT ci_namespace_mirrors.*
          FROM ci_namespace_mirrors
          WHERE (((traversal_ids[1]),(traversal_ids[2]),(traversal_ids[3]),(traversal_ids[4]),(traversal_ids[5])) IN ((9970,11787569,60126506,60423513,69913952),(9970,11787569,60126506,60423513,70405406)))
        )
        UNION
        (
          SELECT ci_namespace_mirrors.*
          FROM ci_namespace_mirrors
          WHERE (((traversal_ids[1])) IN ((60357594),(70410142)))
        )
      ) ci_namespace_mirrors
    ) SELECT ci_runners.*
    FROM ci_runners
    INNER JOIN ci_runner_namespaces ON ci_runner_namespaces.runner_id = ci_runners.id
    WHERE (ci_runner_namespaces.namespace_id IN (SELECT namespace_id FROM cte_namespace_ids))
  )
) ci_runners
ORDER BY "ci_runners"."id" ASC
LIMIT 20
OFFSET 0;

Time: 7.294 ms
  - planning: 5.238 ms
  - execution: 2.056 ms
    - I/O read: 0.000 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 518 (~4.00 MiB) from the buffer pool
  - reads: 0 from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/29909/commands/92949

After:

SELECT "ci_runners".*
FROM (
  (
    SELECT "ci_runners".*
    FROM (
      (
        SELECT "ci_runners".*
        FROM "ci_runners"
        INNER JOIN "ci_runner_projects" "runner_projects" ON "runner_projects"."runner_id" = "ci_runners"."id"
        WHERE runner_projects.project_id IN (40966370, 43592462, 57633207, 57998909, 43723469, 43485681, 44025715, 45049685, 58558240, 58179498)
      )
      UNION
      (
        WITH "cte_namespace_ids" AS MATERIALIZED (
          SELECT "ci_namespace_mirrors"."namespace_id"
          FROM (
            (
              SELECT "ci_namespace_mirrors".*
              FROM "ci_namespace_mirrors"
              WHERE (((traversal_ids[1]),(traversal_ids[2]),(traversal_ids[3]),(traversal_ids[4]),(traversal_ids[5])) IN ((9970,11787569,60126506,60423513,69913952),(9970,11787569,60126506,60423513,70405406)))
            )
            UNION
            (
              SELECT "ci_namespace_mirrors".*
              FROM "ci_namespace_mirrors"
              WHERE (((traversal_ids[1])) IN ((60357594),(70410142)))
            )
          ) ci_namespace_mirrors
        ),
        "cte_project_ids" AS MATERIALIZED (
          SELECT "ci_project_mirrors"."project_id"
          FROM "ci_project_mirrors"
          WHERE (ci_project_mirrors.namespace_id IN (SELECT namespace_id FROM cte_namespace_ids))
        )
        SELECT "ci_runners".*
        FROM "ci_runners"
        INNER JOIN "ci_runner_projects" ON "ci_runner_projects"."runner_id" = "ci_runners"."id"
        WHERE (ci_runner_projects.project_id IN (SELECT project_id FROM cte_project_ids))
      )
      UNION
      (
        WITH "cte_namespace_ids" AS MATERIALIZED (
          SELECT "ci_namespace_mirrors"."namespace_id"
          FROM (
            (
              SELECT "ci_namespace_mirrors".*
              FROM "ci_namespace_mirrors"
              WHERE (((traversal_ids[1]),(traversal_ids[2]),(traversal_ids[3]),(traversal_ids[4]),(traversal_ids[5])) IN ((9970,11787569,60126506,60423513,69913952),(9970,11787569,60126506,60423513,70405406)))
            )
            UNION
            (
              SELECT "ci_namespace_mirrors".*
              FROM "ci_namespace_mirrors"
              WHERE (((traversal_ids[1])) IN ((60357594),(70410142)))
            )
          ) ci_namespace_mirrors
        )
        SELECT "ci_runners".*
        FROM "ci_runners"
        INNER JOIN "ci_runner_namespaces" ON "ci_runner_namespaces"."runner_id" = "ci_runners"."id"
        WHERE (ci_runner_namespaces.namespace_id IN (SELECT namespace_id FROM cte_namespace_ids))
      )
    ) ci_runners
  )
  UNION
  (
    SELECT "ci_runners".*
    FROM "ci_runners"
    INNER JOIN "ci_runner_namespaces" ON "ci_runner_namespaces"."runner_id" = "ci_runners"."id"
    WHERE ci_runner_namespaces.namespace_id = 60357594
  )
  UNION
  (
    SELECT "ci_runners".*
    FROM "ci_runners"
    INNER JOIN "ci_runner_projects" ON "ci_runner_projects"."runner_id" = "ci_runners"."id"
    WHERE ci_runner_projects.project_id = 44097722
  )
) ci_runners
ORDER BY "ci_runners"."id" ASC
LIMIT 20 OFFSET 0;

Time: 4.796 ms
  - planning: 4.161 ms
  - execution: 0.635 ms
    - I/O read: 0.000 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 3 (~24.00 KiB) from the buffer pool
  - reads: 0 from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

https://console.postgres.ai/gitlab/gitlab-production-main/sessions/30289/commands/93950

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Before:

モ curl -H "PRIVATE-TOKEN: $GITLAB_TOKEN" http://gdk.test:3000/api/v4/runners | jq '.'
[]

After:

モ curl -H "PRIVATE-TOKEN: $GITLAB_TOKEN" http://gdk.test:3000/api/v4/runners | jq '.'
[
  {
    "id": 26,
    "description": "example",
    "ip_address": null,
    "active": false,
    "paused": true,
    "is_shared": false,
    "runner_type": "project_type",
    "name": null,
    "online": null,
    "status": "stale"
  },
  {
    "id": 27,
    "description": "example",
    "ip_address": null,
    "active": false,
    "paused": true,
    "is_shared": false,
    "runner_type": "project_type",
    "name": null,
    "online": null,
    "status": "stale"
  },
  {
    "id": 32,
    "description": "",
    "ip_address": null,
    "active": true,
    "paused": false,
    "is_shared": false,
    "runner_type": "group_type",
    "name": null,
    "online": null,
    "status": "stale"
  },
  {
    "id": 39,
    "description": null,
    "ip_address": null,
    "active": true,
    "paused": false,
    "is_shared": false,
    "runner_type": "project_type",
    "name": null,
    "online": null,
    "status": "stale"
  },
  {
    "id": 40,
    "description": null,
    "ip_address": null,
    "active": true,
    "paused": false,
    "is_shared": false,
    "runner_type": "group_type",
    "name": null,
    "online": null,
    "status": "stale"
  }
]

How to set up and validate locally

In rails console enable the experiment fully

Feature.enable(:custom_ability_admin_runners)

Visit any root group
Create group runners at different levels of the group hierarchy.
Create project runners for different projects in and outside of the chosen group hierarchy.
Create a new role role with the base role of Guest and the :admin_runners permission.
Assign the role to a new user.
Generate a PAT for the new user.

Make a curl request to runners API

$ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners"

Verify that all the expected runners are returned in the JSON response.

Edited Jul 27, 2024 by mo khan

Draft: Access runners via `admin_runners` permission

What does this MR do and why?

MR acceptance checklist

Screenshots or screen recordings

How to set up and validate locally

Merge request reports