WIP: Change namespace billable_members_count to include all descendants
What does this MR do?
cc/ @amandarueda Relevant for you and our discussion
Currently billable_members_count
only includes members in the
current namespace plus descendant namespaces. It does not include
members of projects within the group hierarchy. This change
includes direct and indirect users of any and all groups and
projects within the hierarchy, which is more indicative of
total billable users.
This change does not include counting members of groups that a given project in the hierarchy may be shared with. We currently don't have an easy mechanism for pulling this information. We'll have to do a little more digging to figure out a performant way to do that.
Concerns
I'm slightly concerned about the performance on this change. As you can see below it's already more than 3.5x slower with just 4 users in the system. What happens on GitLab.com?
Before we were doing a fairly simple query:
(1.5ms) SELECT COUNT(*) FROM "users" WHERE "users"."id" IN (SELECT "members"."user_id" FROM "members" LEFT OUTER JOIN "users" ON "members"."user_id" = "users"."id" WHERE "members"."type" IN ('GroupMember') AND "members"."source_type" = $1 AND "users"."state" = $2 AND "members"."requested_at" IS NULL AND "members"."source_id" IN (WITH RECURSIVE "base_and_descendants" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."id" = 8)
UNION
(SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT "id" FROM "base_and_descendants" AS "namespaces")) [["source_type", "Namespace"], ["state", "active"]]
Now we're doing a bit more complex query:
(5.8ms) SELECT COUNT(*) FROM ((SELECT "users".* FROM "users" WHERE "users"."id" IN (SELECT "members"."user_id" FROM "members" LEFT OUTER JOIN "users" ON "members"."user_id" = "users"."id" WHERE "members"."type" IN ('GroupMember') AND "members"."source_type" = 'Namespace' AND "users"."state" = 'active' AND "members"."requested_at" IS NULL AND "members"."source_id" IN (WITH RECURSIVE "base_and_ancestors" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."id" = 8)
UNION
(SELECT "namespaces".* FROM "namespaces", "base_and_ancestors" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."id" = "base_and_ancestors"."parent_id")), "base_and_descendants" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."id" = 8)
UNION
(SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT "namespaces"."id" FROM ((SELECT "namespaces".* FROM "base_and_ancestors" AS "namespaces" WHERE "namespaces"."type" IN ('Group'))
UNION
(SELECT "namespaces".* FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."type" IN ('Group'))) namespaces WHERE "namespaces"."type" IN ('Group'))))
UNION
(SELECT "users".* FROM "users" INNER JOIN "members" ON "members"."user_id" = "users"."id" AND "members"."type" IN ('ProjectMember') AND "members"."source_type" = 'Project' AND "members"."requested_at" IS NULL INNER JOIN "projects" ON "projects"."id" = "members"."source_id" INNER JOIN "namespaces" ON "namespaces"."id" = "projects"."namespace_id" AND "namespaces"."type" IN ('Group') AND "namespaces"."type" = 'Group' WHERE "namespaces"."id" IN (WITH RECURSIVE "base_and_descendants" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."id" = 8)
UNION
(SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT "id" FROM "base_and_descendants" AS "namespaces"))) users
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team