Export detailed membership data

What does this MR do and why?

It adds a new service that prepares data for the export of exact memberships of a certain group and its subgroups.

Follow-up MRs (there are subtasks on the original issue for these):

• endpoint to call this service
• export also memberships of the projects

Description of the solution

DetailedDataService is inherited from existing Namespaces::Export::BaseService, it calls the GroupsIterator` to get the data for the export.

The iterator gets the provided group as the root group from which it iterates over the descendants. It uses Batch from any node from the group hierarchy strategy. As it is depth-first algorithm, we can calculate the members of every subgroups based on the parent members combined with direct members. This makes the db query for every group simpler (we have to get only direct + shared members for every subgroup, we calculate the inherited members).

GroupMembersProcessor is the class responsible for combining members of every subgroup. It takes the inherited members and combine them with direct and shared members. The direct member is always taken, the highest access level is taken if a user is member both inherited and shared.

DataTransformer takes the members and transforms them to Namespaces::Export::Member object. We need this new object because we need to know for every group and member not only the source group (that is actually on the member model) but also the "current group" (the group we will show the membership for).

Database queries

Data

• the biggest group is gitlab-com and it has 12368 members (which includes all membership types). This is maximum number of members we will probably work with in this algorithm.
• In regards to namespace counts, the biggest group has 75072 of them.
• more details in an internal note

New queries

Group.order_by_ids_sequence

• source
• query plan
• I took 10 random existing groups, we will have max 100 groups as we do group batches of 100

Existing queries usage

memberships_for_group

• source
• query plan
• this query will run for each group in the structure
• the query plan is for gitlab.com group, id 6543

Follow-up issues

• Detailed membership export: Make sure we export... (#477403)
• Iteration 1.5: Export project memberships for d... (#477404 - closed)

Related MRs

• Add worker to export detailed group memberships... (!159375 - merged)
• Add endpoint to export detailed data about user... (!160096 - merged)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

There are no changes in the functionality yet.

How to set up and validate locally

1. In rails console enable the feature flag

   Feature.enable(:members_permissions_detailed_export)

2. Find a group with a deeper hierarchy or create a group with some subgroups

3. Assign some users to the group and its subgroups, you can play also with custom roles assignment and memberships inheritance

4. Go to the rails console and call the service, check the data

result = Namespaces::Export::DetailedDataService.new(container: group, current_user: User.first).execute
CSV.parse(result.payload, headers: true)

Related to: #460477 (closed), #468889 (closed)