Create a service to generate npm packages metadata on the group and instance level

Description

For the context and discussion please see the thread. In two words: the npm metadata generation might take a lot of time for large group hierarchies. Example query link (internal).

We need to change the strategy of how we generate the metadata when we only know the root namespace or group. Instead of first doing a project lookup and checking for the required permissions, etc., we could filter out the packages by scoped name and than apply all additional logic: namespaces lookup, projects authorizations check and projects features check.

Simplified version:

Packages.where(name: name).each_batch do |scope|
  records = scope.apply_project_filter_and_everything_else
end

🌱 Context

We earlier did these refactors:

• Refactor NPM metadata endpoint (!154577 - merged)
• Inline helper methods to the npm endpoint classes (!154919 - merged) (in progress)

Now that we have a clear separation between the helpers of each endpoint (project, group, instance), we can:

• move the existing generate_metadata_service helper to the project endpoint helpers
• create a new generate_metadata_service helper that implements the each_batch optimization described above.
  • NOTE: This requires a modification to the existing index, see thread