Querying labels for groups times out

When using the label filter, we query the recent labels for a group or project. However, when querying these labels for a group we get a timeout for gitlab-com and gitlab-org groups (likely more). The issue seems to be that we generate a lot of queries:

• This is also likely not a recent change, see: https://log.gprd.gitlab.net/goto/9cf2e850-9b2a-11ed-85ed-e7557b0a598c
• We have permission changes that loop over the groups https://gitlab.com/gitlab-org/gitlab/-/blob/dd1e70d3676891025534dc4a1e89ca9383178fe7/app/finders/concerns/finder_with_group_hierarchy.rb#L71

cc @jprovaznik @engwan (thanks for the debugging info for this issue)