Move NOTES (COMMENTS) to dedicated Elasticsearch index

This issue is for connecting to the primary MR for this EPIC &5197 (closed)

Problem

Today we use a single index for all data types. There are many problems with this described in #3217 (closed) but a major motivating factor for moving to indices for each data type is performance. Based on #3217 (comment 399228783) we can reduce the total response time of searches for issues by 54% if those issues are in a dedicated index rather than the combined index with all the other data types.

We will likely see similar benefits for other data types but we should just start with 1 data type for now to iterate and learn at each point.

Solution

Move the notes to a dedicated index. This will have several challenges that we need to overcome and may require several MRs to accomplish.

Steps

1. Denormalise the notes permission data per &2054 . This may be done over a few MRs:
   1. Denormalise the project issues, merge requests, snippets, and repository visibility level and project visibility level onto the notes document type. This requires a migration to update all the existing notes with this field. It also requires triggering an notes update whenever the project permissions change.
   2. Change project and group-wide searches to use the new denormalized fields along side the inline permission (ie. just confidential field) and the existing project_id field already on the notes.
   3. Change global searches to use the new denormalized fields rather than the join to parent
2. Add support for setting the notes used when indexing based on the record type (the bulk indexers all hardcode gitlab-<ENV> that will need to change. We can still use the single bulk indexer since bulk indexing can support to writing to multiple indices.
3. Add a migration that pauses indexing, moves the notes over to the new index, ensures reads and writes for notes are now coming from the new index and then deletes the notes from the existing index

Proposal (issues will be opened for each)

These should be done in order unless otherwise noted

1. De-normalize the notes (comments) permission data and update notes when permission data changes. <-- will include migration
2. Remove joins from the Elasticsearch query for project/group scoped notes (comments) search
3. Remove joins from the Elasticsearch query for globally scoped notes (comments) search
4. Migration to copy notes (comments) data to new index
5. Migration to remove notes (comments) data from combined index **

** must be released in conjunction with the Migration to copy notes data to new index

Release Notes