Skip to content

Denormalize notes permission data & update when permissions change

Terri Chu requested to merge 300351-remove-joins-from-the-elasticsearch-query-for-project-group-scoped-notes-comments-search into master

What does this MR do?

Related to #300351 (closed)

Migration to denormalize the notes project permission data required to remove the project parent joins from Elasticsearch Notes scoped queries. This MR will

  • add a migration to move permission data to all notes documents
  • update notes when project or project feature permissions are changed

Notes can be tied to multiple types of data so we will only track related permission data on each note to reduce index size. Project permission data that will be tracked:

  • visibility_level (All types)
  • issues_access_level (Issue type)
  • repository_access_level (Commit type)
  • snippets_access_level (Snippet type)
  • merge_requests_access_level (Merge Request type)

Tested locally using ES6 and ES7

Runtime expectations on GitLab.com

There are approximately 34,000,000 notes in the GitLab.com Elasticsearch index.

command run: curl "$CLUSTER_URL/_count?q=type:note"

The migration will run batches of 5,000 notes every 1 minute

Time estimate (GitLab.com): 114 hours

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Terri Chu

Merge request reports