Skip to content

De-normalize issue permission data and add data version to document

Release Post tile: Elasticsearch Migration makes for faster Advanced Search experience

Issue permissions are currently queried using the project_visibility_level and issues_access_level stored in the Project parent documents. We need to move the permission data to the issue documents as well as add a migration version field. It should be possible to just use the issues_access_level to determine whether issues should be searchable based on the options laid out in the table below.

UPDATE: it was found that both project visibility_level AND issues_access_level are required to correctly identify internal project and their associated issues permissions.

setting project_visibility_level issues_access_level (options available)
private 0 10 / 0
internal 10 20 / 10 / 0
public 20 20 / 10 / 0

Proposed solution

should be done over multiple MRs

Note: The migration to add new fields to the index needs to be done in it's own MR before the mapping config and json are updated.

  • issues_access_level (int)
  • visibility_level (int)
  • gitlab_migration_version (text, but we expect to store uuids here in future) (we will not be doing this until it is required, and will track that in work #284603)
  • Update mapping config in code
  • Update json generated for issues to include new field (issues_access_level)
  • Add hooks to kick off updates of issue document data if project's issues access level is updated
  • Update ElasticDeleteProjectWorker so that project deletes also delete issue documents by project_id
  • Add migration to add new data to existing issue documents in combined index. The migration should be written using the new framework. All issues missing the new field should be queued for indexing. Indexing may not need to paused during this migration, but that should be validated during testing.

Epic order

  1. Remove joins from the Elasticsearch query for project/group scoped issues search
  2. De-normalize the issue permission data. <-- will include migration and can be done in multiple MRs
  3. Remove joins from the Elasticsearch query for globally scoped issues search
  4. Migration to copy issue data to new index
  5. Migration to remove issue data from combined index **
  6. Rake task changes to support new index **
  7. Backend changes for supporting new indexes during Zero Downtime Re-indexing **
  8. Admin UI changes to support new index (any time after new index created/data migrated to it)

** must be released in conjunction with the Migration to copy issue data to new index

Release notes

Prior to this release, searching issues in Advanced Search used a combination of project and issue permissions stored in the project and not the issue. These checks would sometimes result in lengthy page load times.

Now the Elasticsearch migration copies the project and issue permissions to the issue documents. You can read more about our plans to expand this capability on,"Migrate blobs to dedicated Elasticsearch index"

https://docs.gitlab.com/ee/user/search/advanced_global_search.html#faster-searches

Edited by Terri Chu