Skip to content

Advanced Search: Support group-level wiki

Problem to solve

Currently, Advanced Search only supports searching wikis on a project level. Our indexing code doesn't have support for repositories that belongs to groups (a wiki is a repository under the hood). Customers are confused when they search for wikis in a group expecting to see group wikis, but they only get project wikis results.

Intended users

Customers who use and create wikis on a group level.

Proposal

When users search within a group only then wikis related to the selected group and wikis related to all the projects inside this group will be shown.

Make group-level wikis searchable. project-level wikis are stored in the main index. There may not be any advantage to using that index since group-level wikis are not associated with a project. Permissions will need to be propagated into the documents

Technical steps:

Consideration to move all project wikis to their own wiki index as a first step. &6979 (closed)

  1. backend Advanced Search migration to create a new index for all wikis and backfill Project wikis over to new index (calling the indexer worker with some flag to trigger it). We would need to index in both old/new indexes until the backfill is completed. Add hashed namespace_id to the new index. The new index mappings should support both project and group wikis. Do not think we should use the default routing, we would hit all shards in the index for a specific top level namespace. For new index, we should use routing of group_<id> (not top level namespace) for GroupWiki and project_<id> for ProjectWiki. The routing should only be used when performing project scoped searches, it cannot be used for group or global searches. We cannot use the routing when searching on a parent group, because it would miss results (we are not using top level namespace so we may miss descendant results)
  2. backend Add functionality to the gitlab-elasticsearch-indexer to index group level wikis.
    • The command line flag --project-path should be deprecated (and scheduled for removal in the next major milestone).
    • A new command line flag --path should be used in it's place to support group and project level indexing of wiki/repositories.
    • Change project_id to a named argument (currently it is an argument) and should be made optional.
    • Add group_id as a named argument, maybe this should be required (all data has an associated group_id, even repositories and project-wikis through the project)
    • Consider traversal_ids (formerly known as namespace_ancestry) optimization at time of indexing (an optional argument). group_id could also be pulled from this argument.
    • We might consider adding a new --blob-type, could be used to determine which index to send to and how to build the document up (will not need this since we are going to use one index)
  3. backend Support indexing for group wikis
    • create new WikiInstanceProxy and WikiClassProxy classes, these will eventually replace the existing ProjectWikiInstanceProxy and ProjectWikiClassProxy but they cannot be removed until the next major release (there will be a migration completed guard to determine which to use). They should be marked as deprecated with a link to an issue with a milestone of 17.0
    • updates for ElasticCommitIndexerWorker
      • use options to pass group_id and need to adjust the code to not return if project_id is nil
    • updates for Gitlab::Elastic::Indexer
      • need to pass group to the elasticsearch_indexer and it will also pass the new arguments for index name to the elasticsearch_indexer
  4. backend Support for removal from index when a group is deleted (like ElasticProjetDeleteWorker but for groups)
    • ElasticProjectDeleteWorker needs to be updated to delete from new wiki index
    • ElasticGroupDeleteWorker needs to be created
  5. backend Support searching for group wikis, behind feature flag
  6. backend Advanced Search migration to backfill data into index
  7. backend Add support for searching via APIs
  8. frontend Support searching for group wikis: #385593
  9. add any deprecation issues required and set to milestone of 17.0

Progress status

Indexer

  1. Add wiki-access-level & WikiPermissions -> !175 ✅
  2. Rename --project-path to --path -> !176 ✅
  3. Replace argument with --project-id -> !178 ✅
  4. Release intermediate indexer version -> !180 ✅
  5. Separate index for wikis (+ add group_id) -> !174 ✅

GitLab

  1. Introduce Elastic::GroupIndexStatus to support group level wikis -> !107579 (merged) ✅
  2. Update Gitlab::Elastic::Indexer to support new parameters (--full-path and --project_id) & update indexer version -> !107668 (merged) ✅
  3. Add a migration to add project permissions to wiki documents -> !116944 (merged) ✅
  4. Add indexing for project wikis to the separate index and add searching for project wiki -> !107365 (merged) ✅
  5. Add indexing for group wikis -> !120171 (merged) ✅
  6. Migrate existing Group Wikis to Elasticsearch -> !122529 (merged) ✅
  7. Add searching logic for the GroupWikis -> !123522 (merged) ✅
  8. Update [documentation](https://docs.gitlab.com/ee/user/project/wiki/group.html -> !123522 (merged) ✅
  9. Delete blob_wiki documents from the main index -> !126538 (merged)

Release notes

The group-level wiki will be included in Advanced global and group wiki searches.

Edited by Ravi Kumar