Skip to content

Allow all projects to be indexed

Terri Chu requested to merge 428070-allow-all-projects-to-be-indexed into master

Background

Related to #428070 (closed)

This change only affects instances where Elasticsearch limiting is enabled and configured to limit indexing to specific namespaces or projects. Note: This is how Advanced search is setup on GitLab.com The MR only makes changes to the indexing process but does not touch any code related to search.

This MR is broken into three parts (all behind the feature flag introduced in this MR)

Merge request What it does
⭐ !134456 (merged) Allow all projects being indexed
!137308 (merged) Support all projects being indexed when performing deletes and transfers to another namespace
!138098 (merged) elastic rake task updates

What does this MR do and why?

This MR allows all projects to be indexed and includes:

  • introduction of search_index_all_projects feature flag, gated to the project
  • a new method to determine if the project's associated data is indexed
  • changes to callbacks and the track! method to determine if the associated data should be indexed

Screenshots or screen recordings

No UI changes. Only changes are to determine if a project record is indexed for Advanced search.

How to set up and validate locally

  1. Setup elasticsearch in gdk
  2. Enable advanced search and Elasticsearch indexing restrictions, I chose one group and project image
  3. Disable the feature flag in rails console: Feature.disable(:search_index_all_projects)
  4. Reindex everything from scratch: bundle exec rake gitlab:elastic:index
  5. Verify that only the expected projects exist in the index, in my case that is 3 projects (2 under the selected group, and 1 selected project) 
       curl --request GET \
  --url http://localhost:9200/gitlab-development-projects/_search \
  --data '{
	"query": {
		"bool": {
			"must": [
				{
					"term": {
						"type": {
							"value": "project"
						}
					}		
				}
	    	]
		}
	}
}'
  6. enable the feature flag for a project: Feature.enable(:search_index_all_projects, Project.find(7)) (flightjs project for me)
  7. update the project settings (I updated the description)
  8. verify in the log/elasticsearch.log that ONLY the project record is queued for indexing (not any associated records)
  9. verify that now one more project exists in the index and no associated data exists 
    curl --request GET \
  --url 'http://localhost:9200/gitlab-development-*/_search' \
  --header 'Content-Type: application/json' \
  --data '{
	"query": {
		"bool": {
		    "must": [
			    {
				    "term": {
					    "project_id": {
						    "value": 7
					    }
				    }
			    }
		    ]
	    }
    }
}'
  10. create a new project
  11. verify that it is not queued for indexing and does not exist in the index
  12. enable the feature flag fully: Feature.enable(:search_index_all_projects)
  13. create a new project
  14. verify in the log/elasticsearch.log that ONLY the project record is queued for indexing (not any associated records)
  15. verify that now one more project exists in the index but no associated data

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Terri Chu

Merge request reports