Add the /help page in robots.txt (!17928) · Merge requests · GitLab.org / GitLab FOSS · GitLab

Achilleas Pipinellis - OOO until November 25th (parental leave) requested to merge robots-help into master Mar 22, 2018

What does this MR do?

Exclude the /help page from being crawled. This change is two-fold:

Do not let search engines index the help page of every GitLab installation out there. Searching for gitlab help now yields many results from the help page of random instances.
The /help page mostly has links to internal docs, and we prefer to link to the external docs website instead.

What are the relevant issue numbers?

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/44433

Edited Mar 22, 2018 by Achilleas Pipinellis - OOO until November 25th (parental leave)