Disallow some project routes

What does this MR do?

This MR disallows some project routes in robots.txt. These routes are:

• /*/-/import
• /*/-/environments
• /*/-/jobs
• /*/-/requirements_management
• /*/-/pipelines
• /*/-/pipeline_schedules
• /*/-/dependencies
• /*/-/licenses
• /*/-/metrics
• /*/-/incidents
• /*/-/value_stream_analytics
• /*/-/analytics
• /*/insights

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Separation of EE specific content

Related to #273148 (closed)