Clarify where to run ChatOps commands when toggling feature flags

The following discussion from !85950 (merged) should be addressed:

• @pgascouvaillancourt started a discussion: (+3 comments)

  thought: This might warrant a follow-up, but we tend to run --staging commands in the #staging channel. Perhaps we should adjust this template accordingly 🤔

---

After some discussion in this issue, it appears that we might not need to enforce stricter guidelines on where to toggle feature flags outside of the production environment. The only outcome of this issue will be to link to https://docs.gitlab.com/ee/development/feature_flags/controls.html#rolling-out-changes from https://docs.gitlab.com/ee/development/feature_flags/ to make that section more discoverable.

Below is the initial proposal that will not be applied after all:

---

There seems to be inconsistencies as to where ChatOps commands should be run when toggling feature flags in Slack.

It seems to be generally understood that toggling a feature flag in production should be done in the #production channel. It seems to be less clear for staging feature flags: !85950 (comment 926847516)

I found 147 messages when searching for "--staging" in #production, and 332 for the same query in #staging

It appears that, most of the time, staging feature flags are being toggled in the #staging channel, but it is quite common to see them being toggled in the #production channel as well.

Additionally, feature flags are sometimes toggled from completely different channels. Theoretically, ChatOps commands can also be run in DMs, so it should be possible to toggle feature flags in a very opaque fashion (to be confirmed).

These inconsistencies make it hard to track feature flag changes. The full logs can be found in https://ops.gitlab.net/gitlab-com/chatops/-/pipelines, but it is generally easier to search for messages in Slack to see when a flag was last updated. There seems to be little documentation on best practices when using ChatOps too. This documentation page shows a few example commands but doesn't provide much info about them: https://docs.gitlab.com/ee/development/feature_flags/.

We might want to clarify the documentation around how we use ChatOps and issue a few reminders to the engineers. Here's a proposal:

• Add a Toggling feature flags' state section in https://docs.gitlab.com/ee/development/feature_flags/. This new documentation section should explain how we use ChatOps and where the commands should be run.
  • I propose that we document the workflow that's already being used by the majority:
    • ChatOps command should only be run in environment-specific channels (#production, #staging).
    • Toggling a feature flag's state in an environment should be done in the corresponding environment-specific channel (enabling a flag in staging should be done in #staging).
• Update the Feature Flag Roll Out issue template to reflect the documentation changes.
• Announce the changes in the Engineering Week-in-Review.
• Announce in the #development channel.