Introduce a feature flag for Resolve "Notifications for when pipelines are fixed""

The following discussion from !16951 (merged) should be addressed:

• @dosuken123 started a discussion: (+2 comments)

  Should we put a feature flag here to allow us to easily rollback the behavior when something went wrong on production?

For the above we'll introduce a separate FF