Ensure that each new codepath of a new methods are executed at least once as part of test suite

What problem are we trying to solve?

We were hit by the very tricky set of merge requests being merged, with a mix of feature flags, and really hard time to discover this failure. (We were lucky that the problematic codepath was mitigated by disabling a feature flag. However, we would have had a more significant problem if we did not have the feature flag). The reason why we did not notice this issue was that there was no coverage for the changed method, due to stubbing, and also exclusion from coverage reports.

Purpose

The purpose of this issue is to discuss and evaluate options for ensuring that each new codepath of a new added method is actually executed at least once as part of test suite.

Proposed Solutions

1. We do have code coverage that we could use for that purpose. Initially it might be good enough to see if the method was actually executed. CONCLUSION: we do have code coverage visualization but it has a bug, opened #345987 (closed)
2. Fix exclusion of lib/gitlab/sidekiq_middleware/ from coverage report !74779 (merged)
3. Make code coverage visualization be more noticeable (or even block the MR from being merged) if there's lines with no coverage hits !74859 (merged)
4. [-] Figure out what gaps we have with respect to enabling OUT OF SCOPE:
   • environment variables
   • if GitLab.com?
   • Rails environment
   • feature flag

Related Incident

2021-11-16 pipelines failing on public and private runners