Extended mirrored repository failures should lead to automatic mirror disablement
On GitLab.com, we see a large number of repository mirror attempts with haven't succeeded in months, or worse, have never succeeded.
These attempts are not only resource intensive for us to carry out over and over with the same outcome, but also make us into a bad neighbour to other Git hosting services. This is because we will continually poll git repositories that don't exist, generating unnecessary traffic. Sometimes the "other" endpoint is GitLab.com, creating even more unnecessary traffic for us.
Over time, this traffic builds up, and in some cases is leading to pagerduty alerts and SLO violations on some of our Gitaly nodes. Slack thread: https://gitlab.slack.com/archives/C101F3796/p1612351719221500
Proposal
If a repository is uncontactable for X days we should disable mirroring for that repository, possibly sending an email to the repository maintainers.