Resolve "Geo: Actively try to correct verification failures on the secondary"

What does this MR do?

Actively try to correct verification failures on the secondary when we detect a checksum failure or mismatch attempting a resync with a progressive backoff.

Are there points in the code the reviewer needs to double check?

Why was this MR needed?

Right now if we detect a checksum mismatch between the primary and secondary, we don't do anything about it requiring administrators to intervene manually.

Does this MR meet the acceptance criteria?

• Changelog entry added, if necessary
• Documentation created/updated
• API support added
• Tests added for this feature/bug
• Conforms to the code review guidelines
  • Has been reviewed by a Backend maintainer
  • Has been reviewed by a Database specialist
• EE specific content should be in the top level /ee folder
• Conforms to the merge request performance guidelines
• Conforms to the style guides
• Conforms to the database guides
• If you have multiple commits, please combine them into a few logically organized commits by squashing them
• Internationalization required/considered
• If paid feature, have we considered GitLab.com plan and how it works for groups and is there a design for promoting it to users who aren't on the correct plan
• End-to-end tests pass (package-and-qa manual pipeline job)

What are the relevant issue numbers?

Closes #5195 (closed)