With downtime upgrades for scaled deployments

What does this MR do?

Adds documentation for upgrading multi-node GitLab deployments with downtime.

pulled from this comment

• restart postgresql during the downtime
• for the duration, Sidekiq and Puma should be shut down, and then formally restarted at the end.

* [ ] Monitor should be shut down as well. Don't want it poking around in things that aren't running, flagging everything red, and trying to run "stale" queries over the database while the schema is being changed.

* [ ] Question: does this mean the exporters need to be down, or just the monitor node(s)?

• Monitor just has to be included somewhere, for now. If we get feedback about problems, we can incorporate changes.

• [ ] While the Praefect database is being restarted, the Praefects need to be shut down. They'll have no clients so that won't matter. Gitaly nodes won't mind, it'll just be very quiet.
  • this will not make the cut as at present we're linking to the zero downtime docs.
• consul to be upgraded while rails is offline.
• Redis to be upgraded while rails is offline
• ensure that when a reconfigure implicity .. a restart is done .. eg: gitaly

---

• nod to how to handle major upgrades with backend and frontend temporarly being on different versions. Answer: frontend is down until it's upgraded and on the same version, so not a problem.

---

• procedure from this comment](omnibus-gitlab#6244 (comment 617714626))

1. Stop writes to the database (shutting down all rails nodes, or at least stopping puma/sidekiq everywhere)
2. Select a rails deploy node that will run the migrations
3. Run the upgrade on the deploy node, (if jumping multiple minor version you must not use SKIP_POST_DEPLOYMENT_MIGRATIONS)
4. Restart the deploy node and ensure all services are up
5. Upgrade all the rest of the nodes and restart them (once the deploy node is done, all the rest can be done at the same time or individually)

Related issues

omnibus-gitlab#6244 (closed)

Author's checklist

• Optional. Consider taking the GitLab Technical Writing Fundamentals course.
• Follow the:
  • Documentation process.
  • Documentation guidelines.
  • Style Guide.
• If you're adding or changing the main heading of the page (H1), ensure that the product tier badge is added.
• If you are a GitLab team member, request a review based on:
  • The documentation page's metadata.
  • The associated Technical Writer.

If you are a GitLab team member and only adding documentation, do not add any of the following labels:

• ~"frontend"
• ~"backend"
• ~"type::bug"
• ~"database"

These labels cause the MR to be added to code verification QA issues.

Reviewer's checklist

Documentation-related MRs should be reviewed by a Technical Writer for a non-blocking review, based on Documentation Guidelines and the Style Guide.

• If the content requires it, ensure the information is reviewed by a subject matter expert.
• Technical writer review items:
  • Ensure docs metadata is present and up-to-date.
  • Ensure the appropriate labels are added to this MR.
  • Ensure a release milestone is set.
  • If relevant to this MR, ensure content topic type principles are in use, including:
    • The headings should be something you'd do a Google search for. Instead of Default behavior, say something like Default behavior when you close an issue.
    • The headings (other than the page title) should be active. Instead of Configuring GDK, say something like Configure GDK.
    • Any task steps should be written as a numbered list.
    • If the content still needs to be edited for topic types, you can create a follow-up issue with the docs-technical-debt label.
• Review by assigned maintainer, who can always request/require the reviews above. Maintainer's review can occur before or after a technical writer review.