gitlab-rails: check dependencies directly instead of via Rake call
Summary
Address gitlab-org/charts/gitlab#1775 (closed)'s observation of the amount of time needed to complete init, by replacing calls to Rake with direct calls to the resources we need to check (Redis / PostgreSQL)
Indicators of "complete"
-
Redis -
Supports Sentinel -
Supports rediss://
-
Checks all configured Redis servers -
Ruby only, as this covers "all" bases of connectivity support within Rails
-
-
PostgreSQL -
Supports single database endpoint -
Supports load-balancing using hosts(not needed) -
Supports load-balancing using service discovery(not needed)
-
-
(Re)starts Ruby as few times as possible, loading as few modules as possible.
Details
Implements the basic concept derived from this comment, which is to directly query the various systems.
- Add scripts to directly access PostgreSQL and Redis dependencies
-
rails-db-check
: Ruby script with absolute minimum viability, limited to only checking the connection, and the state of the migrations according to ActiveRecord. -
rails-redis-ping
: uses the Redis gem to directly connect and "ping" redis, input withresque.yml
and variants.
-
- Update
wait-for-deps
script to call direct access scripts instead of callingrake
.
Impact
Without the need to spin up Rails via the Rake call, we can make a quite large impact on the time it takes to find out if we're "ready".
How big? Observe the below, performed against operational deployments (e.g. "shortest possible"), that have had Rake commands run several times:
git@gke-production-a4b9oa-task-runner-77fcf8cb79-8lgpf:/$ time /scripts/wait-for-deps
Checking database connection and schema version
Database Schema - current: 20200407094923, codebase: 20200407094923
real 0m11.174s
user 0m9.497s
sys 0m1.335s
git@gitlab-task-runner-7459858f85-9cwf7:/srv/gitlab$ time /scripts/wait-for-deps
Checking database connectivity and schema version
Database Schema - current: 20200511145545, codebase: 20200511145545
Checking Redis connectivity
Checking 'resque.yml'
- Attempting to connect to 'redis://gitlab-redis-master:6379' SUCCESS
real 0m3.459s
user 0m2.010s
sys 0m1.367s
Former, with use of `psql -c` directly
git@demo-task-runner-74b4679696-vvpxl:/srv/gitlab$ time /scripts/wait-for-deps Checking database connection and schema version Database Schema - current: 20200313123934, codebase: 20200313123934 Checking Redis connectivity - checking resque.yml GOODreal 0m1.366s user 0m1.034s sys 0m0.309s
cc @skarbek
Edited by Robert Speicher