Backfill runner semver column
What does this MR do and why?
Describe in detail what your merge request does and why.
This MR is part of a sequence of MRs that will ultimately result in being able to efficiently count the number of out-of-date CI runners in a given scope (instance/namespace/project). Please refer to the table below for more details:
Sequence | MR | notes |
---|---|---|
1 | semver version text column (major.minor.patch) on top of the existing ci_runners.version column. |
|
2 | semver column is kept up-to-date based on version value. |
|
3 |
Backfill runner semver column (!89054 - merged) | Adds batched background migration to backfill semver column from version values. |
Screenshots or screen recordings
These are strongly recommended to assist reviewers and reduce the time to merge your change.
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
-
Check that you have some
ci_runner
records with NULLsemver
values through thegdk psql -d gitlabhq_development_ci
command:SELECT id, version, semver FROM ci_runners WHERE semver::cidr IS NULL LIMIT 10
-
Run the migration in this branch:
bundle exec rails db:migrate
-
Check that the migration filled the
semver
column appropriately.
You can also check http://gdk.test:3000/admin/background_migrations?tab=finished&database=ci to see the state of the ScheduleBackfillCiRunnerSemver
batched background migration (should be in the Finished tab).
Reverting the migration should remove the migration from the dashboard.
Query plans
UPDATE query
Details and visualization: https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/10569/commands/37960
EXPLAIN UPDATE
ci_runners
SET semver = CONCAT(v.components[1]::text, '.', v.components[2]::text, '.', v.components[3]::text)
FROM (
SELECT id AS r_id, REGEXP_SPLIT_TO_ARRAY(substring(ci_runners.version
FROM 'v?(\d+\.\d+\.\d+)'), '\.')::smallint[] AS components
FROM "ci_runners"
WHERE "ci_runners"."id" BETWEEN 1 AND 2000
AND (semver::cidr IS NULL)
AND "ci_runners"."id" >= 1
AND "ci_runners"."id" < 200) v
WHERE id = v.r_id
AND v.components IS NOT NULL
Execution plan:
ModifyTable on public.ci_runners (cost=0.86..249.01 rows=54 width=319) (actual time=10.654..10.656 rows=0 loops=1)
Buffers: shared hit=61 read=55 dirtied=19
I/O Timings: read=8.070 write=0.000
-> Nested Loop (cost=0.86..249.01 rows=54 width=319) (actual time=0.943..1.218 rows=1 loops=1)
Buffers: shared hit=24 read=9 dirtied=1
I/O Timings: read=0.543 write=0.000
-> Index Scan using ci_runners_pkey on public.ci_runners ci_runners_1 (cost=0.43..52.18 rows=54 width=17) (actual time=0.847..1.120 rows=1 loops=1)
Index Cond: ((ci_runners_1.id >= 1) AND (ci_runners_1.id <= 2000) AND (ci_runners_1.id >= 1) AND (ci_runners_1.id < 200))
Filter: (((ci_runners_1.semver)::cidr IS NULL) AND ((regexp_split_to_array("substring"((ci_runners_1.version)::text, 'v?(\d+\.\d+\.\d+)'::text), '\.'::text))::smallint[] IS NOT NULL))
Rows Removed by Filter: 73
Buffers: shared hit=20 read=9 dirtied=1
I/O Timings: read=0.543 write=0.000
-> Index Scan using index_ci_runners_on_id_and_semver_cidr on public.ci_runners (cost=0.43..3.45 rows=1 width=277) (actual time=0.019..0.019 rows=1 loops=1)
Index Cond: (ci_runners.id = ci_runners_1.id)
Buffers: shared hit=4
I/O Timings: read=0.000 write=0.000
Background runtime estimates
Based on:
- an estimated 1.3 million
ci_runner
records having nosemver
values - minimum batch size of 500 records: 2600 batches with 2 minutes interval = 3.6 days
- maximum batch size of 10000 records: 130 batches with 2 minutes interval = 4h20m
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Production database points-of-interest
Non-conforming runner versions (will result in null version_* fields)
> SELECT DISTINCT(version) FROM ci_runners WHERE version IS NOT NULL AND semver::cidr IS NULL
+---------------------+
| version |
|---------------------|
| 1 |
| 1.1 |
| 10.0 |
| HEAD |
| HEAD-fd84d97 |
| dev |
| development version |
+---------------------+
Total runners not covered by migration
> SELECT COUNT(*) FROM ci_runners WHERE semver::cidr IS NULL
+-------+
| count |
|-------|
| 15250 |
+-------+
**> SELECT COUNT(*) FROM ci_runners WHERE version IS NULL
+-------+
| count |
|-------|
| 12738 |
+-------+
Part of #339523 (closed)