Add collapsible sections to report
Description
This MR adds collapsible sections to the report, and groups MRs by milestones.
This should make the report more easy to read.
Future ideas for improvement:
- Fix the milestone sorting (e.g. 14.10 is currently listed before 14.9)
- Group Issues by milestones
- Or group both MRs and Issues together by milestone?
- Group by major milestone versions instead?
- Make the ruby code more ruby-like? I don't know ruby and this is the best I could do
Example output report
MRs (total: 330)
14.8 (3)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Update CI/CD data decay blueprint with new content and links | gitlab-org/gitlab!79110 (merged) | %14.8 | @grzesiek |
||
| Remove dangling running entries from ci running builds table | gitlab-org/gitlab!79444 (merged) | %14.8 | @mbobin |
||
| Backfill CI queuing tables | gitlab-org/gitlab!80172 (merged) | %14.8 | @mbobin |
14.9 (3)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Fix query analyzers base context / suppress keys | gitlab-org/gitlab!80824 (merged) | %14.9 | @grzesiek |
||
| Cleanup back-filling CI queuing tables migration | gitlab-org/gitlab!81602 (merged) | %14.9 | @mbobin |
||
| Update CI scaling blueprint status and link remaining work | gitlab-org/gitlab!81808 (merged) | %14.9 | @grzesiek |
14.10 (1)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Draft: CI/CD partitioning PoC | gitlab-org/gitlab!80186 (closed) | %14.10 | @grzesiek |
15.1 (3)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Add pipeline partitioning design details to docs | gitlab-org/gitlab!87683 (merged) | %15.1 | @grzesiek |
||
| Reference pipeline partitioning design in time decay blueprint | gitlab-org/gitlab!89030 (merged) | %15.1 | @grzesiek |
||
| Add timeline to CI/CD time decay blueprint | gitlab-org/gitlab!89034 (merged) | %15.1 | @grzesiek |
15.2 (5)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Add pipeline age Prometheus histogram metric | gitlab-org/gitlab!90027 (merged) | %15.2 | @grzesiek |
||
| Removes Ci::LegacyStage | gitlab-org/gitlab!91060 (merged) | %15.2 | @lauraX |
||
| Drop unused ci builds index | gitlab-org/gitlab!91470 (merged) | %15.2 | @morefice |
Reduced ci_builds size by 0.8GB
|
|
| Remove legacy queuing code | gitlab-org/gitlab!91595 (merged) | %15.2 | @mbobin |
Removed 600 legacy LoC | |
| Replace hardcoded table names with arel values | gitlab-org/gitlab!92535 (merged) | %15.2 | @mbobin |
15.3 (8)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Remove plaintext token index from ci_builds | gitlab-org/gitlab!91258 (merged) | %15.3 | @mbobin |
Reduced ci_builds size by 4.4GB
|
|
| Drop unused scope on ci builds | gitlab-org/gitlab!91473 (merged) | %15.3 | @morefice |
||
| Drop unused index on ci_builds | gitlab-org/gitlab!92747 (merged) | %15.3 | @morefice |
Reduced ci_builds size by 70GB
|
|
| Remove pending builds covering index | gitlab-org/gitlab!92877 (merged) | %15.3 | @mbobin |
||
| Remove token column from ci_builds table | gitlab-org/gitlab!92894 (merged) | %15.3 | @mbobin |
Reduced ci_builds size by 2.2GB
|
|
| Improve efficiency of specs for builds queuing | gitlab-org/gitlab!93270 (merged) | %15.3 | @grzesiek |
||
| Remove pipeline#find_stage_by_name! | gitlab-org/gitlab!95166 (merged) | %15.3 | @morefice |
||
| Add uniqueness index findings to CI data decay | gitlab-org/gitlab!95250 (merged) | %15.3 | @morefice |
15.4 (20)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Add foreign key findings to CI data decay | gitlab-org/gitlab!92781 (merged) | %15.4 | @mbobin |
||
| Remove reads to stage column from build | gitlab-org/gitlab!94941 (merged) | %15.4 | @morefice |
||
| Dedicated file for GenericCommitStatus factory | gitlab-org/gitlab!95696 (merged) | %15.4 | @morefice |
||
| Fetch Ci::Stage by id | gitlab-org/gitlab!95711 (closed) | %15.4 | @morefice |
||
| Document ALTER TABLE behavior for CI partitioning | gitlab-org/gitlab!95944 (merged) | %15.4 | @stomlinson |
||
| Add naming conventions for CI partitioning | gitlab-org/gitlab!96530 (merged) | %15.4 | @mbobin |
||
| Add blueprint docs about partitioning builds metadata first | gitlab-org/gitlab!96741 (merged) | %15.4 | @grzesiek |
||
| Zero partition helper | gitlab-org/gitlab!96815 (merged) | %15.4 | @stomlinson |
||
| Add partition_id column to selected CI database tables | gitlab-org/gitlab!96856 (merged) | %15.4 | @mbobin |
||
| Prepare to remove trace column from ci_builds | gitlab-org/gitlab!96865 (merged) | %15.4 | @mbobin |
||
| Document index maintenance findings | gitlab-org/gitlab!97007 (merged) | %15.4 | @morefice |
||
| Remove ci_read_stage_records feature flag | gitlab-org/gitlab!97030 (merged) | %15.4 | @morefice |
||
| Introduce Ci::PartitioningAnalyzer | gitlab-org/gitlab!97113 (merged) | %15.4 | @morefice |
||
| Cascade partitioning key for pipeline resources during creation/retry | gitlab-org/gitlab!97163 (merged) | %15.4 | @mbobin |
||
| Copy partition_id from jobs when creating new artifacts | gitlab-org/gitlab!97360 (merged) | %15.4 | @mbobin |
||
| Cascade partition id value to pipeline variables | gitlab-org/gitlab!97388 (merged) | %15.4 | @mbobin |
||
| Cascade partition value to generic commit statuses and their stages | gitlab-org/gitlab!97672 (merged) | %15.4 | @mbobin |
||
| Updates spec setup for BuildDependencies and more - 7/10 | gitlab-org/gitlab!97797 (merged) | %15.4 | @lauraX |
||
| Assign partition value to child pipelines from parent pipeline | gitlab-org/gitlab!97822 (merged) | %15.4 | @mbobin |
||
| Prepare Ci::BuildMetadata for partitioning 1/3 | gitlab-org/gitlab!97924 (merged) | %15.4 | @morefice |
15.5 (24)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Draft: PoC for removing partitioned data using foreign keys | gitlab-org/gitlab!92440 (closed) | %15.5 | @mbobin |
||
Draft: Resolve "Partitionci_builds_metadata table" |
gitlab-org/gitlab!97776 (closed) | %15.5 | @morefice |
||
| Prepare Ci::BuildMetadata for partitioning 2/3 | gitlab-org/gitlab!97926 (merged) | %15.5 | @morefice |
||
| Add indexes to ci_builds_metadata | gitlab-org/gitlab!98170 (merged) | %15.5 | @morefice |
||
| Document Zero partition helper | gitlab-org/gitlab!98217 (merged) | %15.5 | @morefice |
||
| Implement generic test case for builds partitioning | gitlab-org/gitlab!98309 (merged) | %15.5 | @mbobin |
||
| Fix typo in CI Data Decay blueprint | gitlab-org/gitlab!98414 (merged) | %15.5 | @morefice |
||
| Update CI scaling blueprints and add CI partitioning timeline | gitlab-org/gitlab!98416 (merged) | %15.5 | @grzesiek |
||
| Schedule index removal for ci_builds_metadata | gitlab-org/gitlab!98429 (merged) | %15.5 | @morefice |
||
| Ensure that stage name and record are in sync for page deployments | gitlab-org/gitlab!98506 (merged) | %15.5 | @mbobin |
||
| GraphQL fetch stage by id | gitlab-org/gitlab!98511 (merged) | %15.5 | @morefice |
||
| Add swap_primary_key database helper | gitlab-org/gitlab!98645 (merged) | %15.5 | @morefice |
||
| Remove trace column from ci_builds table | gitlab-org/gitlab!98646 (merged) | %15.5 | @mbobin |
||
| Document primary key strategy CI partitioning | gitlab-org/gitlab!98762 (merged) | %15.5 | @morefice |
||
| Allow add_concurrent_foreign_key to support multiple columns definition | gitlab-org/gitlab!98974 (merged) | %15.5 | @morefice |
||
| Update documentation for composite primary keys | gitlab-org/gitlab!98987 (merged) | %15.5 | @mbobin |
||
| Drop index on ci_builds_metadata | gitlab-org/gitlab!99483 (merged) | %15.5 | @morefice |
||
| Fix lock retries for the partition helper | gitlab-org/gitlab!100169 (merged) | %15.5 | @mbobin |
||
| Use table_name in SQL queries for Ci::BuildMetadata | gitlab-org/gitlab!100248 (merged) | %15.5 | @mbobin |
||
| Draft: Test scope block argument for partitionable | gitlab-org/gitlab!100696 (closed) | %15.5 | @mbobin |
||
| Split list partitioning migration into separate migrations | gitlab-org/gitlab!100810 (merged) | %15.5 | @mbobin |
||
| Ensure current user owns table to partition | gitlab-org/gitlab!100889 (merged) | %15.5 | @stomlinson |
||
| Update CI partitioning blueprints with the current state and insights | gitlab-org/gitlab!101043 (merged) | %15.5 | @grzesiek |
||
| Document a possible separator in partitioned resource URI | gitlab-org/gitlab!101374 (merged) | %15.5 | @grzesiek |
15.6 (22)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Partition ci_builds_metadata table | gitlab-org/gitlab!100115 (merged) | %15.6 | @mbobin |
||
Extend Ci::PartitioningAnalyzer to check partition_id presence |
gitlab-org/gitlab!100804 (merged) | %15.6 | @morefice |
||
| Switch from partition to routing table | gitlab-org/gitlab!100935 (merged) | %15.6 | @mbobin |
||
| Add TimeoutHelpers | gitlab-org/gitlab!101332 (merged) | %15.6 | @morefice |
||
| Add LockRetriesHelpers | gitlab-org/gitlab!101333 (merged) | %15.6 | @morefice |
||
| Extract ensure_batched_background_migration_is_finished to BatchedBackgroundMigrationHelpers | gitlab-org/gitlab!101370 (merged) | %15.6 | @morefice |
||
| Document setting the sequence name explicitly | gitlab-org/gitlab!101638 (merged) | %15.6 | @mbobin |
||
| Document add_concurrent_partitioned_index helper | gitlab-org/gitlab!101642 (merged) | %15.6 | @morefice |
||
| Add Migration::ConstraintsHelpers | gitlab-org/gitlab!101737 (merged) | %15.6 | @morefice |
||
| Remove ignore rules for ci_builds.trace | gitlab-org/gitlab!101744 (merged) | %15.6 | @mbobin |
||
| Explicitly lock tables for partitioning | gitlab-org/gitlab!101769 (merged) | %15.6 | @mbobin |
||
| Prevent usage of add_concurrent_index on partition table | gitlab-org/gitlab!101924 (merged) | %15.6 | @morefice |
||
| Prevent unnecessary jsonb updates for builds_metadata | gitlab-org/gitlab!102031 (merged) | %15.6 | @mbobin |
||
| Cleanup job token authentication for artifacts download | gitlab-org/gitlab!102161 (merged) | %15.6 | @mbobin |
||
| Fix percentage of time rollouts for routing tables switch | gitlab-org/gitlab!102621 (merged) | %15.6 | @mbobin |
||
| Create the first dynamic partition for builds metadata | gitlab-org/gitlab!102735 (merged) | %15.6 | @mbobin |
||
| Enable Ci::PartitioningAnalyzer | gitlab-org/gitlab!102762 (merged) | %15.6 | @morefice |
||
| Document priority locking strategy | gitlab-org/gitlab!102767 (merged) | %15.6 | @morefice |
||
| Refactor Ci::PartitioningAnalyzer | gitlab-org/gitlab!102861 (closed) | %15.6 | @morefice |
||
| Prevent index removal on partition table | gitlab-org/gitlab!103172 (merged) | %15.6 | @morefice |
||
| Document query analyzer strategy | gitlab-org/gitlab!103487 (merged) | %15.6 | @morefice |
||
| Add partition_id to CI models referenced by foreign keys | gitlab-org/gitlab!103594 (merged) | %15.6 | @morefice |
15.7 (27)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Remove add_column_with_default | gitlab-org/gitlab!103907 (merged) | %15.7 | @morefice |
||
| Draft: Add FK to p_ci_builds_metadata | gitlab-org/gitlab!104226 (closed) | %15.7 | @morefice |
||
Remove the p letter suffix from partition names |
gitlab-org/gitlab!104389 (merged) | %15.7 | @mbobin |
||
| Update steps required for partitioning new tables | gitlab-org/gitlab!104485 (merged) | %15.7 | @mbobin |
||
| Add partition_id to ci_resources | gitlab-org/gitlab!104567 (merged) | %15.7 | @morefice |
||
| Cascade partition_id to Ci::UnitTestFailure | gitlab-org/gitlab!104571 (merged) | %15.7 | @morefice |
||
| Use an exisiting builds metadata partition for tests | gitlab-org/gitlab!104671 (merged) | %15.7 | @mbobin |
||
| Cascade partition_id to Ci::Sources::Pipeline | gitlab-org/gitlab!104678 (merged) | %15.7 | @morefice |
||
| Draft: Cascade partition_id to Ci::Resource | gitlab-org/gitlab!104683 (closed) | %15.7 | @morefice |
||
| Cascade partition_id to Ci::BuildPendingState | gitlab-org/gitlab!104695 (merged) | %15.7 | @morefice |
||
| Cascade partition_id to pending / running build entries | gitlab-org/gitlab!104718 (merged) | %15.7 | @grzesiek |
||
| Cascade partition_id to Ci::BuildReportResult | gitlab-org/gitlab!104834 (merged) | %15.7 | @morefice |
||
| Cascade partition_id to Ci::BuildTraceChunk | gitlab-org/gitlab!104836 (merged) | %15.7 | @morefice |
||
| Use table_name for Ci::Build SQL queries | gitlab-org/gitlab!104942 (merged) | %15.7 | @morefice |
||
| Update Ci::BuildMetadata with association | gitlab-org/gitlab!104945 (merged) | %15.7 | @morefice |
||
| Cascade partition_id to Ci::BuildNeed | gitlab-org/gitlab!104953 (merged) | %15.7 | @morefice |
||
| Describe partitioning for multi-project pipelines | gitlab-org/gitlab!104969 (merged) | %15.7 | @mbobin |
||
| Use the routing table by default for Ci::BuildMetadata | gitlab-org/gitlab!105071 (merged) | %15.7 | @mbobin |
||
| Patch ActiveRecord CRUD actions for partitioning | gitlab-org/gitlab!105103 (merged) | %15.7 | @mbobin |
||
| Cascade partition_id to Ci::BuildTraceMetadata | gitlab-org/gitlab!105314 (merged) | %15.7 | @morefice |
||
| Cascade partition_id to Ci::BuildRunnerSession | gitlab-org/gitlab!105520 (merged) | %15.7 | @morefice |
||
| Cascade partition_id to Ci::JobVariable | gitlab-org/gitlab!105636 (merged) | %15.7 | @morefice |
||
| Prepare Ci::Build for partitioning | gitlab-org/gitlab!106128 (merged) | %15.7 | @morefice |
||
| Prefix partition_id to Ci::Build#token | gitlab-org/gitlab!106179 (merged) | %15.7 | @morefice |
||
| Cascade partition_id to Ci::Resource | gitlab-org/gitlab!106431 (merged) | %15.7 | @mbobin |
||
| Add unique index to ci_builds | gitlab-org/gitlab!106680 (merged) | %15.7 | @morefice |
||
| Refactor query counts for pipeline serialization tests | gitlab-org/gitlab!107190 (merged) | %15.7 | @mbobin |
15.8 (13)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Draft: Wip prep ci_builds partitioning | gitlab-org/gitlab!103516 (closed) | %15.8 | @morefice |
||
| Add FKs including partition_id referencing ci_builds | gitlab-org/gitlab!106717 (merged) | %15.8 | @morefice |
||
| Schedule async index on ci_builds | gitlab-org/gitlab!106952 (merged) | %15.8 | @morefice |
||
| Remove ci_build_partition_id_token_prefix feature flag | gitlab-org/gitlab!107103 (merged) | %15.8 | @morefice |
||
| Draft: Swap ci_builds primary key | gitlab-org/gitlab!107343 (closed) | %15.8 | @morefice |
||
| Disable ci_builds reindexing on Sundays | gitlab-org/gitlab!107448 (merged) | %15.8 | @mbobin |
||
| Bump default partition_id value for CI tables to 101 | gitlab-org/gitlab!107472 (merged) | %15.8 | @mbobin |
||
| Use the same exclusive lease key name for all indexing actions | gitlab-org/gitlab!108099 (merged) | %15.8 | @mbobin |
||
| Check time before starting reindex action | gitlab-org/gitlab!108428 (merged) | %15.8 | @mbobin |
||
| Accept on_update option for add_concurrent_foreign_key helper | gitlab-org/gitlab!108535 (merged) | %15.8 | @mbobin |
||
| Update High Traffic table CI list | gitlab-org/gitlab!108688 (merged) | %15.8 | @morefice |
||
| Specify CI data partitioning DRIs in the blueprint | gitlab-org/gitlab!108795 (merged) | %15.8 | @grzesiek |
||
| Update partitioning docs to include on_update: :cascade for FKs | gitlab-org/gitlab!108972 (merged) | %15.8 | @mbobin |
15.9 (54)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Switch current partition_id value to 101 | gitlab-org/gitlab!107225 (closed) | %15.9 | @mbobin |
||
| Add FK including partition_id to ci_build_report_results | gitlab-org/gitlab!107476 (merged) | %15.9 | @morefice |
||
| Add FK including partition_id to ci_build_needs | gitlab-org/gitlab!107547 (merged) | %15.9 | @morefice |
||
| Add FK including partition_id to ci_builds_runner_session | gitlab-org/gitlab!107549 (closed) | %15.9 | @morefice |
||
| Remove job token on completion | gitlab-org/gitlab!108021 (merged) | %15.9 | @mbobin |
||
| Add gem skeleton for ActiveRecord partitioning patches | gitlab-org/gitlab!108223 (merged) | %15.9 | @mbobin |
||
| Redefine FKs with update statement | gitlab-org/gitlab!108571 (merged) | %15.9 | @morefice |
||
| Ensure build metadata record exists with before_validation | gitlab-org/gitlab!108780 (merged) | %15.9 | @mbobin |
||
| Add tmp indexes for CI partitioning - 1 | gitlab-org/gitlab!108788 (merged) | %15.9 | @morefice |
||
| Add tmp indexes for CI partitioning - 2 | gitlab-org/gitlab!108823 (merged) | %15.9 | @morefice |
||
| Drop unused index in ci_builds | gitlab-org/gitlab!108964 (merged) | %15.9 | @morefice |
||
| Add tmp indexes for CI partitioning - 3 | gitlab-org/gitlab!109084 (merged) | %15.9 | @morefice |
||
| Add partitioning temp indexes to CI tables | gitlab-org/gitlab!109085 (merged) | %15.9 | @mbobin |
||
| Add partitioning tmp indexes to ci_sources_pipelines table | gitlab-org/gitlab!109094 (merged) | %15.9 | @mbobin |
||
| Add unique index to ci_builds | gitlab-org/gitlab!109116 (merged) | %15.9 | @morefice |
||
| Add generator for partitioning FKs | gitlab-org/gitlab!109130 (merged) | %15.9 | @mbobin |
||
| Update table partitioning doc | gitlab-org/gitlab!109220 (merged) | %15.9 | @morefice |
||
| Add FK to ci_running_builds | gitlab-org/gitlab!109308 (merged) | %15.9 | @morefice |
||
| Add FK to ci_pending_builds | gitlab-org/gitlab!109311 (merged) | %15.9 | @morefice |
||
| Update ci_builds_runner_session FK to include partition_id | gitlab-org/gitlab!109316 (merged) | %15.9 | @mbobin |
||
| Update ci_resources FK to include partition_id | gitlab-org/gitlab!109337 (merged) | %15.9 | @mbobin |
||
| Fix partition ids for ci_sources_pipelines records | gitlab-org/gitlab!109364 (merged) | %15.9 | @mbobin |
||
| Drop old FKs on ci_builds | gitlab-org/gitlab!109783 (merged) | %15.9 | @morefice |
||
| Rebalance partition_id for ci_pipelines | gitlab-org/gitlab!109981 (merged) | %15.9 | @morefice |
||
| Fix unique index on ci_build_pending_states | gitlab-org/gitlab!110172 (merged) | %15.9 | @morefice |
||
| Fix partition_id for ci_job_variables | gitlab-org/gitlab!110177 (closed) | %15.9 | @morefice |
||
| Add partitioned index and FK for ci_sources_pipelines and ci_builds | gitlab-org/gitlab!110367 (merged) | %15.9 | @mbobin |
||
| Add partitioned index and FK for ci_build_trace_metadata and ci_builds | gitlab-org/gitlab!110392 (merged) | %15.9 | @mbobin |
||
| Add partitioned index and FK for ci_job_artifacts and ci_builds | gitlab-org/gitlab!110395 (merged) | %15.9 | @mbobin |
||
| Add partitioned index and FK for ci_job_variables and ci_builds | gitlab-org/gitlab!110397 (merged) | %15.9 | @mbobin |
||
| Schedule partial index for ci_build_report_results | gitlab-org/gitlab!110489 (merged) | %15.9 | @morefice |
||
| Drop invalid index from postgres_async_indexes | gitlab-org/gitlab!110494 (merged) | %15.9 | @morefice |
||
| Refactor async index creation tasks | gitlab-org/gitlab!110530 (merged) | %15.9 | @mbobin |
||
| Add validate_partitioned_foreign_key helper | gitlab-org/gitlab!110752 (closed) | %15.9 | @morefice |
||
| Schedule index for ci_build_needs | gitlab-org/gitlab!110780 (merged) | %15.9 | @morefice |
||
| Fix partition ids for ci_job_variables records | gitlab-org/gitlab!111173 (merged) | %15.9 | @mbobin |
||
| CI Partitioning - Document long running migration | gitlab-org/gitlab!111178 (merged) | %15.9 | @morefice |
||
| Rebalance partition_id for ci_builds | gitlab-org/gitlab!111287 (merged) | %15.9 | @morefice |
||
| Add async foreign key validation model | gitlab-org/gitlab!111293 (merged) | %15.9 | @mbobin |
||
| Extract indexing exclusive locking to be used by FKs too | gitlab-org/gitlab!111300 (merged) | %15.9 | @mbobin |
||
| Add migration helpers for async FK validation | gitlab-org/gitlab!111314 (merged) | %15.9 | @mbobin |
||
| Validate foreign keys asynchronously | gitlab-org/gitlab!111330 (merged) | %15.9 | @mbobin |
||
| Schedule ci_builds FK indexes for async creation | gitlab-org/gitlab!111333 (merged) | %15.9 | @mbobin |
||
| Fix partition_id for ci_pipeline_variables | gitlab-org/gitlab!111420 (merged) | %15.9 | @morefice |
||
| Fix partition_id for ci_job_artifacts | gitlab-org/gitlab!111436 (merged) | %15.9 | @morefice |
||
| Fix partition_id for ci_stage | gitlab-org/gitlab!111439 (merged) | %15.9 | @morefice |
||
| Fix partition_id for ci_build_needs | gitlab-org/gitlab!111440 (closed) | %15.9 | @morefice |
||
| Update iterations Gantt chart | gitlab-org/gitlab!111547 (merged) | %15.9 | @mbobin |
||
| Fix partition_id for ci_build_report_results | gitlab-org/gitlab!111548 (merged) | %15.9 | @morefice |
||
| Fix partition_id for ci_build_trace_metadata | gitlab-org/gitlab!111549 (merged) | %15.9 | @morefice |
||
| Fix partition_id for p_ci_builds_metadata | gitlab-org/gitlab!111576 (merged) | %15.9 | @morefice |
||
| Execute FK async validations as part of DB reindexing | gitlab-org/gitlab!112015 (merged) | %15.9 | @mbobin |
||
| Add documentation for async FK validation | gitlab-org/gitlab!112034 (merged) | %15.9 | @mbobin |
||
| Test FK helper updates on build metadata | gitlab-org/gitlab!112205 (closed) | %15.9 | @mbobin |
15.10 (48)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Use format_with_prefix, ensure_prefix options of token auth strategy | gitlab-org/gitlab!109063 (merged) | %15.10 | @tianwenchen |
||
| Validate and replace FK for ci_sources_pipelines and ci_builds | gitlab-org/gitlab!110368 (merged) | %15.10 | @mbobin |
||
| Refactor add_concurrent_partitioned_foreign_key migration helper | gitlab-org/gitlab!110388 (merged) | %15.10 | @mbobin |
||
| Validate and replace FK for ci_job_variables and ci_builds | gitlab-org/gitlab!111158 (merged) | %15.10 | @mbobin |
||
| Validate and replace FK for ci_build_trace_metadata and ci_builds | gitlab-org/gitlab!111163 (merged) | %15.10 | @mbobin |
||
| Validate and replace FK for ci_job_artifacts and ci_builds | gitlab-org/gitlab!111166 (merged) | %15.10 | @mbobin |
||
| Remove FF for token removal | gitlab-org/gitlab!111695 (merged) | %15.10 | @mbobin |
It reduces the size of ci_builds by 175MB for each 1M processed builds, or by about 175GB per year. |
|
| Prepare async foreign key validation for ci_build_needs | gitlab-org/gitlab!112130 (merged) | %15.10 | @tianwenchen |
||
| Prepare async foreign key validation for ci_build_report_results | gitlab-org/gitlab!112131 (merged) | %15.10 | @tianwenchen |
||
| Prepare async foreign key validation for ci_job_artifacts | gitlab-org/gitlab!112132 (merged) | %15.10 | @tianwenchen |
||
| Prepare async foreign key validation for ci_build_trace_metadata | gitlab-org/gitlab!112133 (merged) | %15.10 | @tianwenchen |
||
| Drop CI test partitions before attempting to create more | gitlab-org/gitlab!112293 (merged) | %15.10 | @mbobin |
||
| Update FKs between p_ci_builds_metadata partitions and ci_builds | gitlab-org/gitlab!112316 (merged) | %15.10 | @mbobin |
||
| Scope FK name uniqueness to the table name | gitlab-org/gitlab!112448 (merged) | %15.10 | @mbobin |
||
| Add rake tasks for async FK validation | gitlab-org/gitlab!112455 (merged) | %15.10 | @mbobin |
||
| Validate and replace FK for ci_build_report_results and ci_builds | gitlab-org/gitlab!112461 (merged) | %15.10 | @mbobin |
||
| Validate and replace FK for ci_build_needs and ci_builds | gitlab-org/gitlab!112462 (merged) | %15.10 | @mbobin |
||
| Find FK by name or identifier | gitlab-org/gitlab!112463 (merged) | %15.10 | @morefice |
||
| Prepare async foreign key validation for ci_sources_pipelines | gitlab-org/gitlab!112508 (merged) | %15.10 | @tianwenchen |
||
| Prepare async foreign key validation for ci_job_variables | gitlab-org/gitlab!112511 (merged) | %15.10 | @tianwenchen |
||
| Schedule ci_job_artifacts index for removal | gitlab-org/gitlab!112530 (closed) | %15.10 | @morefice |
Reduced ci_job_artifacts size by 79 GB
|
|
| Remove Rails 6 patches for partitioning | gitlab-org/gitlab!112573 (merged) | %15.10 | @mbobin |
||
| Prepare async index removal of token for ci_builds | gitlab-org/gitlab!112642 (merged) | %15.10 | @tianwenchen |
||
| Concurrently remove index for index_ci_builds_on_token_encrypted | gitlab-org/gitlab!112644 (merged) | %15.10 | @tianwenchen |
||
| Drop FK for ci_pending_builds | gitlab-org/gitlab!112668 (merged) | %15.10 | @morefice |
||
| Drop FK for ci_running_builds | gitlab-org/gitlab!112669 (merged) | %15.10 | @morefice |
||
| Schedule FK async validations for p_ci_builds_metadata partitions | gitlab-org/gitlab!112788 (merged) | %15.10 | @mbobin |
||
| Prepare async FKs for check constraints | gitlab-org/gitlab!112838 (merged) | %15.10 | @mbobin |
||
| Add types to async constraint validations | gitlab-org/gitlab!113101 (merged) | %15.10 | @mbobin |
||
| Add migration helpers for async check constraints | gitlab-org/gitlab!113206 (merged) | %15.10 | @mbobin |
||
| Add executors for async check constraints validations | gitlab-org/gitlab!113239 (merged) | %15.10 | @mbobin |
||
| Refactor check constraints existence | gitlab-org/gitlab!113242 (merged) | %15.10 | @mbobin |
||
| Add docs for creating new partitioned tables | gitlab-org/gitlab!113247 (merged) | %15.10 | @mbobin |
||
| Refactor CI partitioning strategy | gitlab-org/gitlab!113357 (merged) | %15.10 | @mbobin |
||
| Swap async constraints validators | gitlab-org/gitlab!113503 (merged) | %15.10 | @mbobin |
||
| Schedule temporary partitioning indexes removal | gitlab-org/gitlab!113507 (merged) | %15.10 | @mbobin |
||
| Remove temporary partitioning indexes | gitlab-org/gitlab!113513 (closed) | %15.10 | @mbobin |
||
| Validate and replace FK for p_ci_builds_metadata and ci_builds | gitlab-org/gitlab!113655 (merged) | %15.10 | @mbobin |
||
| Define ci_builds primary key as partitioned | gitlab-org/gitlab!113725 (merged) | %15.10 | @mbobin |
||
| Clean up p_ci_builds_metadata schema | gitlab-org/gitlab!113955 (merged) | %15.10 | @morefice |
||
| Reduce the chances of deadlocks for builds migrations | gitlab-org/gitlab!114220 (merged) | %15.10 | @mbobin |
||
| Fix rollback on async foreign key constraints | gitlab-org/gitlab!114224 (merged) | %15.10 | @mbobin |
||
| Use access exclusive lock for FKs on partitioned tables | gitlab-org/gitlab!114455 (merged) | %15.10 | @mbobin |
||
| Rename ensure_prefix to require_prefix_for_validation | gitlab-org/gitlab!114537 (merged) | %15.10 | @tianwenchen |
||
| Test PK swap without dropping the index | gitlab-org/gitlab!114590 (closed) | %15.10 | @mbobin |
||
| Add partitioned CI/CD tables to navigation | gitlab-org/gitlab-docs!3666 (merged) | %15.10 | @mbobin |
||
| Prepare Ci::Build model for primary key swap | gitlab-org/gitlab!114709 (merged) | %15.10 | @mbobin |
||
| Use id as first column for partitioned tables | gitlab-org/gitlab!114870 (merged) | %15.10 | @mbobin |
15.11 (7)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Prepare ci_builds for list partitioning | gitlab-org/gitlab!114077 (merged) | %15.11 | @mbobin |
||
| Support async constraint validation for first list partition | gitlab-org/gitlab!114406 (merged) | %15.11 | @tianwenchen |
||
| Remove partition_id default from ci_builds | gitlab-org/gitlab!115348 (merged) | %15.11 | @mbobin |
||
| Support LFK triggers on partitioning helpers | gitlab-org/gitlab!115515 (merged) | %15.11 | @mbobin |
||
| Validate ci_builds partitioning constraint | gitlab-org/gitlab!115840 (merged) | %15.11 | @mbobin |
||
| Remove partition_id default from ci_builds | gitlab-org/gitlab!115874 (merged) | %15.11 | @mbobin |
||
| Remove partition_id default for p_ci_builds_metadata | gitlab-org/gitlab!116761 (merged) | %15.11 | @tianwenchen |
16.0 (14)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Create routing table for ci_builds | gitlab-org/gitlab!115485 (merged) | %16.0 | @mbobin |
||
| Implement runner backoff for migrations | gitlab-org/gitlab!115679 (merged) | %16.0 | @mbobin |
||
| Change schema_spec to support composite FKs | gitlab-org/gitlab!117499 (merged) | %16.0 | @tianwenchen |
||
| Add document for async partitioning constraint preparation | gitlab-org/gitlab!117857 (merged) | %16.0 | @tianwenchen |
||
| Initialize the migration to convert pk to bigint for ci_pipelines | gitlab-org/gitlab!118136 (merged) | %16.0 | @tianwenchen |
||
| Replace hardcoded table names for ci_builds | gitlab-org/gitlab!118225 (merged) | %16.0 | @mbobin |
||
| Initialize the migration to convert pk for ci_pipeline_variables | gitlab-org/gitlab!118878 (merged) | %16.0 | @tianwenchen |
||
| Use database helpers to create partitioned tables | gitlab-org/gitlab!119338 (merged) | %16.0 | @mbobin |
||
| Suggest partitioning new CI tables | gitlab-org/gitlab!119519 (merged) | %16.0 | @mbobin |
||
| Lock tables before dropping the LFK trigger | gitlab-org/gitlab!120210 (merged) | %16.0 | @mbobin |
||
| Change locking order for retrying jobs | gitlab-org/gitlab!120216 (merged) | %16.0 | @mbobin |
||
| Fix partitioning flaky tests | gitlab-org/gitlab!120232 (merged) | %16.0 | @mbobin |
||
| Do not generate DB docs file for p_ci_builds | gitlab-org/gitlab!120419 (merged) | %16.0 | @mbobin |
||
| Prepare the async index for ci_pipeline_variables | gitlab-org/gitlab!120946 (merged) | %16.0 | @tianwenchen |
16.1 (21)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Swap primary key with bigint column for ci_pipeline_variables | gitlab-org/gitlab!120823 (merged) | %16.1 | @tianwenchen |
||
| Create routing table for ci_builds | gitlab-org/gitlab!120873 (merged) | %16.1 | @mbobin |
||
| Add concurrent index for ci_pipeline_variables | gitlab-org/gitlab!120950 (merged) | %16.1 | @tianwenchen |
||
| Fix background migration progress report on multi-db | gitlab-org/gitlab!121251 (merged) | %16.1 | @mbobin |
||
| Log database name on async index operations | gitlab-org/gitlab!121405 (merged) | %16.1 | @mbobin |
||
| Force partition attachment to happen during the weekend | gitlab-org/gitlab!121554 (merged) | %16.1 | @mbobin |
||
| Replace foreign keys to use the partitioned table | gitlab-org/gitlab!121718 (merged) | %16.1 | @mbobin |
||
| Validate foreign keys targeting p_ci_builds | gitlab-org/gitlab!122087 (merged) | %16.1 | @mbobin |
||
| Replace partitioned foreign keys to use p_ci_builds | gitlab-org/gitlab!122097 (closed) | %16.1 | @mbobin |
||
| Fix doc for swap_primary_key | gitlab-org/gitlab!122303 (merged) | %16.1 | @morefice |
||
| Drop foreign keys before attaching ci_builds | gitlab-org/gitlab!122566 (closed) | %16.1 | @mbobin |
||
| Remove ci_builds partitioning helper code | gitlab-org/gitlab!122584 (merged) | %16.1 | @mbobin |
||
| Remove explicit locking for retrying jobs | gitlab-org/gitlab!122737 (merged) | %16.1 | @igor.drozdov |
||
| Update partitioning docs for new tables to use p_ci_builds | gitlab-org/gitlab!122858 (merged) | %16.1 | @mbobin |
||
| Replace hardcoded table names for ci_builds | gitlab-org/gitlab!123118 (merged) | %16.1 | @mbobin |
||
| Execute partitioning tests | gitlab-org/gitlab!123124 (closed) | %16.1 | @mbobin |
||
| Make migrations skipped in production a no-op | gitlab-org/gitlab!123212 (merged) | %16.1 | @stomlinson |
||
Point introduced_by_url to correct MR for pi_ci_builds
|
gitlab-org/gitlab!123324 (merged) | %16.1 | @mallocke |
||
| Make migrations skipped in production a no-op | gitlab-org/gitlab!123346 (closed) | %16.1 | @mbobin |
||
| Add vacuum type to vaccum activity view | gitlab-org/gitlab!123353 (merged) | %16.1 | @mbobin |
||
| Add API to mark migrations as successful | gitlab-org/gitlab!123408 (merged) | %16.1 | @mbobin |
16.2 (22)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Patch Rails 7 for CI partitioning | gitlab-org/gitlab!111673 (merged) | %16.2 | @mbobin |
||
| Rename foreign keys that use p_ci_builds | gitlab-org/gitlab!122102 (closed) | %16.2 | @mbobin |
||
| Initialize the conversion for ci_pipeline_variables.pipeline_id | gitlab-org/gitlab!123132 (merged) | %16.2 | @tianwenchen |
||
| Finalize conversion for FKs targeting p_ci_builds | gitlab-org/gitlab!123799 (merged) | %16.2 | @mbobin |
||
| Replace foreign keys to use the partitioned table - V3 | gitlab-org/gitlab!123805 (merged) | %16.2 | @mbobin |
||
| Replace foreign keys to use the partitioned table - 4 | gitlab-org/gitlab!123811 (merged) | %16.2 | @mbobin |
||
| Clean up bigint conversion for ci_pipeline_variables.id | gitlab-org/gitlab!124026 (merged) | %16.2 | @tianwenchen |
||
| Finalize conversion for FKs targeting p_ci_builds - self-managed | gitlab-org/gitlab!124154 (merged) | %16.2 | @mbobin |
||
| Finalize conversion for FKs targeting p_ci_builds - 2 | gitlab-org/gitlab!124186 (merged) | %16.2 | @mbobin |
||
| Finalize conversion for FKs targeting p_ci_builds - 2 self-managed | gitlab-org/gitlab!124303 (merged) | %16.2 | @mbobin |
||
| Replace foreign keys to use the partitioned table - 4 self-managed | gitlab-org/gitlab!124306 (merged) | %16.2 | @mbobin |
||
| Finalize conversion for FKs targeting p_ci_builds - 3 | gitlab-org/gitlab!124310 (merged) | %16.2 | @mbobin |
||
| Replace foreign keys to use the partitioned table - 5 | gitlab-org/gitlab!124726 (merged) | %16.2 | @mbobin |
||
| Create FK between p_ci_builds_metadata and p_ci_builds | gitlab-org/gitlab!124865 (merged) | %16.2 | @mbobin |
||
| Revert "Merge branch '414396-partitioned-fks-v7' into 'master'" | gitlab-org/gitlab!125254 (merged) | %16.2 | @mbobin |
||
| Document skipping migrations API | gitlab-org/gitlab!125382 (merged) | %16.2 | @mbobin |
||
| Remove gitlab_active_record vendor gem | gitlab-org/gitlab!125561 (merged) | %16.2 | @morefice |
||
| Initialize the conversion for ci_pipeline_chat_data.pipeline_id | gitlab-org/gitlab!125965 (merged) | %16.2 | @tianwenchen |
||
| Remove SafelyChangeColumnDefault to CI partitioned models | gitlab-org/gitlab!125993 (merged) | %16.2 | @morefice |
||
| Drop devault value CI partition tables | gitlab-org/gitlab!126001 (merged) | %16.2 | @morefice |
||
| Finalize conversion for FKs targeting p_ci_builds - 4 | gitlab-org/gitlab!126122 (merged) | %16.2 | @mbobin |
||
| Finish CI FKs conversion and create new partitions | gitlab-org/gitlab!126701 (merged) | %16.2 | @mbobin |
16.3 (17)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Draft: Swap primary key with bigint column for ci_pipelines | gitlab-org/gitlab!121123 (merged) | %16.3 | @tianwenchen |
||
| Switch table_name in CommitStatus to routing table | gitlab-org/gitlab!122919 (merged) | %16.3 | @mbobin |
||
| Prepare the async index for ci_pipelines | gitlab-org/gitlab!124152 (merged) | %16.3 | @tianwenchen |
||
| Finalize conversion for FKs targeting p_ci_builds - 3 self-managed | gitlab-org/gitlab!124311 (merged) | %16.3 | @mbobin |
||
| Ensure id uniquness across p_ci_builds partitions | gitlab-org/gitlab!125002 (merged) | %16.3 | @mbobin |
||
| Remove ignore_column for ci_pipeline_variables.id_convert_to_bigint | gitlab-org/gitlab!125187 (merged) | %16.3 | @tianwenchen |
||
| Prepare async index for pipeline_id_convert_to_bigint | gitlab-org/gitlab!125188 (merged) | %16.3 | @tianwenchen |
||
| Initialize the conversion for ci_pipeline_messages.pipeline_id | gitlab-org/gitlab!126188 (merged) | %16.3 | @tianwenchen |
||
| Initialize the conversion for ci_pipelines.auto_canceled_by_id | gitlab-org/gitlab!126645 (merged) | %16.3 | @tianwenchen |
||
| Draft: Initialize the conversion for ci_sources_pipelines.pipeline_id | gitlab-org/gitlab!126648 (closed) | %16.3 | @tianwenchen |
||
| Initialize the conversion for ci_stages.pipeline_id | gitlab-org/gitlab!126650 (merged) | %16.3 | @tianwenchen |
||
| Draft: Initialize the conversion for ci_sources_pipelines.source_pipeline_id | gitlab-org/gitlab!126671 (merged) | %16.3 | @tianwenchen |
||
| Remove default partition value from CI tables | gitlab-org/gitlab!126726 (merged) | %16.3 | @mbobin |
||
| Update LFK references for p_ci_builds | gitlab-org/gitlab!127091 (merged) | %16.3 | @mbobin |
||
| Revert changes from 414396-complete-ci-builds-partitioning | gitlab-org/gitlab!127324 (merged) | %16.3 | @mbobin |
||
| Draft: Add foreign key constraint to ci_pipeline_variables | gitlab-org/gitlab!127445 (merged) | %16.3 | @tianwenchen |
||
| Draft: Create sync index for pipeline_id_convert_to_bigint | gitlab-org/gitlab!127449 (merged) | %16.3 | @tianwenchen |
No milestone (11)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Update CI/CD Scaling blueprint with the current state | gitlab-org/gitlab!76111 (merged) | @grzesiek |
|||
| Fix CI/CD time decay blueprint disclaimer | gitlab-org/gitlab!77868 (merged) | @grzesiek |
|||
| Draft: PoC pipelines associations with cascading partition id | gitlab-org/gitlab!84071 (closed) | @grzesiek |
|||
| Add a few examples of iterative CI/CD data partitioning | gitlab-org/gitlab!89521 (merged) | @grzesiek |
|||
| Fix dates on x-axis in pipeline partitioning timeline chart | gitlab-org/gitlab!98558 (merged) | @grzesiek |
|||
| Draft: Handle tables referenced by foreign key in ConvertTableToFirstListPartitionHelper | gitlab-org/gitlab!104792 | @stomlinson |
|||
| Backport Rails 7 preloader and associations for partitioning | gitlab-org/gitlab!105237 (closed) | @mbobin |
|||
| Draft: Fix triggers during list partitioning | gitlab-org/gitlab!109419 | @stomlinson |
|||
| Draft: Start conversion of FK columns to ci_pipelines as bigint | gitlab-org/gitlab!123128 (closed) | @mbobin |
|||
| Backfill builds uniqueness | gitlab-org/gitlab!124145 (closed) | @mbobin |
|||
| Draft: Prepare async index for pipeline_id_convert_to_bigint | gitlab-org/gitlab!127471 (merged) | @tianwenchen |
Backlog (7)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| Copy Rails 7 Preloader files into the partitioning gem | gitlab-org/gitlab!108224 (closed) | %Backlog | @mbobin |
||
| Draft: Add ActiveRecord patches to the partitioning preloading gem | gitlab-org/gitlab!108225 (closed) | %Backlog | @mbobin |
||
| Add configuration options to the partitioning gem | gitlab-org/gitlab!108229 (closed) | %Backlog | @mbobin |
||
| Require partitioning preloader gem | gitlab-org/gitlab!108239 (closed) | %Backlog | @mbobin |
||
| Draft: Rebalance CI partitions | gitlab-org/gitlab!108544 (closed) | %Backlog | @mbobin |
||
| Add validation to Ci::CommitStatus#ci_stage | gitlab-org/gitlab!110346 (closed) | %Backlog | @morefice |
||
| Draft: Fix build with nil stage_id | gitlab-org/gitlab!111912 (closed) | %Backlog | @morefice |
Issues (total: 42)
| Iteration | References | Status / ETA | DRI | Impact | |
|---|---|---|---|---|---|
| index_ci_job_artifacts_on_expire_at_and_job_id is bloated and unused | gitlab-org/gitlab#358042 | %Backlog | |||
| Benchmark partitioned builds schema | gitlab-org/gitlab#361017 (closed) | %Backlog | |||
| Get pipelines / builds queries from the test suite | gitlab-org/gitlab#361018 (closed) | %Backlog | |||
| Benchmark DB workload with CI/CD partitioning | gitlab-org/gitlab#362910 (closed) | %Backlog | |||
Backend: Remove ci_builds.stage column |
gitlab-org/gitlab#364377 | %Backlog | |||
Creating partitioned routing table for ci_pipelines data |
gitlab-org/gitlab#364397 (closed) | %Backlog | |||
| Add CommitStatus#ci_stage validation | gitlab-org/gitlab#371196 (closed) | %Backlog | |||
| Investigate autovacuum performance impacts of CI partitioning | gitlab-org/gitlab#371634 | %Backlog | |||
Add FKs for ci_partitions and routing tables |
gitlab-org/gitlab#372996 | %Backlog | |||
[Feature flag] Rollout of ci_partitioning_analyze_queries_partition_id_check
|
gitlab-org/gitlab#378282 (closed) | %Backlog | |||
| Investigate ci_builds <-> taggings relation in connection to partitioning | gitlab-org/gitlab#382210 (closed) | %Backlog | |||
| Pass unifed partitioned CI resource identifier to Sidekiq workers | gitlab-org/gitlab#383860 | %Backlog | |||
| CI partition rebalancing | gitlab-org/gitlab#387779 | %Backlog | |||
| Switch current partition_id value to 101 | gitlab-org/gitlab#387810 (closed) | %Backlog | |||
| Investigate if single column FK indexes are still needed after partitioning ci_builds | gitlab-org/gitlab#388798 | %Backlog | |||
| Partition manager for CI tables | gitlab-org/gitlab#389234 (closed) | %Backlog | |||
| Rename build#commit_id to build#pipeline_id | gitlab-org/gitlab#389921 (closed) | %Backlog | |||
| Reduced timeout to 9 hours | gitlab-org/gitlab#390538 | %Backlog | |||
| Update async FK validation documentation | gitlab-org/gitlab#391762 | %Backlog | |||
| Add partition_id column and value to non-CI tables | gitlab-org/gitlab#394832 (closed) | %Backlog | |||
| Enable QueryAnalyzers::Ci::PartitioningRoutingAnalyzer for ci_builds | gitlab-org/gitlab#398134 (closed) | %Backlog | |||
| Partition ci_job_artifacts table | gitlab-org/gitlab#403628 (closed) | %Backlog | |||
Update CommitStatus model to use p_ci_builds table |
gitlab-org/gitlab#415321 (closed) | ||||
| Update ci_builds to use routing table | gitlab-org/gitlab#416455 (closed) | ||||
| Create new partition for p_ci_builds | gitlab-org/gitlab#416457 (closed) | %Backlog |
Generated at
2023-07-25 14:30:52 UTC by .
Edited by François Rosé (OOO, back 2024-10-29)