Skip to content

Dependency Proxy TTL workers

Steve Abrams requested to merge 294187-dp-ttl-workers into master

⛰ Context

The Dependency Proxy allows users to cache images pulled from DockerHub in their GitLab storage. Images are made up of blobs and manifests. There are many reasons users may want to clear their cache or delete specific images. Examples include:

  1. Users pull an image like alpine:latest, where the image changes over time, meaning we end up with a bunch of stale blobs and manifests taking up space unnecessarily.
  2. The image that was cached is simply no longer used anywhere.

To help relieve this problem, we are implementing Time-To-Live (TTL) policies where the user can set how long they want any given blob/manifest to be retained before being removed.

🚮 What does this MR do?

In previous MRs we created a model to store the TTL policies and added a "status" to the blob and manifest models. This MR implements the workers that are responsible for:

  1. Updating blobs/manifests to "expired" based on their associated policies
  2. Deleting the expired records/files

(1.) is handled by DependencyProxy::ImageTtlGroupPolicyWorker and (2.) is handled by DependencyProxy::CleanupBlobWorker and DependencyProxy::CleanupManifestWorker.

(1.) is a worker that runs once daily, iterating over all enabled TTL policies and then updating any qualified blobs/manifests to "expired". It kicks off the second set of workers that delete the expired records.

The (2.) workers use the LimitedCapacity::Worker to ensure the queue for deleting files does not get backed up by only allowing a certain number of jobs to run at any given time. This number of jobs is controlled by the new ApplicationSetting: dependency_proxy_ttl_group_policy_worker_capacity. We want to control this because some blobs may be very large and take a long time to be deleted.

We also update the dependency proxy services to ensure no expired blob or manifest is ever served when users pull an image.

💾 Database

🐘 Migration output

Up migration 
== 20210910014741 AddDependencyProxyTtlGroupPolicyWorkerCapacityToApplicationSettings: migrating
-- add_column(:application_settings, :dependency_proxy_ttl_group_policy_worker_capacity, :smallint, {:default=>2, :null=>false})
   -> 0.0055s
== 20210910014741 AddDependencyProxyTtlGroupPolicyWorkerCapacityToApplicationSettings: migrated (0.0056s)

== 20210910015047 AddAppSettingsDepProxyTtlWorkerCapacityCheckConstraint: migrating
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0003s
-- execute("ALTER TABLE application_settings\nADD CONSTRAINT app_settings_dep_proxy_ttl_policies_worker_capacity_positive\nCHECK ( dependency_proxy_ttl_group_policy_worker_capacity >= 0 )\nNOT VALID;\n")
   -> 0.0045s
-- current_schema()
   -> 0.0002s
-- execute("SET statement_timeout TO 0")
   -> 0.0011s
-- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT app_settings_dep_proxy_ttl_policies_worker_capacity_positive;")
   -> 0.0023s
-- execute("RESET statement_timeout")
   -> 0.0012s
== 20210910015047 AddAppSettingsDepProxyTtlWorkerCapacityCheckConstraint: migrated (0.0453s)

== 20210913224558 UpdateDependencyProxyManifestsUniquenessConstraint: migrating
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:dependency_proxy_manifests, [:group_id, :file_name, :status], {:unique=>true, :name=>"index_dep_prox_manifests_on_group_id_file_name_and_status", :algorithm=>:concurrently})
   -> 0.0044s
-- add_index(:dependency_proxy_manifests, [:group_id, :file_name, :status], {:unique=>true, :name=>"index_dep_prox_manifests_on_group_id_file_name_and_status", :algorithm=>:concurrently})
   -> 0.0043s
-- transaction_open?()
   -> 0.0000s
-- indexes(:dependency_proxy_manifests)
   -> 0.0018s
-- remove_index(:dependency_proxy_manifests, {:algorithm=>:concurrently, :name=>"index_dependency_proxy_manifests_on_group_id_and_file_name"})
   -> 0.0030s
== 20210913224558 UpdateDependencyProxyManifestsUniquenessConstraint: migrated (0.0182s)

== 20210914172202 AddStatusIndexToDependencyProxyTables: migrating ============
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:dependency_proxy_manifests, :status, {:name=>"index_dependency_proxy_manifests_on_status", :algorithm=>:concurrently})
   -> 0.0015s
-- add_index(:dependency_proxy_manifests, :status, {:name=>"index_dependency_proxy_manifests_on_status", :algorithm=>:concurrently})
   -> 0.0032s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:dependency_proxy_blobs, :status, {:name=>"index_dependency_proxy_blobs_on_status", :algorithm=>:concurrently})
   -> 0.0014s
-- add_index(:dependency_proxy_blobs, :status, {:name=>"index_dependency_proxy_blobs_on_status", :algorithm=>:concurrently})
   -> 0.0030s
== 20210914172202 AddStatusIndexToDependencyProxyTables: migrated (0.0148s) ===

== 20210928171122 AddGroupIdStatusIdIndexToDependencyProxyTables: migrating ===
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:dependency_proxy_manifests, [:group_id, :status, :id], {:name=>"index_dependency_proxy_manifests_on_group_id_status_and_id", :algorithm=>:concurrently})
   -> 0.0037s
-- execute("SET statement_timeout TO 0")
   -> 0.0007s
-- add_index(:dependency_proxy_manifests, [:group_id, :status, :id], {:name=>"index_dependency_proxy_manifests_on_group_id_status_and_id", :algorithm=>:concurrently})
   -> 0.0077s
-- execute("RESET statement_timeout")
   -> 0.0008s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:dependency_proxy_blobs, [:group_id, :status, :id], {:name=>"index_dependency_proxy_blobs_on_group_id_status_and_id", :algorithm=>:concurrently})
   -> 0.0017s
-- add_index(:dependency_proxy_blobs, [:group_id, :status, :id], {:name=>"index_dependency_proxy_blobs_on_group_id_status_and_id", :algorithm=>:concurrently})
   -> 0.0039s
== 20210928171122 AddGroupIdStatusIdIndexToDependencyProxyTables: migrated (0.0230s)
Down migration 
== 20210928171122 AddGroupIdStatusIdIndexToDependencyProxyTables: reverting ===
-- transaction_open?()
   -> 0.0000s
-- indexes(:dependency_proxy_manifests)
   -> 0.0038s
-- execute("SET statement_timeout TO 0")
   -> 0.0006s
-- remove_index(:dependency_proxy_manifests, {:algorithm=>:concurrently, :name=>"index_dependency_proxy_manifests_on_group_id_status_and_id"})
   -> 0.0039s
-- execute("RESET statement_timeout")
   -> 0.0006s
-- transaction_open?()
   -> 0.0000s
-- indexes(:dependency_proxy_blobs)
   -> 0.0022s
-- remove_index(:dependency_proxy_blobs, {:algorithm=>:concurrently, :name=>"index_dependency_proxy_blobs_on_group_id_status_and_id"})
   -> 0.0020s
== 20210928171122 AddGroupIdStatusIdIndexToDependencyProxyTables: reverted (0.0166s)

== 20210914172202 AddStatusIndexToDependencyProxyTables: reverting ============
-- transaction_open?()
   -> 0.0000s
-- indexes(:dependency_proxy_manifests)
   -> 0.0059s
-- execute("SET statement_timeout TO 0")
   -> 0.0007s
-- remove_index(:dependency_proxy_manifests, {:algorithm=>:concurrently, :name=>"index_dependency_proxy_manifests_on_status"})
   -> 0.0169s
-- execute("RESET statement_timeout")
   -> 0.0006s
-- transaction_open?()
   -> 0.0000s
-- indexes(:dependency_proxy_blobs)
   -> 0.0019s
-- remove_index(:dependency_proxy_blobs, {:algorithm=>:concurrently, :name=>"index_dependency_proxy_blobs_on_status"})
   -> 0.0030s
== 20210914172202 AddStatusIndexToDependencyProxyTables: reverted (0.0357s) ===

== 20210913224558 UpdateDependencyProxyManifestsUniquenessConstraint: reverting
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:dependency_proxy_manifests, [:group_id, :file_name], {:unique=>true, :name=>"index_dependency_proxy_manifests_on_group_id_and_file_name", :algorithm=>:concurrently})
   -> 0.0034s
-- execute("SET statement_timeout TO 0")
   -> 0.0006s
-- add_index(:dependency_proxy_manifests, [:group_id, :file_name], {:unique=>true, :name=>"index_dependency_proxy_manifests_on_group_id_and_file_name", :algorithm=>:concurrently})
   -> 0.0103s
-- execute("RESET statement_timeout")
   -> 0.0006s
-- transaction_open?()
   -> 0.0000s
-- indexes(:dependency_proxy_manifests)
   -> 0.0017s
-- remove_index(:dependency_proxy_manifests, {:algorithm=>:concurrently, :name=>"index_dep_prox_manifests_on_group_id_file_name_and_status"})
   -> 0.0042s
== 20210913224558 UpdateDependencyProxyManifestsUniquenessConstraint: reverted (0.0245s)

== 20210910015047 AddAppSettingsDepProxyTtlWorkerCapacityCheckConstraint: reverting
-- transaction_open?()
   -> 0.0000s
-- execute("ALTER TABLE application_settings\nDROP CONSTRAINT IF EXISTS app_settings_dep_proxy_ttl_policies_worker_capacity_positive\n")
   -> 0.0035s
== 20210910015047 AddAppSettingsDepProxyTtlWorkerCapacityCheckConstraint: reverted (0.0285s)

== 20210910014741 AddDependencyProxyTtlGroupPolicyWorkerCapacityToApplicationSettings: reverting
-- remove_column(:application_settings, :dependency_proxy_ttl_group_policy_worker_capacity, :smallint, {:default=>2, :null=>false})
   -> 0.0107s
== 20210910014741 AddDependencyProxyTtlGroupPolicyWorkerCapacityToApplicationSettings: reverted (0.0141s)

🔎 Query Analysis

There are a number of queries introduced in these workers. To reduce clutter in this description, I've commented them where each query occurs:

  1. !70029 (comment 679903596)
  2. !70029 (comment 679903603)
  3. !70029 (comment 679903612)
  4. !70029 (comment 679903615)
  5. !70029 (comment 679903618)

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

💻 How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

📏 MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #294187 (closed)

Edited by Steve Abrams

Merge request reports