Maven virtual registry: adjust scopes when accessing cached responses

🐗 Context

At the heart of the Maven Virtual registry, we have the model CachedResponse which represents a cached file (that is located in object storage).

Having a reference to an object storage file, we can't use the usual approaches to destroy this model. Rails callbacks should be executed to properly remove the file from the object storage.

Thus, in #479957 (closed), we introduced a system where we can "mark" a cached response for destruction and a background job will take care to clean them up. For its own logic, this worker introduced different status: default, processing and error.

Up to now, we had only the default status. Thus, all the read queries to cached responses can simply ignore the status. Now, that we have different states, we need to adjust these queries to make sure that the default status is used.

Lastly, we have a unique index in the database that will need to take the status into account. Without that change, we could run into situations where the background worker is processing a cached response and a concurrent process will try to introduce a new record which will make the unique index fail 💥

This MR adjusts all these aspects to take the status into account. The related issue is Maven virtual registry: adjust cached response ... (#494247 - closed).

🤔 What does this MR do and why?

• When reading a cached response, make sure that the default scope is used.
• Update the related specs.
• Update the unique index on cached responses from [upstream_id, relative_path] to [upstream_id, relative_path, status].

🗒 MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

✅

🦄 Screenshots or screen recordings

No UI changes.

⚙ How to set up and validate locally

To try using the virtual registries, you can follow these setup directions:

• !163641 (merged)
• !164076 (merged)
• !165195 (merged)

💽 Database review

⤴ Migration up

main: == [advisory_lock_connection] object_id: 129080, pg_backend_pid: 25980
main: == 20240925113134 UpdateUniqueIndexOnVirtualRegistriesPackagesMavenCachedResponses: migrating 
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0328s
main: -- index_exists?(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :relative_path], {:unique=>true, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_default_upt_id_relpath", :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0022s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0005s
main: -- add_index(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :relative_path], {:unique=>true, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_default_upt_id_relpath", :where=>"status = 0", :algorithm=>:concurrently})
main:    -> 0.0037s
main: -- execute("RESET statement_timeout")
main:    -> 0.0004s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0003s
main: -- index_exists?(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :status, :id], {:name=>"idx_vregs_pkgs_mvn_cached_resp_on_upst_id_status_id", :algorithm=>:concurrently})
main:    -> 0.0017s
main: -- add_index(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :status, :id], {:name=>"idx_vregs_pkgs_mvn_cached_resp_on_upst_id_status_id", :algorithm=>:concurrently})
main:    -> 0.0037s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0003s
main: -- indexes(:virtual_registries_packages_maven_cached_responses)
main:    -> 0.0024s
main: -- remove_index(:virtual_registries_packages_maven_cached_responses, {:algorithm=>:concurrently, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_upstrm_id_and_rel_path"})
main:    -> 0.0031s
main: == 20240925113134 UpdateUniqueIndexOnVirtualRegistriesPackagesMavenCachedResponses: migrated (0.0762s) 

main: == [advisory_lock_connection] object_id: 129080, pg_backend_pid: 25980
ci: == [advisory_lock_connection] object_id: 129340, pg_backend_pid: 25982
ci: == 20240925113134 UpdateUniqueIndexOnVirtualRegistriesPackagesMavenCachedResponses: migrating 
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.0004s
ci: -- index_exists?(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :relative_path], {:unique=>true, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_default_upt_id_relpath", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0022s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0003s
ci: -- add_index(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :relative_path], {:unique=>true, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_default_upt_id_relpath", :where=>"status = 0", :algorithm=>:concurrently})
ci:    -> 0.0023s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0002s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.0003s
ci: -- index_exists?(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :status, :id], {:name=>"idx_vregs_pkgs_mvn_cached_resp_on_upst_id_status_id", :algorithm=>:concurrently})
ci:    -> 0.0018s
ci: -- add_index(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :status, :id], {:name=>"idx_vregs_pkgs_mvn_cached_resp_on_upst_id_status_id", :algorithm=>:concurrently})
ci:    -> 0.0012s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.0003s
ci: -- indexes(:virtual_registries_packages_maven_cached_responses)
ci:    -> 0.0020s
ci: -- remove_index(:virtual_registries_packages_maven_cached_responses, {:algorithm=>:concurrently, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_upstrm_id_and_rel_path"})
ci:    -> 0.0017s
ci: == 20240925113134 UpdateUniqueIndexOnVirtualRegistriesPackagesMavenCachedResponses: migrated (0.0322s) 

ci: == [advisory_lock_connection] object_id: 129340, pg_backend_pid: 25982

⤵ Migration down

main: == [advisory_lock_connection] object_id: 128600, pg_backend_pid: 25177
main: == 20240925113134 UpdateUniqueIndexOnVirtualRegistriesPackagesMavenCachedResponses: reverting 
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0349s
main: -- index_exists?(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :relative_path], {:unique=>true, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_upstrm_id_and_rel_path", :algorithm=>:concurrently})
main:    -> 0.0052s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0005s
main: -- add_index(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :relative_path], {:unique=>true, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_upstrm_id_and_rel_path", :algorithm=>:concurrently})
main:    -> 0.0034s
main: -- execute("RESET statement_timeout")
main:    -> 0.0004s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0003s
main: -- indexes(:virtual_registries_packages_maven_cached_responses)
main:    -> 0.0021s
main: -- remove_index(:virtual_registries_packages_maven_cached_responses, {:algorithm=>:concurrently, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_upst_id_status_id"})
main:    -> 0.0022s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0003s
main: -- indexes(:virtual_registries_packages_maven_cached_responses)
main:    -> 0.0016s
main: -- remove_index(:virtual_registries_packages_maven_cached_responses, {:algorithm=>:concurrently, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_default_upt_id_relpath"})
main:    -> 0.0018s
main: == 20240925113134 UpdateUniqueIndexOnVirtualRegistriesPackagesMavenCachedResponses: reverted (0.0782s) 

main: == [advisory_lock_connection] object_id: 128600, pg_backend_pid: 25177
ci: == [advisory_lock_connection] object_id: 128600, pg_backend_pid: 25562
ci: == 20240925113134 UpdateUniqueIndexOnVirtualRegistriesPackagesMavenCachedResponses: reverting 
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.0401s
ci: -- index_exists?(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :relative_path], {:unique=>true, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_upstrm_id_and_rel_path", :algorithm=>:concurrently})
ci:    -> 0.0055s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0005s
ci: -- add_index(:virtual_registries_packages_maven_cached_responses, [:upstream_id, :relative_path], {:unique=>true, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_upstrm_id_and_rel_path", :algorithm=>:concurrently})
ci:    -> 0.0038s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0003s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.0004s
ci: -- indexes(:virtual_registries_packages_maven_cached_responses)
ci:    -> 0.0019s
ci: -- remove_index(:virtual_registries_packages_maven_cached_responses, {:algorithm=>:concurrently, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_upst_id_status_id"})
ci:    -> 0.0024s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.0003s
ci: -- indexes(:virtual_registries_packages_maven_cached_responses)
ci:    -> 0.0018s
ci: -- remove_index(:virtual_registries_packages_maven_cached_responses, {:algorithm=>:concurrently, :name=>"idx_vregs_pkgs_mvn_cached_resp_on_uniq_default_upt_id_relpath"})
ci:    -> 0.0011s
ci: == 20240925113134 UpdateUniqueIndexOnVirtualRegistriesPackagesMavenCachedResponses: reverted (0.0916s) 

ci: == [advisory_lock_connection] object_id: 128600, pg_backend_pid: 25562

⚙ Updated queries

• Find cached response by relative path (with group id).
• Find cached response by relative path (without group id).
• Search through cached responses.