Skip to content

Create a service to build npm metadata cache record

🌱 Context

In !114312 (merged) we introduced a new model to manage npm metadata caches,
that suppose to operate like a cache and to reduce the load and speed up the metadata endpoints.

Now it's time to provide a convenient way of creating/updating npm metadata cache entry, therefore this MR introduces a new service.

🚢 What does this MR do and why?

  • Creates the new service Packages::Npm::CreateMetadataCacheService to create a new entity of npm metadata cache or update the metadata of existing one.

  • Additionally it introduces the new uploader Packages::Npm::MetadataCacheUploader to manage the metadata objects in the storage.

📷 Screenshots or screen recordings

No.

💾 Database migrations

Migration up

$ rails db:migrate:main

main: == [advisory_lock_connection] object_id: 274380, pg_backend_pid: 4527
main: == 20230403093349 EnsurePackagesNpmMetadataCachesIsEmpty: migrating ===========
main: -- quote_table_name("packages_npm_metadata_caches")
main:    -> 0.0000s
main: -- execute("TRUNCATE TABLE \"packages_npm_metadata_caches\"")
main:    -> 0.0067s
main: == 20230403093349 EnsurePackagesNpmMetadataCachesIsEmpty: migrated (0.0775s) ==

main: == 20230405071033 AddObjectStorageKeyToPackagesNpmMetadataCaches: migrating ===
main: -- column_exists?(:packages_npm_metadata_caches, :object_storage_key)
main:    -> 0.0053s
main: -- add_column(:packages_npm_metadata_caches, :object_storage_key, :text, {:null=>false})
main:    -> 0.0007s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0005s
main: -- index_exists?(:packages_npm_metadata_caches, :object_storage_key, {:unique=>true, :name=>"index_packages_npm_metadata_caches_on_object_storage_key", :algorithm=>:concurrently})
main:    -> 0.0021s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0003s
main: -- add_index(:packages_npm_metadata_caches, :object_storage_key, {:unique=>true, :name=>"index_packages_npm_metadata_caches_on_object_storage_key", :algorithm=>:concurrently})
main:    -> 0.0014s
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
main: == 20230405071033 AddObjectStorageKeyToPackagesNpmMetadataCaches: migrated (0.0204s) 

main: == 20230503191056 AddTextLimitToPackagesNpmMetadataCachesObjectStorageKey: migrating 
main: -- transaction_open?()
main:    -> 0.0000s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE packages_npm_metadata_caches\nADD CONSTRAINT check_f97c15aa60\nCHECK ( char_length(object_storage_key) <= 255 )\nNOT VALID;\n")
main:    -> 0.0006s
main: -- execute("ALTER TABLE packages_npm_metadata_caches VALIDATE CONSTRAINT check_f97c15aa60;")
main:    -> 0.0004s
main: == 20230503191056 AddTextLimitToPackagesNpmMetadataCachesObjectStorageKey: migrated (0.0187s) 

main: == [advisory_lock_connection] object_id: 274380, pg_backend_pid: 4527

Migration down

$ rails db:migrate:down:main VERSION=20230405071033
main: == [advisory_lock_connection] object_id: 274160, pg_backend_pid: 3290
main: == 20230405071033 AddObjectStorageKeyToPackagesNpmMetadataCaches: reverting ===
main: -- remove_column(:packages_npm_metadata_caches, :object_storage_key)
main:    -> 0.0050s
main: == 20230405071033 AddObjectStorageKeyToPackagesNpmMetadataCaches: reverted (0.0158s) 

$ rails db:migrate:down:main VERSION=20230403093349
main: == [advisory_lock_connection] object_id: 274120, pg_backend_pid: 3701
main: == 20230403093349 EnsurePackagesNpmMetadataCachesIsEmpty: reverting ===========
main: == 20230403093349 EnsurePackagesNpmMetadataCachesIsEmpty: reverted (0.0039s) ==

main: == [advisory_lock_connection] object_id: 274120, pg_backend_pid: 3701

$ rails db:migrate:down:main VERSION=20230503191056
main: == [advisory_lock_connection] object_id: 274120, pg_backend_pid: 4115
main: == 20230503191056 AddTextLimitToPackagesNpmMetadataCachesObjectStorageKey: reverting 
main: -- transaction_open?()
main:    -> 0.0000s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("            ALTER TABLE packages_npm_metadata_caches\n            DROP CONSTRAINT IF EXISTS check_f97c15aa60\n")
main:    -> 0.0005s
main: == 20230503191056 AddTextLimitToPackagesNpmMetadataCachesObjectStorageKey: reverted (0.0200s) 

main: == [advisory_lock_connection] object_id: 274120, pg_backend_pid: 4115

🔬 How to set up and validate locally

All steps need to be executed in rails console

  1. Create a new npm package:

    def fixture_file_upload(*args, **kwargs)
      Rack::Test::UploadedFile.new(*args, **kwargs)
    end
    
    package = FactoryBot.create(:npm_package, project: Project.first)
  2. Fetch packages as ActiveRecord::Relation:

    packages = Packages::Package.with_name(package.name)
  3. Execute the service to create a new entity of Packages::Npm::MetadataCache:

    Packages::Npm::CreateMetadataCacheService.new(Project.first, package.name, packages).execute
  4. Verify that a new entity of Packages::Npm::MetadataCache is created with the metadata:

    metadata_cache = Packages::Npm::MetadataCache.last
    Gitlab::Json.parse(metadata_cache.file.read)
  5. Create a new tag:

    Packages::Npm::CreateTagService.new(package, "my-new-tag").execute
  6. Trigger the Packages::Npm::CreateMetadataCacheService to update the metadata of existing Packages::Npm::MetadataCache entry:

    Packages::Npm::CreateMetadataCacheService.new(Project.first, package.name, packages).execute
  7. Verify if the metadata gets updated and now contain a new my-new-tag tag:

    metadata_cache = Packages::Npm::MetadataCache.last
    Gitlab::Json.parse(metadata_cache.file.read)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #393635 (closed)

Edited by Dzmitry (Dima) Meshcharakou

Merge request reports

Loading