Enqueue a worker to sync a metadata cache
Context
In #393636 (closed) we added a background worker to generate metadata cache and in #393655 (closed) we started to respond with a metadata cache when the metadata endpoint is hit.
The next step is to define places and enqueue a worker to sync a metadata cache.
What does this MR do and why?
This MR enqueues the Packages::Npm::CreateMetadataCacheWorker
when npm metadata should be updated:
- A package or a package file was deleted.
- A new package was created.
- A tag was added or deleted.
- A metadata endpoint was hit, but metadata cache was missing.
Since the feature still in development, all changes are gated behind the feature flag.
Screenshots or screen recordings
No.
How to set up and validate locally
This is the mix of steps that should be either executed in rails console (ruby commands) or in the terminal.
-
Enable the
npm_metadata_cache
feature flag for chosen project.Feature.enable(:npm_metadata_cache, Project.first)
Note: Here I used
Project.first
but any projects can be used, we need laterproject.namespace.path
andproject.id
In my case it's a@gitlab-org
and1
-
Publish a new npm package.
2.1. Create an empty npm package.
$ mkdir test-metadata-cache && cd $_ $ touch .tool-versions $ echo nodejs 18.16.0 > .tool-versions $ npm init
2.2. Create
.npmrc
file in the same directory with a project -level endpoint.$ touch .npmrc $ echo '@scope:registry=http://gdk.test:3000/api/v4/projects/<project_id>/packages/npm/' >> .npmrc $ echo '//gdk.test:3000/api/v4/projects/<project_id>/packages/npm/:_authToken="${TOKEN}"' >> .npmrc
Note: Replace
scope
with a project's namespace and<project_id>
with project's id.2.3. Edit package's name to contain project's namespace
$ sed -i -e 's/test-metadata-cache/@gitlab-org\/test-metadata-cache/g' package.json
2.4. Publish a package to the Package registry
$ TOKEN=<your_token> npm publish
Note: Replace
<your_token>
with a valid token. -
Validate if a new entry of
Packages::Npm::MetadataCache
was created.Packages::Npm::MetadataCache.find_by(package_name: '@gitlab-org/test-metadata-cache')
-
Create a request to a metadata endpoint and check if a metadata cache was returned.
$ curl --header "PRIVATE-TOKEN: <your-token>" "http://gdk.test:3000/api/v4/projects/<project_id>/packages/npm/<package_name>"
Note: Replace
<your_token>
with a valid token,<project_id>
with a project's id and<package_name>
with a package's name.The easiest way to check if a response comes from a metadata cache is to check
last_downloaded_at
attribute of aPackages::Npm::MetadataCache
entry. We can do that as following:Packages::Npm::MetadataCache.find_by(package_name: '@gitlab-org/test-metadata-cache').last_downloaded_at
The value should match current time.
Now let's check if a metadata cache gets synched after the changes.
-
Add a new tag for the package
$ TOKEN=<your_token> npm dist-tag add @scope/test-metadata-cache@1.0.0 my-tag
Note: Replace
<your_token>
with a valid token and@scope
with a project's namespace. -
Verify if a metadata contains a new tag
$ curl --header "PRIVATE-TOKEN: <your-token>" "http://gdk.test:3000/api/v4/projects/<project_id>/packages/npm/<package_name>"
Note: Replace
<your_token>
with a valid token,<project_id>
with a project's id and<package_name>
with a package's name. -
Remove the most recently added tag
$ TOKEN=<your_token> npm dist-tag rm @scope/hello_world@1.0.0 my-tag
Note: Replace
<your_token>
with a valid token and@scope
with a project's namespace. -
Verify if a metadata does not contain this tag anymore
$ curl --header "PRIVATE-TOKEN: <your-token>" "http://gdk.test:3000/api/v4/projects/<project_id>/packages/npm/<package_name>"
Note: Replace
<your_token>
with a valid token,<project_id>
with a project's id and<package_name>
with a package's name. -
Publish two new versions of the package
sed -i -e 's/1.0.0/2.0.0/g' package.json $ TOKEN=<your_token> npm publish sed -i -e 's/2.0.0/3.0.0/g' package.json $ TOKEN=<your_token> npm publish
Note: Replace
<your_token>
with a valid token. -
Verify if a metadata does not contain this tag anymore
$ curl --header "PRIVATE-TOKEN: <your-token>" "http://gdk.test:3000/api/v4/projects/<project_id>/packages/npm/<package_name>"
Note: Replace
<your_token>
with a valid token,<project_id>
with a project's id and<package_name>
with a package's name. -
Remove a package file from the most recent package's version
Packages::Package.find_by(name: "@gitlab-org/test-metadata-cache", version: "3.0.0").id # note down package's id Packages::Package.find_by(name: "@gitlab-org/test-metadata-cache", version: "3.0.0").package_files.last.id # note down package file's id
$ curl --request DELETE --header "PRIVATE-TOKEN: <your_token>" "http://gdk.test:3000/api/v4/projects/<project_id>/packages/<package_id>/package_files/<package_file_id>"
Note: Replace
<your_token>
with a valid token,<project_id>
with a project's id and<package_id>
with a package's id and<package_file_id>
with a package file's id. -
Verify if a metadata is up to date - doesn't contain a package with a version
3.0.0
.$ curl --header "PRIVATE-TOKEN: <your-token>" "http://gdk.test:3000/api/v4/projects/<project_id>/packages/npm/<package_name>"
Note: Replace
<your_token>
with a valid token,<project_id>
with a project's id and<package_name>
with a package's name. -
Remove a package with the version
2.0.0
Packages::Package.find_by(name: "@gitlab-org/test-metadata-cache", version: "2.0.0").id # note down package's id
curl --request DELETE --header "PRIVATE-TOKEN: <your_token>" "http://gdk.test:3000/api/v4/projects/<project_id>/packages/<package_id>"
Note: Replace
<your_token>
with a valid token,<project_id>
with a project's id and<package_id>
with a package's id. -
Verify if a metadata is up to date - doesn't contain a package with a version
2.0.0
.$ curl --header "PRIVATE-TOKEN: <your-token>" "http://gdk.test:3000/api/v4/projects/<project_id>/packages/npm/<package_name>"
Note: Replace
<your_token>
with a valid token,<project_id>
with a project's id and<package_name>
with a package's name.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #393652 (closed)