Process webhook notifs for container registry events to track usage
What does this MR do and why?
Closes #385497 (closed)
This MR utilizes the existing webhook notifications and processes them to push data to Gitlab::UsageDataCounters::HLLRedisCounter
and adding new events to the Service Ping.
New events:
- i_container_registry_delete_tag_user
- i_container_registry_push_tag_user
- i_container_registry_create_repository_user
- i_container_registry_delete_repository_user
- i_container_registry_push_repository_user
For groupcontainer registry, we want to have weekly
and monthly
aggregations of these events thus we have both monthly and weekly metric definitions. The events that are considered within this MR are the ones from a User
.
To push unique user data, an id
is needed and we get this from the User
with the username
in the actor["name"]
of the event hash. We make sure that the event is from a User
by looking into the actor["user_type"]
. The User
related ones are:
personal_access_token
build
gitlab_or_ldap
See below for the event
structure (reference:
{
"events": [
{
"id": "a582a0f3-e620-43e6-8e98-ff850fc9d984",
"timestamp": "2023-01-25T14:45:54.17327+11:00",
"action": "push",
"target": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 528,
"digest": "sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0",
"length": 528,
"repository": "root/test",
"url": "http://registry.test:5000/v2/root/test/manifests/sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0",
"tag": "more-auth"
},
"request": {
"id": "8ef8d69b-f957-45ab-ba2c-9153291f47b6",
"addr": "172.16.123.1:56969",
"host": "registry.test:5000",
"method": "PUT",
"useragent": "docker/20.10.18 go/go1.18.6 git-commit/e42327a6d3c55ceda3bd5475be7aae6036d02db3 kernel/5.15.68-0-virt os/linux arch/arm64 UpstreamClient(Docker-Client/20.10.22 \\(darwin\\))"
},
"actor": {
"name": "root",
"user_type": "personal_access_token"
},
"source": {
"addr": "Jaimes-MBP:5000",
"instanceID": "45681f21-a006-42f2-ab7c-4cc37d8906b4"
}
}
]
Validating Locally
To validate locally, setup the Service Ping locally with this guide.
1.Verifying the metric definitions appear on usage data
curl http://gdk.test:3000/api/v4/usage_data/metric_definitions | grep -C 5 'i_container_registry'
This would return the metric definitions.
redis_hll_counters
2.Verify that our new counters are part of the Open the rails console:
bundle exec rails c
And then query the following:
usage = GitlabServicePingWorker.new.usage_data;nil
usage.dig('redis_hll_counters', 'user_container_registry').keys
3.Verify that the counts are increased
The counters are incremented whenever the above events are emitted. They are emitted whenever:
- a tag gets deleted
- a tag gets pushed
- a repository gets deleted
- a repository gets created
- a repository gets pushed
To be able to do this, we will have to setup the Container Registry standalone on fresh master (outside the GDK) and enable notifications
through the instructions here.
An alternative is to call the endpoint with the ContainerRegistry event payload format as documented in container-registry!1205 (merged):
To be able to curl without any tokens,
- open
api/container_registry_event.rb
and - comment out the line
before { authenticate_registry_notification! }
curl -d '{
"events": [
{
"id": "a582a0f3-e620-43e6-8e98-ff850fc9d984",
"timestamp": "2023-01-25T14:45:54.17327+11:00",
"action": "delete",
"target": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 528,
"digest": "sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0",
"length": 528,
"repository": "root/test",
"url": "http://registry.test:5000/v2/root/test/manifests/sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0",
"tag": "latest"
},
"request": {
"id": "8ef8d69b-f957-45ab-ba2c-9153291f47b6",
"addr": "172.16.123.1:56969",
"host": "registry.test:5000",
"method": "PUT",
"useragent": "docker/20.10.18 go/go1.18.6 git-commit/e42327a6d3c55ceda3bd5475be7aae6036d02db3 kernel/5.15.68-0-virt os/linux arch/arm64 UpstreamClient(Docker-Client/20.10.22 \\(darwin\\))"
},
"actor": {
"name": "root",
"user_type": "personal_access_token"
},
"source": {
"addr": "127.0.0.1:5000",
"instanceID": "45681f21-a006-42f2-ab7c-4cc37d8906b4"
}
}
]
}' -H "Content-Type: application/vnd.docker.distribution.events.v1+json" -X POST 'http://localhost:3000/api/v4/container_registry_event/events'
And then to verify that the metric was counted, do a:
Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'i_container_registry_delete_tag_user',
start_date: Date.current.beginning_of_week, end_date: Date.current.next_week)
Replace i_container_registry_delete_tag_user
with the event that you tried.
Notes:
- Feel free to replace
name: "root"
to other users in your local machine. - Feel free to replace
"action": "delete"
with"action": "push"
for a push event - Removing the line
"tag": "latest"
and the preceding comma will make it a repository event.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #385497 (closed)