Refactor the helm index presenter (!69223) · Merge requests · GitLab.org / GitLab

David Fernandez requested to merge 337974-fix-helm-metadata-endpoint into master Aug 30, 2021

👾 Context

In %14.1, we introduced support for helm packages within the GitLab package registry.

That support was implemented with several API endpoints. One of those endpoints is what we call the metadata endpoint. Basically, clients (usually $ helm) will ping this endpoint to get the "state" of the registry. By "state", I mean "which packages are available and which versions are available".

Depending on what the client expect, we could seed all the available packages with all the available versions. Here with helm, we need to send back all the available packages given a channel name.

As most of these packages metadata endpoints, pagination is not an option so we should be careful on the amount of objects returned.

Last piece of context, helm packages are usually materialized by a single file (a .tgz file). So when a helm package is pushed, a single file is uploaded. Also, the GitLab package registry allows duplicates uploads. This means that if a user uploads 3 times a helm package with the same package name and version, the GitLab package registry will have 3 files associated with that package. This works because when the file of a given package is requested, the registry will always return the most recent one.

With #337974 (closed), users found that there was something off with the metadata endpoint. Basically, it was found that the finder used for that endpoint would return the 20 most recent package files.

The problem here is twofold:

20 is a too low limit and can be easily be hit
The current finder used works on packages files, disregarding the package objects. This means that duplicated files will be returned and so, the limit is not applied in the right level. It is applied at the "package file" level instead of the "package" level.

🔍 What does this MR do?

Introduce a new helm finder for packages.
- This finder look for packages given a project and channel.
- It will return the 300 most recent packages. This is a hard limit that we impose here.
  - This limit is similar to what we do with the nuget endpoint APIs.
  - We will iterate on this value, meaning that we will revisit this if necessary in the future.
Update the index presenter to work on a set of packages (instead a set of package files).
- The presenter here needs to get the most recent package file. This is a common need so we created a new scope in the Packages::PackageFile.
Created a new scope in Packages::PackageFile to return the most recent package files given a set of packages.
- This will create a query based on a lateral join with package files to select only the most recent package file for each package.
- This scope will only be useful for package types where a push operation is a single file upload. That's the case for helm but not all package types work like this (for example, with maven, the push operation uploads at least 2 files)
Added/Updated the related specs
- Making sure that if the presenter or API endpoint hits packages with duplicated package files, only the most recent package file.

📸 Screenshots or Screencasts (strongly suggested)

Case: many helm packages with duplicates

Given a project, let's add it as a helm repository:

$ helm repo add --username <user> --password <pat_token> gitlab <gitlab_base_url>/api/v4/projects/<project_id>/packages/helm/stable

Let's create 10 different helm packages. (Pardon my poor skills at zsh scripts 😺)

Shell commands

$ repeat 10 { PKG=`base64 </dev/urandom | tr -dc 'a-zA-Z' | head -c20` && helm create foobar-$PKG && helm package foobar-$PKG && helm push foobar-$PKG-0.1.0.tgz gitlab }
Creating foobar-jknYlguxYGAyjrfddEZn
Successfully packaged chart and saved it to: /Users/david/projects/sandbox/helm/mr/foobar-jknYlguxYGAyjrfddEZn-0.1.0.tgz
Pushing foobar-jknYlguxYGAyjrfddEZn-0.1.0.tgz to gitlab...
Done.
Creating foobar-YcnWQxKhgHIhspHWoMkx
Successfully packaged chart and saved it to: /Users/david/projects/sandbox/helm/mr/foobar-YcnWQxKhgHIhspHWoMkx-0.1.0.tgz
Pushing foobar-YcnWQxKhgHIhspHWoMkx-0.1.0.tgz to gitlab...
Done.
Creating foobar-VBLqqIEhiYuyMWKlYsuI
Successfully packaged chart and saved it to: /Users/david/projects/sandbox/helm/mr/foobar-VBLqqIEhiYuyMWKlYsuI-0.1.0.tgz
Pushing foobar-VBLqqIEhiYuyMWKlYsuI-0.1.0.tgz to gitlab...
Done.
Creating foobar-mNdJQHPPIXfeVmRLdNay
Successfully packaged chart and saved it to: /Users/david/projects/sandbox/helm/mr/foobar-mNdJQHPPIXfeVmRLdNay-0.1.0.tgz
Pushing foobar-mNdJQHPPIXfeVmRLdNay-0.1.0.tgz to gitlab...
Done.
Creating foobar-DMABduUqxcmaAwdORQBw
Successfully packaged chart and saved it to: XXX/foobar-DMABduUqxcmaAwdORQBw-0.1.0.tgz
Pushing foobar-DMABduUqxcmaAwdORQBw-0.1.0.tgz to gitlab...
Done.
Creating foobar-WqihEVDQxUZhXrrcKIDu
Successfully packaged chart and saved it to: XXX/foobar-WqihEVDQxUZhXrrcKIDu-0.1.0.tgz
Pushing foobar-WqihEVDQxUZhXrrcKIDu-0.1.0.tgz to gitlab...
Done.
Creating foobar-EfBMzPCWusrdYzGjmung
Successfully packaged chart and saved it to: XXX/foobar-EfBMzPCWusrdYzGjmung-0.1.0.tgz
Pushing foobar-EfBMzPCWusrdYzGjmung-0.1.0.tgz to gitlab...
Done.
Creating foobar-NjLVAYpuOkgnBuFcRdtc
Successfully packaged chart and saved it to: XXX/foobar-NjLVAYpuOkgnBuFcRdtc-0.1.0.tgz
Pushing foobar-NjLVAYpuOkgnBuFcRdtc-0.1.0.tgz to gitlab...
Done.
Creating foobar-nUAnPonhWEVFOzjhvwPB
Successfully packaged chart and saved it to: XXX/foobar-nUAnPonhWEVFOzjhvwPB-0.1.0.tgz
Pushing foobar-nUAnPonhWEVFOzjhvwPB-0.1.0.tgz to gitlab...
Done.
Creating foobar-riHvrqMJgqqRYehpqkVb
Successfully packaged chart and saved it to: XXX/foobar-riHvrqMJgqqRYehpqkVb-0.1.0.tgz
Pushing foobar-riHvrqMJgqqRYehpqkVb-0.1.0.tgz to gitlab...
Done.

Let's add some duplicates here and create 5 packages where we upload 10 times to the same package name and version.

Shell commands

$ repeat 5 { PKG=`base64 </dev/urandom | tr -dc 'a-zA-Z' | head -c20` && helm create foobar-$PKG && helm package foobar-$PKG && repeat 10 { helm push foobar-$PKG-0.1.0.tgz gitlab } }
Creating foobar-NPJtmmIiFXUohgxwAKvG
Successfully packaged chart and saved it to: XXX/foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz
Pushing foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz to gitlab...
Done.
Creating foobar-IuBHpVMmnlNUvXflDLAK
Successfully packaged chart and saved it to: XXX/foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz
Pushing foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz to gitlab...
Done.
Pushing foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz to gitlab...
Done.
Pushing foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz to gitlab...
Done.
Pushing foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz to gitlab...
Done.
Pushing foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz to gitlab...
Done.
Pushing foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz to gitlab...
Done.
Pushing foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz to gitlab...
Done.
Pushing foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz to gitlab...
Done.
Pushing foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz to gitlab...
Done.
Pushing foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz to gitlab...
Done.
Creating foobar-gZaTvQaUdfRLyVRAhBks
Successfully packaged chart and saved it to: XXX/foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz
Pushing foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz to gitlab...
Done.
Pushing foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz to gitlab...
Done.
Pushing foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz to gitlab...
Done.
Pushing foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz to gitlab...
Done.
Pushing foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz to gitlab...
Done.
Pushing foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz to gitlab...
Done.
Pushing foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz to gitlab...
Done.
Pushing foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz to gitlab...
Done.
Pushing foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz to gitlab...
Done.
Pushing foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz to gitlab...
Done.
Creating foobar-xWbrLuqlKvdqGukJwECr
Successfully packaged chart and saved it to: XXX/foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz
Pushing foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz to gitlab...
Done.
Pushing foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz to gitlab...
Done.
Pushing foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz to gitlab...
Done.
Pushing foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz to gitlab...
Done.
Pushing foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz to gitlab...
Done.
Pushing foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz to gitlab...
Done.
Pushing foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz to gitlab...
Done.
Pushing foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz to gitlab...
Done.
Pushing foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz to gitlab...
Done.
Pushing foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz to gitlab...
Done.
Creating foobar-NvbtCHwfWtoQbSItwLKF
Successfully packaged chart and saved it to: XXX/foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz
Pushing foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz to gitlab...
Done.
Pushing foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz to gitlab...
Done.

Let's update the helm repo:

$ helm repo update

Now, let's check the available packages

$ helm search repo foobar  
NAME                              	CHART VERSION	APP VERSION	DESCRIPTION                
gitlab/foobar-dmabduuqxcmaawdorqbw	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-efbmzpcwusrdyzgjmung	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-gzatvqaudfrlyvrahbks	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-iubhpvmmnlnuvxfldlak	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-jknylguxygayjrfddezn	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-mndjqhppixfevmrldnay	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-njlvaypuokgnbufcrdtc	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-npjtmmiifxuohgxwakvg	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-nuanponhwevfozjhvwpb	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-nvbtchwfwtoqbsitwlkf	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-rihvrqmjgqqryehpqkvb	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-vblqqiehiyuymwklysui	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-wqihevdqxuzhxrrckidu	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-xwbrluqlkvdqgukjwecr	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-ycnwqxkhghihsphwomkx	0.1.0        	1.16.0     	A Helm chart for Kubernetes

The 15 packages are properly returned.

Let's curl the metadata endpoint:

curl output

$ curl http://<user>:<pat_token>@<gitlab_base_url>/api/v4/projects/<project_id>/packages/helm/stable/index.yaml
---
apiVersion: v1
entries:
  foobar-jknYlguxYGAyjrfddEZn:
  - name: foobar-jknYlguxYGAyjrfddEZn
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:03:49.010478000Z'
    digest:
    urls:
    - charts/foobar-jknYlguxYGAyjrfddEZn-0.1.0.tgz
  foobar-YcnWQxKhgHIhspHWoMkx:
  - name: foobar-YcnWQxKhgHIhspHWoMkx
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:03:50.314913000Z'
    digest:
    urls:
    - charts/foobar-YcnWQxKhgHIhspHWoMkx-0.1.0.tgz
  foobar-VBLqqIEhiYuyMWKlYsuI:
  - name: foobar-VBLqqIEhiYuyMWKlYsuI
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:03:51.573050000Z'
    digest:
    urls:
    - charts/foobar-VBLqqIEhiYuyMWKlYsuI-0.1.0.tgz
  foobar-mNdJQHPPIXfeVmRLdNay:
  - name: foobar-mNdJQHPPIXfeVmRLdNay
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:03:52.833776000Z'
    digest:
    urls:
    - charts/foobar-mNdJQHPPIXfeVmRLdNay-0.1.0.tgz
  foobar-DMABduUqxcmaAwdORQBw:
  - name: foobar-DMABduUqxcmaAwdORQBw
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:03:54.100017000Z'
    digest:
    urls:
    - charts/foobar-DMABduUqxcmaAwdORQBw-0.1.0.tgz
  foobar-WqihEVDQxUZhXrrcKIDu:
  - name: foobar-WqihEVDQxUZhXrrcKIDu
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:03:55.355456000Z'
    digest:
    urls:
    - charts/foobar-WqihEVDQxUZhXrrcKIDu-0.1.0.tgz
  foobar-EfBMzPCWusrdYzGjmung:
  - name: foobar-EfBMzPCWusrdYzGjmung
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:03:56.623086000Z'
    digest:
    urls:
    - charts/foobar-EfBMzPCWusrdYzGjmung-0.1.0.tgz
  foobar-NjLVAYpuOkgnBuFcRdtc:
  - name: foobar-NjLVAYpuOkgnBuFcRdtc
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:03:57.884538000Z'
    digest:
    urls:
    - charts/foobar-NjLVAYpuOkgnBuFcRdtc-0.1.0.tgz
  foobar-nUAnPonhWEVFOzjhvwPB:
  - name: foobar-nUAnPonhWEVFOzjhvwPB
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:03:59.135067000Z'
    digest:
    urls:
    - charts/foobar-nUAnPonhWEVFOzjhvwPB-0.1.0.tgz
  foobar-riHvrqMJgqqRYehpqkVb:
  - name: foobar-riHvrqMJgqqRYehpqkVb
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:04:00.397702000Z'
    digest:
    urls:
    - charts/foobar-riHvrqMJgqqRYehpqkVb-0.1.0.tgz
  foobar-NPJtmmIiFXUohgxwAKvG:
  - name: foobar-NPJtmmIiFXUohgxwAKvG
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:05:08.898255000Z'
    digest:
    urls:
    - charts/foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz
  foobar-IuBHpVMmnlNUvXflDLAK:
  - name: foobar-IuBHpVMmnlNUvXflDLAK
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:05:25.641067000Z'
    digest:
    urls:
    - charts/foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz
  foobar-gZaTvQaUdfRLyVRAhBks:
  - name: foobar-gZaTvQaUdfRLyVRAhBks
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:05:43.519507000Z'
    digest:
    urls:
    - charts/foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz
  foobar-xWbrLuqlKvdqGukJwECr:
  - name: foobar-xWbrLuqlKvdqGukJwECr
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:06:01.314275000Z'
    digest:
    urls:
    - charts/foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz
  foobar-NvbtCHwfWtoQbSItwLKF:
  - name: foobar-NvbtCHwfWtoQbSItwLKF
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:06:20.532499000Z'
    digest:
    urls:
    - charts/foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz
generated: '2021-08-30T13:09:30.829620000Z'
serverInfo:
  contextPath: "/api/v4/projects/<project_id>/packages/helm"

Let's see if the limit works, we're going to temporarily update the limit for packages to say, 8

$ git diff
diff --git a/app/finders/packages/helm/packages_finder.rb b/app/finders/packages/helm/packages_finder.rb
index 8418f437a1a..8aac9571251 100644
--- a/app/finders/packages/helm/packages_finder.rb
+++ b/app/finders/packages/helm/packages_finder.rb
@@ -5,7 +5,7 @@ module Helm
     class PackagesFinder
       include ::Packages::FinderHelper
 
-      MAX_PACKAGES_COUNT = 300
+      MAX_PACKAGES_COUNT = 8
 
       def initialize(project, channel)
         @project = project

Let's update the repo again:

$ helm repo update

Let's search for our packages again:

$ helm search repo foobar                                                                               
NAME                              	CHART VERSION	APP VERSION	DESCRIPTION                
gitlab/foobar-gzatvqaudfrlyvrahbks	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-iubhpvmmnlnuvxfldlak	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-njlvaypuokgnbufcrdtc	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-npjtmmiifxuohgxwakvg	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-nuanponhwevfozjhvwpb	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-nvbtchwfwtoqbsitwlkf	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-rihvrqmjgqqryehpqkvb	0.1.0        	1.16.0     	A Helm chart for Kubernetes
gitlab/foobar-xwbrluqlkvdqgukjwecr	0.1.0        	1.16.0     	A Helm chart for Kubernetes

Let's curl once again the metadata endpoint:

curl output

$ curl http://<user>:<pat_token>@<gitlab_base_url>/api/v4/projects/<project_id>/packages/helm/stable/index.yaml
---
apiVersion: v1
entries:
  foobar-NjLVAYpuOkgnBuFcRdtc:
  - name: foobar-NjLVAYpuOkgnBuFcRdtc
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:03:57.884538000Z'
    digest:
    urls:
    - charts/foobar-NjLVAYpuOkgnBuFcRdtc-0.1.0.tgz
  foobar-nUAnPonhWEVFOzjhvwPB:
  - name: foobar-nUAnPonhWEVFOzjhvwPB
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:03:59.135067000Z'
    digest:
    urls:
    - charts/foobar-nUAnPonhWEVFOzjhvwPB-0.1.0.tgz
  foobar-riHvrqMJgqqRYehpqkVb:
  - name: foobar-riHvrqMJgqqRYehpqkVb
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:04:00.397702000Z'
    digest:
    urls:
    - charts/foobar-riHvrqMJgqqRYehpqkVb-0.1.0.tgz
  foobar-NPJtmmIiFXUohgxwAKvG:
  - name: foobar-NPJtmmIiFXUohgxwAKvG
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:05:08.898255000Z'
    digest:
    urls:
    - charts/foobar-NPJtmmIiFXUohgxwAKvG-0.1.0.tgz
  foobar-IuBHpVMmnlNUvXflDLAK:
  - name: foobar-IuBHpVMmnlNUvXflDLAK
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:05:25.641067000Z'
    digest:
    urls:
    - charts/foobar-IuBHpVMmnlNUvXflDLAK-0.1.0.tgz
  foobar-gZaTvQaUdfRLyVRAhBks:
  - name: foobar-gZaTvQaUdfRLyVRAhBks
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:05:43.519507000Z'
    digest:
    urls:
    - charts/foobar-gZaTvQaUdfRLyVRAhBks-0.1.0.tgz
  foobar-xWbrLuqlKvdqGukJwECr:
  - name: foobar-xWbrLuqlKvdqGukJwECr
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:06:01.314275000Z'
    digest:
    urls:
    - charts/foobar-xWbrLuqlKvdqGukJwECr-0.1.0.tgz
  foobar-NvbtCHwfWtoQbSItwLKF:
  - name: foobar-NvbtCHwfWtoQbSItwLKF
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: A Helm chart for Kubernetes
    created: '2021-08-30T13:06:20.532499000Z'
    digest:
    urls:
    - charts/foobar-NvbtCHwfWtoQbSItwLKF-0.1.0.tgz
generated: '2021-08-30T14:42:51.320531000Z'
serverInfo:
  contextPath: "/api/v4/projects/47/packages/helm"

The 8 most recent packages are properly returned. 🚀

Case: Same helm packages in different channels

Add a helm repo as above.

Let's check the metadata endpoint for the stable channel:

$ curl http://<username>:<pat_token>@<gitlab_base_url>/api/v4/projects/<project_id>/packages/helm/stable/index.yaml
---
apiVersion: v1
entries: {}
generated: '2021-09-03T10:40:14.109968000Z'
serverInfo:
  contextPath: "/api/v4/projects/47/packages/helm"

No packages for now.

Let's create a helm package and upload three times to three different channels. Before uploading, we update the package description so that it has the channel name.

$ helm create bananas
[ ... update the package description with the stable word ... ]
$ helm package bananas
$ curl --request POST \                                                                                               
     --form 'chart=@bananas-0.1.0.tgz' \
     --user <username>:<pat_token> \
     <gitlab_base_url>/api/v4/projects/<project_id>/packages/helm/api/stable/charts

[ ... update the package description with the alpha word ... ]
$ helm package bananas
$ curl --request POST \                                                                                               
     --form 'chart=@bananas-0.1.0.tgz' \
     --user <username>:<pat_token> \
     <gitlab_base_url>/api/v4/projects/<project_id>/packages/helm/api/alpha/charts

[ ... update the package description with the beta word ... ]
$ helm package bananas
$ curl --request POST \                                                                                               
     --form 'chart=@bananas-0.1.0.tgz' \
     --user <username>:<pat_token> \
     <gitlab_base_url>/api/v4/projects/<project_id>/packages/helm/api/beta/charts

⚠ be sure to wait a few minutes here, a background job needs to run to process the uploaded packages.

Now let's have a look at the metadata response from the different channels

curls outputs

$ curl http://<username>:<pat_token>@<gitlab_base_url>/api/v4/projects/<project_id>/packages/helm/stable/index.yaml
---
apiVersion: v1
entries:
  bananas:
  - name: bananas
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: Hello world from stable channel
    created: '2021-09-03T10:42:39.334112000Z'
    digest:
    urls:
    - charts/bananas-0.1.0.tgz
generated: '2021-09-03T10:50:02.145641000Z'
serverInfo:
  contextPath: "/api/v4/projects/47/packages/helm"

$ curl http://<username>:<pat_token>@<gitlab_base_url>/api/v4/projects/<project_id>/packages/helm/alpha/index.yaml
---
apiVersion: v1
entries:
  bananas:
  - name: bananas
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: Hello world from alpha channel
    created: '2021-09-03T10:43:04.471427000Z'
    digest:
    urls:
    - charts/bananas-0.1.0.tgz
generated: '2021-09-03T10:50:25.457660000Z'
serverInfo:
  contextPath: "/api/v4/projects/47/packages/helm"

$ curl http://<username>:<pat_token>@<gitlab_base_url>/api/v4/projects/<project_id>/packages/helm/beta/index.yaml
---
apiVersion: v1
entries:
  bananas:
  - name: bananas
    type: application
    version: 0.1.0
    apiVersion: v2
    appVersion: 1.16.0
    description: Hello world from beta channel
    created: '2021-09-03T10:43:19.217459000Z'
    digest:
    urls:
    - charts/bananas-0.1.0.tgz
generated: '2021-09-03T10:50:50.234156000Z'
serverInfo:
  contextPath: "/api/v4/projects/47/packages/helm"

For each channel, we know that the right package file has been read because the description match the channel queried. 🎉

⚒ How to setup and validate locally (strongly suggested)

Install helm
Have a project ready (any visibility)
Have a personal access token ready (with scope api)

You can run the same commands as above.

📏 Does this MR meet the acceptance criteria?

Conformity

I have included changelog trailers, or none are needed. (Does this MR need a changelog?)
I have added/updated documentation, or it's not needed. (Is documentation required?)
[-] I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
I have self-reviewed this MR per code review guidelines.
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
I have followed the style guides.
This change is backwards compatible across updates, or this does not apply.

Availability and Testing

I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
[-] I have tested this MR in all supported browsers, or it's not needed.
[-] I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

Label as security and @ mention @gitlab-com/gl-security/appsec
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
Security reports checked/validated by a reviewer from the AppSec team

💾 Database

Migrations up

== 20210831134840 AddPackageFileIdChannelIdxToPackagesHelmFileMetadata: migrating 
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:packages_helm_file_metadata, [:package_file_id, :channel], {:name=>"index_packages_helm_file_metadata_on_pf_id_and_channel", :algorithm=>:concurrently})
   -> 0.0067s
-- execute("SET statement_timeout TO 0")
   -> 0.0006s
-- add_index(:packages_helm_file_metadata, [:package_file_id, :channel], {:name=>"index_packages_helm_file_metadata_on_pf_id_and_channel", :algorithm=>:concurrently})
   -> 0.0172s
-- execute("RESET statement_timeout")
   -> 0.0007s
== 20210831134840 AddPackageFileIdChannelIdxToPackagesHelmFileMetadata: migrated (0.0350s) 

== 20210831135249 AddInstallableHelmPkgsIdxToPackages: migrating ==============
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:packages_packages, [:project_id, :id], {:name=>"idx_installable_helm_pkgs_on_project_id_id", :algorithm=>:concurrently})
   -> 0.0068s
-- add_index(:packages_packages, [:project_id, :id], {:name=>"idx_installable_helm_pkgs_on_project_id_id", :algorithm=>:concurrently})
   -> 0.0052s
== 20210831135249 AddInstallableHelmPkgsIdxToPackages: migrated (0.0137s) =====

== 20210909184349 AddIndexPackageIdIdOnPackageFiles: migrating ================
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- indexes(:package_package_files)
   -> 0.0021s
-- current_schema()
   -> 0.0002s
-- execute("CREATE INDEX CONCURRENTLY index_packages_package_files_on_package_id_id ON packages_package_files (package_id, id)")
   -> 0.0063s
-- execute("RESET statement_timeout")
   -> 0.0008s
== 20210909184349 AddIndexPackageIdIdOnPackageFiles: migrated (0.0132s) =======

Migrations down

== 20210909184349 AddIndexPackageIdIdOnPackageFiles: reverting ================
-- transaction_open?()
   -> 0.0000s
-- indexes(:packages_package_files)
   -> 0.0042s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- remove_index(:packages_package_files, {:algorithm=>:concurrently, :name=>"index_packages_package_files_on_package_id_id"})
   -> 0.0029s
-- execute("RESET statement_timeout")
   -> 0.0005s
== 20210909184349 AddIndexPackageIdIdOnPackageFiles: reverted (0.0100s) =======

== 20210831135249 AddInstallableHelmPkgsIdxToPackages: reverting ==============
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:packages_packages, [:project_id, :id], {:name=>"idx_installable_helm_pkgs_on_project_id_id", :algorithm=>:concurrently})
   -> 0.0084s
-- execute("SET statement_timeout TO 0")
   -> 0.0007s
-- remove_index(:packages_packages, {:name=>"idx_installable_helm_pkgs_on_project_id_id", :algorithm=>:concurrently, :column=>[:project_id, :id]})
   -> 0.0123s
-- execute("RESET statement_timeout")
   -> 0.0007s
== 20210831135249 AddInstallableHelmPkgsIdxToPackages: reverted (0.0243s) =====

== 20210831134840 AddPackageFileIdChannelIdxToPackagesHelmFileMetadata: reverting 
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:packages_helm_file_metadata, [:package_file_id, :channel], {:name=>"index_packages_helm_file_metadata_on_pf_id_and_channel", :algorithm=>:concurrently})
   -> 0.0040s
-- execute("SET statement_timeout TO 0")
   -> 0.0016s
-- remove_index(:packages_helm_file_metadata, {:name=>"index_packages_helm_file_metadata_on_pf_id_and_channel", :algorithm=>:concurrently, :column=>[:package_file_id, :channel]})
   -> 0.0064s
-- execute("RESET statement_timeout")
   -> 0.0008s
== 20210831134840 AddPackageFileIdChannelIdxToPackagesHelmFileMetadata: reverted (0.0165s)

Explain plans

Get all the most recent package files given a set of packages: !69223 (comment 665487817).
Eager load all helm package file metadata: !69223 (comment 665602388)

Edited Sep 10, 2021 by David Fernandez

Refactor the helm index presenter