Support Maven artifact classifiers when checking for duplicates
Context
When a package is uploaded to the GitLab Maven package registry, the clients will not send a single file but a set of files.
As such, the backend receives not one but multiple uploads.
The best way to understand this is to present the files (that are sent) in a tree-like view (simplified):
.
└── my-awesome-company
└── foobar-test-bananas
├── 4.3
│ ├── foobar-test-bananas-4.3.jar
│ └── foobar-test-bananas-4.3.pom
└── maven-metadata.xml
In reality, there a bit more files but for the sake of simplicity, we will keep it like this.
Now, maven packages also use what they call SNAPSHOTS
versions. They are unreleased, ongoing versions. The uploads of those versions have some extra files:
.
└── my-awesome-company
└── foobar-test-bananas
├── 4.2-SNAPSHOT
│ ├── foobar-test-bananas-4.2-20230306.162821-1.jar
│ ├── foobar-test-bananas-4.2-20230306.162821-1.pom
│ └── maven-metadata.xml
└── maven-metadata.xml
Notice that the file names contain a dynamic part which is made up from a timestamp (20230306.162821
) and an increment (1
). This is to make sure that the filenames are really unique. This dynamic part is the same for all the files of the same set.
If we upload a new iteration to version 4.2-SNAPSHOT
, we will have the following result:
.
└── my-awesome-company
└── foobar-test-bananas
├── 4.2-SNAPSHOT
│ ├── foobar-test-bananas-4.2-20230306.162821-1.jar
│ ├── foobar-test-bananas-4.2-20230306.162821-1.pom
│ ├── foobar-test-bananas-4.2-20230307.113057-2.jar
│ ├── foobar-test-bananas-4.2-20230307.113057-2.pom
│ └── maven-metadata.xml
└── maven-metadata.xml
- The dynamic part changed. It's a different timestamp (
20230307.113057
) and the increment is the next one available (2
)
On the other hand, the GitLab Maven package registry provides a feature simply called "duplicates". It allows users to decide if uploading files to an existing package is allowed or not.
This feature currently works by reading the file extension if the file that is being uploaded and checking the existing files. If the file extension already exists and duplicates are not allowed, the upload is not accepted.
🐛 The bug
The problem with the above (maven uploads and duplicates) is that users can choose to upload additional jar
files with a different "classifier". Example:
.
└── my-awesome-company
└── foobar-test-bananas
├── 4.3
│ ├── foobar-test-bananas-4.3.jar
│ ├── foobar-test-bananas-4.3-javadoc.jar
│ ├── foobar-test-bananas-4.3-source.jar
│ └── foobar-test-bananas-4.3.pom
└── maven-metadata.xml
Notice the additional jars with -javadoc
and -source
classifiers. Guess what happens when duplicates are not allowed? Yes, .jar
so the backend will consider the foobar-test-bananas-4.3-javadoc.jar
file as a duplicate of foobar-test-bananas-4.3.jar
but in reality, it's not the case.
This is exactly the typebug described in #325749 (closed).
🚑 Fixing the bug
Looking at the example above, it seems that this is trivial to solve: Oh yeah, just compare the full filename (foobar-test-bananas-4.3.jar
) with the existing ones and that's it.
Unfortunately, that's half of the solution due to the SNAPSHOT
versions. Remember the dynamic part? That part will make all filenames unique = we can't compare the full filename.
In other words, we need to divide the duplicates detection logic in two:
- For non
SNAPSHOT
versions: compare the uploaded filename with the existing ones. - For
SNAPSHOT
versions: strip the dynamic part from the filename, do the same for the existing ones and then compare them.
🤔 What does this MR do and why?
- Update the
spec/services/packages/maven/find_or_create_package_service_spec.rb
so that artifact classifiers are supported during the duplicates check. - Update the related specs (including the requests specs).
📺 Screenshots or screen recordings
I'm going to use a project in a group where duplicates are not allowed. Here is the pom.xml
that I'm going to use:
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>my.awesome.company</groupId>
<artifactId>my-package</artifactId>
<packaging>jar</packaging>
<version>2.0</version>
<name>my-package</name>
<url>http://maven.apache.org</url>
<properties>
<maven.compiler.source>7</maven.compiler.source>
<maven.compiler.target>7</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<repositories>
<repository>
<id>gl_pru</id>
<url>http://gdk.test:8000/api/v4/projects/23/packages/maven</url>
</repository>
</repositories>
<distributionManagement>
<repository>
<id>gl_pru</id>
<url>http://gdk.test:8000/api/v4/projects/23/packages/maven</url>
</repository>
<snapshotRepository>
<id>gl_pru</id>
<url>http://gdk.test:8000/api/v4/projects/23/packages/maven</url>
</snapshotRepository>
</distributionManagement>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.3</version>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Notice that the -javadocs
jar is going to be generated and uploaded.
For each use case below, I'm going to upload the same file twice.
1️⃣ With a non SNAPSHOT
version
💥 using master
$ mvn deploy -s settings.xml
[snip ... snip]
[INFO] --- deploy:3.0.0:deploy (default-deploy) @ my-package ---
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.1/my-package-2.1.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.1/my-package-2.1.pom (1.9 kB at 6.5 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.1/my-package-2.1.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.1/my-package-2.1.jar (2.4 kB at 14 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.1/my-package-2.1-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.248 s
[INFO] Finished at: 2023-03-07T10:17:27+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0:deploy (default-deploy) on project my-package: Failed to deploy artifacts: Could not transfer artifact my.awesome.company:my-package:jar:javadoc:2.1 from/to gl_pru (http://gdk.test:8000/api/v4/projects/23/packages/maven): status code: 400, reason phrase: Bad Request (400) -> [Help 1]
Upload rejected
That's because the -javadoc.jar
file is detected as a duplicate and since duplicates are not allowed, the backend rejects the upload.
🚑 using this MR
$ mvn deploy -s settings.xml
[snip ... snip]
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.pom (1.9 kB at 311 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.jar (2.4 kB at 15 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0-javadoc.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0-javadoc.jar (95 kB at 558 kB/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml (304 B at 746 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.231 s
[INFO] Finished at: 2023-03-07T10:10:45+01:00
[INFO] ------------------------------------------------------------------------
Upload is a success. -javadoc.jar
file was accepted too.
Let's upload a second time:
$ mvn deploy -s settings.xml
[snip ... snip]
[INFO] --- deploy:3.0.0:deploy (default-deploy) @ my-package ---
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.pom
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.jar
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.821 s
[INFO] Finished at: 2023-03-07T10:12:08+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0:deploy (default-deploy) on project my-package: Failed to deploy artifacts: Could not transfer artifact my.awesome.company:my-package:pom:2.0 from/to gl_pru (http://gdk.test:8000/api/v4/projects/23/packages/maven): status code: 400, reason phrase: Bad Request (400) -> [Help 1]
Upload is rejected because it's a duplicate and we don't allow duplicates
1️⃣ With a SNAPSHOT
version
💥 using master
$ mvn deploy -s settings.xml
[snip ... snip]
[INFO] --- deploy:3.0.0:deploy (default-deploy) @ my-package ---
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/my-package-3.1-20230307.091859-1.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/my-package-3.1-20230307.091859-1.pom (1.9 kB at 7.2 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/my-package-3.1-20230307.091859-1.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/my-package-3.1-20230307.091859-1.jar (2.4 kB at 15 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/my-package-3.1-20230307.091859-1-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.010 s
[INFO] Finished at: 2023-03-07T10:19:01+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0:deploy (default-deploy) on project my-package: Failed to deploy artifacts: Could not transfer artifact my.awesome.company:my-package:jar:javadoc:3.1-20230307.091859-1 from/to gl_pru (http://gdk.test:8000/api/v4/projects/23/packages/maven): status code: 400, reason phrase: Bad Request (400) -> [Help 1]
Upload rejected
Same -javadoc.jar
file is detected as a duplicate.
🚑 using this MR
$ mvn deploy -s settings.xml
[snip ... snip]
[INFO] --- deploy:3.0.0:deploy (default-deploy) @ my-package ---
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1.pom (1.9 kB at 11 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1.jar (2.4 kB at 10.0 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1-javadoc.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1-javadoc.jar (95 kB at 540 kB/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml (304 B at 3.5 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/maven-metadata.xml (984 B at 6.6 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml (342 B at 2.1 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.530 s
[INFO] Finished at: 2023-03-07T10:14:12+01:00
[INFO] ------------------------------------------------------------------------
Upload accepted
No issues with the -javadoc.jar
file.
Let's upload again:
$ mvn deploy -s settings.xml
[snip ... snip]
[INFO] --- deploy:3.0.0:deploy (default-deploy) @ my-package ---
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/maven-metadata.xml (984 B at 9.3 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091524-2.pom
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091524-2.jar
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091524-2-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.867 s
[INFO] Finished at: 2023-03-07T10:15:26+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0:deploy (default-deploy) on project my-package: Failed to deploy artifacts: Could not transfer artifact my.awesome.company:my-package:pom:3.0-20230307.091524-2 from/to gl_pru (http://gdk.test:8000/api/v4/projects/23/packages/maven): status code: 400, reason phrase: Bad Request (400) -> [Help 1]
Upload not accepted as it is detected as a duplicated
⚙ How to set up and validate locally
- Have a group, project and a PAT ready.
- Setup the maven authentication as described in https://docs.gitlab.com/ee/user/packages/maven_repository/#authenticate-to-the-package-registry.
- In the group settings, do not allow duplicates for the maven package registry. See https://docs.gitlab.com/ee/user/packages/maven_repository/#do-not-allow-duplicate-maven-packages.
You can now try the scenarios above.
🛵 MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.