Maven dependency proxy is not handling Basic Auth properly
💼 Summary
- The Maven dependency proxy is not handling Basic Auth properly. When an unauthenticated request is sent, maven clients will not receive the
401 Unauthorized
but403 Forbidden
. As such, maven clients will not retry the request with the credentials. - Confirmed happening for
$ mvn
(latest version). - Confirmed not happening for
$ gradle
(latest version). - A workaround is available: use custom headers for the authentication.
🔦 Finding the root cause
⚙ Setup
We're going to use a very simple setup where all dependencies will be pulled through the Maven dependency proxy using the $ mvn
client.
Here are the relevant files of the GitLab public project:
`pom.xml`
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.app</groupId>
<artifactId>my-app</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>gitlab-maven</id>
<url>${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/dependency_proxy/packages/maven</url>
</repository>
</repositories>
</project>
- Single dependency:
junit
`settings.xml`
<settings>
<mirrors>
<mirror>
<id>gitlab-maven</id>
<name>GitLab proxy of central repo</name>
<url>${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/dependency_proxy/packages/maven</url>
<mirrorOf>central</mirrorOf>
</mirror>
</mirrors>
<servers>
<server>
<id>gitlab-maven</id>
<username>gitlab-ci-token</username>
<password>${CI_JOB_TOKEN}</password>
<configuration>
<authenticationInfo>
<userName>gitlab-ci-token</userName>
<password>${CI_JOB_TOKEN}</password>
</authenticationInfo>
</configuration>
</server>
</servers>
</settings>
- Basic auth set up as documented in https://docs.gitlab.com/ee/user/packages/maven_repository/?tab=mvn#basic-http-authentication
- Mirror feature set up to make sure that maven central is not used directly, as documented in https://docs.gitlab.com/ee/user/packages/maven_repository/?tab=mvn#additional-configuration-for-mvn
.gitlab-ci.yml
test_maven:
image: maven:latest
script:
- mvn test -s settings.xml
Lastly, here are the dependency proxy setting we're using:
1️⃣ First run
Job log
Running with gitlab-runner 16.9.0 (656c1943)
on GDK local runner p_zL-mnD, system ID: r_YSZ3Q2HuIcag
Resolving secrets
00:00
Preparing the "docker" executor
00:01
Using Docker executor with image maven:latest ...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:000f3d1826225aa67765be0005a06c29882d902cacef4ced3cdeef8d35230a1f for maven:latest with digest maven@sha256:ef6c85125449082775e012b72b74c61aade088dc27dfb0be8cc12b85438b8b90 ...
Preparing environment
00:01
Running on runner-pzl-mnd-project-291-concurrent-0 via 732189845d2c...
Getting source from Git repository
00:00
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/root/zd-501034/.git/
Checking out d7b02ea8 as detached HEAD (ref is main)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:02
Using docker image sha256:000f3d1826225aa67765be0005a06c29882d902cacef4ced3cdeef8d35230a1f for maven:latest with digest maven@sha256:ef6c85125449082775e012b72b74c61aade088dc27dfb0be8cc12b85438b8b90 ...
$ mvn test -s settings.xml
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------< com.mycompany.app:my-app >----------------------
[INFO] Building my-app 1.0-SNAPSHOT
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from gitlab-maven: http://gdk.test:8000/api/v4/projects/291/dependency_proxy/packages/maven/org/apache/maven/plugins/maven-resources-plugin/3.3.1/maven-resources-plugin-3.3.1.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.197 s
[INFO] Finished at: 2024-02-23T16:17:03Z
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.apache.maven.plugins:maven-resources-plugin:3.3.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-resources-plugin:jar:3.3.1: The following artifacts could not be resolved: org.apache.maven.plugins:maven-resources-plugin:pom:3.3.1 (absent): Could not transfer artifact org.apache.maven.plugins:maven-resources-plugin:pom:3.3.1 from/to gitlab-maven (http://gdk.test:8000/api/v4/projects/291/dependency_proxy/packages/maven): status code: 403, reason phrase: Forbidden (403) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
ERROR: Job failed: exit code 1
The dependency proxy backend will require users to be able to read_package
on the dependency setting. That permission is granted to basically reporter+
users.
If the request is refused, then one probably explanation is that the user was not properly authenticated = no user "attached" to the request. The backend will consider this as an anonymous user = read_package
will not be granted. This ends up with the request being rejected.
2️⃣ Trying something else
To make sure that this is not a deep bug in the Maven dependency proxy, let's use the customer headers for authentication as described in https://docs.gitlab.com/ee/user/packages/maven_repository/?tab=mvn#custom-http-header.
`settings.xml`
<settings>
<servers>
<server>
<id>gitlab-maven</id>
<configuration>
<httpHeaders>
<property>
<name>Job-Token</name>
<value>${CI_JOB_TOKEN}</value>
</property>
</httpHeaders>
</configuration>
</server>
</servers>
<mirrors>
<mirror>
<id>gitlab-maven</id>
<name>GitLab proxy of central repo</name>
<url>${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/dependency_proxy/packages/maven</url>
<mirrorOf>central</mirrorOf>
</mirror>
</mirrors>
</settings>
Here is the job:
Job logs
Running with gitlab-runner 16.9.0 (656c1943)
on GDK local runner p_zL-mnD, system ID: r_7yjilNUjLzUa
Resolving secrets
00:00
Preparing the "docker" executor
00:01
Using Docker executor with image maven:3.6.3-jdk-11 ...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:de7948a941defda38829ce0d898e315dbc6226a3b9f5b39103c85deab3fe5fd0 for maven:3.6.3-jdk-11 with digest maven@sha256:1d29ccf46ef2a5e64f7de3d79a63f9bcffb4dc56be0ae3daed5ca5542b38aa2d ...
Preparing environment
00:01
Running on runner-pzl-mnd-project-291-concurrent-0 via 77407bdbc658...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/root/zd-501034/.git/
Checking out 98e9e1c8 as detached HEAD (ref is main)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:56
Using docker image sha256:de7948a941defda38829ce0d898e315dbc6226a3b9f5b39103c85deab3fe5fd0 for maven:3.6.3-jdk-11 with digest maven@sha256:1d29ccf46ef2a5e64f7de3d79a63f9bcffb4dc56be0ae3daed5ca5542b38aa2d ...
$ mvn test -s settings.xml
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------------< com.mycompany.app:my-app >----------------------
[INFO] Building my-app 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from gitlab-maven: http://gdk.test:8000/api/v4/projects/291/dependency_proxy/packages/maven/org/apache/maven/plugins/maven-resources-plugin/2.6/maven-resources-plugin-2.6.pom
[snip snip]
[INFO] No tests to run.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 55.324 s
[INFO] Finished at: 2024-02-23T16:00:04Z
[INFO] ------------------------------------------------------------------------
Job succeeded
Success
This shows us two things:
- We don't have a generalized
🐛 in the Maven dependency proxy. - We can workaround the situation with custom headers.
⛏ Digging deeper
Back to basic auth, we added some logs to the backend to show the request headers:
{"Host"=>"gdk.test:8000", "User-Agent"=>"Apache-Maven/3.6.3 (Java 11.0.10; Linux 6.5.0-15-generic)", "Accept-Encoding"=>"gzip,deflate", "Cache-Control"=>"no-cache", "Cache-Store"=>"no-store", "Gitlab-Workhorse"=>"11-10-0cfa69752d8-74ffd66ae-ee-254903-ga2cdd6d49faf", "Gitlab-Workhorse-Proxy-Start"=>"1708703135887959000", "Pragma"=>"no-cache", "X-Forwarded-For"=>"172.16.123.1", "X-Request-Id"=>"01HQBA9E4FSF5VAP9JR4V2WCJX", "X-Sendfile-Type"=>"X-Sendfile", "Version"=>"HTTP/1.1"}
What is missing?
The Authorization
header. That is the one that carries the basic auth credentials. This proves that the $ mvn
client is not sending the credentials at all.
This is very settings.xml
file.
⚡ Eureka!
Got a $ nuget
package manager where the first request is sent without credentials and the client requires to receive a 401 Unauthorized
response to send a second request with the credentials.
This was strange because the maven dependency proxy is set up to send such response.
403 Forbidden
response could be returned and this got me to this line. Indeed, that error response is not properly wrapped.
❓ One line fix?
I updated that line from forbidden!
to wrap_error_response { forbidden! }
so the backend has a chance to return the 401 Unauthorized
response.
Let's run the job again:
Job log
[snip snip]
[INFO] No tests to run.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 55.324 s
[INFO] Finished at: 2024-02-23T16:00:04Z
[INFO] ------------------------------------------------------------------------
Job succeeded
Success
Packages are properly created in the package registry (caching).
🤔 Checking $ gradle
Let's quickly check $ gradle
with
`build.gradle`
plugins {
id 'java'
id 'application'
id 'maven-publish'
}
repositories {
maven {
url "${System.getenv("CI_API_V4_URL")}/projects/${System.getenv("CI_PROJECT_ID")}/dependency_proxy/packages/maven"
name "GitLab"
allowInsecureProtocol = true
credentials(PasswordCredentials) {
username = 'gitlab-ci-token'
password = System.getenv("CI_JOB_TOKEN")
}
authentication {
basic(BasicAuthentication)
}
}
}
dependencies {
implementation 'junit:junit:4.12'
}
`settings.gradle`
rootProject.name = 'test'
`.gitlab-ci.yml`
test_gradle:
image: gradle:latest
script:
- gradle install
Let's see the job:
Job logs
Running with gitlab-runner 16.9.0 (656c1943)
on GDK local runner p_zL-mnD, system ID: r_YSZ3Q2HuIcag
Resolving secrets
00:00
Preparing the "docker" executor
00:01
Using Docker executor with image gradle:latest ...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:50b00513fa941de814bc4aa357d2fea12b35035ea9fc672f4f8ffddf87d11f24 for gradle:latest with digest gradle@sha256:d2c6d17a59bab04b9fa5f0b4d46399d5ef502ba0d64c7b368a65ac47201f6366 ...
Preparing environment
00:00
Running on runner-pzl-mnd-project-291-concurrent-0 via 732189845d2c...
Getting source from Git repository
00:00
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/root/zd-501034/.git/
Checking out 95e4bc91 as detached HEAD (ref is main)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:05
Using docker image sha256:50b00513fa941de814bc4aa357d2fea12b35035ea9fc672f4f8ffddf87d11f24 for gradle:latest with digest gradle@sha256:d2c6d17a59bab04b9fa5f0b4d46399d5ef502ba0d64c7b368a65ac47201f6366 ...
$ gradle install
Welcome to Gradle 8.6!
Here are the highlights of this release:
- Configurable encryption key for configuration cache
- Build init improvements
- Build authoring improvements
For more details see https://docs.gradle.org/8.6/release-notes.html
Starting a Gradle Daemon (subsequent builds will be faster)
> Task :compileJava NO-SOURCE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE
> Task :jar
> Task :startScripts
> Task :installDist
BUILD SUCCESSFUL in 4s
3 actionable tasks: 3 executed
Job succeeded
All good
🛠 The fix
Here are the proper fixes that we need:
- The
forbidden!
response from this line should be properly wrapped to return401 Unauthorized
.- The related spec should be updated.
- The documentation should be update to recommend custom for the
$ mvn
as this will trigger about 50% less network requests (1
for each dependency instead of2
).
Everything can be handled in the single MR.
Given that a workaround is available, I would categorize this as a typebug with severity3 and weight 1
.