Skip to content

Resolving docker authentication behaves randomly

Problem

When introducing !2048 (merged) we've changed the behavior of credentials resolving, which now can - in specific case - behave totally randomly.

Before the change credentials were resolved in strict order and first found - however it was defined - was returned:

func (e *executor) getAuthConfig(imageName string) *types.AuthConfig {
	indexName, _ := docker.SplitDockerImageName(imageName)

	resolvers := []authConfigResolver{
		e.getUserAuthConfiguration,
		e.getHomeDirAuthConfiguration,
		e.getBuildAuthConfiguration,
	}

	for _, resolver := range resolvers {
		source, authConfig := resolver(indexName)

		if authConfig != nil {
			e.Println("Authenticating with credentials from", source)
			e.Debugln("Using", authConfig.Username, "to connect to", authConfig.ServerAddress,
				"in order to resolve", imageName, "...")

			return authConfig
		}
	}

	e.Debugln(fmt.Sprintf("No credentials found for %v", indexName))

	return nil
}

Runner iterated over the list of defined resolvers in the defined order and if credentials for the given indexName were found, they were immediately returned. This respected the defined priority of credentials sources.

In the new version, it looks slightly different:

func ResolveConfigForImage(imageName, dockerAuthConfig, username string, credentials []common.Credentials) *RegistryInfo {
	authConfigs := ResolveConfigs(dockerAuthConfig, username, credentials)
	if authConfigs == nil {
		return nil
	}

	indexName, _ := splitDockerImageName(imageName)
	for registry, info := range authConfigs {
		if indexName == convertToHostname(registry) {
			return &info
		}
	}

	return nil
}

func ResolveConfigs(dockerAuthConfig, username string, credentials []common.Credentials) map[string]RegistryInfo {
	resolvers := []authConfigResolver{
		func() (string, map[string]types.AuthConfig) {
			return getUserConfiguration(dockerAuthConfig)
		},
		func() (string, map[string]types.AuthConfig) {
			return getHomeDirConfiguration(username)
		},
		func() (string, map[string]types.AuthConfig) {
			return getBuildConfiguration(credentials)
		},
	}
	res := make(map[string]RegistryInfo)

	for _, r := range resolvers {
		source, configs := r()
		for registry, conf := range configs {
			if _, ok := res[registry]; !ok {
				res[registry] = RegistryInfo{
					Source:     source,
					AuthConfig: conf,
				}
			}
		}
	}

	return res
}

Runner still iterates over the list of defined resolvers in the defined order. But instead of checking found credentials against of the searched indexName immediately, the credentials are added to a map indexed by registry ID. This is needed for Kubernetes executor, where we need to get all credentials for all registries at the beginning, so that information can be added to the Pod configuration. In Docker executor case Runner next iterates over all entries of this map and looks for the registry matching the indexName.

And here is the problem.

Registry name can be defined in two ways:

  • as a hostname:port pair, e.g. gitlab.example.com:5000
  • as an URL, e.g. https://gitlab.example.com:5000/v1

Both ways are valid and both can be found. For that we had before and we're still using the convertToHostname() method to get the canonical name of the registry. But in the new implementation, it's done in wrong place.

Let's take such user case:

A company hosts GitLab with images registry. There is a non-public project containing a definition of an image, that is used as a service in jobs in many other projects. Because the project is non-public, Docker needs credentials to be able to pull the image. Because the image may be used by people that should not have access to it's project (just be able to use the hosted image), the company added the credentials for the GitLab registry in /root/.docker/config.json at runner's host.

The credentials are there defined as auth:{"https://registry.gitlab.example.com/v1/":{...}}

Now, what will happen. GitLab will pass registry credentials automatically in job payload, defining the registry as registry.gitlab.example.com. Runner will iterate over the resolvers and first it will find the home directory one. It will add the registry to the map using https://registry.gitlab.com/v1/ as the index. Next it will find the credentials in job payload and will add them to the map using registry.gitlab.example.com as the index. Now our map contains two entries:

res := map[]stringRegistryInfo{
    "registry.gitlab.example.com": {...},
    "https://registry.gitlab.com/v1/": {...},
}

Because registry names used as map indexes are different, they were both added. The first one contains the job-unique credentials (which may not work if the user doesn't have access to the image project), second one contains the credentials prepared by the runner administrator.

Now, in ResolveConfigForImage() Runner iterates over this map to find the registry for given indexName. But! Accessing maps in Go is random! And both registry.gitlab.example.com and https://registry.gitlab.com/v1/ will be changed to registry.gitlab.example.com by convertToHostname()!

So at the end, we might get different credentials for the same image, and the choice will be done totally randomly!

Workaround

If you're seeing such behavior, make sure that the registry name in ~/.docker/config.json (or ~/.dockercfg) and the registry name sent with job payload are the same. And the value sent by GitLab should be in form of host[:port] where port is optional and is not present when it's 443 (HTTPS). So it could be registry.gitlab.com for GitLab.com where registry is accessible through HTTPS, or gitlab.example.com:5005 when registr for gitlab.example.com is enabled on a custom port.

Fix

diff --git a/helpers/docker/auth/auth.go b/helpers/docker/auth/auth.go
index aaf9e6279..1f3677a3e 100644
--- a/helpers/docker/auth/auth.go
+++ b/helpers/docker/auth/auth.go
@@ -51,13 +51,12 @@ func ResolveConfigForImage(
        }
 
        indexName, _ := splitDockerImageName(imageName)
-       for registry, info := range authConfigs {
-               if indexName == convertToHostname(registry) {
-                       return &info
-               }
+       info, ok := authConfigs[indexName]
+       if !ok {
+               return nil
        }
 
-       return nil
+       return &info
 }
 
 // ResolveConfigs returns the authentication configuration for docker registries.
@@ -83,8 +82,9 @@ func ResolveConfigs(dockerAuthConfig, username string, credentials []common.Cred
        for _, r := range resolvers {
                source, configs := r()
                for registry, conf := range configs {
-                       if _, ok := res[registry]; !ok {
-                               res[registry] = RegistryInfo{
+                       registryHostname := convertToHostname(registry)
+                       if _, ok := res[registryHostname]; !ok {
+                               res[registryHostname] = RegistryInfo{
                                        Source:     source,
                                        AuthConfig: conf,
                                }
diff --git a/helpers/docker/auth/auth_test.go b/helpers/docker/auth/auth_test.go
index 895702442..ec797cbad 100644
--- a/helpers/docker/auth/auth_test.go
+++ b/helpers/docker/auth/auth_test.go
@@ -12,6 +12,7 @@ import (
        "github.com/docker/docker/api/types"
        "github.com/stretchr/testify/assert"
        "github.com/stretchr/testify/require"
+
        "gitlab.com/gitlab-org/gitlab-runner/common"
 )
 
@@ -210,6 +211,42 @@ func TestGetConfigs(t *testing.T) {
        }, result)
 }
 
+func TestGetConfigs_DuplicatedRegistryCredentials(t *testing.T) {
+       registryCredentials := []common.Credentials{
+               {
+                       Type:     "registry",
+                       URL:      "registry.domain.tld:5005",
+                       Username: "test_user_1",
+                       Password: "test_password_1",
+               },
+       }
+
+       cleanup := setupTestHomeDirectoryConfig(t, true)
+       defer cleanup()
+       result := ResolveConfigs("", "", registryCredentials)
+
+       expectedResult := map[string]RegistryInfo{
+               "registry.domain.tld:5005": {
+                       Source: filepath.Join(HomeDirectory, ".dockercfg"),
+                       AuthConfig: types.AuthConfig{
+                               Username:      "test_user_1",
+                               Password:      "test_password_1",
+                               ServerAddress: "https://registry.domain.tld:5005/v1/",
+                       },
+               },
+               "registry2.domain.tld:5005": {
+                       Source: filepath.Join(HomeDirectory, ".dockercfg"),
+                       AuthConfig: types.AuthConfig{
+                               Username:      "test_user_2",
+                               Password:      "test_password_2",
+                               ServerAddress: "registry2.domain.tld:5005",
+                       },
+               },
+       }
+
+       assert.Equal(t, expectedResult, result)
+}
+
 func TestSplitDockerImageName(t *testing.T) {
        remote, image := splitDockerImageName("tutum.co/user/ubuntu")
        expectedRemote := "tutum.co"

We already have one user report mentioning such random credentials resolving. We don't have yet the information how the registry is referenced in config.json file on the host, but currently the above description is the only one that would explain what is happening.

Edited by Tomasz Maczukin