Investigate mismatched SCIM/SAML identities

Summary

SCIM and SAML identities have become mismatched; this may be related to changing from SAML to SCIM authentication.

Azure queries Gitlab on the externalID of the user; based on our reply Azure sends a POST. This POST contains a JSON object with user data, which is rejected with a 409 error. It's not clear why a User has already been taken error is thrown and the authentication is working normally despite this error.

The steps seen in the logs producing the error:

1. Azure does a provisioning sync.
2. Presumably, because SAML extern_uid is not found for the NameID sent by SCIM, it then moves to next step.
3. Azure sends a POST with user data (attempts to create user).
4. GitLab replies with a HTTP 409 response and Azure logs show User has already been taken error.
   • The 409 seems to make sense because the SCIM extern_uid does not match the identifier that Azure sends and can't create a new account because it already exists.
   • Note: The NameID matches the SAML extern_uid but not the SCIM identity tied to the user.

Steps to reproduce

The steps to reproduce are unclear. The flow we believe to be at issue is:

1. Initial SAML configuration with one provider and set of identifiers.
2. Generate a SCIM token.
3. GitLab turns on scim_identities in 2020-02 (?) which generated SCIM identities using SAML identities information.
4. Move to a different provider setting up SAML/SCIM with a different set of identifiers.

It's possible that a customer could end up in this scenario just by changing SAML/SCIM providers.

Example case

First surfaced in this ticket. (internal)

What is the current bug behavior?

With auto-provisioning on Kibana logs (internal) and provisioning logs for Azure show a lot of users failing with a 409 (conflict) error.

Relevant logs and/or screenshots

Kibana log (internal)

• 409 entry for single user as example: https://log.gprd.gitlab.net/goto/5578bf0b3ecac08feda60b940efa259c
• all recent scim requests for org: https://log.gprd.gitlab.net/goto/98d603a94a794488b3468de0390332d7

Example user throwing provisioning error:

SAML identity:

[ gprd ] production> pp u.identities
[#<Identity:0x00007f10976e65a0
  id: 2865780,
  extern_uid: "6fbad272-d5d1-4a46-b28e-5fc7ff93a666",
  provider: "group_saml",
  user_id: 0000000,
  created_at: Thu, 17 Sep 2020 19:06:54 UTC +00:00,
  updated_at: Thu, 17 Sep 2020 19:06:54 UTC +00:00,
  secondary_extern_uid: nil,
  saml_provider_id: 760>]

SCIM identity for same user and relevant group:

[#<ScimIdentity:0x00007f10a4d46280
  id: 6866,
  group_id: 0000000,
  user_id: 0000000,
  created_at: Mon, 10 Feb 2020 20:28:48 UTC +00:00,
  updated_at: Mon, 27 Apr 2020 23:35:34 UTC +00:00,
  active: true,
  extern_uid: "ad|KBRA|1996462a-bad0-47a2-9c6d-e42b4b51eb68">]

Output of checks

This bug happens on GitLab.com, GitLab Enterprise Edition 13.5.0-pre d04ab9f1bf3

Workaround

1. Check that the SAML and SCIM identities do not match, and that the SAML identity was created prior to SCIM.
2. Double check that the current SAML identity matches the NameID provided by customer.
3. Delete SCIM identity for user.