Skip to content

Draft: Add Secret Detection feature as a global pre-receive hook

Yuanchen Lu requested to merge yuanchenlu/gitlab-ee:v15.9.8-ee into 15-9-stable-ee

What does this MR do and why?

This MR adds the capability to preemptively detect secret leaks before they are committed to the remote repo using a global server side git hook.

To start, a project setting checkbox is provided to allow opt-in and opt-out. By default, all projects would opt in unless a project admin chooses to opt out.

The secret detection module used here is https://github.com/Yelp/detect-secrets with a list of plugins to choose from. We can customize by providing a template and add capability to choose what plugin to use on a per project basis.

I realize in the discussions that server hooks have performance issue so this feature is intended to use for self-hosted gitlab instance only that won't have a high rate of git operations. A feature flag will need to added to enable this feature.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests. Project settings include a new checkbox defaulting to checked for newly created projects: image

If a secret leak is detected, push will fail with the following message on the CLI (note I don't have the code for rendering UI message yet):

Detect secrets...........................................................Failed
- hook id: detect-secrets
- exit code: 1

Potential secrets about to be committed to git repo! Please rectify.

Secret Type: JSON Web Token
Location:    jwt:1

Possible mitigations:

  - Mark false positives with an inline `pragma: allowlist secret`
    comment
  - Commit with `--no-verify` if this is a one-time false positive

If a secret has already been committed, visit
https://help.github.com/articles/removing-sensitive-data-from-a-
repository
Before After

How to set up and validate locally

  1. To set up this feature, a server side hook is required. For self hosted gitlab in k8s, I installed the pre-receive hook to Gitaly's /home/git/custom_hooks/pre-receive.d/.

The following script is what I have:

#!/usr/bin/env bash

INSTALL_PYTHON=/usr/bin/python3
PROJECT_ID=`echo ${GL_REPOSITORY} | sed 's/[^0-9]*//g'`
RUN_HOOK=`curl -s -m 2 "<GITLAB_HOSTNAME>/api/v4/projects/${PROJECT_ID}/custom_attributes/push_protection_enabled" \
    -H 'PRIVATE-TOKEN: <ADMIN_TOKEN>' | jq -r '.value'`

# skip secret detection if curl times out
if [ $RUN_HOOK = "1" ]; then
    ARGS=(hook-impl --config=/home/git/custom_hooks/.pre-commit-config.yaml --hook-type=pre-commit)
    HERE="$(cd "$(dirname "$0")" && pwd)"
    ARGS+=(--hook-dir "$HERE" -- "$@")

    if [ -x "$INSTALL_PYTHON" ]; then
        exec "$INSTALL_PYTHON" -mpre_commit "${ARGS[@]}"
    elif command -v pre-commit > /dev/null; then
        exec pre-commit "${ARGS[@]}"
    else
        echo '`pre-commit` not found.  Did you forget to activate your virtualenv?' 1>&2
        exit 1
    fi
else
    exit 0
fi
  1. create a new project and navigate to its project setting to make sure the checkbox is checked
  2. push a commit containing a secret and observe the operation fail
  3. push a commit not containing a secret and observe the operation pass

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports