Errors returned by Git process are not captured for SSH operations
Issue
SSH operations that cause the git
process spawned to fail are logged as successful by Gitaly. Users will receive the error message, but there is no indication of a failure in the logs and the grpc_server_handled_total
prometheus metric will increment counter for the OK
code.
Git failures that occur on HTTP requests are correctly logged.
This prevents admins from understanding the health of their instance as failing git operations cannot be identified without user complaints.
Steps to reproduce
On version 14.3.3:
- Create new project and clone locally using SSH
- Connect to Gitaly server and run
chmod 0700 <REPO_PATH/refs> && chown root <repo_path/refs>
- Run
git ls-remote
orgit pull
on local machine, error below is returned - The child
git
process has exited with code 128 and written to stderr - Check
/var/log/gitlab/gitaly/current
, event is logged as successful
Error received by user
fatal: '/var/opt/gitlab/git-data/repositories/@hashed/26/d2/26d228663f13a88592a12d16cf9587caab0388b262d6d9f126ed62f9333aca94.git' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Log event
{
"command.count": 1,
"command.inblock": 0,
"command.majflt": 0,
"command.maxrss": 192328,
"command.minflt": 118,
"command.oublock": 0,
"command.real_time_ms": 7,
"command.system_time_ms": 2,
"command.user_time_ms": 2,
"correlation_id": "01FJHQZHS3FARG5NZG76CZ3GRJ",
"grpc.code": "OK",
"grpc.meta.auth_version": "v2",
"grpc.meta.client_name": "gitlab-shell-git-upload-pack",
"grpc.meta.deadline_type": "none",
"grpc.meta.method_type": "bidi_stream",
"grpc.method": "SSHUploadPack",
"grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
"grpc.request.glProjectPath": "root/gitaly-mirror",
"grpc.request.glRepository": "project-400",
"grpc.request.repoPath": "@hashed/26/d2/26d228663f13a88592a12d16cf9587caab0388b262d6d9f126ed62f9333aca94.git",
"grpc.request.repoStorage": "default",
"grpc.service": "gitaly.SSHService",
"grpc.start_time": "2021-10-21T15:37:12.815Z",
"grpc.time_ms": 8.291,
"level": "info",
"msg": "finished streaming call with code OK",
"peer.address": "10.150.0.55:39144",
"pid": 1877,
"remote_ip": "100.0.0.1",
"span.kind": "server",
"system": "grpc",
"time": "2021-10-21T15:37:12.823Z",
"user_id": "user-1",
"username": "root"
}
Edited by Will Chandler (ex-GitLab)