Fully support feature flag actors in Gitaly
What does this MR do and why?
Feature flags in Gitaly was in a weird state. GitLab Rails pre-evaluates the flags, then passes to Gitaly via gRPC metadata or indirectly through internal APIs and workhorse. When evaluating gRPC metadata, feature flags are not evaluated with any actors. In other words, all feature flags are global in the context of RPCs. In contrast, feature flags in other locations support actors partially—only Project actor is allowed. This inconsistency creates plenty of troubles while delivering Gitaly changes.
This commit enhances the feature flag situation by:
- Introduce Repository actor, dedicated for Gitaly. This actor uses a repository relative path as its unique identity.
project = Project.find_by_full_path("gitlab-org/gitlab")
Feature.enable(:gitaly_mep_mep, project.repository)
- Implement a collector in Gitaly client to collect possible actors for pre-evaluation
- Apply actor collector to Gitaly CommitService as a POC for this approach
What happens after this MR is merged? Properly nothing changes. At the time of writing, all feature flags used in Gitaly were either not enabled, or rolled out globally. In both cases, this change does not affect them. We'll observe the performance impact for a reasonable amount of time before moving forward and apply the new mechanism to other RPCs.
List of supported actor types
Actor type | Scope |
---|---|
User | Users who are performing the current action. The user is inferred from authentication info. Some examples: - Signed-in users if the action is performed on UI - Owners of SSH public keys used when cloning via SSH - Owners of Personal Tokens used when querying an API<br>- etc. |
Repository | Particular repositories managed by Gitaly |
Group | All project repositories under the scopes of a group |
When a flag is enabled for a percentage of actors using the following command, it means the flag impacts 25% of the repositories. If a RPC does not have a repository, 25% of users are influenced instead. This is a side effect when the flags are pre-evaluated in Rails side. We could not pick the target actors!
/chatops run feature set gitaly-mep-mep 25 --actors
Screenshots or screen recordings
N/A. This MR does not introduce a change on the UI.
How to set up and validate locally
This section defines various testing scenarios. The main goal is to ensure Gitaly receives the correct list of relative feature flags in gRPC metadata. Unfortunately, Gitaly does not support logging feature flags by default. Therefore, we must add featureflag.UnaryInterceptor
and featureflag.StreamInterceptor
to gitaly server, and rebuild Gitaly code base:
diff --git a/internal/gitaly/server/server.go b/internal/gitaly/server/server.go
index e62b4b99c..180b31d70 100644
--- a/internal/gitaly/server/server.go
+++ b/internal/gitaly/server/server.go
@@ -3,6 +3,7 @@ package server
import (
"crypto/tls"
"fmt"
+ "gitlab.com/gitlab-org/gitaly/v15/internal/middleware/featureflag"
"time"
grpcmw "github.com/grpc-ecosystem/go-grpc-middleware"
@@ -110,6 +111,7 @@ func New(
sentryhandler.StreamLogHandler,
cancelhandler.Stream, // Should be below LogHandler
auth.StreamServerInterceptor(cfg.Auth),
+ featureflag.StreamInterceptor,
}
unaryServerInterceptors := []grpc.UnaryServerInterceptor{
grpcmwtags.UnaryServerInterceptor(ctxTagOpts...),
@@ -126,6 +128,7 @@ func New(
sentryhandler.UnaryLogHandler,
cancelhandler.Unary, // Should be below LogHandler
auth.UnaryServerInterceptor(cfg.Auth),
+ featureflag.UnaryInterceptor,
}
// Should be below auth handler to prevent v2 hmac tokens from timing out while queued
for _, limitHandler := range limitHandlers {
Afterward, filter Gitaly logs to capture feature_flags
field in each RPC with the following command:
gdk tail gitaly | jq --stream --unbuffered --raw-input 'split("{")|.[1:]|join("{")|"{" + .|fromjson|{feature_flags, "grpc.request.fullMethod", "grpc.meta.client_name"}'
In the following scenarios, I'll use gitaly_mep_mep
dummy feature flag for demonstration.
Flag is off globally
Click to expand
Feature.disable(:gitaly_mep_mep)
- Clone via SSH:
git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
"grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}
- Clone via HTTP:
git clone http://127.0.0.1:3000/gitlab-org/gitlab-shell.git
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
"grpc.meta.client_name": "gitlab-workhorse"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
"grpc.meta.client_name": "gitlab-workhorse"
}
- Visit the repository on UI
http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/GetCommitSignatures",
"grpc.meta.client_name": "gitlab-web"
}
Flag is on globally
Click to expand
Feature.enable(:gitaly_mep_mep)
- Clone via SSH:
git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
"grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}
- Clone via HTTP:
git clone http://127.0.0.1:3000/gitlab-org/gitlab-shell.git
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
"grpc.meta.client_name": "gitlab-workhorse"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
"grpc.meta.client_name": "gitlab-workhorse"
}
- Visit the repository on UI
http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/GetCommitSignatures",
"grpc.meta.client_name": "gitlab-web"
}
gitlab-org/gitlab-shell
Flag is on for Click to expand
Feature.remove(:gitaly_mep_mep)
Feature.enable(:gitaly_mep_mep, Project.find_by_full_path("gitlab-org/gitlab-shell").repository)
- Clone via SSH:
git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
"grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}
- Clone another repo via SSH:
git clone ssh://git@127.0.0.1:2222/gitlab-org/Flight.git
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
"grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}
- Clone via HTTP:
git clone http://127.0.0.1:3000/gitlab-org/gitlab-shell.git
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
"grpc.meta.client_name": "gitlab-workhorse"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
"grpc.meta.client_name": "gitlab-workhorse"
}
- Clone another repository via HTTP:
git clone http://127.0.0.1:3000/gitlab-org/Flight.git
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
"grpc.meta.client_name": "gitlab-workhorse"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
"grpc.meta.client_name": "gitlab-workhorse"
}
- Visit the repository on UI
http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/GetCommitSignatures",
"grpc.meta.client_name": "gitlab-web"
}
- Visit another repository on UI
http://localhost:3000/gitlab-org/Flight/-/commits/main
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
"grpc.meta.client_name": "gitlab-web"
}
Flag is on for user Root
Click to expand
Feature.remove(:gitaly_mep_mep)
Feature.enable(:gitaly_mep_mep, User.find_by_username("root"))
- Clone via SSH with root's public key
git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
"grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}
- Clone via SSH with another use's public key
git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
"grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}
- Clone via HTTP; this clone could not be associated with any user:
git clone http://127.0.0.1:3000/gitlab-org/gitlab-shell.git
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
"grpc.meta.client_name": "gitlab-workhorse"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
"grpc.meta.client_name": "gitlab-workhorse"
}
- Clone via HTTP with root basic auth:
git clone http://root:password@127.0.0.1:3000/gitlab-org/gitlab-shell.git
. Note that we could not determine the user from any public repository clone.
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/InfoRefsUploadPack",
"grpc.meta.client_name": "gitlab-workhorse"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
"grpc.meta.client_name": "gitlab-workhorse"
}
- Visit the repository on UI using user Root credentials
http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/GetCommitSignatures",
"grpc.meta.client_name": "gitlab-web"
}
- Visit the repository on UI using another user credentials
http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/GetCommitSignatures",
"grpc.meta.client_name": "gitlab-web"
}
- Some more tests with Personal Snippets, Project Snippets, and Designs
gitlab-org
Flag is on for Group Click to expand
Feature.remove(:gitaly_mep_mep)
Feature.enable(:gitaly_mep_mep, Group.find_by_path("gitlab-org"))
- Clone via SSH
git clone ssh://git@127.0.0.1:2222/gitlab-org/gitlab-shell.git
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
"grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}
- Clone via SSH
git clone ssh://git@127.0.0.1:2222/gitlab-org/Flight.git
{
"feature_flags": "true",
"grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
"grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}
- Clone via SSH
git clone ssh://git@127.0.0.1:2222/root/tada.git
{
"feature_flags": "false",
"grpc.request.fullMethod": "/gitaly.SSHService/SSHUploadPack",
"grpc.meta.client_name": "gitlab-sshd-git-upload-pack"
}
-
Repeat similar scenarios with clone via HTTP
-
Visit the repository on UI
http://localhost:3000/gitlab-org/gitlab-shell/-/commits/main
andhttp://localhost:3000/gitlab-org/Flight/-/commits/main
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
"grpc.meta.client_name": "gitlab-web"
}
- Visit the repository on UI
http://localhost:3000/root/tada/-/commits/main
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommits",
"grpc.meta.client_name": "gitlab-web"
}
- Visit Project wiki
http://localhost:3000/gitlab-org/Flight/-/wikis/home
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/TreeEntry",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
"grpc.meta.client_name": "gitlab-web"
}
- Visit Project wiki of another org
http://localhost:3000/flightjs/test-wiki-project/-/wikis/home
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/TreeEntry",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/FindCommit",
"grpc.meta.client_name": "gitlab-web"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/LastCommitForPath",
"grpc.meta.client_name": "gitlab-web"
}
Rollout to 25% of the repositories
Click to expand
It's time-consuming to test this style of rollout, I'll write a small script instead.
Feature.remove(:gitaly_mep_mep)
Feature.enable_percentage_of_actors(:gitaly_mep_mep, 25)
repositories = [
Project.all.map(&:repository),
Project.all.map { |p| p.wiki.repository },
Group.all.map { |g| g.wiki.repository },
Snippet.all.map(&:repository)
].flatten.compact.shuffle
repositories.each do |repository|
begin
repository.ls_files("main")
rescue; end
end
Running the above scripts and analyzing the logs, all the logs have mep_mep
feature flags. The number of mep_mep:true
is 11/34. As the sample size is too small, I did not expect the enable count is equal to 25%. The result is close enough. In fact, it proves that the feature flag works for all repository types.
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
"grpc.meta.client_name": "gitlab-web",
"grpc.request.glRepository": "wiki-8"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
"grpc.meta.client_name": "gitlab-web",
"grpc.request.glRepository": "project-14"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
"grpc.meta.client_name": "gitlab-web",
"grpc.request.glRepository": "project-4"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
"grpc.meta.client_name": "gitlab-web",
"grpc.request.glRepository": "group-25-wiki"
}
{
"feature_flags": "mep_mep:true",
"grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
"grpc.meta.client_name": "gitlab-web",
"grpc.request.glRepository": "wiki-6"
}
{
"feature_flags": "mep_mep:false",
"grpc.request.fullMethod": "/gitaly.CommitService/ListFiles",
"grpc.meta.client_name": "gitlab-web",
"grpc.request.glRepository": "snippet-4"
}
...
---
Total: 34
Enabled: 11
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.