Skip to content

Add the gitlab-experiment gem, with configuration

Jeremy Jackson requested to merge jj-add-gitlab-experiment into master

What does this MR do?

This MR adds a couple pretty impactful concepts, and so I've separated the commits to be a bit easier to get an overview from.

First, we've added a new "experiment" feature flag type, which will help us track experiments over their lifecycle and help the engineering team create the appropriate tracking issues for experiments.

Second, it introduces the gitlab-experiment gem, which makes running experiments easier, and provides the entire company a cohesive and well documented interface on how to introduce and code up our experiments. The objectives here are about allowing us to be consistent and iteratively migrate some of the underlying aspects of experimentation (flipper -> unleash potentially) without having to change the documented interface to them.

The gitlab-experiment gem ships with scientist under the hood, which also allows us to run low level benchmarking experiments.

Third, it adds a very simplistic null hypothesis experiment around issue creation. The objective here is to go through a rollout process, review data collection, and overall vet the concepts we're trying to employ around our experiment library.

The hypothesis of the experiment is that there should be no difference between our control and candidate results. The experiment (also considered an A/A experiment) is intended to be useful in building out a funnel view that outlines the behavior of going to the issues on a project, clicking new (< 5-10% the number of visits to index), and finally creating the new issue (80-90% the number of visits to new). We think our final data should generally show a large number of views on the issues list on projects, dropping down to a much smaller number for wanting to create a new issue, and then seeing some smaller level of drop off in actually creating the new issue.

It's worth noting that we're not tracking "actor" or user on the experiment at all, as we think a potentially more useful way to classify this might be on project and not on user.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

This doesn't add any obvious risks aside from adding a new initializer, so that should be called out. I don't see how this adds any risk, but I think it's important to consider carefully.

  • [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Issue links

Related to #246620 (closed) https://gitlab.com/gitlab-org/growth/experimentation/-/issues/3

Edited by Mayra Cabrera

Merge request reports