Experiment Recording and Control/Experiment Bug Investigation
Issue
We have trial registrations (experiment group) for people in the control
group.
While monitoring the trial_during_signup
experiment, there seems to be something wrong with the assignment of people to the control
vs experiment
groups and/or context
.
We have trial registrations (experiment group) for people in the control
group, which is not acceptable.
gitlabhq_production=> select group_type, context, count(*) from experiment_users where experiment_id=42 group by 1, 2; group_type | context | count
------------+-------------------------+-------
1 | {} | 533
1 | {"trial_chosen": true} | 7
1 | {"trial_chosen": false} | 304
0 | {} | 516
0 | {"trial_chosen": true} | 8
0 | {"trial_chosen": false} | 290
Root cause
- The
tracking_group
is calculated byrecord_experiment_user
is for thecurrent_user
subject, whereas the one calculated by theexperiment_enabled?
is calculated for the cookie subject.
Affected experiments
Any experiments since 2020 December 3 after the MR got deployed
- cookie-based rollout,
- calling
record_experiment_user
- are recording the
group type
(experiment vs control).
Any experiment doing the below without subject:
experiment_enabled?(:foo)
record_experiment_user(:foo)
Solution
I made a draft MR suggestion to fix it
Verification
-
verify the bug, -
root cause, -
impact and -
the solution approach suggested in !51468 (merged)?
Edited by Alper Akgun