Expose a portion of free .com users to the new purchase flow

Experiment Summary

By decreasing the required steps and providing a more intuitive interface we can increase the purchase rate for .com group purchases.

Hypothesis

We will see an increase in the successful purchase rate by providing users with a more intuitive checkout experience.

Supporting Data

This mural documents the current checkout experience which requires a total of 3 screens and 6 user actions between inputs and required scrolling before they enter their credit card information.

The new checkout funnel we will be testing consists of 1 screen and 3 inputs prior to address/credit card details (see existing user portion of this mural for designs).

Experiment Design

• Users to target: free group owners
• Page to target for the experiment: Group billing page
• Roll out: We will start with a roll-out of 10% of eligible free group owners and slowly increase to 50%
• If a user is in the experiment group then we should prefill the group upgrade dropdown in the checkout flow with the name of the group from which they clicked the upgrade button. For example, if I'm a member of 3 groups, one with the name XYZ. If I click "upgrade" from the group billing page of group XYZ then in the checkout funnel the question regarding where the subscription should be applied (question titled: "GitLab Grou") should be prefilled with my group XYZ.

The control free group owners that select to upgrade on a group billing page will be sent to the existing checkout funnel. The experiment free group owners that select to upgrade on a group billing page will be sent to the new checkout funnel design by the groupacquisition team (linked below).

This experiment will not require any design work as the new checkout funnel we will be testing is already live.

This issue by the groupacquisition team covers the part of the new checkout funnel creation where they added the option to select which group the user wants to upgrade which is required for this experiment.

The URL we will utilize in the experiment version to link existing free owners to the new checkout funnel is: https://gitlab.com/-/subscriptions/new

Experiment tracking

In order to successfully track this experiment, we will need to be able to know what bucket a free group owner is in. We will also need to be able to track the conversion rate for both the control from the point of upgrade click to check out complete. If we're unable to track this at the user level then we can run the experiment at the namespace level and track the click to checkout complete rate for the namespace.

Estimated Experiment Length and Impact

Over the last three months, we've averaged 330 subscriptions purchased per month. If we want to reach significance at 95% confidence level with 5% margin of error then we need 178 upgrades in our experiment group. If our experiment is running at 50% of total upgrades this means the experiment will need to run for ~32 days.

If we were to achieve a 5% absolute increase in the checkout completion rate in the experiment group this would represent ~$60,000 in IACV.