Experiment with making the phone number field optional within SaaS trial sign up
Hypothesis
We believe that by making the phone number field optional in the SaaS trial sign up we will increase the trial creation rate and increase the total IACV generated from trials.
Business problem
The required phone number field is only as valuable as the data that it collects and based on our analysis 77.8% of the time it is not providing the business with any value due to fake phone number submissions while reducing the trial conversion rate by up to 5%.
Supporting data
It is a common industry understanding that requiring a phone number field during a form submission reduces the submission rate. Unbounce indicates that on average it reduces form conversions by 5% (source). By requiring a non-validated phone number field we're reducing our form conversion rate without the added benefit of a valid phone number for the sales team.
I analyzed the most recent 1,000 submissions to the trial form and found that 77.8% of the phone number submitted were fake. I defined "fake" in this analysis as any phone number with less than 7 digits or highly visible pattern i.e. 1234567 or 1231234 etc.
Expected outcome
By making the phone number field optional field we believe that we will increase the trial creation rate by 3% and generate more IACV than the control group.
Experiment design & implementation
We will run a 50/50 experiment where 50% of users (control group) will get the standard trial sign up form where the phone number field is required and 50% of users (experiment group) will get a trial sign up form where the phone number field will be optional.
We will measure the trial sign up rates between the control and experiment groups. We will then monitor the trial upgrade rates within 45 days of the trial start date for both the control and experiment groups.
Base | Variant |
---|---|
Current version of https://gitlab.com/-/trials/new | Variant with optional phone number field |
ICE score
Impact | Confidence | Ease | Score |
---|---|---|---|
8 | 7 | 9 | 8 |
Known assumptions
We don't know the actual increase in the trial creation rate we will see by making the phone number field optional, 3% was an estimate based on Unbounce analysis where they saw a 5% increase when the phone number field was removed.
While we know that 77.8% of the phone numbers submitted are fake. The remaining 22.2% could represent a meaningful amount of revenue by running this as a 50/50 experiment we will be able to understand the revenue impact.
This experiment is limited to the SaaS trial form and a second test would need to be run to validate if the results are consistent with the self-hosted trial form.
Results, lessons learned, next steps
(to be filled out when the experiment is completed and results are analyzed
Cleanup
- Cleanup the feature flag and specs !24449 (comment 327281459)