Move search bar to top nav to test usage changes
Problem to solve
We have seen a huge drop in search usage since the new navigation was launched and, while the drop has stabilized, usage has not returned. We also now have command palette metrics that indicate users are not finding what they need from the command palette–they hardly ever click on results of any kind from the command palette.
Proposal
Test moving the search bar back into the header navigation and roll it out to a subset of users (TBD) on GitLab.com.
Success criteria
in the interest of keeping it simple, we can start with the following (for internal users only):
- Increasing the absolute daily count of
click_search_button_to_activate_command_palette
by ≥ 20 per day on average points. Today it is pretty consistently around 90 per day. I'd like to see that persist for at least a 30 day period - Increasing daily ratio of (
press_enter_to_advanced_search
count /click_search_button_to_activate_command_palette
count) by ≥ 10 percentage points. It's at roughly 23% today, so I'd like to see that number get to 33% on average over at least a 30 day period
The thinking is that we give this as try for internal users for 60 days and see how the metrics looked on a rolling 30 days basis during that time. If we're seeing meaningful improvement, even if those metrics just improve and stabilize, we would then gradually roll it out to a subset of users on dot-com.
Do we need a certain quantity of the user base to be exposed for statistical significance?
Yes. I did some calculating and came up with a sample size of 334, at 167 per group (assuming two groups). That should give the test a power of .85, p = 0.05
Is 30 days long enough?
It's probably the minimum, tbh, but I would expect to see some effect in that time period. Maybe 30 days is the cutoff if we're not seeing any effect, but if we're seeing good results, we keep running it.
What is the hypothesized impact to an increase or decrease to this metric?
My hypothesis is that an increase in these metrics indicates that users are again able to search, which, extrapolated out to all our users, should increase our CSAT score and improve subjective research findings about users' ability to find what they're looking for on GitLab. You probably know more than me about how our CSAT is calculated and if my hypothesis for impact there is a reasonable one.