Move Pages Menu entry under "Deploy"

This issue started as an experiment issue.

Experiment summary

We believe that the GitLab Pages Menu entry is not correctly positioned under Project Settings, leading to discoverability issues. It should correctly be placed under the "Deployments" tab.

To verify that, we will temporarily move the GitLab Pages Menu entry to "Deployments".

And we’ll measure the impact by comparing the page open count between two groups of the same size, one with the menu entry traditionally in Settings and one with the new position. Page open events will be tracked with Snowplow and compared in Sisense.

Hypothesis

The GitLab Pages menu entry being under Settings may be confusing. How is it a Setting? Under the assumption that Pages is only an add on, to an existing code repo, eg. a Documentation website, it makes sense. But that is not what Pages is (see the Business Problem Paragraph).

For standalone apps, such as Frontend apps (React or Vue SPAs, Next.js Apps...), the association is not there. Pages is itself the ultimate deployment method.

Positioning it under "Deployments" will lead to a better clarity about the nature of the feature, it will also lead to more people being able to find it.

Business problem

GitLab Pages is a very useful tool for static site deployments. Originally, we had adapted the functionality after GitHub introduced a similar functionality and thus placed its menu entry in a similar place. GitHub Pages is mostly used to create a documentation site for the code hosted in the same repo. By positioning GitLab Pages under "Settings" like GitHub does it, we're suggesting the same usecase. However, GitLab's feature is much broader, as it can not only display html and md, but build with any Static Site Generator. So it is more comparable with Vercel or Netlify than with GitHub. By positioning it under Deployments we can strengthen the Product's Market position.

Supporting data

The biggest GitLab Pages users on gitlab.com are Software repositiores using Pages for documentation, supporting the Hypothesis stated under Business Problem.

Expected outcome

More People are able to find GitLab Pages and start using it.

Experiment design & implementation

We will test a new Menu position during a one-month period between two groups of the same size.

Due to GitLab Pages' currently low use compared to absolute user count, it's recommendable that the experiment is run on the entire user-base, so all users will be in either test- or control group. As FF actor we should use the user. It's less prone to confusion if there is consistent behaviour for one user between projects, as opposed to the other way around.

This means:

• Test Group: 50% of users
• Control Group: 50% of users
• Experiment duration: 2-3 Months

Result tracking:

• The results will be tracked in the Sisense Pages Dashboard
• Compared numbers are:
  • Pages Onboarding screen initial navigation clicks (this means that users have navigated beyond the first Pages setup step, meaning "accidental" navigations to Pages are not counted)
  • New Pages deployment commits

ICE score

Impact	Confidence	Ease	Score
7	8	9	8

Known assumptions

Results, lessons learned, next steps

The results support the Hypothesis. That said, customers have learned the position of Pages, so they will need to be supported learning the new position.

Initially this experiment was published with the information about the menu change in the documentation only. While a significant amount of users were able to find Pages, the control group did open Pages more often than the experiment group.

After this effect stabilised, a callout was added to the Settings page for the Experiment group during the experiment.

This resulted in a significant change in the experiment results. Now more people were able to find Pages if it was located under Deployments.

If we look at the initial navigation clicks (indicating the user starts exploring the Pages deploy process), we see an increase of 4% (552 in the experiment group vs 530 in the control group).

The more interesting data point is the commit step. This can be used as a vector for new Pages deployments. Here the effect is even stronger: we were able to create an increase by about 9% (337 in the experiment group vs. 310 in the control group).

(Note that the absolute numbers are only comparable between the respective experiment/control groups, not between the with/without callout charts because the deployment without callout was live for a longer period than the one with the callout)

In addition to the experiment, I conducted a Survey with a group of Frontend Engineers (n=18, 29.4% have not used GitLab before). I presented the entire sidebar for a Project and asked where they would expect GitLab Pages to be. In accordance to my proposition, 60% of respondents would expect Pages to be located under "Deployments".

Learnings

• More Users are able to find GitLab Pages when located under "Deployments"
• About two thirds of Users intuitively look for a Pages functionality under "Deployments"
• Users have learned the position of Pages under Settings, so they must be supported when the transition happens. A callout has proved to be effective.

Next steps

Make this change permanent.

Checklist

• Fill in the experiment summary and write more about the details of the experiment in the rest of the issue description. Some of these may be filled in through time (the "Result, learnings, next steps" section for example) but at least the experiment summary should be filled in right from the start.
• Add the label of the group:: that will work on this experiment (if known).
• Mention the Product Manager, Engineering Manager, and at least one Product Designer from the group that owns the part of the product that the experiment will affect.
• Fill in the values in the ICE score table ping other team members for the values you aren’t confident about (i.e. engineering should almost always fill out the ease section). Add the ~"ICE Score Needed" label to indicate that the score is incomplete.
• Replace the ~"ICE Score Needed" with an ICE low/medium/high score label once all values in the ICE table have been added.
• Mention the [at]gitlab-core-team team and ask for their feedback.

cc @ericschurter @oregand