Frontend: Experiment Implementation: "Performance Insights" Button
Experiment Summary
This tests if adding a button which opens a (modal/sidebar) with a few stats on it about the pipeline timing results in users looking at the data.
From this idea in the team test discussion issue.
Experiment design/measures of success
- This will run for 2 months/milestones for all users
- We expect at least 5% of active monthly users who visit the page to click the button
- We expect 25% of users who click the button for details to click further into a job
- We expect to collect 10 or more comments from users the feedback issue over the 2 months.
- We will utilize social media and the Field Team to make users aware who have previously requested pipeline timing data.
- Feedback will be collected in a feedback issue
Design
- Add a button with title "Performance Statistics" to the pipeline detail page that opens a (modal/sidebar). (see below)
- Details should show some/all/what we can get added within the implementation time of the following ranked in order of preference.
- Top 5 slowest jobs in the pipeline (using available job
duration
) - - Longest wait for the job - shows how long last job took before it was executed (e.g. waiting for jobs in previous stages to finish) -
started_at - created_at
- A tooltip/explanation of how longest wait and longest queueing time is calculated
- A link to feedback issue - TBD
- Slowest job per stage (using job
duration
) - Longest queueing time - job that spent the longest in
pending
state waiting to be picked by a Runner (using available jobqueued_duration
)
- Top 5 slowest jobs in the pipeline (using available job
Rollout strategy
This is not an A/B test and can be rolled out to all users using our normal Feature Flag rolloutstrategy.
Tracking Details
- Track how often the modal is opened.
- Track how often a detail is clicked
Open Questions
-
Which is faster implementation modal or sidebar for the data? - Design is modal which Dev agreed was doable.
-
Can we link to job details? - Yes the modal links to job details for listed jobs
-
How are we tracking this / how will we report it? - Snowplow tracking (
click_insights_button
action,performance_insights
label)
- Snowplow tracking (
-
What is the criteria for success/we should prioritize making this an MVC and more performant? - Noted in the experiment design.
Edited by James Heimbuck