Daily Run with a Subset of Data in Prod- RCA

🚀 Plan B: Running Daily Subset in Production

📋 Overview

As we progress through the staging environment (Use a non-prod environment for evaluating GitLa... (#346 - closed)), we aim to kick off Plan B by running a subset of daily runs in Production via the new RCA chat endpoint.

💡 Proposal

To accomplish this, we need to:

• 🔑 Generate PATs (Personal Access Tokens) for three users:

  • @rutshah
  • @carolinesimpson
  • @mfanGitLab

• 🔗 Integrate with the slash-troubleshoot endpoint (powered by Claude 3.5 Sonnet) !584 (merged)

• 🔄 Execute daily runs

• 🔄 Increase daily runs upto ~1400 prompts ( July 19th-Target date)

• 📊 Update the Daily Run Dashboard

• 🔍 Conduct spot check analysis and troubleshoot scores of 1 due to error limit

• 🔍 Conduct spot check analysis on prompt and foundational model to give a rough benchmark as well

📚 Further Details

RCA Dashboard: https://lookerstudio.google.com/reporting/e0af7354-fbbd-46db-a541-e791621b49d7