Support Merge Requests as context for Duo Chat responses
PLEASE PROVIDE FEEDBACK
Please, leave a comment in with your feedback on your experience with this feature. Please, give the comment a title and include the following information in the comment:
- The question(s) you asked
- The link to the MR if it is a public MR
- The answer you received (if it doesn't give away confidential information)
- How satisfied you where
This helps us understand how you use the feature and allows us to build up an eval data set of user questions to improve the feature. Your help is much appreciated.
Release notes
We have heard your feedback that you would like chat to be aware of merge requests. You can now ask questions about a merge request to quickly dig into it. While the summarize code changes feature in the MR user interface helps MR authors quickly populate the MR description, so that reviewers can get a general understanding what the MR is about. This new chat capability accelerates reviewers or authors that may have specific questions that go beyond a summary.
https://docs.gitlab.com/ee/user/gitlab_duo_chat/#the-context-chat-is-aware-of
Problem to solve
Currently, users of Duo Chat cannot ask questions about MRs (but they expect to be able to ask questions about things they have access to as explained in the parent epic). See the image below as an example of this situation. Some of these questions can be answered with other Duo features, however users would still expect that chat is also aware of the MR context.
Proposal
When a user is sitting on an MR page (any of the 4 MR tabs) in the GitLab UI. Duo Chat should have the ability to acquire the context of the MR (i.e. its contents) and use it to provide a sensible response to the prompt of the user.
The goal is to answer these questions raised in chat bashes and in the in-product feedback. A lot of questions are "explain this MR" or "summarize this MR". Let's assume both scenarios:
- When the user asks such a question, there is no description and there are no comments. Just code changes. There are two use cases for this:
- The first one is similar to (or the same as) is supported by generate-a-description-by-summarizing-code-changes: As the author of a change, I want to document the change so that others can review it.
- The second one is: I am a reviewer of an MR and there is no or a poor description in the MR, and I am looking for help from Duo to ramp up quickly on understanding the MR.
- When the user asks such a question there is a description and there are comments. As a reviewer of this MR, I am overwhelmed with the description and the many comments and I am looking for help from Duo to ramp up quickly on understanding the MR.
Engineering Notes:
- There is an MR that already implements this functionality via a new tool for a MergeRequestReader.
- The additional work is to prepare the necessary CEF test cases based on this simple data set.
- Optional alternative or addition in this iteration: This data set
👆 is rather basic. We can do better and extract questions from comments in MRs as proposed here. Eventually, we want to do this but it is not a must-have for this first iteration on bringing MR context to the chat. The developer on this issue can decide whether they want to do it for the initial iteration or not, depending work effort. (Note to the developer: at the time when this feature will likely be the implemented the groupai model validation has no capacity to support you on creating this more advanced dataset. But you can create it yourself.)
- Optional alternative or addition in this iteration: This data set
- Also run any local testing to reasonably conclude that the introduction of the new tool won't produce quality regression for the other types of questions
- This is an opportunity to additionally set up these quality evaluations (i.e. for this tool only) in the new LangSmith frameowork we are incorporating into our testing harness/workflows
- The implementation must be thoroughly tested to ensure the quality of the answers meets Chat's GA criteria (i.e. less than 20% of the answers shall be considered poor by an LLM judge (poor = rate 1 or 2 on a scale from 1 to 4).
- If by the time this is being implemented we have already resolved the other issue where Duo Chat tries to search for issues/epics when it actually can't, then we should test that it doesn't try to do the same with the MR reader tool.
- The MR should be updated (or a new MR) to update the relevant documentation areas to include the new MR reader functionality and how it can be used. Make it clear that this feature is currently free for premium and ultimate users but will require an Enterprise Duo add-on in the future.
- This introduces a new type of question in the CEF.
-
The new type of prompt should contribute to the product metrics' event category tracking in the usage dashboard (i.e. "Summarize Something" category).The new event should contribute to the general category pie chart as well as the detailed bar chart of types of things being summarized (i.e. new event "Summarize MR")
Pricing and Packaging: Duo Enterprise add-on required
This feature enables users to ask about MRs in chat. Many such questions can also be answered with the Duo MR features with a button click developed by the groupcode review. While the latter may turn out to be the preferred form of applying AI in the context of MRs users still expect the chat can also answer such MR-related questions.
The Duo MR features are packaged as Duo Enterprise features, we want to make sure the MR-related questions in chat are in the same add-on tier. Hence, the requirement for this feature here is that it adheres to the Duo Enterprise tier and is not available in the Duo Pro add-on.
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.