Resolve "Add used tools to completions rake task output file"
What does this MR do and why?
This MR updates the Duo Chat completions Rake task to include the tool used in the tasks' output. More detailed information can be found in Issue #443676 (closed)
The duo_chat::completions.rake tasks output is obtained from the output of the Gitlab::Duo::Chat::Request.completion method. Which consists of the following format:
{
ref: data_row.ref,
query: query,
resource: resource,
response: response.response_body
}
The idea behind this MR is to allow for the tasks to include the tool_used
within the output so we can ensure our IssueReader
and EpicReader
tools are being called at the proper times.
Closes #443676 (closed)
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- you need to have root group path set up - it needs to be on ultimate or premium plan (so you need to run your instance as SaaS) and it needs to have experiment and beta setting switched on
- you need to clone dataset repo: https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/datasets
- into the root group, you need to import data:
-zip archive with samples - https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/datasets/-/packages/22409459
- Zip archive:
- unzip manually, the root folder contains several other archives that can be imported using GitLab UI.
- Import the groups first, then import the projects
- You can import the exported groups to any root group, however you need to preserve the canonical names of the exported groups and projects.
- We use group and project names and iids to match the queries with resources in our dataset.
- Zip archive:
Then you can run rake task:
AIEF_DATASET=<DATASET_REPO>/duo_chat/v1/jsonl/queries \
AIEF_OUTPUT=duo_chat_completions GITLAB_SIMULATE_SAAS=true bundle exec rake "gitlab:duo_chat:completions[<root group path>, <user id>]"
When looking at the file generated by the rake tasks you should now see that it contains tool_used
and its output should either be nil
, Gitlab::Llm::Chain::Tools::IssueReader::Executor
, or Gitlab::Llm::Chain::Tools::EpicReader::Executor
Changes To Be Made
-
Add tools_used
method to Duo::Chat::Request -
Add tools_used
test cases for Duo::Chat::Request in Duo::Chat::Request spec file -
Update Duo::Chat::Request.completions hash output to include the tools_used
-
Verify updated hash with AI Model Validation Team - @tle_gitlab
-