Combine resource reader and identifier in one tool per resource
What does this MR do and why?
Goal
The aim of this pull request is to enhance the speed of our chat system when dealing with questions related to epics and issues.
Current Process
Up until now, answering questions about epics or issues involved using two separate tools:
- Issue/Epic Identifier: This tool helped identify the specific issue or epic.
- Resource Reader: The Resource Reader tool was used to serialize the resource into a format suitable for the large language model. Previously, this format was JSON, but it has now been changed to XML.
Previous Workflow
The previous workflow involved multiple back-and-forth interactions with the large language model:
- First, we would select the appropriate tool, either the issue or epic identifier.
- Then, we would query the large language model with the results from the identifier to determine which resource reader to use.
- Next, we would pass the identified issue or epic to the resource reader for serialization.
- Finally, we would feed the serialized issue or epic back into the large language model and wait for the final answer.
Proposed Workflow
With the changes introduced in this MR, we've streamlined the process by eliminating the middle step:
- Upon receiving the initial question from the customer, we immediately consult the large language model.
- The model indicates that we should use the epics/issue reader.
- The epics/issue reader identifies the resource (epic or issue) and serializes it instantly.
- We then input this serialized data directly into the large language model, which provides us with the final answer.
Pros and Cons
Pros: The primary advantage of this solution is its speed improvement. It significantly reduces the time taken to respond to queries about epics and issues.
Cons: One drawback is that when we initially designed the chat system, we anticipated having more tools related to epics and issues. The issue or epic identifier was intended to be a common tool for such tasks. Additionally, the resource reader could have been utilized for other models, such as merge requests. However, the current lack of additional tools and the observed latency in the chat system prompted us to prioritize speed over versatility. The tools we used previously are still in the codebase, we can start using them at any time.
- it's worth noting that by merging these tools, we're disregarding a specific use case present in the resource reader. This use case was originally designed to handle scenarios where resources were deeply nested in JSON hierarchies. However, given the substantial token window size in our current tools, this functionality is no longer relevant and can be considered dead code.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Recording comparing speed: internal only: https://drive.google.com/file/d/1Q7XagR3SbDiemY5yCgBvmmDdJdaHzn39/view?usp=drive_link
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
Ask question about epic or issue using reference to the resource (not "this issue" or "current epic"). Observe speed.