Skip to content

Wrap injected code blob inside XML tag

euko requested to merge 423419-questions-about-code-should-not-pick-documentation-tool into master

What does this MR do and why?

Using a XML tag helps LLM to recognize the tag's content as a distinct section within a prompt.

For code-related questions, the practice may help Anthropic Claude avoid choosing a wrong tool based on the content of the code.

For example, the code may reference "GitLab" in its comments. Without the tag there's a chance the injected code blob pollutes the overall context of a prompt.

How to set up and validate locally

*Assumes you have access to Anthropic Claude API

Run the updated specs:

REAL_AI_REQUEST=1 bundle exec rspec ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_real_requests_spec.rb:83 -fdoc

Revert the change to the executor and re-run the specs. The context 'with blob for code containing gitlab references' should fail by attempting pick up the documentation tool.

Unfortunately, the outcome is not deterministic and the test may continue to pass for some runs.

git checkout master -- ee/lib/gitlab/llm/chain/agents/zero_shot/executor.rb

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #423419 (closed)

Edited by euko

Merge request reports