Modify ETV to use anthropic body streaming to fix timeout problems
What does this MR do and why?
During limited rollout of the Anthropic client for Explain This Vulnerability for testing purposes, it was discovered that it seemed to not work at all in Staging or Production. This was later established to have been caused by Net::ReadTimeout
errors being raised due to the api taking too long to service the request.
This MR uses the new body streaming mechanics established by the AI framework team to collect the full response before processing, avoiding the timeout issue, and also opening the possibility to UI response streaming if desired.
If we do somehow still have timeout issues, a more specific failure message is presented in the response to simplify diagnosis.
How to set up and validate locally
Assuming a local environment with SAST vulnerabilities and the ability to execute Anthropic client requests, your requests should still work, but should now no longer timeout (though they may take a very very long time.)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/419597