Add streaming to Anthropic client
What does this MR do and why?
Add streaming to Anthropic client
This adds the option to stream data from Anthropic by calling the
stream
method. We added a new method instead of adding an option to
complete
because streaming does not work with the concept of
exponential backoff.
It also adds the anthropic-version
header with '2023-06-01' as
Anthropic changed the way streaming works and the previous version will
be deprecated. The old version accumulated the response in the stream,
while the new version only sends the next token.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- Make sure you have an Anthropic Token set in
ApplicationSettings
- Execute the following in the rails console:
Gitlab::Llm::Anthropic::Client.new(User.first).stream(prompt: "Write a hello world in ruby:") do |f| puts f["completion"] end
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.