Skip to content

Add streaming to Anthropic client

Nicolas Dular requested to merge nd/anthropic-streaming into master

What does this MR do and why?

Add streaming to Anthropic client

This adds the option to stream data from Anthropic by calling the stream method. We added a new method instead of adding an option to complete because streaming does not work with the concept of exponential backoff.

It also adds the anthropic-version header with '2023-06-01' as Anthropic changed the way streaming works and the previous version will be deprecated. The old version accumulated the response in the stream, while the new version only sends the next token.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. Make sure you have an Anthropic Token set in ApplicationSettings
  2. Execute the following in the rails console:
    Gitlab::Llm::Anthropic::Client.new(User.first).stream(prompt: "Write a hello world in ruby:") do |f|
       puts f["completion"]
    end

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Nicolas Dular

Merge request reports

Loading