Skip to content

Define different service names per LLM client

Patrick Bajao requested to merge pb-circuitbreaker-service-names into master

What does this MR do and why?

Gitlab::Llm::Concerns::CircuitBreaker requires service_name to be defined.

Before this change, we are only using a single service_name and that means all client will use a single circuit. If a single provider fails and the circuit opens, all providers will be affected.

To prevent that, since we have different clients (e.g. OpenAI, Vertex, Anthropic), we define a specific service name per client.

This also includes a fix to ExponentialBackoff concern to raise the correct exception to avoid a NameError.

This is behind the circuit_breaker feature flag. See [Feature flag] Enable `circuit_breaker` feature... (#409619 - closed).

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Peter Leitzen

Merge request reports