Skip to content

feat: add codesuggestions client for litellm

Pam Artiaga requested to merge pam/add-codesuggestions-litellm-client into main

What does this merge request do and why?

We needed to evaluate code suggestions using the litellm SDK for Codestral.

Submitting this as an MR in case this will be useful in future evaluations.

How to set up and validate locally

Run the following evaluation:

poetry run eli5 code-suggestions evaluate \
--experiment-prefix="experiment-123" \
--dataset=code-suggestions-input-testcases-v1 \
--limit=2 \
--source=litellm \
--model-name="codestral@2405" \
--model-provider=vertex_ai

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Pam Artiaga

Merge request reports