feat: add codesuggestions client for litellm
What does this merge request do and why?
We needed to evaluate code suggestions using the litellm
SDK for Codestral.
Submitting this as an MR in case this will be useful in future evaluations.
How to set up and validate locally
Run the following evaluation:
poetry run eli5 code-suggestions evaluate \
--experiment-prefix="experiment-123" \
--dataset=code-suggestions-input-testcases-v1 \
--limit=2 \
--source=litellm \
--model-name="codestral@2405" \
--model-provider=vertex_ai
Merge request checklist
-
Tests added for new functionality. If not, please raise an issue to follow up. -
Documentation added/updated, if needed.
Edited by Pam Artiaga