Skip to content

feat: introduce base evaluators to build custom ones

Alexander Chueshev requested to merge ac/base-evaluators into main

What does this merge request do and why?

This MR introduces base evaluators, which provide a foundation for easily building new custom evaluators, including those that use LLMs as judges.

This MR mainly extracts the base classes introduced in !108 (merged) into a subpackage common for all AI feature evaluation pipelines.

Please, check the updated docs for the additional details - https://gitlab.com/gitlab-org/ai-powered/eli5/-/blob/98545ad28bb24eb1b7000c971d2a01bc567d13a5/doc/evaluators/index.md.

This MR is the first step towards consolidating our evaluation pipelines in the ELI5 project.

Ref: #35

How to set up and validate locally

  1. Check out to this merge request's branch.
  2. Install dependencies.
    poetry run install
  3. Check the existing commands ELI5 provides:
    poetry run eli5 --help

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Alexander Chueshev

Merge request reports

Loading