Rename use_expected_code to use_actuals
What does this merge request do and why?
- Rename
--post-transformation {use_expected => use_actuals}
. - Anticipating Duo Chat evaluations (https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/prompt-library/-/issues/85), there is no need to make the name of this post-transformation code completion specific.
How to set up and validate locally
- Run the pipeline with the post-transformation
GIT_CURRENT_COMMIT=$(git rev-parse --short HEAD)
TODAY=`date +'%Y-%m-%d'`
OUTPUT_TABLE="${TODAY}_${GIT_CURRENT_COMMIT}_bcardoso_actuals"
poetry run python promptlib/eval_codebase.py \
--runner DirectRunner \
--project unreview-poc-390200e5 \
--region us-central1 \
--temp_location "gs://unreview-dataflow/tmp/" \
--save_main_session \
--input-bq-table unreview-poc-390200e5:gl_gitlab_codebase.input_raw_v1 \
--output-bq-table "unreview-poc-390200e5:gl_gitlab_experiments.${OUTPUT_TABLE}" \
--throttle-sec 0.01 \
--num-samples-per-file 3 \
--model code-gecko@latest \
--language python \
--post-transformation use_actuals
- Validate that the average is 1.0
SELECT round(avg(similarity_score), 2) FROM `unreview-poc-390200e5.gl_gitlab_experiments.2023-10-02_ffc73b4_bcardoso_actuals`
# yields 1.0
Merge request checklist
-
I've ran the eval_codebase.py
pipeline to validate that nothing is broken. -
Tests added for new functionality. If not, please raise an issue to follow up. -
Documentation added/updated, if needed.
Edited by Bruno Cardoso