Skip to content

Rename use_expected_code to use_actuals

Bruno Cardoso requested to merge bc/use-actuals into main

What does this merge request do and why?

How to set up and validate locally

  1. Run the pipeline with the post-transformation
  GIT_CURRENT_COMMIT=$(git rev-parse --short HEAD)
  TODAY=`date +'%Y-%m-%d'`

  OUTPUT_TABLE="${TODAY}_${GIT_CURRENT_COMMIT}_bcardoso_actuals"
  poetry run python promptlib/eval_codebase.py \
      --runner DirectRunner \
      --project unreview-poc-390200e5 \
      --region us-central1 \
      --temp_location "gs://unreview-dataflow/tmp/" \
      --save_main_session \
      --input-bq-table unreview-poc-390200e5:gl_gitlab_codebase.input_raw_v1 \
      --output-bq-table "unreview-poc-390200e5:gl_gitlab_experiments.${OUTPUT_TABLE}" \
      --throttle-sec 0.01 \
      --num-samples-per-file 3 \
      --model code-gecko@latest \
      --language python \
      --post-transformation use_actuals
  1. Validate that the average is 1.0
SELECT round(avg(similarity_score), 2) FROM `unreview-poc-390200e5.gl_gitlab_experiments.2023-10-02_ffc73b4_bcardoso_actuals`
# yields 1.0

Merge request checklist

  • I've ran the eval_codebase.py pipeline to validate that nothing is broken.
  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Bruno Cardoso

Merge request reports

Loading