Support prompt templates on eval pipeline

Overview

This MR is the first iteration of allowing requests to setup custom prompts. This lays the foundation for the prompt library to be used as a service.

What does this merge request do and why?

This MR allows evaluations prompt templates with code-suggestions eval pipeline.

• Add a new prompt_template_config argument to code-suggestions eval. This is the path the the prompt templates config file.

• Add an example set of prompts and config to data/prompts

  ❯ tree data/prompts/
  data/prompts/
  ├── code_suggestions_complex.txt.example
  ├── code_suggestions_simple.txt.example
  └── template_config.json.example

• Add a new step ApplyPromptTemplates that is included if the prompt_template_config argument is specified. This step will:

  • Parse and validate the prompt templates config file.
  • Apply input from the test case dataset to the template

How to set up and validate locally

1. Ensure GCP enviroment variables are setup.

2. Check out to this merge request's branch.

3. Copy the template_config.json.example file in the data/prompts/ folder

   cp data/prompts/template_config.json{.example,}

4. Run the follow command to kick off the pipeline.

   poetry run promptlib code-suggestions \
          --runner DirectRunner \
          --project unreview-poc-390200e5 \
          --region us-central1 \
          --temp-location "gs://unreview-dataflow/tmp/" \
          --save-main-session \
          eval \
          --input-bq-table unreview-poc-390200e5.gl_gitlab_codebase.input_testcases_v1 \
          --output-bq-table unreview-poc-390200e5:gl_gitlab_experiments.2023-11-14_wonderful_tests \
          --include-suffix \
          --throttle-sec 0.01 \
          --batch-size 24 \
          --min-length 25 \
          --language=rust \
          --prompt-template-config=data/prompts/template_config.json \
          --model code-gecko@latest

5. Navigate to BigQuery tables on GCP and confirm the outputs include the prompt templates 🎉

Merge request checklist

• I've ran the affected pipeline(s) to validate that nothing is broken. Pipeline(s) ran:
• Tests added for new functionality. If not, please raise an issue to follow up.
• Documentation added/updated, if needed.

Relates to https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/prompt-library/-/issues/93