Skip to content

Pin Vertex AI model versions

Tan Le requested to merge 383-pin-vertex-models into main

What does this merge request do and why?

Pin Vertex AI model versions.

code-bison and text-bison model no longer supports @latest versioning schema and cause 404 errors. This change pins all Vertex AI model versions to the latest known versions 002. This would also allow better predictability and reproducibility of latency and quality moving forward.

To see the latest model versions, please follow the official doc from Google.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Check out to this merge request's branch.
  2. Ensure a local Docker image built successfully.
    docker buildx build --platform linux/amd64 \
      -t ai-gateway:dev .
  3. Run a local service on Docker.
    docker run --platform linux/amd64 --rm \
      -p 5052:5052 \
      -e AUTH_BYPASS_EXTERNAL=true \
      -v $PWD:/app -it ai-gateway:dev
  4. Run a cURL command to test code completions
    curl --request POST \
      --url http://codesuggestions.gdk.test:5052/v2/completions \
      --header 'Content-Type: application/json' \
      --header 'X-Gitlab-Authentication-Type: oidc' \
      --data '{
      "prompt_version": 1,
      "project_path": "awesome_project",
      "project_id": 23,
      "current_file": {
        "file_name": "main.go",
        "content_above_cursor": "func (h* HotelHandler) HandleGetHotels(c *fiber.Ctx) error {\n\thotels, err := h.Store.Hotels.GetHotels(c.Context(), nil)\n\tif err != nil {\n\t\treturn err\n\t}\n\treturn",
        "content_below_cursor": "}"
      }
    }'
    
    {
      "id": "id",
      "model": {
        "engine": "vertex-ai",
        "name": "code-gecko@002",
        "lang": "go"
      },
      "experiments": [
        {
          "name": "exp_truncate_suffix",
          "variant": 0
        }
      ],
      "object": "text_completion",
      "created": 1702604943,
      "choices": [
        {
          "text": " c.JSON(hotels)\n",
          "index": 0,
          "finish_reason": "length"
        }
      ]
    }

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.

Closes #383

Edited by Tan Le

Merge request reports

Loading