Support model name on /v2/code/generations (!523) · Merge requests · GitLab.org / ModelOps / AI Assisted (formerly Applied ML) / Code Suggestions / AI Gateway

Tan Le requested to merge support-model-name-params into main Dec 21, 2023

What does this merge request do and why?

Support model name on /v2/code/generations. This change is backward compatible and can be deployed independently with the change in Rails client.

This adds support for model_name parameter on /v2/code/generations endpoint. This parameter is optional. If it is not provided, we default to claude-2.0 if model_provider is anthropic.

Please note that model_name is ignored if model_provider is vertex-ai.

How to set up and validate locally

Check out to this merge request's branch.

Ensure a local Docker image built successfully.

docker buildx build --platform linux/amd64 \
  -t ai-gateway:dev .

Run a local service on Docker.

docker run --platform linux/amd64 --rm \
  -p 5052:5052 \
  -e AUTH_BYPASS_EXTERNAL=true \
  -v $PWD:/app -it ai-gateway:dev

Run a cURL command to test code generations

Code Generations - Default

curl --request POST \
  --url http://codesuggestions.gdk.test:5999/v2/code/generations \
  --header 'Content-Type: application/json' \
  --header 'X-Gitlab-Authentication-Type: oidc' \
  --data '{
  "current_file": {
    "file_name": "main.go",
    "content_above_cursor": "# Hello world",
    "content_below_cursor": "\n"
  },
  "prompt_version": 2,
  "prompt": "Human: You are a coding autocomplete agent. We want to generate new Go code inside the\nfile '\''hello.go'\'' based on instructions from the user.\nThe existing code is provided in <existing_code></existing_code> tags.\nThe new code you will generate will start at the position of the cursor, which is currently indicated by the <cursor> XML tag.\nIn your process, first, review the existing code to understand its logic and format. Then, try to determine the most\nlikely new code to generate at the cursor position to fulfill the instructions.\n\nWhen generating the new code, please ensure the following:\n1. It is valid Go code.\n2. It matches the existing code'\''s variable, parameter and function names.\n3. It does not repeat any existing code. Do not repeat code that comes before or after the cursor tags. This includes cases where the cursor is in the middle of a word.\n4. If the cursor is in the middle of a word, it finishes the word instead of repeating code before the cursor tag.\n\nReturn new code enclosed in <new_code></new_code> tags. We will then insert this at the <cursor> position.\nIf you are not able to write code based on the given instructions return an empty result like <new_code></new_code>.\n\nHere are a few examples of successfully generated code by other autocomplete agents:\n\n<examples>\n\n  <example>\n    H: <existing_code>\n         package main\nimport (\n  \"net/http\"\n)\nfunc main() {\n  err := http.ListenAndServe(httpPort, nil)\n  if err != nil { panic(err) }\n}\n// create an HTTP handler that fetches the current user preferences and returns them as JSON\n<cursor>\n\n// function to print the current user'\''s name\n       </existing_code>\n\n    A: <new_code>func userPrefsHandler(w http.ResponseWriter, r *http.Request) {\n\n  // Get current user\n  user := getCurrentUser(r)\n\n  // Fetch user prefs from database\n  prefs, err := fetchUserPrefs(user.ID)\n  if err != nil { http.Error(w, \"Error fetching prefs\", 500); return }\n\n  // Encode prefs to JSON\n  jsonPrefs := json.Marshal(prefs)\n  if err != nil { http.Error(w, \"Error encoding prefs\", 500); return }\n\n  w.Header().Set(\"Content-Type\", \"application/json\")\n  w.Write(jsonPrefs)\n}</new_code>\n  </example>\n\n  <example>\n    H: <existing_code>\n         // filter out any non-prime numbers from the list\nfunc filterPrimes(list []int) []int {\n<cursor>\n  return primes\n}\n\n// calculate the square root of a number\n       </existing_code>\n\n    A: <new_code>var primes []int\n  for _, num := range list {\n    isPrime := true\n    for i := 2; i <= num/2; i++ {\n      if num%i == 0 {\n        isPrime = false\n        break\n      }\n    }\n    if isPrime { primes = append(primes, num) }\n  }</new_code>\n  </example>\n\n</examples>\n\n\n<existing_code>\n// Generate a function to print hello world\nfunc print<cursor>\n</existing_code>\n\n\nHere are instructions provided in <instruction></instruction> tags.\n\n<instruction>\nCreate more new code for this file. If the cursor is inside an empty function,\ngenerate its most likely contents based on the function name and signature.\n\n</instruction>\n\n\nAssistant: <new_code>"
}
'

Code Generations - Anthropic - Default

curl --request POST \
  --url http://codesuggestions.gdk.test:5999/v2/code/generations \
  --header 'Content-Type: application/json' \
  --header 'X-Gitlab-Authentication-Type: oidc' \
  --data '{
  "current_file": {
    "file_name": "main.go",
    "content_above_cursor": "# Hello world",
    "content_below_cursor": "\n"
  },
  "prompt_version": 2,
  "prompt": "Human: You are a coding autocomplete agent. We want to generate new Go code inside the\nfile '\''hello.go'\'' based on instructions from the user.\nThe existing code is provided in <existing_code></existing_code> tags.\nThe new code you will generate will start at the position of the cursor, which is currently indicated by the <cursor> XML tag.\nIn your process, first, review the existing code to understand its logic and format. Then, try to determine the most\nlikely new code to generate at the cursor position to fulfill the instructions.\n\nWhen generating the new code, please ensure the following:\n1. It is valid Go code.\n2. It matches the existing code'\''s variable, parameter and function names.\n3. It does not repeat any existing code. Do not repeat code that comes before or after the cursor tags. This includes cases where the cursor is in the middle of a word.\n4. If the cursor is in the middle of a word, it finishes the word instead of repeating code before the cursor tag.\n\nReturn new code enclosed in <new_code></new_code> tags. We will then insert this at the <cursor> position.\nIf you are not able to write code based on the given instructions return an empty result like <new_code></new_code>.\n\nHere are a few examples of successfully generated code by other autocomplete agents:\n\n<examples>\n\n  <example>\n    H: <existing_code>\n         package main\nimport (\n  \"net/http\"\n)\nfunc main() {\n  err := http.ListenAndServe(httpPort, nil)\n  if err != nil { panic(err) }\n}\n// create an HTTP handler that fetches the current user preferences and returns them as JSON\n<cursor>\n\n// function to print the current user'\''s name\n       </existing_code>\n\n    A: <new_code>func userPrefsHandler(w http.ResponseWriter, r *http.Request) {\n\n  // Get current user\n  user := getCurrentUser(r)\n\n  // Fetch user prefs from database\n  prefs, err := fetchUserPrefs(user.ID)\n  if err != nil { http.Error(w, \"Error fetching prefs\", 500); return }\n\n  // Encode prefs to JSON\n  jsonPrefs := json.Marshal(prefs)\n  if err != nil { http.Error(w, \"Error encoding prefs\", 500); return }\n\n  w.Header().Set(\"Content-Type\", \"application/json\")\n  w.Write(jsonPrefs)\n}</new_code>\n  </example>\n\n  <example>\n    H: <existing_code>\n         // filter out any non-prime numbers from the list\nfunc filterPrimes(list []int) []int {\n<cursor>\n  return primes\n}\n\n// calculate the square root of a number\n       </existing_code>\n\n    A: <new_code>var primes []int\n  for _, num := range list {\n    isPrime := true\n    for i := 2; i <= num/2; i++ {\n      if num%i == 0 {\n        isPrime = false\n        break\n      }\n    }\n    if isPrime { primes = append(primes, num) }\n  }</new_code>\n  </example>\n\n</examples>\n\n\n<existing_code>\n// Generate a function to print hello world\nfunc print<cursor>\n</existing_code>\n\n\nHere are instructions provided in <instruction></instruction> tags.\n\n<instruction>\nCreate more new code for this file. If the cursor is inside an empty function,\ngenerate its most likely contents based on the function name and signature.\n\n</instruction>\n\n\nAssistant: <new_code>",
  "model_provider": "anthropic"
}

Code Generations - Anthropic - Claude 2.1

curl --request POST \
  --url http://codesuggestions.gdk.test:5999/v2/code/generations \
  --header 'Content-Type: application/json' \
  --header 'X-Gitlab-Authentication-Type: oidc' \
  --data '{
  "current_file": {
    "file_name": "main.go",
    "content_above_cursor": "# Hello world",
    "content_below_cursor": "\n"
  },
  "prompt_version": 2,
  "prompt": "Human: You are a coding autocomplete agent. We want to generate new Go code inside the\nfile '\''hello.go'\'' based on instructions from the user.\nThe existing code is provided in <existing_code></existing_code> tags.\nThe new code you will generate will start at the position of the cursor, which is currently indicated by the <cursor> XML tag.\nIn your process, first, review the existing code to understand its logic and format. Then, try to determine the most\nlikely new code to generate at the cursor position to fulfill the instructions.\n\nWhen generating the new code, please ensure the following:\n1. It is valid Go code.\n2. It matches the existing code'\''s variable, parameter and function names.\n3. It does not repeat any existing code. Do not repeat code that comes before or after the cursor tags. This includes cases where the cursor is in the middle of a word.\n4. If the cursor is in the middle of a word, it finishes the word instead of repeating code before the cursor tag.\n\nReturn new code enclosed in <new_code></new_code> tags. We will then insert this at the <cursor> position.\nIf you are not able to write code based on the given instructions return an empty result like <new_code></new_code>.\n\nHere are a few examples of successfully generated code by other autocomplete agents:\n\n<examples>\n\n  <example>\n    H: <existing_code>\n         package main\nimport (\n  \"net/http\"\n)\nfunc main() {\n  err := http.ListenAndServe(httpPort, nil)\n  if err != nil { panic(err) }\n}\n// create an HTTP handler that fetches the current user preferences and returns them as JSON\n<cursor>\n\n// function to print the current user'\''s name\n       </existing_code>\n\n    A: <new_code>func userPrefsHandler(w http.ResponseWriter, r *http.Request) {\n\n  // Get current user\n  user := getCurrentUser(r)\n\n  // Fetch user prefs from database\n  prefs, err := fetchUserPrefs(user.ID)\n  if err != nil { http.Error(w, \"Error fetching prefs\", 500); return }\n\n  // Encode prefs to JSON\n  jsonPrefs := json.Marshal(prefs)\n  if err != nil { http.Error(w, \"Error encoding prefs\", 500); return }\n\n  w.Header().Set(\"Content-Type\", \"application/json\")\n  w.Write(jsonPrefs)\n}</new_code>\n  </example>\n\n  <example>\n    H: <existing_code>\n         // filter out any non-prime numbers from the list\nfunc filterPrimes(list []int) []int {\n<cursor>\n  return primes\n}\n\n// calculate the square root of a number\n       </existing_code>\n\n    A: <new_code>var primes []int\n  for _, num := range list {\n    isPrime := true\n    for i := 2; i <= num/2; i++ {\n      if num%i == 0 {\n        isPrime = false\n        break\n      }\n    }\n    if isPrime { primes = append(primes, num) }\n  }</new_code>\n  </example>\n\n</examples>\n\n\n<existing_code>\n// Generate a function to print hello world\nfunc print<cursor>\n</existing_code>\n\n\nHere are instructions provided in <instruction></instruction> tags.\n\n<instruction>\nCreate more new code for this file. If the cursor is inside an empty function,\ngenerate its most likely contents based on the function name and signature.\n\n</instruction>\n\n\nAssistant: <new_code>",
  "model_provider": "anthropic",
  "model_name": "claude-2.1"
}

Merge request checklist

Tests added for new functionality. If not, please raise an issue to follow up.
Documentation added/updated, if needed.

Edited Dec 21, 2023 by Tan Le

Support model name on /v2/code/generations

What does this merge request do and why?

How to set up and validate locally

Merge request checklist

Merge request reports