Fix the v3 code generations dependency injection (!570) · Merge requests · GitLab.org / ModelOps / AI Assisted (formerly Applied ML) / Code Suggestions / AI Gateway

Andras Herczeg requested to merge ah/fix-v3-generations-dependency into main Jan 02, 2024

What does this merge request do and why?

A bug was introduced to the v3 code generation endpoint when we allowed the dynamic selection of anthropic models. This MR fixes this problem with the smallest possible footprint; another MR will follow to introduce the model selection to the v3 endpoint as well.

Steps to reproduce

Check out the main branch

Send a cURL request to the /v3/code/completions endpoint

curl --request POST \
  --url http://codesuggestions.gdk.test:5999/v3/code/completions \
  --header 'Content-Type: application/json' \
  --header 'X-Gitlab-Authentication-Type: oidc' \
  --data '{
    "prompt_components": [
        {
            "type": "code_editor_generation",
            "payload": {
                "file_name": "test",
                "content_above_cursor": "func hello_world(){",
                "content_below_cursor": "",
                "model_provider": "anthropic",
                "language_identifier": "go",
                "prompt": "Human: Write a golang function that prints hello world."
            },
            "metadata": {
                "source": "Gitlab EE",
                "version": "16.3"
            }
        }
    ]
}'

Observe 500 HTTP response and error in the log

{
  "url": "http://codesuggestions.gdk.test:5999/v3/code/completions",
  "path": "/v3/code/completions",
  "status_code": 500,
  "method": "POST",
  "correlation_id": "bc228d57d2bb49a4b59ebdc0d3f1af83",
  "http_version": "1.1",
  "client_ip": "172.17.0.1",
  "client_port": 47568,
  "duration_s": 0.061061542997776996,
  "cpu_s": 0.038238288999998815,
  "user_agent": null,
  "gitlab_instance_id": null,
  "gitlab_global_user_id": null,
  "gitlab_host_name": null,
  "gitlab_saas_namespace_ids": null,
  "gitlab_realm": null,
  "meta.feature_category": "code_suggestions",
  "exception": {
    "message": "CodeGenerations.__init__() missing 1 required positional argument: 'model'",
    "backtrace": "Traceback (most recent call last):\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/anyio/streams/memory.py\", line 98, in receive\n    return self.receive_nowait()\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/anyio/streams/memory.py\", line 93, in receive_nowait\n    raise WouldBlock\nanyio.WouldBlock\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 78, in call_next\n    message = await recv_stream.receive()\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/anyio/streams/memory.py\", line 118, in receive\n    raise EndOfStream\nanyio.EndOfStream\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/app/ai_gateway/api/middleware.py\", line 110, in dispatch\n    response = await call_next(request)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 84, in call_next\n    raise app_exc\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 70, in coro\n    await self.app(scope, receive_or_disconnect, send_no_error)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/authentication.py\", line 48, in __call__\n    await self.app(scope, receive, send)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 108, in __call__\n    response = await self.dispatch_func(request, call_next)\n  File \"/app/ai_gateway/api/middleware.py\", line 287, in dispatch\n    return await call_next(request)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 84, in call_next\n    raise app_exc\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py\", line 70, in coro\n    await self.app(scope, receive_or_disconnect, send_no_error)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py\", line 79, in __call__\n    raise exc\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/middleware/exceptions.py\", line 68, in __call__\n    await self.app(scope, receive, sender)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py\", line 20, in __call__\n    raise e\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py\", line 17, in __call__\n    await self.app(scope, receive, send)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/routing.py\", line 718, in __call__\n    await route.handle(scope, receive, send)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/routing.py\", line 276, in handle\n    await self.app(scope, receive, send)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/routing.py\", line 66, in app\n    response = await func(request)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/fastapi/routing.py\", line 274, in app\n    raw_response = await run_endpoint_function(\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/fastapi/routing.py\", line 191, in run_endpoint_function\n    return await dependant.call(**values)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/starlette/authentication.py\", line 76, in async_wrapper\n    return await func(*args, **kwargs)\n  File \"/app/ai_gateway/api/feature_category.py\", line 21, in wrapper\n    return await func(*args, **kwargs)\n  File \"/app/ai_gateway/api/v3/code/completions.py\", line 50, in completions\n    return await code_generation(payload=component.payload)\n  File \"/opt/venv/ai-gateway-9TtSrW0h-py3.10/lib/python3.10/site-packages/dependency_injector/wiring.py\", line 994, in _patched\n    return await _async_inject(\n  File \"src/dependency_injector/_cwiring.pyx\", line 66, in _async_inject\n  File \"/app/ai_gateway/api/v3/code/completions.py\", line 103, in code_generation\n    engine = code_generations_anthropic()\n  File \"src/dependency_injector/providers.pyx\", line 225, in dependency_injector.providers.Provider.__call__\n  File \"src/dependency_injector/providers.pyx\", line 2689, in dependency_injector.providers.Factory._provide\n  File \"src/dependency_injector/providers.pxd\", line 650, in dependency_injector.providers.__factory_call\n  File \"src/dependency_injector/providers.pxd\", line 608, in dependency_injector.providers.__call\nTypeError: CodeGenerations.__init__() missing 1 required positional argument: 'model'\n"
  },
  "logger": "api.access",
  "level": "info",
  "type": "mlops",
  "stage": "main",
  "timestamp": "2024-01-04T00:10:16.539103Z",
  "message": "172.17.0.1:47568 - \"POST /v3/code/completions HTTP/1.1\" 500"
}

How to set up and validate locally

Check out to this merge request's branch.

Ensure a local Docker image built successfully.

docker buildx build --platform linux/amd64 \
  -t ai-gateway:dev .

Run a local service on Docker.

docker run --platform linux/amd64 --rm \
  -p 5052:5052 \
  -e AUTH_BYPASS_EXTERNAL=true \
  -v $PWD:/app -it ai-gateway:dev

Run the following cURL request and confirm 200 HTTP response and no error in log

Code Generations - Anthropic - Default

curl --request POST \
  --url http://codesuggestions.gdk.test:5999/v3/code/completions \
  --header 'Content-Type: application/json' \
  --header 'X-Gitlab-Authentication-Type: oidc' \
  --data '{
    "prompt_components": [
        {
            "type": "code_editor_generation",
            "payload": {
                "file_name": "test",
                "content_above_cursor": "func hello_world(){",
                "content_below_cursor": "",
                "model_provider": "anthropic",
                "language_identifier": "go",
                "prompt": "Human: Write a golang function that prints hello world."
            },
            "metadata": {
                "source": "Gitlab EE",
                "version": "16.3"
            }
        }
    ]
}'

Merge request checklist

Tests added for new functionality. If not, please raise an issue to follow up.
Documentation added/updated, if needed.

Blocked by !530 (merged)

Edited Jan 04, 2024 by Tan Le

Fix the v3 code generations dependency injection

What does this merge request do and why?

Steps to reproduce

How to set up and validate locally

Merge request checklist

Merge request reports