Skip to content

Make Chat requests go through cloud.gitlab.com

Matthias Käppler requested to merge 431554-cloud-gitlab-com into master

What does this MR do and why?

We would like to route all AI requests through cloud.gitlab.com/ai (Cloud Connector LB) instead of codesuggestions.gitlab.com. We had already started doing this for Code Suggestions. For Duo Chat, which only recently moved behind the AI gateway, this MR updates the client instance to also use the new Cloud Connector route.

This change is behind the same FF use_cloud_connector_lb, which I merely moved elsewhere to cover both call sites.

Local development

We still allow developers to override this behavior in case they need to point their GL instance elsewhere, in which case they can now set AI_GATEWAY_URL. This should also solve #434671 (closed).

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

I verified that with the FF enabled, we now contact the correct host:

Before:

workhorse_1         | time="2024-01-17T14:31:35Z" level=info msg="SendURL: sending" correlation_id=01HMBXBA2CEW19G3KE4SNKJKCG path=/api/v4/code_suggestions/completions url="https://codesuggestions.gitlab.com/v2/code/generations"
workhorse_1         | localhost:3000 192.168.80.1 - - [2024/01/17:14:31:35 +0000] "POST /api/v4/code_suggestions/completions HTTP/1.1" 401 38 "" "curl/8.2.1" 669

After:

workhorse_1         | time="2024-01-17T14:41:19Z" level=info msg="SendURL: sending" correlation_id=01HMBXX4Y70DP72YMNDHVD2QHJ path=/api/v4/code_suggestions/completions url="https://cloud.gitlab.com/ai/v2/code/generations"
workhorse_1         | localhost:3000 192.168.80.1 - - [2024/01/17:14:41:19 +0000] "POST /api/v4/code_suggestions/completions HTTP/1.1" 401 38 "" "curl/8.2.1" 408

For Chat this is harder to track, but I checked that the Sidekiq job making this request still works correctly.

How to set up and validate locally

  1. Enable FF listed above
  2. Curl CS endpoint:
curl -v -H'content-type: application/json' -H'Authorization: Bearer glpat-...' -d'{
    "prompt_version": 1,
    "current_file": {
      "file_name": "test.py",
      "content_above_cursor": "def is_even(n: int) ->",
      "content_below_cursor": "end"
    }
  }' localhost:3000/api/v4/code_suggestions/completions

Related to #431554 (closed)

Edited by Matthias Käppler

Merge request reports