fix(chat): authorization and telemetry of injected context

This MR is high priority item for Create a Unit Primitive for the include-file co... (gitlab-org&14979), which is task forceAI Context initiatives

What does this merge request do and why?

This MR fixes Authorization/Telemetry of Injected Context (include_<category>_context unit primitive).

In addition, tests are added to prevent regressions from happening in the future development. No changes on the business logic, including prompts.

Closes Authorization and Telemetry for include_<catego... (gitlab-org/gitlab#491598 - closed)

Related to Add unit primitive for injected context (gitlab-org/gitlab!168327 - merged)

How to set up and validate locally

Request example:

curl -X 'POST' \
  'http://localhost:5052/v2/chat/agent' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'x-gitlab-enabled-feature-flags: expanded_ai_logging' \
  -d '{
  "messages": [
    {
      "role": "user",
      "content": "How can I log output?",
      "additional_context": [
        { "id": "123", "category": "merge_request", "content": "something", "metadata": {"a": "b"}}
      ]
    },
    {"role": "assistant", "content": "Use print function"},
    {
      "role": "user",
      "content": "How can I log output?",
      "context": { "type": "issue", "content": "This is awesome issue" },
      "current_file": { "file_path": "main.py", "data": "print", "selected_code": true },
      "additional_context": [
        { "id": "456", "category": "file", "content": "helloworld", "metadata": {"c": "d"}}
      ]
    }
  ],
  "options": {
    "agent_scratchpad": {
      "agent_type": "react",
      "steps": []
    }
  }
}'

Tracked internal events in access log example:

{"url": "http://localhost:5052/v2/chat/agent", "path": "/v2/chat/agent", "status_code": 200, "method": "POST", "correlation_id": "415b35b61b004f6c93e22e45802f1ba2", "http_version": "1.1", "client_ip": "127.0.0.1", "client_port": 43502, "duration_s": 5.554389568000261, "duration_request": -1, "request_arrived_at": "2024-10-11T05:02:32.979111+00:00", "response_start_duration_s": 0.08353800800068711, "first_chunk_duration_s": 1.8264594829997804, "cpu_s": 0.23477931499999993, "content_type": "application/x-ndjson; charset=utf-8", "user_agent": "curl/7.81.0", "gitlab_language_server_version": null, "gitlab_instance_id": null, "gitlab_global_user_id": null, "gitlab_host_name": null, "gitlab_version": null, "gitlab_saas_duo_pro_namespace_ids": null, "gitlab_saas_namespace_ids": null, "gitlab_realm": null, "gitlab_duo_seat_count": null, "enabled_feature_flags": "expanded_ai_logging", "meta.feature_category": "duo_chat", "tracked_internal_events": ["request_include_file_context", "request_include_merge_request_context", "request_duo_chat"], "logger": "api.access", "level": "info", "type": "mlops", "stage": "main", "timestamp": "2024-10-11T05:02:38.533580Z", "message": "127.0.0.1:43502 - \"POST /v2/chat/agent HTTP/1.1\" 200"}

If the user (IJWT) is not authorized to access include_<category>_context unit primitive:

Response:

{"detail":"Unauthorized to access include_merge_request_context"}

Access Log:

{"url": "http://localhost:5052/v2/chat/agent", "path": "/v2/chat/agent", "status_code": 403, "method": "POST", "correlation_id": "ecbd699a767b43929a65990a7d4f4eb4", "http_version": "1.1", "client_ip": "127.0.0.1", "client_port": 40662, "duration_s": 0.0357103599999391, "duration_request": -1, "request_arrived_at": "2024-10-09T05:31:27.234866+00:00", "response_start_duration_s": 0.03554826800063893, "first_chunk_duration_s": 0.03561220600022352, "cpu_s": 0.03577610199999981, "content_type": "application/json", "user_agent": "curl/7.81.0", "gitlab_language_server_version": null, "gitlab_instance_id": null, "gitlab_global_user_id": null, "gitlab_host_name": null, "gitlab_version": null, "gitlab_saas_duo_pro_namespace_ids": null, "gitlab_saas_namespace_ids": null, "gitlab_realm": null, "gitlab_duo_seat_count": null, "enabled_feature_flags": "expanded_ai_logging", "meta.feature_category": "duo_chat", "http_exception_details": "403: Unauthorized to access include_merge_request_context", "logger": "api.access", "level": "info", "type": "mlops", "stage": "main", "timestamp": "2024-10-09T05:31:27.270620Z", "message": "127.0.0.1:40662 - \"POST /v2/chat/agent HTTP/1.1\" 403"}

Merge request checklist

• Tests added for new functionality. If not, please raise an issue to follow up.
• Documentation added/updated, if needed.