Skip to content

Ai Gateway client for Duo Chat

Shinya Maeda requested to merge ruby-chat-client-ai-gateway into master

What does this MR do and why?

This MR adds AI Gateway client for GitLab Duo chat.

Currently, GitLab-Rails is directly requesting to 3rd party model provider Anthropic, however, this should request to the AI Gateway in order to enable the features in all GitLab instances (SaaS and On-premises). See AI Gateway blueprint for more information.

This is a high priority MR in order for https://gitlab.com/groups/gitlab-org/-/epics/10585+ and Supporting GitLab Duo (chat) for SM and Dedicated (&11251 - closed).

AI Gateway counter-part: Support streaming in Chat API (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!484 - merged)

Screenshots or screen recordings

Test 1: with Gitlab::Llm::AiGateway::Client

Requesting to the AI Gateway from the GitLab-Rails:

[1] pry(main)> user = User.first
[1] pry(main)> user = User.firstew(user).stream(prompt: "\n\nHuman: Can you sing a song?\n\nAssistant:")
  User Load (1.0ms)  SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT 1 /*application:console,db_config_name:main,console_hostname:shinya-XPS-15-9530,console_username:shinya,line:(pry):1:in `__pry__'*/
  ApplicationSetting Load (2.8ms)  SELECT "application_settings"."id", "application_settings"."default_projects_limit", "application_settings"."signup_enabled", "application_settings"."gravatar_enabled", "application_settings"."sign_in_text", "application_settings"."created_at", "application_settings"."updated_at", "application_settings"."home_page_url", "application_settings"."default_branch_protection", "application_settings"."help_text", "application_settings"."restricted_visibility_levels", "application_settings"."version_check_enabled", "application_settings"."max_attachment_size", "application_settings"."default_project_visibility", "application_settings"."default_snippet_visibility", "application_settings"."user_oauth_applications", "application_settings"."after_sign_out_path", "application_settings"."session_expire_delay", "application_settings"."import_sources", "application_settings"."help_page_text", "application_settings"."shared_runners_enabled", "application_settings"."max_artifacts_size", "application_settings"."runners_registration_token", "application_settings"."max_pages_size", "application_settings"."require_two_factor_authentication", "application_settings"."two_factor_grace_period", "application_settings"."metrics_enabled", "application_settings"."metrics_host", "application_settings"."metrics_pool_size", "application_settings"."metrics_timeout", "application_settings"."metrics_method_call_threshold", "application_settings"."recaptcha_enabled", "application_settings"."metrics_port", "application_settings"."akismet_enabled", "application_settings"."metrics_sample_interval", "application_settings"."email_author_in_body", "application_settings"."default_group_visibility", "application_settings"."repository_checks_enabled", "application_settings"."shared_runners_text", "application_settings"."metrics_packet_size", "application_settings"."disabled_oauth_sign_in_sources", "application_settings"."health_check_access_token", "application_settings"."container_registry_token_expire_delay", "application_settings"."after_sign_up_text", "application_settings"."user_default_external", "application_settings"."elasticsearch_indexing", "application_settings"."elasticsearch_search", "application_settings"."enabled_git_access_protocol", "application_settings"."usage_ping_enabled", "application_settings"."sign_in_text_html", "application_settings"."help_page_text_html", "application_settings"."shared_runners_text_html", "application_settings"."after_sign_up_text_html", "application_settings"."rsa_key_restriction", "application_settings"."dsa_key_restriction", "application_settings"."ecdsa_key_restriction", "application_settings"."ed25519_key_restriction", "application_settings"."housekeeping_enabled", "application_settings"."housekeeping_bitmaps_enabled", "application_settings"."housekeeping_incremental_repack_period", "application_settings"."housekeeping_full_repack_period", "application_settings"."housekeeping_gc_period", "application_settings"."html_emails_enabled", "application_settings"."plantuml_url", "application_settings"."plantuml_enabled", "application_settings"."shared_runners_minutes", "application_settings"."repository_size_limit", "application_settings"."terminal_max_session_time", "application_settings"."unique_ips_limit_per_user", "application_settings"."unique_ips_limit_time_window", "application_settings"."unique_ips_limit_enabled", "application_settings"."default_artifacts_expire_in", "application_settings"."elasticsearch_url", "application_settings"."elasticsearch_aws", "application_settings"."elasticsearch_aws_region", "application_settings"."elasticsearch_aws_access_key", "application_settings"."geo_status_timeout", "application_settings"."uuid", "application_settings"."polling_interval_multiplier", "application_settings"."cached_markdown_version", "application_settings"."check_namespace_plan", "application_settings"."mirror_max_delay", "application_settings"."mirror_max_capacity", "application_settings"."mirror_capacity_threshold", "application_settings"."prometheus_metrics_enabled", "application_settings"."authorized_keys_enabled", "application_settings"."help_page_hide_commercial_content", "application_settings"."help_page_support_url", "application_settings"."slack_app_enabled", "application_settings"."slack_app_id", "application_settings"."performance_bar_allowed_group_id", "application_settings"."allow_group_owners_to_manage_ldap", "application_settings"."hashed_storage_enabled", "application_settings"."project_export_enabled", "application_settings"."auto_devops_enabled", "application_settings"."throttle_unauthenticated_enabled", "application_settings"."throttle_unauthenticated_requests_per_period", "application_settings"."throttle_unauthenticated_period_in_seconds", "application_settings"."throttle_authenticated_api_enabled", "application_settings"."throttle_authenticated_api_requests_per_period", "application_settings"."throttle_authenticated_api_period_in_seconds", "application_settings"."throttle_authenticated_web_enabled", "application_settings"."throttle_authenticated_web_requests_per_period", "application_settings"."throttle_authenticated_web_period_in_seconds", "application_settings"."gitaly_timeout_default", "application_settings"."gitaly_timeout_medium", "application_settings"."gitaly_timeout_fast", "application_settings"."mirror_available", "application_settings"."password_authentication_enabled_for_web", "application_settings"."password_authentication_enabled_for_git", "application_settings"."auto_devops_domain", "application_settings"."external_authorization_service_enabled", "application_settings"."external_authorization_service_url", "application_settings"."external_authorization_service_default_label", "application_settings"."pages_domain_verification_enabled", "application_settings"."user_default_internal_regex", "application_settings"."external_authorization_service_timeout", "application_settings"."external_auth_client_cert", "application_settings"."encrypted_external_auth_client_key", "application_settings"."encrypted_external_auth_client_key_iv", "application_settings"."encrypted_external_auth_client_key_pass", "application_settings"."encrypted_external_auth_client_key_pass_iv", "application_settings"."email_additional_text", "application_settings"."enforce_terms", "application_settings"."file_template_project_id", "application_settings"."pseudonymizer_enabled", "application_settings"."hide_third_party_offers", "application_settings"."snowplow_enabled", "application_settings"."snowplow_collector_hostname", "application_settings"."snowplow_cookie_domain", "application_settings"."user_show_add_ssh_key_message", "application_settings"."custom_project_templates_group_id", "application_settings"."usage_stats_set_by_user_id", "application_settings"."receive_max_input_size", "application_settings"."diff_max_patch_bytes", "application_settings"."archive_builds_in_seconds", "application_settings"."commit_email_hostname", "application_settings"."protected_ci_variables", "application_settings"."runners_registration_token_encrypted", "application_settings"."local_markdown_version", "application_settings"."first_day_of_week", "application_settings"."elasticsearch_limit_indexing", "application_settings"."default_project_creation", "application_settings"."lets_encrypt_notification_email", "application_settings"."lets_encrypt_terms_of_service_accepted", "application_settings"."geo_node_allowed_ips", "application_settings"."encrypted_lets_encrypt_private_key", "application_settings"."encrypted_lets_encrypt_private_key_iv", "application_settings"."required_instance_ci_template", "application_settings"."dns_rebinding_protection_enabled", "application_settings"."default_project_deletion_protection", "application_settings"."grafana_enabled", "application_settings"."lock_memberships_to_ldap", "application_settings"."time_tracking_limit_to_hours", "application_settings"."grafana_url", "application_settings"."login_recaptcha_protection_enabled", "application_settings"."outbound_local_requests_whitelist", "application_settings"."raw_blob_request_limit", "application_settings"."allow_local_requests_from_web_hooks_and_services", "application_settings"."allow_local_requests_from_system_hooks", "application_settings"."asset_proxy_enabled", "application_settings"."asset_proxy_url", "application_settings"."encrypted_asset_proxy_secret_key", "application_settings"."encrypted_asset_proxy_secret_key_iv", "application_settings"."static_objects_external_storage_url", "application_settings"."max_personal_access_token_lifetime", "application_settings"."throttle_protected_paths_enabled", "application_settings"."throttle_protected_paths_requests_per_period", "application_settings"."throttle_protected_paths_period_in_seconds", "application_settings"."protected_paths", "application_settings"."throttle_incident_management_notification_enabled", "application_settings"."throttle_incident_management_notification_period_in_seconds", "application_settings"."throttle_incident_management_notification_per_period", "application_settings"."push_event_hooks_limit", "application_settings"."push_event_activities_limit", "application_settings"."custom_http_clone_url_root", "application_settings"."deletion_adjourned_period", "application_settings"."license_trial_ends_on", "application_settings"."eks_integration_enabled", "application_settings"."eks_account_id", "application_settings"."eks_access_key_id", "application_settings"."encrypted_eks_secret_access_key_iv", "application_settings"."encrypted_eks_secret_access_key", "application_settings"."snowplow_app_id", "application_settings"."productivity_analytics_start_date", "application_settings"."default_ci_config_path", "application_settings"."sourcegraph_enabled", "application_settings"."sourcegraph_url", "application_settings"."sourcegraph_public_only", "application_settings"."snippet_size_limit", "application_settings"."minimum_password_length", "application_settings"."encrypted_akismet_api_key", "application_settings"."encrypted_akismet_api_key_iv", "application_settings"."encrypted_elasticsearch_aws_secret_access_key", "application_settings"."encrypted_elasticsearch_aws_secret_access_key_iv", "application_settings"."encrypted_recaptcha_private_key", "application_settings"."encrypted_recaptcha_private_key_iv", "application_settings"."encrypted_recaptcha_site_key", "application_settings"."encrypted_recaptcha_site_key_iv", "application_settings"."encrypted_slack_app_secret", "application_settings"."encrypted_slack_app_secret_iv", "application_settings"."encrypted_slack_app_verification_token", "application_settings"."encrypted_slack_app_verification_token_iv", "application_settings"."force_pages_access_control", "application_settings"."updating_name_disabled_for_users", "application_settings"."elasticsearch_indexed_field_length_limit", "application_settings"."elasticsearch_max_bulk_size_mb", "application_settings"."elasticsearch_max_bulk_concurrency", "application_settings"."disable_overriding_approvers_per_merge_request", "application_settings"."prevent_merge_requests_author_approval", "application_settings"."prevent_merge_requests_committers_approval", "application_settings"."email_restrictions_enabled", "application_settings"."email_restrictions", "application_settings"."npm_package_requests_forwarding", "application_settings"."container_expiration_policies_enable_historic_entries", "application_settings"."issues_create_limit", "application_settings"."push_rule_id", "application_settings"."group_owners_can_manage_default_branch_protection", "application_settings"."container_registry_vendor", "application_settings"."container_registry_version", "application_settings"."container_registry_features", "application_settings"."spam_check_endpoint_url", "application_settings"."spam_check_endpoint_enabled", "application_settings"."elasticsearch_pause_indexing", "application_settings"."repository_storages_weighted", "application_settings"."max_import_size", "application_settings"."compliance_frameworks", "application_settings"."notify_on_unknown_sign_in", "application_settings"."default_branch_name", "application_settings"."project_import_limit", "application_settings"."project_export_limit", "application_settings"."project_download_export_limit", "application_settings"."group_import_limit", "application_settings"."group_export_limit", "application_settings"."group_download_export_limit", "application_settings"."maintenance_mode", "application_settings"."maintenance_mode_message", "application_settings"."wiki_page_max_content_bytes", "application_settings"."elasticsearch_indexed_file_size_limit_kb", "application_settings"."enforce_namespace_storage_limit", "application_settings"."container_registry_delete_tags_service_timeout", "application_settings"."kroki_url", "application_settings"."kroki_enabled", "application_settings"."elasticsearch_client_request_timeout", "application_settings"."gitpod_enabled", "application_settings"."gitpod_url", "application_settings"."abuse_notification_email", "application_settings"."require_admin_approval_after_user_signup", "application_settings"."help_page_documentation_base_url", "application_settings"."automatic_purchased_storage_allocation", "application_settings"."encrypted_ci_jwt_signing_key", "application_settings"."encrypted_ci_jwt_signing_key_iv", "application_settings"."container_registry_expiration_policies_worker_capacity", "application_settings"."elasticsearch_analyzers_smartcn_enabled", "application_settings"."elasticsearch_analyzers_smartcn_search", "application_settings"."elasticsearch_analyzers_kuromoji_enabled", "application_settings"."elasticsearch_analyzers_kuromoji_search", "application_settings"."secret_detection_token_revocation_enabled", "application_settings"."secret_detection_token_revocation_url", "application_settings"."encrypted_secret_detection_token_revocation_token", "application_settings"."encrypted_secret_detection_token_revocation_token_iv", "application_settings"."domain_denylist_enabled", "application_settings"."domain_denylist", "application_settings"."domain_allowlist", "application_settings"."new_user_signups_cap", "application_settings"."encrypted_cloud_license_auth_token", "application_settings"."encrypted_cloud_license_auth_token_iv", "application_settings"."secret_detection_revocation_token_types_url", "application_settings"."disable_feed_token", "application_settings"."personal_access_token_prefix", "application_settings"."rate_limiting_response_text", "application_settings"."invisible_captcha_enabled", "application_settings"."container_registry_cleanup_tags_service_max_list_size", "application_settings"."git_two_factor_session_expiry", "application_settings"."keep_latest_artifact", "application_settings"."notes_create_limit", "application_settings"."notes_create_limit_allowlist", "application_settings"."kroki_formats", "application_settings"."asset_proxy_whitelist", "application_settings"."admin_mode", "application_settings"."delayed_project_removal", "application_settings"."lock_delayed_project_removal", "application_settings"."external_pipeline_validation_service_timeout", "application_settings"."encrypted_external_pipeline_validation_service_token", "application_settings"."encrypted_external_pipeline_validation_service_token_iv", "application_settings"."external_pipeline_validation_service_url", "application_settings"."throttle_unauthenticated_packages_api_requests_per_period", "application_settings"."throttle_unauthenticated_packages_api_period_in_seconds", "application_settings"."throttle_authenticated_packages_api_requests_per_period", "application_settings"."throttle_authenticated_packages_api_period_in_seconds", "application_settings"."throttle_unauthenticated_packages_api_enabled", "application_settings"."throttle_authenticated_packages_api_enabled", "application_settings"."deactivate_dormant_users", "application_settings"."whats_new_variant", "application_settings"."encrypted_spam_check_api_key", "application_settings"."encrypted_spam_check_api_key_iv", "application_settings"."floc_enabled", "application_settings"."elasticsearch_username", "application_settings"."encrypted_elasticsearch_password", "application_settings"."encrypted_elasticsearch_password_iv", "application_settings"."diff_max_lines", "application_settings"."diff_max_files", "application_settings"."valid_runner_registrars", "application_settings"."encrypted_mailgun_signing_key", "application_settings"."encrypted_mailgun_signing_key_iv", "application_settings"."mailgun_events_enabled", "application_settings"."usage_ping_features_enabled", "application_settings"."encrypted_customers_dot_jwt_signing_key", "application_settings"."encrypted_customers_dot_jwt_signing_key_iv", "application_settings"."pypi_package_requests_forwarding", "application_settings"."throttle_unauthenticated_files_api_requests_per_period", "application_settings"."throttle_unauthenticated_files_api_period_in_seconds", "application_settings"."throttle_authenticated_files_api_requests_per_period", "application_settings"."throttle_authenticated_files_api_period_in_seconds", "application_settings"."throttle_unauthenticated_files_api_enabled", "application_settings"."throttle_authenticated_files_api_enabled", "application_settings"."max_yaml_size_bytes", "application_settings"."max_yaml_depth", "application_settings"."throttle_authenticated_git_lfs_requests_per_period", "application_settings"."throttle_authenticated_git_lfs_period_in_seconds", "application_settings"."throttle_authenticated_git_lfs_enabled", "application_settings"."user_deactivation_emails_enabled", "application_settings"."throttle_unauthenticated_api_enabled", "application_settings"."throttle_unauthenticated_api_requests_per_period", "application_settings"."throttle_unauthenticated_api_period_in_seconds", "application_settings"."jobs_per_stage_page_size", "application_settings"."sidekiq_job_limiter_mode", "application_settings"."sidekiq_job_limiter_compression_threshold_bytes", "application_settings"."sidekiq_job_limiter_limit_bytes", "application_settings"."suggest_pipeline_enabled", "application_settings"."throttle_unauthenticated_deprecated_api_requests_per_period", "application_settings"."throttle_unauthenticated_deprecated_api_period_in_seconds", "application_settings"."throttle_unauthenticated_deprecated_api_enabled", "application_settings"."throttle_authenticated_deprecated_api_requests_per_period", "application_settings"."throttle_authenticated_deprecated_api_period_in_seconds", "application_settings"."throttle_authenticated_deprecated_api_enabled", "application_settings"."dependency_proxy_ttl_group_policy_worker_capacity", "application_settings"."content_validation_endpoint_url", "application_settings"."encrypted_content_validation_api_key", "application_settings"."encrypted_content_validation_api_key_iv", "application_settings"."content_validation_endpoint_enabled", "application_settings"."sentry_enabled", "application_settings"."sentry_dsn", "application_settings"."sentry_clientside_dsn", "application_settings"."sentry_environment", "application_settings"."max_ssh_key_lifetime", "application_settings"."static_objects_external_storage_auth_token_encrypted", "application_settings"."future_subscriptions", "application_settings"."packages_cleanup_package_file_worker_capacity", "application_settings"."container_registry_import_max_tags_count", "application_settings"."container_registry_import_max_retries", "application_settings"."container_registry_import_start_max_retries", "application_settings"."container_registry_import_max_step_duration", "application_settings"."container_registry_import_target_plan", "application_settings"."container_registry_import_created_before", "application_settings"."runner_token_expiration_interval", "application_settings"."group_runner_token_expiration_interval", "application_settings"."project_runner_token_expiration_interval", "application_settings"."ecdsa_sk_key_restriction", "application_settings"."ed25519_sk_key_restriction", "application_settings"."users_get_by_id_limit", "application_settings"."users_get_by_id_limit_allowlist", "application_settings"."container_registry_expiration_policies_caching", "application_settings"."search_rate_limit", "application_settings"."search_rate_limit_unauthenticated", "application_settings"."encrypted_database_grafana_api_key", "application_settings"."encrypted_database_grafana_api_key_iv", "application_settings"."database_grafana_api_url", "application_settings"."database_grafana_tag", "application_settings"."public_runner_releases_url", "application_settings"."password_uppercase_required", "application_settings"."password_lowercase_required", "application_settings"."password_number_required", "application_settings"."password_symbol_required", "application_settings"."encrypted_arkose_labs_public_api_key", "application_settings"."encrypted_arkose_labs_public_api_key_iv", "application_settings"."encrypted_arkose_labs_private_api_key", "application_settings"."encrypted_arkose_labs_private_api_key_iv", "application_settings"."arkose_labs_verify_api_url", "application_settings"."delete_inactive_projects", "application_settings"."inactive_projects_delete_after_months", "application_settings"."inactive_projects_min_size_mb", "application_settings"."inactive_projects_send_warning_email_after_months", "application_settings"."delayed_group_deletion", "application_settings"."maven_package_requests_forwarding", "application_settings"."arkose_labs_namespace", "application_settings"."max_export_size", "application_settings"."encrypted_slack_app_signing_secret", "application_settings"."encrypted_slack_app_signing_secret_iv", "application_settings"."container_registry_pre_import_timeout", "application_settings"."container_registry_import_timeout", "application_settings"."pipeline_limit_per_project_user_sha", "application_settings"."dingtalk_integration_enabled", "application_settings"."encrypted_dingtalk_corpid", "application_settings"."encrypted_dingtalk_corpid_iv", "application_settings"."encrypted_dingtalk_app_key", "application_settings"."encrypted_dingtalk_app_key_iv", "application_settings"."encrypted_dingtalk_app_secret", "application_settings"."encrypted_dingtalk_app_secret_iv", "application_settings"."jira_connect_application_key", "application_settings"."globally_allowed_ips", "application_settings"."container_registry_pre_import_tags_rate", "application_settings"."license_usage_data_exported", "application_settings"."phone_verification_code_enabled", "application_settings"."max_number_of_repository_downloads", "application_settings"."max_number_of_repository_downloads_within_time_period", "application_settings"."feishu_integration_enabled", "application_settings"."encrypted_feishu_app_key", "application_settings"."encrypted_feishu_app_key_iv", "application_settings"."encrypted_feishu_app_secret", "application_settings"."encrypted_feishu_app_secret_iv", "application_settings"."error_tracking_enabled", "application_settings"."error_tracking_api_url", "application_settings"."git_rate_limit_users_allowlist", "application_settings"."error_tracking_access_token_encrypted", "application_settings"."invitation_flow_enforcement", "application_settings"."package_registry_cleanup_policies_worker_capacity", "application_settings"."deactivate_dormant_users_period", "application_settings"."auto_ban_user_on_excessive_projects_download", "application_settings"."max_pages_custom_domains_per_project", "application_settings"."cube_api_base_url", "application_settings"."encrypted_cube_api_key", "application_settings"."encrypted_cube_api_key_iv", "application_settings"."dashboard_limit_enabled", "application_settings"."dashboard_limit", "application_settings"."can_create_group", "application_settings"."lock_maven_package_requests_forwarding", "application_settings"."lock_pypi_package_requests_forwarding", "application_settings"."lock_npm_package_requests_forwarding", "application_settings"."jira_connect_proxy_url", "application_settings"."password_expiration_enabled", "application_settings"."password_expires_in_days", "application_settings"."password_expires_notice_before_days", "application_settings"."product_analytics_enabled", "application_settings"."email_confirmation_setting", "application_settings"."disable_admin_oauth_scopes", "application_settings"."default_preferred_language", "application_settings"."disable_download_button", "application_settings"."encrypted_telesign_customer_xid", "application_settings"."encrypted_telesign_customer_xid_iv", "application_settings"."encrypted_telesign_api_key", "application_settings"."encrypted_telesign_api_key_iv", "application_settings"."disable_personal_access_tokens", "application_settings"."max_terraform_state_size_bytes", "application_settings"."bulk_import_enabled", "application_settings"."allow_runner_registration_token", "application_settings"."user_defaults_to_private_profile", "application_settings"."allow_possible_spam", "application_settings"."default_syntax_highlighting_theme", "application_settings"."search_max_shard_size_gb", "application_settings"."search_max_docs_denominator", "application_settings"."search_min_docs_before_rollover", "application_settings"."deactivation_email_additional_text", "application_settings"."jira_connect_public_key_storage_enabled", "application_settings"."git_rate_limit_users_alertlist", "application_settings"."allow_deploy_tokens_and_keys_with_external_authn", "application_settings"."security_policy_global_group_approvers_enabled", "application_settings"."projects_api_rate_limit_unauthenticated", "application_settings"."deny_all_requests_except_allowed", "application_settings"."product_analytics_data_collector_host", "application_settings"."lock_memberships_to_saml", "application_settings"."gitlab_dedicated_instance", "application_settings"."update_runner_versions_enabled", "application_settings"."encrypted_openai_api_key", "application_settings"."encrypted_openai_api_key_iv", "application_settings"."database_max_running_batched_background_migrations", "application_settings"."encrypted_product_analytics_configurator_connection_string", "application_settings"."encrypted_product_analytics_configurator_connection_string_iv", "application_settings"."silent_mode_enabled", "application_settings"."package_metadata_purl_types", "application_settings"."ci_max_includes", "application_settings"."remember_me_enabled", "application_settings"."encrypted_anthropic_api_key", "application_settings"."encrypted_anthropic_api_key_iv", "application_settings"."diagramsnet_enabled", "application_settings"."diagramsnet_url", "application_settings"."allow_account_deletion", "application_settings"."wiki_asciidoc_allow_uri_includes", "application_settings"."namespace_aggregation_schedule_lease_duration_in_seconds", "application_settings"."container_registry_data_repair_detail_worker_max_concurrency", "application_settings"."vertex_ai_host", "application_settings"."encrypted_vertex_ai_credentials", "application_settings"."encrypted_vertex_ai_credentials_iv", "application_settings"."vertex_ai_project", "application_settings"."instance_level_code_suggestions_enabled", "application_settings"."delete_unconfirmed_users", "application_settings"."unconfirmed_users_delete_after_days", "application_settings"."default_branch_protection_defaults", "application_settings"."gitlab_shell_operation_limit", "application_settings"."elasticsearch_requeue_workers", "application_settings"."elasticsearch_worker_number_of_shards", "application_settings"."protected_paths_for_get_request", "application_settings"."namespace_storage_forks_cost_factor", "application_settings"."package_registry_allow_anyone_to_pull_option", "application_settings"."bulk_import_max_download_file_size", "application_settings"."max_import_remote_file_size", "application_settings"."max_decompressed_archive_size", "application_settings"."sentry_clientside_traces_sample_rate", "application_settings"."prometheus_alert_db_indicators_settings", "application_settings"."ci_max_total_yaml_size_bytes", "application_settings"."decompress_archive_file_timeout", "application_settings"."search_rate_limit_allowlist", "application_settings"."snowplow_database_collector_hostname", "application_settings"."container_registry_db_enabled", "application_settings"."failed_login_attempts_unlock_period_in_minutes", "application_settings"."max_login_attempts", "application_settings"."encrypted_vertex_ai_access_token", "application_settings"."encrypted_vertex_ai_access_token_iv", "application_settings"."project_jobs_api_rate_limit", "application_settings"."math_rendering_limits_enabled", "application_settings"."service_access_tokens_expiration_enforced", "application_settings"."enable_artifact_external_redirect_warning_page", "application_settings"."update_namespace_name_rate_limit", "application_settings"."allow_project_creation_for_guest_and_below", "application_settings"."pre_receive_secret_detection_enabled", "application_settings"."make_profile_private", "application_settings"."can_create_organization", "application_settings"."web_ide_oauth_application_id", "application_settings"."instance_level_ai_beta_features_enabled", "application_settings"."security_txt_content" FROM "application_settings" ORDER BY "application_settings"."id" DESC LIMIT 1 /*application:console,db_config_name:main,console_hostname:shinya-XPS-15-9530,console_username:shinya,line:/app/models/concerns/cacheable_attributes.rb:19:in `current_without_cache'*/
  Group Load (1.1ms)  SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."emails_disabled", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND "namespaces"."path" = 'gitlab-com' AND "namespaces"."parent_id" IS NULL LIMIT 1 /*application:console,db_config_name:main,console_hostname:shinya-XPS-15-9530,console_username:shinya,line:/ee/lib/gitlab/com.rb:26:in `block (2 levels) in gitlab_com_user_ids'*/
=> " I'm an AI assistant created by Anthropic to be helpful, harmless, and honest. I don't have the ability to sing songs, but I can talk with you!"

Corresponding log in AI Gateway:

{
    "url": "http://localhost:5052/v1/chat/agent",
    "path": "/v1/chat/agent",
    "status_code": 200,
    "method": "POST",
    "correlation_id": "64f5c704db344d059c6b176e9b2acaa4",
    "http_version": "1.1",
    "client_ip": "127.0.0.1",
    "client_port": 58286,
    "duration_s": 1.9310308560015983,
    "cpu_s": 0.014569222999999631,
    "user_agent": "Ruby",
    "gitlab_instance_id": "9e210aee-80e2-4f03-86ce-d4bb0fb46455",
    "gitlab_global_user_id": "bqtER3xdKMEUpFY0uIH0fTzf5a5WqJWNIfBDNsg2Y9E=",
    "gitlab_host_name": "gdk.test",
    "gitlab_saas_namespace_ids": null,
    "gitlab_realm": "saas",
    "logger": "api.access",
    "level": "info",
    "type": "mlops",
    "stage": "main",
    "timestamp": "2023-12-13T06:04:36.411070Z",
    "message": "127.0.0.1:58286 - \"POST /v1/chat/agent HTTP/1.1\" 200"
}

Test 2: with Gitlab::Llm::Chain::Requests::AiGateway

Requesting to the AI Gateway via Gitlab::Llm::Chain::Requests::AiGateway:

[6] pry(main)> user = User.first
  User Load (0.5ms)  SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT 1 /*application:console,db_config_name:main,console_hostname:shinya-XPS-15-9530,console_username:shinya,line:(pry):6:in `__pry__'*/
=> #<User id:1 @root>
[7] pry(main)> prompt = { prompt: "\n\nHuman: Hi, How are you?\n\nAssistant:" }
=> {:prompt=>"\n\nHuman: Hi, How are you?\n\nAssistant:"}
[8] pry(main)> Gitlab::Llm::Chain::Requests::AiGateway.new(user).request(prompt)
=> " I'm doing well, thanks for asking!"

Corresponding log in AI Gateway:

{
    "url": "http://localhost:5052/v1/chat/agent",
    "path": "/v1/chat/agent",
    "status_code": 200,
    "method": "POST",
    "correlation_id": "2b48351a68bb408d9698b23dbb3cc010",
    "http_version": "1.1",
    "client_ip": "127.0.0.1",
    "client_port": 47134,
    "duration_s": 0.7762770629997249,
    "cpu_s": 0.013390616999999772,
    "user_agent": "Ruby",
    "gitlab_instance_id": "9e210aee-80e2-4f03-86ce-d4bb0fb46455",
    "gitlab_global_user_id": "bqtER3xdKMEUpFY0uIH0fTzf5a5WqJWNIfBDNsg2Y9E=",
    "gitlab_host_name": "gdk.test",
    "gitlab_saas_namespace_ids": null,
    "gitlab_realm": "saas",
    "logger": "api.access",
    "level": "info",
    "type": "mlops",
    "stage": "main",
    "timestamp": "2023-12-13T06:06:36.281904Z",
    "message": "127.0.0.1:47134 - \"POST /v1/chat/agent HTTP/1.1\" 200"
}

How to set up and validate locally

  1. Run AI Gateway streaming MR Support streaming in Chat API (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!484 - merged)
  2. Run GDK
  3. Enable gitlab_duo_chat_requests_to_ai_gateway feature flag
  4. Ensure Duo Chat is activated in your GitLab instance https://docs.gitlab.com/ee/development/ai_features/index.html#test-ai-features-locally.
  5. Access to the Duo Chat Web UI.

For more detailed instruction, see !138274 (comment 1687912146)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Shinya Maeda

Merge request reports