Skip to content

Resolve "Explain This Vulnerability: Handle Null Prompt situations"

Gregory Havenga requested to merge 416521-explain-this-vulnerability-handle-null-prompt-situations into master

What does this MR do and why?

There is a corner case possible in the implementation of the ExplainThisVulnerability AiAction in which a user may use the include_source_code on a vulnerability which does not have applicable source code to place in the prompt. This behaviour is technically undefined as the interface is capable of identifying that this situation has occurred and makes the request without enforcing the presence of code, however it is still possible to force it.

Attempting to do so results in the template class generating a null prompt, which does not break, but is redundant and a waste of a request to the LLM api, so this MR checks that the prompt output is not null, else providing a default error response.

While here, I have additionally removed the redundant options method from the template class which is no longer needed due to the removal of the OpenAi integration.

How to set up and validate locally

Assuming you have SAST vulnerabilities present in your local GDK and sufficient credentials to test against the LLM API's, use the following snippet in your rails console:

Gitlab::Llm::Completions::ExplainVulnerability.new(Gitlab::Llm::Templates::ExplainVulnerability).execute(User.first, Vulnerabilities::Finding.all.find{|v| !v.source_code?}.vulnerability, {include_source_code: true})

This will call the AiAction behaviours internally, enforcing a prompt containing code while selecting a vulnerability_finding that doesn't have an associated code snippet from your DB and calling it's vulnerability. The printed result expected looks like this:

 (called from execute_all at /home/gregory/Projects/gitlab-development-kit/gitlab/lib/gitlab/graphql/subscriptions/action_cable_with_load_balancing.rb:16)
[ActionCable] Broadcasting to graphql-event::aiCompletionResponse:resourceId:Z2lkOi8vZ2l0bGFiL1Z1bG5lcmFiaWxpdHkvMQ:userId:Z2lkOi8vZ2l0bGFiL1VzZXIvMQ: "{\"wal_locations\":{},\"gql_payload\":{\"id\":\"b5eeb13f-9a3d-4d7a-8574-2eff9156452f\",\"request_id\":null,\"model_name\":\"Vulnerability\",\"content\":\"\",\"errors\":[\"An unexpected error has occurred.\"],\"role\":\"assistant\",\"timestamp\":{\"__timestamp__\":[\"Time\",\"2023-08-21 15:26:58....
=> "{\"error\":{\"message\":\"An unexpected error has occurred.\"}}"

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #416521 (closed)

Edited by Gregory Havenga

Merge request reports