Self-hosted Mistral/Mixtral models for Code generation
What does this MR do and why?
This MR enables the setting/use of self-hosted models (Mistral/Mixtral) for code suggestions.
This MR is based on !152380 (merged) and the FE is being handled in !152107 (merged)
Screenshots or screen recordings
Screen_Recording_2024-05-17_at_13.53.40
Prompt Evaluation Control (~0.85) and Test similarity score (~0.87)
- Control : 0.85
- Test with one setup: 0.87 : Variance: +2%
- Test with second simpler set up: 0.86 : Variance : +1%
- Sample Size:425
- Success: Yes , for both set ups , but to optimise latency choosing simple instructions , the simpler set up as the main aim was to be better than the control?
On GCP
SELECT count(*), avg(similarity_score) FROM `dev-ai-research-0e2f8974.code_suggestion_experiments.mhamda_mistral_20240517_161540__similarity_score
How to set up and validate locally
-
Ensure you have purchased Duo Pro and that the instance is running as an SM instance.
-
Verify the value of
GITLAB_SIMULATE_SAAS
by executingecho $GITLAB_SIMULATE_SAAS
; it should be0
.
If you don't have a Duo Pro record, you can create one from rails c
using:
add_on = GitlabSubscriptions::AddOn.find_or_create_by!(name: "code_suggestions") {|e| e.description = "Test"}
add_on_purchase = GitlabSubscriptions::AddOnPurchase.create!(
add_on: add_on, expires_on: 1.month.from_now, quantity: 5, purchase_xid: 'A-S0001'
)
- In a separate terminal, if
gdk
is running, execute the following:
gdk stop rails-web && GITLAB_RAILS_RACK_TIMEOUT_ENABLE_LOGGING=false PUMA_SINGLE_MODE=true GITLAB_SIMULATE_SAAS=0 gdk rails s
-
In the Rails console, enable the experiment fully:
> Feature.enable(:self_managed_code_suggestions) > Feature.enable(:ai_custom_model) > ApplicationSetting.first.update(duo_features_enabled: true)
This MVP requires the
Ai::SelfHostedModel
record to be created in the database.The table/model exists on the target branch, but the UI doesn't yet exist there: Front-end iteration for self-deployed model con... (!152107 - merged).
Please create a record in the Rails console: ::Ai::SelfHostedModel.create!(model: 0, endpoint: 'http://127.0.0.1:11434/v1', name: 'mistral-7B', api_token: 'token')
-
Log in as an admin.
-
Visit the index page Admin Area > AI-Powered Features > Features.
-
Use Mistral for Code Generations.
Make sure that you have the AI gateway up and running in the GDK.
Set up a locally hosted Mistral model using OLLAMA. In a separate terminal, run ollama serve
.
Follow the above recording to verify the changes.