[QA] Enable code_suggestions_in_web_ide_spec.rb QA spec
Description
The QA spec code_suggestions_in_web_ide_spec.rb was added to fast quarantine since it continued to fail on staging. This blocked deployments, resulting in this incident.
TODO: Let's investigate why this was failing and re-enable the test as soon as possible.
Investigation
- Our first thought was that this was related to the new
web_ide_oauth
Feature Flag. However, this proved to not be the case (internal Slack thread). - This QA spec will fail even if everything is hooked up properly, but the model has no suggestions. Maybe we need to improve the prompt?
- Upon testing, whenever the
code_suggestions/completions
endpoint returns withmodel.name
ascode-bison@latest
, we get no results.code_gecko@latest
performs much better and returns results. Maybe there was a change in the AI Gateway that is routing certain requests to the underperformingcode-bison
model?
Thanks
Thanks @mayra-cabrera && @ddavison for collaborating on this investigation!
Edited by Paul Slaughter