Draft: Code suggestions streaming experiment
What does this MR do and why?
Info: The experiment, not intended to be merged
An experiment of adding a streaming option to /code_suggestions/completions API endpoint
Screenshots or screen recordings
As you can see in the video, the API endpoint streams responses in batches instead of streaming every event. I'm still trying to figure out why this is happening. It could be related to how Workhorse operates.
How to set up and validate locally
- Copy the following script to a .js file
async function fetchDataFromStream() {
const apiToken = 'YOUR_API_TOKEN';
const apiUrl = 'http://127.0.0.1:3000/api/v4/code_suggestions/completions';
try {
const stream = true
const rawResponse = false
const contentAboveCursor = `package main
// Generate the code to create a web server
// that uses a single home handler with an index page
// that displays Hello world on the page
`
const response = await fetch(apiUrl, {
method: 'POST',
headers: {
'Content-Type': 'application/json', // Set the appropriate content type
'Private-Token': apiToken, // Use 'Private-Token' for GitLab PAT
},
body: JSON.stringify({
'current_file': {
'file_name': 'main.go',
'content_above_cursor': contentAboveCursor,
'content_below_cursor': '\n}'
},
'stream': stream
}),
});
if (!response.ok) {
throw new Error(`HTTP error! Status: ${response.status}`);
}
const reader = response.body.getReader();
while (true) {
const { done, value } = await reader.read();
if (done) {
console.log('Streaming response is complete.');
break;
}
const rawContent = new TextDecoder().decode(value);
if (rawResponse) {
console.log(rawContent);
} else {
// Split the content into lines
const lines = rawContent.split('\n');
// Initialize an array to store data objects
// const dataObjects = [];
// Iterate through each line
for (let i = 0; i < lines.length; i++) {
// Check if the line contains "data:"
if (lines[i].includes("data:")) {
// Find the next line with the JSON data
const jsonData = lines[i].replace('data: ', '');
// Check if the jsonData is not empty
if (jsonData) {
// Parse the JSON data
const dataObject = JSON.parse(jsonData);
// dataObjects.push(dataObject);
if (dataObject.completion !== undefined) {
// console.log(dataObject.completion);
process.stdout.write(`${dataObject.completion}`);
}
}
} else if (!stream) {
const jsonData = rawContent
const dataObject = JSON.parse(jsonData);
if (dataObject.completion !== undefined) {
process.stdout.write(`${dataObject.completion}`);
}
}
}
}
// You can process or display the streamed data as needed
}
} catch (error) {
console.error('Error:', error);
}
}
fetchDataFromStream();
- Provide your local GitLab Token
- Run
node filename.js
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Vitali Tatarintev