Better async requests
What does this merge request do and why?
This MR uses aiohttp to manage async requests. Compared to the current way, this approach 2 major benefits
- Better performance.
- Uses a session as a best practice. Current we are using 1 session per request which has some performance hit. For details see doc: https://docs.aiohttp.org/en/stable/http_request_lifecycle.html#using-a-session-as-a-best-practice.
- Use coroutine instead of threading. Current we use
asyncio.to_thread
, in this MR we useasyncio.create_task
.
- Significantly simplify the
ModelHandle
trait implementation. All the common async logic are abstracted away into a separate function which uses aiohttp. This means that each concrete implantation becomes much simpler, we just need to specify the header, json payload and response parsing. Adding a new model becomes simpler now.
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
Merge request checklist
-
I've ran the affected pipeline(s) to validate that nothing is broken. -
Tests added for new functionality. If not, please raise an issue to follow up. -
Documentation added/updated, if needed.
Edited by Hongtao Yang