tqcq/llama.cpp
0
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2025-08-12 19:37:53 -04:00
Code Issues Packages Projects Releases Wiki Activity

server : refactor multitask handling (#9274)

Browse Source 
* server : remove multitask from server_task

* refactor completions handler

* fix embeddings

* use res_ok everywhere

* small change for handle_slots_action

* use unordered_set everywhere

* (try) fix test

* no more "mutable" lambda

* Apply suggestions from code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* use deque

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit is contained in:
Xuan Son Nguyen
2024-09-02 17:11:51 +02:00
committed by GitHub
parent b60074f1c2
commit 6e7d133a5f
5 changed files with 365 additions and 462 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
examples/server/tests/features/steps/steps.py
View File

@@ -818,7 +818,7 @@ async def concurrent_requests(context, f_completion, *args, **kwargs):
for prompt_no in range(context.n_prompts):
shifted_args = [context.prompts.pop(), seeds[prompt_no], *args]
context.concurrent_tasks.append(asyncio.create_task(f_completion(*shifted_args, **kwargs)))
await asyncio.sleep(0.1)
await asyncio.sleep(0.01)
@step('the slot {slot_id:d} is saved with filename "{filename}"')