mirror of
https://github.com/yhirose/cpp-httplib.git
synced 2026-10-19 11:19:36 +00:00
Fix README and docs pages that no longer match the code
Bring the READMEs and the docs pages (en and ja) in line with the current header: - Fix the client sample that does not compile, the Transfer-Encoding statement for default headers, the verifier callback comment, the thread pool spawn rule, the sanitize_filename description, and the inverted Accept-Encoding sentence. - Describe recent behaviour: TLS errors behind Read/Write failures, Host validation, Connection: close for a body without length, content_reader rules, refused CONNECT, WebSocketClient proxies, CPPHTTPLIB_INTERIM_RESPONSE_MAX_COUNT, the SSE event size limit, and the payload limit of the Stream API. - Complete the API references of WebSocket, WebSocketClient and the stream:: helpers, and the SSE reconnect rules. - Fix cookbook pages: keychain loading on every backend, received Close status, a multipart sample that trusted the client's file name, an echo sample that sent binary as text, SNI, pre-request handler scope, stop() with WebSocket handlers, and the thread pool notes. - Restore the mutex in the LLM app's Chapter 5 code.
This commit is contained in:
1 parent
94cf1b5c9a
commit
9bfd2ddf1c
77 files changed
+378
-145
No files matched your search
@@ -59,7 +59,7 @@ Compatibility with existing tools and libraries is a big design decision. We des
|
||||
|
||||
## 7.3 Concurrent Request Handling
|
||||
|
||||
Our server processes requests one at a time. If another request arrives while a translation is in progress, it waits until the previous inference finishes. This is fine for a desktop app used by one person, but it becomes a problem for a server shared by multiple users.
|
||||
Our server runs inference one request at a time. If another request arrives while a translation is in progress, it waits until the previous inference finishes. This is fine for a desktop app used by one person, but it becomes a problem for a server shared by multiple users.
|
||||
|
||||
`llama-server` handles concurrent requests through a mechanism called **slots**.
|
||||
|
||||
@@ -67,7 +67,7 @@ Our server processes requests one at a time. If another request arrives while a
|
||||
|
||||
The key point is that tokens from each slot are not inferred **one by one in sequence**, but rather **all at once in a single batch**. GPUs excel at parallel processing, so processing two users simultaneously takes almost the same time as processing one. This is called "continuous batching."
|
||||
|
||||
In our server, cpp-httplib's thread pool assigns one thread per request, but the inference itself runs single-threaded inside `llm.chat()`. `llama-server` consolidates this inference step into a shared batch processing loop.
|
||||
In our server, cpp-httplib's thread pool serves each connection on its own worker thread, so requests on different connections are handled concurrently. Inference, however, is serialized by `llm_mutex`, so only one `llm.chat()` runs at a time. `llama-server` consolidates this inference step into a shared batch processing loop.
|
||||
|
||||
## 7.4 Differences in SSE Format
|
||||
|
||||
|
||||
Reference in new issue
Block a user