mirror of
https://github.com/yhirose/cpp-httplib.git
synced 2026-11-22 20:11:46 +00:00
Add keep-alive parking (Server::set_keep_alive_parking())
A kept-alive connection holds a worker thread while it waits for its next request, so connections beyond the thread count wait for one to be closed, and some clients get nothing until another connection ends. With keep-alive parking, listen() runs an event loop on the detail::Poller instead of the blocking accept loop. A connection waiting for its next request is parked there and goes back to a worker through the TaskQueue when the request arrives. The handlers, content providers and WebSocket keep their blocking API and still hold a thread while they run. SSLServer supports it too: a worker runs the TLS handshake once the client starts it, and the session stays with the parked connection. It is opt-in and off by default. With it on: - consecutive requests on a connection may run on different threads; - when the queue is full, a connection that has not been served (or has not finished its TLS handshake) is closed as before, while an established one waits in a backlog instead of losing its request; - a parked connection frees its read buffer, and with OpenSSL 3.0.14, 3.1.6, 3.2.2, 3.3.1 or later also its TLS record buffers; - handing a connection over adds roughly 10-20 us to a request; - a subclass that overrides the private process_and_close_socket() is bypassed. On two CPUs with two workers, 50 keep-alive clients of a 2.7 ms handler were all served evenly (the slowest response took about 70 ms instead of 5 s, with 10 clients getting nothing), light requests from 50 clients went from 79k to about 240k per second, and with 10000 idle connections parked, 10 active clients got 95k requests per second where the old loop served none. An idle connection costs under 1 KB, or about 15 KB with TLS. The old accept loop moved unchanged into accept_loop(). KeepAliveConnection holds a parked connection, PlainConnection and SSLServer::TlsConnection add the transport, and SSLServer::start_session() holds the session setup and handshake that both paths use.
This commit is contained in:
1 parent
da42c375ca
commit
530afcc12e
3 files changed
+1139
-130
No files matched your search
@@ -919,6 +919,23 @@ svr.set_keep_alive_max_count(2); // Default is 100, 0 for no limit
|
||||
svr.set_keep_alive_timeout(10); // Default is 5
|
||||
```
|
||||
|
||||
#### Keep-alive parking
|
||||
|
||||
```cpp
|
||||
svr.set_keep_alive_parking(true); // Call it before listen()
|
||||
```
|
||||
|
||||
By default, a kept-alive connection holds a worker thread while it waits for its next request, so connections beyond the thread count wait for one to be closed. With keep-alive parking, a connection waiting for its next request is handed to an event loop instead, and goes back to a worker when the request arrives. An idle connection then costs a socket and a few kilobytes instead of a thread, and connections that outnumber the threads take turns.
|
||||
|
||||
With it:
|
||||
|
||||
- Consecutive requests on a connection may run on different threads.
|
||||
- When the [max queued requests](#max-queued-requests) limit is reached, only a connection that has not been served yet (or, with `SSLServer`, has not finished its TLS handshake) is closed. An established one waits for room.
|
||||
- Each request pays for handing the connection over, so with no more connections than threads and very light handlers, it can serve fewer requests per second.
|
||||
- A handler, a content provider and a WebSocket still hold their thread while they run.
|
||||
- A subclass that overrides the private `process_and_close_socket()` is bypassed.
|
||||
- `SSLServer` runs the TLS handshake on a worker when the client starts it. Until then the connection waits in the event loop and is closed at the keep-alive timeout, not the read timeout, and `ssl_last_error()` is not set for it.
|
||||
|
||||
### Client address behind a reverse proxy
|
||||
|
||||
```cpp
|
||||
@@ -962,7 +979,7 @@ Please see [Server example](https://github.com/yhirose/cpp-httplib/blob/master/e
|
||||
|
||||
When a new task arrives and the queued tasks outnumber the idle threads, a temporary thread is spawned (up to the maximum). A dynamic thread exits automatically once it has waited for the idle timeout without receiving a task. The idle timeout defaults to 3 seconds, configurable via `CPPHTTPLIB_THREAD_POOL_IDLE_TIMEOUT`.
|
||||
|
||||
A connection holds its thread while it is served: between requests up to the keep-alive timeout, through a request body sent slowly but within the read timeout, and for the whole life of a WebSocket. Enough of them leave no thread for anyone else, so in front of untrusted clients, put a reverse proxy that limits connections per client, and reject unwanted WebSocket upgrades in the pre-request handler.
|
||||
A connection holds its thread while it is served: between requests up to the keep-alive timeout (unless [keep-alive parking](#keep-alive-parking) is on), through a request body sent slowly but within the read timeout, and for the whole life of a WebSocket. Enough of them leave no thread for anyone else, so in front of untrusted clients, put a reverse proxy that limits connections per client, and reject unwanted WebSocket upgrades in the pre-request handler.
|
||||
|
||||
If you want to set the thread counts at runtime:
|
||||
|
||||
|
||||
Reference in new issue
Block a user