Add Server::set_keep_alive_parking()

A kept-alive connection holds a worker thread while it waits for its
next request, so connections beyond the thread count wait for one to be
closed, and with a pool smaller than the number of clients some clients
get nothing until the keep-alive timeout or the max count ends another
connection.

With keep-alive parking, listen() runs an event loop on the detail::Poller
instead of the blocking accept loop. A connection waiting for its next
request is parked there, and goes back to a worker through the TaskQueue
when the request arrives. The handlers, content providers and WebSocket
keep their blocking API and still hold a thread while they run.

It is opt-in and off by default. With it on:
- consecutive requests on a connection may run on different threads;
- when the queue is full, a connection that has not been served yet is
  closed as before, but an established one waits in a backlog instead of
  losing its request;
- the loop owns the listening socket while it watches it, and stop()
  hands it over instead of closing it under the poller;
- SSLServer ignores the setting for now, and an override of
  process_and_close_socket() is not called.

On two CPUs with two workers, 50 keep-alive clients of a 2.7 ms handler
were all served evenly (the slowest response took 70 ms instead of 5 s
with 10 clients getting nothing), light requests from 50 clients went
from 82k to 248k per second, and 10 clients were served after 10000
idle connections that had stopped the old loop. With no more clients
than workers, handing the connection over costs about half of the
requests per second of a very light handler.

The old accept loop moved unchanged into accept_loop().
process_server_socket_core() now takes the stream, and
drain_and_close_socket() takes how long to wait for the peer.
This commit is contained in:
yhirose committed 2026-11-20 23:39:33 -05:00
1 parent 1b914e1597
commit 58bd9a1900
3 files changed
+735 -105

No files matched your search

+17 -1
View File
@@ -919,6 +919,22 @@ svr.set_keep_alive_max_count(2); // Default is 100, 0 for no limit
svr.set_keep_alive_timeout(10); // Default is 5
```
#### Keep-alive parking
```cpp
svr.set_keep_alive_parking(true); // Call it before listen()
```
By default, a kept-alive connection holds a worker thread while it waits for its next request, so connections beyond the thread count wait for one to be closed. With keep-alive parking, a connection waiting for its next request is handed to an event loop instead, and goes back to a worker when the request arrives. An idle connection then costs a socket and a few kilobytes instead of a thread, and connections that outnumber the threads take turns.
With it:
- Consecutive requests on a connection may run on different threads.
- When the [max queued requests](#max-queued-requests) limit is reached, only a connection that has not been served yet is closed. An established one waits for room.
- Each request pays for handing the connection over, so with no more connections than threads and very light handlers, it can serve fewer requests per second.
- A handler, a content provider and a WebSocket still hold their thread while they run.
- `SSLServer` ignores it for now, and an override of `process_and_close_socket()` is not called.
### Client address behind a reverse proxy
```cpp
@@ -962,7 +978,7 @@ Please see [Server example](https://github.com/yhirose/cpp-httplib/blob/master/e
When a new task arrives and the queued tasks outnumber the idle threads, a temporary thread is spawned (up to the maximum). A dynamic thread exits automatically once it has waited for the idle timeout without receiving a task. The idle timeout defaults to 3 seconds, configurable via `CPPHTTPLIB_THREAD_POOL_IDLE_TIMEOUT`.
A connection holds its thread while it is served: between requests up to the keep-alive timeout, through a request body sent slowly but within the read timeout, and for the whole life of a WebSocket. Enough of them leave no thread for anyone else, so in front of untrusted clients, put a reverse proxy that limits connections per client, and reject unwanted WebSocket upgrades in the pre-request handler.
A connection holds its thread while it is served: between requests up to the keep-alive timeout (unless [keep-alive parking](#keep-alive-parking) is on), through a request body sent slowly but within the read timeout, and for the whole life of a WebSocket. Enough of them leave no thread for anyone else, so in front of untrusted clients, put a reverse proxy that limits connections per client, and reject unwanted WebSocket upgrades in the pre-request handler.
If you want to set the thread counts at runtime: