yhirose
9bfd2ddf1c
Bring the READMEs and the docs pages (en and ja) in line with the current header: - Fix the client sample that does not compile, the Transfer-Encoding statement for default headers, the verifier callback comment, the thread pool spawn rule, the sanitize_filename description, and the inverted Accept-Encoding sentence. - Describe recent behaviour: TLS errors behind Read/Write failures, Host validation, Connection: close for a body without length, content_reader rules, refused CONNECT, WebSocketClient proxies, CPPHTTPLIB_INTERIM_RESPONSE_MAX_COUNT, the SSE event size limit, and the payload limit of the Stream API. - Complete the API references of WebSocket, WebSocketClient and the stream:: helpers, and the SSE reconnect rules. - Fix cookbook pages: keychain loading on every backend, received Close status, a multipart sample that trusted the client's file name, an echo sample that sent binary as text, SNI, pre-request handler scope, stop() with WebSocket handlers, and the thread pool notes. - Restore the mutex in the LLM app's Chapter 5 code.
7.4 KiB
title, order
| title | order |
|---|---|
| 7. Reading the llama.cpp Server Source Code | 7 |
Over the course of six chapters, we built a translation desktop app from scratch. We have a working product, but it's ultimately a "learning-oriented" implementation. So how does "production-quality" code differ? Let's read the source code of llama-server, the official server bundled with llama.cpp, and compare.
llama-server is located at llama.cpp/tools/server/. It uses the same cpp-httplib, so you can read the code the same way as in the previous chapters.
7.1 Source Code Location
llama.cpp/
├── tools/server/
│ ├── server.cpp # Entry point
│ ├── server-http.cpp # HTTP server (the layer that uses cpp-httplib)
│ ├── server-context.cpp # Inference and slot management
│ └── ...
└── vendor/cpp-httplib/
└── httplib.h # cpp-httplib (bundled version)
The implementation is split across several files by role. Together they run to more than 20,000 lines, but once you understand the structure, you can narrow down the parts worth reading.
7.2 OpenAI-Compatible API
The biggest difference between the server we built and llama-server is the API design.
Our API:
POST /translate → {"translation": "..."}
POST /translate/stream → SSE: data: "token"
llama-server's API:
POST /v1/chat/completions → OpenAI-compatible JSON
POST /v1/completions → OpenAI-compatible JSON
POST /v1/embeddings → Text embedding vectors
llama-server conforms to OpenAI's API specification. This means OpenAI's official client libraries (such as the Python openai package) work out of the box.
# Example of connecting to llama-server with the OpenAI client
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="dummy")
response = client.chat.completions.create(
model="local-model",
messages=[{"role": "user", "content": "Hello!"}]
)
Compatibility with existing tools and libraries is a big design decision. We designed a simple translation-specific API, but if you're building a general-purpose server, OpenAI compatibility has become the de facto standard.
7.3 Concurrent Request Handling
Our server runs inference one request at a time. If another request arrives while a translation is in progress, it waits until the previous inference finishes. This is fine for a desktop app used by one person, but it becomes a problem for a server shared by multiple users.
llama-server handles concurrent requests through a mechanism called slots.
The key point is that tokens from each slot are not inferred one by one in sequence, but rather all at once in a single batch. GPUs excel at parallel processing, so processing two users simultaneously takes almost the same time as processing one. This is called "continuous batching."
In our server, cpp-httplib's thread pool serves each connection on its own worker thread, so requests on different connections are handled concurrently. Inference, however, is serialized by llm_mutex, so only one llm.chat() runs at a time. llama-server consolidates this inference step into a shared batch processing loop.
7.4 Differences in SSE Format
The streaming mechanism itself is the same (set_chunked_content_provider + SSE), but the data format differs.
Our format:
data: "去年の"
data: "春に"
data: [DONE]
llama-server (OpenAI-compatible):
data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"delta":{"content":"去年の"}}]}
data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"delta":{"content":"春に"}}]}
data: [DONE]
Our format simply sends the tokens. Because llama-server follows the OpenAI specification, even a single token comes wrapped in JSON. It may look verbose, but it includes useful information for clients, like an id to identify the request and a finish_reason to indicate why generation stopped.
7.5 KV Cache Reuse
In our server, we process the entire prompt from scratch on every request. Our translation app's prompt is short ("Translate the following text to ja..." + input text), so this isn't a problem.
llama-server reuses the KV cache for the prefix portion when a request shares a common prompt prefix with a previous request.
For chatbots that send a long system prompt and few-shot examples with every request, this alone dramatically reduces response time. The difference is night and day: processing several thousand tokens of system prompt every time versus reading them from cache in an instant.
For our translation app, where the system prompt is just a single sentence, the benefit is limited. However, it's an optimization worth keeping in mind when applying this to your own applications.
7.6 Structured Output
Since our translation API returns plain text, there was no need to constrain the output format. But what if you want the LLM to respond in JSON?
Prompt: Analyze the sentiment of the following text and return it as JSON.
LLM output (expected): {"sentiment": "positive", "score": 0.8}
LLM output (reality): Here are the results of the sentiment analysis. {"sentiment": ...
LLMs sometimes ignore instructions and add extraneous text. llama-server solves this problem with grammar constraints.
curl http://localhost:8080/v1/chat/completions \
-d '{
"messages": [{"role": "user", "content": "Analyze sentiment..."}],
"json_schema": {
"type": "object",
"properties": {
"sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
"score": {"type": "number"}
},
"required": ["sentiment", "score"]
}
}'
When you specify json_schema, tokens that don't conform to the grammar are excluded during token generation. This guarantees that the output is always valid JSON, so there's no need to worry about json::parse failing.
When embedding LLMs into applications, whether you can reliably parse the output directly impacts reliability. Grammar constraints are unnecessary for free-text output like translation, but they're essential for use cases where you need to return structured data as an API response.
7.7 Summary
Let's organize the differences we've covered.
| Aspect | Our Server | llama-server |
|---|---|---|
| API design | Translation-specific | OpenAI-compatible |
| Concurrent requests | Sequential processing | Slots + continuous batching |
| SSE format | Tokens only | OpenAI-compatible JSON |
| KV cache | Cleared each time | Prefix reuse |
| Structured output | None | JSON Schema / grammar constraints |
| Code size | ~200 lines | Over 20,000 lines |
Our code is simple because of the assumption that "one person uses it as a desktop app." If you're building a server for multiple users or one that integrates with the existing ecosystem, llama-server's design serves as a valuable reference.
Conversely, even 200 lines of code is enough to make a fully functional translation app. I hope this code reading exercise has also conveyed the value of "building only what you need."
Next Chapter
In the next chapter, we'll cover the key points for swapping in your own library and customizing the app to make it truly yours.
Next: Making It Your Own