AsanagiDB is a graph database I’ve been building in Zig — LMDB for storage, a Gremlin traversal engine on top, no JVM. It’s currently in v2.0. I run it in production for my own projects daily. Version 2.0 replaced the networking layer, and this is what that looked like.
Why I did it
The old server was one OS thread per connection. The accept loop blocked on
accept(), spawned a thread for the connection, and that thread lived for the
connection’s entire life. A single semaphore capped concurrency at 256 —
except that same semaphore was also standing in for the LMDB read-transaction
budget (MAX_READERS is 512 process-wide), so “how many clients can connect”
and “how many queries can run” were the same number, which is wrong.
It held up until one of the things built on top of it — a listing sync that
streams thousands of records through the Gremlin write path — started losing
connections under load. The engine would close a socket mid-write
(write: broken pipe), the client would reconnect, and because each request
was effectively its own short-lived connection, a few hundred reconnects a
second turned into a stampede the 256-slot cap couldn’t absorb. I shipped a
client-side throttle and a circuit breaker, and those helped, but they were
treating a symptom. The model was the problem: a thread per connection, and a
cap that conflated two unrelated limits.
Why libxev, and why not rewrite everything
I’d already decided the target was an event-loop reactor: one small pool of threads multiplexing many sockets, with the execution budget as a separate layer on top. libxev seemed like the natural fit — cross-platform, small enough to reason about.
I did look at alternatives partway through, when the iteration count got
frustrating. The conclusion was that the problem wasn’t libxev — it was the
immaturity of Zig 0.16’s own std.Io networking, which I was leaning on for
the pieces around it. Swapping event-loop libraries wouldn’t have fixed that.
So I stayed.
The other early decision that saved a lot of time: not rewriting the protocol handlers. AsanagiDB speaks three wire protocols — a WebSocket/GraphSON+GraphBinary path for Gremlin clients, a binary protocol for its own SDKs, and a plain JSON-line path. Every one already operated on a complete in-memory buffer — one assembled WebSocket frame, one line — not a live socket. They took a reader over a byte slice and wrote to a stream. That meant the reactor didn’t need to understand any of them. It needed to assemble a complete message and hand it over, with a buffer to write into instead of a socket. The handler code barely changed.
The Implementation
Three interfaces, each with contracts asserted in debug builds:
StreamTransport — a bidirectional byte stream: read, write (which
queues), close, wantWrite, plus an explicit spot for the socket options
protocols actually need (setNoDelay, peerAddress, peer credentials for the
Unix socket). The point is that no protocol code ever touches a file
descriptor or casts through the abstraction to reach a socket option — the old
code did exactly that in one place to set TCP_NODELAY. Plain TCP and plain Unix socket each own a socket directly; TLS is a decorator that wraps one of those and runs the handshake and record layer on top.
Framer — push(bytes) returns a batch of complete messages plus how many bytes it consumed, or “need more”, or “protocol error”. One per protocol. The parsing logic already existed; WebSocket is the fiddly one — a frame header with the payload length encoded one of three ways depending on size, an XOR mask over the payload, and single messages that can arrive split across several frames. What changed is that it now runs on partial input: the Framer keeps whatever bytes are left over from the last read and emits a message only once it has a whole one.
Executor — submit(job) returns accepted, rejected-queue-full, or
rejected-shutting-down. One implementation: a fixed pool of worker threads
behind a bounded queue with a reject policy.
The flow: the loop thread accepts and creates a heap Conn — its file descriptor (fd), read buffer, a write queue of owned slices with an offset, framer carry-over, auth state, an in-flight counter, a close_after_flush flag. A read completion runs the bytes through the transport into the framer; each complete message becomes a job submitted to the pool. If the pool rejects it, the connection gets a Gremlin 597 “too busy” envelope queued and is marked to close after that flushes — backpressure, not a dropped socket, which is the specific thing that bit me. A worker thread decodes, runs the traversal (the interpreter is unchanged), encodes the response, appends it to the connection’s write queue under a lock, and signals the loop. The loop wakes, writes what’s pending, and on the last byte closes the connection if it was flagged.
Invariants worth stating because they’re what keeps this debuggable:
- A connection is in exactly one of: reading, submitted, writing, closing.
- Pool size plus reserved admin readers must stay under
MAX_READERS. The pool defaults to 64 workers, queue 1024; both are configurable. - If a connection closes while one of its jobs is still running, the job is flagged cancelled. The worker, when it finishes, has to commit or roll back its LMDB transaction and then drop the response — it must not touch the freed connection. Getting that wrong is a use-after-free or a leaked reader slot, and both showed up before the rule was explicit.
The original Zig file went from about 1,775 lines to about 200.
libxev specifics
A few things that weren’t obvious from the outside:
xev.TCP.initis IP-only. For the Unix-socket listener I create theAF_UNIXsocket withstd.posix, bind and listen there, and hand the raw fd toxev.TCP.initFd. From that point it’s the same accept/read/write/close path as TCP;setNoDelayjust becomes a no-op.xev.ThreadPool.schedule()returnsvoid. It takes an unbounded queue and can’t tell you it’s full. Since the whole point of the execution layer is a bounded queue that rejects past a threshold, I hand-rolled the pool — worker threads, a bounded ring buffer, an atomic depth counter, a reject path — rather than usexev.ThreadPool.- On Linux I pin the epoll backend. io_uring is there and libxev supports it, but I haven’t validated this workload against it yet, and epoll is the one I can reason about under load.
- The first thing I built was a throwaway test that spins up a
Loop, aTimer, and anAsync, and asserts the callbacks fire. On kqueue and epoll it passed immediately and I moved on. That turned out to matter, because Windows didn’t seem to like it.
Windows
libxev’s Windows backend is IOCP. On paper the reactor should just use it. In practice:
The engine started, initialized the loop, the async handle, the timer,
registered the listening socket, and printed “listening”. Then loop.run()
panicked — reached unreachable code — before a single client connected.
Reading libxev’s iocp.zig, the accept-submission path has around fifteen
catch unreachable and assert sites: associate_fd(...) catch unreachable,
getsockname / getsockopt asserts, WSAEINVAL => unreachable. My thinking is that it hasn’t been exercised against a listening socket that libxev didn’t
create itself. Fixing the first landmine just moves you to the next one, and
this isn’t code I wanted to debug blind on a release deadline.
I fell back to Zig’s own std.Io.net, which is AFD-ioctl based. That served
requests fine — and then panicked on Ctrl-C, in std’s own
.CANCELLED => unreachable.
So Windows ended up on raw ws2_32 — blocking accept / recv / send, one
thread per connection, which is exactly the model 2.0 was moving away from
everywhere else. The saving grace is the seam: that blocking loop feeds the
same framer and dispatch path and writes through the same buffer
abstraction as the epoll and kqueue reactors, so every wire protocol behaves
identically regardless of what’s underneath. Server.start picks the
implementation at comptime. Windows keeps thread-per-connection; it just
doesn’t get the scaling win, which is an acceptable trade for the deployments
that actually run on Windows — dev boxes, small on-prem.
One bug from that path is worth mentioning because it wasted an afternoon:
bind() kept returning WSAEADDRNOTAVAIL for 127.0.0.1. The address bytes
were going into sockaddr_in via a big-endian readInt, which on a
little-endian host stored them reversed. @bitCast of the four-byte array
straight into the field fixed it.
Where it landed
- Linux/epoll: roughly 72,000 connection open/close cycles and between four and six thousand concurrent connections doing real traversals, with flat memory and no file-descriptor leaks. This is the case that matters — it’s the incident’s own failure mode, and it no longer fails.
- macOS/kqueue: in daily use; a few hundred concurrent connections with churn, clean.
- Windows: functionally verified — 200 concurrent connections, reads and writes, zero errors, flat handle count across repeated churn, clean shutdown — in an emulated VM. Not load-tested at scale; I don’t have a real Windows host for that.
What this explicitly does not fix: write throughput. LMDB has a single writer, so concurrent write traversals still serialize — I measure a couple hundred write commits a second regardless of how many connections are open. The reactor makes the engine stop falling over under connection load; it doesn’t make LMDB writes parallel, and nothing short of a different storage model would.
Protocol behavior was checked against the previous release with the stock
clients — gremlin-go, the Gremlin Console over GraphBinary, the binary SDK, and nc for the JSON path — same results before and after.
If you’re doing something similar
The thing that made this tractable was discovering the handlers already took
complete buffers. If yours do too — if there’s a layer that parses a request
from a []const u8 and writes a response — the reactor is a re-plumb, not a
rewrite. Find that line first.
libxev on epoll and kqueue did what it says. The Zig 0.16 standard library around it is younger, and the Windows story is genuinely rough — if you need Windows and you need the event loop, budget real time for it or plan a blocking fallback from the start.
AsanagiDB 2.0 is at asanagi.ai; I’d rather hear from you where it breaks, than not.