Three C++20 systems projects built for quant/HFT Systems + Trading roles. They exist to answer a question a friend in the industry put to me bluntly: in an era where anyone can generate a competent order book in ninety seconds, what does writing one actually prove?

Nothing, on its own. So these projects are built on a different premise — the code is the artefact, the evidence is the product. What cannot be generated cheaply is the empirical record of engaging with real hardware: measurements taken on a specific machine, predictions that turned out wrong, and the reasoning that closed the gap. A model can assert that false sharing costs performance. It cannot tell you that on my M5 Pro the coherence granule measures 64 bytes while sysctl reports 128 and libc++ reports 256 — because that took an experiment.

Every claim below is either measured or explicitly labelled as unverified. All numbers come from an Apple M5 Pro (arm64), Apple clang 21, -O2 -mcpu=native, and are reproduced by make bench in each repository.

Zero-Allocation NASDAQ ITCH Feed Handler

quant-itch-feed-handler

A parser for NASDAQ TotalView-ITCH 5.0 that reconstructs a limit order book at 25.8 million messages/second on one core — roughly 2.6× the full-feed peak rate of the real exchange — with zero heap allocations on the hot path.

The claim is enforced, not asserted

“No allocation on the hot path” is the easiest claim in systems programming to believe about your own code and be wrong. A std::string outgrowing its small-string buffer, a vector that grows once at message 40,000 — neither is visible by reading the code. So the project replaces the global operator new and aborts if anything allocates while the hot loop is armed:

That line is produced by a test that fails loudly, across 5,000,000 messages. It is worth more than any paragraph of prose.

Where the time actually goes

Cost attribution per message. The parser is not the bottleneck.

Stage	ns/msg	Share
Framing (length prefixes)	2.22	5.7%
Field decoding (byte swaps, dispatch)	4.11	10.6%
Order book update	32.37	83.6%
Total	38.70

Decoding — the part that looks like the hard work, with its byte swaps and unaligned loads — is 10% of the cost. A single end-to-end figure would have hidden that.

What surprised me: my hash table lost to std::unordered_map

The entire point of a flat, open-addressed table is beating the node-based standard container. On first measurement it was 1.17× slower.

The cause: libc++’s std::hash<uint64_t> is the identity function. For exchange-assigned sequential order references that is not a weakness — it is a free, perfect hash. My “strong” splitmix64 hash was paying three multiplies to destroy the very structure that made the workload easy. So I made the hash a policy and measured three:

ns/op, 500k live orders, 90% hit / 10% miss. Lower is better.

hash	insert	find	churn	avg probes	worst run
identity	1.64	8451.11	118107.72	11941.94	500,000
fibonacci ← used	2.56	3.52	2.38	1.05	2
splitmix64	8.42	7.99	15.88	1.54	35
std::unordered_map	23.75	6.95	10.81	—	chained

A 2,399× spread from changing nothing but the hash function — and the “obvious” choice for sequential keys was the worst of the three by three orders of magnitude. Identity hashing gives perfect hits, but packs sequential keys into a contiguous 100%-full block, so a miss has no empty slot to terminate on and walks up to 500,000 slots.

The part I find genuinely interesting: std::unordered_map uses the same identity hash and is unharmed, because chaining has no probe run to walk. Open addressing is what makes hash quality load-bearing. The two design choices interact, and reasoning about either alone gives the wrong answer.

With Fibonacci hashing the final result is 9.3× insert, 1.97× lookup, 4.55× churn and 1.67× less memory than std::unordered_map — and zero allocations against one per insert.

A 431,072× algorithmic win that bought 1.05×

Instrumenting the book’s price-level search showed it examining an average of 431,072 levels per touch update — ITCH carries prices at $0.0001 granularity while quotes are penny-granular, so 99% of the array is permanently empty. Replacing the linear scan with a hierarchical bitmap and hardware bit-scan (CLZ/CTZ) took that to 1.00.

End-to-end, that was worth 1.05×. Two reasons, both worth internalising: touch updates are rare relative to total messages, so Amdahl’s law caps the win; and a linear scan over contiguous, mostly-zero memory streams at many bytes per cycle with no dependent loads. Step counts are not costs. The bitmap is still the right design because it bounds the worst case — but the honest headline is 4%, and quoting 431,000× would not survive a follow-up question.

Lock-Free SPSC Ring Buffer

quant-spsc-ring-buffer

A single-producer/single-consumer queue — the pipe between a network thread and a strategy thread. The interesting result is not the queue, but what measuring it properly revealed.

The benchmark that lied to me

My first version reported that the lock-free queue was slower than a mutex, and that deliberately introducing false sharing made it faster. Both results were real and reproducible. Neither was a fact about the queue.

Two measurements diagnosed it. The queue alone, single-threaded: 3.05 ns/msg. A bare cross-core cache-line ping-pong with no queue at all: ~88 ns one way. The benchmark was measuring the CPU interconnect and attributing it to my data structure. Instrumenting the consumer confirmed it — the drain batch averaged 1.6 messages, so the queue was always near-empty and every message forced a cache line to migrate between cores.

Decomposing into three regimes with different bottlenecks.

Regime	ns/msg	What it measures
[A] single thread, push+pop	3.05	the queue’s instruction path
[A] mutex + same ring, uncontended	15.51	uncontended lock cost — 5.1×
[B] cross-core, 1 msg at a time	98.07	the interconnect — 96% of cost
[B] 8-byte padding (false sharing)	35.87	faster, see below
[C] cross-core, batch 512	1.07	37× over [B]

The lock-free algorithm is worth ~5× over a mutex. Batching is worth 37×. An engineer who has only read about lock-free queues optimises the wrong thing. This is also why production feed handlers hand strategies a batch of ticks rather than one at a time.

And false sharing won because in lockstep producer/consumer the line must migrate every message regardless of padding — so co-locating head and tail means one transfer serves both directions instead of two. An optimisation’s value depends on the access pattern, not on whether it appears on a best-practices list.

Measuring the coherence granule instead of assuming it

“Pad your atomics to 64 bytes” is an x86 fact that got copied into everyone’s mental model. On this machine three authoritative sources disagree:

source	says
sysctl hw.cachelinesize	128
std::hardware_destructive_interference_size	256
measured	64

Only one of those is an experiment. Two threads, two independent atomic counters, nothing varying but the byte separation: a clean step at 64 bytes and a 7.0× penalty below it (17.07 → 2.44 ns/increment).

Memory ordering, at the instruction level

The claim “acquire/release is free on x86 but not on ARM” is repeated constantly and almost never demonstrated. Compiling the same store three ways, on AArch64:

On x86-64 all three stores emit the same mov. That asymmetry is a career hazard, not a curiosity: forgetting a release on x86 produces byte-identical machine code, so the bug compiles, runs, passes every test — then corrupts data the first time it runs on ARM, which today means an Apple Silicon laptop or AWS Graviton.

A sanitizer that never fires proves nothing, so I broke the queue on purpose — downgraded the producer’s release to relaxed — and confirmed ThreadSanitizer catches it. The correct queue is TSan-clean over 2,000,000 messages.

O(1) Pool Allocator

quant-pool-allocator

A fixed-size block allocator and a bump arena for hot paths that cannot afford malloc. The pool allocates in 2.62 ns against malloc’s 20.37 ns — but that ratio is nearly irrelevant.

The tail is the argument, not the mean

ns per allocation. The last column is the headline.

	p50	p99	p99.9	p99.9 / p50
malloc	9.77	343.36	1464.34	150×
pool	3.25	6.52	39.06	12×

At the median the pool is 3× faster. At p99.9 it is ~37× faster. malloc’s distribution is visibly bimodal — the second mode is the arena growing via mmap: a syscall, a page-table update, and a page fault. Microseconds, not nanoseconds, at an unpredictable moment — and that moment is disproportionately likely to be the busy one, because that is when allocation rate peaks.

The pool’s p99.9 is only 12× its median because there is no second code path to fall into. If you benchmark the mean, you conclude this allocator is barely worth the trouble. That is the trap.

What surprised me: “malloc scatters your data” is largely false

I expected the big win to be layout. Walking a 1,048,576-node intrusive list of 64-byte orders (64 MiB, far exceeding L2):

configuration	ns/node	vs best
pool, allocated and walked in order	1.53	—
malloc, scattered by interleaved allocations	3.20	2.1×
pool memory, walked in random order	126.30	82.7×

The allocator was worth only 2.1×. malloc is a size-class allocator, so a million same-sized objects land in a compact ascending range even with decoys interleaved — not adjacent, but ordered, and an ascending walk is still prefetchable.

Access pattern was worth 83×. Same pool, same nodes, same memory — only the link order shuffled. 126 ns is approximately this machine’s DRAM latency, which is the tell: randomised pointer chasing defeats the prefetcher completely, because the address of the next load is the result of the current one.

So the engineering conclusion is not “use a pool allocator” but “make the traversal sequential” — which is precisely why the ITCH order book links orders with 32-bit indices into a flat array rather than with pointers.

What these projects do not claim

A portfolio that overclaims is worse than one that underclaims, so the repositories state their negative space explicitly:

• These are not production trading systems — no matching engine, risk checks, multicast framing, gap recovery or feed arbitration.
• macOS provides no thread-to-core pinning on Apple Silicon, so every cross-core number is labelled indicative rather than publishable. The single-threaded and algorithmic numbers are stable.
• No hardware performance counters. perf is Linux-only and Apple Silicon exposes no supported equivalent, so there are no IPC or cache-miss-rate figures — only wall-clock timing and counters the code maintains itself.
• The prefault() page-fault benefit was not reproducible here; cold beat prefaulted about as often as it lost. The call is kept because the effect is established on Linux, but no number is claimed for it.

A note on measurement

Before any of these numbers meant anything, I had to establish what the clock could resolve. CNTFRQ_EL0 on Apple Silicon reports 1 GHz, which invites you to believe you have nanosecond resolution. The underlying counter is 24 MHz, so the real granularity is 41.67 ns, and every macOS timing API inherits that floor.

Operations costing 2–20 ns therefore cannot be timed individually on this machine. The harness detects this and switches to batched timing; the latency benchmark explicitly refuses to print a p99.9 when the clock is too coarse to support one. Declining to publish a number turned out to be a stronger signal than publishing it.