Building tinyGPT: A Transformer Inference Engine From Scratch in C++

Back in April, I set out with a simple goal: build and understand a Transformer language model end to end — not just training it, but everything that happens after, all the way down to the metal.

Where it started

The spark came from contributing to llama.cpp, where I implemented a SIMD-vectorized version of ggml_vec_dot_q4_1_q8_1 — a quantized dot-product kernel used during inference. That PR taught me a lot about one optimized operation. But it also left me with a bigger question: what happens around that one kernel? How does a model actually get from "trained weights" to "text streaming out token by token," and what does it take to make that fast?

I decided the only real way to find out was to build the whole stack myself — training included.

Phase 1: Training the model

I implemented a decoder-only Transformer in PyTorch and trained a ~30M parameter GPT on the TinyStories dataset, on Kaggle, for 2 epochs. Nothing exotic here — the interesting part of this project was never the training run, it was what came after.

Final architecture

Once training was done, I exported the weights and made a deliberate choice: no torch.jit, no ONNX, no existing runtime. I wanted to write the inference engine myself, in C++, from the ground up.

Phase 2: Writing an inference engine from nothing

The engine grew in distinct stages, each one solving a specific bottleneck I hit in the previous one.

• Stage 1 — Get it working: A minimal C++ engine that loads exported .npy weights and runs autoregressive generation. No optimization, just correctness. I built a custom tensor library from scratch—implementing matrix multiplication, multi-head attention, LayerNorm, softmax, GELU, positional encodings, and the complete autoregressive inference pipeline. The engine loads exported NumPy weights directly and is organized as a set of header-only components (tensor.hpp, runner.hpp, parser.hpp).
• Stage 2 — Make the hot path fast: The obvious bottleneck in any Transformer forward pass is matrix multiplication. I added ARM NEON SIMD vectorization to the core tensor ops (operator+, operator*, LayerNorm, softmax, add_bias), and switched matmul to a cache-aware \(32 \times 32\) tiled implementation to cut down on cache misses and make better use of memory bandwidth.
• Stage 3 — Stop recomputing what you already know: Naive autoregressive decoding recomputes attention over the entire sequence at every step, even though most of it hasn't changed. I added KV caching, storing past key/value tensors so each new token only requires computing attention for the new position — turning an \(\mathcal{O}(n^2)\) generation loop into something closer to \(\mathcal{O}(n)\).
• Stage 4 — Use all the cores: Single-threaded SIMD only gets you so far. I added OpenMP parallelism across matmul, multi-head attention, and other compute-heavy ops, with adaptive thresholds so small tensors don't pay threading overhead for no benefit.

Results

I benchmarked the final engine against a naive PyTorch implementation, on both MPS and CPU, on an Apple M1 MacBook Air (mean of 5 runs):

The MPS comparison looks dramatic, but the more honest and interesting number is the CPU-vs-CPU one: even against PyTorch's own CPU path, the custom engine wins on every metric, purely from tiling, SIMD, KV caching, and threading — no exotic tricks, just doing the well-known things correctly and stacking them.

What actually mattered

If I had to rank the four optimizations by impact:

1. KV caching gave the single biggest algorithmic win — it changes the complexity class of generation, not just the constant factor.
2. Tiled matmul mattered because, at this model size, matrix multiplication dominates the per-token cost, and cache misses were the silent killer in the naive version.
3. SIMD compounds on top of tiling — once the memory access pattern is cache-friendly, vectorizing the arithmetic gives another significant improvement.
4. OpenMP was the smallest but easiest win, and only paid off once the per-operation cost was high enough to amortize threading overhead — which is why the adaptive threshold mattered.

The broader lesson: inference engineering isn't really about writing a forward pass. It's about not doing work you don't have to do (KV caching), respecting the memory hierarchy (tiling), and using the hardware you actually have (SIMD + threads). None of these ideas are novel—they're standard techniques used in mature inference systems such as llama.cpp and vLLM—but implementing them myself, and watching the profiler numbers change as each optimization landed, made the "why" click in a way that simply reading papers never did.

Where it's limited (on purpose)

This is a learning project, not a production runtime, and it's honest about that:

• CPU-only, optimized for Apple Silicon (ARM NEON)
• Batch size of 1
• FP32 only — no quantization yet
• Fixed maximum context length

What's next

The natural next step is weight quantization (starting with INT8 and eventually mixed precision), followed by exploring multi-GPU inference to better understand tensor parallelism, communication costs, and memory-bandwidth tradeoffs at larger scales.

Note: AI assistance was used for debugging, error handling, and some boilerplate during development. The model implementation, training pipeline, inference engine architecture, and optimization strategy are my own work.

---