10. nanoGPT FFN-NPU on Silicon (L2 Hybrid)

If 08 / 09 were L3 single-IP patterns, this page is the deliberate departure: an L2 hybrid offload for putting a real LLM onto a small chip.

Same board (QMTECH XC7Z020), same toolchain, different work split — and the first silicon result in this codebase where the NPU is slower than the CPU, handled honestly.

The Model

examples/silicon_llm/ — nanoGPT shakespeare_char.

Architecture : Transformer decoder, character-level
Parameters   : 10,770,000  (D=384, n_head=6, n_layer=6, V=65, ctx=64)
Training     : RTX 3090, 5000 steps → val 1.59 nats/char
Weights      : INT8 quantized, 10.7 MB → bundled into ARM ELF
Dataset      : TinyShakespeare

Small but a real Transformer — attention, KV cache, FFN. Same shape as GPT-4, just fewer parameters.

Why L2 Hybrid, Not L3

XC7Z020 BRAM is ~140 KB usable. nanoGPT INT8 weights are 6.9 MB. They don't fit. L3 (full graph + weight ROM in one IP) is impossible.

→ L2 hybrid split:

ARM (PS, 666 MHz)
  ├── token + positional embedding
  ├── causal attention + KV cache (variable length, fits ARM)
  ├── LayerNorm
  ├── LM head + argmax sampling

NPU (PL, 90 MHz) — llm_ffn_npu_top
  ├── FFN block: fc1[D→4D] → GELU → fc2[4D→D]
  ├── ap_start once per layer = 6× per token
  └── weights streamed from DDR via m_axi each call

Stateless IP — ARM passes weight pointer + scale + input activation via registers each call. No state in PL fabric. That's what makes KV cache and context length ARM-tunable without re-synthesis.

Silicon Result — Accuracy

demo1: 5 prompts × CPU vs NPU, byte-equal comparison

[0] ROMEO:        CPU: '\nWhat is the wor'  NPU: '\nWhat is the wor'  → MATCH ✓
[1] JULIET:       CPU: '\nWhat is the wor'  NPU: '\nWhat is the wor'  → MATCH ✓
[2] HAMLET:       CPU: '\nWhat shall be s'  NPU: '\nWhat shall be s'  → MATCH ✓
[3] FIRST CITIZEN:CPU: '\nWhat is the wor'  NPU: '\nWhat is the wor'  → MATCH ✓
[4] MERCUTIO:     CPU: '\nThe shall be so'  NPU: '\nThe shall be so'  → MATCH ✓

CPU↔NPU agreement: 5/5    NPU FFN ap_starts: 708
RESULT: PASS

demo2: live 50-character Shakespeare generation

ROMEO:
What is the world the world of the death?
ROMEO:

That output is the result of the NPU running 6 FFN MAC calls per token on actual silicon DSPs.

Silicon Result — Speed (Honest)

This is where it diverges from P06/P07. The NPU is ~2.2× slower than the CPU.

Wrap-safe demo3 per-phase measurement (BENCHMARK.md):

Why the NPU is Slower — Bottleneck Analysis

DMA cost breakdown (per-forward, summed over 6 layers):

98.9 % is the IP's internal compute time. Why is it slow?

HLS C-synth predicted ~55 ms per FFN call; silicon measures 168 ms. A 3× shortfall. The cause is 6 m_axi pointers sharing a single bundle=gmem. The per-call ~1.5 MB INT8 weight burst-reads serialise.

XC7Z020 limit mapping:

ARM is faster not because ARM is computationally strong — but because ARM's cache-line fill pattern is more efficient than 6 contending m_axi masters.

Three Rounds of Retracted Conclusions (Honest)

This project reached the correct measurement through two wrong public conclusions. All recorded:

PMU CCNT is 32-bit @ 666 MHz, wrapping every 6.45 s. Cumulative-sum measurements are always risky. Only per-forward deltas can be trusted.

How to Make It Faster

The four options named in P08 BOARD_NOTES:

1. Split bundle=gmem — give fc1_W / fc2_W / fc1_b / fc2_b their own m_axi masters. Removes serialisation. The single highest- confidence change.
2. Widen HP slave to 128-bit — change BD width. The IP ports already support it.
3. Bigger D — MAC count grows D², DMA grows D — past D ≥ 1024 the DMA tax amortises.
4. ZU-class chip — larger BRAM (cache some weights), wider m_axi, faster PS (Cortex-A53 @ 1.5 GHz → ~2× faster on the ARM-side cost).

Two Silicon Traps (X-1, X-2)

Traps visible only after ap_done was already raising cleanly:

X-1 — Never hardcode the HLS register map

The hand-authored offsets in llm_npu.h disagreed with HLS-generated xllm_*_hw.h. Cause: HLS lays scalar arguments inline between pointer arguments (e.g. fc1_W_scale = 0x28 between fc1_W = 0x1c and fc1_b = 0x30).

→ Fix: copy offsets verbatim from xhw.h. No HLS/Vivado/Vitis changes, ARM .h only.

X-2 — print_callback got raw token IDs

llm_generate(..., print_callback) passed token IDs (0..64) straight to the callback via (char)cast. demo1 (array path) was fine because the caller did LLM_VOCAB_CHARS[id]. demo2 (callback path) printed ASCII garbage.

→ Fix: LLM_VOCAB_CHARS[id] conversion before invoking the callback.

Details: examples/silicon_llm/docs/BOARD_NOTES.md

Reproduce — 3 demos

# Board just-reset
bash examples/silicon_llm/scripts/run_npu_test.sh demo3
python examples/silicon_llm/scripts/parse_benchmark.py
# → examples/silicon_llm/docs/results/BENCHMARK.md auto-updated

# All (demo1 accuracy + demo2 live stream + demo3 wrap-safe benchmark)
bash examples/silicon_llm/scripts/run_npu_test.sh

What This Demo Proves — and Does Not

Proves:

• ✅ The EulerNPU YAML → HLS → Vivado toolchain carries LLMs onto FPGA NPUs
• ✅ A real 10.77 M-parameter nanoGPT fits on a small chip
• ✅ Stateless IP + ARM KV cache + DDR weight streaming works end-to-end
• ✅ 5/5 byte-equal CPU↔NPU MATCH → NPU math correctness

Does not prove:

• ❌ That NPU beats ARM on XC7Z020 — honestly, it loses by 2.2×
• ❌ That the same result holds on other boards — split bundle=gmem + ZU-class will be different
• ❌ Text quality of greedy decoding — val 1.59 nats/char + argmax is naturally repetitive

Detailed References

• examples/silicon_llm/docs/tutorial.md — full flow
• examples/silicon_llm/docs/results/SUMMARY.md — measurements + 3-round retraction log
• examples/silicon_llm/docs/results/BENCHMARK.md — wrap-safe component table
• examples/silicon_llm/docs/BOARD_NOTES.md — X-1 ~ X-5 traps

Wrap-up — The Silicon Ladder

FPGA NPU isn't always faster. But it measures honestly when it is and when it isn't. That measurement is the basis for the next board choice and the next NPU design improvement.