01. The appealing first hypothesis
The compiler's analytics.json surfaced an internal DRAM bandwidth planning figure of 51.2 GB/s. Dividing that number by the as-stored 893 MB external weight file yielded approximately 57.3 tokens/sec—close to an early peak. It seemed to support a hard bandwidth ceiling.
But dimensional plausibility is not a causal test. The calculation predicted that halving the bytes read per token would increase throughput, and that increasing genuine sequence work per dispatch would remain constrained by the same byte rate.
02. Counter-experiments
Two observations contradicted that simple model. First, reduced-size LUT2 weights did not produce the corresponding speedup. Second, sequence/K sweeps moved aggregate useful work more rapidly than the supposed throughput limit permitted. The original “57.3 tok/s ceiling” was retracted as a K=1 coincidence, not treated as an ANE architectural law.
Subsequent work refined the regime analysis: fixed per-call dispatch costs matter at shallow graphs, but at 28-layer production depth per-layer compute dominates more of the total step time. The productive lever is demonstrated multi-lane execution, not arithmetic on weight bytes alone.
03. The measured B sweep
| Real batch shape | Aggregate tok/s | Relative to B=1 | Interpretation |
|---|---|---|---|
| B=1 | 45.4 | 1.00× | Reference |
| B=2 | 68.1 | 1.50× | Amortized work |
| B=4 | 102.1 | 2.25× | Efficiency knee |
| B=8 | 125.8 | 2.77× | Higher memory cost |
These are aggregate throughputs over five-round interleaved steady-state measurements. The B=4 value is the practical default for the 8 GB machine. It is not an individual-stream token rate, nor an end-to-end app UX measurement.
04. What qualifies as a speed result
ACCEPT only if:
1. compiled input tensor has real [B, ...] shape
2. elapsed time scales with actual B work
3. reported tokens = work actually executed
4. source artifact and device are named
REJECT:
fixed [1,1] model + posthoc divide by K
synthetic tokens from a benchmark argument
projected tokens/sec treated as measuredThe project rejected a K=16 throughput claim derived from a shape-locked [1,1] compiled model and a post-hoc divisor. Its reported wall time was flat while its headline token rate increased, indicating arithmetic rather than additional hardware work.
05. What to take forward
On the tested Qwen decode substrate, real batch geometry increases aggregate throughput. At shallow depth dispatch overhead is material; at 28 layers computation load changes the limiting regime. No single constant extracted from a compiler report should be published as a universal throughput ceiling.
Source evidence: ane-re/docs/FINDINGS.md §§5, 7–9, 13; ANE-RE decision ledger D001, D028, D040, D143, D264. Performance numbers depend on the recorded device, software and thermal conditions.