A systems view of local inference: tokenization, orchestration, memory movement, kernels, and why an NPU result is more than a single throughput number.
Inference is a pipeline
Moving one operator to an accelerator does not move the model. Tokenization, model orchestration, memory transfers, unsupported operators, sampling, and API streaming can remain on the CPU even when the compute-heavy path uses an NPU.
Local inference boundary
Tokenizer / API→
Host orchestration→
NPU kernels→
KV + state→
Sampling / stream
Measure the boundary you changed
End-to-end latency is important, but it can hide whether a kernel improvement actually helped. Keep both component-level measurements and end-to-end measurements, then describe which part of the pipeline each number represents.
Correctness is a performance prerequisite
Fast output from a numerically wrong path is not an optimization. A useful accelerator benchmark states the model/checkpoint, precision, input shape or sequence conditions, warmup method, and correctness check used before the timing run.