Hands-On: Reproducing the WhisperKit Conversion Pipeline

Part of the ANE Knowledge Base. This doc records an actual end-to-end run of the pipeline studied in doc 06, executed on 2026-07-12. Scripts, pinned environment and captured evidence: experiments/whisperkit-conversion/.

1. Setup#

Hardware	Apple M4 Pro (macOS 26.5.1)
Toolchain	Python 3.12.8 via uv venv; coremlcompiler from Xcode beta
Package	whisperkit 0.4.2 installed from our pinned submodule (references/whisperkittools, commit 84f77a8)
Key deps (resolved)	argmaxtools 0.1.23 (identical to our vendored snapshot), coremltools 9.0, torch 2.5.0, transformers 4.53.0
Model	openai/whisper-tiny (39M params — smallest full pipeline exerciser)

uv venv --python 3.12 .venv                                    # torch 2.5.0 needs Python <= 3.12
VIRTUAL_ENV=$PWD/.venv uv pip install ../../references/whisperkittools
export HF_HOME="$PWD/hf-cache"                                 # self-contained HF cache
whisperkit-generate-model --model-version openai/whisper-tiny --output-dir ./output

Notes from the run:

• A non-editable install keeps the submodule pristine (editable would drop egg-info inside it).
• The CLI works because setup.py uses find_packages(), which ships tests/ and scripts/ too — whisperkit-generate-model is literally a unittest suite runner (doc 06 §8's "verification as CI" made tangible).
• First dependency install is heavy (~1 GB: torch, scipy, wandb via argmaxtools); a cosmetic scikit-learn 1.9 not supported warning from coremltools is harmless here.

2. What the pipeline actually did#

From results/conversion-log.txt (198 lines, full trace committed):

1. TextDecoder — instantiated Argmax's WhisperTextDecoder (SDPA = Cat, per doc 06 §3), loaded HF weights through the linear_to_conv2d hooks, ran a 447-token full autoregressive parity decode against Hugging Face, traced (888 PyTorch ops → MIL), converted, loaded on CPU_AND_NE, PSNR-tested, compiled with coremlcompiler, and extracted the compute plan.
2. AudioEncoder — same flow with SDPA = SplitHeadsQ.
3. MelSpectrogram — the torch.stft DSP module, converted and compute-planned.

Total test time: ~48 s (32.7 s decoder suite + 15.6 s encoder suite), all tests OK. The .mlpackage intermediates are deleted after compilation — the deployable artifacts are the .mlmodelc bundles.

3. Measured results#

Correctness (PSNR, threshold 35 dB)#

Check	Result
torch2torch decoder (Argmax reimpl vs HF, 447-token decode)	PSNR 136, argmax(logits) accuracy 100%
torch2torch encoder	PSNR 143
torch2coreml TextDecoder	PSNR 42.5 ✅
torch2coreml AudioEncoder	PSNR 68.9 ✅
torch2coreml MelSpectrogram	PSNR 69.4 ✅

The gap between torch2torch (~140) and torch2coreml (~42–69) is the FP16 conversion cost — the decoder's 42.5 dB is the lowest because a full softmax-over-51k-vocab logit tensor amplifies FP16 noise, yet it comfortably clears the 35 dB bar.

ANE dispatch (from MLComputePlan, per-op JSONs in results/)#

Component	Ops	ANE-supported	ANE-dispatched	First load (specialization)	Size
TextDecoder	205	203 (99.0%)	203 (99.0%)	1.07 s	57 MB
AudioEncoder	883	883 (100%)	883 (100%)	3.64 s	16 MB
MelSpectrogram	25	22 (88%)	0 (0%) — all CPU	0.39 s	372 KB

The MelSpectrogram result is the textbook lesson, now measured: its two heaviest ops (the STFT expressed as grouped convs, 21.3% of cost each) are CPU-only ('supported': ['CPU']), so Core ML keeps the entire 25-op graph on CPU rather than bounce data between compute units — the same economics as DistilBERT's 4 embedding ops on CPU (doc 03 §5), at whole-graph scale.

Note the encoder's 883 ops for a 4-layer model — that's Principle 2's chunking (SplitHeadsQ: per-head × per-chunk op explosion) visible in the graph, traded for cache residency.

Latency: ANE vs CPU (our benchmark; whisperkittools ships speed tests disabled)#

tests/test_*.py hardcode TEST_SKIP_SPEED_TESTS = True, so we reproduced the measurement with bench_ane_vs_cpu.py (median of 50 predictions on the compiled models, results/benchmark.txt):

Component	CPU_AND_NE	CPU_ONLY	Speedup
AudioEncoder (1500-token sequence)	6.15 ms	22.04 ms	3.59×
TextDecoder (1 KV-cached token step)	1.62 ms	2.47 ms	1.52×

Both KB predictions confirmed on real hardware:

• Long-sequence, compute-heavy work (encoder) is where the ANE shines → 3.6×.
• A single-token decode step is tiny and bandwidth-bound (Principle 4) → only 1.5×, which is exactly why generate_model.py relaxes its speedup assertion to 0.3× (doc 06 §8) and why compression (doc 07) matters for decoders. At 1.62 ms/token ≈ 600 tokens/s upper bound, whisper-tiny decoding is nowhere near ANE-limited.

4. Reproduction gotchas (what the docs don't tell you)#

1. Python ≤ 3.12 — the torch==2.5.0 pin has no 3.13 wheels; uv venv --python 3.12 solves it cleanly.
2. Install non-editable from the submodule to keep it clean; the console scripts still get everything they need (find_packages()).
3. Speed tests are off by default — TEST_SKIP_SPEED_TESTS = True in both test files; PSNR + compute plan are what you get from the stock pipeline. Bring your own latency bench (ours is ~40 lines with ct.models.CompiledMLModel).
4. Decoder input dtypes matter: input_ids/cache_length are int32 scalars-per-batch; masks are additive float16 (-1e4 convention, doc 02 §P3).
5. The heavy downloads dominate wall-clock (~1 GB pip + ~150 MB HF weights); the conversion itself is under a minute for tiny.
6. coremltools 9.0 handled everything the doc 06-era code needed (the MLComputePlan API used by argmaxtools requires ≥ 8.1).

5. Verdict#

The doc 06 study reproduced end to end, unmodified, two years after publication: 99–100% ANE dispatch on the transformer trunks, PSNR comfortably above threshold, and latency behavior exactly as the four principles predict. The pipeline's own structure (unittest mixins → assets only saved on pass → compute-plan JSON as artifact) remains a model for how to ship ANE conversions with evidence attached.