Hands-On Comparison: WhisperKit/Core ML vs Core AI Export Pipelines

Part of the ANE Knowledge Base. Both pipelines were actually executed on the same machine (Apple M4 Pro, macOS 26.5.1) with the same model (openai/whisper-tiny), on 2026-07-12. Experiments: experiments/whisperkit-conversion (doc 11) and experiments/coreai-whisper-export. Evidence committed under each results/. This is the empirical companion to the conceptual comparison in doc 10.

1. The two runs, side by side#

Dimension	WhisperKit → Core ML (2024-era)	coreai-torch → Core AI (WWDC26-era)
Environment	Python 3.12, torch 2.5.0, transformers 4.53, coremltools 9.0, whisperkit 0.4.2 + argmaxtools 0.1.23	Python 3.12, torch 2.11.0, transformers 4.57.3, coreai-core 1.0.0b2, coreai-torch 0.4.1
Model source	Hand-re-authored Whisper (Argmax): Conv2d/BC1S attention, KV cache as I/O, 3 components	Stock Hugging Face AutoModelForSpeechSeq2Seq, unmodified
Export API	torch.jit.trace per fixed shape	torch.export with Dim("dec_seq_len", 1, 448) — dynamic decoder length in one graph
Author-side code	whisperkittools + argmaxtools (thousands of lines: custom attention, hooks, test harness)	~100-line script (adapted from Apple's ~180-line recipe)
Artifacts	TextDecoder.mlmodelc (57 MB) + AudioEncoder.mlmodelc (16 MB) + MelSpectrogram.mlmodelc (372 KB) ≈ 73 MB (fp16)	Single whisper-tiny_float16.aimodel = 72 MB (fp32: 144 MB); contents: metadata.json + main.mlirb (MLIR binary) + main.hash
Architecture choice	3 graphs, KV-cached single-token decoder — inference-optimal	1 graph, no KV cache, full forward per call — recipe-simple
Conversion wall-clock	~48 s of test suites (plus trace/convert inside)	67 s fp32 (6 s for fp16 with cached weights)
Built-in verification	PSNR suite (42.5–69.4 dB), 447-token parity decode, per-op compute plans	Manual but trivial: logits parity max|diff| = 1.2e-4, argmax match
ANE dispatch measured	99–100% on decoder/encoder; MelSpectrogram all-CPU (compute-plan JSONs)	Not measurable on this OS — see §3
Latency measured	Encoder 6.15 ms (ANE, 3.59× vs CPU); decoder step 1.62 ms	~1,400 ms full enc+dec on the in-package CPU runtime (fp16 ≈ fp32)
Runs on macOS 26.5?	Fully (convert + ANE execution)	Export ✅; inference only via bundled CPU runtime; ANE/GPU delegates need macOS 27

2. What the numbers mean (don't compare 1.4 s to 8 ms naively)#

The two latency figures measure different things:

• The Core ML pipeline ran on the ANE with a KV-cached architecture: encoder once (6.15 ms) + 1.62 ms per token.
• The Core AI run used the bundled CPU fallback runtime (no delegates on macOS 26.5) with a cache-less graph that recomputes the full encoder + 4-token decoder every call. Same-machine CPU_ONLY Core ML encoder alone was 22 ms — so ~1.4 s for a cache-less fp32/fp16 full forward on a non-optimized CPU runtime is unsurprising, and says nothing about Core AI-on-ANE performance.

The honest conclusions this comparison does support:

1. Authoring effort collapsed ~50× (a script vs a toolkit) for the same model family — because the re-authoring work moved into Apple's decomp tables and platform primitives (doc 09 §3).
2. The KV-cache architecture is still the author's decision. Apple's simple Whisper recipe skips it; their LLM recipes (coreai-models/models/qwen3 + primitives/ios/cache.py) do implement it. Inference-optimal Whisper on Core AI would need the same encoder/decoder split + cache states WhisperKit pioneered — the concepts from doc 06 transfer intact.
3. Artifact sizes converge (72 vs 73 MB at fp16) — weights dominate; formats don't.
4. Verification is easier but thinner by default in the recipe path: whisperkittools ships PSNR gates and compute plans; with Core AI you get clean parity checking in 10 lines but must ask for it (or use the Debugger/save_intermediates on macOS 27).

2.5 Why "stock HF" and "hand-rewritten" are equivalent: where the surgery moved#

Both pipelines compute the same function with the same weights; they differ in who performs the ANE-friendly re-expression and when.

Core ML era — the converter was a faithful photocopier. torch.jit.trace records the ops exactly as the source code executes them (HF Whisper: nn.Linear on (B,S,C), view/transpose head splits, one fused softmax), and coremltools translated that graph op-for-op. Nothing re-architected it, so the ANE compiler received hostile shapes (small last axis → 64-byte padding, cache-missing attention tensors, copy-inducing transposes) and fell back or ran slow — Apple's own 2022 baseline: 10× slower, 14× more memory. The only available lever was rewriting the model source so the trace already had the right shape: that's whisperkittools/argmaxtools. The linear_to_conv2d_map hooks are the proof that this is pure re-expression: the same (out, in) Linear weight, unsqueezed to (out, in, 1, 1), computes identically as a 1×1 conv on BC1S.

Core AI — the surgery became a compiler pass. Three changes:

1. torch.export produces a typed, symbolically-shaped functional graph — a real compiler IR, not a photograph.
2. run_decompositions(get_decomp_table()) uses a table designed to preserve composite ops: HF's F.scaled_dot_product_attention survives as a single SDPA node (coreai_torch/composite_ops/_sdpa.py), likewise RMSNorm/RoPE — instead of shattering into matmuls the compiler couldn't recognize.
3. At specialization, the compiler pattern-matches those nodes and emits the per-hardware lowering: fused kernels on GPU; on ANE, the very per-head / channels-first / last-axis-aligned schedule that used to be hand-written. The reference form of that lowering is visible in Apple's own coreai-models/primitives/ios/sdpa.py — the same algorithm as ml-ane-transformers (2022) and argmaxtools (2023), now platform-owned. Layout assignment and memory planning (the old "minimize copies" discipline) happen inside .optimize() + specialization over the .aimodel's MLIR (main.mlirb).

The equivalence was always mechanical — Linear(W) on (B,S,C) ≡ Conv2d1×1(W[:,:,None,None]) on (B,C,1,S); one big softmax ≡ N per-head softmaxes — and this experiment verifies it three ways: identical fp16 artifact sizes (72 vs 73 MB — same weights), logits parity 1.2e-4 (same function), and the 2022 hand-written attention reappearing verbatim in the 2026 platform primitives (same lowering).

The boundary: the compiler absorbed op-level optimization, not architectural optimization. KV caches, encoder/decoder splits, and context prefill remain author decisions (§2, conclusion 2) — math went to the compiler; inference architecture stayed with you.

3. The compute-unit question, answered empirically#

Does Core AI let you pin compute units like Core ML's .cpuAndNeuralEngine? Yes — with an even richer API — but placement is decided at specialization time, not load time:

from coreai.runtime import AIModel, ComputeUnitKind, SpecializationOptions

ComputeUnitKind            # .cpu | .gpu | .neural_engine | .available_kinds()
opts = SpecializationOptions.from_preferred_compute_unit_kind(ComputeUnitKind.neural_engine)
# also: SpecializationOptions.allowed_compute_unit_kinds  <- the true CPU_AND_NE analogue (allowed *set*)
#       SpecializationOptions.cpu_only(), .default, .with_debug
model = await AIModel.load("model.aimodel", specialization_options=opts)   # baked at specialization

	Core ML	Core AI
Mechanism	MLModelConfiguration.computeUnits at load	SpecializationOptions at specialization (AIModel.load / Swift AIModel.specialize)
Semantics	Allowed set (.all, .cpuOnly, .cpuAndGPU, .cpuAndNeuralEngine)	Both preferred unit (from_preferred_compute_unit_kind) and allowed set (allowed_compute_unit_kinds)
Device introspection	MLModel.availableComputeDevices	ComputeUnitKind.available_kinds() (returned 3 kinds on the M4 Pro)
Verified on this Mac	✅ used throughout doc 11	API present; delegates unavailable: SpecializationOptions.is_supported() == False, and USE_OS_COREAI=1 → "Core AI Framework is not available for this version of macOS"

The probe (evidence: experiments/coreai-whisper-export/results/specialization-probe.txt) also revealed the runtime architecture: coreai-core ships an in-package portable runtime (CPU; runs anywhere — how our macOS 26.5 inference worked) while hardware delegates (ANE/GPU) live in the OS's Core AI framework (macOS 27+ / iOS 27+, opted into from Python with USE_OS_COREAI=1). Core ML has no such split — it is an OS framework, which is why it runs everywhere but can't be pip-installed.

4. Platform-gate summary for this machine (macOS 26.5 + Xcode beta < 27)#

Capability	Available?
Core ML: convert, compile, ANE execution, compute plans	✅ all of it
Core AI: torch.export → .aimodel toolchain	✅
Core AI: bundled-runtime inference (CPU) + parity checks	✅
Core AI: ANE/GPU delegate specialization	❌ needs macOS 27
Core AI: xcrun coreai-build AOT compile	❌ not in this Xcode beta

Follow-up experiment once macOS 27 lands on this Mac: rerun run_aimodel.py with USE_OS_COREAI=1 and preferred_compute_unit_kind=neural_engine, and benchmark the fp16 .aimodel against doc 11's 6.15 ms Core ML encoder — the last unmeasured cell in this comparison.

5. Takeaway#

Same model, same machine, two generations of tooling: Core ML + WhisperKit demanded expertise but delivered measured 99–100% ANE execution today; Core AI delivered a 50× simpler authoring path and a portable runtime, with its hardware story gated behind the OS upgrade. The optimization knowledge didn't disappear — it moved: from your code (docs 02–07) into Apple's primitives and compiler, with the KV-cache/architecture decisions still firmly yours.