Apple Neural Engine

The ANE is a FP16, channels-first accelerator whose buffers require the last tensor axis to be contiguous and 64-byte aligned, supports at most 5D tensors, pays real costs for reshapes and transposes, and is often bandwidth-bound at small batch sizes. This knowledge base traces — from Apple's 2022 research to the 2026 Core AI stack — exactly what that means for transformers, with runnable experiments measured on real Apple silicon.

Chapters

1. 01 Apple Neural Engine (ANE): Overview and Hardware Constraints What the ANE is and the hardware constraints that drive every optimization: 4D channels-first layout, 64-byte alignment, FP16, max 5D, memory copies, bandwidth-boundness.
2. 02 The Four Principles for Optimizing Transformers on the ANE Apple's four principles with real code: BC1S layout + Linear→Conv2d 1×1, per-head chunking + split softmax, minimizing copies with einsum, and handling bandwidth limits.
3. 03 Case Study: Optimizing Hugging Face DistilBERT for the ANE Retrofitting a third-party model without retraining: the subclass-and-swap pattern, state-dict hooks, FP16 hygiene, and measured results (up to 10× latency, 14× memory).
4. 04 Deploying Attention-Based Vision Transformers on the ANE DeiT vs MOAT, local window attention, the ≤5D 'relay' window partition, NHWC vs NCHW, split softmax, and position-embedding design (RPE / single-head RPE / LePE).
5. 05 Deployment Workflow: PyTorch → Core ML → ANE End-to-end pipeline: PyTorch → torch.jit.trace → coremltools (ML Program) → Xcode performance reports, with a pitfalls checklist for reproducing Apple's exports.
6. 06 Case Study: WhisperKit — Production ASR on the ANE (Argmax) Production ASR on the ANE: multi-component models, pluggable SDPA strategies, KV-cached decoding with static shapes, MLState, context-prefill LUTs, and CI verification.
7. 07 Model Compression for the ANE: Palettization, Mixed-Bit Recipes, Outlier Decomposition Palettization, per-layer sensitivity profiling and mixed-bit recipes, sparse outlier decomposition — compression as a latency optimization for bandwidth-bound regimes.
8. 08 WWDC Timeline: Core ML Evolution 2022–2025 The official platform timeline: performance reports (2022), ct.optimize compression (2023), stateful models / MLTensor / fused SDPA (2024), and the 2025 plateau.
9. 09 Core AI (WWDC26): Apple's New On-Device Inference Stack Apple's new inference stack: .aimodel, specialization + AOT compilation, the Swift API, coreai-torch / coreai-opt / coreai-models, and how the 2022 ANE principles survive.
10. 10 Core ML vs Core AI: A Side-by-Side Comparison Side-by-side comparison: trace vs export, who owns the optimization, KV-cache lineage, compilation moving left, verification as a product — plus a migration mapping.
11. 11 Hands-On: Reproducing the WhisperKit Conversion Pipeline Doc 06 reproduced for real on an M4 Pro: whisper-tiny → Core ML, PSNR 42–69 dB, 99–100% ANE dispatch on the trunks, and our measured ANE-vs-CPU benchmark.
12. 12 Hands-On Comparison: WhisperKit/Core ML vs Core AI Export Pipelines Same model, same machine, both pipelines executed: ~50× less authoring code with coreai-torch, artifact sizes converge, and the compute-unit question answered empirically.
13. 13 Core AI Agent Skills: Apple's ANE Playbook, Machine-Readable Apple's ANE playbook as agent skills: the K@Q convention, −40000 vs −inf, 'no fused SDPA path', stride rules, the readonly-vs-stateful KV fork, and PSNR failure signatures.

About this knowledge base

Thirteen documents, each traceable to a source: an Apple ML Research article, a WWDC session, an official page, or a file in the reference implementations. Hands-on chapters reproduce the pipelines on an Apple M4 Pro (macOS 26.5.1) and report measured evidence.