◧ ANE Knowledge Base
EN ES

Chapters · 01

Apple Neural Engine (ANE): Overview and Hardware Constraints

Part of the ANE Knowledge Base. Sources: Apple ML Research articles "Deploying Transformers on the Apple Neural Engine" (June 2022) and "Deploying Attention-Based Vision Transformers to Apple Neural Engine" (2024).

1. What is the ANE?#

The Apple Neural Engine is Apple's dedicated neural network accelerator (NPU), integrated into Apple Silicon SoCs. It is energy-efficient, specialized hardware designed to run ML inference at low power, freeing the CPU and GPU for other workloads.

History and availability#

Generation Chip Device Compute
1st A11 Bionic (2017) iPhone X 0.6 TFlops (FP16)
5th A15 Bionic (2021) iPhone 13 Pro 15.8 TFlops (FP16) — ~26× the first generation
  • iPhone: ANE present since A11; Apple's optimized-transformer device spec targets A14 and newer.
  • iPad: ANE available starting with the A12 chip.
  • Mac: ANE available starting with the M1 chip (and all later M-series).

How you access it#

There is no direct public API for the ANE. Models reach it through Core ML:

  • You convert a model (e.g., a traced PyTorch model) with coremltools into an .mlpackage (ML Program).
  • At load time, Core ML builds a hybrid execution plan that "seamlessly blends CPU, GPU, and ANE (if available)" — individual ops may be dispatched to different compute units.
  • ct.ComputeUnit.ALL (default) lets Core ML use all compute units, including the ANE. Whether an op actually runs on ANE depends on whether the op and its tensor layout are ANE-compatible.
  • Xcode performance reports (Performance tab on an .mlpackage) show per-op compute-unit dispatch and measured latency on a real device — this is the ground truth for "is my model actually on the ANE?"

Implication: getting a model to run is easy; getting all (or most) of its ops to map onto the ANE is what the optimization principles in this knowledge base are about. A single incompatible op in the middle of the graph can force expensive fallbacks to CPU/GPU and memory copies between compute units.

2. Hardware characteristics that drive every optimization#

These are the ANE facts from which all the transformer optimization principles (see doc 02) derive:

2.1 Preferred data format: 4D, channels-first#

"The most conducive data format for the ANE (hardware and software stack) is 4D and channels-first" — i.e. (B, C, 1, S) for sequences (called BC1S in Apple's code) or (N, C, H, W) for images. This is why ANE-optimized transformers replace nn.Linear (which wants (B, S, C)) with 1×1 nn.Conv2d (which wants (B, C, H, W)).

2.2 The last axis is not packed: 64-byte alignment#

The defining ANE buffer constraint:

"The last axis of an ANE buffer is not packed; it must be contiguous and aligned to 64 bytes."

Every 64 bytes of the last dimension is processed in one batch; if the last dimension holds less than 64 bytes of data, it is padded up to 64 bytes. Consequences:

  • If the last axis is a singleton (size 1), an FP16 tensor gets padded 32× (64× in 8-bit precision) — that is 32×/64× the memory cost and up to 32× slower effective processing.
  • Concrete example from the vision article: a 7×7 attention window in NCHW layout leaves only 7 FP16 elements (14 bytes) in the last dim → 50 bytes of padding per row.
  • Therefore: put a large, ideally 64-byte-multiple dimension last. For sequence models that is the sequence length S; for window partitioning in vision models it is the channel dim (hence NHWC there — see doc 04).

2.3 FP16 by default#

The ANE computes in half precision (float16). Practical implications:

  • Numerical constants tuned for FP32 can break. Example: DistilBERT's LayerNorm epsilon of 1e-12 underflows in FP16; Apple's port uses 1e-7 (references/ml-ane-transformers/ane_transformers/huggingface/distilbert.py).
  • Additive attention masks should use -1e4 (not -inf or -1e9) so several masks can be summed while staying in FP16 range.
  • Optional input clamping (clip_mag in LayerNormANE) can be used to reduce overflow risk.

2.4 Maximum tensor rank: 5D#

"ANE supports a maximum of 5D tensors." Ops that naturally use 6D tensors (e.g., window partition/reverse in Swin/MOAT-style vision transformers, shaped (N, C, Nh, Nw, Hw, Ww)) must be re-expressed as a chain of ≤5D reshapes/permutes ("relay" partitioning — see doc 04).

2.5 Reshapes and transposes may trigger memory copies#

"Reshape and transpose operations are likely to trigger memory copies unless specifically handled." Memory copies are pure overhead, so ANE-optimized code:

  • avoids reshape/transpose wherever possible,
  • when unavoidable, pays the cost once (e.g., a single transpose of the key tensor before the QK matmul; one NHWC transpose before all window partitioning rather than per window),
  • prefers einsum formulations whose operand layouts map directly to hardware batched-matmul without intermediate rearrangement.

2.6 L2 cache residency and multicore utilization#

The ANE is a multicore engine with an L2 cache. Large intermediate tensors (e.g., a full multi-head QKV projection or a giant attention matrix) spill out of cache and serialize work. Chunking big tensors into per-head pieces:

  • increases the chance each piece stays L2-resident,
  • lets the Core ML compiler parallelize the pieces across ANE cores.

This is also why split softmax (softmax applied per attention head on smaller tensors) is much faster than one huge softmax.

2.7 Compute-bound vs. bandwidth-bound regimes#

For small workloads the ANE is often bandwidth-bound, not compute-bound: "large parameter tensors are being fetched from memory, only to be applied on too few inputs before the next parameter tensor is fetched."

Observed evidence: DistilBERT latency stays roughly constant across sequence lengths 32 → 64 → 128 (batch 1) although the attention compute quadruples. Remedies:

  • Increase batch size (batch inference workloads) — more arithmetic per parameter fetch.
  • Shrink the parameters via quantization or pruning — less data to fetch.

3. Quick constraint checklist#

Use this as a pre-flight check when targeting the ANE. (In 2026 Apple published its own expanded, official version of this list as an agent skill — see doc 13 for the additions: K@Q convention, -40000 masks, stride rules, dtype set fp16/int8/int16, and more.)

# Constraint What to do
1 4D channels-first preferred Use (B, C, 1, S) / NCHW; swap nn.Linear → nn.Conv2d 1×1
2 Last axis contiguous + 64-byte aligned Keep a large dim (seq len, or channels ≥ multiple of 32) last; never a singleton
3 FP16 compute Audit epsilons (≥ ~1e-7), mask values (−1e4), watch overflow
4 Max 5D tensors Decompose 6D ops into staged ≤5D reshapes
5 Reshape/transpose → memory copies Minimize; batch them; use einsum with hardware-friendly layouts
6 L2 cache + multicore Chunk large tensors (per-head QKV, split softmax)
7 Bandwidth-bound at small batch/seq Batch more, or quantize/prune weights
8 No direct API; hybrid dispatch Convert via coremltools (mlprogram); verify dispatch in Xcode performance reports
9 Some ops are better on CPU E.g., embedding lookups — a few CPU ops are fine and expected

4. Where to go next#

Generated from the knowledge base markdown — every claim traces to a cited source.