Core AI Agent Skills: Apple's ANE Playbook, Machine-Readable

Part of the ANE Knowledge Base. Source: the skills/ tree of references/coreai-models (commit 85e2f2d) — three agent skills Apple ships for Claude Code (.claude-plugin/marketplace.json), Codex CLI and Gemini CLI. They are the most explicit statement of ANE authoring rules Apple has ever published — including hardware facts absent from the 2022/2024 research articles and the WWDC sessions.

Apple now distributes its on-device-ML expertise as agent skills: markdown rule-books + tested helper scripts that a coding agent loads on demand. The skill headers state the philosophy: "hard-won empirical knowledge… the rules here are stable across Core AI releases — they reflect hardware behavior, not API shapes." This doc catalogs what each skill teaches and — most importantly — what it adds to docs 01–07.

1. The three skills#

Skill	Role	Key files
working-with-coreai	Orchestrator: the AUTHOR → COMPRESS → EXPORT → COMPILE → RUN pipeline, platform/sizing guidance, onboarding protocol	SKILL.md, references/guidance.md
model-authoring	The hardware rule-book: per-compute-unit authoring patterns, verification gates, KV-cache conventions	SKILL.md, references/neural_engine_rules.md (479 lines), references/gpu_rules.md, references/common_issues.md
model-compression-exploration	An automated compression-sweep protocol over coreai-opt (~30 configs + refinements, JSONL + scatter report)	SKILL.md, 4 reference docs, 2 unit-tested scripts

Notable workflow prescriptions: "Run code, don't read code — running gives ground truth instantly" (architecture discovery via forward hooks); author primitives bottom-up (norm → projections → attention → MLP → block), verifying each before composing; "start with export, add authoring or compression only if needed" — re-authoring is now the exception, not the default; and for agents, "your first response is always a conversation, not code."

2. New ANE facts (not in docs 01–07 until now)#

neural_engine_rules.md confirms every principle this KB documented — BC1S, Conv2d 1×1, 64-byte last axis, 32×/64× padding penalties, per-head chunking for L2 residency, the bchq,bkhc->bkhq einsum — and adds hardware knowledge that was previously folklore or unknown:

#	New fact	Detail
1	Supported dtypes: fp16, int8, int16	fp32 anywhere → GPU/CPU fallback. Any Python float literal (x * 1.0) creates an f32 buffer; torch.exp upcasts to f32; F.silu lowers to cast(f32) → mps.swish → cast(f16) (fix: x * torch.sigmoid(x))
2	-40000.0, never -inf	"Neural Engine hardware does not handle IEEE -inf correctly in softmax" — first official statement of why the KB's -1e4 convention exists
3	ANE computes K @ Q, transposed from GPU's Q @ Kᵀ	Hence the mask shape (1, key_seq, 1, query_seq) — transposed vs GPU. Wrong orientation is the #1 cause of ~15–30 dB PSNR
4	"There is no fused SDPA path" — verbatim	Per-head attention is "fundamental to Neural Engine hardware", settling the doc 09 §3.5 question: the compiler cannot lower fused SDPA to ANE; the per-head form must exist in the source graph
5	Softmax placement rule	Softmax on a spatial dim limits the compiler's ability to split work spatially → put softmax on the channel dim (retroactively explains 2022's softmax(dim=1))
6	Layer-design rules (need retraining)	Conv strides must factor into 2s and 3s (4,6,8,9,12,16,24,32); palettized kernels: stride ≤ 2; decompose big kernels (k_fused = k1+k2−1); fuse activation-free conv chains; factor dilations into 2s/3s; pooling stride 2 or 4 only
7	RoPE stays outside the graph	Precompute cos/sin in Python, pass as 4D (1, head_dim, 1, S) inputs — in-graph 2D table gathers produce 3D output the ANE rejects
8	Embeddings are externalized	Shape (vocab, 1, hidden), exported as a separate program so they quantize independently (gather_embeddings_{N} entrypoint)
9	Multi-entrypoint artifacts	One dynamic torch.export → shape-specialized static functions: extend_{ctx}_{len} (decode), prompt_opt_{ctx}_{len} (prefill, no logits computed), gather_embeddings_{N}
10	fp16 drift in long prefills	Sequential 1-token prefill accumulates fp16 rounding across steps×layers; beyond ~50 tokens use chunked prefill (S_q=64) or fp32 cache tensors host-side
11	Vector LUT palettization	Newer ANE generations support vector-valued lookup-table entries; LUTs can span multiple output channels

3. The KV-cache pattern fork (refines docs 06/09)#

The skills draw a sharp platform split that our earlier reading of primitives/ only hinted at:

	Neural Engine (iOS)	GPU (macOS)
Cache shape	[n_layers, B, H_kv·D, 1, max_S] — seq on dim 4	[n_layers, B, H_kv, max_S, D] — seq on dim 3
Pattern	Readonly functional I/O: the graph contains no cache writes; each call gets the full past cache, does cat([k_cache, key_rope], dim=-1), and returns the new K/V tokens as outputs; the host writes them into the cache	Stateful export wrapper: register_buffer + coreai::mutable_slice_update (in-place eager / functional meta) + mutable_arg_action="hoistToArg"
Critical pitfall	Cache post-RoPE keys — caching pre-RoPE K makes later steps attend to un-rotated keys, "PSNR collapses to ~20 dB"	Don't use stateful-transform APIs for generation: "state resets between inference calls"

Evolutionary note for the KB: WhisperKit's one-hot mask blend (doc 06 §4) → MLState (2024) → and now two Core AI idioms, chosen per compute unit. The ANE one is closest in spirit to WhisperKit's — host-managed, functional, static — vindicating that design.

4. GPU rules worth recording (the anti-ANE)#

gpu_rules.md is a mirror-image world: standard (B,S,D) layout, nn.Linear, fused QKV (one projection — the exact opposite of Apple's 2022 "separate projections for cache residency"), native fused SDPA, -inf masks allowed, fp32 intermediates OK, dynamic shapes fine. Plus production techniques: fused Q/K-norm+RoPE on the packed QKV slice, up_proj-before-gate_proj ordering for throughput, MoE via SwitchLinear/GatherMM (all experts in one (sets, experts, out, in) tensor, uint16 indices), and memory-efficient 7B+ loading (meta-device init, load_state_dict(assign=True), safetensors streamed one layer at a time).

The lesson the KB predicted: optimization advice is compute-unit-relative. What's mandatory on ANE (per-head splits, separate projections) is an anti-pattern on GPU, and vice versa — now stated by Apple in two parallel rule files.

5. PSNR as a diagnostic language#

The skills formalize verification numbers this KB had observed empirically ([docs 06](06-case-study-whisperkit.md §8)/11):

Gate	Threshold
Re-authored vs source (torch)	> 70 dB
ANE layout vs GPU layout (torch)	> 70 dB
Compiled vs torch (fp16)	≥ 40 dB
After 4-bit palettization	≥ 35 dB

And, remarkably, failure signatures: ~15–30 dB → mask orientation; 20–30 dB → wrong activation (SiLU/GELU/QuickGELU are not interchangeable); ~18 dB → M-RoPE pattern mismatch; ~20 dB → pre-RoPE keys cached. PSNR ranges as an error-code table. (Our doc 11 run: compiled TextDecoder 42.5 dB, AudioEncoder 68.9 dB — squarely in the healthy band.)

Runtime gotchas catalogued in common_issues.md: .contiguous() on everything before NDArray (the runtime reads raw memory, ignoring strides); dtype descriptor is "si32" not "i32"; filter export inputs to USER_INPUT/BUFFER kinds; tanh → 2·sigmoid(2x)−1 to avoid an f32 op; xcrun coreai-build compile --preferred-compute neural-engine — an AOT-time compute-unit flag complementing the runtime SpecializationOptions of doc 12 §3.

6. Platform guidance (from guidance.md)#

• iOS: models < 2 GB; static shapes; little/no control flow; palettization (2/4/6/8-bit) or int8/int4 per-channel; variable lengths → chunk into multiple static functions.
• macOS: leave ≥ 6 GB RAM headroom; dynamic shapes/control flow fine; int4 per-block quantization recommended.
• Query os_proc_available_memory() before loading; prefer .default specialization unless you align the model representation to a pinned unit (ANE ↔ palettized/static; GPU ↔ linear-quant/dynamic).

7. The compression-sweep protocol (skill #3, condensed)#

A fully proceduralized version of the argmaxtools mixed-bit methodology (doc 07 §3): three experiment groups (per-channel quant int8/int4 × 3 schemes; per-block int4 × block sizes 16/32/128; palettization 4/6/8-bit × group sizes), ~30 configs swept with unit-tested size/quality scripts, then refinement by layer-skipping (first/last/smallest-parameter-type) seeded from the 95th/75th-percentile configs, results in JSONL + a 5-anchor frontier table + scatter plot. Notable engineering details: per-block scale overhead makes int4/bs=16 ≈ 5 effective bits; divisibility silently skips layers (pre-check); real inputs mandatory — "random inputs produce meaningless PSNR." The skill even prescribes parallel subagents per group — Apple writing multi-agent orchestration guidance.

8. Meta-lesson#

Apple's optimization knowledge has now traversed: research articles (2022) → reference repos (2022–23) → third-party toolkits (2023–24) → compiler + platform primitives (2026) → agent skills (2026). The end state is knowledge packaged for machines to apply — the same bet this KB makes. For this repo, neural_engine_rules.md supersedes scattered folklore as the canonical ANE rule reference; docs 01–07 explain why those rules exist.