Running AI Agents On-Device: The 2026 Stack

Here's the thing almost every “AI agent” demo quietly depends on: a fast, reliable trip to a data center. The model lives in the cloud, your prompt flies there, tokens fly back. For a single chatbot reply, you barely notice the network tax. For an agent, you notice everything, because an agent isn't one call. It's a chain.

A real agentic task, read a screen, decide, call a tool, read the result, decide again, can fire off 10 to 30 model calls before it hands you an answer. Every one of those calls pays the round-trip tax. And the round-trip tax is brutal. Measured on-device time-to-first-token comes in around 15 to 80 ms. The same first token from the cloud runs 180 to 600 ms. That's a 4x to 13x gap, and it comes almost entirely from eliminating the network hop, not from a smarter model.

Multiply the gap by the number of calls and the math gets loud. An agent that would feel instant locally feels like it's thinking through molasses over the network. That's the real case for running agents on-device in 2026, and it's finally a case you can actually ship, because the silicon and the frameworks caught up at the same time.

Latency is the product, not a nice-to-have

I want to sit on this point because it's the whole argument. When people compare local and cloud models, they reach for benchmark scores, who wins on reasoning, who writes better code. For an agent, that's often the wrong axis. The thing that decides whether an agent feels alive or dead is how long the loop takes, and the loop is dominated by the per-call latency floor.

Think about what an agent actually does. It doesn't answer once. It observes, plans, acts, observes the result, re-plans. Each step is a model call. If each call carries a 300 ms network penalty and the task needs 20 of them, you've spent six seconds just waiting on the network before you count any actual compute. Move that on-device and the same 20 calls pay near-zero network cost. The agent stops feeling like it's buffering.

There's a privacy angle too, and it's not a throwaway. An agent that reads your messages, your calendar, your screen is handling the most sensitive data you own. On-device means that data never leaves the device to get reasoned over. That's not a compliance checkbox, it's a genuinely different trust model. But even if you didn't care about privacy at all, the latency math alone would push agents local.

Apple: a 3B model in every app, and a 20B one that hides on flash

Apple made the most consequential move here, and it's easy to undersell. At WWDC 2025, alongside iOS 26, they shipped the Foundation Models framework: a native Swift API that hands any third-party app direct access to the same on-device model that powers Apple Intelligence. It's roughly a 3-billion-parameter model, and it runs entirely on the device. No API key, no per-token bill, no server. The model is just there, on hundreds of millions of phones, addressable from your app.

What makes it usable for agents isn't the raw model, it's the two features wrapped around it. First, guided generation. You annotate a Swift type with the @Generable macro and @Guide constraints, and the framework forces the model to emit output that fits that type. Type-safe structured output, on-device, without prompt-wrangling JSON out of a text blob. Second, tool calling via a Swift Tool protocol. You define tools, and the framework automatically orchestrates parallel and serial tool-call graphs on-device. That last part is the agent loop, built into the OS.

Now the catch, and it's a big one: the on-device model gives you about a 4,096-token context window. Four thousand tokens. The 32K window only exists on the server-side Private Cloud Compute model. If you're used to stuffing a fat system prompt, a pile of tool definitions, and a long history into context, 4K feels like a straitjacket. This is the single biggest design constraint of building on Apple's local stack, and it forces a discipline that's honestly good for you: summarize aggressively, keep tool schemas tight, don't hoard history. Agents that respect the window work. Agents that fight it fall back to the cloud and lose the whole latency advantage.

Then at WWDC 2026, announced June 8, Apple showed where this is going. The third-generation models include AFM 3 Core Advanced, a 20-billion-parameter model. Twenty billion, on a phone. The trick is that it's sparsely activated: it only lights up 1 to 4 billion parameters for any given prompt, using a technique Apple calls Instruction-Following Pruning, and it swaps the routed experts between NAND flash and DRAM as needed. That's a genuinely clever bit of systems engineering. Flash is huge and cheap and slow; DRAM is small and fast. Keeping the full 20B on flash and paging only the experts you need into DRAM lets a phone punch far above its memory weight.

The quality jump is real, not just a spec bump. Apple's own human-evaluation benchmarks show AFM 3 Core was preferred on 45.6% of side-by-side text comparisons against 23.3% for the 2025 baseline, and its text-to-speech mean opinion score climbed to 4.15 general and 4.24 conversational on a 5-point scale. Treat vendor evals with the usual salt, but the direction is clear: on-device quality is not the toy tier anymore.

Qualcomm: the NPU grew up, and so did the tooling

Apple owns its stack top to bottom, so its story is clean. The Android side is messier and, in some ways, more interesting, because Qualcomm has to serve a thousand different device makers. On September 24, 2025, at its Snapdragon Summit, Qualcomm announced the Snapdragon 8 Elite Gen 5, built on TSMC's 3nm N3P process. The headline for our purposes is the Hexagon NPU: Qualcomm says it's 37% faster than the previous Snapdragon 8 Elite generation, and it adds INT2 precision support.

INT2 is worth pausing on. Quantization is how you shrink a model to fit and run fast on a phone, trading numeric precision for speed and memory. Most on-device work lives at INT4 or INT8. INT2 is two bits per weight, which is aggressive, and hardware support for it means you can push models smaller and faster than before without falling off a cliff. Paired with the faster NPU, Qualcomm demonstrated on-device inference exceeding 200 tokens per second at a 128K-token context length, entirely on the device. Compare that 128K to Apple's local 4K and you see the two philosophies: Apple keeps the local model tiny and leans on the cloud for big context, Qualcomm pushes the whole thing onto the device.

The part developers actually touch is Qualcomm AI Hub, and it's the unglamorous piece that makes the platform real. It offers more than 300 optimized models, over 175 of them pre-optimized including 15 leading LLMs, and it lets you convert PyTorch or ONNX models into on-device runtimes like LiteRT, ONNX Runtime, or the Qualcomm AI Runtime. Qualcomm says more than 1,800 companies, including Meta, Samsung, and Amazon, use it. That last number is the tell. On-device AI stopped being a research demo and became a supply chain with real customers deploying real models.

Google: the parameter count is a lie, in a good way

Google's angle is the cleverest bit of misdirection in the space. Gemma 3n models carry raw parameter counts of 5B and 8B. On paper that sounds too heavy for a phone. But thanks to Per-Layer Embeddings, or PLE, they run with a dynamic memory footprint of roughly 2GB and 3GB, comparable to what you'd expect from a 2B and 4B model. The parameters exist, they just don't all have to sit in precious memory at once. It's the same instinct as Apple's flash-to-DRAM paging, solved a different way.

You can feel this yourself. Google shipped AI Edge Gallery, an open-source on-device playground now in open beta on the Play Store, that runs Gemma models locally via the MediaPipe LLM Inference API. One 529MB model reaches a prefill rate of up to 2,585 tokens per second, and the app does offline audio transcription through Audio Scribe. Prefill is how fast the model reads your prompt, and 2,585 tokens per second means a long input gets ingested almost instantly. For an agent chewing on a screenful of context every loop, prefill speed matters as much as generation speed.

So what does shipping a local agent actually take

Strip away the marketing and the job comes down to a few honest decisions. The first is the split. Almost no serious agent is going to be 100% local in 2026. The realistic architecture is a router: keep the fast, private, high- frequency loop on-device, and escalate the rare hard call to the cloud. Apple basically ships this pattern for you, tiny local model plus Private Cloud Compute for the 32K jobs. On Qualcomm you can push much more local, but you still want an escape hatch for the task that genuinely needs a frontier model.

The second is context discipline, and it's where most teams will bleed. If you target Apple's 4K window, you cannot be lazy about what goes into the prompt. Every tool schema, every scrap of history, every instruction competes for those tokens. The agents that work will summarize their own state, prune tool definitions down to what's relevant, and treat context as the scarce resource it is. This is a design skill, not a config flag.

The third is the loop itself. On Apple, the Foundation Models framework orchestrates your tool-call graph, so you define tools and let the OS run the agent. On Android you assemble more of it yourself through AI Hub and a runtime like LiteRT or ONNX Runtime. More control, more work. Either way the payoff is the same: when the model call is local, you can afford to loop. You can let the agent try, check, and retry without watching a latency meter tick up 300 ms a step.

The way I read 2026, on-device agents crossed the line from “interesting research” to “the obvious default for a large class of tasks.” Not because the local models beat the frontier on quality, they don't. Because for anything that loops, the latency floor and the privacy story matter more than the last few points of benchmark score. Apple put a capable model and an agent runtime in every app. Qualcomm made the NPU fast enough and the tooling broad enough that 1,800 companies are shipping on it. Google proved you can cheat the memory ceiling. The pieces are on the table. The interesting work now is design, not waiting for the hardware.