A 760M-Parameter Model That Beats GPT-5-High on Math
Let’s talk about a paper that quietly dropped on arXiv this week and absolutely deserves more attention than it’s getting.
Zyphra, a Palo Alto startup founded in 2021 that most people outside the local LLM bubble have never heard of, released ZAYA1-8B. On paper, it looks like a typical small MoE model. Eight billion total parameters, 760 million active per forward pass. You could easily dismiss it as another entry in the crowded small-model pack.
But the benchmarks are genuinely startling.
With test-time compute, ZAYA1-8B hits 91.9% on AIME 2025 and 89.6% on HMMT 2025. Those numbers beat Claude 4.5 Sonnet (79.2% on HMMT) and GPT-5-High (88.3% on HMMT). Not by a hair. By a wide margin.
And it’s doing this with under one billion active parameters. The models it’s beating have 30 to 50 times that many.
This is the kind of result that makes you sit up and pay attention. Not because small models are a novelty anymore, but because the gap between 760M active parameters and frontier-tier math performance has been collapsing faster than almost anyone expected.
What Makes ZAYA1-8B Different
Zyphra didn’t just squeeze a standard transformer with a few MoE layers and call it a day. The model uses what they call MoE++, which introduces three architectural changes worth understanding:
Compressed Convolutional Attention (CCA). Instead of the standard multi-head attention mechanism, CCA performs sequence mixing in a compressed latent space. The result is an 8x reduction in KV-cache size compared to full attention. For anyone who’s ever tried running a 70B model on consumer hardware and watched VRAM explode as context length grows, this is the kind of engineering that matters. It’s not a marginal optimization. It’s a fundamental rethinking of how attention works at scale.
ZAYA1 Router. Standard MoE models use linear routers to decide which experts to activate. Zyphra replaced that with a multi-layer MLP design that uses a PID controller-inspired bias-balancing scheme. The goal is straightforward: keep expert utilization uniform so you don’t end up with 28 of 32 experts doing nothing. The math is elegant. The gradient for each expert’s bias term is simply the difference between actual utilization and uniform utilization. No KL penalties, no heuristics.
Learned Residual Scaling. Across the model’s 40 layers, residual-norm growth is controlled through learned bias and gating coefficients. The overhead is negligible, roughly 4 times the layer dimension, but it prevents the gradient vanishing and explosion problems that typically plague deeper MoE models.
These aren’t incremental improvements. They’re the kind of architectural decisions that suggest the authors actually thought about what’s broken in standard MoE transformers and tried to fix it.
The Training Story
What’s perhaps more impressive than the architecture is the training pipeline. ZAYA1-8B was trained from scratch specifically for reasoning, not bolted onto a general-purpose base model like so many recent releases.
The pretraining data strategy includes something called Answer-Preserving Trimming. When training on long chain-of-thought traces, the model tends to learn reasoning without conclusions, because the final answer gets buried at the end of a 32K token sequence. Zyphra’s solution is to truncate from the tail, keeping the problem decomposition and final answer while removing the middle reasoning steps. The model learns the problem-solution relationship without drowning in intermediate text.
The post-training follows a four-stage RL cascade that’s unusually disciplined:
Reasoning warmup on math and puzzle datasets.
A 400-task curriculum called RLVE-Gym with adaptive environments.
Domain-specific RL on math, code, and test-time compute traces.
Behavioral RL for chat and instruction following.
They use DPPO Binary-TV instead of standard PPO clipping, Muon optimizer with zero momentum for matrix weights, and a token efficiency reward that biases toward concise correct answers. None of this is groundbreaking individually, but the combination suggests a team that has actually done this work before and knows what works.
Markovian RSA
Here’s where things get genuinely clever. ZAYA1-8B includes a test-time compute method called Markovian RSA (Recursive Self-Aggregation).
The basic idea: generate multiple independent reasoning traces in parallel, reduce each to its final 4K tokens, then recursively aggregate those tails across multiple rounds. The key insight is that you don’t need to carry forward the entire reasoning chain. You only need the conclusions, and those are bounded.
The default configuration generates 16 parallel rollouts, samples 4 tails per aggregation round, and runs 2 aggregation rounds. Each rollout has a 40K token thinking budget. The prefill context stays bounded at roughly the query length plus 4K tokens, regardless of how many reasoning rounds you run.
This is a practical solution to a real problem. Most test-time compute methods for reasoning models hit the same wall: the context window fills up with reasoning traces, and you can’t afford to keep generating. Markovian RSA sidesteps this by only carrying forward the tails.
The result is a model that can effectively “think longer” without requiring a larger context window or more VRAM.
The AMD Connection
ZAYA1-8B was trained entirely on AMD Instinct MI300 GPUs with Pollara 400 networking. This matters more than it might seem.
Everyone knows NVIDIA dominates AI training. But AMD’s MI300X has been quietly gaining ground, and training a model this capable entirely on AMD hardware is a strong signal that the ecosystem is maturing. Zyphra’s investors include AMD itself, along with IBM and Bison Ventures, so there’s a business incentive to prove AMD’s viability. But the results speak for themselves.
The model requires custom forks of vLLM and transformers for inference, which is a minor friction point. Tensor parallelism isn’t supported for the CCA mechanism yet. But the core inference path works, and the model is available on Hugging Face with 11 quantization variants already.
Why This Matters for Local LLM Enthusiasts
ZAYA1-8B is Apache 2.0 licensed. You can run it commercially, modify it, redistribute it, no strings attached.
With 760M active parameters, the inference cost is comparable to running a 4B dense model. On a single consumer GPU, you can run reasonable batched inference. On Apple Silicon, it should work well with MLX once the custom attention mechanism gets proper support.
The model’s sweet spot is reasoning tasks. It’s not going to beat a 70B model on general knowledge or creative writing. But for math, code generation, and structured reasoning, it punches far above its weight.
More importantly, it proves that architectural innovation and training methodology can close the gap with models that have 30 to 50 times the active parameters. The “bigger is better” narrative isn’t dead, but it’s definitely taking some serious dents.
The Bigger Picture
Zyphra is a small company. 31 employees. $110M Series A. Their CEO is Krithik Puthalath and their chief scientist is Beren Millidge, an Oxford postdoc who’s a pioneer in active inference and the free-energy principle. The technical report lists 20 authors, many of whom have strong academic backgrounds in ML.
The model’s Twitter/X reach hit about 1 million views in 24 hours. The community reaction has been overwhelmingly positive, particularly around the AMD training stack and the efficiency of the reasoning cascade.
But the real question is whether this represents a trend or an outlier. Can other labs replicate these results? Can the architecture generalize beyond math and code? Can the Markovian RSA approach be adapted for non-reasoning tasks?
Those are open questions. But the direction is clear: intelligence density, not parameter count, is becoming the critical metric. And models like ZAYA1-8B are pushing that frontier faster than anyone predicted.