35 Billion Parameters, Three Billion Active: Why Qwen3.6 Runs So Fast Locally

A 35-billion-parameter model, around 150 tokens per second in single-stream decode, and only two RTX 3090s to run it: that combination was not what we expected. On paper, Qwen3.6-35B-A3B looked like one more large model with a serious appetite for VRAM. In practice, it became the model we kept returning to.

The reason was not a spectacular top score in a general benchmark table. On our local systems, Qwen3.6-35B-A3B brought together three qualities that rarely align this well: fast responses, enough model quality for serious work, and practical operation on hardware configurations that are unusually small or inexpensive for a model in this size class.

All figures below come from our own single-stream measurements with the specified quants, engines, and GPU topologies. They describe our setups, not a universal model benchmark. That caveat matters particularly when comparing the FP8 path in vLLM with the GGUF path in llama.cpp.

The Part of the Name That Matters Is A3B

The 35B refers to the model's approximate total parameter count. During token generation, however, this Mixture-of-Experts model uses only a subset of its experts. According to the official model description, around three billion parameters are active per token. That is what A3B means.

This distinction changes the inference math substantially. We still have to store all model weights and distribute them appropriately across the hardware. During decode, however, speed depends heavily on how many active weights must actually be read and processed for each new token. Qwen3.6-35B-A3B therefore retains the knowledge and capacity envelope of a 35B MoE while sending only a small share of those weights through the compute paths for each token.

Our comparison with the dense Qwen3.6-27B showed how much of a difference this makes. Although the A3B variant is nominally larger, it decoded more than twice as fast in our local measurements. With the dense 27B model, a much larger proportion of the weights is involved in every step. The model name alone consequently tells us little about how quickly it can generate new tokens.

Two RTX 3090s Already Delivered Nearly All the Throughput

On two RTX 3090s, we ran the official FP8 variant with TP=2. With approximately 192,000 tokens of configured context, it reached around 150 tok/s. For an interactive local agent, that is far beyond the point where raw generation speed still feels tight.

On our full 8×3090 rig, the best single-stream decode result rose to approximately 164.8 tok/s. That run used TP=4 + PP=2 and a context of 262,000 tokens. Moving from two GPUs to eight therefore bought us just under ten percent more decode throughput.

This is poor scaling if the goal is anything close to linear acceleration from additional GPUs. As a practical finding about the model, though, it is extremely useful. Two 3090s are already enough to extract most of its single-stream speed. We do not have to spread the model across eight cards before its architecture starts paying off.

Here are the relevant vLLM setups:

That made TP=4 + PP=2 our best eight-GPU configuration. It combined the highest decode rate we measured with the full 262k context and a distribution that worked well on our rig. TP=2 + PP=4 showed the tradeoff from the other side: additional pipeline stages make placement easier, but pipeline bubbles add latency to a single request.

With this model, extra GPUs mainly gave us context headroom, more placement options, and an easier fit for the complete set of weights. Single-stream throughput improved only slightly. For an MoE with around three billion active parameters, there simply is not much additional compute available to offset the communication overhead of distributing the model ever more widely.

The Single-GPU GGUF Path Was a Different Experiment

The model was remarkably fast even on a single RTX 3090. Using a local GGUF from the Unsloth quant family, we measured approximately 131.6 tok/s.

That figure cannot be compared cleanly with the FP8 vLLM runs. The engine, quantization format, and distribution all differ, which changes several causes of the measured throughput at once. The single-GPU result answers a different question: can the small active parameter set produce a responsive system on one good consumer GPU? In our setup, the answer was a clear yes.

This made the quantization path more than just a way to fit the model into memory. Previous experiments with very large quantized models had taught us to treat low bit widths with caution. MiniMax M2.7 under long active context showed how quietly a quant that initially appears fast and convincing can later drift into loops, switch languages, or become inconsistent in its judgments.

We did not encounter that pattern with Qwen3.6-35B-A3B in our everyday tasks. The IQ quants we tested did not seem measurably less intelligent in the workloads that mattered to us. This finding is narrowly limited to those quantizations and tasks, and says little about other variants or about arbitrarily long contexts. Within our runs, we found no quality loss that would have outweighed the substantial gains in memory use and local practicality.

A Local Workhorse Does Not Need Eight GPUs

At approximately 150 tok/s on two 3090s, the model's role changed. It was no longer an occasional experiment reserved for the full rig, but a fast local worker we could assign to a specific job.

That opens the door to smaller setups: a separate machine for interactive agents, an inexpensive node for coding tasks, a voice-agent system, or a shared environment where the eight-GPU rig handles the heavy jobs while a smaller machine takes care of fast conversations. The right fit still depends on context, quant, and workload. What is interesting is that these options are worth considering at all for a 35B model.

Larger frontier models remain better at some demanding tasks. Qwen3.6-35B-A3B offers a different combination of speed, quality, and hardware demands:

• Decode is fast enough for interaction that feels immediate.
• Quality is sufficient for serious local work rather than demonstrations alone.
• The low active compute requirement gives us unusual flexibility in our hardware choices.

In daily use, that combination often beats a model that scores perhaps five percent higher in an isolated evaluation but takes three times the effort to run. A local model does not do its work in a leaderboard. It waits for prompts, holds context, calls tools, and occupies real GPUs while doing so.

The Next Useful Test: 2×3060

The single-GPU GGUF run and the weak scaling beyond two RTX 3090s led us to the next question: how far can we reduce the hardware cost before the practical advantage disappears?

Two RTX 3060s make for a serious test. Our expectations stay modest: lower throughput than on the 3090s and possible quality loss from more aggressive quantization. The architecture still makes the setup plausible. If only around three billion parameters are active per token and a good Q4 or IQ quant preserves enough quality, a much cheaper 2×3060 system could still be fast enough for a local assistant.

We would find the same experiment far less attractive with a dense 35B model. With A3B, getting the model to launch successfully is not enough. It has to stay responsive and dependable enough on the cheaper hardware that we would trust it with daily work.

Between MiniMax and HY3, Operations Matter

Our MiniMax and HY3 experiments had exposed two different limits. MiniMax could look very strong on short tasks, then become qualitatively unstable under long agent workloads. HY3 remained more convincing in the long-context runs we observed, but paid for it with steeply declining decode throughput. The details and measurement conditions are in our article on long agent workloads.

Qwen3.6-35B-A3B answered a different question. Its architecture made a capable model unusually straightforward to operate: high local decode speed, usable quants, and little additional single-stream benefit from multiplying the GPU count. Its technical profile mattered almost as much as the model quality itself.

This gives us a practical rule for choosing models. A few additional percentage points of quality in an isolated test do not automatically justify an operating setup that is considerably more expensive, slower, or less reliable. Once a model works on real hardware every day, speed, context headroom, quantizability, and sensible GPU topologies become part of what it can do.

The 2×3060 experiment will therefore tell us more than another screenshot of a high token rate. If Qwen3.6-35B-A3B retains enough quality and remains practically fast on that setup, a capable local worker would run without a large GPU rig. That shift is what makes A3B more interesting to us than the bare 35B figure.

Models and Engines

• Qwen3.6-35B-A3B, official model
• Qwen3.6-35B-A3B-FP8, official quantization
• Qwen3.6-27B, official model
• Unsloth Qwen3.6-35B-A3B GGUF
• vLLM
• llama.cpp