Qwen3.5 122B on Eight RTX 3090s: The First Large Model That Felt Like Infrastructure

By the time we had worked through reasoning loops, unexpected language switches, erratic rankings, and hardware faults masquerading as model problems, we had settled on a fairly sober standard: a model that barely squeezes into 192 GB of VRAM is not yet a usable server. It needs enough headroom on that hardware to run reliably from day to day.

By that standard, Qwen3.5-122B-A10B was the first genuinely large local model in this project that we could treat as infrastructure. We had run larger and more spectacular models before, but Qwen was the first to stay uneventful over long sessions instead of demanding the constant supervision an experiment needs.

A 120 GB Model on 192 GB of VRAM Still Leaves Room for Real Choices

Our local LLM rig pairs seven RTX 3090s with one RTX 3090 Ti for about 192 GB of VRAM in total, wired through a consumer PCIe topology without NVSwitch. For this test we used the dynamic FP8 quant RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic, roughly 120 GB on disk, served with vLLM 0.25.1.

That size was decisive for the test. With MiniMax M2.7 AWQ and HY3 GPTQ, the weights had claimed so much of the available memory that the KV cache and alternative distributions had almost no room left, and we were stuck with whichever configuration would start at all. Qwen122 left enough memory free to test two serious topologies and to choose between throughput and KV capacity.

Two KV Heads Change the Memory Equation

Qwen122 combines 122 billion total parameters with about 10 billion active parameters in a mixture-of-experts architecture. Those headline figures were less important to our long-context results than the attention design: only one in four of its 48 language-model layers runs full attention, while the rest use linear or recurrent attention, and the model has just two KV heads.

This makes the expensive KV work grow far more slowly with the active context than on the other large models in our test series. MiniMax M2.7 and HY3 use full attention in every layer, with eight KV heads each; our previous article on sustained agent workloads covers the full comparison and the limits we ran into with both models.

We found that parameter count alone says surprisingly little about the practicality of running a large model locally. Even when two models occupy the same amount of memory and advertise identical context windows, their usable KV headroom and long-session decode costs can be worlds apart.

TP=8: About 105 Tokens per Second, but Less KV Headroom

We started with the obvious configuration, distributing the model across all eight GPUs through tensor parallelism. TP=8 reached about 104.9 tok/s in a short single-stream decode, a comfortably high figure for a model of this size. Responses arrived quickly enough to make the interaction feel immediate.

The wide TP distribution was not a particularly elegant fit for the model's mere two KV heads. In this setup, vLLM reported a total KV pool of 474,777 tokens, which works out to roughly 1.81 fully occupied sessions of 262,144 tokens each. For a single fast session, that is a very usable configuration. As a server for several long workstreams, it reaches its memory limit sooner.

TP=2 Plus PP=4: Less Speed, More Than Triple the KV Pool

For the second configuration, we paired the GPUs for tensor parallelism and divided the model across four pipeline stages. The short decode dropped to about 72.6 tok/s, roughly 32 tokens per second behind TP=8, though still a solid speed for a single stream.

The payoff was in the KV cache:

With 1,551,018 available KV tokens, the pipeline-heavy topology provided more than triple the KV capacity — an unusually comfortable margin after the tight budgets we had worked with on MiniMax and HY3.

1.55 Million KV Tokens Are Not a Context Window

The big number needs an important qualification. Qwen122 has a hard per-request context limit of 262,144 tokens, so a KV pool of just over 1.55 million tokens does not allow a single conversation of 1.55 million tokens.

What it does allow is several long sessions at once: alongside a substantial coding session, a separate experimental run, a long research context, and further idle conversations can all stay in the cache without every new request immediately touching off a scramble for the memory that is left. Fully utilized, the budget covers almost six requests at the model's maximum on paper; real sessions will of course vary in size.

The operational value lies in that concurrency, not in an impressive cache figure on a screenshot. Keeping several serious workstreams available side by side changes how we can use the server, turning the model from a one-off big demo into a shared resource.

Long Sessions Became Uneventful for the First Time

With MiniMax and HY3, every large active context cost us extra diagnostic work. We had to watch for longer reasoning loops, drifting language and judgments, or a sharp drop in decode throughput beyond 100,000 active tokens. The problems showed up differently on the two models, but long sessions demanded attention either way.

Our Qwen122 runs were calmer. Throughput in short decodes was high, the pipeline-heavy topology provided ample KV headroom, and long workstreams felt less fragile as a result. This is a qualitative observation from our particular setup, not a universal stability benchmark. For daily use it still mattered at least as much as the measured token rate.

Qwen122 still has clear limitations: it can be more verbose than necessary, it remains slower than smaller models, and it cannot replace current frontier cloud models across the board. Nor does a large KV pool guarantee response quality. It merely removes an operational bottleneck, one that had overshadowed every long session with the other large local models.

Why Qwen Became Our Everyday Model

Our earlier tests had exposed two different limits. MiniMax's impressive performance on short tasks gave way to less stable judgment under precisely the sustained agent workloads we cared about. HY3 remained more convincing over long runs, but paid for its full-attention architecture with steeply rising decode costs; its GPTQ path on vLLM also left very little context headroom.

Qwen122 filled the less glamorous, operationally more valuable role. It was fast enough, its topology could be adapted to the workload, and it handled long contexts more economically. That made it a tool we could simply leave available rather than prepare for one special run.

Getting 122 billion parameters into memory was only the starting condition. Qwen's hybrid attention architecture and two KV heads lowered the price the system paid for long active histories, while enough memory stayed free for us to pursue different operational goals with TP and PP. On the same rig, a similarly sized model built entirely around full attention can be considerably more cumbersome.

Which Topology We Use for Which Workload

For a single interactive session or a demonstration, TP=8 is the clear choice: about 105 tok/s gives us better responsiveness, and the KV pool still holds at least one request at the full context length, with some room to spare.

On a shared agent server, we prefer TP=2 plus PP=4. About 73 tok/s remains perfectly usable for a single stream, while the much larger KV pool keeps several substantial sessions active without dedicating the entire server to one maximum-length context.

The right topology therefore follows the workload. TP=8 favors single-stream speed, TP=2 plus PP=4 the server's available working capacity. Once several real sessions are running in parallel, we value that headroom more highly than the additional throughput of a short single-stream benchmark.

This experiment has changed how we evaluate models. From now on, alongside the per-request context limit, we will document the total KV pool and the concurrency it provides. The 262,144 tokens on the model card tell us how long one session may become; whether the model can actually be used as a server on eight consumer GPUs depends on how many times that session fits into the available memory.