#17 · Qwen 3.8 Flash Next 176B: Our Workhorse’s Well-Read Brother

There is a saying about doctors: surgeons know nothing and do everything, internists know everything and do nothing. Our workhorse, the small Qwen 3.8 27B from #14, plays the surgeon here: it codes superbly and handles tools reliably. It just doesn’t know much. At only 27 billion parameters that is no surprise, and plenty of reports have it drifting into hallucination on knowledge questions. Qwen3.8-Flash-Next is considerably larger and can close part of that gap while matching it on tool calling and coding — which matters to us in project analysis, code review and requirements gathering.

On our rig the large model now runs at 83.9 tok/s with 100,000 tokens of context, in a context window of 524,288 tokens, on six of ten graphics cards. The remaining four stay free for production work. At the vendor, Artificial Analysis measures 71.9 tok/s, and there the model costs $0.15 per million input tokens and $0.47 per million output tokens. The two speeds are not cleanly comparable: different measurement procedure, different load point.

On the knowledge question there is only one outside number so far: Artificial Analysis puts Qwen3.8-Flash-Next at 42 on its Intelligence Index and the 27B at 34. The eight-point gap is real, but the 42 is explicitly flagged there as an estimate, and independent verification is still pending. It is not a head-to-head anyway, because Artificial Analysis measures the two against different peer groups: the 27B against models of its own size, the large one against considerably larger ones.

Where the knowledge sits

The picture of the surgeon has a solid technical counterpart: knowledge sits in the stored parameters, skill shows up in the computation. The 27B has to fit both into 27 billion parameters, and every token runs through all of them. Giving the model more knowledge means adding parameters there, and those parameters then get read for every single token. The more it knows, the slower it gets.

Qwen3.8-Flash-Next breaks that coupling. Most of the model consists of 512 subnetworks, the experts. For each token a small upstream stage, the router, picks ten of them; one further expert always runs as well. The model stores 176 billion parameters — Artificial Analysis lists 180, because it counts the small draft head as well — and computes six billion of them per token. That gives the model many times more room for knowledge than the 27B, without computing any more slowly for it.

Of those 176 billion, 51.2 billion go to a lookup table alone; the remaining 125 billion or so are the model proper. In operation the table sits in the machine’s main memory rather than on the graphics cards, roughly 102 GB of it. That clears up a number that doesn’t add up at first: the download of the build we run is 180 GB, and only about 73 GiB of that ends up on the cards.

The six billion active parameters are enough to work out what should be possible in theory. At roughly one byte per parameter the card reads about 6 GB per token, and a 3090 has 936 GB/s of memory bandwidth:

6 GB read per token
at 936 GB/s memory bandwidth
    = about 156 tok/s, on paper, on a single card

What we actually measured, on eight cards, was 51 tok/s. Where that gap comes from is the more interesting part of the work; it is further down.

From 28 to 84 tok/s

We started with the inference engine ik_llama.cpp and a build squeezed down to four bits in the GGUF format, the weight format of the llama.cpp family: 28.1 tok/s. Four bits means that every weight value is stored with four bits instead of sixteen. At full resolution the model would be 354 GB.

The very first number we saw was a prettier one: 58.94 tok/s, reported by the engine itself, measured at 21 prompt tokens and therefore at the friendliest operating point imaginable. Under our own measurement procedure — roughly 1,000 tokens of prompt, 256 output tokens, median of three runs, the short run from here on — what remained was 28.1. Self-reported figures are not comparable with the benchmark.

That engine never gave us more than 29.8 tok/s, and that was already the best of the speculation variants we tried. The jump came from switching to a vLLM fork in which the model’s architecture was already supported: 51.8 tok/s with the eight-bit build on eight cards. The four-bit build added 56.1 on four cards.

The first four values come from the short run, the last two from the run with 100,000 tokens of context. Both measurement points belong in the picture: the final configuration delivers 71 tok/s in the short run and 83.9 at 100,000 tokens. The long run is faster here than the short one.

The next big step came from speculative decoding. The model brings along a small draft head for it: the head proposes several tokens at once, the large model checks them in a single pass and keeps the hits. That turned 56.1 into 79.7 tok/s. The rest came from widening the context window, more on that further down.

A warning in passing: six extra flags, all of them straight out of the fork’s own documentation, cost 23 percent of throughput when switched on together. We did not measure them individually; we have run without them since.

More cards make it slower

The finding that runs hardest against intuition: adding cards does not help. With the four-bit build, eight cards delivered 51.9 tok/s and four cards 56.1. With the eight-bit build, ten cards came in at 46.3 tok/s, behind the eight at 51.8. There are four reasons for that.

A lot of it is held twice on every card. The router with its 512 entries, the running states of the layers and the tables that translate tokens into number vectors sit at full size on every card involved. You can see it in the memory footprint: eight cards held 165 GiB between them, even though only 125 GiB of the eight-bit build have to be on the cards at all. Every additional card therefore costs a fixed surcharge in memory before it has computed its first token.

With a single request the computation is too small. Each of the ten active experts gets exactly one token. The matrix being multiplied has exactly one row — and the tensor cores of a 3090 only get going properly at around 32 rows.

The merging runs over PCIe. After every computation step the cards have to reconcile their intermediate results. Our cards have no NVLink, and direct card-to-card transfer is switched off on GeForce anyway. Every merge therefore goes through main memory, and the more cards are involved, the more stages it takes.

36 of the 48 layers are recurrent. They carry forward a state that the next token needs. That cannot be parallelized across tokens in principle, no matter how many cards stand ready.

That leaves the question of why it ended up being six cards in particular. The number of cards a weight matrix is split across has to divide 16: the model’s vision part has 16 attention heads, so 16 parallel part-computations that can only be spread evenly. Six does not divide 16, and startup aborts with a message saying exactly that. Six cards therefore only work in two stages: two cards each share the matrices, and three such pairs in sequence share the model’s 48 layers. That arrangement is what delivers the 83.9 tok/s — and it leaves four cards free for other models.

Four proposals, two hits

Speculative decoding comes with one knob: how many tokens the draft head should propose at once. The values in between are instructive, because they refute an obvious assumption.

The highest hit rate belongs to three proposals, the highest throughput to four. What counts is the product of chain length and rate: four proposals at 58.2 percent give 2.3 accepted tokens per cycle, three proposals at 67.1 percent only 2.0. A configuration picked by acceptance rate would have been the slower one.

There is a ceiling to it. At six proposals it tips over to 70.8 tok/s, because checking the longer chain costs more than the extra hits bring in. We run four.

Four bits is not a defined measure

Before the very first startup came the question of which build to download at all. We catalogued 214 public repositories with ready-made quantizations and added up their file sizes. The result is a warning that reaches beyond this model: the names of the quantization levels are not comparable between providers.

The same label — Q4_K_M — stands for 88.9 GB at one provider, 119.2 at another, 135.4 at a third and 238.3 GB at a fourth. That last value is above the model’s eight-bit build, which measures 188.2 GB, and is probably just broken. The cause is always the same: every provider treats the large lookup table differently. Some quantize it along with the rest, some leave it in the package at full resolution — and the sizes under one and the same name drift apart accordingly. Anyone who picks a build by its name is buying sight unseen.

On quality there is more to say than you would expect, but only for some of the builds. Structurally, every provider quantizes the layer of 512 experts and nothing else; the attention layers, the token translation tables and the draft head stay untouched. Intel measures 99.64 percent of the full build’s quality for its own four-bit version, averaged over four benchmarks. For the GGUF quantizations, on the other hand, we know of no published quality numbers. Every usable measurement comes from builds in the safetensors format that vLLM and related engines load; carrying those over to the GGUF files is plausible and unproven.

The context window you have to switch on yourself

Qwen3.8-Flash-Next ships with 262,144 tokens of context. The million you often read about only comes about through YaRN — a method that stretches the positional encoding and hands the model a larger window than it was trained for. Nothing turns that on by itself; it has to be configured at startup.

We run 524,288 tokens, so a factor of 2. Throughput does not suffer for it, quite the opposite: the 83.9 tok/s at 100,000 tokens were measured with the stretched window and sit above the 79.7 without it. What is untested is the output quality at 300,000 or 500,000 tokens. Working seriously in that range means measuring the quality on your own tasks first.

The KV cache holds 793,921 tokens, spread across all concurrent sessions. At 50,000 tokens per session that is about 15 conversations side by side, at 200,000 still four, at 400,000 exactly one.

It gets interesting when more is requested than fits. We tested that with a separate random text for every request, so the service could not reuse anything: 405,031 prompt tokens per request against a KV cache of 793,921.

The service works through the requests cleanly, one after another. None was evicted, none failed, and the wait grows linearly with the queue. Asking for too much therefore means waiting longer; nobody gets turned away.

The first attempt at this measurement was useless, and instructively so: all six requests carried the same text. The service recognized the beginning it had already processed and was done after 6.5 seconds. What we had measured was cache reuse, not the overflow.

When no error message comes

At startup we gave one parameter a name the software does not know. The consequence: exit code 0, no error message, no entry in the container’s log, and a log file that breaks off after a good three kilobytes in the middle of initialization. From the outside that looks like a clean exit. Because the container was set to bring itself back up, that produced 35 start attempts in a row.

The obvious suspicion was that memory had run out. It was wrong; the system log showed not a single entry to that effect. The software’s documentation, incidentally, describes a visible error message for this case — this container behaves differently. When it dies without any message at all, there is a typo in a startup parameter behind it, not a memory problem.

The second case is more insidious. The knob that sets what share of the graphics memory the service claims for itself can be set to 0.95. The service starts with that, builds a larger cache and reports itself ready. Every single inference request after that fails with out-of-memory and HTTP 500. A readiness check that only asks whether the server answers considers this state healthy. The maximum is 0.92.

What the small model keeps doing

The 27B stays in service, and one number from Artificial Analysis explains nicely why we don’t use it everywhere: for the Intelligence Index it burned 200 million output tokens against a median of 54 million in its size class. It is remarkably wordy. That matches what we measured ourselves in the addendum to #15 — the model spends much of its output budget in the thinking part, before the actual answer even begins. Two independent routes, the same finding.

We looked at the large model’s output budget as well, though only with a spot check. The switch for thinking effort has several levels. A single short question produced 235 output tokens on the lowest level and 123 on the standard level — almost twice as many for the instruction to keep it short. A single short question is not a measurement series. We drew nothing from it and are staying on standard.

A few things stay open, and that belongs to the picture:

• The vision part sits inside the model; we have never addressed it. For our clients the model is registered as a pure text model.
• Output quality at very long context is untested. The 524,288 tokens are a throughput statement.
• There are no published quality numbers for the GGUF quantizations. That is why we run a four-bit build in the safetensors format, the one format for which measurements exist at all.
• And the core question from the opening is still open: whether the large model really closes those knowledge gaps is something we will only know once we have measured it against our own tasks. Until then the Artificial Analysis index is an estimate, and our expectation is only an expectation.