#14 · Thinking Without Experts: A Dense Qwen 3.8 27B on a Single RTX 3090

A dense model with 27 billion parameters is normally the unattractive choice on a single consumer GPU. It has no experts it could skip at decode time; for every single token, the whole set of weights has to travel through memory bandwidth. Our budget rig built from eBay parts ran into exactly that once before: a dense 27B from an earlier Qwen generation never got past roughly 18 tok/s there on two RTX 3060.

Qwen 3.8 27B is dense as well: a hybrid model without Mixture-of-Experts. We put it on a single RTX 3090 with 24 GB, ran a benchmark campaign across twelve configurations and moved one profile into continuous operation at the end of it. Its medians are 57.31 tok/s on a short prompt and 58.64 tok/s with roughly 81,000 tokens of context. The second number is the more interesting one.

What “dense” costs on a single card

At decode time, a dense model reads its complete set of weights for every token it produces. A Mixture-of-Experts model of the same total size touches only a fraction of that per token. This is why a dense 27B turns out so awkward on narrow memory bandwidth, and why a 35B MoE pulls away from it on the same hardware.

On the server side, the main remedy for this bottleneck is speculative decoding. A fast draft path proposes several tokens at once, and the target model verifies them together. With correct verification, the output distribution of the target model is preserved; only the computation of that distribution is accelerated. There is a dedicated MTP drafter for Qwen 3.8, which we load as a full external file in Q4_0. That makes the qualified profile look like this:

The second RTX 3090 in the same machine serves a different model and takes no part in these measurements. All numbers below come from one card.

How we measured

For each target, our canonical harness runs one warmup and three decode runs of 256 output tokens each. The stored payloads override the server defaults: temperature 1.0, top_p 0.95, top_k 0, min_p 0, no repetition penalty. Every value in the tables is the median of those three runs. Measure with different sampling, a different prompt, a differently warmed cache or a different output length, and you are measuring something else.

A word about the label: the “100k” profile is called that because it is the standardized target name in the measurement protocol. After tokenization, the actual runs for this model came out between 80,900 and 82,000 prompt tokens. The first cold prefill covered 80,873 tokens and went through at 824.96 tok/s in 98.03 seconds.

The run that got the job

So at 81,000 tokens of context, this profile does not decode any slower than on a short prompt. The acceptance rate moves in step with it, which is the striking part: 53.3 percent on the short prompt, 76.3 percent on the long one. With a lot of history behind it, the drafter guesses right far more often. That this higher hit rate outweighs the extra cache work is our reading of the two tables; we have not run a controlled proof of it.

Against the original baseline profile with 262k context, q4 KV and the built-in MTP drafter at depth 2, the short median gains 7.8 percent and the long median 63.9 percent. The price for that is a server-side context window cut in half: 131,072 instead of 262,144 tokens.

The real lever sits in the patch stack

The largest single contribution comes from the engine, ahead of the choice of drafter. With identical q4 KV, 131k context and the built-in MTP drafter at depth 4:

On a short prompt practically nothing happens; at deep context the median climbs by about 51 percent. Of the eight patches, the ones that matter here are the Ampere MMQ path, the quantized-KV flash attention path, the configurable MMVQ/MMQ crossover and the small-batch MMQ tuning; the patch stack is public on GitHub. A note for anyone who reflexively updates to the latest upstream: here, the patched build is the one with the lower number.

Three draft tokens, not four

The obvious assumption that a deeper draft buys more does not survive the measurement. In the long profile, the full drafter at depth 3 had 529 of 693 proposed tokens accepted. Depth 4 managed 520 of 969, or 53.7 percent, and depth 5 only 488 of 1,367, or 35.7 percent. The additional draft steps create more compute than their hits save.

A second variant failed on language. A leaner test drafter passed all 26 technical structure checks, but on our German-language test load at long context it reached only 388 of 1,486 accepted tokens, or 26.1 percent. Its vocabulary selection had been built for English technical text and code, and it does not carry over well enough to a German-language load. It did not go into service.

215.8 tok/s in the session, 42.74 in the benchmark

Stacked speculation from MTP plus an n-gram module was the most tempting arm of the whole campaign. Across an eight-part file editing session, throughput rose on average from 92.7 to 215.8 tok/s, and over the last three turns from 93.6 to 230.4 tok/s. That is roughly a factor of 2.3, and it fits the mechanism: the method looks for the most recently generated token sequence in the existing context and proposes whatever followed it there. An editing session supplies exactly that kind of repetition.

In the canonical long profile, the same arm dropped from 58.64 to 42.74 tok/s, with a wide spread. Repetition-heavy prompts trigger n-gram hits whose verification can cost more than an ordinary MTP step. We therefore keep the mode available outside the standard profile and do not start it automatically. A technique whose payoff depends this strongly on the workload belongs in the hands of whoever knows the workload.

What the thinking costs

Reasoning is on by default in this profile, and the engine hands out the thinking portion in a separate reasoning_content field instead of mixing it into the answer. For short answers it can be switched off per request:

"chat_template_kwargs": {"enable_thinking": false}

In the verified smoke test, the model then answered with exactly OK. Without that option, a tightly cut max_tokens budget can be used up entirely by the thinking portion. A model that thinks can spend its whole answer budget on thinking and have nothing left to say afterwards. Two further limits of the profile matter just as much in practice: the profile has exactly one slot, so concurrent long requests wait for one another, and input and output share the same 131,072 tokens.

What we explicitly did not measure

The checkpoint we qualified is a fine-tune called Qwen3.8-27B-Uncensored-OrcaRouter. “Uncensored” is the name of the fine-tune and not an assurance: it does not promise that every request will be answered, nor that answers are safe, legal or factually correct. Quantization and fine-tuning can change behavior and factual accuracy.

We have no systematic, reproducible behavioral comparison between this fine-tune and the official Qwen3.8-27B. The campaign qualified runtime, stability and throughput. On the technical side, all we have observed so far is that reasoning happens by default, that switching off the thinking portion produced the terse answer we had asked for, and that the German technical benchmark prompts were understood and read as continuous list tasks. That is not a quality eval, and certainly not a safety eval.

Other points stay open as well. The qualification was purely text-based; vision is not active. The engine accepts tool schemas through the chat template, but a performance campaign is no substitute for a full tool-calling eval. The internal endpoint carries no authentication of its own and therefore has no business sitting unprotected on the network. And anyone who needs a guaranteed 262k of context has to move to a different profile; the speed gain measured here applies to 131k.

For the comparison still ahead of us, we have imposed the same discipline that earlier models required: identical system and user prompt, with model ID, sampling parameters and thinking mode all on record, and at least three repetitions under stochastic sampling. A single fine answer is an impression, not a property of the model.

What remains

A dense 27B decodes on a single RTX 3090 at 81,000 tokens of context as fast as it does on a short prompt, with a 131k context window and reasoning active. The gain did not come from the model but from everything around it: a patched engine, q4_0 KV for target and drafter, a full external MTP drafter and a draft depth of three. Four was already too many.

About the answer quality of this fine-tune we can say nothing solid so far. The measurements in this campaign do not reach that far; that is exactly where the next one starts.