#14 · A Full-Fledged Office Assistant on a Single GPU: Qwen 3.8 27B

27 billion parameters, a large context window, reasoning switched on: before it writes the actual answer, the model first produces a chain of thought. And underneath all of that sits a single RTX 3090 with 24 GB. This combination runs here in continuous operation and decodes at 57.31 tok/s on a short prompt and 58.64 tok/s with roughly 81,000 tokens of context.

Qwen 3.8 27B stands out in that size class. Artificial Analysis — an analysis firm that puts language models through one uniform test procedure and condenses the result into a single figure — lists it at 34 in its Intelligence Index, which puts it first among 140 open-weight models in its size class, where the median is 7. The bare number says little on its own, because it depends on which version of the index produced it; the rank says a great deal. Its strong suits there are coding and tool calling, meaning the ability to invoke tools and systems through defined interfaces. That is a published third-party rating, not a measurement of our own — but those are exactly the two disciplines office automation is built on. Anything expected to produce documents, operate systems and work through a task in an orderly way has to be able to call tools reliably above all else.

It is also a dense model, and that property decides whether it fits on a single card at all. Density cuts both ways here: it costs speed, and it makes the model tolerate the very quantization that gets it onto the card in the first place.

The downside first. Dense means no Mixture-of-Experts: there are no experts the model could skip while it produces a token. Every single token has to pull the entire set of weights through memory bandwidth. Our budget rig built from eBay parts ran into exactly that once before: a dense 27B from an earlier Qwen generation never got past roughly 18 tok/s there on two RTX 3060.

What “dense” costs on a single card

At decode time, a dense model reads its complete set of weights for every token it produces. A Mixture-of-Experts model of the same total size touches only a fraction of them per token. That is why a dense 27B turns out so awkward on narrow memory bandwidth, and why a 35B MoE pulls away from it on the same hardware.

On the server side, the main remedy for this bottleneck is speculative decoding. A fast side path — the drafter — proposes several tokens at once, the target model checks them as a batch and keeps whatever fits. With correct verification, the output distribution of the target model is preserved; only computing it gets faster. For Qwen 3.8 there is a drafter that follows the MTP approach, short for Multi-Token Prediction: predicting several tokens in one step. It can be run in two forms, and our benchmark campaign put both of them up against each other — the model’s built-in drafter, listed in the tables as “built-in MTP”, and a full drafter loaded from a separate file. In continuous operation, that separate file runs in Q4_0.

A fixed package of model, drafter, engine and server flags is what we call a profile from here on. The one we adopted looks like this:

Two of those rows need a word of explanation. One slot means the server handles exactly one request at a time. The KV cache is the buffer that holds the context the server has already read, so that it does not have to recompute it for every new token; here it is quantized as well, in q4_0. The second RTX 3090 in the same machine serves a different model and takes no part in this measurement: every number below comes from one card.

The upside: dense models tolerate Q4

A four shows up three times in that table: at the target model, at the drafter and at the KV cache. The whole exercise hangs on it. As a back-of-the-envelope figure, a 27-billion-parameter model at eight bits per weight takes up roughly 27 GB — more than the card has to begin with. At four bits it is half that, and only then is there room next to the weights for the drafter and for 131,072 tokens of context.

Which raises the obvious question of whether halving the precision makes the model dumber. On a dense model the loss stays small, and the reason is the same one that hurts so much at decode time: all weights get used evenly, so the quantization error spreads out evenly too. Published benchmarks for dense models typically put Q8 and Q4 one to two percent apart, and our own operating experience fits that. On another dense 27B from the Qwen line we noticed no perceptible difference in quality between Q4 and Q8. We have not measured that systematically.

With Mixture-of-Experts models we have seen something markedly different. There, a small router decides which experts handle a token — eight out of 256 in the case of MiniMax M2.5. Quantize that router too coarsely and it picks the wrong experts, at which point it no longer helps that the experts themselves are intact. On top of that, an individual expert is small, around 0.6 billion parameters in this model, and at three to four bits it has barely any precision left for its specialization. Both effects were visible in operation: at 3.28 bits per weight, MiniMax M2.5 stopped following simple instructions, and at 4.08 bits it lost the thread of a conversation within a single turn.

For the question of what fits on one card, that is the difference that actually matters. On paper a MoE model looks like the better candidate for tight memory, because fewer weights move per token — it still has to be loaded in full, though, and it takes the quantization needed for that far less well. The dense 27B is more expensive at decode time, and in exchange it can be squeezed down to four bits.

Where a model like this already works for us

For us this has not stayed a laboratory question. The dense previous generation has been running here in Q4 for months in everyday office work, in exactly those places where the data should not leave the building. Our front-office assistant drafts business letters with it and uses the model in browser-based sessions to maintain and look up data in the CRM system. The feedback from the front office: enthusiastic.

That is operating experience, not a benchmark — but it is months of it, on real tasks, with a dense 27B at four bits.

How we measured

All numbers in this article come from the same setup. A measurement script starts each configuration, sends a warmup run ahead of it and then measures three runs of 256 output tokens each; in the tables they appear as TG1 through TG3. Each configuration goes through two scenarios, which we will call the short run and the long run: once with a short prompt, once with roughly 81,000 tokens of context already in place. All table values are medians of those three runs.

The requests from the script bring their own sampling parameters along and override the server defaults with them: temperature 1.0, top_p 0.95, top_k 0, min_p 0, no repetition penalty. Measure with different sampling, a different prompt, a cold cache instead of a warm one or a different output length, and you are measuring something else.

The long run needs one more note, about bookkeeping. In our measurement protocol it carries the fixed name “100k” — a standardized label, not a measured quantity. After tokenization, the actual runs for this model came out between 80,900 and 82,000 prompt tokens. Reading such a prompt in once, the prefill, covered 80,873 tokens on the first, cold run and went through at 824.96 tok/s in 98.03 seconds.

The run that got the job

The last column of both tables shows the acceptance rate: the share of tokens proposed by the drafter that the target model kept when it checked them. Whether speculation pays off hangs on that number.

So at 81,000 tokens of context this profile decodes no slower than on a short prompt. What stands out is that the acceptance rate moves along with it: 53.3 percent in the short run, 76.3 percent in the long one. With a lot of history behind it, the drafter guesses right far more often. That this higher hit rate outweighs the extra work on the larger KV cache is our reading of the two tables; we have not produced a controlled proof of it.

Against the starting point of the campaign — 262k context, q4 KV, built-in MTP drafter at depth 2 — the short run gains 7.8 percent and the long run 63.9 percent. The price for that is a server-side context window cut in half: 131,072 instead of 262,144 tokens.

The real lever sits in the engine

The largest single contribution comes from the engine, ahead of the choice of drafter. To pin that down we compared two builds of llama.cpp with everything else held identical — q4 KV, 131k context, built-in MTP drafter at depth 4. “Stock” is the unmodified upstream build, “Patch” our own build with eight additional patches on top:

On a short prompt practically nothing happens; at deep context the median climbs by about 51 percent. Four of the eight patches carry most of that; in the code they are called the Ampere MMQ path, the quantized-KV flash attention path, the configurable MMVQ/MMQ crossover and the small-batch MMQ tuning, and the patch stack is public on GitHub. A note for anyone who reflexively updates to the latest upstream: here, the patched build is the one with the lower number.

Three draft tokens, not four

The obvious assumption that a deeper draft buys more does not survive the measurement. In the long run, the target model accepted 529 of 693 tokens proposed by the full drafter at depth 3, or 76.3 percent. Depth 4 managed 520 of 969, which is 53.7 percent, and depth 5 only 488 of 1,367, or 35.7 percent. The extra draft steps create more compute than their hits save.

The second attempt at more speed failed on language. A leaner test drafter passed all 26 technical structure checks, but on our German-language test prompts in the long run it reached only 388 of 1,486 accepted tokens, or 26.1 percent. Its vocabulary selection had been built for English technical text and code, and it does not carry over well enough to a German-language load. It did not go into service.

215.8 tok/s in the session, 42.74 in the benchmark

The most tempting configuration of the whole campaign stacks two speculation methods on top of each other: the MTP drafter plus an n-gram module. That module looks for the most recently generated token sequence in the existing context and proposes whatever followed it there.

In a file editing session over eight turns, that paid off. Throughput rose on average from 92.7 to 215.8 tok/s, and over the last three turns from 93.6 to 230.4 tok/s. That is roughly a factor of 2.3, and it fits the mechanism: a session in which the same file is reworked step by step delivers exactly that kind of repetition.

In the long run of our measurement script, the picture flipped. The same combination fell from 58.64 to 42.74 tok/s there, with a wide spread between the individual runs. Repetition-heavy prompts trigger n-gram hits as well, and having the target model check those hits can cost more than an ordinary MTP step. In our profile, the n-gram module therefore stays switched off: it can be enabled when needed, but it does not start on its own. Whether it pays off only shows in a measurement run with your own typical prompts.

What the thinking costs

Reasoning is on by default in this profile. The engine hands out the thinking portion in a separate reasoning_content field instead of mixing it into the answer. For short answers it can be switched off per request:

"chat_template_kwargs": {"enable_thinking": false}

In the smoke test, the model then answered with exactly OK. Without that option, a tightly cut max_tokens budget can be used up entirely by the thinking portion — the request then ends before the actual answer begins. Two further limits of the profile matter just as much in practice: it has exactly one slot, so concurrent long requests wait for one another, and input and output share the same 131,072 tokens.

What we explicitly did not measure

The campaign tested runtime, stability and throughput, not the substance of the answers. We also have not measured how much this particular model gives up at Q4 compared to Q8; that the loss is small on dense models is broadly documented, but our own verdict on it comes from day-to-day operation. On the technical side, all we have observed so far is that reasoning happens by default, that switching off the thinking portion produced the terse answer we had asked for, and that the German technical benchmark prompts were understood and read as continuous list tasks. That is not a check of answer quality, and certainly not a safety review.

Other points stay open as well. We tested text only; vision is not active. The engine accepts tool schemas through the chat template, but a performance campaign is no substitute for a full check of tool calling. The internal endpoint carries no authentication of its own and therefore has no business sitting unprotected on the network. And anyone who needs a guaranteed 262k of context has to move to a different profile; the speed gain measured here applies to 131k.

For the comparison still ahead of us, we have imposed the same discipline that earlier models required: identical system and user prompt, with model ID, sampling parameters and thinking mode all on record, and at least three repetitions — because under stochastic sampling a single good answer can simply be luck.

What remains

A dense 27B with a 131k context window and reasoning active runs on a single RTX 3090 and decodes at 81,000 tokens of context as fast as it does on a short prompt. Two decisions carry that. Q4_K_M is what gets the weights onto the card at all, and because the model is dense, that quantization costs little as far as we can see so far. The rest comes from the surroundings: a patched engine, q4_0 KV for target and drafter, a full external MTP drafter and a draft depth of three. Four was already too many.

What none of this shows is how good the answers are. We do not have that measurement series yet; that is exactly where the next one starts.