#15 · One Model for the Team: Qwen3.8-27B on Two Four-GPU Clusters

Since late August, Qwen3.8-27B has been the endpoint several of our developers work against at the same time with their coding agents. Behind it sit two clusters of four RTX 3090 each and a proxy of our own that hands out the requests. For the clients, only the address changed; the model name and the interface stayed the same.

In #14 the same model ran on a single card. This article is about the step up from there, and about the question that took the most work along the way: what happens when several agents turn up at the same moment, each carrying a 120,000-token prompt?

The setup

The service occupies eight RTX 3090 in two groups of four. Each group runs its own instance of SGLang, the inference engine that serves the model. Each instance uses tensor parallelism 4, TP=4 for short: every weight matrix is split into four parts, and after each step the four cards merge their intermediate results. The FP8 weights take up 7.62 GB per card.

On top of that comes speculative decoding with DFlash2. A small draft model proposes eight tokens at once, the large model checks them in a single pass and keeps 3.9 of them on average. That is why a single session still reaches 124 to 133 tok/s with roughly 120,000 tokens of context.

The token pool and the request limit are two independent ceilings. The pool holds the cached conversation history of all sessions; the request limit decides how many of them compute at the same time. Our planning assumes five people with two to three subagents each, so 10 to 15 sessions of roughly 100,000 tokens — that does not fit on one group of four. Together, the two clusters carry 1.45 million tokens and twelve concurrent requests.

Why two clusters instead of one big one

A request goes through two phases. First the server reads the prompt in, the prefill, then the answer is produced token by token, the decode. SGLang gives the prefill priority, and under speculative decoding it forcibly switches off mixed operation, the mode in which prefill and decode run together. The reason is that a decode step here is not a single token but a draft plus its verification across eight tokens.

In practice: one prefill of a good 120,000 tokens stalls every answer already running on the same instance for about 136 seconds. With two instances, that hits only half the users. That halving is the real payoff of the split; the extra throughput matters less.

Putting all eight cards into a single instance would have been simpler, and it was measurably worse. In the short run — our measurement profile with a prompt of roughly 400 tokens — tensor parallelism 8 came in 31 percent below TP=4. At decode with about 80,000 tokens of context the gap was 38 percent, with three concurrent sessions 26 percent. The cards sit on PCIe without NVLink, and direct GPU-to-GPU transfer is broken on this board and switched off. Every merge step therefore goes through main memory, and more cards per instance mostly just create more traffic there.

The collapse that forced the split

On August 21 the service fell over under real agent load. Six sessions of a good 120,000 tokens each delivered 7.78 tok/s between them, individual sessions 0.87 to 2.54 tok/s, and the first token arrived after more than 190 seconds. Memory was not the problem: the token pool was 54 percent full. SGLang lines prefill jobs up one after another and lets almost no decode through in between, and six cold sessions are 732,000 tokens of prefill — roughly ten minutes in which nothing else moves.

Two settings fixed that. --prefill-max-requests 2 caps how many prefills run at once, and --enable-session-radix-cache keeps the already-processed part of an active session from being evicted from the cache.

A factor of 30, measured on the slower of the two GPU groups. That took care of the collapse. The 136 seconds of standstill per large prefill stayed; only the second instance did anything about those.

The proxy: why no off-the-shelf router

The two clusters are configured differently. One runs the native context window of 262,144 tokens, the other 524,288 via YaRN — a method that stretches the positional encoding and hands the model a larger window than it was trained for.

SGLang ships a router of its own. It prefers to send a request to the cluster that already has the beginning of that conversation cached, and otherwise it evens out the load. It does not know how long a prompt is, and it does not know the context limits of its own instances either. A request that is too long comes back from the standard cluster as HTTP 400 — and the router retries only on 408, 429 and 5xx.

Since September 4 there has been a proxy of our own in front of the two clusters. It decides in this order:

Rule 4 is why we built the proxy at all. Without it, ten short sessions fight over the six slots on the standard cluster while the other one sits empty. Measured on September 4: ten concurrent short requests, six on the standard cluster, four overflowed to the other, all of them served within 9.7 seconds, none of them queued.

Rule 2 prevents the most expensive mistake, because moving a session to the other cluster means a cold prefill: 67 seconds, measured at roughly 80,000 tokens. The proxy recognizes a session by a hash of its first 4,000 characters — the beginning stays stable across every turn while the conversation grows at the far end. Two sessions with an identical opening would therefore count as one; we have not seen that happen so far.

If the long-context cluster fails, or if we shut it down to use its four cards for something else, everything keeps running over the standard cluster; a watchdog checks availability every five seconds. Requests that are too long then get an HTTP 503 with an explanation.

What the stretched context window costs

Qwen documents the extension beyond 262,144 tokens itself and warns about it in the same breath. Every common framework implements YaRN statically, so the scaling factor applies to a 20k prompt as well and can hurt the quality of shorter texts. Qwen’s advice is to change the positional encoding only when long contexts are genuinely needed.

How much a factor of 2 costs has not been measured anywhere, not by Qwen either. The only numbers come from the YaRN paper on Llama-2, at factors 16 and 32: 0.48 to 1.05 percentage points at short context there. That a factor of 2 costs less is an assumption. Splitting the service sidesteps the question instead of answering it — short requests run on the unmodified model, and only genuinely long ones on the stretched one.

As far as throughput goes, the stretching is free.

Both deviations sit inside the spread; in the short run alone the standard deviation is 9.99 tok/s. The source code fits that picture: the stretched position values are computed once, at startup.

Capacity, on the other hand, drops noticeably. A single session with the full 512k window occupies 524,288 of 650,000 tokens, 81 percent of that cluster. Setting every client to 512k across the board would therefore be a mistake: all sessions would crowd into the long-context cluster, which in practice holds one of them, while the standard cluster runs empty.

The biggest single win had nothing to do with throughput

With zero requests running, the eight cards drew a continuous 1,030.6 W. SGLang’s scheduler keeps asking for work while idle: eight server processes were spinning at 90 to 99 percent CPU load and holding the cards at boost clocks. With --sleep-on-idle, the scheduler waits on a blocking call instead.

775 W saved, 75 percent, which on a rig that spends most of its time waiting comes to roughly 18 kWh a day. The price is 32 milliseconds more on the first token, which nobody notices next to a prefill that runs for seconds. Our assumption that the cards would stay in P2 anyway because a CUDA context was already open turned out to be wrong: it is the constant polling that keeps them up there.

What we have measured under team load — and what we have not

The most solid measurement comes from August 24: 15 to 20 minutes with three coding agents running in parallel. Both clusters were configured identically back then; the stretched context window and our own proxy did not exist yet.

Roughly nine out of ten prompt tokens came from the cache, because every follow-up request only has to read in the part of the conversation that is new. The actual prefill load worked out to about 167 tok/s against a capacity of 1,050 tok/s; the setup was nowhere near saturation.

The limits of those numbers are just as clear. Twenty minutes say nothing about days or about more than three concurrent agents, and the proxy itself has not been measured under real team load at all yet. One incident from the same day is still unexplained: one of three agents started at the same moment went into an infinite loop, which points to a corrupted recurrent state rather than to the scheduler. We rolled back two settings we had introduced shortly before; in the 20 minutes that followed it did not happen again — an indication, not proof.

Reading these numbers correctly means knowing the spread. Most of our measurement series come from the previous rig with the same cards; on the current one we sit about 11.7 percent below that, across 54 runs. The spread is 11.20 tok/s, so three runs per configuration give us ±13 tok/s. Effects smaller than 15 percent are therefore not demonstrable, the YaRN measurement above included. What lies below that line, and what the proxy carries in day-to-day use, will only show up over longer operation.

Addendum, September 8, 2026: Without a cap on the thinking phase, no answer comes back

Qwen3.8 thinks before it answers: ahead of the actual reply, the model produces an internal train of reasoning, the thinking phase. On hard tasks that reasoning never finds an end. In five runs out of five, the model spent the entire output budget inside the thinking phase — about 32,700 thinking tokens out of 32,768 allowed — and after an average of 277 seconds it had produced not one line of code. The model does not stop on its own; it fills whatever it is given.

This is the same failure that hit our coding agents in late August: the orchestration layer above them reported empty responses, and for a long time we could not say why. Now we can reproduce it.

The obvious lever against this would be the reasoning_effort setting, which we had left on medium for a long time. In our setup it does nothing. The model’s chat template — the template that turns the messages of a conversation into the actual prompt text — knows only three levels and raises an error on anything else, yet our server accepts high and max without complaint. So the value never reaches the template at all: SGLang reduces it beforehand to a yes/no on thinking. The measurements on one and the same question fit that: xhigh 959, medium 958, low 996 thinking tokens. That also disproves an earlier note of our own, which had medium coming out three and a half times faster.

What does work is a hard budget on the token choices. At every step a mask decides which tokens may be drawn at all; once the thinking budget is used up, the only thing left in that mask is the token that closes the thinking phase. The model has to stop thinking and start answering — not because the prompt asks it to, but because sampling leaves it nothing else. The setting lives in the environment variable SGLANG_MAX_THINK_TOKENS, running at 5000 since September 8. A per-request value exists only on the engine’s native path, not on the OpenAI-compatible endpoint; clients cannot send one along, and the budget applies to everyone.

That mask has to be enforced somewhere, and that comes at a price: the budget forces a change to the grammar backend, the component that constrains which token sequences are allowed. Ours had been switched off and now runs a real implementation. We checked: the switch costs no GPU memory, the usage figures stayed where they were, and the model still calls tools.

We measured on a task that leaves no room for generous grading: a function that converts duration strings like 1h30m into seconds, checked against 38 real test cases. The generated code gets executed, and what counts is the number of tests it passes. 16 of the cases expect an error to be raised, so a stub that always throws would score 16 out of 38. That is the bar to clear. Five runs per level:

5000 has the best mean and the best worst run, for eight seconds more than 4000. At 7000 it tips: one run converted 1h30m into 1800 seconds instead of 5400, breaking the most basic rule of the task. That matches what the paper “When More Thinking Hurts” (ACL Findings 2026) calls overthinking — the model discards an intermediate result that was already correct, after weighing it one more time. The paper puts the tipping point at roughly 7000 tokens as well, measured on a different model and on competition math. Two independent routes arriving at the same order of magnitude make the number easier to trust.

This series does not carry much: five runs per level, one single task. The differences between 4000, 5000 and 6000 come to one or two tests out of 38 and are not significant. Two statements hold up: 7000 is worse than the 4000 to 6000 range, and without a budget nothing arrives at all.

The budget is a ceiling, not a target. Simple requests get by on 500 to 1900 thinking tokens and never come near it. It only bites in the cases that run away anyway — which are exactly the ones that used to come back empty.

One side finding to close on: two identical requests at temperature 0, a setting that should draw the same token every time, came out at 996 and 1144 thinking tokens — 15 percent spread. Speculative decoding is not bit-exact on our setup. That matters for measurements, including the five runs per level above.