#15 · One Model for the Team: Qwen3.8-27B on Two Four-GPU Clusters

Since late August, Qwen3.8-27B has been the endpoint several of our developers work against at the same time with their coding agents. Behind it sit two clusters of four RTX 3090 each and a proxy of our own that hands out the requests. For the clients, only the address changed; the model name and the interface stayed the same.

In #14 the same model ran on a single card. This article is about the step up from there, and about the question that took the most work along the way: what happens when several agents turn up at the same moment, each carrying a 120,000-token prompt?

The setup

The service occupies eight RTX 3090 in two groups of four. Each group runs its own instance of SGLang, the inference engine that serves the model. Each instance uses tensor parallelism 4, TP=4 for short: every weight matrix is split into four parts, and after each step the four cards merge their intermediate results. The FP8 weights take up 7.62 GB per card.

On top of that comes speculative decoding with DFlash2. A small draft model proposes eight tokens at once, the large model checks them in a single pass and keeps 3.9 of them on average. That is why a single session still reaches 124 to 133 tok/s with roughly 120,000 tokens of context.

The token pool and the request limit are two independent ceilings. The pool holds the cached conversation history of all sessions; the request limit decides how many of them compute at the same time. Our planning assumes five people with two to three subagents each, so 10 to 15 sessions of roughly 100,000 tokens — that does not fit on one group of four. Together, the two clusters carry 1.45 million tokens and twelve concurrent requests.

Why two clusters instead of one big one

A request goes through two phases. First the server reads the prompt in, the prefill, then the answer is produced token by token, the decode. SGLang gives the prefill priority, and under speculative decoding it forcibly switches off mixed operation, the mode in which prefill and decode run together. The reason is that a decode step here is not a single token but a draft plus its verification across eight tokens.

In practice: one prefill of a good 120,000 tokens stalls every answer already running on the same instance for about 136 seconds. With two instances, that hits only half the users. That halving is the real payoff of the split; the extra throughput matters less.

Putting all eight cards into a single instance would have been simpler, and it was measurably worse. In the short run — our measurement profile with a prompt of roughly 400 tokens — tensor parallelism 8 came in 31 percent below TP=4. At decode with about 80,000 tokens of context the gap was 38 percent, with three concurrent sessions 26 percent. The cards sit on PCIe without NVLink, and direct GPU-to-GPU transfer is broken on this board and switched off. Every merge step therefore goes through main memory, and more cards per instance mostly just create more traffic there.

The collapse that forced the split

On August 21 the service fell over under real agent load. Six sessions of a good 120,000 tokens each delivered 7.78 tok/s between them, individual sessions 0.87 to 2.54 tok/s, and the first token arrived after more than 190 seconds. Memory was not the problem: the token pool was 54 percent full. SGLang lines prefill jobs up one after another and lets almost no decode through in between, and six cold sessions are 732,000 tokens of prefill — roughly ten minutes in which nothing else moves.

Two settings fixed that. --prefill-max-requests 2 caps how many prefills run at once, and --enable-session-radix-cache keeps the already-processed part of an active session from being evicted from the cache.

A factor of 30, measured on the slower of the two GPU groups. That took care of the collapse. The 136 seconds of standstill per large prefill stayed; only the second instance did anything about those.

The proxy: why no off-the-shelf router

The two clusters are configured differently. One runs the native context window of 262,144 tokens, the other 524,288 via YaRN — a method that stretches the positional encoding and hands the model a larger window than it was trained for.

SGLang ships a router of its own. It prefers to send a request to the cluster that already has the beginning of that conversation cached, and otherwise it evens out the load. It does not know how long a prompt is, and it does not know the context limits of its own instances either. A request that is too long comes back from the standard cluster as HTTP 400 — and the router retries only on 408, 429 and 5xx.

Since September 4 there has been a proxy of our own in front of the two clusters. It decides in this order:

Rule 4 is why we built the proxy at all. Without it, ten short sessions fight over the six slots on the standard cluster while the other one sits empty. Measured on September 4: ten concurrent short requests, six on the standard cluster, four overflowed to the other, all of them served within 9.7 seconds, none of them queued.

Rule 2 prevents the most expensive mistake, because moving a session to the other cluster means a cold prefill: 67 seconds, measured at roughly 80,000 tokens. The proxy recognizes a session by a hash of its first 4,000 characters — the beginning stays stable across every turn while the conversation grows at the far end. Two sessions with an identical opening would therefore count as one; we have not seen that happen so far.

If the long-context cluster fails, or if we shut it down to use its four cards for something else, everything keeps running over the standard cluster; a watchdog checks availability every five seconds. Requests that are too long then get an HTTP 503 with an explanation.

What the stretched context window costs

Qwen documents the extension beyond 262,144 tokens itself and warns about it in the same breath. Every common framework implements YaRN statically, so the scaling factor applies to a 20k prompt as well and can hurt the quality of shorter texts. Qwen’s advice is to change the positional encoding only when long contexts are genuinely needed.

How much a factor of 2 costs has not been measured anywhere, not by Qwen either. The only numbers come from the YaRN paper on Llama-2, at factors 16 and 32: 0.48 to 1.05 percentage points at short context there. That a factor of 2 costs less is an assumption. Splitting the service sidesteps the question instead of answering it — short requests run on the unmodified model, and only genuinely long ones on the stretched one.

As far as throughput goes, the stretching is free.

Both deviations sit inside the spread; in the short run alone the standard deviation is 9.99 tok/s. The source code fits that picture: the stretched position values are computed once, at startup.

Capacity, on the other hand, drops noticeably. A single session with the full 512k window occupies 524,288 of 650,000 tokens, 81 percent of that cluster. Setting every client to 512k across the board would therefore be a mistake: all sessions would crowd into the long-context cluster, which in practice holds one of them, while the standard cluster runs empty.

The biggest single win had nothing to do with throughput

With zero requests running, the eight cards drew a continuous 1,030.6 W. SGLang’s scheduler keeps asking for work while idle: eight server processes were spinning at 90 to 99 percent CPU load and holding the cards at boost clocks. With --sleep-on-idle, the scheduler waits on a blocking call instead.

775 W saved, 75 percent, which on a rig that spends most of its time waiting comes to roughly 18 kWh a day. The price is 32 milliseconds more on the first token, which nobody notices next to a prefill that runs for seconds. Our assumption that the cards would stay in P2 anyway because a CUDA context was already open turned out to be wrong: it is the constant polling that keeps them up there.

What we have measured under team load — and what we have not

The most solid measurement comes from August 24: 15 to 20 minutes with three coding agents running in parallel. Both clusters were configured identically back then; the stretched context window and our own proxy did not exist yet.

Roughly nine out of ten prompt tokens came from the cache, because every follow-up request only has to read in the part of the conversation that is new. The actual prefill load worked out to about 167 tok/s against a capacity of 1,050 tok/s; the setup was nowhere near saturation.

The limits of those numbers are just as clear. Twenty minutes say nothing about days or about more than three concurrent agents, and the proxy itself has not been measured under real team load at all yet. One incident from the same day is still unexplained: one of three agents started at the same moment went into an infinite loop, which points to a corrupted recurrent state rather than to the scheduler. We rolled back two settings we had introduced shortly before; in the 20 minutes that followed it did not happen again — an indication, not proof.

Reading these numbers correctly means knowing the spread. Most of our measurement series come from the previous rig with the same cards; on the current one we sit about 11.7 percent below that, across 54 runs. The spread is 11.20 tok/s, so three runs per configuration give us ±13 tok/s. Effects smaller than 15 percent are therefore not demonstrable, the YaRN measurement above included. What lies below that line, and what the proxy carries in day-to-day use, will only show up over longer operation.