← Insights

DE — Deutsche Version

Beyond 60,000 Tokens: MiniMax M2.7 Under Sustained Agent Load

· Filip

With 83 to 90 tokens per second in a single stream, around 181 tokens per second across four parallel requests, and remarkably strong performance on short tasks, our hybrid AWQ of MiniMax M2.7 initially looked like an excellent balance of speed and quality. For local inference on eight consumer GPUs, those were numbers we could work with.

Then the sessions got longer.

Our actual workloads rarely consist of one question followed by a short answer. Agents read files, call tools, compare results, correct intermediate steps, and carry the entire conversation history forward. An evaluation of eleven profiles, complete with ranking, Markdown report, and several form-style documents, can quietly push the active context into the range of 60,000 to well over 100,000 tokens. That was where MiniMax started to behave differently: reasoning loops grew longer, evaluations varied between identical runs, and Chinese or other foreign-language fragments occasionally appeared in German output.

That shifted the question we were trying to answer. The maximum context window had become little more than a capacity figure. For production use, we needed to know how quickly, coherently, and consistently the model could still perform with 60,000, 80,000, or 100,000 genuinely active tokens.

This article covers practical runs on one specific rig, using the documented quants, engines, and launch parameters for each configuration. These results are not universal model benchmarks. They show how these setups behaved under our sustained agent workload, including the points at which model, quantization, runtime, and hardware became almost impossible to disentangle cleanly.

Eight Consumer GPUs Are Not One Pool of Memory

Our local LLM rig contains seven RTX 3090 cards and one RTX 3090 Ti. Together, they provide around 192 GB of VRAM. They sit on a consumer PCIe topology without NVSwitch and, in our build, without a usable NVLink fabric. Large models may fit into memory, but the work still has to be distributed across eight independent GPUs and several PCIe root complexes.

Four variables therefore interacted in every serious test:

  • the distribution of model weights and KV cache across the cards,
  • PCIe communication overhead,
  • quantization format and engine,
  • decode throughput with a long active context.

The last point barely registered in short benchmarks. A short prompt mainly measures how quickly the engine can move weights and process a small KV cache. A long agent run makes every new token perform additional work over a large active history. Depending on the attention architecture, that cost can grow very differently.

The Hybrid AWQ Solved the Quantization Problem, at First

In our previous article on quantization, we explained why a nominal bit count tells us very little about MiniMax. The fast candidate we eventually chose for production was ayysasha/MiniMax-M2.7-AWQ-G32-STRIX-2H: INT4 experts save most of the memory, while attention, the router, and other sensitive areas remain at higher precision.

The key specifications of our first production path were:

Component Configuration
Expert tensors INT4
Attention BF16
Router/Gate BF16 or high precision
KV cache int8_per_token_head
Parallelism TP=8 plus Expert Parallelism
Engine vLLM
Single stream, short context approximately 83 to 90 tok/s
four parallel requests, synthetic approximately 181 tok/s aggregate

On short technical-specification tasks, the model produced coherent output and working tool calls. For this path, that effectively settled the quantization question: the model was fast, and its capabilities had not suffered the same obvious damage we saw with the blanket W4A16 quant. What remained unanswered was whether it could sustain that quality through a long, tool-heavy session.

A Context Window Can Be Huge and Still Barely Usable

There is a substantial gap between a server accepting a long prompt and a model continuing to work reliably inside it. The configured maximum initially tells us whether the weights, KV cache, and runtime buffers fit in memory. It says little about how three other factors develop over the course of a session:

  1. How far does the decode rate drop when every new token has to account for a very long history?
  2. Do reasoning, language, and tool calls remain stable?
  3. Does the model produce comparable judgments when an evaluation task is repeated?

For agent workloads, the third question matters more than one spectacular output. A model can write every file correctly and formally complete the process. If the same candidates jump several places in the ranking when the task is unchanged, the run has finished technically, but its decision cannot be trusted.

MiniMax began to degrade between 60,000 and more than 100,000 active tokens, and the symptoms took several forms:

  • The reasoning repeated the same analysis with slightly different wording.
  • Individual reasoning blocks grew to many thousands of tokens.
  • German output contained Chinese tokens or other multilingual fragments.
  • Tool calls failed to terminate cleanly or fell into repeated write or edit attempts.
  • Identical evaluation tasks produced markedly different rankings.

Not every run showed every symptom, which made the diagnosis harder. A deterministic crash is comparatively easy to assign to a particular layer. A session that works for forty minutes, saves six correct files, and reaches a fundamentally different conclusion from the day before is more polite operationally and more problematic on the merits.

vLLM 0.25.1 Stopped the Worst Loops, but Not the Instability

We upgraded the MiniMax path to vllm/vllm-openai:v0.25.1, used the native minimax_m2 parser for reasoning and tool calls, and switched the KV cache to int8_per_token_head. We also applied this sampling recipe:

{
  "temperature": 0.6,
  "top_p": 0.95,
  "top_k": 40,
  "min_p": 0.01,
  "repetition_penalty": 1.1,
  "frequency_penalty": 0.4
}

The change produced a visible improvement. The earlier outright garbling and the worst verbatim reasoning loops became less frequent. Version 0.25.1 also fixed the parser regression we had encountered in an interim build. What remained was less obvious: very large reasoning blocks, trouble completing some tool calls, and above all, volatile evaluations.

For a controlled comparison, we ran the same eleven-profile task five times on the same vLLM stack. All five repetitions produced the six expected artifacts. By a purely operational measure, the success rate was therefore five out of five.

The actual content told a different story:

Observation across five repetitions Result
complete deliverables 5 out of 5
highest total score awarded per run 158, 79, 75, 105, 80
first place person_11 three times, plus person_09 and person_06
example placement of one profile 5 -> 6 -> 3 -> 1 -> 7
reasoning tokens per run approximately 6,700 to 12,900
largest individual reasoning block approximately 10,070 tokens
runtime per repetition approximately 18 to 38 minutes

Some variation in the exact order is expected from stochastic generation. Here, however, the winner, scoring scale, and placement of individual candidates moved so much that the five runs did not produce a stable ranking on the merits. The model could create the files and execute the workflow; under this sustained load, its evaluations were not reproducible enough.

A counter-test with SGLang left the result unchanged. SGLang was around 26 percent faster in the short decode microbenchmark. In the real high-context run, eager prefill made it slower and more prone to runaways. The instability therefore extended beyond vLLM. Engine, KV format, and parser shaped the failure pattern without eliminating the long loops or inconsistent judgments.

The Reconstructed IQ5_K Path Was Slower but Surprisingly Solid

We also had an older fallback based on the 5-bit GGUF ubergarm/MiniMax-M2.7-GGUF in IQ5_K and ik_llama.cpp. Post 2 covers its differences from the hybrid AWQ in detail; only two points matter operationally here. The quant delivered usable quality, and its performance depended surprisingly heavily on the exact launch recipe.

For a while, we believed that this path was no longer reproducible. We later recovered the command we had actually used from a local OpenCode session database. Several seemingly minor differences had skewed our reconstructions:

  • q4_0 instead of q8_0 for the key and value cache,
  • the balanced tensor split,
  • leaving graph split disabled,
  • the original mmap and cache settings,
  • the image ik-llama-cuda:main-20260612-full.

With that setup restored, the earlier figures returned:

  • approximately 49.5 tok/s with a short context,
  • approximately 350 tok/s prompt processing at around 80,000 tokens,
  • approximately 13.1 tok/s decode at around 80,000 tokens of active context,
  • coherent output that was usable for the tasks we observed.

This path was clearly slower than the hybrid AWQ on vLLM. It also showed how quickly an incorrect reconstruction can lead to a false conclusion about a model or quant. In a distributed inference system, “the same model” is not a complete experimental description.

--max-gpu Had a Measurable but Modest Effect

For graph mode, the ik_llama fork provides --max-gpu N and -grt, among other options. --max-gpu 4 does not restrict the run to four cards. All eight GPUs can still hold parts of the model globally, but any single layer is processed on no more than four cards. In theory, this reduces the number of participants in layer-local reductions. With eight KV heads, a group of four seemed entirely plausible on our rig.

We compared the MiniMax IQ5_K path with --max-gpu 4 and --max-gpu 2. Both variants booted cleanly and performed almost identically. In their respective A/B comparisons, short-context throughput rose from about 49.5 to around 50.3 tok/s. At approximately 80,000 active tokens, the variants reached about 13.07 and 13.08 tok/s respectively, compared with around 12.3 tok/s in the corresponding baseline run. Depending on the measurement point, the gain was therefore somewhere between roughly one and six percent.

The effect was real, but it was no structural breakthrough. The switch nudged a boundary; it did not turn low-double-digit decode into a 40-tok/s long-context path.

HY3 Separated the Quality Problem from the Speed Problem

To determine which symptoms were specific to MiniMax, we tested Tencent HY3, a 295B-A21B MoE with 80 layers and eight KV heads. Two quants ran into very different limits.

GPTQ Int4 on vLLM: Fast Enough, but Almost No Context Headroom

Our first attempt used AngelSlim/Hy3-GPTQ-Int4 on vLLM 0.25.1 with TP=8. With a short context, the model reached approximately 56.3 tok/s. Once it was loaded, however, each TP rank, and therefore each GPU, had only around 0.47 GiB of usable KV memory left. vLLM estimated a maximum context of roughly 12,400 tokens for this configuration.

That ruled out the path for our long agent sessions. Decode rate was not the issue; the session simply did not fit into the remaining KV cache. As expected, a nominal --max-model-len 262144 did nothing to increase the available memory.

UD128 on llama.cpp: A Good Run with Steep Context Costs

The GGUF YanissAmz/Hy3-295B-A21B-GGUF in its UD128 variant gave us more room. Our older llama.cpp image did not initially recognize the architecture and exited with unknown model architecture: 'hy_v3'. After switching to a newer ghcr.io/ggml-org/llama.cpp:server-cuda, the model launched with a configured context of 262,144 tokens, q8_0 KV, Flash Attention, and an even tensor split across all eight GPUs.

The Non-MTP baseline at the time was:

active prompt context prompt processing decode
short – approximately 44.8 tok/s
approximately 100,024 tokens approximately 643 tok/s approximately 12.3 tok/s
approximately 200,024 tokens approximately 216 tok/s approximately 4.6 tok/s

These figures apply to the run documented below, without Multi-Token Prediction enabled. They are neither a general verdict on HY3 nor a claim about every later runtime optimization.

In terms of output quality, HY3 was the more revealing counter-test. During a real run at around 90,000 tokens, the model remained coherent, wrote correct German, and identified missing points in the ongoing analysis. Its failure profile therefore differed markedly from the MiniMax runs: the output remained convincing while decode throughput collapsed as the context grew.

Split mode also made a measurable difference. Layer split reached around 40 tok/s with a short context, while tensor split achieved approximately 45.7 tok/s, an increase of about 14 percent. HY3 has eight KV heads, which suits distribution across eight GPUs. The tensor split made better use of the aggregate memory bandwidth, despite the reductions over PCIe. Neither f16 nor bf16 KV fit in this tensor path, even at substantially smaller test contexts; q8_0 was the configuration that worked for the 262k setup.

HY3 answered an important diagnostic question for us. A large full-attention model can remain qualitatively stable at long context and still become too slow to operate. MiniMax added loops, language leakage, and unstable judgments on top of those context costs.

Qwen3.5-122B Made the Architectural Costs Visible

Our control group was RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic, a dynamically quantized variant of the official Qwen3.5-122B-A10B. The model also ran on vLLM 0.25.1 and the same 8×3090 rig.

Two topologies showed the usual tradeoff between single-stream speed and KV headroom:

Topology short decode reported KV pool theoretical 262k sessions
TP=8 approximately 104.9 tok/s 474,777 tokens approximately 1.81
TP=2 + PP=4 approximately 72.6 tok/s 1,551,018 tokens approximately 5.9

The larger pool does not extend the context of an individual request beyond the model limit of 262,144 tokens. It allows several long sessions to run side by side. For an agent server, that is often more useful than the highest number from a single short benchmark.

Architecture was the decisive factor. Qwen uses 48 language-model layers, only every fourth of which performs full attention. The remaining layers use linear or recurrent attention. The model also has only two KV heads. MiniMax and HY3, by contrast, perform full attention in every layer, with eight KV heads each.

Model full-attention layers KV heads head dimension relative BF16 KV work per token
Qwen3.5-122B-A10B 12 out of 48 2 256 1x
MiniMax M2.7 62 out of 62 8 128 approximately 10.3x
HY3 80 out of 80 8 128 approximately 13.3x

The final column is an architecture-based comparison, not measured throughput. It does explain the direction of our observations. With MiniMax and HY3, every generated token has to traverse all full-attention layers over a growing history. Qwen carries much of that history in a more compact state and pays the full KV cost in only one quarter of its layers.

As a result, Qwen felt less sensitive in long sessions. It was not the strongest model for every task, nor was it free of verbosity. On our rig, however, it was much easier to plan around: high short-context throughput, substantial KV headroom in the PP-heavy topology, and a less dramatic loss of decode performance with a long active context.

Hardware Failures Wrote Their Own Model Reviews

Some of our supposed software problems originated outside the software. Different GPUs disappeared from the PCIe tree during different runs. After changes to power distribution, the failure moved to another card or initially stopped occurring. In other states, cards returned with only a Gen1 link. HY3 with eight-way tensor split could trigger a GPU failure, while Qwen3.6-35B-A3B on TP4+PP2 ran for hours. The two configurations stressed the system differently, so their different behavior was not reliable evidence of a model defect.

Our diagnosis led us to power distribution, transient load spikes, riser contacts, and degraded PCIe links. We documented the sequence of the hardware investigation in detail in “When a GPU Gets Run Over by the Bus”. For the model tests, the order of events mattered most:

  • GPU 1 at 41:00.0, corresponding to SLOT2, disappeared first.
  • After changes to power allocation, the affected devices included GPU 6 at c1:00.0, corresponding to SLOT3.
  • With a different load distribution, several cards on the same power branch failed.
  • After the hardware changes, some links temporarily appeared only as Gen1 x8, Gen1 x4, or Gen1 x16.

Above the driver layer, failures like these look like dead workers, RPC timeouts, CUDA problems, or a crashed engine. They can also distort measurements without immediately causing a hard failure. A PCIe link that has fallen back to Gen1 and thermally throttled GDDR6X memory are quite capable of manufacturing an apparent engine regression.

Since then, we have separated three checks before evaluating a model: Are all eight cards present? Do their links deliver plausible bandwidth under load? Do power, temperatures, clocks, and AER counters stay within expected ranges throughout the run? Only then does the inference software get its chance to be at fault.

What We Changed for Production

These experiments left us with a few sober rules for day-to-day operation.

We Measure at the Target Context

A short benchmark remains useful because it exposes broad differences in kernels and split modes. When selecting an agent model, we now add measurement points at approximately 60k, 80k, 100k and, where useful, 200k of active context. These cover decode rate, Time to First Token, prompt throughput, and a qualitative task spanning several tool steps.

We Separate Operational Completion from Evaluation Stability

Files, exit codes, and completed tool calls answer the operational question. Repeated identical runs answer the question of trust. For evaluations and rankings, one successful output is not enough. We compare winners, scoring scales, changes in rank, and rationales across multiple repetitions.

We Break Up Long Evaluation Tasks When the Model Drifts Near Its Limit

For MiniMax, splitting the eleven profiles into two or three groups is a practical countermeasure. This tends to keep the active context below the degradation zone we observed. It is no substitute for a more stable model path, but it prevents one session from having to carry every profile, intermediate analysis, form, and reasoning token at once.

We Preserve Working Launches as Complete Artifacts

A model name and quant are not enough. A reproducible run requires the image tag or digest, complete launch command, KV dtype, context size, split mode, tensor split, cache options, container name, port, and measurement log. Recovering the IQ5_K launch was the clearest reminder: our memory had reconstructed a plausible command. The machine insisted on the real one.

We Draw Conclusions About Models Only After a Hardware Stress Test

Roaming GPU failures, AER errors, Gen1 retraining, and thermal clock drops make any model comparison worthless. One short, healthy run is not enough when the eventual workload involves a long prefill, eight parallel GPU paths, and load swings of several kilowatts.

Reproducibility: The Launch Configurations That Matter

The following details belong directly with the measurements in this article. Where the documentation did not contain a complete Docker launch, we provide the exact server arguments we had saved rather than inventing a plausible wrapper.

MiniMax Hybrid AWQ on vLLM 0.25.1

Property Value
Container minimax_m27_awq_upgrade
Image vllm/vllm-openai:v0.25.1
Compose profile ayysasha_upgrade
Host/server port 8000
maximum model context 180000
KV cache int8_per_token_head
Split TP=8 plus Expert Parallelism

Saved server arguments:

/models/MiniMax-M2.7-AWQ 
  --served-model-name MiniMax-M2.7-AWQ-hybrid-int8kv mm minimax-m2.7-awq 
  --tensor-parallel-size 8 
  --enable-expert-parallel 
  --max-model-len 180000 
  --kv-cache-dtype int8_per_token_head 
  --attention-backend TRITON_ATTN 
  --disable-custom-all-reduce 
  --gpu-memory-utilization 0.95 
  --max-num-seqs 4 
  --max-num-batched-tokens 4096 
  --generation-config auto 
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":40,"repetition_penalty":1.1,"frequency_penalty":0.4}' 
  --enable-prefix-caching 
  --reasoning-parser minimax_m2 
  --tool-call-parser minimax_m2 
  --enable-auto-tool-choice 
  --trust-remote-code 
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":16}' 
  --host 0.0.0.0 
  --port 8000

Associated environment variables:

VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
NCCL_P2P_DISABLE=1
NCCL_IB_DISABLE=1
NCCL_CUMEM_ENABLE=0
VLLM_ALLREDUCE_USE_SYMM_MEM=0
VLLM_WORKER_MULTIPROC_METHOD=spawn
VLLM_ENABLE_CUDA_COMPATIBILITY=0

The JSON sampling recipe reproduced separately above also contains min_p: 0.01 and describes the parameters used for the five agent runs. The saved server-side override line did not include this value, so we list the two layers separately.

MiniMax IQ5_K on ik_llama

docker run -d --name minimax_iq5k --gpus all --ipc=host --shm-size=64g 
  -e NVIDIA_DISABLE_REQUIRE=1 
  -p 8088:8080 
  -v /bigData/llama-models/ubergarm/MiniMax-M2.7-GGUF:/models 
  --entrypoint /app/llama-server 
  ik-llama-cuda:main-20260612-full 
  --model /models/IQ5_K/MiniMax-M2.7-IQ5_K-00001-of-00005.gguf 
  --alias MiniMax-M2.7-IQ5_K 
  --host 0.0.0.0 --port 8080 
  -c 163840 
  -fa 1 -ngl 999 
  -ts 1,1.008,1,1.008,1.008,1.008,1,0.97 
  -ctk q4_0 -ctv q4_0 
  -muge -gr 
  -b 2048 -ub 256 
  --threads 1 --parallel 1 
  --no-mmap --jinja 
  --cache-ram 32768 --prompt-cache-all

The --max-gpu comparisons used the same launch with --max-gpu 4 on host port 8089 and --max-gpu 2 on 8090, respectively. All other parameters remained unchanged.

HY3 GPTQ Int4 on vLLM 0.25.1

docker run -d --name hy3_gptq_int4 --gpus all --shm-size 32g --ipc host 
  -e NVIDIA_VISIBLE_DEVICES=all 
  -e NVIDIA_DISABLE_REQUIRE=1 
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID 
  -e NCCL_P2P_DISABLE=1 
  -e NCCL_IB_DISABLE=1 
  -e NCCL_CUMEM_ENABLE=0 
  -e VLLM_ALLREDUCE_USE_SYMM_MEM=0 
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn 
  -e VLLM_ENABLE_CUDA_COMPATIBILITY=0 
  -e VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm 
  -v /bigData/hf-cache/AngelSlim-Hy3-GPTQ-Int4:/models/AngelSlim-Hy3-GPTQ-Int4 
  -v /bigData/vllm/data/torch-extensions-v024:/root/.cache/torch_extensions 
  -v /bigData/vllm/data/triton-v024:/root/.triton 
  -v /bigData/vllm/data/cuda-cache-v024:/root/.cache/cuda 
  -v /bigData/vllm/data/vllm-cache-v024:/root/.cache/vllm 
  -p 8005:8000 
  vllm/vllm-openai:v0.25.1 
  /models/AngelSlim-Hy3-GPTQ-Int4 
  --served-model-name hy3-gptq-int4 
  --tensor-parallel-size 8 
  --max-model-len 12000 
  --gpu-memory-utilization 0.92 
  --max-num-seqs 8 
  --max-num-batched-tokens 4096 
  --disable-custom-all-reduce 
  --enable-prefix-caching 
  --tool-call-parser hy_v3 
  --reasoning-parser hy_v3 
  --enable-auto-tool-choice 
  --trust-remote-code 
  --host 0.0.0.0 
  --port 8000

HY3 UD128 on llama.cpp, Non-MTP Baseline

docker run -d --name hy3_ud128_ctx --entrypoint /app/llama-server 
  --gpus all --ipc host --shm-size 64g 
  -p 8005:8005 
  -v /bigData/llama-models/YanissAmz-Hy3-UD128:/models 
  ghcr.io/ggml-org/llama.cpp:server-cuda 
  --model /models/Hy3-UD128-00001-of-00003.gguf 
  --host 0.0.0.0 
  --port 8005 
  --ctx-size 262144 
  --flash-attn on 
  --n-gpu-layers 999 
  --cache-type-k q8_0 
  --cache-type-v q8_0 
  --batch-size 2048 
  --ubatch-size 512 
  --split-mode tensor 
  --tensor-split 1,1,1,1,1,1,1,1 
  --threads 24 
  --parallel 1 
  --chat-template-file /models/hy3-chat-template.jinja

Qwen3.5-122B-A10B-FP8-dynamic on vLLM 0.25.1, TP=8

docker run -d --name qwen122b_fp8dyn_tp8 --gpus all --shm-size 32g --ipc host 
  -e NVIDIA_VISIBLE_DEVICES=all 
  -e NVIDIA_DISABLE_REQUIRE=1 
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID 
  -e NCCL_P2P_DISABLE=1 
  -e NCCL_IB_DISABLE=1 
  -e NCCL_CUMEM_ENABLE=0 
  -e VLLM_ALLREDUCE_USE_SYMM_MEM=0 
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn 
  -e VLLM_ENABLE_CUDA_COMPATIBILITY=0 
  -v /bigData/hf-cache/Qwen3.5-122B-A10B-FP8-dynamic:/models/Qwen3.5-122B-A10B-FP8-dynamic 
  -v /bigData/vllm/data/torch-extensions-v024:/root/.cache/torch_extensions 
  -v /bigData/vllm/data/triton-v024:/root/.triton 
  -v /bigData/vllm/data/cuda-cache-v024:/root/.cache/cuda 
  -v /bigData/vllm/data/vllm-cache-v024:/root/.cache/vllm 
  -p 8004:8000 
  vllm/vllm-openai:v0.25.1 
  /models/Qwen3.5-122B-A10B-FP8-dynamic 
  --served-model-name qwen3.5-122b-fp8dyn 
  --tensor-parallel-size 8 
  --max-model-len 262144 
  --gpu-memory-utilization 0.92 
  --max-num-seqs 8 
  --max-num-batched-tokens 4096 
  --disable-custom-all-reduce 
  --enable-prefix-caching 
  --reasoning-parser qwen3 
  --language-model-only 
  --enable-auto-tool-choice 
  --tool-call-parser qwen3_coder 
  --trust-remote-code 
  --host 0.0.0.0 
  --port 8000

The TP=2-plus-PP=4 variant used the same model weights and the same maximum context, but distributed the model with --tensor-parallel-size 2 --pipeline-parallel-size 4. It produced the 72.6 tok/s reported above and the substantially larger KV pool.

The Operational Limit Is the Model’s Real Size

After these experiments, we no longer consider a model production-ready just because its server can reserve a large context and answer a short prompt quickly. It is ready only if it continues to make usable decisions under the intended workload: after many tool calls, with a large active KV cache, on the real PCIe topology, and while the power delivery system sustains eight consumer GPUs for the entire run.

MiniMax M2.7 showed us that high short-context throughput and impressive individual answers can coexist with unstable behavior over long sessions. HY3 isolated the other half of the problem: a model can maintain its output quality and still be defeated by the decode cost of its full-attention architecture. Qwen revealed how much that equation changes with fewer full-attention layers and only two KV heads.

The result is not another favorite number in a benchmark table. We now choose the model according to the length and shape of the work. Short, demanding tasks can reward different strengths from a six-hour agent session. And once an evaluation has real consequences, the ranking from the fifth identical run matters at least as much as the token rate from the first short prompt.

Models and Engines

Write a comment

Your e-mail address will not be published. A first comment is approved manually.