After our extended experiments with MiniMax M2.7, Tencent HY3 was an obvious next candidate. The 295B-A21B MoE had a strong reputation for coding and reasoning, a promising mixed GGUF quant was available locally, and our eight GPUs had enough combined VRAM to make the idea seem at least somewhat sensible.
What followed was unusually instructive for a local inference experiment. HY3 was neither a straightforward success nor a spectacular failure. It performed remarkably well on our tasks, yet became slow enough at large active contexts to put its practical value in doubt. We later discovered that the GGUF already contained an additional MTP layer that our original runtime simply did not use. Once the runtime caught up, the practical limit moved considerably, though the cost of the architecture remained.
Our previous article on sustained agent workloads compares MiniMax, HY3, and Qwen more broadly. This article covers the full HY3 diagnostic trail: two quants, several split modes, KV cache limits, Multi-Token Prediction, and a benchmark state that stubbornly refused to collapse into a single number.
The setup: 192 GB of VRAM, but no shared memory
Our local LLM rig contains seven RTX 3090 cards and one RTX 3090 Ti. Together they provide roughly 192 GB of VRAM on a consumer PCIe topology. There is no NVSwitch turning those eight cards into one tightly coupled accelerator.
At this model size, every choice becomes expensive:
- the inference engine,
- the quantization format,
- the KV cache data type,
- the split mode,
- and, occasionally, the power distribution inside the rig.
HY3 has 295 billion parameters, with 21 billion active per token. It uses 80 regular layers, eight KV heads, and a native 256K context window. With a suitable quantization, the model fits across our cards. That settles the capacity question. Whether eight GPUs can still generate tokens fast enough at 100,000 or 200,000 active tokens is an entirely different calculation.
The fast vLLM path left almost no room for context
Our first attempt was designed to keep HY3 in the fast stack:
AngelSlim/Hy3-GPTQ-Int4,- vLLM 0.25.1,
- Tensor Parallelism across all eight GPUs,
max_model_len=262144,gpu_memory_utilization=0.92.
At short context, this configuration reached about 56 tok/s. That was respectable for a model in this class. The decode kernel did not worry us. The remaining KV memory did.
Once the model was loaded, vLLM reported only about 0.47 GiB of usable KV memory per TP rank, or per GPU. In this exact configuration, that limited a single request to an effective context of roughly 12,400 tokens. Our agent workloads routinely grow beyond 100,000 active tokens, so this path was effectively closed. Configuring max_model_len=262144 does not create memory that no longer exists after the weights have been loaded.
The GPTQ path therefore gave us a useful short-context TG data point and little else. As a permanent local working model for our workload, HY3 was not viable this way.
UD128 and llama.cpp launched HY3 at 262K
The more interesting experiment used YanissAmz/Hy3-295B-A21B-GGUF, specifically the mixed UD128 variant, with llama.cpp.
At first, nothing launched. Our older container image stopped with this message:
unknown model architecture: 'hy_v3'
The build simply lacked support for the architecture. HY3 loaded correctly after we switched to a newer llama.cpp image. This belongs near the top of the diagnostic checklist for new open-weight models: an apparently broken quant may just reflect the gap between the model and the toolchain.
Our working baseline configuration used:
split-mode tensor,flash-attn on,q8_0for the key and value caches,ctx-size 262144.
HY3 started cleanly with those settings. The contrast with vLLM and GPTQ was substantial: on the same rig, the GGUF reserved the full 262K window and served requests. HY3 was therefore capable of running on eight consumer GPUs in principle. For the first time, the experiment looked like a potential local working model rather than a successful loading exercise.
The first decode rate looked promising
At short context, UD128 reached about 44.8 tok/s with the faster tensor split. This historical baseline used standard autoregressive decoding. The additional MTP block already present in the GGUF file was not being executed at the time.
For a quant of this size on consumer GPUs, that was fast enough to justify more measurements. The test that mattered, however, started with a large active history.
At 100K and 200K, the full-attention bill arrived
Our agent sessions combine long prompts, tool output, and a history that keeps growing. We therefore tested HY3 with prompt sizes much closer to the real workload.
At approximately 100,024 prompt tokens, without MTP, we measured:
- Prompt processing: approximately 643 tok/s,
- Decode: approximately 12.3 tok/s.
At approximately 200,024 prompt tokens, throughput dropped further:
- Prompt processing: approximately 216 tok/s,
- Decode: approximately 4.6 tok/s.
The short-context rate of almost 45 tok/s first became 12.3 and then 4.6 tok/s. At 200K, the model remained usable in the most literal sense, but it was hardly pleasant as an interactive long-running agent. Every new token had to carry the large active history through 80 full-attention layers.
This was where HY3 differed from our more troublesome MiniMax runs. HY3 became slow without visibly losing its mind.
Model quality was the strongest part of the experiment
On a real analysis task with a large context, HY3 remained coherent, wrote correct German, stayed on task, and even found gaps in the ongoing analysis. It survived the context formally and continued to work with it substantively.
At serious context lengths, UD128 appeared qualitatively stronger than any quantized MiniMax variant we had run locally up to that point. Under sustained load, MiniMax repeatedly added loops, language leakage, and inconsistent judgments to its already high decode cost. In the run we observed, HY3 showed good instincts and poor token economics.
That distinction matters when diagnosing the system. Speeding up an incoherent model only makes it incoherent faster. HY3 gave us something worth rescuing: the technical output was convincing, while execution cost limited its practical use.
Layer split and tensor split used the rig differently
We also compared split modes. Their names sound like implementation details, but on a PCIe system they represent two very different ways of using the hardware.
With layer split, each GPU owns complete transformer layers, including the corresponding portions of the KV cache. A decode step proceeds largely in sequence through these layer groups: one GPU processes its layers, then passes the hidden state to the next. Communication between cards remains relatively limited, but much of the aggregate memory bandwidth sits idle while only one subset of the model is active.
With tensor split, weights, attention or KV heads, and the work inside each layer are distributed across all eight GPUs. Every card can contribute to the same layer. The price is collective reductions over consumer PCIe; the benefit is access to more aggregate memory bandwidth during decode.
HY3 has eight KV heads, exactly one per GPU under an eight-way tensor split. Real kernels and placement rules are more complicated than this sketch, but the geometry divides neatly.
For our UD128 setup at short context, we measured:
- Layer split: approximately 40 tok/s,
- Tensor split: approximately 45.7 tok/s.
Tensor split won this TG test by roughly 14 percent. With MiniMax, the wrong split mode had sometimes halved performance or caused other problems. In our HY3 build, the benefit of parallel bandwidth outweighed the additional PCIe reductions.
This result applies to this quant, this llama.cpp revision, these flags, and our topology. Tensor mode was experimental in the build we used, so the 14 percent result does not establish a general ranking of split modes.
The intriguing TP4/PP-like hybrid remained theoretical
The comparison suggests a middle ground. Pure layer split leaves bandwidth unused. An eight-way tensor split keeps every GPU busy in every layer, but also pays for eight-way reductions over PCIe in every layer. A potentially attractive layout would let four GPUs work on a layer in parallel while distributing different layers or graph regions across all eight cards.
The graph splitter in ik_llama.cpp promises something close to this. The upstream llama.cpp build that worked for us offered none, layer, row, and the experimental tensor split, but no graph mode. ik_llama.cpp additionally provides -sm graph, --max-gpu N, -grt, and -ts.
On a host with eight GPUs, --max-gpu 4 does not mean that only four cards are used. All eight may still participate in the model globally, but any individual layer can span no more than four GPUs. Consecutive layers or subgraphs can reside on different groups of four. In simplified form, the idea looks like this:
Layer/subgraph A -> GPUs 0,1,2,3 (four-way parallelism within the layer)
Layer/subgraph B -> GPUs 4,5,6,7 (four-way parallelism within the layer)
later regions -> other groups of four, chosen by the graph placer
This resembles TP4 within a layer combined with PP-like distribution across eight GPUs. On consumer PCIe, such a compromise could offer more parallel bandwidth than layer split while keeping reduction groups smaller than TP8.
Equating it with vLLM-style TP4+PP2 would still be wrong. There are no two explicit pipeline stages, no conventional microbatch schedule, and no corresponding model of overlap and pipeline bubbles. -grt controls the data type used for graph reductions; -ts affects distribution and placement. Both are graph and communication options, not alternative names for conventional Pipeline Parallelism.
We wanted to test precisely this configuration, but our runtime was the wrong vintage. The installed ik_llama build failed in the same place:
unknown model architecture: 'hy_v3'
We therefore have no valid HY3 comparison between graph split and tensor split, and no defensible HY3 figure for --max-gpu. A HY3-specific community fork existed, but we did not validate it locally. Row mode likewise produced no clean result that we would publish as a measurement.
As a basic plausibility check of the mechanism, MiniMax runs with --max-gpu 2 and 4 gained approximately one to six percent. That keeps the idea alive but cannot be transferred to a different architecture. Eight KV heads make the HY3 hybrid theoretically elegant. Without a compatible, validated run, it remains theory.
Higher-precision KV cache would not fit
The q8_0 KV cache raised an obvious question: could f16 or bf16 simplify the compute path and improve decode throughput?
Our experiments initially answered only the memory question:
f16KV did not start,bf16KV did not start,- both failed at 262K,
- and both also failed in this tensor path at the substantially smaller context sizes we tested.
Of the higher-precision KV variants we tested, q8_0 was therefore the only option that allowed the intended high-context setup. A future test with Q4 target KV presents a separate speed-versus-quality tradeoff. What we could not do was fix the slowdown by moving to a higher-precision KV data type.
A different engine does not automatically flatten the context curve
We also examined the obvious escape route: was the slowdown merely a llama.cpp issue that Transformers, KTransformers, SGLang, or TensorRT-LLM had already solved?
For this particular class of full-attention model, we found no solid evidence that any of these engines simply eliminates the single-stream collapse at 100K to 200K. In a short MiniMax benchmark, SGLang was temporarily about 26 percent faster than vLLM, but it ran more slowly and less reliably on a real high-context agent task. The microbenchmark gain did not survive the target workload.
Prefix Caching, Radix Caching, PagedAttention, and Chunked Prefill are real optimizations. They save prefill work or improve memory use. They do not reduce the active KV state that full attention processes on every new decode step.
The rising cost with context is therefore not just the result of an awkward runtime. Specialized split-K or graph attention could improve the constants, and the graph hybrid remained an interesting direction. The lever we eventually tested on HY3 was a different one: speculative decoding with the native MTP layer.
The extra layer was already in the file
At this point, the story seemed complete. HY3 was smart, fit into memory with a 262K window, delivered decent throughput at short context, and became painfully slow with a large active history.
The UD128 GGUF contained 81 blocks in total:
block_count = 81
80 target layers
1 NextN / MTP layer
The additional quantized block occupied roughly another 2 GB in the file. In other words, we were carrying roughly 2 extra GB for a layer that was simply asleep. We already had the quant, the model download, and the GPUs. What we lacked was a runtime capable of executing it.
Support for the hy_v3 architecture, including --spec-type draft-mtp, landed in upstream llama.cpp in July 2026 (PR #25395). With a current build, the performance story changed significantly.
What Multi-Token Prediction does here
MTP stands for Multi-Token Prediction. In our setup, the additional NextN layer proposes upcoming tokens, and the full 295B target model verifies those proposals.
- A valid proposal is accepted.
- An incorrect proposal is rejected.
- The full target model retains final authority.
The draft layer performs advance work for the full target model, which then verifies the result. The draft acceptance rate determines the potential speedup. It cannot slip unverified tokens into the output, so correctly implemented speculative decoding should not systematically reduce quality.
We used Q8 KV on both sides:
- Q8 for the target KV cache,
- Q8 for the draft KV cache.
Quantizing the draft cache may affect how useful its proposals are. The target model still verifies the output.
More speculative tokens were much slower at first
Our first instinct was also wrong. If predicting one token helps, three should surely help more. The initial attempt used:
n_max = 3
p_min = 0.75
At short context, this configuration managed only about 18 tok/s, less than half the speed of normal decoding. In a later controlled sequence of three measurements, the same parameter set averaged only 15.6 tok/s.
A speculative depth of one helped in another exploratory run. As long as the confidence threshold remained at 0.75, however, throughput reached only about 26 to 27 tok/s. We did not repeat this specific point under the later formal sweep protocol, so it should be treated as directional evidence only.
The useful setting was the least dramatic combination:
n_max = 1
p_min = 0
Each speculative step produced no more than one draft token. With no confidence threshold suppressing the attempt, the target model could reject unhelpful proposals itself. Our current production path uses upstream llama.cpp, an even eight-way tensor split, ctx-size 262144, Q8 target and draft KV, n_max=1, p_min=0, and ubatch=256.
At short context, three measurements after a separate warmup produced:
| Run | Decode rate |
|---|---|
| 1 | 54.62 tok/s |
| 2 | 54.51 tok/s |
| 3 | 53.86 tok/s |
| Mean | 54.33 tok/s |
Weighted draft acceptance was approximately 63 percent. On the same hardware, the same target quant therefore rose from roughly 45 to about 54 tok/s.
The parameter sweep punished aggressive settings
A partial sweep confirmed the finding. Each table entry is the mean of three measured runs after a separate warmup; TG denotes token-generation throughput.
n_max |
p_min |
short TG | short acceptance | 100K TG | 100K acceptance |
|---|---|---|---|---|---|
| 1 | 0 | 54.33 | 63.2 % | 23.48 | 74.6 % |
| 2 | 0 | 45.60 | 43.2 % | 14.68 | 47.6 % |
| 3 | 0 | 47.92 | 42.1 % | 13.87 | 36.2 % |
| 2 | 0.75 | 15.75 | 87.7 % | 13.17 | 93.5 % |
| 3 | 0.75 | 15.60 | 85.6 % | 13.69 | 92.3 % |
| 1 | 0.25 | 35.35 | 62.1 % | 13.57 | 72.4 % |
n_max limits the number of draft candidates generated recursively through HY3's single NextN layer. On our rig, deeper drafts were consistently slower. Additional draft work, synchronization, and the declining usefulness of later candidates are plausible explanations. The sweep establishes the result more reliably than it establishes the exact mechanism.
The high acceptance rates at p_min=0.75 did not help either. The work required to achieve them cost more than it saved. We could not choose this optimization by selecting the most aggressive or theoretically tidy setting; we had to measure it.
At 100K, controlled throughput nearly doubled
Our historical non-MTP measurement at approximately 100K active context was 12.3 tok/s. With n_max=1, p_min=0, Q8 target and draft KV, and the server still configured for 262K, the formal three-run sweep produced:
| Run | Decode rate |
|---|---|
| 1 | 24.68 tok/s |
| 2 | 23.26 tok/s |
| 3 | 22.51 tok/s |
| Mean | 23.48 tok/s |
Compared with the earlier 12.3 tok/s, that is close to a doubling. Both results used the same current llama.cpp image, the same model, Q8 KV, tensor split, and the same 100K prompt format. The MTP run did, however, use the safer ubatch=256 and a stricter warmup protocol.
We therefore still need a direct --spec-type none control on the exact same build and under the exact same protocol. The observed path is substantially faster, but attributing every percentage point to MTP requires that control measurement.
For eight consumer GPUs and 100K active tokens, the reproducible result still changes the practical assessment. Roughly 23.5 tok/s is more than a minor benchmark correction. It turned a painfully slow model into a plausible long-context worker.
A fresh run reached 31.5 tok/s
A later fresh run landed between the controlled mean and our strangest warm state. It was not operating solely on an already cached prefix: the server processed the full prompt and generated immediately afterward.
- Prompt: 100,024 tokens,
- Prompt processing: 548.87 tok/s,
- Decode: 31.50 tok/s,
- Output: 256 tokens,
- Draft acceptance: 103 / 151 = 68.21 percent,
- Runtime: 190.53 seconds.
The 31.5 tok/s result is a real observation, and the full prefill makes it more interesting than a purely cached warm measurement. It still does not replace the controlled mean of 23.48 tok/s. Warm state and protocol details moved this system substantially; for planning purposes, the repeatable series matters more than the fastest individual run.
The 37 tok/s state remained a mystery
In a separate, heavily warmed-up sequence with nominally identical MTP settings and a cached prefix of approximately 100,023 tokens, we measured:
| Run | Decode rate | accepted drafts |
|---|---|---|
| 1 | 37.81 tok/s | 107 / 148 |
| 2 | 37.06 tok/s | 107 / 147 |
| 3 | 36.15 tok/s | 107 / 148 |
| Mean | 37.01 tok/s | 321 / 443 |
That was roughly three times the original 100K rate without MTP. It was genuine decode throughput against an active cached prefix, not a confusion with prompt processing. Draft acceptance was around 72 percent.
The later controlled sweep did not reproduce this state. Despite similar nominal settings and approximately 75 percent weighted acceptance, it settled at around 23.5 tok/s. Fresh runs made the picture messier still: the first run after a full prefill at roughly 510 tok/s decoded at 24.95 tok/s, while a later full prefill led to 31.50 tok/s. Those runs produced real states between the controlled result and the heavily warmed extreme. We do not yet know which of them occurs reliably in normal operation.
The suspects include:
- the number of preceding decode warmups,
- reuse of CUDA Graphs,
- KV and prompt cache state,
- allocator or cache layout,
- GPU clocks and general hardware state.
A benchmark result is not a fixed property printed on the model's packaging. It emerges from the model, runtime, graph state, cache state, topology, and measurement protocol.
Our defensible conclusion is therefore narrower: the new MTP path reproducibly moved the measured HY3 configuration at 100K from the earlier range around 12 tok/s to approximately 23.5 tok/s. One fresh run reached 31.5 tok/s, and one conspicuous warm sequence reached 37 tok/s. We still need to explain and reproduce the higher states, and the non-speculative control on the same build remains outstanding.
A server that starts is still not a reliable server
Our first MTP server used a microbatch size of 512. It launched, answered short prompts, and appeared healthy. The real 100K prompt then caused a CUDA out-of-memory error on GPU0.
Changing
ubatch = 512 -> 256
stabilized the 262K-configured instance under the 100K workload. We deliberately did not reduce the maximum context to 131K merely to create friendlier benchmark conditions. The setup still had to expose the full 262K window.
HY3 thus repeated one of the basic rules of our project: a successful launch and a few short answers do not prove that a configuration will survive its intended workload.
MTP reduces the cost; it does not abolish it
The architecture explains why the faster path still hits a context wall:
| Model | Full-attention layers | KV heads | Head dimension | relative BF16 KV work |
|---|---|---|---|---|
| Qwen3.5-122B | 12 of 48, remainder linear/recurrent | 2 | 256 | 1x |
| MiniMax M2.7 | 62 of 62 | 8 | 128 | approximately 10.3x |
| HY3 | 80 of 80 | 8 | 128 | approximately 13.3x |
HY3 performs full attention in all 80 target layers. Every generated token makes those layers operate across the active history. Based on layer count, KV heads, and head dimension, the architecture produces a comparison figure of approximately 13.3 times the KV work per token relative to Qwen3.5-122B. Even compared with MiniMax, HY3 is higher by a factor of about 1.3.
The eight KV heads primarily mean more local KV data and attention work per token. PCIe traffic does not automatically increase by a factor of eight. Tensor-parallel and graph collectives add to the constants; the linear increase with context comes from the attention pattern itself.
Qwen3.5-122B uses full attention in only one quarter of its 48 layers and has two KV heads. The rest of its hybrid stack carries history in compact recurrent state. Qwen therefore pays a fundamentally different long-context bill.
Even a single unified-memory GPU would slow down with increasing context on the same full-attention model. Our eight consumer GPUs, PCIe synchronization, and topology amplify an effect already built into the architecture.
Then the hardware impersonated a model failure
Different GPUs disappeared during different runs. On one occasion it was GPU1 at 41:00.0, or SLOT2; on another it was GPU6 at C1:00.0, or SLOT3. After changes to the power distribution, the fault moved.
That argued against one defective card or one bad slot. Instead, it pointed to a broader interaction involving:
- PSU distribution,
- power rails,
- transient load spikes,
- PCIe link retraining under load.
We cover the diagnostic timeline in detail in “When the GPU Gets Run Over by the Bus”. For HY3, the consequence was simple: model comparisons become worthless when cards drop out under load or their links fall back into degraded states. Above the driver layer, power and link problems quickly resemble dead workers, CUDA errors, or an unstable inference engine.
An eight-way tensor split in particular stresses the system differently from a smaller TP or PP layout. Before blaming software for an unusual decode rate, we therefore need all eight cards to be present, their links to behave plausibly under load, and the power supply, temperatures, clocks, and AER counters to remain unremarkable.
What HY3 means in practice on consumer hardware
The right assessment depends heavily on the intended workload.
Short context
UD128 is far from absurdly slow on our rig. Standard decoding reached around 45 tok/s, and the tuned MTP path after warmup reached approximately 54 tok/s. vLLM with GPTQ was similarly fast at about 56 tok/s, but our configuration provided only roughly 12,400 tokens of usable context.
Large active contexts
A model that launches with a 262K context window will not necessarily run comfortably throughout it. Without MTP, HY3 dropped into the low double digits at approximately 100K. With n_max=1 and p_min=0, the controlled result was approximately 23.5 tok/s.
We have documented the separate 37 tok/s sequence, but it should not support a purchase or deployment decision. The 4.6 tok/s at 200K also came from the older non-MTP path. We have not yet completed a comparable MTP sweep at 200K, and extrapolating the 100K gain directly would not be defensible.
Intelligence per token
HY3 deserves respect. It remained coherent on our large analysis task, wrote good German, and found genuine gaps. This was not a huge local model that merely occupied memory. Its qualitative strength was precisely what made the optimization worthwhile.
A general-purpose daily model
For long, mixed agent workloads, our answer still leaned toward Qwen rather than HY3. Qwen's hybrid attention architecture handles large active contexts more economically. MTP made the comparison more interesting, but it did not erase the architectural difference.
HY3 presents an unusually clear tradeoff: better qualitative performance than we expected from the local quant, substantially better long-context economics with MTP, and still 80 full-attention layers charging a real price for every active token.
Our verdict changed, but not completely
HY3 UD128 taught us something different from MiniMax. Under sustained load, MiniMax repeatedly made us question the model's judgment. HY3 slowed down while retaining our technical respect. In the runs we observed, its main bottleneck was operational.
The native MTP path moved the boundary that matters: short-context decode rose from the mid-40s to the mid-50s tok/s, while the controlled 100K result moved from the earlier low double digits to around 23.5 tok/s. HY3 remains architecturally distinct from Qwen, and its 80 full-attention layers still process the active history for every token. These results do not establish that all 262,144 tokens can be used comfortably. A direct --spec-type none control on the same build also remains outstanding.
Had vLLM with GPTQ remained our only path, we would have dismissed HY3 for long-running work fairly quickly. UD128 showed that the model could load at 262K and perform extremely well on the actual task. MTP support finally got that sharp mind in its slow body moving considerably faster.
Across at least part of the large-context range, an interesting technical demonstration became a plausible serious worker. The decisive performance lever had been sitting inside the model file all along as an extra layer, waiting for the runtime to understand it.
Nerd appendix: the exact MTP setup
The working runtime was:
llama.cpp version: 10103 (c588c4f47)
full commit: c588c4f47683e73ad2d69f50480bec6cc85fd0f7
image: ghcr.io/ggml-org/llama.cpp:server-cuda
image digest: sha256:72ca3a0323ec891af9332afc8d501396557a45eca2b1b3712828d5cf57e36743
The exact launch configuration:
docker run -d --name hy3_ud128_mtp \
--entrypoint /app/llama-server \
--gpus all --ipc host --shm-size 64g \
-p 8005:8080 \
-v /bigData/llama-models/YanissAmz-Hy3-UD128:/models \
ghcr.io/ggml-org/llama.cpp:server-cuda \
--model /models/Hy3-UD128-00001-of-00003.gguf \
--host 0.0.0.0 --port 8080 \
--ctx-size 262144 \
--flash-attn on --n-gpu-layers 999 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--batch-size 2048 --ubatch-size 256 \
--split-mode tensor --tensor-split 1,1,1,1,1,1,1,1 \
--threads 24 --parallel 1 \
--chat-template-file /models/hy3-chat-template.jinja \
--spec-type draft-mtp \
--spec-draft-n-max 1 --spec-draft-p-min 0 \
--spec-draft-type-k q8_0 --spec-draft-type-v q8_0
The measurement protocol is part of the result:
- The server remained configured for 262,144 tokens throughout.
- Target and draft KV both used Q8 throughout.
- Each measurement group began with one unscored warmup.
- Three measured runs followed.
- Each run generated 256 tokens.
- Temperature was set to zero for reproducible output.
- The long-context measurement used approximately 100,024 prompt tokens.
- For cached-prefix runs, we verified
cache_n=100023.
The remaining work is deliberately narrow:
- a
--spec-type nonecontrol using the same protocol and this exact build, - an explanation for the 23, 31.5, and 37 tok/s states,
- the missing data point for
p_min=0.5, - a test with
ubatch=384without changing the 262K configuration, - the same MTP measurement sequence at 200K active context.
Until those items are complete, we plan around 23.5 tok/s at 100K. The measured 37 tok/s is a research target, not a promise.