← Insights

DE — Deutsche Version

#8 · Laguna S 2.1 INT4 on 8×3090: A Fast Specialist with Sharp Edges

· Filip

Most models we have run on our 8×3090 rig have fallen into one of two camps: HY3 was smart and painfully slow, while the early MiniMax quants were fast but not always technically up to the job. We explored how those differences play out under sustained agent workloads in our comparison of MiniMax, HY3, and Qwen.

Laguna S 2.1 from Poolside breaks that pattern. It is fast enough that we checked our measurements first. At the same time, it handles precisely the work that often defeats other local models: difficult bugs, code reviews, and long analyses of entire repositories.

That does not make Laguna a universal replacement. The word that keeps appearing in candid field reports is “spiky”: its performance varies unusually sharply by task. After testing it ourselves, we understand why.

This article is exclusively about Laguna. Every performance figure comes from our rig, not the model card.

What Laguna S 2.1 Is, Technically

Poolside released Laguna S 2.1 on July 21, 2026 and described its architecture, training, and internal evaluations in the release article. The model was only a few days old when we tested it, and its weights, configurations, and runtime support were changing just as quickly.

We ran the official poolside/Laguna-S-2.1-INT4 variant with the following specifications:

Feature Value
Architecture 117.6B MoE total, around 8.5B active
Layers 48
Experts 256 routed experts, Top-10 plus Shared Expert
Attention 12 Full-Attention plus 36 Sliding-Window layers with 512 tokens
KV heads 8, head dimension 128
Weights Compressed-Tensors INT4, group size 32, symmetric
Kernel path Marlin W4A16 on SM86
Local size around 67 GiB across 15 Safetensors shards
Engine vllm/vllm-openai:v0.25.1

For decode performance, the 117B figure matters less than the number of parameters active per token. Laguna activates roughly 8.5 billion. The principle resembles Qwen3.6-35B-A3B: a sparse Mixture-of-Experts model touches only part of its weights for each token. Laguna has substantially more total parameters, however, and the version we tested also compressed the routed MLP weights to four bits.

The local quantization_config primarily targeted the routed w1, w2, and w3 weights. Attention projections, the LM Head, Shared Expert, and the dense first MLP layer were on the ignore list. We therefore did not test a blanket four-bit quant, but a targeted compression of the memory-intensive areas. That kept decode costs unusually low despite the 117 billion total parameters.

An Important Note on Reproducibility

Our measurements apply to the 256K version we tested locally on July 26, 2026. Poolside replaced the INT4 repository in August with new weights and a new 1M configuration. The current commit history and model card explicitly identify that checkpoint as the successor to the earlier version. The following results, YaRN overrides, and failure modes must not be assumed to apply to today’s checkpoint without retesting.

Speed on Four GPUs

Here, TG refers to pure token generation during single-user decode after the first token. We used the same protocol throughout: one warm-up run, three measured runs, 256 output tokens, temperature=0, and streaming. Time to First Token is not included in the TG figures.

TP4 Was the Practical Sweet Spot

Topology Short-context TG TG at 100k active context KV capacity
TP4, FP8-KV, util 0.92 105.8 97.2 377,571 (1.44x at 262k)
TP4, BF16-KV, util 0.96 109.1 85.6 394,028 (1.50x at 262k)
TP2+PP3, BF16-KV, 6 GPUs 89.3 67.2 –

Even 105.8 tokens per second at short context is welcome performance from a 117B model on consumer Ampere hardware. The unusual result, though, came with 100,000 active tokens: decode dropped only from 105.8 to 97.2 tokens per second, or roughly 8 percent.

For comparison, HY3 fell from around 45 to roughly 12 tokens per second in our earlier run. MiniMax fell into long loops under a similar load. Laguna stayed fast because only 12 layers read the full history through Full Attention; the remaining 36 use a Sliding Window of 512 tokens.

FP8-KV Overtook BF16-KV Only at Long Context

At short context, BF16-KV was slightly faster at 109.1 versus 105.8 tokens per second. At 100,000 active tokens, the order reversed: FP8-KV reached 97.2, while BF16-KV managed only 85.6 tokens per second. BF16 has twice as much KV data to read. That barely matters while the cache is small, but memory bandwidth becomes the bottleneck at long context.

At this measurement point, FP8-KV was about 13.5 percent faster and also left more VRAM headroom. BF16-KV could serve the full 262k context only with --gpu-memory-utilization 0.96. That left about 0.85 GiB free on each GPU. With that little margin, a single badly timed moment during CUDA Graph capture is enough to kill the server with an OOM.

For the version under test, we therefore settled on TP4 with FP8-KV and 262k context. It delivered the best 100k result, retained more headroom, and matched Poolside’s quality configuration at the time.

How Far the Context Actually Reached

At the time of testing, Poolside called the weights “1M Checkpoints” but shipped them configured for 262,144 tokens by default. The model card then explicitly recommended this setting for the best output quality and warned that quality could deteriorate at 1M. When a vendor warns about its own maximum, we do not treat that maximum as a mode to run without a second thought.

TP4 Physically Fit More Than 262k

The configured context limit and the physically available KV memory were two different things. Across four GPUs, the server reported a KV capacity of 377,571 tokens with FP8 and 394,028 tokens with BF16. Anyone who needed more than 262k could, in practice, fit roughly 370,000 to 390,000 tokens on TP4 with the tested checkpoint.

Reaching 1M Required TP8 and a YaRN Override

For the 1M experiment, we left the local config.json untouched and overrode the values at server startup instead. --max-model-len 1048576 alone would not have been enough. The YaRN factor and attention factor had to match the 1M configuration, or the positional information would have been scaled incorrectly.

{
  "max_position_embeddings": 1048576,
  "rope_parameters": {
    "full_attention": {
      "rope_type": "yarn",
      "factor": 128.0,
      "original_max_position_embeddings": 8192,
      "attention_factor": 1.4852030263919618,
      "partial_rotary_factor": 0.5
    }
  }
}

With TP8 and FP8-KV, the server then reported a KV capacity of 3,604,506 tokens, equivalent to 3.44 concurrent 1M contexts. Our long-context sweep produced the following results:

Stage Short-context TG TG at filled context
524k around 108 62.1
786k around 104 42.9
1.048M around 112 34.6

Short-context decode remained stable at roughly 104 to 112 tokens per second, whether the server was configured for 262k or 1M. Merely reserving the larger window did not slow it down. The extra work appeared only as the KV cache filled: every new token made the Full-Attention layers read the growing history.

All three 1M outputs remained structurally coherent. We saw no collapse in positional processing, no attention failures, and no CUDA or OOM errors. With 1,048,006 prompt tokens, Laguna even recognized that we had repeated the same German test sentence many times and understood the final instruction.

That was still not enough for a recommendation. Coherent output does not prove reliable comprehension.

The Limits of Our 1M Validation

Our long prompt was deliberately synthetic and consisted mainly of one repeated sentence. It tested memory, positional processing, and a very simple semantic response. It did not establish difficult retrieval, multi-step reasoning, or repository work across 1M tokens.

For the tested version, “1M context” therefore needed to be broken down more carefully:

1M allocatable and bootable:                      demonstrated
1M prompt processed, coherent output possible:    demonstrated locally
1M simple verbatim retrieval:                     not properly documented in public
1M reliable for multi-hop logic and repo coding:  unknown
1M at the same quality as 262k:                   undocumented; the vendor warned against it

As of July 26, 2026, there was no publicly reproducible Laguna run using RULER at 500k or 1M, no multi-needle test, and no real repository evaluation across 1M tokens. Poolside’s published long-context table at the time covered the base model before post-training and ended at 128k. Even there, RULER declined gradually from 81.6 percent at 32k to 72.8 percent at 128k.

For this checkpoint, 262k remained the operating limit we could actually defend. Running between 500k and 1M was technically real, but semantic reliability was unproven. For genuinely large datasets, RAG, deliberate file selection, and context compression had a stronger evidence base.

The August checkpoint released since then is configured natively for 1M and, according to the current model card, calibrated for that configuration. The card still notes possible quality degradation at long context. Our old 1M experiments are no substitute for a fresh semantic evaluation of the current weights.

The PP3 Bug That Cost Us an Hour

For a six-GPU setup, we tested TP2 plus PP3, using Pipeline Parallelism across three stages. With FP8-KV, the server started without any obvious problem, reported 1,885,184 KV tokens, passed its health check, and produced useless output on the first real prompt.

The cause was buried deeply enough to look like a model failure at first. In the pipeline layers, k_scale and v_scale were not loaded because the tensor name contained a duplicated attn prefix. vLLM 0.25.1 silently fell back to 1.0 for the missing scales, leaving the FP8-KV cache mathematically mis-scaled.

The obvious --calculate-kv-scales switch did not help either. It was marked as deprecated in vLLM 0.25.1 and had no effect in this case.

TP2+PP3, FP8-KV, 88 tokens:      87.27 TG  -> invalid, garbled output
TP2+PP3, FP8-KV, 100k tokens:    78.22 TG  -> invalid, garbled output

The numbers look perfectly respectable on paper and are still worthless. Only BF16-KV was valid for PP3:

--tensor-parallel-size 2 --pipeline-parallel-size 3 \
--max-model-len 262144 --kv-cache-dtype bfloat16

Even then, PP3 was the worse choice for a single agent stream. TP4-BF16 was about 22 percent faster at short context and roughly 27 percent faster at 100k, despite using two fewer GPUs. PP3 gained KV headroom and paid for it in pipeline latency.

The failure reinforced a rule our rig keeps teaching us: a server that starts is not necessarily a server that works. A health check and a TG result are insufficient. Every startup needs a control prompt whose coherent answer we can inspect. TP8 behaved normally in this experiment; the scale-loading bug was specific to the pipeline path.

Our Daily-Use Launch Configuration

The following is the actual TP4, FP8-KV, 262k command we used for the checkpoint under test. It occupies four GPUs on devices 3, 4, 6, and 7, leaving the other four cards available for a second model.

docker rm -f laguna_s21_int4 2>/dev/null || true
docker run -d --name laguna_s21_int4 --entrypoint vllm \
  --gpus '"device=3,4,6,7"' --ipc host --shm-size 64g \
  -e NVIDIA_DISABLE_REQUIRE=1 -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e NCCL_P2P_DISABLE=1 -e NCCL_IB_DISABLE=1 -e NCCL_CUMEM_ENABLE=0 \
  -e VLLM_ALLREDUCE_USE_SYMM_MEM=0 -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e VLLM_ENABLE_CUDA_COMPATIBILITY=0 \
  -v /bigData/hf-cache/Laguna-S-2.1-INT4:/models/Laguna-S-2.1-INT4 \
  -v /bigData/vllm/data/torch-extensions-v024:/root/.cache/torch_extensions \
  -v /bigData/vllm/data/triton-v024:/root/.triton \
  -v /bigData/vllm/data/cuda-cache-v024:/root/.cache/cuda \
  -v /bigData/vllm/data/vllm-cache-v024:/root/.cache/vllm \
  -p 8005:8000 \
  vllm/vllm-openai:v0.25.1 \
  serve /models/Laguna-S-2.1-INT4 --served-model-name laguna-s-2.1-int4 \
  --tensor-parallel-size 4 --max-model-len 262144 \
  --kv-cache-dtype fp8 --gpu-memory-utilization 0.92 \
  --disable-custom-all-reduce --enable-prefix-caching --generation-config auto \
  --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 \
  --default-chat-template-kwargs '{"enable_thinking":true}' --trust-remote-code \
  --max-num-seqs 1 --max-num-batched-tokens 4096 \
  --host 0.0.0.0 --port 8000
until curl -fsS http://127.0.0.1:8005/health; do sleep 3; done

Several flags are essential for this setup:

  • --reasoning-parser poolside_v1 and --tool-call-parser poolside_v1: Laguna has its own format for reasoning and tool calls. A generic parser judges the model on the wrong basis. Native tool calls are one of its strengths when the correct parser is in place.
  • --default-chat-template-kwargs '{"enable_thinking":true}': According to Poolside, thinking improves performance substantially, but consumes a great many tokens. In the release article, mean completion lengths for the reported benchmarks range from roughly 129,000 to 249,000 tokens. We therefore have to monitor reasoning budgets and finish_reason.
  • --enable-prefix-caching: Agent and tool loops mostly extend the existing context at the end. Prefix Caching avoids redoing that work.

The cold start took several minutes because of Triton’s JIT and torch.compile. That is why the command waits for a successful health check instead of guessing a fixed sleep duration.

For our 1M launch at the time, we added --gpus all, --tensor-parallel-size 8, and the YaRN block documented above through --hf-overrides. Everything else remained unchanged. The recipe for the current August checkpoint needs to be checked against the current model card.

Technically Strong, Operationally Spiky

Inference was the easy part with Laguna. The model was fast and technically impressive. Public tests, however, showed a very uneven coding profile: its strengths clustered around particular kinds of work, and the results depended heavily on the configuration.

A public SparkRun on comparable hardware showed that unevenness especially clearly:

Test Laguna NVFP4 Qwen 3.6 27B FP8 Qwen 3.6 35B-A3B FP8
HumanEval pass@1 36.6% 70.7% 50.6%
MBPP pass@1 76.4% 40.2% 49.8%
BFCL-v4 Overall 21% 28% 27%
BFCL, parallel calls 4 to 8% 92 to 96% 88 to 96%

Laguna finished last on HumanEval in this run and comfortably first on MBPP. Two coding benchmarks that look similar at a glance thus produced almost opposite rankings. That was evidence neither of general superiority nor of general weakness, but of a distinctly jagged capability curve.

The more substantial field reports repeatedly highlighted several strengths:

  • Difficult algorithms and debugging: In one community test, Laguna solved an in-place transformation in Julia that had defeated Qwen 122B and 27B. The attempt used more than 60,000 thinking tokens, after which the tests passed.
  • Native tool calls: One user documented 209 to 270 clean tool calls against real business objects. He later revised his initially euphoric verdict to “decent,” which fits the uneven overall picture rather well.
  • Long repository analyses and reviews: Several reports identified these tasks as a strength.

The recurring weaknesses were just as clear:

  • Frontend, UI, and ordinary Python: Laguna finished well behind Qwen and Gemma in a blind comparison.
  • Overthinking and failure to finish: In one test, the NVFP4 variant ended 12 out of 12 runs without a final answer after spending its entire budget on reasoning. The INT4 quant entered a loop after around 20,000 tokens and repeatedly revisited the same folders.
  • Planning and timely completion: In an OpenCode coding test, the agent stopped several times and had to be prompted to continue.

Together, these reports suggest a consistent operating profile. Laguna can be exceptionally capable on difficult coding tasks with testable outcomes. Qwen is the more predictable everyday model, handles frontend work more efficiently, and works better with generic tool harnesses. With Laguna, external tests and hard stopping criteria belong in the configuration from the start.

Two Findings from Our DOCX Stress Test

Our tender benchmark asks a model to evaluate 11 full-stack profiles against 180 criteria, then use Word-MCP to generate form-like documents. It is a long, multi-stage, tool-heavy task, exactly the pattern that had reliably sent MiniMax into sprawling reasoning loops before.

In the TP4 and FP8-KV configuration, Laguna completed the DOCX benchmark three times in three runs. We saw no loops, garbling, or premature termination. For a model only a few days old running on consumer Ampere hardware, this was the most pleasant surprise of our test series, particularly against the zero out of three and two out of three successful runs achieved by other local models on the same task.

Laguna also thought for a long time before delivering. Its reasoning phases consumed a substantial number of tokens. That remained manageable in the clean 262k setup, but confirmed the externally observed tendency to overthink. Without a capped reasoning budget and monitoring of finish_reason, the persistence that solves a difficult bug can turn into a monologue on a simple task.

Why the Early Reports Contradicted One Another

Between July 21 and 24, the model and runtime environment changed continuously:

  • FP8 received a spinquantless checkpoint with the commit note “fixes agentic looping.”
  • INT4 and NVFP4 were updated to new 256k configurations.
  • The YaRN attention factor and chat template were corrected for GGUF.
  • The DFlash drafter was temporarily incompatible with the target model’s version.
  • In vLLM, implicit and explicit thinking behaved differently across multiple turns.

During that period, reports without the exact checkpoint commit, quant, runtime, chat template, parser, sampling settings, and KV dtype were barely comparable with any other run. Our PP3-FP8 failure is the problem in miniature: the server started and returned plausible performance figures, yet its output was unusable. Incorrect YaRN metadata, broken templates, or missing KV scales could easily make Laguna look as though the model itself were at fault.

The August replacement of the INT4 checkpoint makes this requirement even more important for older reports. A result from July remains useful when the tested version is clearly identified. Without that attribution, we may be comparing different weights under the same model name.

Conclusion: Laguna Is Our Scalpel, Not Our Swiss Army Knife

Laguna S 2.1 INT4 reached roughly 106 tokens per second at short context on four GPUs and still delivered 97 tokens per second with 100,000 active context tokens. With the July checkpoint we tested, 262k was the sensible operating point, around 390k physically fit on TP4, and even 1M produced structurally coherent output on TP8. Solid evidence for semantic reliability at 1M was still missing.

Laguna could produce brilliant results on demanding debugging, review, and algorithm work. It was far less dependable on routine tasks, frontend work, and finishing on time. Its reasoning was both its strength and an operational risk.

That led us to a clear division of labor:

Qwen remains our everyday model. Laguna is the scalpel for the difficult bug: TP4, 262k, FP8-KV, with every result backed by tests or a second reviewer.

Where tests can tell us what is true and a hard token budget decides when the work ends, Laguna is an unusually capable local specialist. As a general-purpose frontend agent, the same model is mostly an excellent way to wonder what everyone else is so excited about.

Write a comment

Your e-mail address will not be published. A first comment is approved manually.