#10 · The LLM Rig Built from eBay Parts: 2 x RTX 3060, 120 Euro for Board and CPU, 77 tok/s

August 13, 2026 · Filip

Our large local LLM rig has eight RTX 3090s, an EPYC, and a power supply that could probably handle some light metalworking on the side. For that build, throwing hardware at the problem was part of the brief. This time, we wanted to find the opposite limit: how cheap can a local LLM machine get before it becomes little more than a hobby project?

The answer was cheaper than we expected. A 59 Euro motherboard, a 25 Euro Xeon, and two used RTX 3060s with 12 GB each were enough to run Qwen3.6-35B-A3B with its 35 billion total parameters and its native context of 262,144 tokens at roughly 77 tok/s.

This is not a machine whose flaws we have to excuse because it was cheap. In our real, multi-stage DOCX workflow, it turned into a usable agent machine. Getting there, however, taught us more than the shopping list did. A supposedly over-aggressive 4-bit quant turned out to be a serving problem. A model producing 107 tok/s lost to a slower one at 91 tok/s. And a string of apparent model failures eventually came down to a misconfigured $PWD.

Old server hardware makes the foundation almost absurdly cheap

At the heart of the machine is an ASUS Z10PA-U8 from the LGA-2011-3 era. Its Intel C612 chipset, eight DDR4 slots, and workstation-grade PCIe layout date back to a time when boards like this sat in corporate machines. Today they turn up on the used market at prices that, with current hardware, would sometimes not even cover shipping.

Component	Configuration	eBay price
Motherboard	ASUS Z10PA-U8 (LGA 2011-3, Intel C612, 8 x DDR4)	approx. 59 Euro
CPU	Intel Xeon E5-2680 v4 (14C/28T)	approx. 25 Euro
RAM	32 GB DDR4 ECC	used, a few Euro
GPU0	RTX 3060 12 GB (GA104), PCIe Gen3 x16	used
GPU1	RTX 3060 12 GB (GA106), PCIe Gen3 x16	used
Operating system	Ubuntu 26.04	0 Euro

With 14 cores and 28 threads, the Xeon sounds more important than it is in this build. During decode, the MoE model resides entirely on the GPUs. The CPU has to keep the runtime fed without holding it back, but it does no work that would justify a modern high-performance socket. An E5-2680 v4 handled the job easily. An even cheaper CPU from the v3 generation would be a plausible option too.

The board mattered more. Both RTX 3060s run over proper PCIe Gen3 x16 links. We needed neither bifurcation tricks nor an x1 riser salvaged from a retired mining rig. A layer split generates little traffic between the cards anyway, since each GPU computes its own layers in sequence. Even so, full x16 links eliminate an entire category of possible faults before troubleshooting begins.

Despite sharing the same product name, the cards use different chips: one GA104 and one GA106. That is not unusual among used RTX 3060s, and it caused no trouble for the layer split. What mattered to the runtime was that each card had 12 GB of VRAM and working CUDA support.

Even 16 GB rather than 32 GB of system memory would be sufficient for this GPU-only path. We would not, however, buy just any board simply because it was the cheapest listing. Eight DDR4 slots, stable PCIe lanes, and the integrated ASPEED BMC are worth more in this machine than the last few Euro saved.

The board and CPU cost roughly 120 Euro together. The two used 3060s were the only major expense. At the time of our purchase, they were the cheapest straightforward route to 24 GB of total VRAM without resorting to specialist cards or dubious adapter contraptions.

Drivers, Docker, and one necessary reboot

The basic installation was uneventful. Two details later determined whether a failure looked like a quantization issue or a problem with the state of the system.

We deliberately installed the proprietary NVIDIA 580 driver and then rebooted the machine completely:

sudo apt-get install -y nvidia-driver-580   # → proprietary 580.173.02 (license NVIDIA, DKMS), CUDA UMD 13.0
sudo reboot                                  # NOT a live modprobe — a real reboot

At that point, ubuntu-drivers wanted to install 595-open instead. We stayed with the proprietary 580 driver for the documented setup. The reboot was not just a courtesy to Linux. Forcing a live driver change with modprobe left our system in an intermediate state where certain quantization paths produced unusable output. That failure disappeared after a clean reboot.

Next came Docker and the NVIDIA Container Toolkit:

sudo apt-get install -y docker.io
# nvidia-container-toolkit via the NVIDIA repo, then:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# GPU test:
docker run --rm --gpus all --entrypoint nvidia-smi ghcr.io/ggml-org/llama.cpp:server-cuda -L

Once nvidia-smi listed both 3060s inside the container, the hardware side was largely done. For the runtime, we used the official CUDA server image from llama.cpp, ghcr.io/ggml-org/llama.cpp:server-cuda. Our image was based on CUDA 12.8. That version detail became unexpectedly important later.

Why a 35B model fits on two RTX 3060s

We cover the model architecture and our more extensive experiments with Qwen3.6-35B-A3B in a separate article in this series. For the budget machine, the important part of the name is the suffix: A3B.

Qwen3.6-35B-A3B has roughly 35 billion parameters, but as a Mixture-of-Experts model it activates only around three billion of them per token. Decode therefore does not have to read every expert weight for each new token, which substantially reduces the demand for memory bandwidth. An RTX 3060 delivers about 360 GB/s, far below an RTX 3090. With only around three billion active parameters, though, the setup still runs fast enough to be a pleasure to use.

We used Unsloth's roughly 17.7 GB UD-IQ4_XS quant with the following complete launch command:

docker rm -f qwen36_a3b_q4 2>/dev/null
docker run -d --name qwen36_a3b_q4 \
  --entrypoint /app/llama-server \
  --gpus all --ipc host --shm-size 16g \
  -p 8080:8080 \
  -v /home/lyth/models:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  --model /models/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf \
  --host 0.0.0.0 --port 8080 \
  --ctx-size 262144 --parallel 1 \
  -fa on -ngl 999 -sm layer \
  -ctk q8_0 -ctv q8_0 \
  -b 2048 -ub 256 \
  --jinja --swa-full --threads 14

Each of the key parameters has a specific job:

-sm layer distributes complete layers across both GPUs. This is not tensor parallelism. Card 0 computes the lower portion of the model, then card 1 the upper portion. With only around three billion active parameters, this sequential arrangement remained fast enough.
-ctk q8_0 -ctv q8_0 stores the key and value caches at 8-bit precision. That allows the long context to fit in the remaining VRAM.
--ctx-size 262144 reserves the model's native context of 262,144 tokens.
-ub 256 limits the physical micro-batch. It costs some prefill performance but creates the necessary VRAM headroom.
-fa on enables Flash Attention, while --jinja uses the chat template embedded in the GGUF through Jinja.
--swa-full instructs llama.cpp to use a full-size cache for Sliding Window Attention. This option is separate from the general prompt cache. It became important for stable conversations in our multi-stage runs.

Where the small KV cache actually comes from

The model weights occupied roughly 8.85 GB per card. Even so, a q8 KV cache for 262,144 tokens still fit within the combined 24 GB of VRAM. In the same configuration, a dense 27B model hit the memory limit much sooner.

The reason becomes clearer when we separate two architectural features. The MoE design reduces the weights active per token and therefore primarily lowers the decode workload. The small KV cache, on the other hand, results from Qwen's hybrid attention architecture. Of its 40 layers, only ten use full Gated Attention; the rest use Gated DeltaNet. The full-attention layers also have only two KV heads. According to our runtime figures, the GQA KV cache in f16 required roughly 0.69 GiB per 32k tokens. With q8_0, the entire native context therefore fit into memory.

Both effects need to be considered separately when planning the hardware. MoE reduces the compute workload, while the attention architecture shrinks the cache. In Qwen3.6, the small set of active expert weights, the low number of full-attention layers, and the two KV heads all work in our favor: little computation per token, ample context, and manageable VRAM requirements.

77 tok/s in a single stream

With a short prompt, the rig delivered roughly 77 tok/s decode in a single stream. For a model with 35 billion total parameters running on two RTX 3060s, that is more than a pleasing benchmark figure. Output arrives faster than we can read it, which makes an interactive session feel immediate.

The result came surprisingly close to our FP8 path using the same base model on two RTX 3090s in the large rig. One measurement is not a general comparison between the 3060 and 3090. It does show how well this particular A3B architecture tolerates the lower memory bandwidth, and how much the quant, engine, and parallelization method shape the outcome.

When the 4-bit quant suddenly seemed stupid

Our first agentic runs were much less encouraging than the short decode test. We had deliberately chosen an unpleasant test task: read files, interpret instructions, edit tables and rankings, coordinate several tool calls, and finally produce a DOCX document.

With the 4-bit A3B model, we observed three types of failure:

repetition loops in which the model kept extending the same line of thought,
hallucinations in the middle of tool calls,
output that unraveled and ended without a clean conclusion.

The obvious suspicion was that UD-IQ4_XS had been quantized too aggressively. A larger quant would have consumed more VRAM, reduced the available long context, and required another large download. Before accepting those costs, we reviewed community discussions, GGUF notes, and issue trackers.

Two changes broke the deadlock in our setup:

We added --swa-full to the llama.cpp launch command. The official argument description specifically calls this a "full-size SWA cache." In our multi-stage sequence, this cache configuration prevented the loops and truncated responses we had seen before. For our server version, that was a clear diagnosis. A single setup still proves nothing about every Qwen quant.
We allowed OpenCode to access the actual working directory. Without that permission, the agent hit an automatic denial when writing to /tmp in headless operation. From the outside, this looked like a failed model step. It was actually permission management:

"external_directory": { "*": "ask", "/tmp/**": "allow" }

The difference was substantial. In the DOCX benchmark, clean runs increased from 0 out of 3 to 2 out of 3. The model and quant file were identical. Only the serving configuration and permissions had changed.

The public discussions about the Unsloth GGUFs also show that tool-calling problems can occur at higher precision, and that templates, sampling, and the runtime all belong in the diagnosis. Some users eliminated loops by switching to a different Qwen chat template. Those reports do not rule out the quant as a possible cause. They do, however, give us enough reason not to treat "more bits" as the first and only answer.

A second finding explained why we had recorded the container's CUDA version. Users reported unusable output with UD-IQ4_XS on CUDA 13.2, with CUDA 13.3 fixing the issue again. Our llama.cpp image with CUDA 12.8 avoided that specific path. On a fresh 13.2 stack, the same failure could once again have looked like a broken quant.

The practical lesson is narrower and more useful: when Qwen3.6 behaves unreliably in an agentic session, we first check the server version, SWA cache, chat template, sampling, driver state, CUDA version, and tool permissions. Only then do we increase quantization precision. In this case, that order saved VRAM, download time, and several misguided rebuilds.

Speculative decoding versus long context

The 77 tok/s figure came from a short prompt. Once an agent session exceeded 100,000 active tokens, decode throughput fell to roughly 35 tok/s. It remained usable, but the difference was easy to feel. With a long context, each new token requires reading more cached data, making the KV cache the dominant consumer of memory bandwidth.

Current versions of llama-server provide two methods well suited to this machine. Both are forms of speculative decoding: a fast path proposes several tokens, and the main model verifies them as a batch. Correct verification preserves the target model's output distribution; only the computation is accelerated.

Prompt lookup with n-grams costs virtually no VRAM

--spec-type ngram-simple requires no draft model. The runtime searches the existing context for n-grams that have already appeared and proposes the tokens that followed them. This works particularly well when the output repeats passages from the prompt verbatim.

Our DOCX workflow contains plenty of those: checkbox labels, text from table cells, and recurring template strings. We added these parameters:

--spec-type ngram-simple --spec-ngram-simple-size-n 8 \
--spec-ngram-simple-size-m 32 --spec-draft-n-max 32

On the real task, the acceptance rate ranged from 19 to 37 percent. A summary composed mainly of newly phrased text achieved only about five percent, which is consistent with the mechanism. A complete clean DOCX run took 1,097 seconds with n-grams, compared with roughly 1,900 seconds without speculative decoding. That works out to roughly 1.7x the end-to-end speed, with no additional model and virtually no extra VRAM.

MTP delivers the next jump

Qwen3.6-35B-A3B was trained with Multi-Token Prediction. Its configuration contains mtp_num_hidden_layers: 1. The regular UD-IQ4_XS GGUF omits the relevant tensors, so for this path we used the separate unsloth/Qwen3.6-35B-A3B-MTP-GGUF build. Its IQ4_XS file is approximately 450 MB larger and includes the additional next-token head. llama-server enables it with --spec-type draft-mtp.

Unlike prompt lookup, MTP can propose tokens at every step, including within newly generated reasoning passages. The two methods can also be combined:

--spec-type draft-mtp,ngram-simple

The additional weights and small draft KV cache consumed just enough VRAM to upset the automatic layer split. At a 262k context with q8 KV, GPU1 had only 357 MiB free, and prefill ended in OOM. A manual split fixed it:

-sm layer -ts 52,48

GPU0 then used 11,417 MiB and GPU1 used 11,541 MiB, leaving roughly 380 MiB of headroom on each card. The full 262k context and MTP now fit in memory at the same time.

One initially plausible idea made no difference: connecting the monitor to the motherboard's VGA output to free framebuffer memory on GPU0. The board's ASPEED BMC was already driving the console. lspci | grep VGA showed 09:00.0 ASPEED Graphics Family, and /app/llama-server was the only GPU process on either 3060. The unequal usage came entirely from the automatic split, so only -ts could change it.

The resulting decode rates were:

Configuration	short decode	decode at 100k
Baseline without speculative decoding	76	35
n-gram only	77	39
MTP + n-gram	91	63

At 100,000 active tokens, throughput rose to 1.8x the baseline. Short interactive prompts also benefited, rising from 77 to 91 tok/s. Across the complete DOCX run, however, the additional gain over n-grams alone was much smaller at roughly ten percent. Faster decode does nothing for prefill, Python work, or pauses between tool calls. That is why the end-to-end figure matters more to us than an isolated peak.

Gemma was faster and still did a worse job

Gemma 4 26B-A4B was the obvious challenger. The model has roughly 25.2 billion total parameters and activates around 3.8 billion. Architecturally, that makes it a much better match for two RTX 3060s than the dense 31B Gemma 4 model.

The Unsloth GGUFs loaded without any special workarounds:

File	Size
`gemma-4-26B-A4B-it-UD-IQ4_XS.gguf`	12.66 GiB
`gemma-4-26B-A4B-it-UD-Q4_K_S.gguf`	15.36 GiB
`gemma-4-26B-A4B-it-UD-Q4_K_M.gguf`	15.78 GiB

Our initial Q4_K_M result was:

Configuration	short decode	decode at 100k
Gemma 26B-A4B Q4_K_M	69	33

That already put Gemma close to Qwen A3B. An external MTP sidecar is also available, mtp-gemma-4-26B-A4B-it-Q8_0.gguf, at only about 0.43 GiB. The key was selecting the correct speculative decoding path: draft-mtp, not draft-simple.

--spec-type draft-mtp \
--spec-draft-model /models/gemma26a4b/mtp/MTP/mtp-gemma-4-26B-A4B-it-Q8_0.gguf \
--spec-draft-n-max 2 \
--spec-draft-device CUDA1

At a 131k context, this setup achieved:

Configuration	short decode	decode at 100k
Gemma 26B-A4B Q4_K_M + MTP	107	74

After further tuning, Gemma even supported a 240k context on the machine. In raw throughput, it was now ahead of Qwen A3B. But we still lacked the measurement that mattered for model selection: the actual work.

Gemma's agentic performance collapsed in the same multi-stage DOCX benchmark:

Run	Score	Runtime
1	35	2,155 s
2	0	1,876 s

Neither run passed completely. The first omitted only the combined document; the second also damaged the ranking format. Both outcomes were worse than the Qwen path, despite Gemma generating more tokens per second.

That produced the following ranking on this machine:

Model	Configuration	short decode	decode at 100k	Agent task result
Qwen3.6-35B-A3B	MTP + n-gram, 262k	91	63	clean runs, best path for the task
Gemma 4 26B-A4B	MTP, 240k	107	74	faster decode, worse on the real task

Gemma remains a very fast candidate. Two runs are far too few for any general judgment of model quality. On this machine and in this structured tool task, though, the Qwen path won. Gemma still needs to prove its utility in more real workloads; a high TG figure cannot substitute for that test.

The benchmark itself faked a model failure

During a later campaign, our harness reported zero successes across nine runs. We spent hours investigating the model's behavior because the symptoms resembled the truncated responses we had seen earlier.

The fault was in the Python wrapper. subprocess.run(cwd=X) sets the process working directory but does not automatically update the $PWD environment variable. OpenCode read $PWD internally. Every run therefore wrote its DOCX to the same /home/lyth/benchmark/result/ directory and overwrote the previous file, while the scorer inspected the empty campaign directories and duly reported one failure after another.

The fix was small:

subprocess.run(["opencode","run","--dir",wd, ...],
               cwd=wd, env={**os.environ, "PWD": wd})

The very next run achieved a PASS with 100 out of 100 points. Every diagnosis from that campaign that blamed the model for unraveling output rested on measurement artifacts. Since then, we have evaluated the generated files and their actual destination paths in agent runs, rather than relying only on exit codes and log output. When launching processes from Python, cwd and $PWD may need to be set together.

Two obvious routes that did not work on this rig

Negative results are especially valuable in a budget build. They reveal which supposedly cheap alternatives consume only time, memory, or both.

The dense Qwen3.6-27B was four times slower

A 27B model initially sounds like a better fit for 24 GB of VRAM than a 35B model. We therefore tested a roughly 17 GB Q4_K_M GGUF from Bartowski, again with 8-bit KV and a layer split.

The result was approximately 18 tok/s.

A dense model has to read all of its weights for every token. The layer split compounds the bottleneck because the two 3060s compute their halves of the model one after the other: card 0 first, then card 1. For comparison, a single V100 produced roughly 25 tok/s with the same dense model. In this configuration, the architecture constrained throughput more than the model's nominal total size.

On the same hardware, the dense 27B path was therefore more than four times slower than 35B-A3B. It also lost on context length: q8 KV reached only around 128k. With q4 KV and further adjustments, we made it to roughly 192k before hitting OOM. We deleted the model again. It was smaller, slower, and offered less context, a remarkably comprehensive defeat.

EXL3 remained less stable than the optimized llama.cpp path

We also tested an EXL3 variant at 4.00 bpw and roughly 19.4 GB using ExLlamaV3 and TabbyAPI. In our runs, this path was markedly less stable than llama.cpp with --swa-full. For our test, no directly comparable option for a full-size SWA cache was available there.

EXL3 provides an interesting quantization method and a runtime designed for consumer GPUs. Even so, llama.cpp was the route that gave us a reliably repeatable agent task. Again, this is a result for these versions and this workload, not a general verdict on ExLlamaV3.

Power draw, Euro, and the limits of the comparison

Under load, each RTX 3060 drew approximately 170 watts, falling to around 26 watts at idle. With both GPUs, the Xeon, the board, and the remaining components, the complete budget machine drew roughly 400 watts under load.

That runs off an ordinary wall outlet without special planning. The build needs neither a 1,600-watt server PSU nor a dedicated circuit. Its load transitions are also much less dramatic than those of our 3090 rig, whose hardware failures already filled an article of their own.

For Qwen3.6-35B-A3B, our documented paths produced the following rough comparison:

Machine	Quant	Speed	Cost class
2 x RTX 3060, budget rig	UD-IQ4_XS on llama.cpp	approx. 77 tok/s	approx. 120 Euro for board and CPU plus two used 3060s
3090 path on the large rig	FP8 on vLLM	approx. 171 tok/s	complete machine costing several thousand Euro

The faster path was roughly 2.2x as fast. The budget rig cost and consumed only a fraction of what the large rig demands, and it still delivered more than a third of its throughput. This is deliberately not a universal price-performance table: used prices fluctuate, the quants differ, and the large rig can handle workloads that fundamentally cannot fit into 24 GB of VRAM.

For maximum speed, large models, and genuine multi-user load, the 3090 rig remains in a different class. As a single fast local agent, the small machine is considerably more economical. Its advantage comes from matching the architecture to the workload. An RTX 3060 remains an RTX 3060.

The cheap machine worked only after the expensive diagnosis

The hardware story can be reduced to 24 GB of VRAM from two used RTX 3060s, a 25 Euro Xeon, and a 59 Euro motherboard. The decisive part starts after that. With roughly three billion active parameters, Qwen3.6-35B-A3B makes this modest platform fast enough. Its hybrid attention architecture keeps the KV cache small enough to fit even the native 262k context.

The machine became useful for real work only after we examined each layer separately. --swa-full corrected the SWA cache for our long conversation path. An OpenCode permission eliminated an apparent model failure. The CUDA 12.8 image avoided a CUDA 13.2 issue with IQ4_XS. A manual tensor split freed the final hundred MiB for MTP, and the correct $PWD variable ultimately cleared a model that our own scorer had falsely accused.

The result is more than a curiosity for the workbench. At 77 tok/s in the baseline configuration, 91 tok/s with MTP and n-grams, and 63 tok/s with 100k active context, the machine handles real local agent work. The hardware was the cheap part. The diagnosis cost what it costs on any other inference server: careful measurement and the willingness not to blame the quant first.

Models, quants, and engines

Update, 27 August 2026: What Became of the 3060 Rig

The article above stands as published. We have kept measuring the same hardware since, and part of our original diagnosis does not survive the newer data. The correction belongs down here rather than upstairs in the text, because how a plausible explanation gives way to a measured one is part of the story.

The trigger was the closing `</think>`

The original damage – seven mangled string literals, one dropped character, a 61 KB monologue that went nowhere until we killed it – allowed for several readings. One came down to loop length: a model losing the thread across many agentic turns. Controlled runs have since taken that explanation apart.

We sent identical payloads to three stacks: our local llama.cpp at Q4, plus DeepInfra and Parasail serving the same model at FP8. The defect reproduced everywhere, and on FP8 it was more pronounced. There the generation ended with finish_reason=stop and no content at all, the reasoning breaking off mid-sentence exactly where the literal would have appeared. That rules out quantization, llama.cpp and our local configuration.

The size series then misled us for a while. Real production code at four lengths looked like a threshold: 400 bytes came through clean, anything from 900 bytes up fell over. But the 400-byte window held only the opening tag; the closing one first appears at 900. Size and tag content were confounded. Strip the structural literal and the same code passed at every length, 2.6 KB of it reproduced character for character in twelve runs out of twelve.

The deciding test held size constant and varied only the tags: four windows of roughly 420 bytes from the same file, with the tag-free control at 580 bytes the largest of the four excerpts.

Variant	local Q4	Parasail FP8
no tag (580 B)	3/3 exact	3/3 exact
opening only	3/3 exact	0/3
closing only	0/3	0/3
both	0/3	0/3

Locally, the closing tag shreds the output into 38 to 49 divergent blocks. Because the failure reproduces on three independent stacks, it belongs to the model. <|im_start|> and <|im_end|> behave the same way: both are single tokens, and unprotected they came through intact in none of nine runs.

What the shield fixes and what stays open

A plugin now encodes these literals before they reach the model and decodes them afterwards. Measured end to end against the live endpoint, the closing-tag case went from 0/5 to 5/5, and the both-tags case did the same. It is not free: in the opening-tag case, which had been clean before, the shield cost two defects across ten runs, a skipped line of code. Against that it removes some two hundred in the cases that had been unusable. It covers Qwen3.6-35B-A3B only, because that is the one model we have measurements for.

Verbatim pass-through turned out to be the easier half. In real agentic work the shielded subagent still achieved nothing: two dispatches with zero edits, and a third that produced 142 KB of monologue and froze the OpenCode interface. The channel was the prompt itself. With the file encoded identically and only the instruction text differing, a raw marker in the prompt gave one correct run out of three plus two doom loops, while the encoded marker gave three out of three with a deterministic 6,215-byte output. Once we closed that channel, the very dispatch that had produced two empty diffs and a frozen interface produced exactly the edit we had asked for.

Two points remain open, and we would rather name them than round them off. We still have not measured short loops against long ones. What is established is a structural-tag defect that strikes immediately and needs no accumulation of turns; about length itself our data say nothing. And one symptom of the original damage is still unexplained: end▁of▁sentence came back as end▁of▁entence. Those are seven ordinary tokens with no structural token among them, the plugin never touches them, and our own measurements predict the opposite for text of that kind. At least it fails loudly – code with a character missing from a literal will not compile.

Methodologically, the control run was what mattered: same model, same class of task, file without markers, edit performed correctly. Without that counterpart, every one of these failures looks like a model that simply cannot do the job.

Qwen runtime on the same two cards

Alongside the defect hunt we pushed the runtime, with the server context fixed at 262,144 tokens throughout. A llama.cpp fork with a profiled MoE expert cache got the larger Q6_K quant onto 2 x 12 GB; it ran, and it stayed slow. Where we ended up is Q4_K_XL with an MTP draft and the same expert cache.

Configuration at 262k context	short	formal 100k target
Q6_K, expert cache	33.98 tok/s	26.60 tok/s
Q4_K_XL, MTP + expert cache	57.68 tok/s	44.27 tok/s
IQ4_XS, earlier setup	82.53 tok/s	55.43 tok/s

Both Q4_K_XL targets passed the standard benchmark without warnings, at a standard deviation of 0.43 on the short run, and the container kept running healthy afterwards, no OOM and no restarts. The IQ4_XS figures come from a different setup and are good for rough orientation only. We read the table as a trade: the higher weight precision costs roughly 30 percent on short and 20 percent at 100k. Whether that pays off is settled by days of real use – a synthetic quality benchmark was not part of the experiment.

We examined two alternatives to MTP deliberately. DFlash had up to about 97 percent of its drafts accepted depending on position and reached mean accepted lengths of up to 4.88; the extra drafting work and the CPU offload needed to make it fit ate the gain again. The best variant fit for continuous operation stopped at 51.73 tok/s on short. DSpark only fitted at 262k with the smaller Q4_K_M sidecar and reached 44.26. Neither matched the MTP reference, so neither earned a 100k final run or a promotion. That verdict covers this fit on 2 x 12 GB at 262k and nothing beyond it.

Nemotron-Cascade-2-30B-A3B at 400K context

The second substantial result on the rig involves a different model. NVIDIA’s Nemotron-Cascade-2-30B-A3B – 31.6 billion parameters in total, 3 billion active per token, a hybrid of Mamba-2 and transformer MoE – runs as an IQ4_XS GGUF with a 409,600-token context entirely on the two cards, with nothing offloaded to the CPU. That is the ceiling: at 458,752 tensors move to the CPU. Occupancy sits at about 10.8 and 10.7 GB of VRAM.

Context	TG median	TTFT median	Prefill
short	91.55 tok/s	0.44 s	0.71 s
100K	47.23 tok/s	0.86 s	34.6 s
200K	29.25 tok/s	1.17 s	102.2 s
350K	18.91 tok/s	1.71 s	272.0 s

Spread across all four targets stayed between 0.04 and 0.34, so the curve is reproducible rather than a lucky sample. At 350K the machine is still usable and no longer interactive: just under 19 tokens per second serves slow work, and four and a half minutes of prefill come before the answer.

None of the usual levers made it faster. Flash attention landed slightly below baseline, n-gram speculative decoding clearly below at 100K, a q4 KV cache likewise, and a batch size of 2048 pushed short throughput down to 55.16. --split-mode tensor is not implemented for the nemotron_h_moe architecture, and --split-mode row fails because the CUDA path in use reports no split-buffer support for the 3060. In this measured setup the plain configuration won: llama.cpp’s auto-fit across both cards and a q8_0 KV cache, sampled at temperature 1.0 and top_p 0.95 on NVIDIA’s recommendation.

The reason is undramatic. At 100K context every generated token reads roughly 450 MB of KV cache, and an RTX 3060 offers 360 GB/s of memory bandwidth. Where bandwidth sets the ceiling, no kernel trick raises it. On quality we say nothing here: the competition scores on the model card are NVIDIA’s claims, not our measurements.

The revised conclusion

The article’s premise holds. Twenty-four gigabytes of VRAM from used parts still buy a great deal; a machine that carries a 31.6-billion-parameter model at 400K context and profiles several Qwen configurations besides remains a compelling result for us.

The more valuable return of these weeks is methodological. We had folded four separate things into a single diagnosis: the hardware, a defect in the model, the behavior of the runtime, and the fitness of our own measurement harness. All four turned out to be separable, and each gave a different answer. The model breaks on a structural token, regardless of stack or quantization. The runtime yields between 34 and 83 tokens per second at short context, depending on quant and drafting method. The hardware caps long context through memory bandwidth. And our harness invented a size threshold that was never there.

What the rig became was not the plan. It was meant to be an appliance for one model. It has turned into a test bench: the place where we put quants, drafting methods and context lengths against each other, and the cheapest way we have of killing a hypothesis before it hardens into a recommendation.