#10 · The LLM Rig Built from eBay Parts: 2 x RTX 3060, 120 Euro for Board and CPU, 77 tok/s
Our large local LLM rig has eight RTX 3090s, an EPYC, and a power supply that could probably handle some light metalworking on the side. For that build, throwing hardware at the problem was part of the brief. This time, we wanted to find the opposite limit: how cheap can a local LLM machine get before it becomes little more than a hobby project?
The answer was cheaper than we expected. A 59 Euro motherboard, a 25 Euro Xeon, and two used RTX 3060s with 12 GB each were enough to run Qwen3.6-35B-A3B with its 35 billion total parameters and its native context of 262,144 tokens at roughly 77 tok/s.
This is not a machine whose flaws we have to excuse because it was cheap. In our real, multi-stage DOCX workflow, it turned into a usable agent machine. Getting there, however, taught us more than the shopping list did. A supposedly over-aggressive 4-bit quant turned out to be a serving problem. A model producing 107 tok/s lost to a slower one at 91 tok/s. And a string of apparent model failures eventually came down to a misconfigured $PWD.
Old server hardware makes the foundation almost absurdly cheap
At the heart of the machine is an ASUS Z10PA-U8 from the LGA-2011-3 era. Its Intel C612 chipset, eight DDR4 slots, and workstation-grade PCIe layout date back to a time when boards like this sat in corporate machines. Today they turn up on the used market at prices that, with current hardware, would sometimes not even cover shipping.
| Component | Configuration | eBay price |
|---|---|---|
| Motherboard | ASUS Z10PA-U8 (LGA 2011-3, Intel C612, 8 x DDR4) | approx. 59 Euro |
| CPU | Intel Xeon E5-2680 v4 (14C/28T) | approx. 25 Euro |
| RAM | 32 GB DDR4 ECC | used, a few Euro |
| GPU0 | RTX 3060 12 GB (GA104), PCIe Gen3 x16 | used |
| GPU1 | RTX 3060 12 GB (GA106), PCIe Gen3 x16 | used |
| Operating system | Ubuntu 26.04 | 0 Euro |
With 14 cores and 28 threads, the Xeon sounds more important than it is in this build. During decode, the MoE model resides entirely on the GPUs. The CPU has to keep the runtime fed without holding it back, but it does no work that would justify a modern high-performance socket. An E5-2680 v4 handled the job easily. An even cheaper CPU from the v3 generation would be a plausible option too.
The board mattered more. Both RTX 3060s run over proper PCIe Gen3 x16 links. We needed neither bifurcation tricks nor an x1 riser salvaged from a retired mining rig. A layer split generates little traffic between the cards anyway, since each GPU computes its own layers in sequence. Even so, full x16 links eliminate an entire category of possible faults before troubleshooting begins.
Despite sharing the same product name, the cards use different chips: one GA104 and one GA106. That is not unusual among used RTX 3060s, and it caused no trouble for the layer split. What mattered to the runtime was that each card had 12 GB of VRAM and working CUDA support.
Even 16 GB rather than 32 GB of system memory would be sufficient for this GPU-only path. We would not, however, buy just any board simply because it was the cheapest listing. Eight DDR4 slots, stable PCIe lanes, and the integrated ASPEED BMC are worth more in this machine than the last few Euro saved.
The board and CPU cost roughly 120 Euro together. The two used 3060s were the only major expense. At the time of our purchase, they were the cheapest straightforward route to 24 GB of total VRAM without resorting to specialist cards or dubious adapter contraptions.
Drivers, Docker, and one necessary reboot
The basic installation was uneventful. Two details later determined whether a failure looked like a quantization issue or a problem with the state of the system.
We deliberately installed the proprietary NVIDIA 580 driver and then rebooted the machine completely:
sudo apt-get install -y nvidia-driver-580 # → proprietary 580.173.02 (license NVIDIA, DKMS), CUDA UMD 13.0
sudo reboot # NOT a live modprobe — a real reboot
At that point, ubuntu-drivers wanted to install 595-open instead. We stayed with the proprietary 580 driver for the documented setup. The reboot was not just a courtesy to Linux. Forcing a live driver change with modprobe left our system in an intermediate state where certain quantization paths produced unusable output. That failure disappeared after a clean reboot.
Next came Docker and the NVIDIA Container Toolkit:
sudo apt-get install -y docker.io
# nvidia-container-toolkit via the NVIDIA repo, then:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# GPU test:
docker run --rm --gpus all --entrypoint nvidia-smi ghcr.io/ggml-org/llama.cpp:server-cuda -L
Once nvidia-smi listed both 3060s inside the container, the hardware side was largely done. For the runtime, we used the official CUDA server image from llama.cpp, ghcr.io/ggml-org/llama.cpp:server-cuda. Our image was based on CUDA 12.8. That version detail became unexpectedly important later.
Why a 35B model fits on two RTX 3060s
We cover the model architecture and our more extensive experiments with Qwen3.6-35B-A3B in a separate article in this series. For the budget machine, the important part of the name is the suffix: A3B.
Qwen3.6-35B-A3B has roughly 35 billion parameters, but as a Mixture-of-Experts model it activates only around three billion of them per token. Decode therefore does not have to read every expert weight for each new token, which substantially reduces the demand for memory bandwidth. An RTX 3060 delivers about 360 GB/s, far below an RTX 3090. With only around three billion active parameters, though, the setup still runs fast enough to be a pleasure to use.
We used Unsloth's roughly 17.7 GB UD-IQ4_XS quant with the following complete launch command:
docker rm -f qwen36_a3b_q4 2>/dev/null
docker run -d --name qwen36_a3b_q4 \
--entrypoint /app/llama-server \
--gpus all --ipc host --shm-size 16g \
-p 8080:8080 \
-v /home/lyth/models:/models \
ghcr.io/ggml-org/llama.cpp:server-cuda \
--model /models/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf \
--host 0.0.0.0 --port 8080 \
--ctx-size 262144 --parallel 1 \
-fa on -ngl 999 -sm layer \
-ctk q8_0 -ctv q8_0 \
-b 2048 -ub 256 \
--jinja --swa-full --threads 14
Each of the key parameters has a specific job:
-sm layerdistributes complete layers across both GPUs. This is not tensor parallelism. Card 0 computes the lower portion of the model, then card 1 the upper portion. With only around three billion active parameters, this sequential arrangement remained fast enough.-ctk q8_0 -ctv q8_0stores the key and value caches at 8-bit precision. That allows the long context to fit in the remaining VRAM.--ctx-size 262144reserves the model's native context of 262,144 tokens.-ub 256limits the physical micro-batch. It costs some prefill performance but creates the necessary VRAM headroom.-fa onenables Flash Attention, while--jinjauses the chat template embedded in the GGUF through Jinja.--swa-fullinstructsllama.cppto use a full-size cache for Sliding Window Attention. This option is separate from the general prompt cache. It became important for stable conversations in our multi-stage runs.
Where the small KV cache actually comes from
The model weights occupied roughly 8.85 GB per card. Even so, a q8 KV cache for 262,144 tokens still fit within the combined 24 GB of VRAM. In the same configuration, a dense 27B model hit the memory limit much sooner.
The reason becomes clearer when we separate two architectural features. The MoE design reduces the weights active per token and therefore primarily lowers the decode workload. The small KV cache, on the other hand, results from Qwen's hybrid attention architecture. Of its 40 layers, only ten use full Gated Attention; the rest use Gated DeltaNet. The full-attention layers also have only two KV heads. According to our runtime figures, the GQA KV cache in f16 required roughly 0.69 GiB per 32k tokens. With q8_0, the entire native context therefore fit into memory.
Both effects need to be considered separately when planning the hardware. MoE reduces the compute workload, while the attention architecture shrinks the cache. In Qwen3.6, the small set of active expert weights, the low number of full-attention layers, and the two KV heads all work in our favor: little computation per token, ample context, and manageable VRAM requirements.
77 tok/s in a single stream
With a short prompt, the rig delivered roughly 77 tok/s decode in a single stream. For a model with 35 billion total parameters running on two RTX 3060s, that is more than a pleasing benchmark figure. Output arrives faster than we can read it, which makes an interactive session feel immediate.
The result came surprisingly close to our FP8 path using the same base model on two RTX 3090s in the large rig. One measurement is not a general comparison between the 3060 and 3090. It does show how well this particular A3B architecture tolerates the lower memory bandwidth, and how much the quant, engine, and parallelization method shape the outcome.
When the 4-bit quant suddenly seemed stupid
Our first agentic runs were much less encouraging than the short decode test. We had deliberately chosen an unpleasant test task: read files, interpret instructions, edit tables and rankings, coordinate several tool calls, and finally produce a DOCX document.
With the 4-bit A3B model, we observed three types of failure:
- repetition loops in which the model kept extending the same line of thought,
- hallucinations in the middle of tool calls,
- output that unraveled and ended without a clean conclusion.
The obvious suspicion was that UD-IQ4_XS had been quantized too aggressively. A larger quant would have consumed more VRAM, reduced the available long context, and required another large download. Before accepting those costs, we reviewed community discussions, GGUF notes, and issue trackers.
Two changes broke the deadlock in our setup:
- We added
--swa-fullto thellama.cpplaunch command. The official argument description specifically calls this a "full-size SWA cache." In our multi-stage sequence, this cache configuration prevented the loops and truncated responses we had seen before. For our server version, that was a clear diagnosis. A single setup still proves nothing about every Qwen quant. - We allowed OpenCode to access the actual working directory. Without that permission, the agent hit an automatic denial when writing to
/tmpin headless operation. From the outside, this looked like a failed model step. It was actually permission management:
"external_directory": { "*": "ask", "/tmp/**": "allow" }
The difference was substantial. In the DOCX benchmark, clean runs increased from 0 out of 3 to 2 out of 3. The model and quant file were identical. Only the serving configuration and permissions had changed.
The public discussions about the Unsloth GGUFs also show that tool-calling problems can occur at higher precision, and that templates, sampling, and the runtime all belong in the diagnosis. Some users eliminated loops by switching to a different Qwen chat template. Those reports do not rule out the quant as a possible cause. They do, however, give us enough reason not to treat "more bits" as the first and only answer.
A second finding explained why we had recorded the container's CUDA version. Users reported unusable output with UD-IQ4_XS on CUDA 13.2, with CUDA 13.3 fixing the issue again. Our llama.cpp image with CUDA 12.8 avoided that specific path. On a fresh 13.2 stack, the same failure could once again have looked like a broken quant.
The practical lesson is narrower and more useful: when Qwen3.6 behaves unreliably in an agentic session, we first check the server version, SWA cache, chat template, sampling, driver state, CUDA version, and tool permissions. Only then do we increase quantization precision. In this case, that order saved VRAM, download time, and several misguided rebuilds.
Speculative decoding versus long context
The 77 tok/s figure came from a short prompt. Once an agent session exceeded 100,000 active tokens, decode throughput fell to roughly 35 tok/s. It remained usable, but the difference was easy to feel. With a long context, each new token requires reading more cached data, making the KV cache the dominant consumer of memory bandwidth.
Current versions of llama-server provide two methods well suited to this machine. Both are forms of speculative decoding: a fast path proposes several tokens, and the main model verifies them as a batch. Correct verification preserves the target model's output distribution; only the computation is accelerated.
Prompt lookup with n-grams costs virtually no VRAM
--spec-type ngram-simple requires no draft model. The runtime searches the existing context for n-grams that have already appeared and proposes the tokens that followed them. This works particularly well when the output repeats passages from the prompt verbatim.
Our DOCX workflow contains plenty of those: checkbox labels, text from table cells, and recurring template strings. We added these parameters:
--spec-type ngram-simple --spec-ngram-simple-size-n 8 \
--spec-ngram-simple-size-m 32 --spec-draft-n-max 32
On the real task, the acceptance rate ranged from 19 to 37 percent. A summary composed mainly of newly phrased text achieved only about five percent, which is consistent with the mechanism. A complete clean DOCX run took 1,097 seconds with n-grams, compared with roughly 1,900 seconds without speculative decoding. That works out to roughly 1.7x the end-to-end speed, with no additional model and virtually no extra VRAM.
MTP delivers the next jump
Qwen3.6-35B-A3B was trained with Multi-Token Prediction. Its configuration contains mtp_num_hidden_layers: 1. The regular UD-IQ4_XS GGUF omits the relevant tensors, so for this path we used the separate unsloth/Qwen3.6-35B-A3B-MTP-GGUF build. Its IQ4_XS file is approximately 450 MB larger and includes the additional next-token head. llama-server enables it with --spec-type draft-mtp.
Unlike prompt lookup, MTP can propose tokens at every step, including within newly generated reasoning passages. The two methods can also be combined:
--spec-type draft-mtp,ngram-simple
The additional weights and small draft KV cache consumed just enough VRAM to upset the automatic layer split. At a 262k context with q8 KV, GPU1 had only 357 MiB free, and prefill ended in OOM. A manual split fixed it:
-sm layer -ts 52,48
GPU0 then used 11,417 MiB and GPU1 used 11,541 MiB, leaving roughly 380 MiB of headroom on each card. The full 262k context and MTP now fit in memory at the same time.
One initially plausible idea made no difference: connecting the monitor to the motherboard's VGA output to free framebuffer memory on GPU0. The board's ASPEED BMC was already driving the console. lspci | grep VGA showed 09:00.0 ASPEED Graphics Family, and /app/llama-server was the only GPU process on either 3060. The unequal usage came entirely from the automatic split, so only -ts could change it.
The resulting decode rates were:
| Configuration | short decode | decode at 100k |
|---|---|---|
| Baseline without speculative decoding | 76 | 35 |
| n-gram only | 77 | 39 |
| MTP + n-gram | 91 | 63 |
At 100,000 active tokens, throughput rose to 1.8x the baseline. Short interactive prompts also benefited, rising from 77 to 91 tok/s. Across the complete DOCX run, however, the additional gain over n-grams alone was much smaller at roughly ten percent. Faster decode does nothing for prefill, Python work, or pauses between tool calls. That is why the end-to-end figure matters more to us than an isolated peak.
Gemma was faster and still did a worse job
Gemma 4 26B-A4B was the obvious challenger. The model has roughly 25.2 billion total parameters and activates around 3.8 billion. Architecturally, that makes it a much better match for two RTX 3060s than the dense 31B Gemma 4 model.
The Unsloth GGUFs loaded without any special workarounds:
| File | Size |
|---|---|
gemma-4-26B-A4B-it-UD-IQ4_XS.gguf |
12.66 GiB |
gemma-4-26B-A4B-it-UD-Q4_K_S.gguf |
15.36 GiB |
gemma-4-26B-A4B-it-UD-Q4_K_M.gguf |
15.78 GiB |
Our initial Q4_K_M result was:
| Configuration | short decode | decode at 100k |
|---|---|---|
| Gemma 26B-A4B Q4_K_M | 69 | 33 |
That already put Gemma close to Qwen A3B. An external MTP sidecar is also available, mtp-gemma-4-26B-A4B-it-Q8_0.gguf, at only about 0.43 GiB. The key was selecting the correct speculative decoding path: draft-mtp, not draft-simple.
--spec-type draft-mtp \
--spec-draft-model /models/gemma26a4b/mtp/MTP/mtp-gemma-4-26B-A4B-it-Q8_0.gguf \
--spec-draft-n-max 2 \
--spec-draft-device CUDA1
At a 131k context, this setup achieved:
| Configuration | short decode | decode at 100k |
|---|---|---|
| Gemma 26B-A4B Q4_K_M + MTP | 107 | 74 |
After further tuning, Gemma even supported a 240k context on the machine. In raw throughput, it was now ahead of Qwen A3B. But we still lacked the measurement that mattered for model selection: the actual work.
Gemma's agentic performance collapsed in the same multi-stage DOCX benchmark:
| Run | Score | Runtime |
|---|---|---|
| 1 | 35 | 2,155 s |
| 2 | 0 | 1,876 s |
Neither run passed completely. The first omitted only the combined document; the second also damaged the ranking format. Both outcomes were worse than the Qwen path, despite Gemma generating more tokens per second.
That produced the following ranking on this machine:
| Model | Configuration | short decode | decode at 100k | Agent task result |
|---|---|---|---|---|
| Qwen3.6-35B-A3B | MTP + n-gram, 262k | 91 | 63 | clean runs, best path for the task |
| Gemma 4 26B-A4B | MTP, 240k | 107 | 74 | faster decode, worse on the real task |
Gemma remains a very fast candidate. Two runs are far too few for any general judgment of model quality. On this machine and in this structured tool task, though, the Qwen path won. Gemma still needs to prove its utility in more real workloads; a high TG figure cannot substitute for that test.
The benchmark itself faked a model failure
During a later campaign, our harness reported zero successes across nine runs. We spent hours investigating the model's behavior because the symptoms resembled the truncated responses we had seen earlier.
The fault was in the Python wrapper. subprocess.run(cwd=X) sets the process working directory but does not automatically update the $PWD environment variable. OpenCode read $PWD internally. Every run therefore wrote its DOCX to the same /home/lyth/benchmark/result/ directory and overwrote the previous file, while the scorer inspected the empty campaign directories and duly reported one failure after another.
The fix was small:
subprocess.run(["opencode","run","--dir",wd, ...],
cwd=wd, env={**os.environ, "PWD": wd})
The very next run achieved a PASS with 100 out of 100 points. Every diagnosis from that campaign that blamed the model for unraveling output rested on measurement artifacts. Since then, we have evaluated the generated files and their actual destination paths in agent runs, rather than relying only on exit codes and log output. When launching processes from Python, cwd and $PWD may need to be set together.
Two obvious routes that did not work on this rig
Negative results are especially valuable in a budget build. They reveal which supposedly cheap alternatives consume only time, memory, or both.
The dense Qwen3.6-27B was four times slower
A 27B model initially sounds like a better fit for 24 GB of VRAM than a 35B model. We therefore tested a roughly 17 GB Q4_K_M GGUF from Bartowski, again with 8-bit KV and a layer split.
The result was approximately 18 tok/s.
A dense model has to read all of its weights for every token. The layer split compounds the bottleneck because the two 3060s compute their halves of the model one after the other: card 0 first, then card 1. For comparison, a single V100 produced roughly 25 tok/s with the same dense model. In this configuration, the architecture constrained throughput more than the model's nominal total size.
On the same hardware, the dense 27B path was therefore more than four times slower than 35B-A3B. It also lost on context length: q8 KV reached only around 128k. With q4 KV and further adjustments, we made it to roughly 192k before hitting OOM. We deleted the model again. It was smaller, slower, and offered less context, a remarkably comprehensive defeat.
EXL3 remained less stable than the optimized llama.cpp path
We also tested an EXL3 variant at 4.00 bpw and roughly 19.4 GB using ExLlamaV3 and TabbyAPI. In our runs, this path was markedly less stable than llama.cpp with --swa-full. For our test, no directly comparable option for a full-size SWA cache was available there.
EXL3 provides an interesting quantization method and a runtime designed for consumer GPUs. Even so, llama.cpp was the route that gave us a reliably repeatable agent task. Again, this is a result for these versions and this workload, not a general verdict on ExLlamaV3.
Power draw, Euro, and the limits of the comparison
Under load, each RTX 3060 drew approximately 170 watts, falling to around 26 watts at idle. With both GPUs, the Xeon, the board, and the remaining components, the complete budget machine drew roughly 400 watts under load.
That runs off an ordinary wall outlet without special planning. The build needs neither a 1,600-watt server PSU nor a dedicated circuit. Its load transitions are also much less dramatic than those of our 3090 rig, whose hardware failures already filled an article of their own.
For Qwen3.6-35B-A3B, our documented paths produced the following rough comparison:
| Machine | Quant | Speed | Cost class |
|---|---|---|---|
| 2 x RTX 3060, budget rig | UD-IQ4_XS on llama.cpp | approx. 77 tok/s | approx. 120 Euro for board and CPU plus two used 3060s |
| 3090 path on the large rig | FP8 on vLLM | approx. 171 tok/s | complete machine costing several thousand Euro |
The faster path was roughly 2.2x as fast. The budget rig cost and consumed only a fraction of what the large rig demands, and it still delivered more than a third of its throughput. This is deliberately not a universal price-performance table: used prices fluctuate, the quants differ, and the large rig can handle workloads that fundamentally cannot fit into 24 GB of VRAM.
For maximum speed, large models, and genuine multi-user load, the 3090 rig remains in a different class. As a single fast local agent, the small machine is considerably more economical. Its advantage comes from matching the architecture to the workload. An RTX 3060 remains an RTX 3060.
The cheap machine worked only after the expensive diagnosis
The hardware story can be reduced to 24 GB of VRAM from two used RTX 3060s, a 25 Euro Xeon, and a 59 Euro motherboard. The decisive part starts after that. With roughly three billion active parameters, Qwen3.6-35B-A3B makes this modest platform fast enough. Its hybrid attention architecture keeps the KV cache small enough to fit even the native 262k context.
The machine became useful for real work only after we examined each layer separately. --swa-full corrected the SWA cache for our long conversation path. An OpenCode permission eliminated an apparent model failure. The CUDA 12.8 image avoided a CUDA 13.2 issue with IQ4_XS. A manual tensor split freed the final hundred MiB for MTP, and the correct $PWD variable ultimately cleared a model that our own scorer had falsely accused.
The result is more than a curiosity for the workbench. At 77 tok/s in the baseline configuration, 91 tok/s with MTP and n-grams, and 63 tok/s with 100k active context, the machine handles real local agent work. The hardware was the cheap part. The diagnosis cost what it costs on any other inference server: careful measurement and the willingness not to blame the quant first.