#20 · New Inference Engines Bring Huge Models to Modest Hardware for Small Teams: DeepSeek with a 1M Context Window on 8× RTX 2080 Ti
748 billion parameters, eight older graphics cards, and a rig costing under €6,000: we run DeepSeek-V4.1-Flash-1017 locally on eight modified RTX 2080 Ti cards with 22 GB each. New community-built inference software is a major reason this setup is usable for a small team. It distributes model components across VRAM and system RAM, puts the CPU to work, and can quickly park long sessions. Space-efficient quantizations have kept quality differences small in our tasks so far.
Community inference software is changing what these cards can do. Together, they provide 176 GiB of VRAM; even quantized, the model is considerably larger. Our article on importing 22 GB RTX 2080 Ti cards from China covers how we sourced them. Here, we look at what now runs on them and how a team can share the rig.
Putting system RAM to work
Fitting a large model outside VRAM only solves the capacity problem. For usable speeds, software must also handle weights spread across memory tiers. Projects including Strata, Maya, and Avalon address this. These are inference engines: software that runs models, not model developers or quantization providers.
Mixture-of-Experts models consist of specialized subnetworks called experts. Only a small subset works on each token; a cache can keep frequently needed weights available. Avalon, a llama.cpp fork for Qwen Flash Next, runs experts in RAM on the CPU alongside GPU computations and preloads weights. Strata also uses CPU experts and caches for Qwen. We use Maya for GLM and DeepSeek; for DeepSeek, it delivered by far the highest speeds on our Turing cards.
The engines coordinate GPU, CPU, and memory differently, making models whose weights exceed VRAM capacity usable. Depending on the engine, the CPU handles some computation while weights are made available across memory tiers. This avoids reading the entire model from SSD for every token.
27B: One card already helps
A 27B model hardly needs the whole rig. Qwen 3.8 27B reached around 92 output tokens per second on one card using Turboderp’s EXL3 quantization at the standard 4.00 bits per weight. For comparison, we also ran vLLM with Unsloth’s mixed 4- and 8-bit NVFP4 quantization. Both setups used inference engines adapted for Turing GPUs. DFlash and MTP use speculative decoding: tokens are proposed in advance for the model to check.
All speeds measure output tokens per second, including thinking tokens, not total time to a completed answer. “100k” denotes the long-context test configuration: Qwen actually had around 81,000 preceding tokens, GLM around 85,000. The context window applies per session; the shared context pool must accommodate all sessions held concurrently. Each speed below comes from a single active request.
| Cards | Engine / quantization | Tokens/s: short / 100k | Window per session | Context pool: configured / tested |
|---|---|---|---|---|
| 1 | EXL3 DFlash / Turboderp 4.00 bpw | 91.64 / 68.20 | 131,072 | 131,072 configured |
| 1 | EXL3 MTP / Turboderp 4.00 bpw | 68.05 / 60.67 | 262,144 | 262,144 configured |
| 2 | EXL3 MTP / Turboderp 4.00 bpw | 84.72 / 83.75 | 262,144 | 1,179,648 configured; 1,061,070 tested with five long sessions |
| 2 | vLLM DFlash / Unsloth NVFP4 mix | 93.98 / 69.34 | 262,144 | 376,329 reported |
| 4 | EXL3 MTP / Turboderp 4.00 bpw, two power supplies | 107.43 / 93.48 | 262,144 | 2,097,152 configured, not filled |
| 4 | vLLM DFlash / Unsloth NVFP4 mix, pool configuration | 85.87 / 69.56 | 262,144 | 1,373,441 reported; 1,234,782 tested with six long sessions |
One card already delivers fluid output. Additional cards mainly provide room for multiple long sessions; we tested the configured pool capacities only to the extent described.
GLM already ran on four cards
GLM 5.3 Flash has around 321 billion parameters—and already ran on four cards here. Peasantsmith’s Maya-S24 quantization totals 94.7 GB, or about 88.2 GiB, in files. Four cards provide around 88 GiB of VRAM, which must also accommodate caches. Maya therefore distributes model components across GPU and host memory, keeping expert components in RAM or on SSD as needed.
| GLM on four cards | Tokens/s: short | Tokens/s: 100k |
|---|---|---|
| Early setup | 55.45 | 45.93 |
| After RAM upgrade and further optimizations | 65.62 | 53.14 |
| Later, same hardware | 67.53 | Not remeasured |
With CPU offloading now so important, we replaced the slow RAM: four 32-GB DDR4-3200 modules replaced a single 32-GB DDR4-2133 module. The CPU now has 128 GiB of faster RAM across four memory channels. Together with software improvements, this raised speeds from around 55 to 67 tokens/s over time.
We ordered four additional cards to expand further. Initially, the motherboard recognized only six. Even this intermediate setup had enough VRAM for all 12,096 GLM experts, so the CPU no longer needed to compute them. A separate comparison using team configurations showed single-request speed rising from 63.10 to 74.04 tokens/s, a 17.3% increase. With four concurrent short requests, combined throughput rose from 141.5 to 162.5 tokens/s.
748B on six cards: It ran, but we wanted more speed
DeepSeek-V4.1-Flash-1017 initially delivered 22.98 output tokens per second on six cards. That was usable, but too slow for the team rig we had in mind. After tuning parallel distribution and speculative acceleration, we reached 53.0 tokens/s with short context and 48.2 in the 100k configuration using Maya. This setup provided one active slot with a context window of 262,144 tokens.
The model’s scale is still unusual for this hardware: DeepSeek has a backbone with 552 billion parameters and an Engram component with another 196 billion, totaling 748B. As an MoE model, it activates only a small subset of its experts for each token. The remaining weights still need somewhere to reside and must be accessible when needed.
We use peasantsmith’s Maya-S quantization. The backbone—the part of the model that performs the computations—averages about 2.6 bits per weight in this quantization. Precision is mixed; individual components such as attention remain at 8 bits. The Engram lookup table uses Q6_K. Altogether, the files total 339.5 GB: 178.2 GB for the backbone and 161.3 GB for Engram.
Engram resides on SSD, but each token requires only a few table rows. A RAM cache keeps frequently needed data available, so the SSD does not have to keep transferring hundreds of gigabytes. With six cards, many experts also remain in system RAM. Our host, with 128 GiB of DDR4-3200, therefore takes part in the computation rather than just loading data for the GPUs.
The compact weights make it possible to fit the model. Maya makes its distribution across GPU, RAM, and SSD fast enough to use. The next question was whether that much compression cost too much quality for our tasks.
How much quality do we retain locally?
Higher scores mean better results in our in-house benchmark; we will explain the tasks and scoring in the next article. Our local GLM scored 74%, versus 75% for the FP8 version served by a provider. DeepSeek V4.1 Flash (1017), using the Maya-S quantization, scored 78% in the new run. In our task mix so far, the large local models were therefore close to the provider offerings compared here, such as those available through OpenRouter.
| Model / setup | In-house benchmark score (%) |
|---|---|
| GPT 6.1 Sol | 90% |
| DeepSeek V4.1 Flash (1017), Maya-S | 78% |
| Qwen 3.8 Flash Next, our own server, AWQ | 77% |
| Claude Haiku 5.5 | 76% |
| Qwen 3.8 Flash, OpenRouter | 76% |
| GLM 5.3 Flash, OpenRouter, FP8 | 75% |
| GLM 5.3 Flash, local, Maya-S24 | 74% |
| Qwen 3.8 Flash Next, IQ3 | 73% |
| Qwen 3.8 27B, maximum 5,000 thinking tokens | 73% |
| Ornith 1.5 35B A3B | 59% |
| Qwen 3.6 27B, Q4 | 51% |
| Qwen 3.6 35B A3B, OpenRouter, FP8 | 47% |
For Qwen Flash Next, our AWQ version scored 77%, ahead of the IQ3 setup at 73%; for GLM, the gap between the local model and provider-served FP8 was smaller. Close results on our tasks do not imply universally equal quality. The rough ranking fits our impression from Artificial Analysis—our interpretation, not an equivalence between scoring scales. For our tasks, though, this comparison gives us a clear indication of whether a quantization costs too much quality.
Getting all eight cards working meant disabling onboard graphics
The motherboard initially refused to recognize two of the eight cards. The additional GPUs were installed but unavailable for inference. Only after we disabled onboard graphics on the Supermicro H12SSL-i did it recognize all eight.
The rig has had no local display output since then; we operate it remotely. In return, Maya now has access to all eight RTX 2080 Ti cards, with 176 GiB of VRAM combined. DeepSeek runs across all eight as one compute group, technically TP8.
That let us move beyond a single slot: how many requests could we serve concurrently, and how much context should each session get? In Maya, that choice already affects memory allocation at startup.
Eight cards: fast solo, shareable as a team
With all eight cards, DeepSeek reached 87.3 output tokens per second with short context and 80.1 in the 100k configuration, using one active slot with a 262,144-token context window. That is a substantial step up from the latest six-card results of 53.0 and 48.2 tokens/s. For shared use, though, we were more interested in performance with multiple concurrent requests.
The following table compares configurations tested on November 18. “Slots” are sessions that can be used concurrently, not additional model copies. Team speeds represent the combined output of all running requests.
| Configured slots | Context window per slot | Single short request, tokens/s | 4 active requests at 100k each, total tokens/s | 8 active requests at 100k each, total tokens/s |
|---|---|---|---|---|
| 8 | 131,072 | 80.7 | 113.3 | 144.5 |
| 4 | 524,288 | 76.6 | 107.2 | – |
| 8 | 524,288 | 75.1 | – | 127.2 |
| 4 | 1,048,576 | 76.3 | Pending | – (four concurrent requests maximum) |
Four configured slots allow at most four concurrent requests, so eight-request results are not applicable to those rows. Other dashes indicate unmeasured cases. “Pending” marks the four-request test still to come with the current million-token configuration.
With a 512k window per slot, 107.2 tokens/s across four configured and active slots works out to 26.8 tokens/s per session. With eight configured and active slots, 127.2 divided by eight gives 15.9. These are average shares of total throughput, not guaranteed individual output rates.
Our current production configuration provides four slots, each with a 1,048,576-token context window. A subsequent production check on November 19 yielded 71.5 tokens/s with short context and 75.9 at 100k. We still need to test how well four concurrent sessions work with very long contexts. The 512k windows in the table also held only 100k each during the team tests. Eight slots with a million tokens each exhausted system RAM during loading.
A large context window has a cost before the first long prompt
Configuring eight slots with long context windows makes Maya allocate the corresponding memory at startup. Its DeepSeek implementation reserves memory for the full context window of every configured slot, even when that slot is idle. A short prompt does not automatically release that reservation. This differs from vLLM’s dynamically shared KV pool.
That reservation competes with model weights for VRAM. Four slots at 512k each reserved roughly 13 GB, with around 53.4 GB of expert weights in RAM. Eight such slots required about 31 GB of KV reservation and pushed expert weights in RAM to 69.5 GB. Four slots at a million tokens each were similar: around 31 GB reserved and 69.1 GB of expert weights in RAM.
Teams should therefore choose the number of concurrent slots and the desired context length deliberately. Eight configured slots consume memory even when only one person is asking a question. Four may suffice if people mostly take turns and can still keep their previous sessions.
The game changer: parking long sessions instead of reprocessing them
When all active slots are occupied, another session needs one to become available. Recomputing the previous context later can take time: in our 450k-context test, it took 588 seconds to get an answer—almost ten minutes. That was not an isolated measurement of input processing.
Maya’s session-retention feature, enabled with MAYA_DS4_SLOT_KEEP=1, avoids recomputing that previous context. When another request needs an occupied slot, the engine can park the existing session between requests and reassign the slot. It does not interrupt an answer in progress. When the parked session makes another request, Maya restores its saved context state.
For a session containing around 355k tokens, saving to RAM took 0.2 seconds and restoring took 0.3 seconds. Processing 733 new tokens then took another 3.3 seconds. Restoring the previous state and processing new input are separate steps. The 0.3 seconds is not a promise for the entire next answer.
If the allocated RAM budget is insufficient, sessions can also be parked on SSD. This differs from offloading model weights: here, the session’s already-computed context is preserved. More employees or agents can therefore take turns than there are active slots. Four slots still serve at most four sessions concurrently.
Reuse works only if the previous context is recognized. If the client changes its beginning, such as the system prompt, the expensive reprocessing may be necessary again.
A 700B model comes within reach of small teams
A model with 748 billion parameters running on our sub-€6,000 rig becomes an interesting option for small teams taking turns. Specialized engines distribute weights and computation across GPUs and the host; parking keeps long sessions ready for the next request. Initial input processing can still take minutes. We have yet to try four fully populated million-token windows together, and eight such slots do not fit in this machine’s RAM. We will cover our in-house benchmark in more detail in the next article.