#11 · DeepSeek-V4-Flash on 8x RTX 3090: One Million Tokens, but No Tensor Parallelism

August 11, 2026 · Filip

A model can run on eight GPUs. That statement, however, can mean two very different things.

In the ideal case, all eight cards work together on every layer. They divide the work within each layer and pool their memory bandwidth, which makes inference substantially faster. In the other case, the model is simply too large for a single GPU: card 0 computes its layers and passes the hidden state to card 1, followed by cards 2 through 7. Every card contributes, but mostly one after another.

On our 8×3090 rig, DeepSeek-V4-Flash currently follows the second pattern. That is what makes this one of the stranger results in our local inference series.

We loaded a 150.75 GiB Q8 GGUF into 192 GB of consumer VRAM, then configured the server for 512k, 768k, and finally the full 1,048,576-token context window. All three configurations started successfully. The 1M server processed a real 120k-token prompt without an out-of-memory error. In a separate run with 500k active context, the model produced all 256 requested output tokens.

The good part of the result:

A 150.75 GiB DeepSeek-V4-Flash Q8 can, in fact, be served across eight RTX 3090-class cards with a one-million-token context window.

Fitting into memory is not the same as parallelization, however. The weights and KV cache are distributed across eight cards, so their combined capacity is sufficient. Throughput falls short of what one would expect from eight GPUs, because neither llama.cpp nor regular vLLM currently offers a clean, validated tensor-parallel path for DeepSeek4 in this setup.

So we reached our slightly absurd capacity target while every token made its way through the rig almost stage by stage. Only a specialized fork and MTP made that serial path considerably less painful.

Integrating the model as an OpenCode tool model also sent us down an unexpected side path. Our Laguna coding agents repeatedly failed on literal <think> strings in the chat template and output parser. Debugging uncovered both a model defect in Laguna and a separate bug in vLLM's reasoning parser that can silently discard generated content. Neither is a DeepSeek4 inference error. We found them only because the DeepSeek integration put structural tokens directly into the code under test. That investigation continues in Post #12: The Six Strings Laguna Cannot Write.

The Setup

We used the same local LLM rig as in the other articles in this series:

8x RTX 3090-class GPUs
192 GB VRAM total
Ampere, SM86
no NVLink
four separate PCIe root complexes
CUDA P2P deliberately disabled

The model was:

DeepSeek-V4-Flash
the Unsloth GGUF UD-Q8_K_XL
five shards
150.75 GiB total size

We deliberately chose the large Q8 target over an IQ2 or Q2 build. A more aggressively quantized expert model can be faster and fit on fewer cards. Our test was meant to answer a different question: How far can eight 3090s take the large Q8 if we also want to retain its enormous context window?

For capacity, the answer was farther than expected. Raw parallel throughput was a more sobering story.

The First Success: The Model Loaded

The configuration in regular llama.cpp was straightforward:

split mode:        layer
flash attention:   on
GPU layers:        all
KV cache:          q8_0 / q8_0
parallel slots:    1
batch:             1024
microbatch:        128

With 512k configured context, the complete model loaded cleanly across all eight cards. At short context, decode reached:

31.6 tok/s

At roughly 100k active context, it retained:

23.9 tok/s

Those numbers were hardly spectacular, but they were stable and internally consistent. The quant was intact, the context window was real, and the server worked. We kept increasing the reservation.

Configured server context	Loaded?	Free VRAM on the tightest GPU
524,288	yes	approximately 1.58 GiB
786,432	yes	approximately 1.18 GiB
1,048,576	yes	approximately 0.77 GiB

The one-million-token server did more than pass a health check. We sent the same 120,021-token workload to the 768k and 1M configurations:

Server context	Active prompt	Prefill	Decode	Total duration
786,432	120,021	301.52 tok/s	22.38 tok/s	409.7 s
1,048,576	120,021	290.68 tok/s	22.27 tok/s	424.6 s

This comparison supports two separate conclusions.

At 120k active context, configuring the server for 1M caused no meaningful decode penalty: 22.27 rather than 22.38 tok/s on the 768k server. Prefill dropped from 301.52 to 290.68 tok/s, so the larger reservation is not free. Even so, decode depends far more on the context that is actually active than on the value of --ctx-size.

The Q8 also had enough VRAM headroom to remain operational with the full 1M allocation. We did not have to fall back to Q4 KV, CPU offload, or a smaller target quant. That was the clearest capacity win in the entire experiment.

One Million Configured Does Not Mean One Million Understood

As in our Laguna 1M test, we separate technical allocation from semantic performance. We established the following:

1M server context allocatable:                  yes
1M-configured server processes a 120k prompt:  yes
500k-active prompt processable:                yes
server remains healthy and returns 256 tokens: yes

We did not establish any of the following:

full 1M active prompt processed end to end:    not tested
1M needle retrieval:                           not tested
1M multi-hop repo reasoning:                   not tested
same semantic quality at 1M as at 262k:        unknown

So this is not another 1M-context claim resting entirely on a health check that happened to turn green once. It is still a significant result: the Q8 model, KV allocation, compute buffers, and complete serving stack all fit across eight 24 GB cards while the server was configured for the full trained context window. Without that prerequisite, no serious 1M test can even begin.

The 500k Run: Serial Decode Starts to Hurt

Our most demanding completed run with active long context used:

500,021 active tokens
256 generated output tokens
warm-prefix reuse at cache_n=500017

Decode reached:

12.20 tok/s

Across the interrupted and later resumed sequence, prefill added up to roughly 78 minutes. There is no flattering way to present that number.

What matters is the nature of the limit. The model neither ran out of VRAM nor crashed. The server remained operational and returned all 256 requested tokens. We did not evaluate that output for retrieval or semantic quality. The run therefore demonstrates operational viability at 500k, not reliable semantic performance at 500k. What we primarily measured was the compute and attention cost of an enormous active context.

The mainline path followed this curve:

Active context	Decode rate
short	31.6 tok/s
approximately 100k	23.9 tok/s
120k on the 1M server	22.27 tok/s
approximately 500k	12.20 tok/s

That is the context tax in almost pure form. Every new token must work against an ever-growing active history. The current implementation does not turn the eight cards into a single pool of memory bandwidth. We already saw what that distinction means in practice with MiniMax under sustained agent workloads; DeepSeek4 lets us pinpoint its origin in the multi-GPU path particularly well.

Why Eight GPUs Still Work Mostly in Series

“Serial” does not mean seven GPUs sit idle. All eight cards hold weights and perform their assigned work. In the setup we tested, their combined capacity accommodates the model, compute buffers, and large context. The problem lies in the timing of those computations.

Layer Split: Capacity First

With a layer split, complete transformer layers are assigned to individual GPUs, or each card receives a group of layers. A generated token takes roughly this path:

token state
  -> GPU 0 computes its layers
  -> GPU 1 computes its layers
  -> GPU 2 computes its layers
  -> GPU 3 computes its layers
  -> GPU 4 computes its layers
  -> GPU 5 computes its layers
  -> GPU 6 computes its layers
  -> GPU 7 computes its layers
  -> final logits

Each layer needs the output of the one before it, and the next token depends on the result of the current token. In autoregressive batch-one decode, that dependency chain becomes the defining constraint.

The layer split gives us:

combined VRAM capacity,
small hidden-state transfers between stages,
relatively little collective communication,
good tolerance of a weak consumer PCIe topology.

What it cannot provide is the combined memory bandwidth of all eight GPUs operating on every layer at once. While GPU 3 reads the weights for one layer, GPUs 0, 1, 2, 4, 5, 6, and 7 cannot independently compute the next dependent layer. They are waiting for the hidden state. That is how a 150 GiB model can fit comfortably in memory and still decode at only 31 tok/s.

Tensor Parallelism: Bandwidth First

True tensor parallelism distributes the work within a layer across multiple GPUs:

one layer
  -> GPU 0 computes shard 0
  -> GPU 1 computes shard 1
  -> GPU 2 computes shard 2
  -> ... simultaneously
  -> collective reduction combines the result
  -> next layer

Several cards now read different parts of the same layer simultaneously. Only then can a single token use their combined VRAM bandwidth.

The cost is communication. Every split layer requires reductions or other collectives. On A100 or H100 systems with NVLink or NVSwitch, that overhead may be manageable. Across eight GeForce cards and four PCIe root complexes, it can become the main problem.

Even so, a correct TP path is the obvious way to turn eight cards that provide capacity into eight cards that improve single-stream speed. Another option would be a correct graph splitter that parallelizes selected operations across smaller GPU groups. For DeepSeek4, neither route is currently available to us in a stack we could deploy responsibly.

`llama.cpp`: The TP Flags Exist, but the DeepSeek4 Path Does Not

This is where we encountered the first trap. llama.cpp supports several split modes, but the existence of a flag says nothing about whether every architecture can use it.

`--split-mode tensor`

The current mainline path rejects tensor mode for DeepSeek4. A different VRAM distribution or a better tensor-split ratio cannot fix that. The production path lacks the graph and kernels needed to split DeepSeek4 correctly.

`--split-mode row`

The historical row split offered no escape either. The CUDA backend reported that the device did not support the required split buffers in this path.

ik_llama, Graph Mode, and `--max-gpu`

The most interesting theoretical compromise for consumer PCIe lives in ik_llama.cpp:

a graph split across the complete model,
limiting each operation to a smaller GPU group with --max-gpu,
distributing later graph regions across the remaining cards.

In principle, this could produce TP4-like processing within each region while distributing the regions across eight GPUs more like pipeline parallelism. This is exactly the arrangement we wanted to test.

Our older local ik_llama image did not recognize the architecture:

unknown model architecture: 'deepseek4'

Newer revisions recognize DeepSeek4, but the graph splitter still falls back to a layer split for this architecture. Once that fallback occurs, --max-gpu no longer controls a genuine graph split and stops helping our single-stream problem. The interesting flag is there; the DeepSeek4 path it needs is not finished.

The Experimental Tensor-Split PR

There is an experimental tensor-split proof of concept for DeepSeek4 in llama.cpp. On a similar Q8 setup with eight 3090s, reported results were roughly:

35 to 37 tok/s at short context
approximately 32 tok/s at around 40k context

The numbers are interesting, but this state is not production-ready. In the path we reviewed, tensors are assigned incorrectly, KV states are duplicated, and the quality validation is insufficient for a 150-GiB reasoning model. Its advantage over our later Q8+MTP result with a layer split is also too small to justify the correctness risk. An experimental graph with a plausible token rate simply is not a correct TP runtime yet.

vLLM: Fast When Hardware and Kernels Match

After reaching the limits of the GGUF path, the obvious question was whether we should stop wrestling with llama.cpp and switch to vLLM. For DeepSeek-V4-Flash on Ampere, however, the regular vLLM stack is not the shortcut it sounds like.

The official model paths focus on FP4 and FP8 formats, along with specialized attention and MoE kernels for newer datacenter architectures. On SM86, the result is not merely somewhat slower: the expected kernels and numeric formats are often missing altogether. Stock vLLM currently has no convincing, clean DeepSeek-V4-Flash TP8 path for eight RTX 3090s.

Specialized forks make the picture more nuanced:

Hardware	Runtime path	Reported result
8x RTX 3080 20 GB	Lasimeri Ampere vLLM fork, heavy CPU offload	4.3 to 4.4 tok/s
8x A800 80 GB	patched vLLM prototype	40.6 tok/s
8x A100 80 GB	patched vLLM	85.5 tok/s
8x A100 80 GB	patched vLLM plus DSpark	170 tok/s

These reports show that DeepSeek4 can be fast when runtime, quant, kernels, memory capacity, and interconnect are designed to work together. They do not show that the same path transfers cleanly to eight 24 GB SM86 cards without P2P. A100 and A800 systems are not simply larger 3090s; supported data types, memory capacity, and interconnect change the entire calculation.

Why Expert Parallelism Did Not Save Us

As a mixture-of-experts model, DeepSeek initially makes expert parallelism sound appealing: different experts reside on different GPUs, and each token travels only to the cards holding its selected experts.

The catch is extensive all-to-all communication. Tokens and expert outputs must frequently move between ranks in every MoE layer. That trade can pay off on a fast datacenter fabric. Across four consumer PCIe root complexes without usable P2P, it is risky.

One public DeepSeek4 experiment made the effect visible:

Prefill became faster.
Decode fell from 36.34 to 23.82 tok/s.

An EP-to-layer variant can improve decode behavior again, but it still requires a functioning high-speed communication path and primarily benefits prefill. Our real bottleneck was single-stream decode at large active context. An optimization that improves prefill while slowing decode by 34 percent does not solve that problem.

AppMana: The Most Ambitious Consumer-GPU Path

AppMana is the strongest counterexample to blanket claims that consumer cards cannot run DeepSeek4 usefully. Its customized path combines:

custom INT4/INT8 quantization,
pipeline parallelism,
sparse MLA kernels,
speculative decode with DSpark,
custom NCCL and runtime work.

The published consumer setup used eleven or twelve RTX 30-class cards with 24 GB each. Including DSpark, the quant is approximately 170.06 GiB. The reported results were:

33.3 tok/s without DSpark
56.1 tok/s with DSpark

That is real progress. It is only partially transferable to our current TP question, though: AppMana uses more cards and a different combination of quant and runtime in a heavily customized pipeline setup. With ten local 3090s, a PP10 experiment would be arithmetically conceivable. There is no validated PP10 run for us to reproduce, and the operational burden is far higher than launching a GGUF server.

We will keep watching this approach. For now, it is not a directly usable TP8 path for our Q8.

SGLang, KTransformers, TensorSharp, and the Rest of the Field

Our search did not stop at llama.cpp and vLLM.

SGLang Plus KTransformers

A working Ampere setup using SGLang and KTransformers ran on:

2x RTX A6000 48 GB
two Xeon Gold 6530
768 GB DDR5
CPU offload of the experts via AMXINT4

Reported throughput was 30 tok/s without MTP and 36 tok/s with MTP.

Technically, that is impressive. Yet its throughput lands almost exactly at our short-context result; the setup needs a massive CPU and RAM subsystem plus a different weight conversion, and it offers no comparable data point at 100k. It would not be an upgrade for our machine.

TensorSharp

TensorSharp remains the alternative we would still test in practice. The engine reads the Unsloth DeepSeek-V4 GGUF directly, and a verified result exists for four A40s with Q8 and DSpark:

26.4 tok/s baseline
37.1 tok/s with DSpark
69 percent speculative acceptance

There are four caveats:

The published prompt was approximately 15k, not 100k.
The weights are still placed layer by layer; this is not true TP.
The run requires a separate DSpark support GGUF of approximately 6 GB.
No meaningful advantage over our 32.39 tok/s at 100k has been shown yet.

TensorSharp is therefore a sensible experiment for later, but not evidence that the TP problem has already been solved.

Transformers

Transformers recognizes the architecture. Since the FP4/FP8 paths are unsupported on SM86, however, the weights expand to 567 GiB in BF16. That is not a serving configuration; it is an order for a different machine.

TensorRT-LLM, LMDeploy, Stock SGLang, and ExLlamaV3

The short version:

Engine	DeepSeek4 situation on our rig
TensorRT-LLM	relevant path targets Blackwell
LMDeploy	DeepSeek4 work targets Hopper
stock SGLang	blocked by FP8, FlashMLA, and MoE Ampere paths
ExLlamaV3 fork	experimental, separate quant, no reliable performance data

So there was no quietly mature engine that we had somehow overlooked. The ecosystem simply is not ready yet for DeepSeek4 with tensor parallelism across eight consumer Ampere cards.

The Turnaround: A Fork Optimizes the Serial Path

At this point, we could have ended the article by saying that the model fits but runs serially and slowly. That would have been accurate, while also omitting the best result of the experiment.

The ds4-longctx branch of alesha-pro/llama.cpp attacks a different part of the problem. It does not claim to have a correct TP path already. Instead, it accelerates the existing layer path.

The fork includes:

constant-shape decode graphs,
sparse top-k Flash Attention,
union lists for sparse-attention work,
causal skipping in the Lightning Indexer,
fused and resident MoE paths,
GLU fusion,
a fused decode indexer,
radix-select top-k for decode,
batched radix top-k for long-context prefill,
speculative K=1 MTP decode within the graph.

For long context, the sparse-attention path matters most. Regular DeepSeek4 code can apply top-k as a mask while still performing a substantial amount of dense-attention work across the active context. The specialized fork gathers only the KV positions the indexer actually selected for the sparse chunks.

The setup remains a layer split, but each serial stage performs less wasted work. We could also test the path with our existing Q8 rather than putting an experimental TP quant in the middle.

The Matching MTP Sidecar

The target model remained unchanged:

Unsloth UD-Q8_K_XL
150.75 GiB

We added the matching K=1 MTP sidecar for the original Flash version:

DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf
size: 3.545 GiB
SHA-256: afd481ee689dce9037f70f39085fcdae5a5b096d521cdad43b19fa52bf8f4083

The MTP layer proposes one candidate token at a time. The full Q8 target model verifies the proposal. Accepted candidates save work; rejected candidates never reach the output. The smaller draft component therefore never gets the final say and cannot alter the response distribution without verification.

The Optimized Results

Our measurement protocol was deliberately boring:

one warmup,
three measured requests,
256 forced output tokens each,
temperature zero,
the same Q8 target,
one short prompt and approximately 100k active context.

The results:

Context	Mainline layer split	Optimized Q8+MTP path	Gain
short	31.6 tok/s	35.98 tok/s	+13.9%
approximately 100k	23.9 tok/s	32.39 tok/s	+35.5%

The individual optimized runs:

Context	Warmup	Measured runs	Mean	MTP acceptance
short	35.84	35.97 / 35.99 / 35.98	35.98	78.87%
100,021	32.80	32.53 / 32.92 / 31.71	32.39	89.77%

A fresh 100k prefill reached:

409.59 tok/s

In a normal chat smoke test, the model answered 17 * 23 correctly with 391. The server logs confirmed that all intended optimization paths were active.

This changed our practical assessment. The short-context gain is welcome, but the 100k result is what matters. Moving from 23.9 to 32.39 tok/s does not retroactively turn a layer split into TP. It does remove a substantial part of DeepSeek's sluggishness, precisely in the context range where the model becomes interesting for agent work.

The Cost: FP16 Cache and Almost No Headroom

The optimized path comes with one major constraint. The fork forces FP16 for DeepSeek4's compressed or indexer cache and ignores the requested Q8 KV setting there. After the 100k test, the tightest GPU had only:

157 MiB free

With 157 MiB of headroom, this configuration has no real safety margin left. The measured 100k workload ran stably. For larger active contexts, we would need a better layer split, smaller batch and microbatch values, or another cache improvement before considering the optimized MTP fork safe.

This also explains why the article's two maximum results come from different configurations:

Mainline Q8: very large context capacity, including starting a 1M server
Optimized Q8+MTP fork: substantially better throughput at 100k, but more pressure from the FP16 cache

Honestly, no single configuration combines every best number in this article. That is what benchmarks are for: exposing tradeoffs rather than hiding them in a marketing line.

What the Measurements Tell Us

The blanket statement “DeepSeek-V4-Flash is slow on RTX 3090” would miss the point. More precisely, DeepSeek-V4-Flash currently lacks a clean, validated path that uses eight RTX 3090s in parallel for a single decode stream.

The Q8 quant works. It fits in memory, produces output, and permits exceptional context reservations. The optimized path also passed a basic arithmetic coherence test. Sparse attention and MTP substantially increase throughput. The missing piece is the runtime geometry.

For now, we can choose among several imperfect forms:

Layer split: correct, simple, and high-capacity, but serial during decode
Experimental tensor split: more parallel, but affected by correctness and KV-placement issues
Expert parallelism: communication-heavy and slower in decode in the published experiment
Customized vLLM and AppMana forks: promising, but using different quants, more hardware, and no clean TP8 path for this rig
CPU offload of experts: functional on other systems, but dependent on a very large CPU and RAM subsystem

No variation of “set --tensor-parallel-size 8 and call it done” can replace the missing path.

What Still Makes This Experiment Remarkable

The missing tensor-parallel path should not obscure the capacity result. We distributed a 150.75 GiB DeepSeek-V4-Flash Q8 across eight 24 GB Ampere cards and started it successfully with:

512k configured context
768k configured context
1,048,576 tokens configured context

The 1M server remained operational and processed a real 120k-token workload. In a separate run, the model handled 500k active context, produced all 256 output tokens, and reached 12.20 tok/s.

We needed no CPU offload for the weights, no retreat to a tiny expert quant, and no context setting that ended in an OOM on the first real prompt. For consumer Ampere, that is a strong capacity result.

We then kept the same Q8 target, added the matching MTP head and specialized DeepSeek4 fork, and raised decode at 100k from 23.9 to 32.39 tok/s. The eight GPUs still pass each token through the model stage by stage. They are doing so while carrying a very large model through a very large active context, and the measured workloads complete stably.

Conclusion: The Model Fits Before the Parallelism Does

DeepSeek-V4-Flash on eight 3090s makes the difference between memory scaling and compute scaling unusually clear. Eight GPUs solved our memory problem. They did not automatically accelerate decode by the same factor.

The layer split turned 192 GB of separate VRAM into enough capacity for a 150.75 GiB Q8 and the allocation of a one-million-token context window. Every autoregressive token still traverses the layer groups in dependency order. Without a correct tensor or graph split within the layers, most of the combined bandwidth cannot work on the same layer simultaneously.

The specialized ds4-longctx fork showed how much room remains even along the serial path. Sparse top-k attention, fused MoE and indexer work, constant-shape graphs, radix top-k, and K=1 MTP delivered 35.5 percent more throughput at 100k without replacing the Q8 target. This is the configuration we would use today for our measured 100k workload.

The real objective remains a correct tensor- or graph-parallel path for DeepSeek4 that uses all eight 3090s at once without corrupted transfers, duplicated KV state, or a chain of unsupported Ampere kernels. We are therefore watching developments in llama.cpp, ik_llama, vLLM, SGLang, TensorSharp, AppMana, and the smaller Ampere forks.

Once a reliable TP or graph path becomes available for this hardware class, we will test it with the same Q8, the same short/100k protocol, and the same strict distinction between configured and actually tested long context. The model already fits. Now the software needs to learn how to make all eight GPUs compute together.

Nerd Appendix: The Exact Optimized Path

The validated optimized build:

fork:   alesha-pro/llama.cpp
branch: ds4-longctx
commit: 393c388a
image:  llama-ds4-longctx:393c388a-sm86-runtime

Target model:

/bigData/llama-models/unsloth/DeepSeek-V4-Flash-GGUF/UD-Q8_K_XL/
DeepSeek-V4-Flash-UD-Q8_K_XL-00001-of-00005.gguf

MTP sidecar:

/bigData/llama-models/unsloth/DeepSeek-V4-Flash-GGUF/
DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf

The Q8-compatible optimization environment:

DSV4_CONSTANT_SHAPE=1
DSV4_SPARSE_FA=1
DSV4_FA_UNION=1
DSV4_IDX_SKIP=1
DSV4_MOE_TILE=1
DSV4_MOE_RESIDENT=1
DSV4_GLU_FUSE=1
DSV4_MOE_FUSE=1
DSV4_DECODE_FUSED_IDX=1
DSV4_DECODE_RADIX_TOPK=1
DSV4_PREFILL_RADIX_TOPK=1
DSV4_MTP_SPEC=1
DSV4_MTP_GGUF=/models/DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf

Two options are deliberately absent:

No GGML_CUDA_P2P=1. P2P across multiple root complexes previously produced corrupted null transfers on this rig and remains disabled.
No DSV4_MMVQ_SMALLK=1. This optimization targets the published expert tensors in IQ2_XXS or Q2_K, not our Q8.

The corresponding main server command is:

docker rm -f deepseek_v4_flash 2>/dev/null || true
docker run -d --name deepseek_v4_flash \
  --entrypoint /app/llama-server \
  --gpus all --ipc host --shm-size 64g \
  -p 8009:8080 \
  -e DSV4_CONSTANT_SHAPE=1 \
  -e DSV4_SPARSE_FA=1 \
  -e DSV4_FA_UNION=1 \
  -e DSV4_IDX_SKIP=1 \
  -e DSV4_MOE_TILE=1 \
  -e DSV4_MOE_RESIDENT=1 \
  -e DSV4_GLU_FUSE=1 \
  -e DSV4_MOE_FUSE=1 \
  -e DSV4_DECODE_FUSED_IDX=1 \
  -e DSV4_DECODE_RADIX_TOPK=1 \
  -e DSV4_PREFILL_RADIX_TOPK=1 \
  -e DSV4_MTP_SPEC=1 \
  -e DSV4_MTP_GGUF=/models/DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf \
  -v /bigData/llama-models/unsloth/DeepSeek-V4-Flash-GGUF:/models:ro \
  llama-ds4-longctx:393c388a-sm86-runtime \
  -m /models/UD-Q8_K_XL/DeepSeek-V4-Flash-UD-Q8_K_XL-00001-of-00005.gguf \
  --alias deepseek-v4-flash-q8-mtp \
  --host 0.0.0.0 --port 8080 \
  -ngl 999 --split-mode layer --flash-attn on --no-repack \
  --ctx-size 262144 --batch-size 4096 --ubatch-size 512 \
  --spec-type dsv4-mtp --parallel 1 \
  --jinja --reasoning on --reasoning-format deepseek

The active optimized configuration uses automatic layer placement. Before deploying it above the measured 100k workload, we would balance the layer split manually or reduce batch pressure. After the test, the tightest card had only 157 MiB free.

The mainline setup for the capacity test used smaller batches:

--ctx-size 524288 --parallel 1 \
--flash-attn on --n-gpu-layers 999 --split-mode layer \
--cache-type-k q8_0 --cache-type-v q8_0 \
--batch-size 1024 --ubatch-size 128

For the 768k and 1M runs, we changed only the configured context. Those mainline experiments produced both the successful 1M server start and the 120k-active-context measurement on that server. They must not be conflated with the tighter MTP variant configured for 262k.