#13 · DeepSeek V4 Flash on TensorSharp: The Server Path Is Ready, the OpenCode Configuration Isn’t
At the end of Post #11, we made a deliberately unglamorous promise:
The model already fits. Now the software has to learn how to use all eight GPUs together.
And then:
We'll report back.
This is that report.
The engine we described as a candidate back then has become a validated TensorSharp server path for our machine. We have tested its compatibility with OpenCode, including tool loops, and the server side is ready to use. The OpenCode client configuration currently loaded on the machine does not yet reflect that production setup, however. It still declares a 290k context, temperature 0, and no small_model.
We did not get here through a sudden burst of telepathy among eight RTX 3090s, nor did a magical --tp 8 turn the pipeline into a tensor-parallel supercomputer. TensorSharp still runs DeepSeek-V4-Flash as a layer split. Every autoregressively generated token passes through complete groups of layers in dependency order. The eight cards provide the necessary capacity, but they do not pool their memory bandwidth for each matrix multiplication.
What changed was the software around that serial path. It wastes less time, preserves state reliably, allocates memory by measured bytes rather than by optimism, and no longer forces an agent session to prefill a hundred thousand tokens again merely because another conversation took a turn in between. DeepSeek4 now also behaves like a real OpenCode model: tools arrive as structured tool_calls, reasoning remains separate from visible content, and an exactly appended continuation can retain the session's KV state.
The result is a real endpoint, not a screenshot of a model that loaded successfully. In the current comparable benchmark under the new TG-v1 protocol, it reaches a median of 44.96 tok/s at short context and 33.75 tok/s at 100k active context. In a deliberately narrow operating mode with 524k context, the validated server can retain two long-lived conversations. It can park a 500k-token session, handle a short independent session, and then return to the long one without repeating a 115-minute prefill.
Before this can become our current OpenCode production configuration, we still need to change the context, sampling, and small_model settings there and restart OpenCode. The constraints define what this endpoint actually does.
What Has Not Changed Since Post #11
Post #11 was primarily about capacity. The 150.75 GiB Unsloth UD-Q8_K_XL GGUF of DeepSeek-V4-Flash fit across eight 24 GB RTX 3090-class cards. The main branch of llama.cpp started with a configured context of one million tokens, processed a real 120k-token prompt in that configuration, and separately completed a run with 500k active tokens. That remains an unusually strong capacity result for consumer Ampere hardware.
Cleanly parallelized decoding remained out of reach. We can now summarize the paths we investigated at the time:
| Path | What it established | Why it was not the solution |
|---|---|---|
| Stock vLLM on SM86/Q8 | That the obvious TP stack does not work here | No cleanly validated DeepSeek4 Q8 path for this eight-GPU Ampere configuration |
| llama.cpp mainline | Q8 capacity, 1M configured context, and a working run with 500k active tokens | The layer split remains serial across complete layer groups; tensor and row split were unavailable for DeepSeek4 here |
ik_llama in graph mode |
That DeepSeek4 is recognized there by now | The run still fell back to layer split; --max-gpu did not create a real graph split |
ds4-longctx with K=1 MTP |
Sparse Attention, fusion, and speculation can materially improve the serial path | An important historical step, but not today's server mode, and very tight with the compressed FP16 cache |
Post #11 still contains the full technical archaeology for anyone who wants it. The connection to this article takes one sentence: our search did not produce a validated true TP8 path on this rig. TensorSharp improves the serial route this hardware actually offers, making it substantially more efficient and operationally useful.
The hardware conditions are unchanged as well. Eight RTX 3090-class GPUs sit behind four PCIe root complexes. There is no NVLink acting as a general interconnect, and P2P remains disabled. Earlier experiments produced corrupt zero transfers on this topology. Confidence is not the missing configuration flag that will make that production-ready.
TensorSharp's command line invites one particular misunderstanding. For DeepSeek4, --tp 8 selects eight GPUs for the specialized executor. That executor places complete, contiguous layer groups on the cards. The matrix work within an individual layer stays on one GPU, with no shared reduction across all eight afterward. Each token moves through a pipeline. The flag has a generic name; the execution geometry is anything but generic.
That distinction separates a defensible result from a suspiciously tidy diagram.
Three Ordinary Bugs Prevented the First Forward Pass
During our early TensorSharp attempt, three individually mundane conditions combined to prevent every startup.
First, the native GGML bridge built successfully but did not reliably make it into the published runtime package. The corrected process mounts the finished library read-only, runs the publish step in a temporary server directory, and copies that exact library into the output. Without this mount, the server does not run the native build described here. The sentence "It built on my machine" only gets more expensive once CUDA is involved.
Then placement failed. The solver understood byte capacities but not DSpark's special constraint. TensorSharp's DSpark support GGUF is about 5.99 GB, not the old 3.545 GB llama.cpp sidecar. It targets trunk layers 40 through 42 and must reside on the last GPU together with the output head. An early split put layer 40 on GPU 6 and the head on GPU 7. The solver now treats the first target layer as a hard boundary and places it, the subsequent relevant layers, DSpark, and the head together. An aesthetically balanced split that cannot execute is still just decoration.
The third problem appeared only after the model loaded successfully. The fused graph across eight GPUs registered 17 scheduler backends: eight CUDA backends, eight fused TensorSharp backends, and one CPU fallback. The embedded ggml was built for 16. The first real forward pass therefore failed. Raising the compiled maximum to 32 leaves actual headroom instead of pretending that the 17th backend does not exist.
After those fixes, the forward pass worked across eight GPUs with fused operations and Flash Attention. The smoke test correctly returned 391 for 17 * 23. The apparently unused 15.5 GiB also turned out to be an artifact of different device ordering in CUDA and nvidia-smi. With 43 trunk layers, ceil(43 / 8) = 6; at least three GPUs must therefore carry six layers. Viewed correctly in PCI bus order, the split already met that lower bound.
A One-Shot Was Fast. A Conversation Wasn't.
Once the server started, TensorSharp reached 53.3 tok/s at short context and 40.4 tok/s around 100k in its early CLI or one-shot path. A later A/B test over streaming HTTP, using its own procedure with temperature 0, 40 prompt tokens, and 256 output tokens, measured 69.33 tok/s at short context.
Those are historical observations, not entries for the current comparison table. The prompts, warmups, sampling, and execution paths differed. The same applies to ds4-longctx at 35.98 tok/s short and 32.39 tok/s around 100k, llama.cpp mainline at 12.20 tok/s with 500k, and TensorSharp's later extreme single-slot run at 18.92 tok/s. Sharing the unit tok/s does not make two measurements comparable. Benchmarks are fussy that way.
The more serious problem was that conversations could not really be continued back then. In the CLI multi-turn path, a continuation at 100k could trigger another full prefill. Every subsequent turn then took about 342 to 345 seconds. There are few more thorough ways to misuse a KV cache.
The first cause was subtle. The HTTP server correctly transferred the target model's KV state, but during the ownership change it lost the executor-managed context of MtpSpeculativeExecution. A continued request retained part of its prefix state on the target side, yet dropped out of DSpark-assisted speculative execution. The server had kept the books and misplaced the librarian.
The fix treats this speculative context as move-only state. Ownership changes only for a clean, exact, exclusively running append-only continuation whose sequence identity and physical position both match. In the process, the previous owner gives up stale pending state, while accepted speculative tails are trimmed to the published position. Cancellation, preemption, divergence, an incompatible position, or a failed rewind causes a controlled fallback to target-only work or a clean reinitialization. Speculation never runs against KV state that might belong to another session.
This is more than defensive programming. A draft verified against the wrong holder can accept incorrect tokens without necessarily crashing. A crash at least has the courtesy to announce itself.
The difference was unambiguous in the repaired HTTP path at 100k. Two exactly appended follow-up turns reused 99,892 and 99,925 tokens. Client-side TTFT was 0.422 and 0.390 seconds instead of the duration of a complete prefill. DSpark performed real verification on both turns. The claim that "every continuation processes 100k again" describes the historical failure, not the current server.
DeepSeek4 Had to Learn OpenCode's Language
A fast HTTP endpoint is not enough for a coding agent. The template now carries tools, assistant history, unchanged JSON, and reasoning. The parser recognizes deepseek4, emits structured tool_calls, and separates reasoning from visible content. A forced tool call returned finish_reason: tool_calls together with {"a":17,"b":23}; no stray </think> appeared in the content.
That parser work led to the project's strangest detour. We used Laguna agents to implement and review DeepSeek4. Naturally, the code under review contained the literal strings <think> and </think>. A series of silent failures first gave us evidence of a narrowly scoped structural-token defect in Laguna. We then found an unrelated bug in the base reasoning parser of vLLM: if a start tag appeared inside generated content, everything before it could disappear.
Neither defect belongs to DeepSeek4 or TensorSharp. The DeepSeek implementation merely supplied the code that exposed both. The token analysis, cross-provider confirmation, workaround, vLLM report, and subsequent discussion of the fix are in Post #12: The Six Strings Laguna Cannot Write.
Cache reuse remains a very strict contract. The system prompt, tools, serialized history, and generation prompt must form an append-only extension that matches exactly at both the byte and token level. Raw generated tokens are preserved across the boundary so that tools do not casually destroy the cache. A changed schema, differently ordered JSON, an unrelated request, or any turn outside that append-only rule can still end reuse. Only declared tool names become calls. DSML offers no escaping mechanism inside the data stream, so a collision with a declared name remains a known limitation.
The Cache Counted Graphs Like Identically Sized Potatoes
Even after state transfer was correct, several classes of bugs remained: unnecessary work in a hot loop, a cache with no concept of resource cost, and two lifetime bugs that looked similar from a distance.
At temperatures above zero, the sampler copied, indexed, partially sorted, and scaled all 129,280 logits. With top_k=0, it even performed a full sort. DSpark can invoke this path for several verification positions. The result was 63.82 tok/s at temperature 0, but only 31.12 tok/s at 0.1 and 30.81 tok/s at 1.0. A sweep across top_k values 0, 40, 200, and 1000 was flat.
The new bounded selector starts with 256 candidates and expands the selection only when Top-p or Min-p requires more probability mass. Temperature is applied to the remaining candidates afterward. The sampling chain is preserved, as are the restored in-place penalties.
We kept the old reference path as an oracle: 160 references and 74,000 adversarial cases. Ties are now resolved by ascending token ID. More important was a numerical edge case that even a majority of reviewers initially judged incorrectly. An early packed sort placed positive NaN above +Infinity, turning a numerical error into a guaranteed candidate. The final ordering puts NaN below -Infinity, which is the safe failure direction. In isolated historical tests, decode at temperature 1.0 rose from 30.8 to 56.2 tok/s. That, too, is development evidence rather than a TG-v1 comparison result.
Next, the graph cache crashed on a request with 35,921 tokens followed by one with 14,000. Its entry-count limit treated a large prefill graph, a one-token decode graph, different DSpark verification widths, and another session's graph as identically sized potatoes. A graph's memory cost follows its shape, not the fact that it counts as one entry.
The cache now records the actual scheduler-buffer bytes per device, checks free VRAM on the tightest GPU, and evicts LRU entries until it has a reserve. Protected graphs belonging to active slots remain in place, and the most constrained device is recalculated after every eviction. With a 512 MiB reserve, five out of five switches completed correctly at 50.9 tok/s, with no useless trims and at least 519 MiB free. A 1,024 MiB reserve, by contrast, caused eight pointless trims and reduced throughput to 43.2 tok/s. The current 524k mode uses graph cache 1 and one pin, but the essential repair was the byte-and-reserve model.
The first use-after-free involved GPU work that was still in flight. graph_build_result could release scheduler buffers without first synchronizing its use_events. Eviction would then free memory while asynchronous GPU work was still reading it. We now synchronize every event before releasing the scheduler. This can block, but it prevents premature deallocation. A later review incorrectly identified this already-fixed bug as the pinning defect; the destructor context contradicted that diagnosis.
The later bug really was a dangling pointer to the cache entry just created. With zero pins, a large reserve, and an old LRU candidate, the trim inside acquire could delete its own fresh result before returning the pointer. The outcome was double free or corruption. A temporary local guard during acquire now prevents that. Pins for active slots are a separate retention policy. A Red/Green swap based solely on the SHA established causality: the old library exited with 139, while the final build returned HTTP 200 and remained healthy. Counting reviewers would have been easier. It would not have been right.
A Conversation Needs Somewhere to Park
One working HTTP continuation was only the beginning. An OpenCode workflow moves between conversations: a long main session, a small side task, tool results, and other traffic. Prefilling a long conversation again after every switch would be technically correct and operationally absurd.
DeepSeek4's compressed attention path has neither a shared, batched Paged-KV pool nor a copyable snapshot of KV state. The generic Retained-Fused cache had inferred from an unlimited reusable-prefix value that such a shared pool existed. DeepSeek4 inherited that value without implementing any of the required Retain, Rebind, and Discard methods. Two individually plausible assumptions combined into one bug.
The Retained-Fused fix introduces capability-based handling for models that operate in fused mode per sequence but have neither a pool nor snapshots. A holder consists of nothing but the native slot ID. Retain moves that ID into a Retained map, Rebind moves it back into a new active request, and an LRU Discard releases the native slot. No 500k KV payload travels across the bus, and sparing ourselves that copy was the whole point.
The planner must also route a single sequential conversation onto the fused path in the first place. Otherwise retention would be a nicely tested mechanism with no caller. The running system can now preserve exact append-only conversations. We added Fused MTP to this retained path as a strict opt-in. Its guards bind the slot immediately before every step, hold the GPU compute lock, compare holder identity as well as sequence identity and position, and discard speculative state on any mismatch. Position alone is insufficient. Two incorrect holders can occupy the same position, and that is exactly when a purely position-based check would look reassuring.
QA produced another useful finding here. After we had removed three guard conditions, the test suite initially stayed green. The test double recognized a wrong slot by its different position, leaving it blind to a dangerous swap at the same position. The tests now assert identity explicitly, and every central guard has a test that turns red when the guard is removed. A green suite proves exactly what it checks. Whether it also covers the thing we were actually worried about is not something its color can tell us.
The current retained budget has to be read literally:
TS_RETAINED_FUSED_CACHE_MAX = N
means N completed retained holders + one separate Primary slot.
Budget 2 therefore occupies three physical cache slots at rest, not two in total. That is the minimum for A -> B -> A because A must remain retained while B is running. The validated server mode uses MAX_CONTEXT=524288, Ubatch 128, retained budget 2, Fused MTP, Prefix Cache, graph cache 1, one graph pin, and no P2P. DeepSeek occupies all eight GPUs exclusively.
A separate Qwen3-1.7B service also exists for future assignment as small_model. Title generation and other small auxiliary requests should then no longer evict a valuable DeepSeek session. That assignment is not yet loaded in the current OpenCode configuration.
Even "warm" has a narrow meaning: KV state remains available for an exactly appended continuation. A modified prompt, an unrelated request that evicts the state, or a session that was not retained may still require a cold prefill. Graph and MTP warmth are not shared evenly among holders either. This was an expensive fact to establish, but at least it is unambiguous.
We First Needed a New Yardstick
All final throughput claims use TG-v1, our current canonical three-run protocol. This deliberately breaks with the earlier collection of attractive but incompatible tok/s numbers.
The container starts fresh for each context size. A nonce-protected cold warmup builds the context and generates 256 tokens. TG1 through TG3 are genuine, topically distinct append-only extensions. Each run requests 256 tokens with the validated production sampling settings: temperature 1.0, Top-p 0.95, Top-k 0, Min-p 0, and Repetition Penalty 1.0.
We report TTFT separately. Decode throughput is calculated as:
(completion tokens - 1) / (time of last output event - time of first output event)
The first token belongs to TTFT; SSE chunks are not tokens. Dividing output tokens by total duration folds prefill and decode into a metric that measures neither cleanly. During the benchmark, we monitor the API, health, OOM events, restart counters, and local VRAM. A fast automatic restart is still a failure.
Before accepting the final results, we tested several plausible explanations. Six were wrong:
| Plausible explanation | What disproved it |
|---|---|
| The layer split left roughly 15.5 GiB unused | Solver simulation and corrected CUDA/PCI enumeration showed that the split already met the six-layer lower bound |
| A leading assistant newline broke the prefix | Turns with the same newlines achieved full reuse |
| MTP was rarely active | Its context is activated once per conversation and then persists; sparse activation logs did not imply sparse speculation |
| Session switches caused 87 percent of the cost | That was true for one TTFT view, but not for wall-clock cost, where they accounted for about 8 percent |
top_k=0 caused the non-greedy slowdown |
The sweep across 0, 40, 200, and 1000 was flat; the temperature path was responsible |
| Two retained slots would fit because we counted two | Graphs existed per slot and were limited by count rather than memory cost |
A separate fidelity test across 203,622 tokens queried facts at 1, 3, 5, 10, 25, 50, 75, 90, and 99 percent of context depth. The model answered 9 out of 9 correctly. For those facts alone, this result eases concerns about Prefix Cache and compressed attention. It establishes neither universal quality nor comprehension across a full active million tokens; both remain untested.
These are the current, directly comparable TensorSharp results:
| TG-v1 metric | Short context | 100k active context |
|---|---|---|
| Cold warmup prompt | 409 tokens | 100,044 tokens |
| Warmup prefill | 1.99 s / 205.28 tok/s | 475.00 s / 210.62 tok/s |
| TG1 / TG2 / TG3 decode | 44.96 / 44.24 / 45.28 tok/s | 33.75 / 37.72 / 33.43 tok/s |
| Median decode | 44.96 tok/s | 33.75 tok/s |
| Mean / standard deviation | 44.83 / 0.43 tok/s | 34.97 / 1.95 tok/s |
| Median TTFT | 0.58 s | 1.13 s |
| Accepted by MTP | 478 / 644 = 74.2% | 504 / 648 = 77.8% |
| Lowest free VRAM during TG1-TG3 | 1,170 MiB | 1,038 MiB |
The 100k column prevents us from celebrating decode in isolation. The cold prefill took 475.00 seconds at 210.62 tok/s. That delay returns whenever no exactly matching retained prefix is available. TTFT and decode appear separately because both matter in operation.
The canonical raw data are in short.json and 100k.json. The same directory also contains the archived files current-container-inspect.json, current-models.json, deployed-native-sha256.txt, and SHA256SUMS.
Our previous temperature-0 configuration produced repetitive Doom Loops. That was our configuration error, not a defect in TensorSharp, Prefix Cache, or compressed attention. The measured production setting is 1.0 / 0.95 / Top-k 0 / Min-p 0 / Repeat 1.0.
500,000 Tokens, Two Sessions, and No Memory Reserve
Our most operationally interesting long-context test used the Fused2 configuration: retained budget 2, maximum context 524k, Ubatch 128, graph-cache limit 1, one graph pin, Prefix Cache, Fused MTP, P2P disabled, and eight otherwise idle GPUs.
On a freshly started server, we ran the sequence A, B, A, B, A:
- A started with a cold prompt of 500,027 tokens. Client-side TTFT was 6,913.347 seconds, or about 115.2 minutes.
- B was a short, independent context.
- The first return to A took 1.666 seconds client-side; the second took 1.680 seconds. Server-side TTFT was 990 ms and 967 ms.
- On those two return steps, A reused 500,037 of 500,080 and 500,090 of 500,133 tokens. Server-side
kvReusedreported 99.991%. - The warm return to B reused 1,358 of 1,400 tokens and achieved a client-side TTFT of 0.260 seconds.
The native holder was rebound, the exact prefixes were reused, and all five responses were correct in routing and transport. However, cached_tokens and kvReused are two views of the same telemetry. The continuations also merely reproduced the expected secret. This test is therefore not independent evidence of KV integrity.
Only KV state stayed warm for both holders. A had to rebuild its graph and later lost it again; only B retained the global MTP context. We make no claim of symmetric graph or MTP warmth.
The memory boundary is equally clear. The native low-water mark was 205 MiB, while the host sampler observed 211 MiB on the tightest GPU. Budget 2 works under exclusive conditions, but a 205 MiB reserve is not enough for an unattended service. Unrelated CUDA work on these cards would be a very short experiment under those conditions.
Budget 3 kept three short sessions warm, confirming the budget semantics. Yet only 105 to 209 MiB remained on the tightest GPU before accounting for a known 500k trunk graph of about 333 MiB. That is a directly demonstrated memory shortfall. We therefore rule out a third retained holder at 500k; we did not have to provoke an OOM deliberately to establish that.
Separate from this is a historical extreme test. With MAX_CONTEXT=1048576, graph cache 1, one pin, and fused per-sequence mode disabled, a single-slot run processed 500,008 prompt tokens and generated 128 completion tokens. Prefill took 6,691.227 seconds at 74.7259 tok/s, decode reached 18.9210 tok/s, and free VRAM fell to 9 MiB.
That was a capacity test at the edge, not the current multi-session production mode, not safe unattended operation, and not evidence of comprehension across a full active million tokens. Its outputs provided a limited capacity oracle. We did not test retrieval across one million tokens, needle finding, or multi-hop reasoning.
The question of a full active million remains open. Configuring context allocation for one million tokens, completing a 500k capacity run, and passing a fidelity test at 203k are three different claims. We are not combining them into one especially marketable million-token number.
The Summary Without a Marketing Tagline
This endpoint's suitability depends on a set of explicit conditions, not one heroic number:
| Claim or operating characteristic | What it actually means | What it does not mean |
|---|---|---|
--tp 8 in TensorSharp |
Eight GPUs hold the complete DeepSeek4 layer pipeline | True TP8; the cards do not compute each layer together |
| 44.96 short / 33.75 at 100k as TG-v1 medians | Current comparable decode rates for warm continuations under validated server sampling | Cold-prompt latency or a ranking against historical numbers |
| 210.62 tok/s during cold 100k prefill | A 100k prompt can be built in the tested mode | A short wait; it took 475.00 seconds |
| A/B/A switching at 500k with retention | Two holders can keep their KV state warm under exact append-only reuse | Symmetrically warm graphs and MTP state, or independent proof of KV integrity |
| Budget 2 | Two completed retained sessions plus a Primary slot fit functionally | A general multi-user service or a safe third 500k holder |
| Low-water mark of 205 MiB | The Fused2 test completed without an OOM | Headroom for unrelated GPU work or unattended operation |
| 1M single-slot configuration | Capacity can be pushed to a prompt of 500,008 tokens | Quality, retrieval, or comprehension across a full active million |
Once the client is configured accordingly, this local path is suitable for a small number of disciplined, long-lived OpenCode conversations. Arbitrary shared traffic is outside its intended use. Tools and prompts must form exact append-only continuations, and unrelated requests can evict state. Protection through a separate Qwen small_model is planned but not yet loaded in OpenCode. All eight GPUs remain exclusively occupied, P2P remains disabled, and the working tree is uncommitted on base e765938. We claim neither a merge, a commit, nor upstream adoption.
What Generalizes, and What Belongs to DeepSeek4
These changes are not one monolithic "DeepSeek patch." Some address general runtime engineering and should remain separate from model-specific execution.
| Area | General lesson | DeepSeek4-specific part |
|---|---|---|
| Non-greedy sampler | Bounded candidate selection avoids fully sorting the vocabulary in the hot decode path; NaN must fall in the safe direction | DSpark verification positions amplify the measured effect |
| Graph cache | Entry-count limits are not a memory model; actual bytes per GPU must be checked against free VRAM on the tightest device | DeepSeek4 graph shapes, pipeline parity, and pinning of active slots |
| Scheduler destruction | Backend events in use must be synchronized before scheduler buffers disappear | The specific lifecycle of graph_build_result in this executor |
| Fresh cache entry | acquire must not return an entry that its own trim has already destroyed |
Acquire paths and pins in DeepSeek4's graph cache |
| Retained holder | Capability checks must distinguish an absent pool from an unlimited shared pool | Moving native DeepSeek4 slots without snapshot support |
| Speculative continuation | Speculative state may move only when identity, position, and ownership continuity have all been confirmed | DSpark, its placement at the output head, and slot-local ring buffers |
| Agent adapter | Raw representations and token-exact boundaries can preserve cache fidelity | Jinja-like DeepSeek4 template, DSML parser, reasoning, and tools |
This separation keeps a generally useful fix from disappearing inside a model-specific package. It also prevents us from presenting DeepSeek4-specific behavior as a universal runtime solution. We apply the same evidentiary limits to testing: the focused C# regression gate passed 376 out of 376 tests. Six known failures unrelated to the patch remained in the broader suite. Five result from a missing native library in the test environment; one is a pre-existing floating-point tolerance test. None of this turns a local, uncommitted working tree into an upstream change.
Operational limits and measurements are recorded in the canonical operating reference. Build, lifetime, test, and implementation details are in the TensorSharp development log. Neither document promises that the changes have been merged anywhere.
The Software Learned More Than the GPUs Did
Post #11 ended with a hardware finding: the model fit, but the runtime did not yet use the eight GPUs well enough. TensorSharp has changed nothing about that finding. Execution remains a layer split, and we still have no true TP8 result. The serial dependency path limits single-stream decode, especially with a large active context. Eight GPUs provide a very large aggregate memory budget, not an eight-way distributed matrix engine.
The software did eliminate a long list of concrete problems. It now starts with the actual native library, treats DSpark placement as a solver constraint, and transfers every required state during HTTP continuations under strict identity checks. DeepSeek4 now presents tool calls and reasoning in the format OpenCode expects without casually losing cache boundaries when the prompt is rendered again. The sampler no longer sorts 129,280 logits repeatedly for every non-greedy selection. The graph cache counts memory rather than entries. We separated two similar-looking lifetime bugs and fixed their actual causes. A retained session now has a native parking space.
Above all, this work repeatedly reminded us that a plausible explanation, a green test suite, or a majority of agents is no substitute for a measurement.
This machine now has a server-side validated path: current TG-v1 medians of 44.96 tok/s at short context and 33.75 tok/s at 100k, structured tools, separate reasoning, validated production sampling, and two retained long-lived conversations within deliberately narrow limits. It will become the current OpenCode production configuration only after the client changes.
A cold 500k prefill still costs about 115 minutes. The 500k mode supports only two retained sessions. A minimum reserve of 205 MiB is insufficient for unattended operation. The 100k prefill may run at 210.62 tok/s, but it still takes 475 seconds in total. Reuse demands exact append-only prompts, and unrelated traffic can evict state. The answers in the 500k routing test are not independent evidence of KV integrity, while comprehension across a full active million tokens remains untested. The most visible open item is the loaded OpenCode configuration with 290k, temperature 0, and no small_model. The client will match the validated server mode only after we update it and restart OpenCode.
Those constraints do not diminish the endpoint. They describe it.
The model fit first. The software did not make eight consumer GPUs behave like a data-center interconnect. It made the available serial path waste far less time and finally remember a conversation.
Nerd Appendix: The Current Operating Mode
The model remains the 150.75 GiB, five-shard Unsloth DeepSeek-V4-Flash-UD-Q8_K_XL. TensorSharp uses its own DeepSeek4 DSpark support GGUF as the draft model: about 5.99 GB, three stages, block size five, and target layers 40 through 42. llama.cpp's old K=1 MTP sidecar is neither the same file nor the same mechanism and must not be used here.
The validated server configuration is deliberately tight. It does not yet match the currently loaded OpenCode client configuration:
8 visible RTX 3090-class GPUs, exclusive; P2P disabled
MAX_CONTEXT=524288 TS_DSV4_UBATCH=128
retained fused budget=2 prefix cache=on
fused MTP=on graph cache=1, graph pins=1
sampling: temperature=1.0, top_p=0.95, top_k=0, min_p=0, repeat=1.0
future small_model: separate Qwen3-1.7B service, not DeepSeek
The extreme single-slot mode is a different configuration: a maximum context of one million tokens, graph cache 1, one pin, and fused per-sequence mode disabled. It exists for capacity experiments and is not an upgrade path for the retained-session production setup.
Future reproducible comparisons should use the documented TG-v1 protocol. Older greedy scripts and particularly successful one-shot prompts do not belong in the same table. TG-v1 builds a nonce-protected context cold, then measures three different warm extensions. TTFT, prefill, cache reuse, decode, MTP statistics, liveness, and local VRAM are reported separately wherever available.
Any future runtime claiming a correct and validated graph- or tensor-parallel path for DeepSeek4 on this Ampere system will need to meet the same Q8 target, the same server-side validated sampling, separate TTFT and decode measurements, and the same skepticism toward long-context claims.
We promised to report back. This is the report.