← Insights

DE — Deutsche Version

#9 · Three Models, Eight GPUs: How We Split 192 GB of VRAM

· Filip

Local inference often revolves around one question: how do we get this one large model running? One model, all eight GPUs, TP=8, done. It makes a good story for the first photo of the server in action.

Our day-to-day workload looks different. We need a large workhorse for demanding coding tasks, a fast model for interactive queries, and possibly a third for office work, language, or image processing. At that point, fitting a model into 192 GB of VRAM is no longer the real problem. We have to decide which cards each model gets, how much context that leaves, and which combinations survive more than a brief test run.

This article is about what may be the least glamorous but most operationally useful part of the rig: memory accounting. Every number comes from measurements on our system. Along the way, two hardware faults kept us busy in very different ways: an overheating card whose core temperature looked perfectly harmless, and a network cable that spent months doing a convincing impression of Gigabit Ethernet.

The Rig at a Glance

Our local LLM rig contains seven RTX 3090 cards and one RTX 3090 Ti. For a multi-model setup, these are the details that matter most:

  • 8x Ampere with 24 GB each, for roughly 192 GB of VRAM in total. All cards are SM 8.6, so they have no hardware support for FP8 or FP4. With INT4, Marlin merely decompresses the weights.
  • An EPYC 7443P, 256 GB of DDR4, and a Supermicro H12SSL-i, with no NVLink and four PCIe root complexes in domains 00, 40, 80, and c0.
  • The proprietary nvidia-driver-580, version 580.126.20. We deliberately disabled P2P because, with GeForce cards spread across separate root complexes in our setup, it returns nothing but zeros anyway.

Those 192 GB do not form a shared pool. They are eight physically separate 24 GB regions. Each model receives a fixed group of cards and must stay within the VRAM budget of every individual card in that group.

Device ID and Physical Slot Are Not the Same Thing

The --gpus '"device=N"' argument in Docker refers to the CUDA device ID, not the physical slot. Confusing the two is a remarkably reliable way to inspect the wrong card during thermal troubleshooting. This is the mapping for our rig:

Device PCI BDF Slot Card
0 01:00.0 SLOT6 3090
1 41:00.0 SLOT2 3090
2 42:00.0 SLOT4 3090
3 81:00.0 SLOT7 3090
4 82:00.0 SLOT1bif 3090 Ti
5 83:00.0 SLOT1bif 3090
6 c1:00.0 SLOT3 3090
7 c2:00.0 SLOT5 3090

The first row became particularly important: Device 0 is installed in SLOT6. We will return to that card later.

Four GPUs for Laguna, Four Still Available

Our target layout was easy to describe: the large main model runs on four GPUs, leaving the other four available for one or two smaller models.

The large model is Poolside Laguna S 2.1 INT4, our daily driver for coding tasks. It runs with TP=4 on devices 3, 4, 6, and 7 and is available on port 8005. These four cards can run Laguna with the full 262,144-token context, BF16 KV, and a usable KV reserve. This layout also leaves Device 0 in SLOT6 unused — exactly the card whose cooling becomes problematic under sustained load.

We measured the following results for Laguna:

Topology Context Short decode Decode at 100k active context KV capacity
TP=4, BF16 KV, utilization 0.96 262144 109.1 tok/s 85.6 tok/s 394,028 tokens (1.50x at 262k)
TP=4, FP8 KV, utilization 0.92 262144 105.8 tok/s 97.2 tok/s 377,571 tokens (1.44x at 262k)

BF16 KV was 3.1 percent faster than FP8 KV during the short decode. At 100,000 tokens of active context, however, it dropped from 109.1 to 85.6 tok/s, putting it behind the FP8 result of 97.2 tok/s. It also left only about 0.85 GiB of headroom per GPU. We still used BF16 KV in production because FP8 KV has a set of much less pleasant limitations on this Ampere stack.

This is the complete command for starting the model on exactly those four GPUs:

docker rm -f laguna_s21_int4 2>/dev/null || true
docker run -d --name laguna_s21_int4 --entrypoint vllm \
  --gpus '"device=3,4,6,7"' --ipc host --shm-size 64g \
  -e NVIDIA_DISABLE_REQUIRE=1 -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e NCCL_P2P_DISABLE=1 -e NCCL_IB_DISABLE=1 -e NCCL_CUMEM_ENABLE=0 \
  -e VLLM_ALLREDUCE_USE_SYMM_MEM=0 -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e VLLM_ENABLE_CUDA_COMPATIBILITY=0 \
  -v /bigData/hf-cache/Laguna-S-2.1-INT4:/models/Laguna-S-2.1-INT4 \
  -v /bigData/vllm/data/torch-extensions-v024:/root/.cache/torch_extensions \
  -v /bigData/vllm/data/triton-v024:/root/.triton \
  -v /bigData/vllm/data/cuda-cache-v024:/root/.cache/cuda \
  -v /bigData/vllm/data/vllm-cache-v024:/root/.cache/vllm \
  -p 8005:8000 \
  vllm/vllm-openai:v0.25.1 \
  serve /models/Laguna-S-2.1-INT4 --served-model-name laguna-s-2.1-int4 \
  --tensor-parallel-size 4 --max-model-len 262144 \
  --kv-cache-dtype bfloat16 --gpu-memory-utilization 0.96 \
  --disable-custom-all-reduce --enable-prefix-caching --generation-config auto \
  --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 \
  --default-chat-template-kwargs '{"enable_thinking":true}' --trust-remote-code \
  --max-num-seqs 1 --max-num-batched-tokens 4096 \
  --host 0.0.0.0 --port 8000
until curl -fsS http://127.0.0.1:8005/health; do sleep 3; done

The wall of NCCL_* and VLLM_* variables is not years' worth of accumulated command-line superstition. It disables the exact P2P paths that do not work across our four separate root complexes. Without --disable-custom-all-reduce, vLLM performs its own P2P check and ignores NCCL_P2P_DISABLE=1 in the process. An earlier debugging session cost us an evening over that lesson; the flag has been part of the launch recipe ever since.

Once Laguna is running, devices 0, 1, 2, and 5 remain free.

What Fits on the Other Four GPUs

Qwen3.6-35B-A3B-FP8: About 171 tok/s with TP=4

The obvious candidate is Qwen3.6-35B-A3B-FP8. This MoE has 35 billion parameters in total but activates only about 3 billion per token. On the four free GPUs, devices 0, 1, 2, and 5, it reaches approximately 171 tok/s with TP=4 and a short context. The full 262,144-token context fits as well. vLLM reports a KV capacity of 1,182,514 tokens, equivalent to 4.51 concurrent 262k contexts. The weights consume just 8.82 GiB per GPU.

docker rm -f qwen36_35b_a3b 2>/dev/null || true
docker run -d --name qwen36_35b_a3b --entrypoint vllm \
  --gpus '"device=0,1,2,5"' --ipc host --shm-size 64g \
  -e NVIDIA_DISABLE_REQUIRE=1 -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e NCCL_P2P_DISABLE=1 -e NCCL_IB_DISABLE=1 -e NCCL_CUMEM_ENABLE=0 \
  -e VLLM_ALLREDUCE_USE_SYMM_MEM=0 -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e VLLM_ENABLE_CUDA_COMPATIBILITY=0 \
  -v /bigData/hf-cache/Qwen3.6-35B-A3B-FP8:/models/Qwen3.6-35B-A3B-FP8 \
  -v /bigData/vllm/data/torch-extensions:/root/.cache/torch_extensions \
  -v /bigData/vllm/data/triton:/root/.triton \
  -v /bigData/vllm/data/cuda-cache:/root/.cache/cuda \
  -v /bigData/vllm/data/vllm-cache:/root/.cache/vllm \
  -p 8007:8000 \
  vllm/vllm-openai:latest \
  serve /models/Qwen3.6-35B-A3B-FP8 --served-model-name qwen3.6-35b-a3b \
  --tensor-parallel-size 4 --max-model-len 262144 --gpu-memory-utilization 0.92 \
  --max-num-seqs 1 --disable-custom-all-reduce --enable-prefix-caching \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --trust-remote-code --host 0.0.0.0 --port 8000
until curl -fsS http://127.0.0.1:8007/health; do sleep 3; done

This model exposed a hard architectural limit that is easy to miss in a typical model overview: TP=8 is mathematically impossible with this FP8 block quantization.

The block quantization uses a fixed block_k=128, while the hidden dimension is 512. With TP=8, each partition would receive only 512 / 8 = 64 values, fewer than one complete block of 128. vLLM therefore aborts startup with ValueError: Weight input_size_per_partition = 64 is not divisible by weight quantization block_k = 128. This is neither a memory problem nor a configuration error. It has no practical impact on our layout because we intended to give the model exactly four GPUs anyway. With TP=4, the partitioning works.

Qwen3.6-27B-FP8: Smaller, but Considerably Slower

The dense Qwen3.6-27B-FP8 is nominally smaller. On the same four GPUs, it was nevertheless much slower on cold hardware:

Model Active parameters Layout Short decode
Qwen3.6-35B-A3B about 3B TP=4, devices 0/1/2/5 about 171 tok/s
Qwen3.6-27B dense model, correspondingly more active parameters TP=4, devices 0/1/2/5 about 65 to 67 tok/s, cold

The 35B model was more than 2.5 times as fast as the 27B model. Decode performance depends primarily on how many parameters must pass through the memory pipeline for each token. The A3B MoE reads about 3 billion active parameters; the dense 27B model has to process all its weights. The total size in the model name says little about that.

The word "cold" in the table matters. Under sustained load, this measurement collapsed because of a thermal problem described below.

Both Qwen variants occupy the same four GPUs in this configuration, so they cannot run simultaneously. Port numbers can be assigned freely; that does not double the VRAM on a card that is already occupied.

Three Models at Once: Splitting the GPUs into Pairs

With two models, the arithmetic is still simple: four GPUs for Laguna and four for Qwen. A third endpoint forces us to split the second half of the rig into two pairs. TP=2 saves cards, but it also shrinks the jointly available VRAM and, most importantly, the budget for the KV cache.

This was our concurrent configuration:

Model GPUs Topology and KV Context Port Short decode
Laguna S 2.1 INT4 3, 4, 6, 7 TP=4, BF16 KV, utilization 0.96 262144 8005 109.1 tok/s
Qwen3.6-27B-FP8 0, 2 TP=2, BF16 KV, utilization 0.96 200000 8007 45.17 tok/s
Gemma 4 31B FP8-block 1, 5 TP=2, BF16 KV, utilization 0.95 76000 8008 40.6 tok/s

Laguna takes the large group of four, Qwen the 0/2 pair, and Gemma the 1/5 pair. All eight GPUs are occupied, and each model has its own OpenAI-compatible endpoint. The 46.2 tok/s we measured for Qwen on pair 1/5 came from a separate comparison run; 1/5 cannot be used by Qwen and Gemma at the same time.

Context length proved to be the key constraint. A 262,144-token context no longer fit either dense model with TP=2. At startup, vLLM calculated the limits down to the individual token.

Why Qwen27B Stops at 200,000 Tokens with TP=2

A direct TP=2 launch with 262,144 tokens failed after profiling:

262144 required:         8.16 GiB KV per GPU
available:               6.33 GiB KV per GPU
calculated maximum:      202272 tokens

With TP=4, Qwen27B still had a KV capacity of 823,332 tokens. TP=2 distributes the model across two cards rather than four. Each GPU must therefore hold more of the weights while the total VRAM base is cut in half. In this configuration, enough room remains in the KV cache for roughly 202,000 tokens. We set the context directly to 200000 and kept a small reserve instead of working through a series of values that were guaranteed to be too large.

The following command documents the separately measured run on GPU pair 1/5. For the concurrent three-model configuration, we launched Qwen on 0/2 instead.

docker rm -f qwen36_27b_fp8 2>/dev/null || true
docker run -d --name qwen36_27b_fp8 --entrypoint vllm \
  --gpus '"device=1,5"' --ipc host --shm-size 64g \
  -e NVIDIA_DISABLE_REQUIRE=1 -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e NCCL_P2P_DISABLE=1 -e NCCL_IB_DISABLE=1 -e NCCL_CUMEM_ENABLE=0 \
  -e VLLM_ALLREDUCE_USE_SYMM_MEM=0 -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e VLLM_ENABLE_CUDA_COMPATIBILITY=0 \
  -v /bigData/hf-cache/Qwen3.6-27B-FP8:/models/Qwen3.6-27B-FP8 \
  -v /bigData/vllm/data/torch-extensions:/root/.cache/torch_extensions \
  -v /bigData/vllm/data/triton:/root/.triton \
  -v /bigData/vllm/data/cuda-cache:/root/.cache/cuda \
  -v /bigData/vllm/data/vllm-cache:/root/.cache/vllm \
  -p 8007:8000 \
  vllm/vllm-openai:latest \
  serve /models/Qwen3.6-27B-FP8 --served-model-name qwen3.6-27b-fp8 \
  --tensor-parallel-size 2 --max-model-len 200000 --gpu-memory-utilization 0.96 \
  --max-num-seqs 1 --disable-custom-all-reduce --enable-prefix-caching \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --trust-remote-code --host 0.0.0.0 --port 8000
until curl -fsS http://127.0.0.1:8007/health; do sleep 3; done

Why Gemma Stops at 76,000 Tokens

Gemma 4 31B is dense as well and needs a large KV cache. The attempt to use 262,144 tokens failed by an even wider margin:

262144 required:        11.96 GiB KV per GPU
available:               5.16 GiB KV per GPU
calculated maximum:      83936 tokens

An initial launch with 80,000 tokens and gpu-memory-utilization set to 0.96 worked at first. vLLM reported 82,397 KV tokens. During the next launch, however, CUDA graph capture failed on an allocation of just 20 MiB because only about 3 MiB remained free. The calculated maximum was clearly not a reliable operating point.

With 76,000 tokens, utilization set to 0.95, and a reported KV capacity of 77,050 tokens, Gemma started cleanly every time. Three free MiB are not headroom; they are an invitation for the next restart to turn into a debugging session.

docker rm -f gemma4_31b 2>/dev/null || true
docker run -d --name gemma4_31b --entrypoint vllm \
  --gpus '"device=1,5"' --ipc host --shm-size 64g \
  -e NVIDIA_DISABLE_REQUIRE=1 -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e NCCL_P2P_DISABLE=1 -e NCCL_IB_DISABLE=1 -e NCCL_CUMEM_ENABLE=0 \
  -e VLLM_ALLREDUCE_USE_SYMM_MEM=0 -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e VLLM_ENABLE_CUDA_COMPATIBILITY=0 \
  -v /bigData/hf-cache/gemma-4-31B-it-FP8-block:/models/gemma-4-31B-it-FP8-block \
  -v /bigData/vllm/data/torch-extensions:/root/.cache/torch_extensions \
  -v /bigData/vllm/data/triton:/root/.triton \
  -v /bigData/vllm/data/cuda-cache:/root/.cache/cuda \
  -v /bigData/vllm/data/vllm-cache:/root/.cache/vllm \
  -p 8008:8000 \
  vllm/vllm-openai:latest \
  serve /models/gemma-4-31B-it-FP8-block --served-model-name gemma-4-31b-fp8 \
  --tensor-parallel-size 2 --max-model-len 76000 --gpu-memory-utilization 0.95 \
  --max-num-seqs 1 --max-num-batched-tokens 4096 \
  --disable-custom-all-reduce --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4 \
  --trust-remote-code --host 0.0.0.0 --port 8000
until curl -fsS http://127.0.0.1:8008/health; do sleep 3; done

FP8 KV Does Not Work on This Ampere Stack

FP8 for the KV cache would be the obvious way to fit more context into the same amount of memory. Compared with BF16, it cuts the memory requirement in half. This option is useful on Hopper hardware. Our Ampere cards with SM 8.6 rejected every available variant in their own particular way:

  • --kv-cache-dtype fp8 uses E4M3. The selected Triton path does not support this type on SM86 and terminates with fp8e4nv not supported.
  • With --kv-cache-dtype fp8_e5m2, the kernel type would be available, but vLLM blocks the combination with FP8 weight checkpoints.
  • --kv-cache-dtype fp8_inc is intended exclusively for Intel Gaudi or HPU in our installed vLLM version.

That left BF16 KV as the only option for Gemma and Qwen on this stack. The tight KV budget and the TP=2 maxima that fall short of 262k follow directly from this hardware and software constraint.

Pipeline parallelism added another trap. With TP=2, PP=3, and --kv-cache-dtype fp8, Laguna produced fluent nonsense. The doubled attention prefix prevented the k_scale and v_scale values from loading. --calculate-kv-scales did not help and is marked as deprecated in vLLM 0.25.1 anyway. With pipeline parallelism, BF16 KV was therefore required for semantically valid output. A clean startup and grammatical sentences are not enough to prove that a model works when the values in its cache have been scaled incorrectly.

These findings gave us four rules for memory planning on Ampere:

Situation Consequence
FP8 KV on SM86 E4M3, E5M2, and fp8_inc are blocked on this stack; use BF16 KV
PP with FP8 KV Output becomes unusable; BF16 KV is required
TP=2 with 262k 262k does not fit the dense models we tested; Qwen reaches about 202k, Gemma about 76k reliably
A3B MoE with TP=8 Impossible because the partition is not divisible by block_k

The Cost of TP=2 for Qwen27B

Qwen3.6-27B ran in both the four-GPU and two-GPU layouts, making the price of the additional endpoint directly visible:

Layout GPUs Context Short decode Note
TP=4 4 262k about 65 to 67 tok/s, cold KV capacity of 823,332 tokens
TP=2 2 200k 46.2 tok/s Half the VRAM base, leaving room for another model

Moving from TP=4 to TP=2 cost about 30 percent in speed and roughly a quarter of the configurable context, from 262k to 200k. In return, Qwen occupies only two cards and leaves two more for Gemma. In a multi-model setup, free GPU pairs are the real currency.

For a dense model, that price is noticeable. 46 tok/s is usable, but hardly generous. An A3B MoE moves far fewer active weights during decode and would be a more economical tenant for a small GPU pair.

PCIe Width Did Not Decide Decode Performance

The GPU pair with the widest PCIe connections initially seemed like the obvious choice for TP=2. We launched the same Qwen27B container on three pairs and changed only the devices:

GPU pair PCIe connection Short decode
1/5 x8 + x8 46.20 tok/s
6/7 x16 + x16 45.30 tok/s
0/2 x16 + x4 45.17 tok/s

Of all things, the x8+x8 pair was the fastest. The x16+x16 pair trailed by 1.9 percent, and the mixed x16 and x4 connection came in just behind it. The spread across all three pairs was only slightly more than two percent and negligible in practice.

In batch-1 decode, local VRAM bandwidth is the primary bottleneck in this setup. The two TP ranks exchange reductions, but those transfers are small compared with the amount of data each GPU reads from its own memory for every token. PCIe width has a greater impact during the prefill of a long prompt. For decode performance, we could choose pairs based on temperature and topology without preserving x16 connections at all costs.

When Hardware Lies More Convincingly Than Software

SLOT6 Throttled on Memory Temperature, Not GPU Core Temperature

The cold result of 65 to 67 tok/s for Qwen3.6-27B with TP=4 did not hold under sustained load. After a while, throughput dropped to 17 to 20 tok/s. There was no crash, error message, or OOM. The model simply became about three times slower.

Device 0 in SLOT6 was the cause. Its GDDR6X memory, or more specifically its memory junction, became too hot under sustained load. The card reported a SW Thermal Slowdown and reduced its core clock to 300 MHz. At that point, the displayed core temperature was only 76 degrees Celsius. Watching that value alone in nvidia-smi provides no obvious cause for concern. The emergency brake came from the memory.

Tensor parallelism turned one slow card into four slow cards. Every rank waits for the slowest participant, so one throttled GPU drags down the entire TP group.

A 200 W power limit did not help. In fact, it reduced available bandwidth because the core now throttled unnecessarily during normal load as well. Software settings could not fix the underlying cause either. The cards are packed too closely together, SLOT6 receives too little airflow, and the VRAM thermal pads are at their limit. The appropriate repair is new pads and more airflow, not a different --gpu-memory-utilization setting.

We therefore keep Device 0 out of the large Laguna layout and treat it as the part of the system that needs special attention under sustained load. With short chat requests and pauses in between, the card stayed unremarkable and cooled down again between calls. Only continuous decode reproduced the problem reliably.

Our search for the heat source also produced a particularly human misdiagnosis. One GPU remained at full load even though the production model had long been idle according to the dashboard. For quite a while, we blamed the wrong model. In reality, a client had disconnected while vLLM continued generating the request in the background. The orphaned request heated the card while nobody was left to receive its output. Since then, when a GPU stays hot despite supposedly being idle, we check for active and stuck requests first. That is quicker than redesigning the cooling system on impulse.

A Faulty Cable Negotiated Its Own Link Speed

A second fault limited model downloads and transfers between our rigs to approximately 12 MB/s for weeks, the level of a 100 Mbit connection. At that speed, copying a 150 GiB model took a good three and a half hours, even though every component involved was rated for Gigabit Ethernet.

The eno1 interface was not permanently stuck at 100 Mbit. It jumped between 1000 and 100 Mbit, returned to 1000 for a while, and dropped again later. That flapping was precisely what prolonged the diagnosis. A consistently broken link is easy to find. An intermittent one offers a brief burst of hope after a restart, then sends the investigation toward drivers, switch ports, and kernel settings.

In the end, the cable itself was faulty. After replacing it with a Cat6 cable, eno1 remained stable at 1000 Mbit, and the transfer rate rose to 112 MB/s, almost ten times faster. Our long search for a supposed download problem ended with a three-euro cable.

We had seen the pattern before while troubleshooting disappearing GPUs and downgraded PCIe links in "When the GPU Gets Run Over by the Bus". When a system suddenly slows down without a clear error message, we inspect the physical layer first: temperatures, clocks, cables, risers, and link width. Software may be complex, but copper is still part of the stack.

Our Rules for Running Multiple Models

The measurements and debugging sessions led us to six operating rules:

  1. We plan in cards, not total VRAM. The 192 GB consists of eight separate 24 GB regions. Practical layouts are built from GPU pairs and groups of four.
  2. MoE models with few active parameters are especially easy to divide. The A3B model loses less decode performance on a small topology than a dense 27B or 31B model.
  3. We budget for BF16 KV on our Ampere stack. The three FP8 KV variants we tested failed for different reasons, which meant that 262k would not fit with TP=2.
  4. We do not validate pipeline parallelism with FP8 KV by startup alone. Laguna produced fluent nonsense with that combination. BF16 KV was required for the documented PP path.
  5. PCIe width is a secondary criterion for batch-1 decode. The difference between x8+x8, x16+x16, and x16+x4 was only slightly more than two percent in our Qwen27B test. Temperature and a sensible allocation of cards mattered more.
  6. When performance drops for no apparent reason, we inspect the hardware and running requests before the model. Hot GDDR6X memory, an orphaned request, and a flapping network cable produced symptoms that no change to the quantization could fix.

Running three models at once on a used 8×3090 rig comes down to exact device assignments, realistic KV reserves, and the discipline not to assume that a GPU that is free on paper is also thermally healthy. That is how we run three independent OpenAI-compatible endpoints on one system. The rest is accounting, until a cable decides it wants a line item too.

Write a comment

Your e-mail address will not be published. A first comment is approved manually.