AMD Radeon AI PRO R9700: Dual-GPU AI Inference Performance

How do two AMD Radeon™ AI PRO R9700 GPUs perform for local LLM inference and image generation?

Introduction

In our Intel Arc™ Pro B70 article, we explored what a VRAM-first, multi-GPU inference workstation looks like when built around Intel’s 32 GB cards. This article asks the same question of AMD’s entry in that fight.

The AMD Radeon™ AI PRO R9700 is the first RDNA 4 professional card positioned squarely at local AI inference. With 32 GB of GDDR6 VRAM per card and 640 GB/s of memory bandwidth, it occupies the same VRAM tier as the Arc Pro B70. With higher bandwidth and AMD’s mature ROCm software stack backing it. At roughly $1,880 (Puget Systems pricing as of July 2026; AMD’s MSRP is $1,299), the R9700 substantially undercuts NVIDIA’s GeForce RTX™ 5090 (~$4,130) while matching it on raw VRAM capacity. Two R9700 cards deliver 64 GB of aggregate VRAM. That’s the same total capacity as two RTX 5090s, at less than half the cost.

In light of that we set out to answer these questions:

• Can two R9700 cards deliver production-quality local LLM inference and image generation?
• And how does AMD’s approach stack up against the four-card B70 configuration we already tested?

We installed two R9700 cards in a Puget Systems workstation and tested single-GPU baselines, multi-GPU serving, generative image workloads, and a 27B-parameter model that requires both cards to run. Along the way, we found that the obvious multi-GPU path (stock vLLM) does not work on these cards at all. We also measured GPU power draw during each benchmark, to calculate real-world cost-per-token and compared it against the full range of today’s cloud APIs, where output pricing spans $1.20/1M tokens (GPT-5.6 Luna) to $30/1M tokens (GPT-5.6 Sol).

Test Setup

Component	Spec
GPUs	2× AMD Radeon™ AI PRO R9700 (RDNA 4 / gfx1201)
VRAM per GPU	32 GB GDDR6
Total VRAM	64 GB
Compute Units	64 per GPU
AI Accelerators	128 per GPU
AI Performance	766 TOPS (INT8) / 1531 TOPS (INT4) per GPU
Memory Bandwidth	640 GB/s per GPU
Infinity Cache	64 MB per GPU
Total Board Power	300W per GPU
Host System	Puget Systems workstation: AMD Ryzen™ Threadripper™ PRO 5995WX (64 cores / 128 threads), 128 GB RAM, ASUS Pro WS WRX80E-SAGE SE
Host OS	Rocky Linux 10.1
Test Environment	Both R9700s passed through (VFIO) to a KVM guest running Ubuntu 26.04, with PCIe peer-to-peer enabled. Benchmarks ran inside the guest, which was allocated 24 vCPUs and 48 GB of RAM.
PCIe	PCIe 5.0 x16 for each GPU (2 slots used in total)

Inference Software: Single-GPU LLM benchmarks used vllm/vllm-openai-rocm:v0.20.2 with unquantized FP16 weights and HIP graphs enabled. Multi-GPU inference used llama.cpp’s ROCm backend, because stock vLLM multi-GPU does not run on these cards. Both Tensor Parallelism and Pipeline Parallelism fail during RCCL collective initialization on RDNA 4 (see What Doesn’t Work), so llama.cpp (which distributes work over direct HIP transfers, no RCCL) is the working multi-GPU path. Image generation used ComfyUI with rocm/pytorch:latest and the Z-Image Turbo model.

Benchmark Tool: We drove every LLM test with NVIDIA GenAI-Perf in --streaming mode. Each prompt was sized at 500 input and 500 output tokens. Standard models ran a 30-second measurement window over 50 prompts, while reasoning models (Qwen3 8B) needed a longer 120-second window over 20 prompts to let the thinking phase finish before the timer mattered. To keep the numbers honest, each run discarded 3 warmup requests so cold-start latency never lands in a reported figure, and we swept three concurrency levels: 1, 4, and 8 simultaneous users.

Power Monitoring: GPU power draw was measured throughout each benchmark by polling the AMD kernel driver’s sysfs hwmon interface (power1_average, reported in microwatts) every 2 seconds. This captures real power consumption under inference load, not just total board power (TBP) ratings.

What Models Fit?

Performance numbers only mean something once you know which models the hardware can actually hold. The ROCm-native vLLM container supports unquantized FP16 and GPTQ/AWQ quantized weights. Our single-GPU benchmarks use full FP16 weights, matching our Intel Arc Pro B70 results. The 27B multi-GPU test uses 4-bit (Q4_K_M) llama.cpp, which, as we’ll see, is currently the only working multi-GPU path on these cards.

Model	Type	Params	FP16 VRAM	Fits 1× R9700? (32 GB)	Fits 2× R9700? (64 GB)
Qwen2.5 3B Instruct	Dense	3B	~6 GB	✅ Yes	✅ Yes
Qwen3 8B	Dense (thinking)	8B	~16 GB	✅ Yes	✅ Yes
Llama 3.1 8B Instruct	Dense	8B	~16 GB	✅ Yes	✅ Yes
DeepSeek R1 Distill 8B	Dense	8B	~16 GB	✅ Yes	✅ Yes
Qwen3.6-27B	Dense	27B	~54 GB	❌ No	✅ Tested (Q4 via llama.cpp)
Qwen3.6-35B-A3B	MoE	35B (3B active)	~70 GB	❌ No	❌ No
Gemma 4 31B	Dense	31B	~62 GB	❌ No	⚠️ Requires bfloat16
Llama 4 Scout	MoE	109B (17B active)	~218 GB	❌ No	❌ No
DeepSeek V4 Flash	MoE	284B (13B active)	~568 GB	❌ No	❌ No

A single R9700 comfortably runs everything up to 8B parameters. The 27B tier, which is currently the sweet spot for capable local inference, requires both cards working together. Models above 64 GB (35B MoE and up) would need additional cards or quantization. For context, the largest open-weight models (DeepSeek V4 Flash at 568 GB) remain firmly out of reach for any workstation-class hardware — those are datacenter territory.

Single-GPU Performance

We started by establishing what a single R9700 can deliver. Each model was tested at concurrency levels 1, 4, and 8 to measure both interactive responsiveness and throughput under load.

How to read these numbers:

• Throughput is the total generation rate in tokens per second
• Time to first token (TTFT) is how long before the model starts replying
• Inter-token latency (ITL) is the gap between streamed tokens
• Average latency is the total time to finish a request

As rules of thumb, a TTFT under about 200 ms feels instant. Since people read at roughly 7 to 13 tokens per second, any ITL below 100 ms outpaces reading speed (under 30 ms feels glass smooth). Throughput is the metric that scales with concurrency: one user rarely consumes a GPU’s full output rate, but several together can.

Summary: Single-GPU Results (Concurrency = 1)

Model	Throughput	TTFT	ITL	Avg Latency
DeepSeek R1 Distill 8B	61.7 tok/s	111 ms	16 ms	10.8 s
Qwen2.5 3B Instruct	53.3 tok/s	60 ms	19 ms	6.1 s
Llama 3.1 8B Instruct	29.1 tok/s	113 ms	34 ms	13.0 s
Qwen3 8B	27.7 tok/s	92 ms	36 ms	18.1 s

All tests: 500 input tokens, 500 output tokens, FP16, HIP graphs enabled, single R9700, max-model-len 16384. Qwen3 8B used a 120-second measurement interval to accommodate its reasoning/thinking token generation.

The performance metrics in the data shows the following:

• DeepSeek R1 8B delivers 61.7 tok/s, the fastest 8B model we tested. Despite being a reasoning-distilled model, it outpaces the Llama family on raw throughput. A smooth 16 ms ITL (Inter-Token Latency) means tokens arrive well ahead of reading speed.
• Qwen2.5 3B reaches 53.3 tok/s with the quickest first token of any model at 60 ms — competitive with cloud API response times. You might expect a 3B model to run several times faster than the 8B models, but it does not: it trails DeepSeek R1 8B (61.7 tok/s) and sits only modestly ahead of Llama 3.1 8B and Qwen3 8B. Single-stream decode is memory-bandwidth-bound on RDNA 4, so once a model fits in VRAM, parameter count matters far less than the R9700’s 640 GB/s bandwidth ceiling, which all of these models push up against.
• Llama 3.1 8B hits 29.1 tok/s, which is usable for interactive chat. The 113 ms TTFT and 34 ms ITL mean responses start quickly and tokens arrive at a comfortable reading pace.
• Qwen3 8B delivers 27.7 tok/s. As a thinking/reasoning model, Qwen3 generates internal reasoning tokens before visible output, which requires an extended measurement window (120 seconds vs. the standard 30) to capture properly; see the note below for more details. Its throughput is the strongest under concurrency of any model tested. Concurrency results for every model are in the expandable tables below.

A note on benchmarking reasoning models: Qwen3 8B initially appeared to produce 0.0 tok/s in our standard 30-second measurement window. The model’s thinking phase, where it generates internal reasoning tokens before visible output, consumed the entire measurement interval, causing GenAI-Perf to report zero completed requests. Extending the measurement window to 120 seconds revealed the true throughput: 27.7 tok/s with a 36 ms ITL. This is a cautionary example for anyone benchmarking thinking models: standard measurement configurations can fundamentally misrepresent their performance. The 27.7 tok/s we report for Qwen3 8B throughout this article uses that corrected 120-second configuration, which we have published alongside our other benchmark configs.

Detailed Single-GPU Benchmark Tables (click to expand)

A note on the concurrency rows: throughput is aggregate, summed across all simultaneous users, while TTFT and ITL are per-request averages across every request completed in the measurement window. As concurrency rises, total system throughput climbs even though each individual request waits a little longer. P99 latency is the worst-case full-request time any single user experienced.

Qwen2.5 3B Instruct (FP16, single GPU)

Concurrency	Throughput	TTFT avg	ITL avg	Avg Latency	P99 Latency
1	53.3 tok/s	60 ms	19 ms	6.1 s	7.5 s
4	183 tok/s	76 ms	21 ms	7.2 s	10.5 s
8	251 tok/s	389 ms	30 ms	10.4 s	18.8 s

Throughput scales from 53 tok/s (single user) to 183 tok/s (4 users). That is 3.4× the single-user throughput, with only a small latency penalty. At 8 users the system reaches 251 tok/s as compute saturation sets in, with TTFT climbing to 389 ms and ITL roughly 1.6× the single-user figure.

Llama 3.1 8B Instruct (FP16, single GPU)

Concurrency	Throughput	TTFT avg	ITL avg	Avg Latency	P99 Latency
1	29.1 tok/s	113 ms	34 ms	13.0 s	17.2 s
4	102 tok/s	162 ms	37 ms	12.2 s	18.5 s
8	153 tok/s	405 ms	47 ms	15.8 s	25.2 s

Llama 3.1 scales from 29.1 tok/s to 102 tok/s at 4 users (3.5× the single-user throughput) while latency stays essentially flat. At 8 concurrent users, throughput reaches 153 tok/s, demonstrating that vLLM’s continuous batching extracts strong parallelism from the R9700’s compute units.

DeepSeek R1 Distill Llama 8B (FP16, single GPU)

Concurrency	Throughput	TTFT avg	ITL avg	Avg Latency	P99 Latency
1	61.7 tok/s	111 ms	16 ms	10.8 s	14.6 s
4	248 tok/s	139 ms	16 ms	13.2 s	15.5 s
8	329 tok/s	429 ms	22 ms	18.4 s	22.4 s

DeepSeek R1 8B is the throughput champion at 61.7 tok/s single-user. At 4 concurrent users, it delivers 248 tok/s with only 28 ms of additional TTFT penalty. The 16 ms ITL means token delivery is exceptionally smooth. At 8 users it reaches 329 tok/s, which is the highest aggregate throughput of any model we tested, and it does so while generating the longest reasoning sequences.

Qwen3 8B (FP16, single GPU) – Thinking Model

Concurrency	Throughput	TTFT avg	ITL avg	Avg Latency	P99 Latency
1	27.7 tok/s	92 ms	36 ms	18.1 s	18.1 s
4	104 tok/s	110 ms	38 ms	19.3 s	19.3 s
8	157 tok/s	383 ms	50 ms	25.4 s	29.0 s

Qwen3 8B is a thinking/reasoning model that generates internal reasoning tokens before visible output. It runs at 27.7 tok/s single-user, scaling to 157 tok/s at 8 concurrent users. That is 5.7× the single-user throughput, and the best concurrency scaling of any model tested — suggesting the R9700’s compute units handle batched reasoning workloads particularly well. The 92 ms TTFT and 36 ms ITL make it fully suitable for interactive use.

Dual-GPU Performance

The R9700’s real value proposition emerges when two cards work together. With 64 GB of aggregate VRAM, models up to 27B parameters — which simply cannot run on a single 32 GB card — become accessible. But how you span two cards matters enormously on RDNA 4, because the obvious path does not work.

Stock vLLM Multi-GPU Doesn’t Run on RDNA 4 (Yet)

Unfortunately, the standard approach to dual GPUs for LLMs fails here: stock vLLM cannot serve a model across two R9700s at all. In fact, both common multi-GPU strategies fail. Tensor Parallelism (TP=2) has long been known to break on PCIe-connected RDNA 4. No XGMI/NVLink exists on these cards, and the RCCL releases in current ROCm vLLM images do not launch collectives reliably on gfx1201. We confirmed that, and then also found that Pipeline Parallelism (PP=2), which used to be the working fallback, now fails the same way. This one is a regression worth understanding. PP=2 ran on vLLM’s older v0 engine, which split a model’s layers sequentially across GPUs with no collective operation at all. The image we tested (v0.20.2) runs the newer v1 engine, which performs an RCCL all-reduce during distributed initialization regardless of whether you request TP or PP. On gfx1201 that collective aborts (HIP failure: 'the operation cannot be performed in the present state') before the server ever loads a model. The v0 engine has since been removed from vLLM, so there is no supported way back to the collective-free PP path. Enabling PCIe peer-to-peer at the platform level, forcing collectives over host sockets, and attempting to pin the old engine all failed. This is a software-maturity gap in the RCCL/RDNA 4 stack (tracked in vLLM #40980 and ROCm rocm-systems #5480), not a configuration problem, and it means the two-card vLLM recipe that worked on older engine builds no longer holds on current images.

There is good news, though: you do not need vLLM for multi-GPU on these cards. llama.cpp’s ROCm backend distributes a model across both GPUs over direct HIP transfers, with no RCCL collectives involved, and it works reliably. Paired with 4-bit GGUF quantization it is also faster than the FP16 path would have been. That is what we benchmark below.

For teams that specifically want vLLM-style TP with int4/FP8, the SGLang inference server with community RDNA 4 patches has a demonstrated same-hardware TP=2 result. We expect the stock RCCL gap to close as fixes land in released ROCm images.

Qwen3.6-27B on Two Cards: llama.cpp with Q4 Quantization

Qwen3.6-27B is a 27-billion-parameter dense model and one of the most capable open-weight models available as of mid-2026. In FP16 it needs ~54 GB and thus cannot fit on a single 32 GB card. Quantized to Q4_K_M it weighs ~17 GB, small enough to fit on one R9700, but splitting it across both cards is what buys the KV-cache headroom that makes concurrent serving practical (more on that below). This is the configuration we would actually deploy for the 27B tier on this hardware.

Qwen3.6-27B Q4_K_M (llama.cpp, 2× R9700, layer split)

Concurrency	Throughput	TTFT avg	ITL avg	Avg Latency	P99 Latency
1	23.4 tok/s	382 ms	41 ms	8.4 s	8.9 s
4	55.7 tok/s	884 ms	64 ms	13.5 s	17.7 s
8	56.0 tok/s	2,668 ms	126 ms	27.6 s	30.3 s

Benchmark: 500 input tokens, 200 output tokens, Q4_K_M GGUF (bartowski/Qwen_Qwen3.6-27B-GGUF), --split-mode layer across 2× R9700, 16384 context, 120-second measurement interval. Measured in the same passthrough-VM environment as the single-GPU results, and both GPUs verified running at full clocks.

Using layer split, we found that at 23.4 tok/s single-user the 27B answers faster than you can read, with a 382 ms TTFT and a 41 ms ITL. It also holds up under load: 55.7 tok/s at 4 users, 56 tok/s at 8, with TTFT stretching to about 2.7 seconds as the batch fills.

The second card roughly doubles usable context. One R9700 loads this model at 32K tokens and fails at 64K; the pair loads 64K and fails at 128K. That headroom, not raw speed, is what the second card is really buying at this tier.

The alternative is llama.cpp’s --split-mode row, which computes every token on both GPUs. It is faster at scale on Qwen2.5-32B (60 tok/s at 8 users) but aborts under concurrent decode on Qwen3.6-27B (see What Doesn’t Work), so layer split is the safe choice.

Q4 is not a concession here. Decode is memory-bandwidth-bound, so reading a quarter as much weight per token makes the quantized model faster than FP16 would be, not just smaller.

Scaling Summary

Model	Precision	Engine	Config	Throughput	Notes
DeepSeek R1 8B	FP16	vLLM	1× R9700	61.7 tok/s	Fastest 8B; 16 ms ITL
Qwen2.5 3B	FP16	vLLM	1× R9700	53.3 tok/s	Quickest TTFT (60 ms)
Llama 3.1 8B	FP16	vLLM	1× R9700	29.1 tok/s	Interactive chat
Qwen3 8B	FP16	vLLM	1× R9700	27.7 tok/s	Thinking model; best concurrency scaling (5.7×)
Qwen3.6-27B	Q4_K_M	llama.cpp	2× R9700 (layer split)	23.4 tok/s	27B tier; needs both cards’ VRAM headroom

All single-user numbers at Concurrency=1. 8B-and-under models run single-GPU FP16 in vLLM, while the 27B tier runs across both cards in llama.cpp because stock vLLM multi-GPU does not work on RDNA 4 (see above).

The results tell a clear story across two tiers:

1. 8B models and smaller run on a single R9700 in FP16, and that is where you want them. Decode is memory-bandwidth-bound, so these fit comfortably in one card’s 32 GB and 640 GB/s. DeepSeek R1 8B leads at 61.7 tok/s, followed by Qwen2.5 3B at 53.3, Llama 3.1 8B at 29.1, and Qwen3 8B at 27.7. The thinking model, Qwen3 8B, has the best concurrency scaling of any model tested (5.7× at 8 users).
2. The 27B tier requires both cards — and, on RDNA 4 today, requires llama.cpp rather than vLLM. Qwen3.6-27B at Q4_K_M runs at 23.4 tok/s single-user across two cards, usable for interactive and production inference. Full FP16 at 27B is out of reach here not because of VRAM (64 GB is plenty) but because stock vLLM multi-GPU does not run on these GPUs.

Cost of Inference: Local vs. Cloud

Raw throughput numbers look great on paper, but the real question every IT lead and engineering manager asks is simple: what does it actually cost to run these models locally, and how does that compare to paying for a cloud API? To answer it with real data rather than rules of thumb, we instrumented our benchmark suite with direct GPU power monitoring, polling the AMD kernel driver’s power1_average sensor every 2 seconds throughout each test run.

Before diving into the numbers, two core realities set the baseline:

This is a cost comparison, not a quality one. The open models we tested do not match frontier APIs on hard reasoning. On the third-party Artificial Analysis Intelligence Index , Qwen3.6-27B in reasoning mode scores 37 against Gemini 3.1 Pro’s 46. Published head-to-head comparisons put even inexpensive cloud tiers ahead of it on several reasoning benchmarks. We are not claiming parity. The economics below matter when an open model is already good enough for the job (summarization, classification, chat, tool use, bulk generation) and per-token meter is the thing you want to stop paying.

Cloud pricing spans a wide range, so which model you compare against decides the answer. Output pricing verified 2026-08-12:

Tier	Model	$/1M output
Frontier	GPT-5.6 Sol	$30.00
Frontier	Claude Opus 5	$25.00
Frontier	Gemini 3.1 Pro (≤200K), GPT-5.6 Terra	$12.00
Mid	Claude Sonnet 5	$10.00
Mid	Gemini 3.6 Flash	$7.50
Mid	Claude Haiku 4.5	$5.00
Economy	Gemini 3.1 Flash-Lite	$1.50
Economy	GPT-5.6 Luna	$1.20

A workstation that beats GPT-5.6 Sol on cost may still lose to GPT-5.6 Luna. Which row you belong on depends on the tier your workload actually needs, so we price against all of them.

Assumptions

Every figure below comes from these five inputs. Swap in your own and the arithmetic follows:

Input	Value used	Basis
Workstation price	$18,775	Puget T142-XL configurator, as-tested specification, August 2026
Electricity	$0.1354/kWh	EIA Electric Power Monthly, US commercial average (May 2026 data)
System overhead	~300 W	CPU package measured at ~120 W under load; RAM, drives, fans and PSU losses estimated. Conservative — see footnote
Hardware life	3 years	Straight-line, no residual value
Cloud prices	See table above	Vendor pricing pages, verified 08-12-2026
Local model	Qwen3.6-27B (Q4_K_M) and the FP16 8B tier	Benchmarked July 2026

Measured GPU Power Under Load

Power is not a single number per model: it climbs with concurrency as batching fills the GPU. We segmented each capture by benchmark phase so every cost figure below uses the power actually drawn at that concurrency level.

Model	Config	c=1	c=4	c=8	Peak
DeepSeek R1 8B	1× R9700	221W	249W	283W	298W
Qwen3 8B	1× R9700	205W	223W	261W	269W
Llama 3.1 8B	1× R9700	205W	227W	259W	278W
Qwen2.5 3B	1× R9700	182W	200W	218W	226W
Qwen3.6-27B (Q4, llama.cpp)	2× R9700	339W (138W + 201W)	—	—	353W

GPU-only power from the AMD kernel driver (sysfs hwmon), averaged over each concurrency phase. Single-GPU rows are the draw of the one active card; the second card sits idle and is not included. The dual-GPU row is both cards summed, with the per-card split shown in parentheses, averaged over a 120-second single-user decode window. Total system wall power adds CPU, RAM, and PSU overhead: we estimate ~300W for this workstation.

None of these workloads come close to maxing-out the R9700’s 300W TBP. Single-card draw ranges from 182W to 283W, because decode is memory-bandwidth-bound and leaves compute units partly idle. Concurrency costs surprisingly little power: going from one user to eight raises draw by 20-28% while multiplying throughput several times over. The two-card 27B is the only configuration that meaningfully raises total draw, at 339W across both GPUs. That split is uneven (138W on the first card, 201W on the second), reflecting how llama.cpp’s layer split distributes work: the card holding the later layers and the bulk of the KV cache stays busier. Even so, each card sits well under its TBP.

Cost Per Million Output Tokens

Electricity alone, using measured per-phase power plus an estimated 300W of system overhead, at the EIA US commercial average of $0.1354/kWh¹:

Model	c=1	c=4	c=8
DeepSeek R1 8B	$0.32	$0.08	$0.07
Qwen2.5 3B	$0.34	$0.10	$0.08
Llama 3.1 8B	$0.65	$0.19	$0.14
Qwen3 8B	$0.69	$0.19	$0.13
Qwen3.6-27B (Q4, 2 GPUs)	$1.03	$0.43	$0.43

Dollars per million output tokens, electricity only. Hardware is handled separately below, and dominates the overall cost analysis.

Serving eight users concurrently costs roughly a fifth as much per token as serving just one, because token throughput multiplies dramatically while power draw creeps up less than 30%. In practice, keeping the hardware actively saturated is what drives down local inference costs.

An 8B model under load draws about $0.07 per million tokens, some seventeen times below the cheapest cloud tier on the market. Electricity is not the only cost of running a workstation, though, and the hardware is where the real money sits.

The Real Driver: Hardware, Not Electricity

A workstation like the one we tested, configured today with a 24-core Threadripper PRO 9965WX, 128 GB of RAM, and two R9700s, comes to $18,775². Amortized over three years, that dwarfs the power bill. Calculating the all-in cost per million output tokens involves two terms³:

At 329 tokens/sec, running this workstation saturated over a 3-year lifespan yields ~185 million tokens for every weekly active hour. The cost calculation boils down to spreading the $18,775 machine cost across active serving hours, plus a flat ~$0.07 per million tokens in electricity.

The figures below track DeepSeek R1 8B on a single card at eight concurrent users: 283 W of GPU draw plus roughly 300 W of system overhead, for 583 W total. At 329 tok/s, every weekly hour of saturated serving pushes out 184.8 million tokens over a three-year hardware lifespan, and electricity works out to a fixed 0.492 kWh per million tokens. That power figure remains constant per token regardless of schedule, because running twice as long burns twice the energy to produce twice the tokens. Spreading the machine’s purchase price across active hours is what moves the needle:

$/kWh	8 h/wk	20 h/wk	40 h/wk	80 h/wk	168 h/wk
$0.10	$12.75	$5.13	$2.59	$1.32	$0.65
$0.1354 (US commercial avg)	$12.77	$5.15	$2.61	$1.34	$0.67
$0.1844 (US residential avg)	$12.79	$5.17	$2.63	$1.36	$0.70
$0.25	$12.82	$5.20	$2.66	$1.39	$0.73

All-in dollars per million output tokens: $18,775 amortized over 3 years plus measured electricity, at 329 tok/s and 583W total system draw.

Moving from the cheapest power in the country to among the most expensive shifts overall cost per token by only 0.6% to 11%. By contrast, moving from 8 hours a week of usage to full-time serving slashes token costs by 19×. If your team runs automated batch processing or continuous agent workflows overnight, local hardware hits break-even almost immediately. If it’s just occasional ad-hoc developer chats, cloud APIs remain the cheaper path.

Break-Even: How Busy Does It Need to Be?

An AI workstation earns its price by displacing tokens you would otherwise rent. Here is how many hours per week of sustained eight-user serving that takes, over a three-year life, against each cloud tier:

Cloud model	$/1M output	DeepSeek R1 8B (c=8)	Qwen3.6-27B Q4 (c=8)
GPT-5.6 Sol	$30.00	3.4 h/week	20.2 h/week
Claude Opus 5	$25.00	4.1 h/week	24.3 h/week
Gemini 3.1 Pro / GPT-5.6 Terra	$12.00	8.5 h/week	51.6 h/week
Claude Sonnet 5	$10.00	10.2 h/week	62.4 h/week
Gemini 3.6 Flash	$7.50	13.7 h/week	84.4 h/week
Claude Haiku 4.5	$5.00	20.6 h/week	130.6 h/week
Gemini 3.1 Flash-Lite	$1.50	70.8 h/week	never
GPT-5.6 Luna	$1.20	89.6 h/week	never

Hours per week of saturated 8-concurrent serving needed to beat each API on cost, at $0.1354/kWh over a 3-year hardware life. “Never” means the workstation cannot reach that price point within a 168-hour week.

Against frontier pricing the bar is low. Three and a half hours a week of real load beats GPT-5.6 Sol, and a single workday a week beats Gemini 3.1 Pro. A team pushing steady work through an 8B model at flagship rates is overpaying by a wide margin.

Against the economy tiers it is a different story. Beating GPT-5.6 Luna at $1.20/1M takes nearly 90 hours a week of saturated serving, and the 27B never gets there: at 56 tok/s it cannot produce tokens fast enough to amortize an $18,775 machine down to $1.20.

The Cards Are the Upgrade, Not the Machine

Those figures charge the entire $18,775 against inference, as if the workstation did nothing else all day. That is the harshest possible accounting, and it is not the situation most buyers are in. A 24-core Threadripper PRO with 128 GB of RAM might have been a workstation you were buying anyway. The AI capability is the pair of R9700s, and specifying them in place of the base configuration’s entry-level card adds over $3,000 to the build.

Priced as an upgrade to a machine you already need, the break-even line drops through the floor:

Cloud model	$/1M output	As a whole machine	As a GPU upgrade
GPT-5.6 Sol	$30.00	3.4 h/week	0.6 h/week
Claude Opus 5	$25.00	4.1 h/week	0.7 h/week
Gemini 3.1 Pro / GPT-5.6 Terra	$12.00	8.5 h/week	1.5 h/week
Gemini 3.6 Flash	$7.50	13.7 h/week	2.4 h/week
Gemini 3.1 Flash-Lite	$1.50	70.8 h/week	12.4 h/week
GPT-5.6 Luna	$1.20	89.6 h/week	15.7 h/week

Calculations based on a $3,283 upgrade price for the pair of R9700s, as of 2026-08-12.

Thirty-six minutes a week of sustained serving covers the cards against GPT-5.6 Sol. At ten hours a week they pay for themselves in two months against flagship pricing and five months against Gemini 3.1 Pro. Even against the cheapest model on the market, fifteen hours a week clears it – while the whole-machine accounting never did.

That is the practical business case for the R9700 in particular. A pair of cards delivers 64 GB of VRAM to handle the 27B tier for a $3,283 upgrade from a basic video card — less than a single high-end workstation GPU from team green. Or, if you are pricing an upgrade for an existing workstation, $3,760 as of August 2026. Unlike cloud API bills that compound indefinitely with every prompt and completion, local hardware is a one-time capital investment that caps your inference costs.

What Never Shows Up in a Token Price

Cost is only part of why engineering teams bring inference in-house. Beyond basic ROI, local hardware solves real day-to-day operational headaches: API rate limits during traffic spikes, models being deprecated or quietly altered mid-project, unexpected pricing updates, and strict compliance or legal barriers around sending sensitive IP to third-party endpoints. For regulated, confidential, or high-volume workflows, these operational guarantees are often the main reason to build locally.

Image Generation: ComfyUI + Z-Image Turbo

Inference isn’t only about language models, so we put the R9700 through a generative image workload as well: ComfyUI running Z-Image Turbo, a distilled diffusion model that renders 1024×1024 images in just four sampling steps.

ComfyUI ran on a current ROCm 7.2 PyTorch build and, once the container had access to the GPU device nodes, saw both R9700s as native HIP devices without needing code changes to ComfyUI or the model. Generation used a single card, with the diffusion model and its Qwen-based text encoder resident in about 18 GB, comfortably inside the card’s 32 GB.

Results

Prompt: “A red fox standing alert in a vibrant autumn forest at golden hour, surrounded by dense fiery-orange and crimson foliage filling the entire frame, a thick carpet of fallen leaves, shafts of warm morning light through the trees, deep depth of field with the whole scene in sharp focus, richly detailed nature photography”

Metric	Value
Iterations	10/10 passed (0 failures)
Cold Start (iter 1)	17.4s (model load from disk + HIP JIT)
Steady State (iter 2–10)	3.6s average
Mean (all 10)	5.0 s
p50	3.5 s
Throughput (steady state)	16.8 images/min
VRAM Used	~18 GB of 32 GB

The steady-state number is what matters: once the first iteration loads the ~18 GB model, the R9700 renders a 1024×1024 image every 3.6 seconds, about 17 images per minute. That leaves roughly 14 GB of headroom on the card for larger models, higher resolutions, or multi-model pipelines, and all 10 runs completed without a single failure or artifact.

Sample Output

These images were generated on the R9700 with Z-Image Turbo using the fox prompt above, each in about 3.6 seconds at steady state:

They were all generated from the same prompt and settings, varying only the random seed. The R9700 produced clean, detailed 1024×1024 results with stable HIP inference throughout.

Comparison: R9700 vs. Arc Pro B70

Both the R9700 and the Arc Pro B70 target the same market: professional AI inference at 32 GB per card. They come from different architectural families, though, with different tradeoffs. We tested both cards on equivalent workloads using our benchmark framework, allowing direct comparison.

Methodology note: Both articles use NVIDIA GenAI-Perf with --streaming, 500 input / 500 output tokens, 50 prompts, at concurrency 1, 4, and 8. Please note that the R9700 numbers here were re-measured with HIP graphs enabled and warmup requests discarded (our corrected methodology) while the B70 numbers are from our previously published article. The figures below are indicative, but a definitive, methodology-matched, head-to-head belongs in a dedicated comparison post. Treat single-digit-percent gaps as ties. One asymmetry is especially worth naming: the B70 figures were captured before we adopted warmup discarding, so they carry some cold-start drag and would likely improve slightly under our current methodology. That bias runs in the B70’s favor here, meaning its lead on these models is a conservative reading rather than an inflated one, though only a matched re-run on both cards can settle it. The original figures are in our Intel Arc Pro B70 article.

Hardware Comparison

GPU Hardware Comparison	AMD Radeon AI PRO R9700	Intel Arc Pro B70
Architecture	RDNA 4 (gfx1201)	Xe2-HPG (Battlemage)
VRAM	32 GB GDDR6	32 GB GDDR6 (ECC)
Memory Bandwidth	640 GB/s	608 GB/s
AI TOPS (INT8)	766 TOPS	367 TOPS
TBP	300W	230W
Price per card (as of July 2026)	~$1,880	~$1,110
Cards Tested	2	4
Total VRAM	64 GB (~$3,760)	128 GB (~$4,450)
Inference Stack	ROCm + vLLM (single-GPU); llama.cpp (multi-GPU)	XPU + vLLM / LLM Scaler
Multi-GPU Method	llama.cpp layer split (stock vLLM multi-GPU non-functional)	Tensor Parallelism (TP)

Single-GPU Throughput (Concurrency = 1)

Model	R9700	B70	Difference
DeepSeek R1 8B	61.7 tok/s	66.9 tok/s	B70 +8%
Llama 3.1 8B	29.1 tok/s	35.4 tok/s	B70 +22%
Qwen2.5 3B	53.3 tok/s	72.9 tok/s	B70 +37%
Qwen3 8B (thinking)	27.7 tok/s	34.7 tok/s	B70 +25%

On these single-GPU numbers the B70 leads across all four models, from an 8% edge on DeepSeek to 37% on the small 3B. Two honest caveats temper that:

• Methodology asymmetry: the R9700 figures were freshly re-measured with our corrected methodology (HIP graphs on, warmup discarded) on the passthrough-VM system described above, while the B70 figures come from its earlier article. We have not yet re-run the B70 under the identical harness, so some of this gap may be measurement rather than silicon. The definitive answer requires a matched head-to-head, which we will run separately.
• Both are memory-bandwidth-bound: with near-identical bandwidth (640 vs. 608 GB/s), neither card should hold a large architectural decode advantage. Where the numbers diverge more than bandwidth would predict — the 3B especially — software maturity (kernel efficiency in each vendor’s stack) is the likely driver, and both stacks are moving quickly.

Treat this as “the B70 is currently at least competitive and often ahead on single-GPU throughput” — with the matched comparison still to come.

Multi-GPU: 27B Dense Model

This is where the software gap between the two platforms matters most, and the comparison is not like-for-like:

27B Dense Model	Radeon AI PRO R9700 (2 cards)	Arc Pro B70 (4 cards)
Working multi-GPU path	llama.cpp layer split	vLLM Tensor Parallelism
27B result	23.4 tok/s (Q4_K_M)	13.1 tok/s (FP16)
Precision	4-bit	16-bit
Total GPU Cost (as of July 2026)	~$3,760 (2 cards)	~$4,450 (4 cards)

The B70 can serve the 27B in full FP16 across four cards using working tensor parallelism, something the R9700 cannot currently do at all, because stock vLLM multi-GPU does not run on RDNA 4. The R9700’s working 27B path is llama.cpp with 4-bit quantization, which lands at 23.4 tok/s. That is faster than the B70’s FP16 result, but at 4-bit rather than 16-bit precision — so this is a difference in approach, not a clean speed win. The practical takeaway for buyers: if full-precision 27B multi-GPU matters to you today, Intel’s stack does it and AMD’s stock stack does not. If 4-bit quantization is acceptable, and it usually is for inference, the R9700 delivers strong 27B throughput on two cards via llama.cpp.

The B70’s 4-card configuration also offers 128 GB of total VRAM — enough for 35B MoE models and beyond — while the R9700 2-card setup is capped at 64 GB. Teams that need models larger than 27B will need either more R9700 cards or the B70’s larger aggregate pool.

Image Generation

Image Generation	AMD Radeon AI PRO R9700	Intel Arc Pro B70
Steady State	3.6s per image	3.9s per image
Throughput (steady)	16.8 img/min	15.4 img/min
VRAM Used	~18 GB	19.3 GB

The R9700 is approximately 8% faster at image generation, consistent with its bandwidth advantage. Both cards handle ComfyUI + Z-Image Turbo without issue.

What Doesn’t Work

Three issues required workarounds during testing:

Stock vLLM multi-GPU (both TP and PP) fails on RDNA 4. vLLM cannot serve a model across two R9700s in either multi-GPU mode. Tensor Parallelism (TP=2) fails during the RCCL all-reduce collective on these PCIe-connected cards. This is a known ROCm issue that our testing shows is deeper than PCIe topology. Even with platform-level GPU peer-to-peer enabled and verified, the RCCL collective kernels in current ROCm vLLM images do not launch on gfx1201, and NCCL_P2P_DISABLE=1 alone does not help. Critically, Pipeline Parallelism (PP=2) now fails the same way: the vLLM image we tested (v0.20.2) runs the v1 engine, which performs an RCCL all-reduce during distributed initialization regardless of TP vs PP, so the collective aborts (HIP failure: 'the operation cannot be performed in the present state') before the model loads. Forcing collectives over host sockets and pinning the older engine both failed. The working paths are outside stock vLLM: llama.cpp (benchmarked above; multi-GPU over direct HIP transfers, no RCCL), or SGLang with RDNA 4 patches for true TP=2 with int4/FP8. We expect AMD to close the RCCL gap as the ROCm stack matures for RDNA 4; until then, plan on llama.cpp for multi-GPU on these cards.

llama.cpp row split is architecture-dependent. --split-mode row ran our full concurrent sweep on Qwen2.5-32B without a single failure, but on Qwen3.6-27B the server aborts under concurrent decode with GGML_ASSERT(!(split && ne02 < ne12)). The row-split matrix-multiplication path does not yet support the broadcast shapes Qwen3.6’s architecture produces in batched decode. Until that is addressed upstream, use --split-mode layer for Qwen3.6-class models (it is what we benchmarked above) and treat row mode as per-architecture: verify it on your model before deploying. We also saw row-split instability under concurrency in earlier testing on a platform without GPU peer-to-peer enabled. If you hit crashes, layer split is the conservative choice.

Container permissions for /dev/kfd. The rocm/pytorch:latest container requires privileged: true and user: root to access /dev/kfd (AMD’s kernel fusion driver device node). Without these, torch.cuda.is_available() returns False even with device passthrough configured. This is a container configuration issue, not a hardware limitation.

How Do Two R9700 GPUs Perform for AI Inference?

The AMD Radeon AI PRO R9700 delivers genuine AI inference capability on RDNA 4 silicon — and the economics make a strong case for local deployment:

• 8B models at production-usable speeds on a single card: DeepSeek R1 8B hits 61.7 tok/s, Qwen2.5 3B reaches 53.3 tok/s, Llama 3.1 8B delivers 29.1 tok/s, and Qwen3 8B — a thinking/reasoning model — runs at 27.7 tok/s with the best concurrency scaling tested (157 tok/s at 8 users). These are honest, full-clock numbers measured with HIP graphs enabled. On single-GPU throughput, the Arc Pro B70 is currently competitive-to-ahead, pending a methodology-matched head-to-head.
• The 27B tier runs well on two cards: Qwen3.6-27B at Q4_K_M serves at 23.4 tok/s single-user across two cards, scaling to 56 tok/s aggregate at 8 concurrent users. This is the practical deployment path for the 27B class on this hardware, and the second card roughly doubles the usable context window.
• A real caveat buyers must know: stock vLLM multi-GPU does not work on RDNA 4 — both tensor and pipeline parallelism fail at RCCL initialization. Multi-GPU on these cards means llama.cpp (or patched SGLang) today, not vLLM. Plan accordingly.
• Reliable image generation: 1024×1024 images in 3.6 seconds steady-state via Z-Image Turbo, with zero failures across 10 runs.
• Cheap to run, but the hardware is what you are paying for: measured GPU draw of 182–283W puts electricity at $0.07–$0.69 per million output tokens depending on concurrency. Hardware amortization dwarfs it. Against frontier APIs the machine clears break-even quickly, at 3.4 hours per week of saturated serving versus GPT-5.6 Sol and 8.5 hours versus Gemini 3.1 Pro. Against economy tiers it is a much harder case: beating GPT-5.6 Luna at $1.20/1M takes nearly 90 hours per week, and the 27B never gets there. Your duty cycle decides this, not your power rate.
• Where it sits against Intel: the B70’s four-card configuration brings double the aggregate VRAM (128 GB) and can run 27B in full FP16 via working tensor parallelism. The R9700 gets to the 27B tier on half the cards and half the slots, but only via quantized llama.cpp. Single-GPU throughput currently favors the B70 on our numbers. The definitive cross-card comparison — same harness, both cards — is a separate head-to-head we plan to publish.

Here is how the three cards compare side by side:

GPU Comparison	AMD Radeon AI PRO R9700	Intel Arc Pro B70	NVIDIA RTX 5090
Architecture	RDNA 4 (gfx1201)	Xe2-HPG (Battlemage)	Blackwell
VRAM	32 GB GDDR6	32 GB GDDR6 (ECC)	32 GB GDDR7
Memory Bandwidth	640 GB/s	608 GB/s	1,792 GB/s
AI TOPS (INT8)	766 TOPS	367 TOPS	3,352 TOPS (FP4 sparse)
TBP	300 W	230 W	575 W
Price per card (configured, July 2026)	~$1,880	~$1,110	~$4,130
Tested Config VRAM	64 GB (2 cards, ~$3,760)	128 GB (4 cards, ~$4,450)	64 GB (2 cards, ~$8,260)
8B FP16 tok/s (single card)	61.7 (DeepSeek R1)	66.9 (DeepSeek R1)	~140–200
27B tok/s (multi-card)	23.4 (Q4, 2 cards, llama.cpp)	13.1 (FP16, TP=4, 4 cards)	N/A (single), not tested (multi)
$/1M tokens (8B, electricity)	$0.29	Not measured	Not measured
Multi-GPU Method	llama.cpp layer split (vLLM multi-GPU broken)	Tensor Parallelism	Tensor Parallelism

The RTX 5090 is roughly 3–4× faster per GPU on decode-bound workloads, driven by nearly 3× the memory bandwidth. But in today’s supply-constrained market, that speed carries a steep premium: two R9700 cards deliver the same aggregate VRAM for less than half the cost of two RTX 5090s. The B70 offers the most VRAM per dollar at 128 GB across four cards, and currently a single-GPU throughput edge, but with more cards and slots to feed.

The caveats are real: stock vLLM multi-GPU does not work on RDNA 4 (use llama.cpp), and container permissions for the image-generation stack need explicit configuration. Once configured, though, single-GPU vLLM and multi-GPU llama.cpp both ran with zero crashes across our complete benchmark suite.

For teams running models up to the 27B class where privacy, cost control, or volume matter, the R9700 dual-card configuration is a compelling option, and volume is the deciding lever. Run the numbers at a fixed $6,262/year, which is this workstation amortized over three years plus power, because you pay for the machine whether it is busy or not. A team generating 5 million output tokens per month is buying that $6,262 to displace $720/year of Gemini 3.1 Pro traffic or $1,800/year of GPT-5.6 Sol. On cost alone that is a clear loss, and at that volume the honest case for local is privacy and control, not payback. Scale to 50 million tokens per month, roughly ten hours a week of saturated serving, and the same $6,299 stands against $7,200/year for Gemini 3.1 Pro, $15,000 for Claude Opus 5 and $18,000 for GPT-5.6 Sol. That is where the machine starts winning on cost, and it wins decisively against flagship traffic. It still loses to GPT-5.6 Luna at $720/year, which is the comparison to make if an economy model does your job.

Teams needing larger VRAM pools (35B+ Mixture of Experts models, for example) should consider adding more R9700 cards or evaluating the 4-card B70 configuration.

Appendix: Setup Guide for Practitioners

Docker Compose Reference (vLLM)

The vllm/vllm-openai-rocm:v0.20.2 image has entrypoint vllm serve, so the container command is <model> <flags> (not python3 -m vllm.entrypoints…, which older docs show). Use this for single-GPU serving, the reliable vLLM path on RDNA 4:

services:
  inference:
    image: vllm/vllm-openai-rocm:v0.20.2
    privileged: true
    shm_size: "32g"
    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri
    environment:
      - VLLM_TARGET_DEVICE=rocm
      - HIP_FORCE_DEV_KERNARG=1
    command:
      - Qwen/Qwen2.5-3B-Instruct
      - --dtype=float16
      - --gpu-memory-utilization=0.95
      - --max-model-len=16384

Leave HIP graphs on. Do not pass --enforce-eager unless you hit a stability issue, as eager mode roughly halves decode throughput on these cards.

Multi-GPU: use llama.cpp, not vLLM

Stock vLLM cannot span two R9700s: both --tensor-parallel-size=2 and --pipeline-parallel-size=2 abort during RCCL collective initialization on gfx1201 (see What Doesn’t Work). The working multi-GPU path is llama.cpp, which distributes work over direct HIP transfers with no RCCL and no special platform configuration:

# llama.cpp: multi-GPU over direct HIP transfers (layer split)
docker run -d --device /dev/kfd --device /dev/dri \
  --security-opt seccomp=unconfined --group-add video --group-add render \
  -p 8000:8000 --entrypoint /app/llama-server \
  ghcr.io/ggml-org/llama.cpp:server-rocm \
  -hf bartowski/Qwen_Qwen3.6-27B-GGUF:Q4_K_M \
  -ngl 99 --split-mode layer -c 16384 --parallel 8 \
  --host 0.0.0.0 --port 8000 --jinja

--split-mode row (both GPUs per token) is faster at high concurrency on architectures that support it, but aborts under concurrent decode on Qwen3.6-class models, so verify against your model first (see What Doesn’t Work). For true vLLM-style TP=2 with int4/FP8, SGLang with RDNA 4 patches is the demonstrated path on this hardware.

ComfyUI Docker Compose (Image Generation)

services:
  comfyui:
    image: rocm/pytorch:latest
    privileged: true
    user: root
    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri
    shm_size: "16g"

---

Tested on a Puget Systems workstation (AMD Threadripper PRO 5995WX, ASUS Pro WS WRX80E-SAGE SE, Rocky Linux 10.1) with 2× AMD Radeon AI PRO R9700 passed through to an Ubuntu 26.04 KVM guest with PCIe peer-to-peer enabled. Benchmarks ran inside the guest with the GPUs verified at full clocks. Single-GPU LLM benchmarks used vllm/vllm-openai-rocm:v0.20.2 with FP16 weights and HIP graphs enabled (not --enforce-eager, which halves decode throughput); every measurement discarded 3 warmup requests. Multi-GPU 27B benchmarks used ghcr.io/ggml-org/llama.cpp:server-rocm with Q4_K_M weights and --split-mode layer; stock vLLM multi-GPU does not run on these cards. Image generation used rocm/pytorch:latest with ComfyUI and Z-Image Turbo. GPU power measured via sysfs hwmon (power1_average). Cloud API pricing as of June 2026, while GPU hardware pricing reflects Puget Systems configured-system pricing as of July 2026.

1. Cost per million output tokens is electricity only: (measured GPU power for that concurrency phase + ~300 W estimated system overhead) × $0.1354/kWh, divided by the measured throughput at that concurrency level. The rate is the US commercial average published by the U.S. Energy Information Administration in Electric Power Monthly, Table 5.3 (May 2026 data). Commercial rates are the relevant basis for a business deployment; the residential average is materially higher at $0.1844/kWh, and the January–May 2026 commercial average is $0.1379/kWh, so a reader on residential power should scale the electricity column up accordingly. System overhead is partly measured: CPU package draw was sampled at ~120 W under load (99.7 W idle) via the kernel’s RAPL interface, and the remainder (RAM, drives, fans, PSU conversion losses) is estimated. The 300 W figure we use is deliberately conservative; the components we can account for total closer to 250 W, so these cost figures if anything overstate local inference. Hardware amortization is handled separately in the next section. ↩︎
2. Our test system uses a prior-generation Threadripper PRO 5995WX, which is no longer sold. Rather than price hardware that is unavailable, this figure reflects a current Puget Threadripper PRO configured comparably (24-core 9965WX, 128 GB RAM, 2× R9700) at August 2026 pricing. Every cost figure in this section is derived from the five inputs listed in the assumptions box above; substitute your own and the arithmetic follows. ↩︎
3. For readers who want the exact algebraic equation: Cost per 1M tokens ($) = [$18,775 ÷ (184.8 × H)] + (0.492 × R), where H is hours per week of saturated 8-user serving over a 3-year hardware lifespan, and R is the electricity rate in $/kWh. ↩︎