How to Benchmark a Local LLM on Strix Halo
You can benchmark your own local LLMs on an AMD Strix Halo machine with one Python script, one official llama.cpp release and a GGUF model. This page builds the script section by section. The script runs the same tests, flags and prompts as my results, so you can put your numbers next to mine.
My machine has an AMD Ryzen AI Max+ 395 processor, a Radeon 8060S GPU and 128 GB of unified memory, with Windows 11. The script expects Windows and the same GPU.
What you need
- Windows with the current AMD GPU driver.
- Python 3.12.
- Free disk space for the model files. A 65B model at Q8_0 needs about 64 GiB.
1. Create the Python environment
Open PowerShell and run these commands. The packages are the Hugging Face download library and the AMD ROCm libraries:
py -3.12 -m venv "$HOME\llm-bench"
& "$HOME\llm-bench\Scripts\Activate.ps1"
python -m pip install --extra-index-url https://repo.amd.com/rocm/whl-multi-arch "rocm[libraries,device-gfx1151]==7.14.1" "huggingface_hub[hf_xet]"
The official llama.cpp ROCm zip for Windows contains the HIP runtime. The zip does not contain the hipBLAS and rocBLAS libraries or the kernels for the gfx1151 GPU. The rocm package from AMD supplies them. My ROCm results use version 7.14.1, which is the version in the command above.
2. Start the script and choose the model
Create a file named strix_bench.py. Paste each Python section from this page into the file, in order. The split-model example in this step is the only exception: use it in place of the first MODEL setting, not in addition to it.
The first section holds the settings. Set MODEL to the model that you want to test. BUILD is the llama.cpp release. Each result page names the build that I used. A newer build can be faster or slower for the same model, so use my build when you want a direct comparison.
"""Strix Halo LLM benchmark. Run: python strix_bench.py [step ...]"""
import csv, hashlib, importlib.metadata, io, json, os, re
import subprocess, sys, sysconfig, time, urllib.request, zipfile
from datetime import datetime
from pathlib import Path
BUILD = "b12360" # a tag from https://github.com/ggml-org/llama.cpp/releases
BUILDS = Path.home() / "llama-builds"
VULKAN = BUILDS / f"{BUILD}-vulkan"
ROCM = BUILDS / f"{BUILD}-rocm"
MODEL = {
"key": "mage-vl-q8",
"repo": "ggml-org/Mage-VL-Cowork-GGUF",
"include": ["Mage-VL-Cowork-Q8_0.gguf"],
"file": "Mage-VL-Cowork-Q8_0.gguf",
}
MODEL_DIR = Path.home() / "models" / MODEL["key"]
MODEL_FILE = MODEL_DIR / MODEL["file"]
RESULTS = Path("results.csv")
PORT = 8080
def save(step, metric, value):
"""Add one value to results.csv at once, so a crash later in the run loses nothing."""
new = not RESULTS.exists()
with RESULTS.open("a", newline="", encoding="utf-8") as f:
w = csv.writer(f)
if new:
w.writerow(["model", "step", "metric", "value", "time"])
w.writerow([MODEL["key"], step, metric, value, datetime.now().isoformat(timespec="seconds")])
print(f"{step}: {metric} = {value}")
A large model can be a split GGUF, with file names that end in -00001-of-00002.gguf. For a split model, put a pattern in include that matches every part, and put the first part in file:
MODEL = {
"key": "ripple-q8",
"repo": "AtomicChat/Ripple-65B-Preview-GGUF",
"include": ["Ripple-65B-Preview-Q8_0*.gguf"],
"file": "Ripple-65B-Preview-Q8_0-00001-of-00002.gguf",
}
3. Install the llama.cpp build
The setup step downloads the Windows Vulkan and ROCm zips for BUILD from GitHub. The step checks each zip against the sha256 digest that GitHub publishes, then unpacks the zip.
def setup():
req = urllib.request.Request(f"https://api.github.com/repos/ggml-org/llama.cpp/releases/tags/{BUILD}",
headers={"User-Agent": "strix-bench"})
assets = json.load(urllib.request.urlopen(req, timeout=60))["assets"]
BUILDS.mkdir(parents=True, exist_ok=True)
for backend, dest, pattern in [("vulkan", VULKAN, r"llama-b\d+-bin-win-vulkan-x64\.zip"),
("rocm", ROCM, r"llama-b\d+-bin-win-(rocm|hip)[^ ]*-x64\.zip")]:
asset = next(a for a in assets if re.fullmatch(pattern, a["name"]))
zip_path = BUILDS / asset["name"]
if not zip_path.exists():
print(f"downloading {asset['name']}")
urllib.request.urlretrieve(asset["browser_download_url"], zip_path)
want = (asset.get("digest") or "").removeprefix("sha256:")
if want and hashlib.sha256(zip_path.read_bytes()).hexdigest() != want:
sys.exit(f"sha256 mismatch for {asset['name']}: delete the zip and run setup again")
with zipfile.ZipFile(zip_path) as z:
z.extractall(dest)
save("setup", f"{backend}_zip", asset["name"])
save("setup", "build", BUILD)
4. Check the GPU and record the driver
The ROCm build needs the AMD library folders on PATH. The rocm_env function adds those folders for ROCm commands only, so the Vulkan commands use the normal PATH.
The check step asks each build to list its devices. Both lists must include the Radeon 8060S. If the ROCm list does not include the GPU, the ROCm build uses the CPU instead, and a large model then needs hours. The step also records the GPU driver version, the AMD Software version and the ROCm library version. Each result page lists the same three values for my machine, so you can see where your setup differs from mine.
SITE = Path(sysconfig.get_paths()["purelib"])
ROCM_LIBS = [SITE / "_rocm_sdk_libraries" / "bin", SITE / "_rocm_sdk_core" / "bin"]
def rocm_env():
env = os.environ.copy()
env["PATH"] = os.pathsep.join([str(p) for p in ROCM_LIBS] + [env["PATH"]])
return env
def powershell(command):
return subprocess.run(["powershell", "-NoProfile", "-Command", command], capture_output=True, text=True).stdout.strip()
def check():
for backend, build, env in [("vulkan", VULKAN, None), ("rocm", ROCM, rocm_env())]:
p = subprocess.run([str(build / "llama-bench.exe"), "--list-devices"], capture_output=True, env=env)
devices = (p.stdout + p.stderr).decode("utf-8", "replace")
print(devices)
save("check", f"{backend}_gpu_seen", "8060S" in devices)
gpu = ('$g = Get-CimInstance Win32_VideoController | Where-Object Name -like "*8060S*" | Select-Object -First 1;'
'"$($g.DriverVersion) $($g.DriverDate.ToString(\'yyyy-MM-dd\'))"')
save("check", "gpu_driver", powershell(gpu))
save("check", "amd_software", powershell('(Get-ItemProperty "HKLM:\\SOFTWARE\\AMD\\CN").CNVersion'))
save("check", "rocm_libs", importlib.metadata.version("rocm-sdk-libraries"))
5. Download the model
The download step downloads the files that match include. For a split GGUF, the step adds up the size of every part.
def download():
from huggingface_hub import snapshot_download
snapshot_download(repo_id=MODEL["repo"], allow_patterns=MODEL["include"], local_dir=MODEL_DIR)
split = re.match(r"(.*)-00001-of-(\d{5})\.gguf$", MODEL_FILE.name)
parts = ([MODEL_FILE.with_name(f"{split[1]}-{i:05d}-of-{split[2]}.gguf") for i in range(1, int(split[2]) + 1)]
if split else [MODEL_FILE])
save("download", "file_gib", f"{sum(p.stat().st_size for p in parts) / 2**30:.2f}")
6. Measure prompt and generation speed
The llama_bench function runs llama-bench with CSV output and saves the mean speed and the standard deviation for each test:
pp4096is prompt processing: how fast the model reads a 4096-token prompt.tg512is text generation: how fast the model writes 512 new tokens.
The bench step runs the speed test on the Vulkan build. The backend step runs the same test on the ROCm build.
BENCH_ARGS = ["-p", "4096", "-n", "512", "-b", "512", "-ub", "512", "-t", "16", "-ngl", "99",
"-sm", "layer", "--flash-attn", "1", "-r", "3"]
def llama_bench(build, args, step, prefix="", env=None):
p = subprocess.run([str(build / "llama-bench.exe"), "-m", str(MODEL_FILE), *args, "-o", "csv"],
capture_output=True, env=env)
if p.returncode:
sys.exit(p.stderr.decode("utf-8", "replace"))
out = p.stdout.decode("utf-8", "replace")
for r in csv.DictReader(io.StringIO(out[out.find("build_commit"):])):
name = f"pp{r['n_prompt']}" if r["n_gen"] == "0" else f"tg{r['n_gen']}"
if int(r.get("n_depth") or 0):
name += f"_d{r['n_depth']}"
save(step, f"{prefix}{name}_tps", f"{float(r['avg_ts']):.2f}")
save(step, f"{prefix}{name}_sd", f"{float(r['stddev_ts']):.2f}")
def bench():
llama_bench(VULKAN, BENCH_ARGS, "bench")
def backend():
llama_bench(ROCM, BENCH_ARGS, "backend", "rocm_", rocm_env())
-ngl 99 puts every layer on the GPU. -t 16 matches the 16 CPU cores of the Ryzen AI Max+ 395. -sm layer is the llama.cpp default split mode. With one GPU, the split mode has no effect. The Vulkan backend refuses -sm row with "device Vulkan0 does not support split buffers".
7. Measure speed at longer context
Generation gets slower when the context already holds many tokens. The context step measures the speed with 0, 8192 and 32768 tokens already in the context. The step then repeats the 8192 test with a q8 KV cache, which uses less memory.
def context():
depth = ["-p", "512", "-n", "128", "-ngl", "99", "--flash-attn", "1", "-r", "2"]
llama_bench(VULKAN, depth + ["-d", "0,8192,32768"], "context", "depth_")
llama_bench(VULKAN, depth + ["-d", "8192", "-ctk", "q8_0", "-ctv", "q8_0"], "context", "kvq8_")
8. Measure load time, GPU memory and server speed
llama-server runs the model behind a local chat API on 127.0.0.1. The server step records three values:
- The load time, from server start until the
/healthendpoint returnsok. - The GPU memory that the model uses.
- The prompt and reply speed for a short question, a 4,000-token document and a coding task.
Windows reports GPU memory only for the whole system. The gpu_mem_gib function adds the "GPU Adapter Memory" counters for dedicated and shared memory. The step reads the counters before the server starts and again after the model loads. The difference is the memory that the model uses.
def gpu_mem_gib():
total = powershell('(Get-Counter "\\GPU Adapter Memory(*)\\Dedicated Usage","\\GPU Adapter Memory(*)\\Shared Usage")'
'.CounterSamples | Measure-Object CookedValue -Sum | ForEach-Object Sum')
return float(total) / 2**30
def http(path, body=None, timeout=1800):
data = json.dumps(body).encode("utf-8") if body is not None else None
req = urllib.request.Request(f"http://127.0.0.1:{PORT}{path}", data=data,
headers={"Content-Type": "application/json; charset=utf-8"})
with urllib.request.urlopen(req, timeout=timeout) as r:
return json.load(r)
def start_server(extra=()):
t0 = time.monotonic()
p = subprocess.Popen([str(VULKAN / "llama-server.exe"), "-m", str(MODEL_FILE), "-ngl", "99", "-c", "16384",
"--flash-attn", "on", "--host", "127.0.0.1", "--port", str(PORT), *extra],
stdout=subprocess.DEVNULL, stderr=open("server.log", "ab"))
while p.poll() is None:
try:
if http("/health", timeout=2).get("status") == "ok":
return p, time.monotonic() - t0
except Exception:
time.sleep(2)
sys.exit("llama-server stopped while it loaded the model: see server.log")
def ask(content, max_tokens=512):
return http("/v1/chat/completions", {"model": "local", "max_tokens": max_tokens, "temperature": 0,
"messages": [{"role": "user", "content": content}]})
DOC = "\n".join(f"Section {i}: The kitchen inventory lists flour, sugar, eggs, butter and milk, "
f"with storage notes for item number {i}." for i in range(1, 221))
CODE_TASK = ("Write a Python function parse_measure(s) that converts strings like '2 1/2 tbsp' or '1 stakan' "
"(a Russian glass, 250 ml) into millilitres. Include three pytest tests.")
PROMPTS = {
"chat": "Explain in three sentences why memory bandwidth limits token generation on an integrated GPU.",
"doc4k": f"Summarize in five bullets:\n{DOC}",
"code": CODE_TASK,
}
def server():
before = gpu_mem_gib()
p, load_s = start_server()
try:
save("server", "load_time_s", f"{load_s:.1f}")
save("server", "gpu_mem_gib_used", f"{gpu_mem_gib() - before:.1f}")
for name, text in PROMPTS.items():
t = ask(text)["timings"]
save("server", f"{name}_prompt_tps", f"{t['prompt_per_second']:.1f}")
save("server", f"{name}_gen_tps", f"{t['predicted_per_second']:.1f}")
save("server", f"{name}_prompt_ms", f"{t['prompt_ms']:.0f}")
finally:
p.kill()
p.wait()
The chat prompt is short, so its prompt speed changes a lot from run to run. Use the chat reply speed, and treat its prompt speed as approximate.
9. Run the benchmark
The last section of the script runs the steps that you name, or every step in order:
STEPS = {"setup": setup, "check": check, "download": download, "bench": bench, "context": context,
"server": server, "backend": backend}
if __name__ == "__main__":
sys.stdout.reconfigure(encoding="utf-8")
for step in sys.argv[1:] or STEPS:
STEPS[step]()
Close other apps that use the GPU. Then run the script from the environment of step 1:
python strix_bench.py setup check
python strix_bench.py download bench context server backend
The first command installs the build and checks the GPU. Make sure that both builds list the Radeon 8060S before you run the second command. To test another model, change MODEL and run the second command again. Every value goes into results.csv with the step, the metric and the time.
10. Compare your numbers with mine
I publish the numbers that I get from this script on my machine. Each model has its own page in this section, after this page. Each model page lists the test date, my driver and the llama.cpp build for that model.
Use my numbers as a reference for your own runs. Small differences are normal. A large difference for the same model and quantization can point to a problem in your setup, for example a model that runs partly on the CPU.