2.7× faster local LLM on one AMD GPU: three vLLM fixes, an overnight AI agent, and what we sent upstream

Why we run a model on our own GPU

It worked, but slowly: 38 tokens per second for one request and 96 for four requests in parallel. A different local runtime with a similar model gave about 97 tokens per second on the same card. The hardware was clearly not the limit. The software setup was.

Three problems, one root cause each

When we looked closely, the slowness came from three separate problems.

1. The fast kernel crashed, so we used a slow one. vLLM has a fast kernel for this GPU family, but it crashed at start-up with our model's quantization format. The workaround was to switch it off and use a generic kernel. The cause was a small bug: a data field that the kernel must not receive still reached it. A fix already existed upstream in an open pull request (#52637, by a community contributor), not yet merged.

2. A speed-up feature would not load. The model ships with an extra "draft" layer for MTP, a technique that guesses the next tokens and checks them in one step. vLLM could not load it, because in this model the draft layer is not compressed while the rest is. Again, a fix already existed in an open pull request (#49539, by another contributor).

3. The GPU had no tuned settings. vLLM picks the tile sizes for its mixture-of-experts kernel from tuned files per GPU model. For our GPU and model shape there was no file, so vLLM used a generic default and logged a warning.

The lesson for us: two of the three fixes already existed. Before writing our own fix, we checked open pull requests. That saved work, and it changed our job from "write a fix" to "prove the fix works on our hardware and report the evidence".

How we worked: an AI agent with guardrails

Most of the work was done by an AI coding agent (Claude Code), with one engineer making the decisions.

• Plans first, reviewed by other AI vendors. Each fix started as a written plan. Before any work, the plan went through a review gate where models from other vendors read it and raised objections. The agent had to accept or reject every finding with a reason.
• Predictions before measurements. For every test, the expected result was written down before the run. Before and after runs alternated (A, B, A, B, A, B), and we report medians, not best runs.
• Overnight, unattended. Tuning the GPU took about 11.5 hours at full load. The agent ran it alone overnight, in chunks, so that a stop would lose only one chunk.
• A person approves everything public. Every upstream comment and pull request text was shown to the engineer first and posted only after a "yes".

The hardware surprise

During the first heavy GPU runs, the computer crashed more than once. The Windows log showed 671 "bad block" errors on the system SSD in under 30 minutes. A failing disk was the obvious suspect. The cause we found was a PCIe power-saving setting (Link State Power Management) in our external-GPU setup. With that setting off, we saw zero disk errors during all later runs, including the 11.5-hour tuning.

Because this cost us crashes once, the agent got a simple rule: before each tuning chunk, check the Windows log for new disk errors, and stop at once if there is one.

What we sent upstream

Status as recorded in our notes on 27 October 2026; upstream issues and pull requests were open, not merged, at that point.

• Bug reports with reproductions: vLLM issue #63597 and #63598 — open.
• Test results from our GPU on the two existing fixes, #52637 and #49539 — posted as comments.
• Regression tests for fix 1: PR #63825 — they fail without the fix (5 of 5) and pass with it (5 of 5); open, waiting for a maintainer label.
• The tuned settings file for this GPU: PR #64045 — open, waiting for a maintainer label.

Until the fixes are merged, our server uses them as local patches. The start script applies them only to the exact vLLM version they were tested on, so an upgrade cannot silently mix old patches with new code.

Results

We measured tokens per second at each step on the same hardware:

• Before (workarounds): 37.8 tokens/s for one request; 96.2 tokens/s for four requests.
• Fast kernel fixed: 60.1 tokens/s for one request; 143.4 for four.
• + MTP loaded: 96.8 tokens/s for one request; 236.1 for four.
• + tuned GPU settings: 101.5 tokens/s for one request; 256.0 for four.

That is about 2.7× faster on the same hardware. Accuracy on a standard math benchmark (GSM8K, 200 questions) did not change in a way we could measure: 87.0% with the old kernel and 86.5% with the fast one, and 88.5% both before and after the tuned settings. The tuning alone made the core kernel 7–42% faster, and the server 18% faster with four parallel requests.

What did not go to plan

We think the misses are worth writing down too.

• An output test failed its own rule. With MTP on, 15 of 20 test answers were word-for-word the same as without MTP. We had set the bar at 17 of 20 before the run, so the test failed. All differences were at near-ties, where two next tokens are almost equally likely. We then guessed at two configuration causes and tested them. The guess was wrong: neither changed a single answer. The most likely cause is how MTP itself checks tokens, but we have not proven it, and we say so in the record.
• We over-reported one number by mistake. A comment we posted upstream quoted the fastest of three runs for a latency figure (216 ms) instead of the median (227 ms). We corrected both public texts the same day.
• Our own script stopped healthy servers. A readiness check failed because a container tool cuts long names in its list. It cost two test starts, and the fix was one flag.

What this means for teams running their own models

• Check open pull requests before writing a fix. Popular open-source projects often have the fix already, waiting for someone to test it on real hardware.
• Default settings are not tuned for your hardware. Our GPU had no tuned file; one night of tuning gave up to 18% more throughput.
• An AI agent can do the long, careful parts, such as overnight runs, repeated measurements and evidence collection, if it works with written plans, independent review, safety stops and a person who approves anything public.