redline
Release 1 · v0.3.2

redline

An exactness-first, model-specific batch-1 inference engine for Qwen3.8-27B on one AMD Instinct MI300X. Rust host, hand-written HIP kernels, no PyTorch at runtime.

The honest headline. At 1,024-token prompts, redline's fast mode is the fastest decoder we measured in this setup: 5.56 ms per token, about 20% below vLLM and 17% below SGLang. At 128 tokens it is level with vLLM, about 2.5% behind. The weak point is time to first token: 526 ms at 1,024 tokens against about 145 ms for vLLM.

Speed

Decode time per token

Batch 1, greedy, bf16, 512 output tokens, one MI300X. Two prompt lengths, 128 and 1,024 tokens. Median over repeated runs. Lower is better.

End to end

Completion time for 512 tokens (seconds)

Prefill plus decode, as a user waits for the full answer. Lower is better.

redline's fast mode is 9% faster end to end than vLLM MTP-2 at 1,024 tokens (3.37 s against 3.71 s) and 1.9% slower at 128 tokens. redline MTP-3 (exact) was not timed on the full harness; see the limitations.

Weakness

Time to first token

The first token is the price of prefill. redline's batched prefill is the largest open item in this release.

Engine and modeFirst token, 128 (ms)First token, 1,024 (ms)

Weakness: fast mode reaches the first token in 526 ms at 1,024 tokens, 3.6 times vLLM's 145 ms. The exact modes take about 5.5 s, roughly 5.4 ms per prompt token.

Exactness

Matches to the Hugging Face reference

The reference is the Hugging Face transformers bf16 eager greedy decode, 512 tokens, cut at the first end-of-sequence token. There are 16 cases: 4 categories (code, math, chat, English), 2 prompts each, at 128 and at 1,024 prompt tokens. No engine matches on all 16.

Engine and modeIdentical to HF (of 16)Run-to-run

Run-to-run is the share of repeated runs that gave identical tokens. redline 80 of 80. SGLang NEXTN-3 130 of 160 (the other engines 160 of 160).

Near ties

The four divergent cases are close calls. At the four redline divergences we checked, the Hugging Face top-1 and top-2 logits differ by 0.06 to 0.38. At these magnitudes one bf16 step is 0.125, so the gap is half a step to three steps. An engine's rounding can flip the choice.

Speculation costs zero accuracy

Exact DFlash2 produces the same tokens as plain redline decoding on all 16 timed outputs. The proof tests show the same result bit for bit: hidden states, recurrent state and KV cache.

Fast is not exact

The fast mode uses the MFMA verify path. Its tokens differ from the exact path on 4 of 16 prompts, and it matches HF on 10 of 16. Use exact mode when you need the exact path's output.

The four divergent cases, teacher-forced through HF

CaseIndexHF top-1 vs top-2 logitWhich path matches HF top-1
Long context

Up to 200K tokens

v0.2 was capped at 4,096 positions. v0.3 adds split-sequence flash attention, so each key and value byte is read once per wave, not six times, and the cache is sized with --ctx. These runs use the c8bb90c build, a validation build of the same kernels, one run per length.

ContextPrefill time (s)Prefill (tok/s)Decode (ms/token)

On 26 of 26 prompts (13 at 128 tokens, 13 at 1,024), the long-context path gives identical 64-token outputs to the short path. Decode slows as the context grows, as expected from reading more cache per token.

How it is fast

What is in the engine

Own kernels

The host is Rust. The kernels are hand-written HIP, from the NEKA core. There is no PyTorch at runtime.

Block drafting

DFlash2 drafts a block of 8 or 16 tokens on the hybrid Gated-DeltaNet model. The target model verifies the whole block in one pass.

Two verify paths

VALU verify is the exact path. MFMA verify is the fast path on matrix cores. It reads the weights once for up to 16 rows.

Versioned state

Recurrent-state snapshots let a rejected draft roll the DeltaNet state back exactly, with no re-computation.

On-device rounds

Draft acceptance runs on the GPU, and each round is graph-captured, so the host stays out of the loop.

Batched prefill

Prefill runs 64 to 128 tokens per weight read. The DeltaNet pass no longer writes a snapshot bank for each chunk, and the output stays bit-identical to the 16-token chunks it replaces.

Also measured and dropped: split-K GEMV (no gain at these shapes), tree verify (slower), and the single-row MFMA GEMV (slower than the vector path in v0.2; v0.3 reworked it). Details in the results document.

Limitations

What this release does not do

  • First token is slow. 526 ms in fast mode and about 5.5 s in exact modes at 1,024 tokens.
  • Short prompts gain less. At 128 tokens the fast mode is level with vLLM, not ahead.
  • Fast mode is not exact. It differs from the exact path on 4 of 16 prompts.
  • MTP-3 exact is not timed. It passes the HF gate, 4 of 4 prompts at 64 tokens, at about 9.7 ms per token. It did not run the full harness.
  • One model, one GPU, batch 1. Other models and batches are untested.
  • Long-context prefill is sequential in DeltaNet to keep the bits exact, so at 200K it is the largest first-token cost.
  • Tree verify is refused past 4,096 positions. Chain verify and DFlash2 work at any length.
  • The 512-token exact check did not complete. Its prompt file was missing in the release validation run. The exact-to-VALU identity shown here comes from the timed outputs.
Method

How the numbers were measured

Hardware and model. One AMD Instinct MI300X, ROCm. Qwen3.8-27B in bf16, batch 1, greedy decoding, 512 output tokens for timed runs.

Engines. vLLM 0.29.1 (ROCm) and SGLang 0.5.23 (ROCm), each with 3 warm-up runs and 10 measured runs, timed by a local HTTP streaming client. redline v0.3.2, 1 warm-up and 5 measured runs, timed inside the process; this excludes model load and HTTP, which favours redline by a small margin.

Exactness. Outputs are cut at the first end-of-sequence token (ids 248046 and 248044) before comparison. Mean length after the cut is 134 tokens. The reference is the Hugging Face eager bf16 greedy decode. Teacher-forced logit margins come from the same reference.

Long context. One run per length on the c8bb90c validation build. Prefill is the whole prompt; decode is measured over 64 tokens.

Reproduce. The benchmark scripts, raw outputs and reports are in the repository under results/release1/. Full write-up: docs/RELEASE1.md.

Project
Authors

Maadhav VH (@vhmaadhav) and Sachin (@Sachin0496).