bench(concurrency): add ragged paired benchmark

[Graffioh]

Dependency

Depends on #595 (packed ragged prefill), transitively on #594.

Scope

Adds benchmark-only tooling under harness/benchmarks/concurrency/ for reproducible Qwen3.6 concurrency measurements:

• A paired ragged workload runner comparing Lucebox K=8, Lucebox K=1, and llama.cpp continuous batching.
• A canonical-suite runner covering HumanEval, GSM8K, Math500, and agent prompts.
• Fresh-server measurements with same-concurrency warmups, disjoint prompts, exact token-usage checks, and reproducibility metadata.

Benchmark setup

run_qwen36_concurrency.sh generates 29 deterministic, disjoint prompts for C=1,4,8,16. Each case starts a fresh server, waits for health, runs a same-concurrency warmup, then records the measurement. Variant order is rotated; the runner rejects prompt reuse and ambient GPU/backend tuning variables.

run_qwen36_canonical_concurrency.sh runs the checked-in prompt suites in complete fixed-width waves: C=1,2,5,10 for ten-case suites and C=1,2,3,6 for the six-case agent suite. It defaults to VARIANTS=ar; blog-ddtree and adaptive-ddtree are optional and require a server build that emits per-response [concurrency-metrics] telemetry.

Each run writes prompts, server logs, warmup and benchmark reports, exact server commands, binary/model/prompt hashes, shared-library hashes, and output summaries under OUT. Use at least five paired repeats for a publication-quality claim; the defaults are intended for screening.

Running the benchmarks

Paired ragged workloads

MODEL=/path/Qwen3.6-27B-Q4_K_M.gguf REPEATS=5 harness/benchmarks/concurrency/run_qwen36_concurrency.sh

Decode-heavy short-prompt comparison:

MODEL=/path/Qwen3.6-27B-Q4_K_M.gguf WORKLOADS=short MAX_TOKENS=256 VARIANTS=luce-k8,llama REPEATS=1 harness/benchmarks/concurrency/run_qwen36_concurrency.sh

Canonical suites

MODEL=/path/Qwen3.6-27B-Q4_K_M.gguf REPEATS=5 harness/benchmarks/concurrency/run_qwen36_canonical_concurrency.sh

Optional DDTree run:

MODEL=/path/Qwen3.6-27B-Q4_K_M.gguf DRAFT_MODEL=/path/dflash-draft-3.6-q8_0.gguf VARIANTS=blog-ddtree harness/benchmarks/concurrency/run_qwen36_canonical_concurrency.sh

Both scripts accept --help. The main options are:

Option	Paired runner	Canonical runner
Model	MODEL required	MODEL required; optional DRAFT_MODEL for DDTree
Server binary	LUCE_SERVER_BIN, LLAMA_SERVER_BIN	SERVER_BIN
Workload/suite selection	WORKLOADS=short,medium,long	SUITES=he-raw,he,gsm,math,agent
Variants	VARIANTS=luce-k8,luce-k1,llama	VARIANTS=ar, blog-ddtree, or adaptive-ddtree
Concurrency	CLIENTS=1,4,8,16	CLIENTS=1,2,5,10 by default; agent uses 1,2,3,6
Repeats	REPEATS=1 by default	REPEATS=1 by default
Output length	MAX_TOKENS=64, WARMUP_TOKENS=8	MAX_TOKENS=128, WARMUP_TOKENS=8
Case selection	—	CASE_LIMIT=<n>; use complete waves
Output directory	OUT=...	OUT=...
Lifecycle controls	SLOTS=16, PORT=18114, HEALTH_TIMEOUT_SECONDS=600, COOLDOWN_SECONDS=3	SLOTS=16, PORT=18116, HEALTH_TIMEOUT_SECONDS=600, COOLDOWN_SECONDS=3

Metrics

Output goodput is exact completion tokens divided by level wall time. It includes queueing, prefill, and decode; it is not decode-only throughput.

Output-window tok/s starts at the earliest observed first output and ends at the final request completion. It removes the initial all-prefill interval but can include staggered prefill, so it is decode-facing rather than pure kernel decode throughput. Request decode tok/s is the median per-request estimate (completion_tokens - 1) / (end - first_output).

Strix Halo screening measurements vs llama.cpp

One fresh screening repeat was run on Strix Halo (Radeon 8060S) after integrating the gfx1151 hipBLASLt default and Q4_K 64-row/four-warp RDNA MMQ specialization from #595. It used Qwen3.6-27B Q4_K_M, Q4_0 K/V cache, greedy sampling, and fixed 64-token outputs. Every row uses a fresh server process and same-concurrency warmup.

Input workload	C	Lucebox output tok/s	llama.cpp output tok/s	Lucebox gain	Max TTFT Luce / llama
Short ≈470 tokens	1	9.46	8.65	+9.4%	1.69 / 1.91 s
Short ≈470 tokens	4	20.49	14.69	+39.5%	5.45 / 8.39 s
Short ≈470 tokens	8	27.01	17.49	+54.4%	11.20 / 17.32 s
Short ≈470 tokens	16	31.32	19.29	+62.4%	22.85 / 34.43 s
Medium ≈1,140 tokens	1	7.44	6.20	+19.9%	3.55 / 4.71 s
Medium ≈1,140 tokens	4	12.40	8.98	+38.1%	13.39 / 19.26 s
Medium ≈1,140 tokens	8	15.09	10.00	+50.9%	26.05 / 38.97 s
Medium ≈1,140 tokens	16	16.03	10.50	+52.7%	53.93 / 78.88 s
Long ≈3,380 tokens	1	4.16	3.31	+25.8%	10.30 / 13.60 s
Long ≈3,380 tokens	4	5.35	3.92	+36.4%	40.57 / 55.66 s
Long ≈3,380 tokens	8	5.79	4.07	+42.1%	80.39 / 112.82 s
Long ≈3,380 tokens	16	5.90	4.14	+42.7%	163.51 / 229.11 s

All 116 measured requests completed successfully with exact 64-token usage counts. These are one-repeat screening measurements, not publication-quality estimates; use at least five paired repeats for a final performance claim.

Decode-heavy Strix Halo screening

A fresh one-repeat paired screen used the same short ragged cohorts with 256 forced output tokens after integrating the gfx1151 hipBLASLt default and Q4_K RDNA MMQ specialization. Generation therefore dominates more of each request.

• Q4_0 K/V cache, greedy sampling, EOS ignored, fresh process and same-concurrency warmup for every measured case
• One economical screening repeat at C1/C4/C8/C16; use five paired repeats for a publication claim
• All 58 measured requests completed with exact 256-token usage

C	Output-window tok/s Luce / llama	Decode-facing gain	End-to-end gain	Request decode tok/s Luce / llama	Max TTFT Luce / llama
1	11.95 / 11.39	+4.9%	+5.5%	11.91 / 11.34	1.74 / 1.97 s
4	33.69 / 24.66	+36.6%	+26.4%	8.42 / 6.91	5.63 / 8.61 s
8	61.69 / 36.43	+69.3%	+47.5%	7.76 / 5.17	11.85 / 17.04 s
16	81.43 / 37.97	+114.4%	+67.5%	6.05 / 3.16	23.49 / 34.43 s

Validation

• Full 64-token Lucebox/llama.cpp screen: 116/116 measured requests completed with exact token usage
• Decode-heavy 256-token Lucebox/llama.cpp screen: 58/58 measured requests completed with exact token usage
• Fresh server process and same-concurrency warmup for every measured row
• Python benchmark harness: 13/13 tests passed
• git diff --check
• HIP build and sequence batch-plan tests: 60/60 passed
• Q4_K MMQ specialization build and focused qwen35 test passed

[cubic-dev-ai]

All reported issues were addressed across 75 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 75 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 86 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Graffioh]

Added canonical workload concurrency coverage in b98d3c3:

• exact raw 10-prompt HumanEval corpus from server/scripts/bench_he.py with identity templating for blog parity
• checked-in HumanEval-chat, GSM8K, Math500, and agent suites with message roles preserved
• full-suite fixed-width waves (10-case: C=1/2/5/10; agent: C=1/2/3/6), with scheduler-retirement barriers so nominal C does not accumulate hidden live requests
• AR vs blog-DDTree variants (Q8_0 draft, SWA=2048, budget=22, fast rollback, adaptive disabled, 128 forced tokens)
• fail-closed response-ID telemetry proof for DDTree steps, target forwards, mean accepted length, and acceptance rate
• consolidated summary plus 25/25 unit tests

Single-repeat Strix Halo screening on the exact raw HumanEval corpus (server binary SHA-256 31664e6f184b7cc37df9e719179d34c2b527ab36e02fd518cf08aef45aa65389):

C	AR goodput	DDTree goodput	DDTree vs AR	DDTree AL	Acceptance
1	11.59	15.17	+30.9%	5.40	33.8%
2	16.66	22.32	+34.0%	5.60	35.0%
5	37.50	25.05	-33.2%	5.59	34.9%
10	56.13	24.48	-56.4%	5.59	34.9%

All rows are 10/10 requests with exactly 128 output tokens. This is a screen, not a publication result; use REPEATS=5 on a clean target PR build for final numbers. The flat acceptance makes the C>=5 crossover an implementation-cost signal rather than a draft-quality failure.

[cubic-dev-ai]

All reported issues were addressed across 12 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Graffioh]

C=3/C=4 follow-up (Strix Halo, Qwen3.6-27B Q4_K_M target + Q8_0/SWA=2048 blog drafter, budget 22, 96 forced output tokens, five repeats):

Suite	C	Cases	AR goodput	Fixed DDTree	Median paired delta	AL	Acceptance
GSM8K	3	9	18.88 tok/s	24.46 tok/s	+29.4%	5.92	37.0%
Math500	3	9	19.03 tok/s	28.71 tok/s	+50.9%	7.00	43.8%
GSM8K	4	8	31.73 tok/s	27.84 tok/s	-12.3%	5.57	34.8%
Math500	4	8	31.81 tok/s	32.92 tok/s	+3.1%	7.10	44.4%

An additional five-repeat C=4 adaptive-DDTree screen measured 30.36 tok/s on GSM8K (-4.4% vs AR) and 31.53 tok/s on Math500 (-1.0% vs AR). It reduces the low-yield GSM penalty, but did not beat AR in either C=4 suite.

Practical result: fixed blog-DDTree is clearly best through C=3 in these reasoning workloads. At C=4, fixed DDTree is worthwhile only for a known high-acceptance Math-like workload; AR is the safer choice for GSM8K or unknown/mixed traffic. Existing HumanEval screening favors AR at C>=4.

All requests completed with exact 96-token accounting and no failures. Output hashes varied across repeats for both AR and DDTree, so this is explicitly reported as a concurrent reproducibility warning rather than hidden. These remain screening measurements because the measured local PR605 binary/worktree was not a clean publication build.

Commit 4527e85 adds CASE_LIMIT for exact full C=3 waves, adaptive-ddtree as an independently selectable variant, robust interrupt cleanup, output-stability reporting, tests, and the measured guidance.

[cubic-dev-ai]

All reported issues were addressed across 5 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 12 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>