Skip to content

fix(ds4): surface the DSpark AR fallback and the fused-verify preconditions - #608

Open
geometric[bot] wants to merge 2 commits into
Luce-Org:mainfrom
GeometricAGI:ds4/dspark-ar-fallback-visibility
Open

fix(ds4): surface the DSpark AR fallback and the fused-verify preconditions#608
geometric[bot] wants to merge 2 commits into
Luce-Org:mainfrom
GeometricAGI:ds4/dspark-ar-fallback-visibility

Conversation

@geometric

@geometric geometric Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

A DSpark server prints DSpark spec-decode ENABLED at startup and then decodes
plain autoregressive for any request the greedy-only verifier cannot serve —
silently. Nothing in the log distinguishes "speculating" from "quietly not".

The sampler case is a live trap in this tree. When a request omits
temperature, the HTTP layer falls back to the model card's sampling defaults,
and share/model_cards/deepseek-v4-flash-0731-rocmfpx.json sets
temperature: 1.0. So a temperature-less benchmark measures AR while the log
still claims speculation.

Measured — same binary, same flags, same prompt:

  • no temperature field: 290 tokens in 289 steps (one step per token, pure AR)
  • "temperature": 0.0: 268 tokens at accept_rate 0.69 (speculating)

This cost a day of chasing a spec-decode "regression" between two builds that
turned out to differ only in whether they shipped that model card. Anyone
benchmarking DSpark without an explicit temperature is measuring the wrong
thing and cannot currently tell.

No behaviour changes — the fallback logic is untouched. This makes it visible,
and documents what a DSpark measurement needs to be comparable.

Impact

No benchmark impact reported.

Implementation

  • fix(ds4): the decode gate logs one line naming which condition routed the request to AR — force_ar / budget stop tokens / sampling — with the effective sampler values (temp, rep_pen, freq_pen, pres_pen).
  • fix(ds4): the fused-verify gate reports every precondition, including verify_hooks, out_logits and expert_runtime. The previous version gated its own warning on two of the pointers whose absence it was meant to report, so a null there stayed invisible.
  • Both diagnostics fire once per process, not per request.
  • docs(ds4): DS4.md gains the fallback conditions, the tell in the log, the acceptance envelope, and an idle-box warning.

Runtime Flags / Configuration

No new runtime flags or configuration switches.

Validation

Check Result
AR vs speculative, same binary and flags, temperature omitted vs 0.0 290 tok / 289 steps (AR) vs 268 tok at accept_rate 0.69 (spec) — the trap reproduces
Clean configure + build of dflash_server from this branch on main pass (CUDA, sm_90)
Fused-verify diagnostic on gfx1151 across the DSpark serving matrix never fired — fused verify was active throughout; the check is preventive and the comment says so

Notes

The acceptance envelope, which is why this is a docs change too.
Speculative throughput is acceptance-bound and acceptance is workload-bound. On
a gfx1151 host, from the same server under the same flags: code/math prompts
(accept 0.83-0.90, ~2.4 committed tokens per verify step) measured 24-27 tok/s,
while open-ended prose (accept 0.64-0.69, ~1.1 committed) measured 16-18 tok/s.
A spec tok/s figure is therefore not comparable across prompt mixes, and DS4.md
now says so next to the numbers.

Measure on an idle box. Strix Halo decode is unified-memory bandwidth
bound: an unrelated co-tenant compile moved an identical request from 18.2 to
3.8 tok/s with acceptance rate and step count unchanged — which reads
exactly like a model or kernel regression. DS4.md now asks for loadavg beside
any tok/s figure.

Scope: no commits to the verifier, no change to when AR is chosen, no change to
the sampler contract. If you would rather the model card not default
temperature to 1.0, that is a separate and arguably better fix — this PR only
makes the current behaviour observable.

Mentioning @Luce-Org/maintainers per CONTRIBUTING now that this is ready for
review.

Review in cubic

DeanoC added 2 commits August 13, 2026 13:43
…itions

A DSpark server prints 'spec-decode ENABLED' at startup and then decodes pure AR
for any request the greedy-only verifier cannot serve, saying nothing. The
sampler case is a live trap in this tree: when a request OMITS temperature the
HTTP layer falls back to the model card's sampling defaults, and
share/model_cards/deepseek-v4-flash-0731-rocmfpx.json sets temperature 1.0 — so a
temperature-less benchmark measures autoregressive decode while the log still
claims speculation.

Measured, same binary and flags, same prompt: no temperature field gives 290
tokens in 289 steps (one step per token, i.e. pure AR); "temperature": 0.0
gives 268 tokens at accept_rate 0.69. This cost a day of chasing a spec-decode
'regression' between two builds that differed only in whether they shipped that
card.

The decode gate now logs one line naming which condition routed the request to
AR — force_ar / budget stop tokens / sampling — with the effective sampler
values. The fused-verify gate reports every precondition, including
verify_hooks, out_logits and expert_runtime; the previous version gated its own
warning on two of the pointers whose absence it was meant to report, so a null
there stayed invisible. Both fire once. The fused-verify one has never fired on
gfx1151 — it is preventive, and the comment says so rather than implying it
caught something.
Three things a DSpark measurement needs and DS4.md did not say.

The fallback conditions and the tell in the log (decode tokens=N steps=N-1
instead of [ds4-spec] gen=... steps=...), so 'spec decode is broken' can be
checked before it is reported.

The acceptance envelope, because speculative throughput is acceptance-bound and
acceptance is workload-bound: on a gfx1151 host, code/math prompts (accept
0.83-0.90, ~2.4 committed tokens per verify step) measured 24-27 tok/s where
open-ended prose (accept 0.64-0.69, ~1.1 committed) measured 16-18 tok/s from the
SAME server under the same flags. A spec tok/s figure is not comparable across
prompt mixes.

And an idle-box warning: Strix Halo decode is unified-memory bandwidth bound, so
an unrelated co-tenant compile moved an identical request from 18.2 to 3.8 tok/s
with acceptance and step count UNCHANGED — which reads exactly like a model or
kernel regression. Record loadavg beside any tok/s figure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant