perf(qwen35): add SpecLA factor-capture speculative decoding

[Graffioh]

Current gain over main

Qwen3.6-27B Q4_K_M target, Q8_0 DFlash draft, DDTree budget 22, top-k 4, tau=6, ten HumanEval-style prompts, and 128 generated tokens:

Configuration	Mean decode throughput
Current main	25.14 tok/s
SpecLA	27.19 tok/s

SpecLA gain over current main: +8.2%.

[Graffioh]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

14 issues found across 33 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/common/specla_commit_cuda.h">

<violation number="1" location="server/src/common/specla_commit_cuda.h:25">
P2: The state contract says `[S_v, S_v, H]`, but `specla_commit_fused` uses an `[S_k, S_v, H]` plane. For asymmetric key/value dimensions, callers following this declaration can allocate the wrong size and the kernel can access state memory incorrectly. Document the first dimension as `S_k`.</violation>
</file>

<file name="server/src/common/specla_mode.h">

<violation number="1" location="server/src/common/specla_mode.h:42">
P3: `specla_tree_topk()` parses `DFLASH_SPECLA_TOPK` with `std::atoi`, which uses undefined behavior on integer overflow and gives no way to detect a bad parse. A value like `4abc` silently yields 4, and a value outside `int` range (e.g. `999999999999`) triggers overflow UB. Use `std::strtol` with `endptr` validation so a malformed or out-of-range value falls back to the documented default of 4 instead of silently misconfiguring tree width.</violation>
</file>

<file name="server/deps/llama.cpp/ggml/src/ggml.c">

<violation number="1" location="server/deps/llama.cpp/ggml/src/ggml.c:5888">
P1: The HLD convolution is rejected by CUDA capability checks for normal verify lengths, because `src[0]->ne[1]` is `n_tokens` rather than `d_inner` (for example, a 22-token verify is rejected). The scheduler can therefore run the node on the generic CPU `SSM_CONV` implementation, which ignores `state`, `hld`, and `factor_ptrs` and cannot produce the packed SpecLA result. Update backend capability/dispatch handling for the `op_params[0] == 1` layout and prevent a generic-backend fallback.</violation>

<violation number="2" location="server/deps/llama.cpp/ggml/src/ggml.c:6750">
P2: ggml_gated_delta_net_specla only asserts that q's inner/head dims match v, but not that the token counts agree. The CUDA kernel launch_gated_delta_net_specla takes n_tokens = v->ne[2] and indexes q/k/v with that count (q_t = q + node*sq2 + h_idx*sq1, k, v similarly). If a caller passes q/k with fewer tokens than v, the kernel reads out of bounds silently. Add GGML_ASSERT(q->ne[2] == T && k->ne[2] == T) (and the same token-count check for g/beta) so a mismatch fails loudly instead of producing corrupted results.</violation>

<violation number="3" location="server/deps/llama.cpp/ggml/src/ggml.c:6763">
P3: The n_boundaries op parameter set here (and mirrored as param 1 in ggml_ssm_conv_specla) is never read by the CUDA kernels. launch_gated_delta_net_specla reads op params 4-9 only, and the conv launcher reads params 2-7; both derive boundary layout from the HLDS meta array (meta[4]) instead. The param is dead state that can drift out of sync with the meta-packed value, which is a latent inconsistency. Drop the unused n_boundaries op param (and its ggml_ssm_conv_specla cousin) or actually consume it.</violation>
</file>

<file name="server/src/common/ddtree.cpp">

<violation number="1" location="server/src/common/ddtree.cpp:216">
P3: The new `build_ddtree_conditional` re-implements the same tree-construction logic as the existing `build_ddtree` almost verbatim: the identical chain-seed loop (break on the tau margin, add top-1 node, push siblings), the identical best-first heap-pop loop with the same `q_star - logw > tau_tree` early stop and the same per-node child-map/token/depth wiring, and a byte-for-byte copy of the ancestor-only `visibility` matrix construction (10+ duplicated lines). This doubles the surface that must stay in sync when the pruning or visibility semantics change. Consider factoring the shared best-first/visibility scaffolding into a helper (parameterized by a per-node top-k source) so the two builders differ only in how they obtain each node's distribution.</violation>

<violation number="2" location="server/src/common/ddtree.cpp:275">
P2: When `--ddtree-tau` is negative, the root candidate fails this check and the conditional Qwen path aborts generation as a proposal failure. Reject negative tau or clamp it to zero before applying the window in both builders.</violation>
</file>

<file name="server/src/common/dflash_target.h">

<violation number="1" location="server/src/common/dflash_target.h:71">
P1: When layer-split fast rollback hits a context-fatal CUDA error, this default makes both speculative loops attempt restore+replay on a poisoned CUDA context. Add an override returning `!last_rollback_context_fatal()` (or make the default fail-safe) so unrecoverable failures abort.</violation>
</file>

<file name="server/docs/SPECLA.md">

<violation number="1" location="server/docs/SPECLA.md:174">
P2: The headline gain is not against a "matched" reference. The table's control row ("SpecLA off reference") runs at top-8, while every SpecLA row (including the recommended 27.19 tok/s route) runs at top-4. The +8.2% = 27.19/25.14 therefore mixes the tree-width change (top-8→top-4) in with the SpecLA/HLD effect, so the claim overstates what SpecLA alone contributes. The doc's own "completed HLD, no pruning | top-4" row (26.05 tok/s) shows that width change alone accounts for ~3.6% before HLD is even added. Label the control as top-8 explicitly in the headline comparison, or add a same-top-k control row so the SpecLA effect is isolated.</violation>
</file>

<file name="server/src/qwen35/qwen35_target_graph.cpp">

<violation number="1" location="server/src/qwen35/qwen35_target_graph.cpp:670">
P1: When the factorized SpecLA fallback is used, `specla_commit_accepted()` commits stale bank-0 factors on the first verify and leaves the durable convolution state unchanged. Commit the bank selected by the verify and apply the accepted convolution factors before enabling this fallback.</violation>
</file>

<file name="server/test/test_dflash.cpp">

<violation number="1" location="server/test/test_dflash.cpp:817">
P3: `--ddtree-tau=` accepts any input with no validation, so `--ddtree-tau=garbage` or a negative value becomes a finite tau (atof returns 0.0) and flips the code into the pruned-tree path (`std::isfinite(ddtree_tau)`), producing a degenerate tree for the round. The adjacent `--ddtree-temp` and `--ddtree-budget` flags both clamp invalid values; mirror that here (e.g. reset to the infinity default when the parsed value is not a finite positive number).</violation>
</file>

<file name="server/test/test_delta_net_specla.cpp">

<violation number="1" location="server/test/test_delta_net_specla.cpp:214">
P3: In `run_hld_specla`, the two early returns (`if (!env.alloc_and_run(backend)) return false;` and the graph-compute-failure return) skip the `ggml_backend_buffer_free(fp_buf)` / `ggml_free(fp_ctx)` at the end of the function, leaking the factor-bank buffer and context on error paths. Move the frees ahead of both returns or use a scope guard.</violation>

<violation number="2" location="server/test/test_delta_net_specla.cpp:452">
P3: `CHECK_MSG(env.alloc_and_run(backend), ...)` only prints on allocation failure; it does not abort, so the test then calls `set_f32`/`ggml_backend_tensor_set`/`ggml_backend_graph_compute` on a graph whose tensors were never allocated to a backend buffer, which asserts or crashes instead of reporting a clean failure. Every other `run_*` in this file guards allocation with an early `return false`. Also, on that path `fp_buf` and `fp_ctx` are never freed. Make the allocation failure fatal and free the fp resources.</violation>
</file>

<file name="server/src/qwen35/qwen35_dflash_target.cpp">

<violation number="1" location="server/src/qwen35/qwen35_dflash_target.cpp:609">
P2: The SpecLA bank-rotation / `specla_compact_fused` / `specla_flush_raw_fused` logic is copied nearly verbatim between this file and `server/test/test_dflash.cpp`. `rollback_to_tree` (walked_sibling compaction + `current_bank`/`old_pending_bank` rotation + `pending_count` update), `finish_speculative_state` (bank selection via `specla_pending_bank` + `specla_flush_raw_fused`), and `rollback_to_specla` (bank toggle + count) each have a hand-maintained mirror in the bench harness. These are the two most state-sensitive routines in the PR; any divergence between the production target and the harness (e.g. the `ne` bounds, channel mapping, or bank-toggle logic) silently changes acceptance behavior. Pull the shared commit/flush helpers into a common function (e.g. in `specla_commit_cuda.h/.cu`) and have both call sites invoke it.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: The HLD convolution is rejected by CUDA capability checks for normal verify lengths, because src[0]->ne[1] is n_tokens rather than d_inner (for example, a 22-token verify is rejected). The scheduler can therefore run the node on the generic CPU SSM_CONV implementation, which ignores state, hld, and factor_ptrs and cannot produce the packed SpecLA result. Update backend capability/dispatch handling for the op_params[0] == 1 layout and prevent a generic-backend fallback.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/deps/llama.cpp/ggml/src/ggml.c, line 5888:

<comment>The HLD convolution is rejected by CUDA capability checks for normal verify lengths, because `src[0]->ne[1]` is `n_tokens` rather than `d_inner` (for example, a 22-token verify is rejected). The scheduler can therefore run the node on the generic CPU `SSM_CONV` implementation, which ignores `state`, `hld`, and `factor_ptrs` and cannot produce the packed SpecLA result. Update backend capability/dispatch handling for the `op_params[0] == 1` layout and prevent a generic-backend fallback.</comment>

<file context>
@@ -5844,6 +5844,56 @@ struct ggml_tensor * ggml_ssm_conv_tree(
+    ggml_set_op_params_i32(result, 5, layer);
+    ggml_set_op_params_i32(result, 6, pending_bank);
+    ggml_set_op_params_i32(result, 7, max_parallel_chains);
+    result->op = GGML_OP_SSM_CONV;
+    result->src[0] = x;
+    result->src[1] = c;
</file context>

[cubic-dev-ai]

P1: When layer-split fast rollback hits a context-fatal CUDA error, this default makes both speculative loops attempt restore+replay on a poisoned CUDA context. Add an override returning !last_rollback_context_fatal() (or make the default fail-safe) so unrecoverable failures abort.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/common/dflash_target.h, line 71:

<comment>When layer-split fast rollback hits a context-fatal CUDA error, this default makes both speculative loops attempt restore+replay on a poisoned CUDA context. Add an override returning `!last_rollback_context_fatal()` (or make the default fail-safe) so unrecoverable failures abort.</comment>

<file context>
@@ -62,6 +62,14 @@ struct DFlashTarget {
+
+    // Whether restore+replay remains safe after rollback_to() returns false.
+    // In-place commit implementations override this while active.
+    virtual bool rollback_failure_is_recoverable() const { return true; }
+
     // Roll back recurrent state to position `commit_n` within the last
</file context>

[cubic-dev-ai]

P1: When the factorized SpecLA fallback is used, specla_commit_accepted() commits stale bank-0 factors on the first verify and leaves the durable convolution state unchanged. Commit the bank selected by the verify and apply the accepted convolution factors before enabling this fallback.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen35/qwen35_target_graph.cpp, line 670:

<comment>When the factorized SpecLA fallback is used, `specla_commit_accepted()` commits stale bank-0 factors on the first verify and leaves the durable convolution state unchanged. Commit the bank selected by the verify and apply the accepted convolution factors before enabling this fallback.</comment>

<file context>
@@ -519,9 +662,177 @@ bool migrate_prefill_cache(const TargetWeights & w,
+    // stand-alone per-layer tensors ([.., max_q] with a cross-layer token
+    // stride) so the capture path treats them like any other cache tensor.
+    if (specla) {
+        ggml_tensor * Fk = cache.factor_k_all;
+        ggml_tensor * Fv = cache.factor_v_new_all;
+        ggml_tensor * Fg = cache.factor_g_ps_all;
</file context>

[cubic-dev-ai]

P2: The state contract says [S_v, S_v, H], but specla_commit_fused uses an [S_k, S_v, H] plane. For asymmetric key/value dimensions, callers following this declaration can allocate the wrong size and the kernel can access state memory incorrectly. Document the first dimension as S_k.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/common/specla_commit_cuda.h, line 25:

<comment>The state contract says `[S_v, S_v, H]`, but `specla_commit_fused` uses an `[S_k, S_v, H]` plane. For asymmetric key/value dimensions, callers following this declaration can allocate the wrong size and the kernel can access state memory incorrectly. Document the first dimension as `S_k`.</comment>

<file context>
@@ -0,0 +1,70 @@
+namespace dflash::common {
+
+// ssm_ptrs_dev: DEVICE array of n_delta pointers, one per delta layer's
+//               [S_v, S_v, H] f32 state tensor (ne0 = k-dim rows).
+// fk/fv/fg:     consolidated factor buffers, f32, token axis outermost:
+//               fk [S_k, H, n_delta, T], fv [S_v, H, n_delta, T],
</file context>

[cubic-dev-ai]

P2: When --ddtree-tau is negative, the root candidate fails this check and the conditional Qwen path aborts generation as a proposal failure. Reject negative tau or clamp it to zero before applying the window in both builders.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/common/ddtree.cpp, line 275:

<comment>When `--ddtree-tau` is negative, the root candidate fails this check and the conditional Qwen path aborts generation as a proposal failure. Reject negative tau or clamp it to zero before applying the window in both builders.</comment>

<file context>
@@ -200,6 +213,120 @@ DDTree build_ddtree(const float * top_log_probs,
+        std::vector<int32_t> cur_ids = ids;
+        for (int depth = 1; depth <= L && tree.n_nodes < budget; ++depth) {
+            const float next_logw = cumulative + cur_lp[0];
+            if (q_star - next_logw > tau_tree) break;
+            push_children(parent, depth, cumulative, prefix, cur_lp, cur_ids, 1);
+            const int node = tree.n_nodes + 1;
</file context>

[cubic-dev-ai]

P3: --ddtree-tau= accepts any input with no validation, so --ddtree-tau=garbage or a negative value becomes a finite tau (atof returns 0.0) and flips the code into the pruned-tree path (std::isfinite(ddtree_tau)), producing a degenerate tree for the round. The adjacent --ddtree-temp and --ddtree-budget flags both clamp invalid values; mirror that here (e.g. reset to the infinity default when the parsed value is not a finite positive number).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/test/test_dflash.cpp, line 817:

<comment>`--ddtree-tau=` accepts any input with no validation, so `--ddtree-tau=garbage` or a negative value becomes a finite tau (atof returns 0.0) and flips the code into the pruned-tree path (`std::isfinite(ddtree_tau)`), producing a degenerate tree for the round. The adjacent `--ddtree-temp` and `--ddtree-budget` flags both clamp invalid values; mirror that here (e.g. reset to the infinity default when the parsed value is not a finite positive number).</comment>

<file context>
@@ -809,6 +814,9 @@ int main(int argc, char ** argv) {
             ddtree_budget = std::atoi(argv[i] + 16);
             if (ddtree_budget <= 0) ddtree_budget = 64;
         }
+        else if (std::strncmp(argv[i], "--ddtree-tau=", 13) == 0) {
+            ddtree_tau = (float)std::atof(argv[i] + 13);
+        }
</file context>

Suggested change

	else if (std::strncmp(argv[i], "--ddtree-tau=", 13) == 0) {
	else if (std::strncmp(argv[i], "--ddtree-tau=", 13) == 0) {
	ddtree_tau = (float)std::atof(argv[i] + 13);
	if (!(ddtree_tau > 0.0f)) ddtree_tau = std::numeric_limits<float>::infinity();
	}

[cubic-dev-ai]

P3: In run_hld_specla, the two early returns (if (!env.alloc_and_run(backend)) return false; and the graph-compute-failure return) skip the ggml_backend_buffer_free(fp_buf) / ggml_free(fp_ctx) at the end of the function, leaking the factor-bank buffer and context on error paths. Move the frees ahead of both returns or use a scope guard.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/test/test_delta_net_specla.cpp, line 214:

<comment>In `run_hld_specla`, the two early returns (`if (!env.alloc_and_run(backend)) return false;` and the graph-compute-failure return) skip the `ggml_backend_buffer_free(fp_buf)` / `ggml_free(fp_ctx)` at the end of the function, leaking the factor-bank buffer and context on error paths. Move the frees ahead of both returns or use a scope guard.</comment>

<file context>
@@ -0,0 +1,764 @@
+    ggml_set_output(result);
+    ggml_build_forward_expand(env.gf, result);
+
+    if (!env.alloc_and_run(backend)) return false;
+    set_f32(q, in.q); set_f32(k, in.k); set_f32(v, in.v);
+    set_f32(g, in.g); set_f32(b, in.b); set_f32(s, in.s0);
</file context>

[cubic-dev-ai]

P3: CHECK_MSG(env.alloc_and_run(backend), ...) only prints on allocation failure; it does not abort, so the test then calls set_f32/ggml_backend_tensor_set/ggml_backend_graph_compute on a graph whose tensors were never allocated to a backend buffer, which asserts or crashes instead of reporting a clean failure. Every other run_* in this file guards allocation with an early return false. Also, on that path fp_buf and fp_ctx are never freed. Make the allocation failure fatal and free the fp resources.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/test/test_delta_net_specla.cpp, line 452:

<comment>`CHECK_MSG(env.alloc_and_run(backend), ...)` only prints on allocation failure; it does not abort, so the test then calls `set_f32`/`ggml_backend_tensor_set`/`ggml_backend_graph_compute` on a graph whose tensors were never allocated to a backend buffer, which asserts or crashes instead of reporting a clean failure. Every other `run_*` in this file guards allocation with an early `return false`. Also, on that path `fp_buf` and `fp_ctx` are never freed. Make the allocation failure fatal and free the fp resources.</comment>

<file context>
@@ -0,0 +1,764 @@
+        schedule.max_parallel_chains);
+    ggml_set_output(result);
+    ggml_build_forward_expand(env.gf, result);
+    CHECK_MSG(env.alloc_and_run(backend), "conv HLD graph allocation failed");
+    set_f32(tx, x); set_f32(tw, weight); set_f32(ts, state);
+    ggml_backend_tensor_set(meta, schedule.packed.data(), 0,
</file context>

[cubic-dev-ai]

P3: The new build_ddtree_conditional re-implements the same tree-construction logic as the existing build_ddtree almost verbatim: the identical chain-seed loop (break on the tau margin, add top-1 node, push siblings), the identical best-first heap-pop loop with the same q_star - logw > tau_tree early stop and the same per-node child-map/token/depth wiring, and a byte-for-byte copy of the ancestor-only visibility matrix construction (10+ duplicated lines). This doubles the surface that must stay in sync when the pruning or visibility semantics change. Consider factoring the shared best-first/visibility scaffolding into a helper (parameterized by a per-node top-k source) so the two builders differ only in how they obtain each node's distribution.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/common/ddtree.cpp, line 216:

<comment>The new `build_ddtree_conditional` re-implements the same tree-construction logic as the existing `build_ddtree` almost verbatim: the identical chain-seed loop (break on the tau margin, add top-1 node, push siblings), the identical best-first heap-pop loop with the same `q_star - logw > tau_tree` early stop and the same per-node child-map/token/depth wiring, and a byte-for-byte copy of the ancestor-only `visibility` matrix construction (10+ duplicated lines). This doubles the surface that must stay in sync when the pruning or visibility semantics change. Consider factoring the shared best-first/visibility scaffolding into a helper (parameterized by a per-node top-k source) so the two builders differ only in how they obtain each node's distribution.</comment>

<file context>
@@ -200,6 +213,120 @@ DDTree build_ddtree(const float * top_log_probs,
     return tree;
 }
 
+DDTree build_ddtree_conditional(const DDTreeConditionalTopK & next_topk,
+                                int L, int K, int budget,
+                                bool chain_seed,
</file context>

[cubic-dev-ai]

P3: The n_boundaries op parameter set here (and mirrored as param 1 in ggml_ssm_conv_specla) is never read by the CUDA kernels. launch_gated_delta_net_specla reads op params 4-9 only, and the conv launcher reads params 2-7; both derive boundary layout from the HLDS meta array (meta[4]) instead. The param is dead state that can drift out of sync with the meta-packed value, which is a latent inconsistency. Drop the unused n_boundaries op param (and its ggml_ssm_conv_specla cousin) or actually consume it.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/deps/llama.cpp/ggml/src/ggml.c, line 6763:

<comment>The n_boundaries op parameter set here (and mirrored as param 1 in ggml_ssm_conv_specla) is never read by the CUDA kernels. launch_gated_delta_net_specla reads op params 4-9 only, and the conv launcher reads params 2-7; both derive boundary layout from the HLDS meta array (meta[4]) instead. The param is dead state that can drift out of sync with the meta-packed value, which is a latent inconsistency. Drop the unused n_boundaries op param (and its ggml_ssm_conv_specla cousin) or actually consume it.</comment>

<file context>
@@ -6673,6 +6723,62 @@ struct ggml_tensor * ggml_gated_delta_net_tree_persist(
+        ggml_new_tensor_1d(ctx, GGML_TYPE_F32, packed);
+    ggml_set_op_params_i32(result, 1, 1);
+    ggml_set_op_params_i32(result, 2, 1);
+    ggml_set_op_params_i32(result, 3, n_boundaries);
+    ggml_set_op_params_i32(result, 4, n_chains);
+    ggml_set_op_params_i32(result, 5, n_waves);
</file context>