Skip to content

perf(qwen35): optimize packed concurrent prefill on Strix Halo - #595

Open
Graffioh wants to merge 8 commits into
Luce-Org:mainfrom
Graffioh:codex/packed-ragged-prefill
Open

perf(qwen35): optimize packed concurrent prefill on Strix Halo#595
Graffioh wants to merge 8 commits into
Luce-Org:mainfrom
Graffioh:codex/packed-ragged-prefill

Conversation

@Graffioh

@Graffioh Graffioh commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Depends on #594. Optimizes Qwen3.6 concurrent prefill on Strix Halo while preserving per-request paged KV and recurrent state isolation.

Area Optimization
Packed prefill Fuses up to 8 FIFO prompt slices with live decode in one graph; batches projections, norms, FFN, and output work while running DeltaNet per ragged segment
Paged attention Adds per-query causal positions so ragged prompt and decode rows share the same paged-attention traversal
Commit path Runs the LM head only for prompt tails/decode rows; fixes M-RoPE axis-major staging; adds a 32x32 LDS concat transpose for short history and single-token decode
Adaptive planner Uses a 2K mixed budget for short prompts and 4K at >=768 prompt tokens; keeps a 4K idle budget and 512-token quantum; all thresholds are environment-configurable
gfx1151 kernels Enables hipBLASLt by default for wide concurrent shapes and tunes Q4_K MMQ tiles for narrow prefill/decode
Profiling Adds ROCTX ranges around planning, graph build/upload/compute/readback, prefill, and decode phases

Validation

Check Result
Release HIP/gfx1151 server build Passed
Slot manager 152 checks passed
Batch planner 62 checks passed
Engine contract 14 checks passed
Diff/whitespace check Passed

Review in cubic

@Graffioh
Graffioh marked this pull request as ready for review August 10, 2026 10:41

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 67 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/deps/llama.cpp/ggml/src/ggml-cuda/set-rows.cu
Comment thread server/src/server/scheduler.cpp
Comment thread server/deps/llama.cpp/ggml/include/ggml.h
Comment thread server/src/server/server_main.cpp Outdated
Comment thread server/deps/llama.cpp/ggml/src/ggml-cuda/paged-attn.cu Outdated
Comment thread server/src/common/feature_gate.cpp Outdated
Comment thread server/test/test_paged_attention.cpp
Comment thread server/CMakeLists.txt
Comment thread server/CMakeLists.txt
Comment thread server/src/server/http_server.cpp Outdated
@Graffioh
Graffioh marked this pull request as draft August 10, 2026 11:07
@Graffioh
Graffioh force-pushed the codex/packed-ragged-prefill branch 2 times, most recently from cdba460 to 20ceb72 Compare August 10, 2026 12:32
@Graffioh
Graffioh marked this pull request as ready for review August 10, 2026 12:53

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 73 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/deps/llama.cpp/ggml/src/ggml-cpu/ops.cpp">

<violation number="1" location="server/deps/llama.cpp/ggml/src/ggml-cpu/ops.cpp:5008">
P1: Padded concurrent-decode rows crash on Metal and can write out of bounds on Vulkan because masked `SET_ROWS` semantics exist only in the CPU/CUDA paths. Add the same negative-index skip or reject/fallback behavior to every backend advertising `GGML_OP_SET_ROWS` before using this op-param.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/src/server/http_server.cpp
Comment thread server/deps/llama.cpp/ggml/src/ggml-cpu/ops.cpp
Comment thread server/src/qwen35/concurrency/qwen35_seq_engine.cpp Outdated
Comment thread server/test/test_batched_gdn.cpp Outdated
Comment thread server/tests/test_server_parallel.py
Comment thread server/tests/test_server_parallel.py
Comment thread server/test/test_concat_transpose.cpp Outdated
@Graffioh
Graffioh force-pushed the codex/packed-ragged-prefill branch 5 times, most recently from c98dbc0 to fecfb20 Compare August 11, 2026 09:13
@Graffioh
Graffioh force-pushed the codex/packed-ragged-prefill branch 3 times, most recently from a351ef8 to 045e3e3 Compare August 11, 2026 16:54

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread server/src/qwen35/qwen35_roctx.cpp Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread server/deps/llama.cpp/ggml/src/ggml-cuda/mmq.cuh Outdated
@Graffioh
Graffioh force-pushed the codex/packed-ragged-prefill branch 3 times, most recently from ef959a5 to a90ffe4 Compare August 12, 2026 15:32
@Graffioh
Graffioh force-pushed the codex/packed-ragged-prefill branch from a90ffe4 to e2f1742 Compare August 13, 2026 12:55

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 78 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/src/server/scheduler.cpp
Comment thread server/src/qwen35/qwen35_roctx.cpp Outdated
Comment thread server/src/common/concurrency/seq_engine.h Outdated
Comment thread server/src/server/server_main.cpp Outdated
Comment thread server/src/server/server_main.cpp Outdated
Comment thread server/src/qwen35/concurrency/qwen35_slot_manager.cpp
Comment thread server/src/qwen35/graph_builders.cpp Outdated
Comment thread server/src/qwen35/qwen35_backend.cpp
Comment thread server/deps/llama.cpp/ggml/src/ggml-cuda/mmq.cuh Outdated
Comment thread server/tests/test_server_parallel.py
@Graffioh
Graffioh force-pushed the codex/packed-ragged-prefill branch from e2f1742 to a873953 Compare August 14, 2026 06:41
@Graffioh
Graffioh force-pushed the codex/packed-ragged-prefill branch from a873953 to 7207c3c Compare August 14, 2026 14:05
@Graffioh Graffioh changed the title perf(paged): pack continuous ragged prefill perf(qwen35): optimize packed concurrent prefill on Strix Halo Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant