feat(ds4) Restore DeepSeek4 HIP backend safely

[howard0su]

Restore the DeepSeek4 full-HIP backend while keeping the active request path stable and preserving batched prefill throughput.

Key changes:

• Re-enable cfg-driven batched prefill chunks for full-HIP and layer-split paths instead of forcing single-token prefill.

• Keep the cached single-token attention graph disabled on HIP because the ggml_set_rows state-update path faults on longer prompts.

• Snapshot existing raw SWA rows before writing the current batched chunk into the ring, avoiding stale/future KV reads while retaining batched execution.

• Reset full-HIP request state between requests, preserve last logits in snapshots, and restore exact-prefix generation without extra prefill.

• Propagate BudgetHook force-close behavior and structured GenerateResult errors through fresh and restored generation paths.

Validation:

• Local: server/build/test_deepseek4_unit and server/build/test_server_unit passed.

• Remote HIP: build-hip/test_deepseek4_unit passed on howardsu@100.126.37.90.

• Remote full-HIP perf with --chunk 512: 525 prompt tokens at 105.21 tok/s, 1045 prompt tokens at 109.44 tok/s.

[cubic-dev-ai]

5 issues found across 22 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/deepseek4/deepseek4_backend.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_backend.cpp:146">
P2: Plain `park`/`unpark` and `park all`/`unpark all` now silently do nothing for this backend because both methods reject selectors the daemon interface defines as valid. Accept empty and `all` as target requests so standard lifecycle commands still release/reload the model.</violation>

<violation number="2" location="server/src/deepseek4/deepseek4_backend.cpp:406">
P2: Restored generation bypasses `req.on_token`, so callback-based streaming and disconnect cancellation do not work on cache hits. Route prefill/decode through `io.with_token_callback(req.on_token)`, matching the other backend paths.</violation>
</file>

<file name="server/src/deepseek4/deepseek4_layer_split_adapter.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_layer_split_adapter.cpp:483">
P2: Local multi-shard forwards invoke undefined behavior when passing HC output to the next shard: the next step memcpy's `hc_state_` onto itself. Preserve the boundary in a separate buffer or make the layer-range copy conditional when pointers match.</violation>
</file>

<file name="server/src/qwen35/qwen35_layer_split_adapter.h">

<violation number="1" location="server/src/qwen35/qwen35_layer_split_adapter.h:63">
P2: Intermediate prefill chunks always compute and transfer full logits even when need_logits is false. The forward function conditionally skips logit work when its logits_out pointer is null, so passing nullptr for intermediate chunks would save the vocab×tokens matmul and GPU→CPU transfer on every chunk except the last.</violation>
</file>

<file name="server/CMakeLists.txt">

<violation number="1" location="server/CMakeLists.txt:438">
P2: CUDA builds now always take the full-logits D2H/CPU top-K fallback because dispatch still checks `DFLASH27B_HAVE_DRAFT_TOPK`. Keep the existing macro name here or update every consumer guard to the renamed CUDA-specific name.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Restored generation bypasses req.on_token, so callback-based streaming and disconnect cancellation do not work on cache hits. Route prefill/decode through io.with_token_callback(req.on_token), matching the other backend paths.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_backend.cpp, line 406:

<comment>Restored generation bypasses `req.on_token`, so callback-based streaming and disconnect cancellation do not work on cache hits. Route prefill/decode through `io.with_token_callback(req.on_token)`, matching the other backend paths.</comment>

<file context>
@@ -715,56 +362,87 @@ void DeepSeek4Backend::snapshot_free(int slot) {
+    }
+    std::vector<int32_t> remaining(req.prompt.begin() + snap_pos, req.prompt.end());
+    const auto t0 = Clock::now();
+    const int committed = remaining.empty() ? snap_pos : do_prefill(remaining, io, snap_pos);
+    if (committed < 0) {
+        result.fail(GenerateErrorCode::PrefillFailed);
</file context>

[cubic-dev-ai]

P2: Plain park/unpark and park all/unpark all now silently do nothing for this backend because both methods reject selectors the daemon interface defines as valid. Accept empty and all as target requests so standard lifecycle commands still release/reload the model.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_backend.cpp, line 146:

<comment>Plain `park`/`unpark` and `park all`/`unpark all` now silently do nothing for this backend because both methods reject selectors the daemon interface defines as valid. Accept empty and `all` as target requests so standard lifecycle commands still release/reload the model.</comment>

<file context>
@@ -309,233 +104,68 @@ DeepSeek4Backend::~DeepSeek4Backend() {
-        free_deepseek4_snapshot(snapshots_[i]);
-    }
-    last_logits_.clear();
+    if (what != "target") return false;
     free_deepseek4_cache(cache_);
-    stream_engine_.destroy();
</file context>

[cubic-dev-ai]

P2: Local multi-shard forwards invoke undefined behavior when passing HC output to the next shard: the next step memcpy's hc_state_ onto itself. Preserve the boundary in a separate buffer or make the layer-range copy conditional when pointers match.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_layer_split_adapter.cpp, line 483:

<comment>Local multi-shard forwards invoke undefined behavior when passing HC output to the next shard: the next step memcpy's `hc_state_` onto itself. Preserve the boundary in a separate buffer or make the layer-range copy conditional when pointers match.</comment>

<file context>
@@ -479,13 +473,14 @@ bool DeepSeek4LayerSplitAdapter::run_forward(
             return false;
         }
         if (timing) add_step_tel(tel_acc, step_tel);
+        shard_input = hc_state_.data();
     }
 
</file context>

[cubic-dev-ai]

P2: Intermediate prefill chunks always compute and transfer full logits even when need_logits is false. The forward function conditionally skips logit work when its logits_out pointer is null, so passing nullptr for intermediate chunks would save the vocab×tokens matmul and GPU→CPU transfer on every chunk except the last.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen35/qwen35_layer_split_adapter.h, line 63:

<comment>Intermediate prefill chunks always compute and transfer full logits even when need_logits is false. The forward function conditionally skips logit work when its logits_out pointer is null, so passing nullptr for intermediate chunks would save the vocab×tokens matmul and GPU→CPU transfer on every chunk except the last.</comment>

<file context>
@@ -59,7 +59,8 @@ class Qwen35LayerSplitAdapter : public LayerSplitAdapter {
     bool prefill(const std::vector<int32_t> & prompt,
-                 int base_pos, int & last_tok) override;
+                 int base_pos, int & last_tok,
+                 bool need_logits = true) override;
     bool decode_ar(int last_tok, int committed, int n_gen,
                    std::vector<int32_t> & out_tokens,
</file context>

[cubic-dev-ai]

P2: CUDA builds now always take the full-logits D2H/CPU top-K fallback because dispatch still checks DFLASH27B_HAVE_DRAFT_TOPK. Keep the existing macro name here or update every consumer guard to the renamed CUDA-specific name.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/CMakeLists.txt, line 438:

<comment>CUDA builds now always take the full-logits D2H/CPU top-K fallback because dispatch still checks `DFLASH27B_HAVE_DRAFT_TOPK`. Keep the existing macro name here or update every consumer guard to the renamed CUDA-specific name.</comment>

<file context>
@@ -447,9 +434,8 @@ elseif(DFLASH27B_GPU_BACKEND STREQUAL "cuda")
-    # name as the HIP branch above (backend-neutral).
-    target_compile_definitions(dflash_common PUBLIC DFLASH27B_HAVE_DRAFT_TOPK=1)
+    # and take the GPU draft top-K path instead of the CPU fallback.
+    target_compile_definitions(dflash_common PUBLIC DFLASH27B_HAVE_DRAFT_TOPK_CUDA=1)
     # GPU port of the sample_logits chain. Compiled in by default; the path is
     # then opted into at runtime via the DFLASH_GPU_SAMPLE env var. Turn the
</file context>

Suggested change

	target_compile_definitions(dflash_common PUBLIC DFLASH27B_HAVE_DRAFT_TOPK_CUDA=1)
	target_compile_definitions(dflash_common PUBLIC DFLASH27B_HAVE_DRAFT_TOPK=1)