fix(tasks): preserve current waits after stale generation completion - #567
Open
lizzjin wants to merge 1 commit into
Open
fix(tasks): preserve current waits after stale generation completion#567lizzjin wants to merge 1 commit into
lizzjin wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
当同一个任务已经从 generation N 进入 generation N+1 后,generation N 的旧连接仍可能延迟送达
TurnComplete(cancelled)。数据库状态本身已经由
task_id + run_seq + status的 CAS 保护:旧 generation 的事件不能把新 generation 改成canceled。但TaskEngine::on_turn_complete原先会在执行该数据库 CAS 前,直接按task_id删除整个awaiting集合:这里的关键问题是:
index能区分连接及其 generation,而awaiting只以task_id为 key。因此,旧连接完成时会误删当前 generation 的所有待处理 permission/question 请求。一个确定性的失败过程如下:
permission-a和permission-b两个未处理请求,任务状态为awaiting_input;TurnComplete(cancelled);awaiting集合已经被旧事件清空;permission-a后,track_request看到空集合,错误地把任务恢复为running;permission-b实际仍未处理,agent 仍然阻塞,界面状态与真实运行状态不一致。这与 #546 修复的数据库状态回退问题相关,但发生在数据库 CAS 之前的内存清理阶段。#546 保证旧事件不能取消当前 generation;本 PR 补上“旧事件也不能清除当前 generation 的等待请求”这一层保护。
改动
TaskEngine::on_turn_complete中复用已有的retire_connection(conn_id, task_id),替代对index、awaiting和 delegation children 的三段直接清理。retire_connection会先移除结束连接的索引,并在同一index锁保护下确认该任务是否仍有其他连接:awaiting集合;a_previous_generation_cancel_does_not_clear_current_waits,构造两个 generation 和两个当前等待请求,覆盖完整状态变化。实现保持为单文件的最小修改,没有改变数据库 schema,也没有扩大
awaiting的 key 结构。修复结果
新增测试验证了以下行为:
AwaitingInput;cancelled完成事件不会取消当前 generation,也不会清空它的等待集合;AwaitingInput;Running。修复前,该回归测试稳定失败,实际状态为
Running、期望状态为AwaitingInput;连续复现 5 次均失败。应用本修改后测试通过,因此本 PR 覆盖并解决了 #564 描述的状态损坏路径。测试
以下验证均已完成:
定向回归测试:
cargo test --locked --no-default-features --lib \ a_previous_generation_cancel_does_not_clear_current_waits -- --nocapture结果:
1 passed; 0 failed。服务器端
work_task相关测试集:cargo test --locked --no-default-features --bin codeg-server --lib work_task结果:library target
143 passed; 0 failed;codeg-serverbinary target0 tests。服务器端严格 Clippy:
结果:通过。
桌面 feature 定向回归测试:
cargo test -j 1 --lib --features test-utils \ a_previous_generation_cancel_does_not_clear_current_waits -- --nocapture结果:
1 passed; 0 failed。git diff --check:通过。补充说明:Rust 1.98.0 下运行全仓库
cargo fmt --all -- --check会报告大量本 PR 之前已存在的格式差异,因此没有用它改写无关文件;本 PR 仍保持仅修改src-tauri/src/work_task/engine.rs,且 diff 空白检查通过。测试环境
d6eb97e9da508266053911561f17deae81a2f582v0.28.1-6-gd6eb97e9reposteward-runner-rust:latest(arm64)rustc 1.98.0cargo 1.98.0--no-default-features-j 1避免并发链接内存压力请维护者审查这个连接感知的清理方案,尤其是
retire_connection在旧、新 generation 连接并存时保留任务级等待集合的语义,以及现有index -> awaiting锁顺序是否符合预期。如需将awaiting进一步按(task_id, run_seq)分区,我也可以根据审查意见继续调整。Fixes #564