jobs: background shell stuck running after process death; stale UI item cannot be cleared #56804

Activity

1. opencode-agent commented on Oct 18, 2026

   

   opencode-agentBot

   on Oct 18, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the detailed report. I reproduced this on 2.0.30, and it still happens on the latest dev (da9d436).

   Steps (Linux, simulated model):

   1. Start a background shell with the shell tool (sleep 600, background: true).
   2. Tear down the location's services through DELETE /api/debug/location. That's the same path the 60-minute idle eviction uses ([Bug] Location inactivity eviction SIGTERMs live background shells after 60 minutes #51828).

   Result:

   • The process is killed.
   • GET /api/shell/{id} returns 404 ShellNotFoundError and shell.list is empty.
   • DELETE /api/shell/<any unknown id> returns success (204).
   • No completion notification ever reaches the session, so the background job is never reported finished. That matches the state you describe.

   The recording and script below show this.

   Two things I couldn't confirm:

   • Killing the process from outside: on Linux, killing the background process with kill -9 is handled correctly; the shell shows as exited and the "Shell finished" notification arrives. So in your case the shell record most likely disappeared some other way, such as the location being torn down. I can't test a Windows-specific kill path here.
   • The stuck job.background record: I couldn't inspect it. From the code, a job that ends this way is marked cancelled, and cancelled jobs aren't written back to that record, so it would stay running. That matches what you saw.

   Related: #51828 (idle eviction kills background shells) and #56729 / #56780 (background shell completion notifications lost when the shell plugin is rebuilt).

   recording.mp4

   

   Reproduction script (drive-evict.ts)

   import { Effect } from "effect"
   import { execSync } from "node:child_process"
   import { readFileSync } from "node:fs"
   import { Llm, OpenCodeDriver } from "opencode-drive"
   import { Database } from "bun:sqlite"
   import { dirname, join } from "node:path"
   
   // Background shell via the shell tool, then tear down the location's services
   // (same path as the 60-minute idle eviction, triggered via DELETE /api/debug/location).
   // The shell record disappears (shell.get -> 404) but the background job is never finalized.
   const PIDFILE = "/tmp/opencode/bg-evict.pid"
   const alive = (pid: string) => {
     try {
       execSync(`kill -0 ${pid}`)
       return true
     } catch {
       return false
     }
   }
   
   export default OpenCodeDriver.use(
     {
       ...(process.env.OC_DEV ? { opencode: { dev: process.env.OC_DEV } } : {}),
       project: { git: true, files: { "README.md": "repro\n" } },
       tui: { recording: true },
     },
     ({ ui, llm, opencode }) =>
       Effect.gen(function* () {
         yield* llm.queue(
           Llm.toolCall({
             index: 0,
             id: "call_bg",
             name: "shell",
             input: { command: `echo $$ > ${PIDFILE}; exec sleep 600`, background: true },
           }),
           Llm.finish("tool-calls"),
         )
         yield* llm.queue(Llm.text("Started the background command."))
         yield* ui.submit("Run a long command in the background")
         yield* ui.waitFor("Started the background command.")
         yield* Effect.sleep(2000)
         const pid = readFileSync(PIDFILE, "utf8").trim()
         const before = yield* opencode.shell.list()
         console.log("shell.list before evict:", JSON.stringify(before))
         const id = (before as any).data?.[0]?.id ?? (before as any)[0]?.id
         yield* ui.screenshot("before-evict")
   
         yield* opencode.debug.location.evict()
         console.log("evicted location services")
         yield* Effect.sleep(5000)
   
         console.log("process alive after evict:", alive(pid))
         const get = yield* opencode.shell.get({ id } as any).pipe(Effect.exit)
         console.log("shell.get after evict:", JSON.stringify(get).slice(0, 300))
         console.log("shell.list after evict:", JSON.stringify(yield* opencode.shell.list()))
         const bogus = yield* opencode.shell.remove({ id: "sh_doesnotexist" } as any).pipe(Effect.exit)
         console.log("shell.remove bogus id:", JSON.stringify(bogus))
         // A finished/cancelled background shell normally posts a "Shell ..." notification to the session.
         const notified = yield* ui.waitFor("Shell ").pipe(Effect.timeout(10000), Effect.exit)
         console.log("completion notification shown:", notified._tag === "Success")
         yield* ui.screenshot("after-evict")
         // Persisted background marker (kv job.background/*) in the server's database.
         const file: string = (before as any).data[0].file
         const db = join(dirname(dirname(dirname(file))), "opencode.db")
         try {
           const rows = new Database(db, { readonly: true }).query("select * from kv where key like 'job.background/%'").all()
           console.log("kv job.background records:", JSON.stringify(rows))
         } catch (e) {
           console.log("kv not readable (drive may use an in-memory database):", String(e))
         }
         if (alive(pid)) execSync(`kill -9 ${pid}`)
       }),
   )

   Run it with opencode-drive: opencode-drive run ./drive-evict.ts. It drives opencode from your PATH; set OC_DEV to the path of an OpenCode checkout to run that from source instead.
2. opencode-agent

   added

   reproduced

   on Oct 18, 2026
3. reynold-hu commented on Oct 18, 2026

   

   reynold-hu

   on Oct 18, 2026

   More actions

   Opened #56809 for the API half: DELETE /api/shell/:id now returns 404 ShellNotFoundError for an unknown id (the endpoint declares the error too), matching shell.get. The test fails on the parent commit with 204. I also confirmed from the code that explicit jobs.cancel persists a cancelled background marker, while the teardown path (scope close) marks the job cancelled without writing that state back - the marker then stays running until restart recovery. Settling the session timeline when a Location is evicted looks like the remaining piece; happy to follow up if a maintainer points at the preferred owner.
4. reynold-hu commented on Oct 19, 2026

   

   reynold-hu

   on Oct 19, 2026

   More actions

   Following the review on #56809: I closed that API change because the idempotent 204 is an intentional contract (V2_HTTP_API_AUDIT entry 135). The remaining fix is settling the session/timeline state when a Location is torn down, so the ghost chip clears without a shell removal. From the code, explicit jobs.cancel persists a cancelled marker while the scope-close path in settle skips that write; that marker mismatch is separate from the client's stuck row.
5. esafak

   marked zombie processes #57634 as a duplicate of this issue on Oct 21, 2026