Skip to content

Cancellation does not reach the model-catalog fetchers: unabortable OpenRouter request, in-flight entry held until settle #1615

Description

@DaubnerF

Problem

When Zoo needs model metadata for a provider, the request goes through the model cache and out to the provider's catalog endpoint over plain HTTP. Nothing about that outbound request is cancellable: a caller that gives up (a timeout, a cancelled task, a closed preview) cannot stop the HTTP request, and the cache keeps the shared in-flight entry alive until the request settles. The fetcher layer under src/api/providers/fetchers/ accepts no cancellation input.

Observed in version v3.82.1

Evidence

  • The catalog refresh is a shared single-flight. inFlightRefresh is a module-level map keyed per provider cache key (src/api/providers/fetchers/modelCache.ts:45), and dedupedFetch deletes the entry only in a .finally() on the fetch promise (src/api/providers/fetchers/modelCache.ts:363-380). A waiter that stops waiting does not affect the entry or the request; only settle removes it.
  • The OpenRouter catalog fetch is a bare axios.get with no request timeout and no signal (src/api/providers/fetchers/openrouter.ts:102). Against a hung or slow endpoint the request has no bound.
  • No caller-supplied cancellation reaches the catalog fetch path. Neither the model-cache entry points nor the OpenRouter fetcher accepts a signal, so whatever cancellation a caller applies at its own layer, the shared cached fetch and the provider HTTP call keep running. (Some other providers' fetchers build their own timeout controllers; none forwards a caller's cancellation.)

Consequence

A hung or slow catalog endpoint leaves the shared in-flight entry pending until the request settles. Callers that stop waiting leave the HTTP request running; repeated triggers against a misconfigured endpoint accumulate stale fetches behind the single-flight map. Previews can advertise stale model capabilities because a superseded fetch is still the one in flight. A timed-out catalog fetch can outlive every caller that wanted it.

Proposed approach

Only the fetcher layer: src/api/providers/fetchers/. Thread the caller's cancellation through the model-cache single-flight down to the fetchers' HTTP calls (including the OpenRouter axios.get), give the catalog request a request timeout, and release the in-flight entry when an aborted request settles. Where multiple waiters share one fetch, reference-count it so the underlying request aborts when the last waiter cancels. Caller-side bounded waits and cancellation checks are outside this issue.

Acceptance criteria

  • A waiter that aborts detaches from the shared fetch; when the last waiter is gone, the underlying request is aborted at the network level.
  • The in-flight entry is released on abort, not only on settle.
  • A timed-out or superseded catalog fetch cannot outlive its callers.
  • Tests hold the HTTP layer pending, abort the caller, and assert the request was aborted and the entry released.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions