-
Notifications
You must be signed in to change notification settings - Fork 0
feat: implement the code mode runtime Actor #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
MQ37
wants to merge
46
commits into
master
Choose a base branch
from
feat/code-mode-runtime
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
46 commits
Select commit
Hold shift + click to select a range
096ea5b
feat: per-run workerd sandbox that runs MCP Code Mode programs
MQ37 6abfe19
test: add apify binding smoke test (tests/binding-smoke.js + test.sh)
MQ37 b669c05
docs: tighten README copy and add compact apify binding reference
MQ37 f20b489
docs: move apify binding section above Learn more; trim intro copy
MQ37 63590fd
docs: drop run-token sentence from Permissions & safety
MQ37 8d0c7ff
docs: add docs/API.md detailed reference and link it from README
MQ37 4c3a430
docs: document exact output shapes and link each method to its Apify …
MQ37 db006c3
feat: add exitCode to run output
MQ37 6486cd0
fix: remove nodejs_compat to close sandbox egress and token leaks
MQ37 7a091e2
test: assert fetch allowlist covers apify.com + subdomains only
MQ37 2493f54
fix: block WebSocket and EventSource egress in guard.js
MQ37 5461aee
fix: close realFetch allowlist bypass via one-shot claim in guard.js
MQ37 c36f402
refactor: rename single-letter locals and de-duplicate the loopback port
MQ37 4607d26
fix: close redirect-following allowlist bypass, lock globalThis.fetch
MQ37 11e7c96
fix: catch usercode.js compile failures, scope run.abort, freeze bind…
MQ37 5bc3374
refactor!: rename actor.getDetails() to actor.get()
MQ37 d59cec3
refactor: migrate worker/*.js to TypeScript, add CI type-checking
MQ37 9020141
docs: drop TypeScript claim for the user's code input
MQ37 976d151
refactor!: rename actor.run/runAndGetItems, run.wait, dataset.getSchema
MQ37 8e18d6b
docs: self-contained actor.json description, dataset schema, README
MQ37 9d251d4
refactor: migrate tests/*.js probes to TypeScript, fix stale run.wait…
MQ37 2169a10
refactor!: apify.store top-level binding, kvs -> keyValueStore
MQ37 f23e4e3
feat: forward X-Apify-Request-Origin: MCP to sub-runs when this run i…
MQ37 2b84280
fix(ci): allow workerd postinstall build script under pnpm 11
MQ37 5fbbc53
docs: add Chain Actors recipe, dataset/keyValueStore usage examples
MQ37 0f61079
fix: use draft-07 dataset schema, not draft 2020-12
MQ37 854d536
fix: Docker builder stage doesn't need tests/ compiled
MQ37 c9d05b5
docs: warn against dangling-promise pattern and wrong API shape in co…
MQ37 6c52256
feat: opt into fullReadmeOnly, bypass the auto-generated summary
MQ37 77a3d8d
docs: lead schema-check guidance with fetch-actor-details, not just i…
MQ37 dae5b1d
docs: make dataset-ID reuse imperative, clarify non-terminal wait status
MQ37 776c581
feat!: dual-mode listItems/store, remove dataset.iterate()
MQ37 83eb3ae
docs: nudge Actor description/README with data-backed usage thresholds
MQ37 7346884
fix: trim actor.json description to fit under 300 chars
MQ37 a767a0e
docs: cut internal-telemetry section, dedupe repeated facts, tighten …
MQ37 2302300
docs: trim Code Mode/API-plumbing mentions from README
MQ37 2812c9c
chore: remove fullReadmeOnly actor.json flag
MQ37 8930b95
fix: gate realFetch claim on request handling, not import order
MQ37 100481c
fix: remove guard.js capability-theft path, add execution safeguards …
MQ37 11699fd
fix: restore guard.js import, close TOCTOU race, add real-workerd int…
MQ37 11cae8a
fix: close global-URL-hijack SSRF bypass, NaN budget corruption, harn…
MQ37 a491443
fix: harden every security check against builtin prototype/static-met…
MQ37 fbc7c5e
fix: extend builtin-capture hardening to accessors, encodeURIComponen…
MQ37 0e89669
style: trim redundant comments
MQ37 dccbcd6
fix: use supported input schema keyword
MQ37 46deff8
ci: run remote smoke tests on master
MQ37 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| { | ||
| "actorSpecification": 1, | ||
| "name": "code-runtime", | ||
| "title": "Code Runtime", | ||
| "description": "Runs one JS script in a sandboxed Actor with an apify binding (Actors, datasets, KV stores). Worth it for bulk work: 50+ record filter/sort/aggregate, or 10+ item fan-out (e.g. many page visits). Skip for a single <10-item lookup \u2014 call the Actor directly instead. Read the README first.", | ||
| "version": "0.1", | ||
| "buildTag": "latest", | ||
| "usesStandbyMode": false, | ||
| "defaultRunOptions": { | ||
| "timeoutSecs": 900, | ||
| "memoryMbytes": 1024 | ||
| }, | ||
| "input": { | ||
| "title": "Code Runtime Input", | ||
| "description": "The program to run inside the sandbox.", | ||
| "type": "object", | ||
| "schemaVersion": 1, | ||
| "properties": { | ||
| "code": { | ||
| "title": "Code", | ||
| "type": "string", | ||
| "description": "JavaScript executed in the sandbox with `apify` and `console` globals; only console output is captured and pushed to the dataset as { stdout, stderr, exitCode, statusMessage } — a top-level `return` value is NOT captured. Before writing code that calls a specific Actor, check its real input field names first — via fetch-actor-details (outside this script, before you write it) or apify.actor.get({ actorId }) (inside it, for an Actor picked at runtime). Do not guess field names from memory; a wrong one throws a fast 400, but costs a wasted round trip. Print a small JSON summary of the result — never dump full datasets. Write top-level `await` statements directly in the script; do NOT wrap your logic in an async function you call without awaiting (e.g. `async function main(){...}; main()`) — the script returns as soon as the top-level body finishes, silently discarding anything still pending, with no error. Every apify.* method takes ONE options object keyed by id, e.g. apify.actor.call({ actorId, input }), apify.dataset.listItems({ datasetId, limit }) — this is NOT the public apify-client SDK's curried apify.actor(id).call(input) shape. If a prior attempt already logged a nested run's defaultDatasetId/defaultKeyValueStoreId (visible in your own earlier turns), reuse it — do NOT re-run the same Actor call with identical input, that wastes compute on a call that already succeeded. apify.actor.call/run.waitForFinish may return non-terminal (READY/RUNNING) once the 60s wait cap elapses — that is NOT a failure, poll again instead of throwing.", | ||
| "editor": "javascript" | ||
| }, | ||
| "maxActorRuns": { | ||
| "title": "Max Actor runs", | ||
| "type": "integer", | ||
| "description": "Caps how many Actor runs this script may start in total across actor.start/actor.call/actor.callAndGetItems. Starting one more once the limit is reached throws inside the script. Omit for no limit.", | ||
| "minimum": 1 | ||
| }, | ||
| "maxTotalChargeUsd": { | ||
| "title": "Max total charge (USD)", | ||
| "type": "number", | ||
| "description": "Execution-wide spending budget across every Actor run this script starts — distinct from a single call's own maxTotalChargeUsd, which only caps that one run. Each run's own cap is clamped so the combined total never exceeds this budget; starting a run once it's exhausted throws inside the script. Omit for no limit.", | ||
| "minimum": 0 | ||
| }, | ||
| "defaultTimeoutSecs": { | ||
| "title": "Default Actor run timeout (seconds)", | ||
| "type": "integer", | ||
| "description": "Applied as timeoutSecs to actor.start/actor.call/actor.callAndGetItems calls that don't specify their own. Omit to use the Apify API's own default.", | ||
| "minimum": 1 | ||
| } | ||
| }, | ||
| "required": ["code"] | ||
| }, | ||
| "output": { | ||
| "actorOutputSchemaVersion": 1, | ||
| "title": "Code Runtime Output", | ||
| "description": "One dataset item { stdout, stderr, exitCode, statusMessage } (exitCode: 0=returned, 1=threw); see the exitCode field description for the timeout/OOM case.", | ||
| "type": "object", | ||
| "properties": { | ||
| "output": { | ||
| "type": "string", | ||
| "title": "Execution output", | ||
| "template": "{{links.apiDefaultDatasetUrl}}/items" | ||
| } | ||
| } | ||
| }, | ||
| "storages": { | ||
| "dataset": { | ||
| "actorSpecification": 1, | ||
| "description": "Present only if the script ran to completion (returned or threw). A run-level timeout or OOM kill produces zero items for that run — the calling Actor-run's own status (SUCCEEDED vs FAILED/TIMED-OUT) is the signal for that case, not item presence.", | ||
| "fields": { | ||
| "$schema": "http://json-schema.org/draft-07/schema#", | ||
| "type": "object", | ||
| "properties": { | ||
| "stdout": { | ||
| "type": "string", | ||
| "title": "stdout", | ||
| "description": "Captured console.log/console.info output." | ||
| }, | ||
| "stderr": { | ||
| "type": "string", | ||
| "title": "stderr", | ||
| "description": "Captured console.error/console.warn output, plus the thrown error (or compile failure) message when exitCode is 1." | ||
| }, | ||
| "exitCode": { | ||
| "type": "integer", | ||
| "enum": [0, 1], | ||
| "title": "Exit code", | ||
| "description": "0 = script returned normally, 1 = script threw (or failed to compile)." | ||
| }, | ||
| "statusMessage": { | ||
| "type": "string", | ||
| "title": "Status message", | ||
| "description": "Prose form of the same signal: 'Script completed' / 'Script threw: ...' / 'Failed to compile: ...'." | ||
| } | ||
| }, | ||
| "required": ["stdout", "stderr", "exitCode", "statusMessage"] | ||
| } | ||
| } | ||
| }, | ||
| "dockerfile": "../Dockerfile" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| node_modules | ||
| .git | ||
| *.log | ||
| .DS_Store |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| name: Remote smoke tests | ||
|
|
||
| on: | ||
| push: | ||
| branches: [master] | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: remote-smoke-${{ github.ref }} | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| remote-smoke: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| steps: | ||
| - name: Check out source | ||
| uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 | ||
|
|
||
| - name: Set up pnpm | ||
| uses: pnpm/action-setup@f40ffcd9367d9f12939873eb1018b921a783ffaa # v4 | ||
|
|
||
| - name: Set up Node.js | ||
| uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 | ||
| with: | ||
| node-version: 24 | ||
| cache: pnpm | ||
|
|
||
| - name: Install dependencies | ||
| run: pnpm install --frozen-lockfile | ||
|
|
||
| - name: Set up Apify CLI | ||
| uses: apify/setup-apify-cli-action@8b19e6a52312e8948b3b10cdb06bd3fa39e8db4a | ||
| with: | ||
| token: ${{ secrets.APIFY_TOKEN }} | ||
|
|
||
| - name: Run remote smoke tests | ||
| run: ./test.sh |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| name: Typecheck and test | ||
|
|
||
| on: | ||
| push: | ||
| branches: [master] | ||
| pull_request: | ||
|
|
||
| jobs: | ||
| typecheck: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: pnpm/action-setup@v4 | ||
| - uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: 24 | ||
| cache: pnpm | ||
| - run: pnpm install --frozen-lockfile | ||
| - run: pnpm run typecheck | ||
|
|
||
| # Token-free unit tests for guard and redirect logic. | ||
| test: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: pnpm/action-setup@v4 | ||
| - uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: 24 | ||
| cache: pnpm | ||
| - run: pnpm install --frozen-lockfile | ||
| - run: pnpm run test | ||
|
|
||
| # Real workerd integration tests with a local API mock. | ||
| test-integration: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: pnpm/action-setup@v4 | ||
| - uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: 24 | ||
| cache: pnpm | ||
| - run: pnpm install --frozen-lockfile | ||
| - run: pnpm build | ||
| - run: pnpm run test:integration |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| node_modules/ | ||
| # Generated at container startup by entrypoint.sh (never checked in): | ||
| worker/usercode.js | ||
| # Compiled from worker/*.ts and tests/*.ts by `pnpm build` (tsconfig.json emits next to source): | ||
| worker/runner.js | ||
| worker/guard.js | ||
| tests/*.js | ||
| tests/unit/*.js | ||
| tests/integration/*.js | ||
| *.log | ||
| .DS_Store |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # Build with Node; run with only workerd and required utilities. | ||
| FROM node:24-bookworm-slim AS builder | ||
| WORKDIR /build | ||
| COPY package.json pnpm-lock.yaml tsconfig.json ./ | ||
| COPY worker/ ./worker/ | ||
| # The binary ships in an optional dependency; compile worker files directly. | ||
| RUN corepack enable \ | ||
| && pnpm install --frozen-lockfile --ignore-scripts \ | ||
| && pnpm exec tsc -p tsconfig.json \ | ||
| && BIN="$(node -e "process.stdout.write(require('workerd').default)")" \ | ||
| && cp "$BIN" /workerd \ | ||
| && chmod +x /workerd | ||
|
|
||
| # Minimal runtime image: workerd, compiled JS, and certificates. | ||
| FROM debian:bookworm-slim | ||
|
|
||
| # curl fetches input; jq extracts user code. | ||
| RUN apt-get update \ | ||
| && apt-get install -y --no-install-recommends ca-certificates curl jq \ | ||
| && rm -rf /var/lib/apt/lists/* | ||
|
|
||
| COPY --from=builder /workerd /usr/local/bin/workerd | ||
|
|
||
| WORKDIR /app | ||
| COPY worker/entrypoint.sh worker/config.capnp ./worker/ | ||
| COPY --from=builder /build/worker/runner.js /build/worker/guard.js ./worker/ | ||
|
|
||
| ENTRYPOINT ["sh", "/app/worker/entrypoint.sh"] |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider addition of tests into CI:
Claude:
Testing gap: no behavioral tests in CI — the sandbox boundary itself is ungated
CI runs typecheck only (
.github/workflows/typecheck.yml— one job,pnpm run typecheck). Every behavioral test intest.shrequiresapify push+apify call+ a token, so nothing about thesecurity boundary runs in CI. Locally, the guard logic has no token-free test at all: the allowlist paths of
isAllowedHost/validateUrlare only exercised live on-platform(
tests/sandbox-isolation.ts), andguardedFetch's redirect re-validation — the entire reason that function exists — has no test of any kind, live or local.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed. Added token-free unit tests and real-workerd integration tests with a local API mock to CI.