Repository navigation
Conversation
A third part of runtime.js, active only when the builder loads the page in its invisible test frame (named puter-test-runner) and started only by a message from the builder's origin. It runs the tests the builder sends against the real page and posts each result back. From the moment it runs, the page can't touch anything the user would notice: localStorage, sessionStorage and puter.kv writes stay in memory over the real data, AI calls get placeholder answers (no cost), and puter.fs changes, sign-in and sign-out, hosting/app/worker management and requests that change data on the app's workers are refused. Dialogs are answered without showing and media plays muted. The test API is small: test(), click/fill/press, text/find/visible, waitFor, expect, plus dialogs and setAIReply. Failures name the values and the line in the tests. No async syntax in the file, so a parse error can never take the badge and the picker down with it.
For apps with behaviour worth protecting, the AI saves a few tests with the new AppTests tool. They live in the project's .puter/ folder, which the Code view, GitHub sync, the zip download and the AI's file listing already skip, and which version snapshots, restores and Duplicate copy, so each version keeps the tests that match it. Publishing drops them from the release. update_preview runs them through verifyPreview: the tests start in a hidden, sandboxed frame (no dialogs, popups, downloads or device access, and it never keeps keyboard focus) as soon as the preview has loaded, alongside the existing watch for runtime errors. Runtime errors win; failing tests come back to the AI as tests_failed, sharing the three fix rounds, with a note to fix the app unless the test is wrong, never weaken a test, and never mention tests to the user. Anything inconclusive is never a failure: no runner in the page, a stopped turn, a background tab, or the 45s budget running out. A group that fails is run once more, and only tests failing both times count. An app without tests costs one file read, made while the preview reloads. FEATURE_FLAGS.appTests turns it all off. AppTests is not strict, keeping the strict-tool count under Claude's limit of 20.
scripts/test-app-tests.mjs runs the runner (source and minified) against a stub DOM: inert outside its frame, started only by the builder, the storage/kv/AI/worker sandbox, and pass, fail, app error, timeout, setup and grouping results. It drives the builder side against a simulated frame (unavailable, retries, other pages, navigation, forged messages, stopping, background tabs, focus) and runs the real verifyPreview, update_preview and AppTests code. Breaking each guard fails it. test-mcp counted every registered tool as offered; it now counts the built-in tools a turn offers, since AppTests is left out when its flag is off.
When a test focused a field, the guard moved focus back to the composer from inside the blur event, while Chrome was still moving focus into the frame. The composer then looked focused, but Chrome kept sending keys to the hidden app: in real Chrome, 36 of 40 keys typed during a run went there. The check now runs just after the blur, when taking focus back works (37 to 40 of 40 reach the composer).
The runner compiles the tests with all of its helpers (sleep, text, click, …) as parameters, but the save-time check only used test and expect. Tests with their own helper such as `const sleep = …` were saved, then failed in the app with "Identifier 'sleep' has already been declared", which cost a whole preview round. The check now uses the runner's list, and a test keeps the two lists in step.
The test frame swapped window.localStorage for an in-memory layer, but runtime.js is deferred, so an app or library that had already kept the real store in a variable (const store = localStorage) still wrote the user's real data during tests. In real Chrome, a test that clicked "Add" left TEST-WRITE in the app's localStorage. Storage.prototype's methods and length now send calls on an overlaid store to its layer, so those writes stay in memory too.
Only fetch and puter.workers.exec were guarded, so an app that talks to its worker with XMLHttpRequest (axios, older code) changed the user's real backend data during tests: in real Chrome, a test's POST reached the worker, once per run and again on the retry. XMLHttpRequest writes to *.puter.work now fail like a request that never reached the server, sendBeacon to a worker returns false, and a form posted to a worker is stopped unless the app already handles it itself. Reads still go through.
click() ended with el.click(), which SVG elements don't have, so a test that clicked a board cell or a chart bar drawn in SVG failed with "el.click is not a function" and sent a working app back to the AI. An element without click() now gets a dispatched click event instead.
fill() refused range and color inputs as "not a text field", so a test of a tip slider or a settings screen failed on the test API itself and cost an AI round on an app that worked. fill() now sets them, and the AppTests description says it does.
press() built the code as 'Key' + the key, so the space bar arrived as code "Key " (not "Space"), digits as "Key1" (not "Digit1"), "Space" arrived as the key "Space" (not " "), and keyCode/which were always 0. A game reading e.code === 'Space' or e.keyCode === 32 ignored the test and its working controls were reported to the AI as broken. press() now sends what a US keyboard sends, accepts the usual aliases (Space, Esc, Up), clicks buttons and checkboxes on the space bar, and clicks a button-type input on Enter instead of submitting its form.
The visible preview may use the clipboard and go full screen, but the
test frame has no permissions, so navigator.clipboard.writeText and
requestFullscreen rejected there. In real Chrome, a working Copy button
and a full-screen button both failed their tests, which sends a healthy
app back to the AI. The frame now has an in-memory clipboard (including
execCommand('copy') and ClipboardItem), which never replaces what the
user copied and lets a test read what the app copied, and fullscreen
requests resolve without changing the page. The AppTests description
tells the AI about the clipboard.
The AppTests description said the user's data is never touched, but only localStorage, sessionStorage and puter.kv are kept in memory. In real Chrome, a test that saved a note to IndexedDB left it in the app's real database. An app opens its database before the deferred runner exists, so those writes can't be redirected reliably; the description now says so and tells the AI never to test flows that change data kept there.
The test frame has no autoplay permission, so its AudioContext stays suspended and resume() waits for a real click that never comes. In real Chrome, a game whose Start button awaits ctx.resume() timed out, and the working game would be sent back to the AI. resume() now resolves at once in the test frame; the context stays suspended, so nothing is heard.
The test frame is sandboxed with an empty allow list and no popups, unlike the visible preview, so getUserMedia and geolocation are refused there and window.open returns null. A test of a location or camera flow would fail on an app that works for the user. The AppTests description now says these aren't available, as it already did for file changes and sign-in.
The test frame loads each page with a ?__pv= cache-buster, and the runner named an error's file by the last part of its address. An error in an inline script was reported as "(?__pv=1793755905277_k1kr96:3)", so the AI was pointed at a file that doesn't exist. The query is dropped, and the site's root reads as index.html.
…ests puter.js keeps clear as its own copy of the real flush, so replacing flush left clear able to empty the user's real store from a hidden test. It now goes to the in-memory flush, and kv/fs share and unshare are refused like the other writes.
A hidden test that clicks a Send or Remind me button could send a real email, push a real notification to the user's devices, or reset a team member's password through puter.js. Those are refused like the other writes, and the AI is told.
puter.ai.convo opens a billed realtime session that the placeholders didn't cover, so a test of a voice-chat button cost real money. It is refused now. The faked txt2speech also dropped the listVoices and listEngines puter.js hangs on it, so a voice picker broke only under test; the fakes keep them.
puter.js trims a list pattern, drops a trailing * and matches it as a
prefix, but the test overlay matched it exactly, so list('todo:')
missed the todos a test had just added and a working app looked
broken. The { pattern, returnValues } form skipped the overlay and an
options object was taken as a request for values. All three now behave
as they do outside tests.
The test frame shares an origin with the visible preview, so an app that keeps its tabs in step with BroadcastChannel sent what a test did to the preview, which could then save it to the user's real data. Channels opened in the test frame now talk only to each other, and one opened before the runner loaded sends nothing out.
Nothing cancels a promise, so a test that ran out of time kept clicking, filling and sleeping through the tests after it, failing them for no fault of the app (and one left waiting, un-awaited, failed whichever test was running when it gave up). Each step now remembers the test it started in and never finishes once that test is over.
A click or waitFor the test forgot to await surfaced as an unhandled rejection, which was reported as "The app threw an error during this test", sending the AI to fix an app that works. The runner now marks its steps' errors and says the step wasn't awaited.
text() took the first match even when it was hidden, so in an app
that hides its other screens text('h2') read a hidden screen's heading
instead of the one the user sees. It now reads the first visible
match, and a hidden one only when that's all there is.
dialogs and setAIReply carried over from one test to the next, so a test that checked dialogs had length 1, or relied on the default AI reply, passed or failed depending on the tests before it. Both are reset when a test ends: the first test still sees dialogs from page load, and a reply set at the top level holds for every test.
The layer was a Proxy over a plain object that sent every name to
getItem, so localStorage.hasOwnProperty('theme'), String(localStorage)
and instanceof Storage threw or lied only while tests ran, and calling
Storage.prototype.getItem on it failed. It now sits on Storage's
prototype, whose members win over stored keys as on the real object,
and the prototype's methods reach the layer.
The focus guard learned where focus was only from focusin, which never fires when the user clicks into the visible preview. If a test then pulled focus into the hidden frame, it was handed to the composer, so what the user typed for their app went into the chat (and Enter could send it). The guard now follows focus on every check.
Once a test had taken focus and it was given back, the end of the run focused the composer again even if the user had since left it, which on a phone brings the keyboard back up. Focus is now restored only when the hidden frame still holds it as it is removed.
The retry merge kept one result per name, so when two tests shared a name (a copy-pasted name, or two cut to the same 120 characters) and the first passed, the second counted as passed even though it failed both times. Both runs go through the page's tests in the same order, so tests are now matched by position.
verifyPreview waited for the tests file with no limit before starting its error watch, so a slow read delayed the check and a read that never settled left update_preview, and the turn, waiting until the user pressed Stop. The tests now start whenever the read is done, alongside the watch, and a read not back within 5s of the load counts as no tests.
The update_preview description told the AI it runs AppTests' tests even with FEATURE_FLAGS.appTests off, when the tool isn't offered. The clause now follows the flag, like the prompt rule, so with the flag off the description is the one from before tests.
click(), fill() and press() checked only the element's own disabled property, which stays false inside a <fieldset disabled>. The browser ignores a click there, so the test went on and failed later with a misleading message instead of saying the control is disabled. They now ask the browser, with :disabled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What it does
For apps with behaviour worth protecting (forms and validation, calculations, games, carts and checkout, saved data, a backend), the AI now writes a few automated tests of the main features.
update_previewruns them after every change in an invisible copy of the app, and failures go back to the AI, so it fixes a broken feature before the user ever sees it. Pages that only show content (a landing page, a portfolio, a menu) get no tests.The user never sees the tests:
How it works
.puter/builder/tests.jsin the project. That folder is already skipped by the Code view, GitHub sync, zip download, readdir and SearchFiles, and already copied by version snapshots, restores and Duplicate, so a version keeps the tests that match it. Publishing deletes the file from the release.js/app-tests.js,stagePublishReleasetools/apps_and_sites/app_tests.jsputer-test-runnerloads the preview, the tests are sent bypostMessage, and results come back per test. Each page and phone size runs in its own frame. A group that fails is run once more.js/app-tests.jsruntime.js. It does nothing anywhere else and starts only for the builder's origin.src/runtime.jsverifyPreviewstarts the tests once the preview has loaded, alongside the 3s error watch. Runtime errors win.js/ui.jsupdate_previewsendstests_failed/tests_persist, which share the 3 fix rounds with runtime errors.update_preview.jsFEATURE_FLAGS.appTeststo turn everything off.prompt.js,helpers.jsThe user's data, money and screen are protected
Inside the test frame:
localStorage,sessionStorageandputer.kvwrites stay in memory, includingputer.kv.clear()and method calls through a reference the app kept before the runner loaded. Reads still see the real data, including puter.js's session. The in-memorylocalStorageis aStorageto the app, andkv.listpatterns match the way puter.js sends them (a prefix).puter.ai.*returns placeholders, so tests cost nothing. Live conversations (puter.ai.convo) are refused.puter.fswrites, kv and fs sharing, sign-in and sign-out, hosting/app/worker/team management, email, notifications, and writes to*.puter.work(fetch, XMLHttpRequest,sendBeacon, form posts) are refused.BroadcastChannelmessages stay inside the test frame, so they never reach the visible preview (same origin).AudioContext.resume()resolves while the context stays silent.alert/confirm/promptare answered without showing, and media is muted.The frame itself:
allowfeatures, so there's no autoplay and no permission prompts.Conservative, like
verifyPreviewNone of these ever counts as a failure:
A test that fails once and passes on the rerun counts as passed. A test that runs out of time is frozen at its current step, so it can't click through the tests after it.
Cost and speed
Tests
scripts/test-app-tests.mjs:verifyPreview,update_previewand AppTests code, plus the prompt and wiring checks.test-mcp.mjsnow counts the built-in tools a turn offers, since AppTests is left out when its flag is off.dist/runtime.js, with the builder and the app on different origins. Scenarios: passing, failing with line numbers, app errors, navigation, other pages at phone size, setup errors, a missing runner, hidden elements, timeouts, stopping, focus,requestAnimationFrame, streaming AI and dialogs.npm run build(all tests, then the build) passes.Rollout note
The runner ships in
runtime.js, which every app loads from builder.puter.com. Until that deploy is live, runs come back "unavailable" and nothing changes for the user.