Skip to content

feat: capture requests - in-app screenshots, screen recordings and logs - #42

Merged
boehlerlukas merged 25 commits into
masterfrom
claude/capture-requests
Oct 1, 2026
Merged

feat: capture requests - in-app screenshots, screen recordings and logs#42
boehlerlukas merged 25 commits into
masterfrom
claude/capture-requests

Conversation

@boehlerlukas

@boehlerlukas boehlerlukas commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What this adds

Android support for capture requests. When a workflow, an AI agent or a teammate asks for a screenshot or a screen recording, the customer answers from the widget card and the SDK captures the app. Logs requests are collected in the background.

Capture, in the app only, with no permission

  • No MediaProjection, no foreground service and no new permission, manifest entry or dependency. The library manifest is unchanged.
  • Screenshots:
    • every app window: the activity plus dialogs and bottom sheets (WindowInspector on API 29+, reflection below);
    • PixelCopy per window (API 26+), with SurfaceViews such as maps, video or GL copied separately and drawn underneath;
    • View.draw below API 26.
  • Recordings: the same window pipeline at an adaptive 2–8 fps (main-thread cost kept under 30 %), encoded as H.264 MP4 with MediaCodec / MediaMuxer.
    • Long edge ≤ 1280, letterboxed on rotation, a keyframe every 2 s.
    • Paused while the app is in the background; stops cleanly on low memory; hard cap 180 s.
  • Masking, before anything leaves the device:
    • password fields, card-number and one-time-code fields (autofill hints, API 26+);
    • FLAG_SECURE windows;
    • views passed to maskView.
    • Masks are padded by 8 dp and cover the view's previous position too, so nothing leaks during fast scrolling.
    • If the masks can't be worked out, the screenshot or frame is dropped rather than sent unmasked.
    • The Flutter host bitmap callback is only used below API 24; it gets the same masks.
  • Capture bar: a small dark pill above the app that can be dragged anywhere.
    • Ready: grip, then Capture or Start, then a round ×.
    • Recording: only the red dot, the timer and a round red Stop.
    • Instruction: a caption above the bar for about 4 s.
    • After a drag: it stays inside the safe area and snaps to the centre. Its position is remembered for the app process and re-clamped on rotation and keyboard changes.
    • Touches outside the pill reach the app (Android 14+).
  • Preview:
    • dark full screen, rounded video, tap to play or pause, thin scrubber;
    • × at the top right, which cancels a running upload;
    • ↺ Retake and Send ➤ pills.
  • Text: every word comes from the widget's translated capture-start.labels.
  • Widget during a capture: it closes and reopens on the conversation, and the app gets no widget opened / closed callbacks for that.
  • Recording files: kept in noBackupFilesDir (not the cache, which Android clears when storage is low) and deleted on every end path. Leftovers are deleted when the SDK starts.
  • In-app messages that arrive during a capture are held and shown afterwards.

Logs requests

  • Taken from the SDK websocket (capture-request) or the ping answer (cr). The SDK sends ws: true in every ping as before, and reads cr from the answer.
  • Flow: claim → flush handler (≤ 500 ms) → build the bundle from the existing collectors → gzip → /logs. Once per request id.
  • Failures: up to 3 attempts (now, +2 min, +8 min), then /event failed.
  • No replay frames: they can't be masked.
  • Bug-report screenshots and replays are unchanged.

Public API

Gleap.getInstance().setCaptureEnabled(boolean enabled);              // false: upload only; stops a running capture
Gleap.getInstance().setRemoteLogCollectionEnabled(boolean enabled);  // false: logs requests answered "unsupported"
Gleap.getInstance().maskView(View view);                             // weak reference
Gleap.getInstance().unmaskView(View view);
Gleap.getInstance().setLogFlushHandler(GleapLogFlushHandler handler); // wrappers hand over buffered logs; main thread, 500 ms cap

Fixes found on the way

  • Screenshots on Android 7.1 with hardware acceleration (API 25 returned none).
  • Attachment uploads are copied in chunks instead of whole files.

Example app: a capture demo screen (masked fields, a secure dialog, a GL view) and a debug-only local stub mode (adb shell setprop debug.gleap.stub http://10.0.2.2:8787).

Verification

  • ./gradlew clean build: library release, example app, 203/203 unit tests (sizing math, logs bundle/gzip/dedupe, origin and request-id checks). Lint: 0 errors. LeakCanary: no leaks.
  • Phone emulator (API 36.1), against a local stub server:
    • Screenshots include dialogs and GL content, with masks applied.
    • Recordings:
      • limit, rotation, background cut, low memory;
      • Send / Retake / Cancel, Gleap.close() mid-capture and setCaptureEnabled(false) mid-capture;
      • stop watchdog (5 s): an injected stall in finishing gave failed with no crash;
      • main-thread budget with an injected 250 ms frame cost;
      • fast scrolling: all 131 frames checked, no leak.
    • Logs over the websocket and the ping fallback; retries ending in failed; in-app messages held during a capture.
  • Tablet emulator (1600×2560): screenshot, rotation and recording, from the first round.
  • UI round, phone and tablet, portrait, landscape and RTL:
    • every bar and preview state, with screenshots;
    • drag to a corner, then the next capture starts there;
    • rotation re-clamp, keyboard and dialogs;
    • × during upload cancels it;
    • a recording killed before Send is deleted on the next start.
    • Forced API 26–33 window-copy path and the software-draw path both produce correct dialog screenshots.

Not covered

  • No physical devices.
  • Nothing on API < 24.
  • Freeform window resizing.
  • Black content: API 21–23 can't capture SurfaceView content; it comes out black.
  • Not auto-masked: Compose text fields, WebView inputs and Flutter-drawn fields aren't native password views. Apps can maskView their host view, and the customer reviews every capture before sending.
  • A stalled stop shows no "finishing" indicator for up to 5 s.

Release

Version bump, CHANGELOG and the dashboard's hardcoded Android version happen at release. The wrappers (React Native, Flutter, Capacitor) need this release; their draft PRs bump the pin.

The whole program: capture requests ("Show me the issue")

Workflows, AI agents (Kai, Kai Resolve, custom agents) and teammates can now ask a customer for a screenshot (with annotation) or a screen recording in the middle of a conversation, and can collect logs (console, network, custom data, environment data, events) from the customer's app in the background.

One server object, the capture request, backs all three. The customer sees a card in the widget; the SDK in their app does the capture; the result lands in the thread and resumes the workflow or wakes the agent. Background log requests travel over the SDK websocket every SDK already keeps open.

How each platform captures:

  • Web, desktop Chrome/Edge: screenshot via one-click "Share this tab" (exact pixels), in-browser render as fallback; recording via tab capture, "Record this page" fallback when sharing is blocked.
  • Web, Firefox/Safari/phones/tablets: in-browser screenshot; window/screen recording on desktop, page recording (rrweb) on phones.
  • iOS / Android apps (incl. React Native, Flutter, Capacitor): in-app capture of the app's own windows: no OS prompt and no new permissions (no ReplayKit, no MediaProjection). Recordings are H.264 MP4 built from frames.
  • Older SDKs, email, WhatsApp & co.: upload / reply-with-attachment fallback, so a request never dead-ends.

Design: https://claude.ai/artifact/5i88BeWfcsWUFhhJmPwK5V (private)

PRs in this program (one per repo):

🤖 Generated with Claude Code

boehlerlukas and others added 18 commits September 30, 2026 21:42
@boehlerlukas @claude
fix: take screenshots on Android 7.1 with hardware acceleration
f21bbcb 
API 25 went to generateBitmap, which skips hardware accelerated windows (all of them by
default), so hardware accelerated apps on Android 7.1 got no screenshot. Below API 26
the activity window is now drawn into the bitmap, as on API 24 and lower.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: copy attachment uploads in chunks instead of whole files
075314a 
addFilePart read each file into one array sized to the whole file (and a single read()
could return less than the file). It now copies the file in 64 KB chunks; a file that
cannot be read still leaves its part empty, and errors writing the request still fail
the upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
feat: background log collection for capture requests
a147630 
When the server asks for the app's logs (a capture request of kind "logs", pushed on the
WebSocket as capture-request in any widget state, or returned as "cr" with a ping), the
SDK claims the request, gives a wrapper SDK up to 500 ms to hand over its buffered logs
(setLogFlushHandler), and sends what a ticket carries (console log incl. the logcat
snapshot, network logs, custom data, env data, events; the replay only when requested
and enabled, its frames uploaded through /uploads/sdksteps first) gzip compressed to
POST /v3/shared/capture-requests/{id}/logs. Each request id is sent once per process; a
request that failed on the server side is tried again when it is pushed again. With
setRemoteLogCollectionEnabled(false) such requests are answered with an "unsupported"
event and nothing is collected.

The SDK announces what it can capture: caps on the WebSocket url and in the ping body.

Replay frames are no longer recycled when they drop out of the replay: an upload may still
be reading them. Nothing runs before a request arrives; the work runs on two lazily
started daemon threads that end when idle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
feat: in-app screenshots and screen recordings for capture requests
3ba1c0b 
The widget (Gleap messenger) can ask the app for a screenshot or a screen recording
(capture-start). The widget cannot stay open while the customer uses the app, so it is
closed and the request is kept by the SDK, which shows a capture bar: a small panel
window of the app (no overlay permission) on the topmost window of the current activity,
above dialogs and bottom sheets, moved along with every activity, rotation and window
change, above system bars, cutouts and the keyboard, draggable, labelled with the
widget's localized texts (English when missing) for TalkBack. Nothing is captured before
the customer taps the bar; capture commands are only taken from the messenger's origin.

- Screenshot: every visible window of the activity (activity, dialogs, popups; via
  WindowInspector on API 29+, WindowManagerGlobal below) is copied with PixelCopy on
  API 26+ (View.draw below, or when a copy fails), SurfaceViews are copied separately and
  drawn beneath their window, dialogs get their dim; FLAG_SECURE windows, password fields
  and views masked with Gleap.maskView are painted black; a wrapper's GetBitmapCallback
  (Flutter) provides the picture instead. JPEG 85, long edge at most 2560. The widget is
  opened again on the conversation and gets capture-image and capture-state (preview)
  after its ping; the logs around it are sent when the request attaches logs.
- Recording (API 26+): the same window capture at the video's size, about 4 frames a
  second (2 to 8, keeping the main thread under 30 % of a frame interval), drawn onto a
  MediaCodec H.264 input surface (hardware canvas, software canvas where it misbehaves;
  another encoder when the default one fails) and muxed to MP4 on a background thread:
  long edge at most 1280, even sizes, about 1 Mbps, a key frame every 2 s, no audio,
  letterboxed after a rotation or resize. It pauses (and the pause is cut) while the app
  is in the background, stops at the request's time limit and on critical memory, then
  shows a preview with Send / Retake / Cancel. Send streams the file to
  /uploads/attachments with progress (OkHttp), completes the request, opens the widget
  on the conversation and reports capture-state done; the file is deleted on every path
  and left-overs of a dead process at the next start.
- Cancel, a refused start and failures report capture-state cancelled / unsupported /
  failed to the widget and the matching event to the server.

While a capture runs the SDK's overlay is hidden, activation methods wait and in-app
messages do not pop up. Public API: setCaptureEnabled, maskView, unmaskView.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
feat: capture demo screen and local stub mode in the example app
aeb0ebe 
A "Capture Demo" screen to try capture requests on: a GLSurfaceView (SurfaceView
content), a password field and a view masked with Gleap.maskView, a dialog with a
password field, a running clock, a way to open the same screen again, and switches for
setCaptureEnabled and setRemoteLogCollectionEnabled. The app registers a log flush
handler the way a wrapper SDK would.

Debug builds only: with adb shell setprop debug.gleap.stub http://10.0.2.2:8787 the app
sends every request (API, WebSocket, widget page, banners, modals) to a local stub server
instead of Gleap, with a fake SDK key; cleartext is allowed for 10.0.2.2 / localhost in
debug builds. The error callback prints the stack trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: tell the server in pings whether the WebSocket is connected
ca3c76f 
The ping body always said ws: true. The server pushes pending log requests on the socket
for ws: true pings and only returns them ("cr") for ws: false pings, so with the socket
down the HTTP fallback never fired. The ping now says whether the SDK's WebSocket is open
right now (from onOpen until it closes or fails); the rest of the ping answer is ignored
as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: no widget closed / opened callbacks for a capture's own close an…
3bfe95d 
…d reopen

A capture closes the widget so the customer can use the app and opens it again on the
conversation afterwards; to the app that is one widget session. The widget closed
callback no longer fires for that close, and the widget opened callback and the unread
count reset no longer fire for the reopen (a count that changed meanwhile is still
reported). Real opens and closes report as before.

The app still hears that the widget closed when it does not open again after the capture
(e.g. no screen of the app is shown), and Gleap.close() during a capture (to the app the
widget is open) ends the capture and reports the close.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
chore: log the widget callbacks and call Gleap.close() in the example…
51c5a2e 
… app

The capture demo logs the widget opened / closed and unread count callbacks (tag
GleapDemo) and has a Gleap.close() button, to check that a capture's own close and
reopen stay silent while real opens and closes are reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: mask app-provided pictures like the SDK's own captures
885ef6f 
With a GetBitmapCallback (the Flutter plugin always sets one) the capture used the app's
bitmap as it was: views masked with maskView, password fields and FLAG_SECURE windows
were not blacked out. The same masks are now drawn over the app's picture (mapped into
its coordinates; a FLAG_SECURE window's whole area is black), for screenshots and every
recorded frame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: stop a running capture when capture is turned off
d349918 
setCaptureEnabled(false) only stopped new captures. A screenshot or recording that runs
now stops too: nothing of it is kept (the encoder and its surface are released, the file
is deleted, an upload is cancelled), the bar goes away, the widget opens again on the
conversation with capture-state failed, and the request gets a failed event with the
reason capture-disabled. The capabilities report nothing from then on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
chore: app bitmap callback switch and a secure dialog in the example app
7ba9090 
The capture demo can hand the SDK its own picture of the screen (setBitmapCallback, as
the Flutter plugin does) and show a dialog with FLAG_SECURE, to check the masks on both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: send ws: true in every ping again
29e3b2c 
Reverts "tell the server in pings whether the WebSocket is connected" (ca3c76f). For a
ws: false ping the server puts the pending surveys, news, checklists, notifications and
the unread count into the ping answer and marks them sent, but the SDK does not read that
answer, so in-app messages triggered while the socket reconnects were lost. The server
pushes pending log requests over the socket when it connects, which covers capture
requests; a "cr" in a ping answer is still read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: captures are masked or dropped, and recordings stay bounded
c3b6420 
From a stability review of the capture pipeline:
- Masks on every path, failing closed: with a wrapper's bitmap callback (Flutter) the
  capture used that picture unmasked. Captures now always copy the windows (the
  SurfaceView copy takes Flutter's content); the callback is only used below API 24, where
  a SurfaceView cannot be copied, with the same masks. When the masks cannot be worked out
  (an error, or more views than can be checked) the screenshot or frame is dropped, never
  delivered without them.
- Masks are 8 dp bigger, and in a recording each covers where its view was in the previous
  frame too, so fast scrolling does not show a field between the scan and the copy.
- Payment card and one-time code fields (autofill hints) are masked like passwords.
- A recording never has more than two frames waiting for the encoder (a stuck encoder no
  longer collects frames), counts the main-thread time of every frame including the
  redraw after a failed copy, and keeps it under 30 %: at 2 fps a costly frame pushes the
  next one further out.
- Stopping a recording has a 5 s watchdog: if the encoder does not finish, the recording
  is given up (encoder, surface and muxer are released on a thread of their own), nothing
  is kept, and the widget opens again with the failure. Results of an earlier recorder are
  ignored.
- A started muxer without samples is stopped before it is released, and the encoder's
  input surface is released when the encoder does not start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: logs requests retry three times, then report failed; no replay f…
805346a 
…rames

- A logs request that fails (claim, upload or a server error) is tried again after 2 and
  8 minutes, at most three times. After the third failure the SDK reports
  /event failed with the reason and does not collect the logs again, however often the
  request comes back with a ping or over the websocket.
- The logs bundle never includes replay frames, also when the request asks for replays:
  they are screenshots taken in the background without masks. The bundle reports the
  other sources as before, so the server and the agent see that replays were not sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: in-app messages that arrive during a capture show after it
187b837 
The server marks in-app messages as sent when it pushes them, so dropping them while the
customer captures the app lost them. They now wait (at most 20, for 10 minutes) and are
shown once the capture is over and the widget is closed again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: capture edge cases from the stability review
613a9ef 
- Turning captures off during a capture opens the widget again without the screenshot it
  takes for tickets.
- A capture result that is not delivered (the widget was not opened again) is freed after
  its time to live instead of staying in memory until the next ping.
- The preview's video asks for no audio focus, so it never pauses the customer's music.
- The capture bar starts its attach attempts over for a new screen and after giving up.
- The timer, the host and reopen checks and the posted results catch their errors: an
  error is reported instead of crashing the app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
chore: capture demo shows a card number and a one-time code field
a5481fb 
Both are masked in screenshots and recordings (autofill hints).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: a recording that is given up is released on its encoder thread
0ecbf11 
The stop watchdog released the encoder and the muxer on a thread of its own while the
encoder thread could still be using them (a slow, not a stuck, encoder), which can crash in
native code. The watchdog now deletes the file and drops the result; the encoder thread
skips finishing the file and releases the encoder, its surface and the muxer when it gets
there (never the main thread).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 1, 2026
Merged
Merged
Merged
Merged
Merged
boehlerlukas and others added 7 commits October 1, 2026 07:25
@boehlerlukas @claude
feat: minimal draggable capture bar and full-screen recording preview
fccdf16 
One visual language with the web bar: a dark pill (#16161A at 97 %, white 10 %
hairline, soft shadow), fully rounded, sized to its content and never wider than the
screen minus 32 dp.

Bar
- Content 44 dp high: a six-dot grip, the main action (Capture / Start, a fully rounded
  36 dp pill in the widget's color, white with dark text before the configuration is
  known) and a circular x; while recording a red dot pulsing between 40 and 100 %, the
  time in tabular digits and a circular red Stop (icon only, its label for accessibility,
  a 48 dp touch target). While the SDK works (taking the screenshot, finishing the
  recording) the main button shows a spinner in place of its icon; nothing moves.
- The hint is a small caption above the pill (below it when there is no room): same dark
  style, 13 sp, at most two lines and 280 dp, gone after 4 s or the first drag or tap. It
  is its own window that takes no touches; TalkBack announces it when the bar appears.
- Dragged from the grip or any part of the pill that is not a button; it follows the finger,
  stays in the safe area (system bars, display cutout, keyboard; 8 dp margins), snaps to
  the horizontal center within 24 dp and keeps its place (a point in the safe area) for the
  rest of the process, so the next capture starts there. It is placed again in the new safe
  area after a rotation, a resize or the keyboard. Default: bottom center, 16 dp above the
  navigation bar.
- Since API 34 only the pill (and Stop's target) takes touches: next to it they reach the
  app. Still on the topmost window (above dialogs and bottom sheets), never captured.

Preview
- Full screen on #0B0B0C; the video centered with 12 dp corners and a hairline, tap to
  pause or play (a play glyph while paused) and a thin scrubber; x at the top end, the
  title as a small centered caption; Retake (white 12 %) and "Send >" (icon after the
  label, mirrored right to left) as pills, a full-width row on phones, at the end on
  tablets. While uploading Send shows a spinner and the percentage and everything else
  waits; an error shows above the buttons.

Every visible or spoken string comes from capture-start.labels (English fallbacks for
older widgets): barStart is "Start" now, new keys barDragHint ("Drag to move"),
previewPlay ("Play") and previewPause ("Pause"); the windows' titles are labels too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: the recording time reads left to right in RTL layouts
77c9f63 
Right to left the bidi algorithm swapped "00:04 / 00:20" into "00:20 / 00:04".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: the recording preview comes back after a rotation
4c31af8 
The new activity's window is not attached yet when it resumes, so the preview was not
shown again and the customer was left without it. It now tries again shortly (like the
bar) until the window is there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: keep recordings out of the cache and delete every one the SDK le…
1ea0bdb 
…aves

On a device low on storage the system cleared the cache, recording included, between
Stop and Send (seen on the emulator: the preview stayed black and Send would fail).
Recordings are now written to no-backup app storage, which the system does not clear,
so the SDK deletes each one itself: sent, retaken, cancelled, failed, given up by the
watchdog, Gleap.close() and capture turned off all go through one place. When the SDK
starts, every file in the folder that does not belong to the running capture is deleted
(a process that ended while recording or sending leaves one).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
fix: x in the recording preview cancels a running upload
6860010 
Like iOS and the web bar: x stays enabled while the recording uploads and cancels it (the
request is aborted, the file deleted, /event released and capture-state cancelled, the
widget opens again). Retake and the video still wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
feat: Retake and Send icons like iOS and the web SDK
42fec9f 
Retake gets a leading counter-clockwise arrow and Send the web SDK's outline paper plane
after its label (Lucide "rotate-ccw" and "send", 24 x 24, stroke 1.8, round caps), drawn
from their SVG path data by a small parser (M, L, H, V, C, Q, A, Z).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas @claude
docs: release notes for capture requests
6c4a925 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boehlerlukas
boehlerlukas merged commit 001229d into master Oct 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@boehlerlukas