Fix/kgsl present wait fence - #96
Conversation
Keep deferred submit objects alive until KGSL has consumed their command lists and populated the shared kernel timestamp. This prevents BO teardown from freeing GPU addresses that the merged submission still references.
|
In my testing with an Adreno 730 device, when the graphics API of Minecraft Java Edition is OpenGL, the version built by this PR fixes the game screen flickering issue compared to the version before modification. However, the scores of It is recommended to keep the PR status as "draft" so that people with similar needs can obtain the build artifacts more easily. |
|
Should I investigate a solution which doesn't plummet the performance of EDIT: This does not fix the artifacts in #44 |
Currently, it seems that the flickering issue is particularly severe on the Adreno 740, and I am unable to reproduce this issue on either Adreno 730 or 830. |
|
Just some more updates about the 740 situation. It seems he following now render correctly: Firefox with GPU acceleration, OpenXray, OpenMW, OpenJK, reVC and etLegacy (alongside of Minecraft). No more flickering, albeit performance is reduced. However, it seems to be better than the zink variant. I'll run some numbers when I get some more time. |
|
So I tried the #96 build artifact on an Adreno 740 (Termux:X11, Debian trixie proot).. With it installed, my 32-bit OpenGL application running under box64 kills Termux:X11 within a few seconds under kgsl. logcat shows Scudo exhausting 33 size classes in sequence, from 48 bytes up to 131088, over about 7.4 seconds, in the LorieNative process - popped counts in the hundreds of thousands with pushed at zero, and rss 0K on every region. Controls: the same application under zink is fine, glxgears and glmark2 under kgsl are also fine and the same application under kgsl on the unpatched 20260709 release produces no scudo output at all. IIt' might be worth noting that 'm running stock Termux:X11, not the fc534e5 build. Is that commit required for #96 or should the Mesa half work standalone? Appreciate your help. |
|
Yes, build I may be able to upload the necessary APK here if it would be easier for you; EDIT: Here it is, of a different commit but it contains the fix: |
|
Superb reply and thank you. Your post explains the log better than a plain leak would - the allocation growth accelerated across 33 size classes in about 7 seconds rather than climbing steadily in one and every region reported rss 0K, which fits requests being duplicated rather than filled. It also fits my controls: glxgears and glmark2 under kgsl were both fine on the same build and they presented at a steady cadence. The APK is super helpful as I have no Android toolchain set up here. I'll reinstall the #96 Mesa build alongside it and report back on whether the crash goes away and what the artefacts do. Thanks again! |
|
A disappointing night's testing.. 😑 With the APK installed, alongside the #96 Mesa build, the crash was numerically identical to the stock Termux:X11 run: 33 Scudo OOM aborts across the same size classes, 1353 scudo lines and about 7 seconds from the first abort to the X server going down. Both halves were verified before the run.. the installed base.apk hashes to the file you'd kindly uploaded (5171db856d00788452223b33538a5608) and glxinfo reported git-91f7e8c6f0. My controls were left unchanged: zink is fine, glxgears and glmark2 under kgsl are fine and the unpatched 20260709 Mesa produces no scudo output at all with the same application. So the multiplication doesn't appear to be the callback path fc534e5 fixes or there's a second route to it. If you've got any ideas I can absolutely run anything that would narrow it down. I really thought this might fix it! |
|
Oops, I seem to have uploaded an experimental file then, and unfortunately it was the wrong APK. Your checksum confirmed that the attachment contained the prepatch backup, so your test never exercised the callback fix. I am sorry for wasting your testing time. Here is the corrected archive. Its ZIP MD5 is 707143af560e0b6639d39cd43800eace, and the APK inside should have MD5 04bf47a9794b123a379ef2b13b1a730a. Could you please repeat the same controlled test with this one? |
|
You've got absolutely nothing to apologise as far as I'm concerned. I'll download the new file, test it as soon as I can and then report back - thanks again for all your help. |
|
The corrected APK fixes it completely 😎 With your build (md5 04bf47a9794b123a379ef2b13b1a730a, verified against the installed base.apk) plus the #96 Mesa artifact (glxinfo: git-91f7e8c6f0), zero scudo lines and zero Scudo OOM aborts, against 1353 and 33 before. It ran the game code without crashes, compared to dying at seven seconds before. The black corruption bands are gone too. Those are the artefacts in lfdevs #99. I'd partly assumed they were unrelated given #96 says it doesn't address #44. Frame rate, 30-second sampled window on a fixed attract demo, Adreno 740: 52.58 fps silent, 47.38 with audio. My pre-#96 figures on the same scene were 54.4-56.6 and 47.0-47.8, so within noise on the configuration that matters. The ~20% you measured on glxgears doesn't seem to transfer to this workload. You've done some great work here! |
89da277 to
91f7e8c
Compare
|
That's great news! I'm glad it works, since yours is now the first KGSL wait-fence done in the wild. I've also opened a PR for the Termux:X11 situation upstream, so that it will be included in future builds. It looks to me like the Termux side is an upstream bug from Xorg, so at least we'll have that one merged. |
|
Very happy to hear the Termux:X11 side is going upstream - I found termux/termux-x11#1114, so I'll follow that one. My own Termux:X11 build carries a small local patch, so knowing which tree it lands in is useful here. One thing on 89da277 from reading it rather than running it - fd_bo_handle() now returns bo->funcs->kms_handle(bo) whenever the hook is present, and .kms_handle is registered unconditionally in kgsl_bo.c's bo_funcs. kgsl_bo_kms_handle() falls through to kgsl_bo_dmabuf(), which returns -1 unless the BO is KGSL_BO_IMPORT with a valid import_fd. So, for a shared BO that KGSL allocated itself, fd_bo_handle() returns 0 where it previously returned bo->handle. Neither FD_KGSL_USE_KMS_DUMB nor MESA_KGSL_X11_SHM_BRIDGE gates that. Is it meant to be unconditional? Separately, if it's of any use to you: I can price the SHM bridge on a real workload. My case is a 2005 arcade game under box64 at 1280x720. It's around 780 draw calls per frame with 94% of the frame on the main thread and the GPU idle 69% of the time. A full-surface copy plus a synchronous round trip per frame all lands on that main thread, so glxgears will show almost nothing of what that path costs whereas mine will. A run-to-run spread on my bed is 0.70 fps, so the number would be a real one. I'd be happy to run it with the flag on and off if you'd want that and it'd benefit me too. |
|
What I'm about to share here cuts against my own points above.. I measured #96's cost on glmark2 properly this time - same boot, A-B-A, version printed on every run: baseline (9452d1d) #96 (91f7e8c) 48-61%, roughly three times the ~20% you saw on glxgears. So my "doesn't transfer" line was true of the game and not of the benchmarks. This pair of results together is more useful than either alone imho: the cost is large on trivial thousand-fps benchmarks and unmeasurable on a real workload On the game, it's 52.58 silent / 47.38 with audio against 54.4-56.6 and 47.0-47.8 before #96, with a 0.70fps run-to-run spread. So if #96 is being held on performance grounds then benchmarks may be the wrong instrument for that. Separately: the sysmem corruption I reported as lfdevs #99 is gone on #96, across four glmark2 scenes. I confirmed that is a real fix rather than the lower frame rate hiding it - the baseline build still corrupts at 904fps, |
|
Sorry for the delay. It seems the Did you happen to find any cadence issues in your testing? Did you notice any frame timing issue? I used a USB capture card for my device, and an SDL2 Gray-band encoder to figure out whether frametimes were consistent, and they were sadly not, but these tests were done on the native X session. I got a bit distracted on this topic 😄 |
|
Ha! Distractions are the eternal problem, believe me I get it.. and thanks for the fd_bo_handle answer too. On cadence - yes, frame times are inconsistent here too. Within a single 30-second window on the attract demo I see per-frame times ranging from about 2.6ms to about 37.7ms against a 19.6ms mean. I can't cleanly separate presentation jitter from workload, though: this game's cost tracks draw-call count and the scene varies a lot and my frame-time counter is in-process rather than captured, so only the minimum and maximum are trustworthy. More usefully, I'm confident that there's a visible stutter under kgsl that I don't see under zink, same build, same scene. It's brief and always happens at the exact same points. That's observational only, no attribution. On benchmarks: zink and freedreno now measure level for me, 51.0 fps each with sound at 1280x720 (even though zink spends far longer in the swap). Is the X transport work likely to apply to the Termux:X11 path or is it native-Xorg only? (Fingers crossed you say X11 here..) |
|
I have been playing around with some fixes, and I found out that part of the performance drop comes from flushing the outgoing back buffer twice. The fenced swap already flushes it, but I have a local fix which avoids that duplicate flush when the outgoing buffer was already successfully fenced. The fences themselves remain in place, and failed fence exports still take the original fallback path. This applies to I rebuilt
These were uncapped tests, with So this recovers quite a bit of performance, but not all of it. Compared to the PR alone, As for native The pixel checks passed, including resizing, MSAA and context switching. The SDL/SurfaceFlinger cadence test also stayed around I haven't pushed this into the PR yet. There is still a performance gap to investigate, and I would like to profile that before adding anything else; So far so good, though! |
A native KGSL BO must keep its owning-device handle even when dma-buf export is unavailable. Document that contract and add a hardware regression test covering repeated lookup before and after sharing and export attempts. The old KMS-handle override is absent from this branch, so no runtime handle replacement is needed. The test submits no GPU work and skips when KGSL is unavailable. Validated with host build/skip, AArch64 build, and a passing native Adreno 740 run using a non-exportable BO.
The explicit Present-fence path flushes the back attachment and exports a fence before passing the image to the loader. At the next image rebind, dri2_allocate_textures flushes that outgoing resource again. On KGSL the explicit-present context flag has already been reset, so this second flush falls back to a synchronous CPU-read wait for GPU completion. Record when a swap successfully flushed a back attachment with an exported fence. Skip only that attachment's redundant replacement flush, consuming the marker on replacement or when validation hands attachments back to the state tracker. Clear it at the start of every drawable flush and exclude shared-buffer mode. Other attachments and unfenced/failed exports retain the existing path. GPU fences and both submission/Present workers remain unchanged. On an Adreno 740 with Termux:X11, alternating clean control/candidate runs improved uncapped gears from about 1000 to 2100 FPS and high-poly glmark2 bump from 720-732 to 1293-1328 FPS. Actual X pixels were checked after Present completion with repeated resize/recreation, shared-context switches, 4x MSAA, and front-buffer rendering. This remains a local review candidate; export-failure fault injection, shared-buffer mode and other drivers have not been runtime validated. (cherry picked from commit 2c0dcd26feaaa2a4f727fdb3c978f3b851ff7ffe)
A locally submitted native fence retains the userspace completion marker. Check that marker before entering sync_wait: KGSL kernel traces show that sync-file signaling can lag command retirement, and profiles measured waits even when the GPU marker had already completed. Use the existing zero-timeout fd_pipe_wait_timeout check. Imported native fences have no userspace sequence and retain the sync-file path, as does a wrapped zero sequence. On Adreno 740 with the PR96 fence path and duplicate-flush fix, isolated high-poly tests improved from 1280 to 1768-1843 FPS. Combined validation recovered uncapped glxgears to within 0.6% of the release and improved the full glmark2 score by 10.5%. Imported pending-fence and pixel checks passed. (cherry picked from commit 64dfdcc29dfc3a9553d5072536c459887ce21672)
Present can skip a pixmap and release it while its render-fence worker is still pending. Reusing the per-buffer X Sync fence at that point allows overlapping trigger/reset sequences, observed as BadMatch from ResetFence and a stalled client with disable_throttling enabled. Track worker completion with a per-buffer util_queue_fence. Wait before resetting and requeueing the X fence, and before freeing its buffer. Drawable shutdown already drains the queue. This also replaces the queue-wide wait when freeing one buffer. Validated on Termux:X11 with a 30-second high-poly stress run that previously stalled, pixel checks with normal rendering, MSAA, resizing, front updates and context switches, and SurfaceFlinger cadence at 60.13 Hz with no intervals over 25 ms. (cherry picked from commit b11704292b6bc4b5e251b5f62adb50f016c4be35)
With the render-wait bridge active, two COPY-mode buffers can both remain held during render completion and server copying. Profiling Termux:X11 refraction showed approximately 30% of main-thread time waiting for an available buffer, versus 5% in the unpatched release. Allow one additional buffer on demand for COPY presentation at swap interval zero when present_sync is active. Synchronized swaps retain the existing limit. FLIP and SKIP policies are unchanged. A buffer-depth ablation scored 661/587 with two buffers, 988 with three and 992 with four. The clean three-buffer build, combined with the previous fixes, averaged 2582 FPS in glxgears and 2128 in glmark2 versus release 2282/1842.5 in A/E/E/A runs. Pixel, MSAA, resize, context-switch, queued-frame, imported-fence and cadence checks passed. (cherry picked from commit 609e727193cb42ad9b56baa2e978e18e145aa676)
|
I think I got it figured out, at least for the benchmark regression 😄 I continued profiling after the duplicate-flush fix, and there were a few separate things contributing to it:
With all of these together, the fresh comparison is:
That is about Pixel, MSAA, resize, context-switch, queued-frame and pending imported-fence checks passed. SDL/SurfaceFlinger presentation remained around I pushed the fixes as separate commits to this PR, without the temporary profiling code. They replayed cleanly onto its existing It would be interesting to see what this does on your game workload now, especially whether any of the presentation stutter changes. |
|
Nice turnaround on that regression :) A datapoint from a real workload, since you asked. I'm running OutRun 2 SP SDX under box64 in a Debian proot on Termux:X11, 1280x720, Adreno 740. When #96 went in, my frame rate didn't measurably change - the cost you measured on glxgears didn't transfer. I'm confident that's because this workload is CPU-bound rather than presentation-bound: the GPU sits around 21% busy while the main thread runs at about 90% of a core. I'm currently on around 56 fps (silent, sound comes later) on kgsl. On the stutter, to be super-precise about what I'm seeing: at specific, repeatable points in one of the attract demos, the motion stops flowing for a split second and the frame rate visibly dips. It happens at the same points every run, rather than being random. I don't see it under zink with the same Mesa and the same X server. If you can share a build as a tarball I would be very happy to extract into the container, test it and report frame rate, per-second minimums and whether those stutter points change. Appreciate what you're doing here! |
|
That's awesome, I'm really curious how this performs. With the HDMI PR, containing both these fixes as well as the HDMI ones, I saw a subjective performance boost (did not measure yet) in OpenGL games; They feel very smooth and actually playabale now. Looks like we've been blessed with a workflow trigger, which means that you should be able to find artifacts of this build to test now, they were built by GitHub. Thank you for helping out with this, a sample size of two is much better than just one 😄 |
|
So I've read through #101 and #44 properly - both the composer handoff and the plane sanitisation are well past what I'd have guessed at. Your comment yesterday - flashing gone on the leased display, still there on Termux:X11 - has read to me like the two paths have genuinely diverged. Was your "very smooth, actually playable" observation on the leased display too, or on the phone's own panel? It's worth flagging that I'm running on a third configuration you may not have: Retroid Pocket 6, unrooted proot, but output over USB-C to an external monitor through Android's own composer rather than a lease. xrandr inside the session reports a single "builtin" output at 1280x720 / 59.86. The X server has no idea the monitor is there. So the leasing aspect is inert for me, but I'm not the internal-panel case either.. or at least, not until I've got my stack where I need it to be. The monitor is just to make testing and debugging easier before I can reliably start playing on the move. Eight of the commits are in dri3, which every client goes through and the titles read as pipelining work rather than bridge work. If any of that reaches ordinary clients, there's a real mechanism for a gain on my setup. I note that your ordinary-path A/B was short uncapped gears, which runs without vsync and can't show pacing in either direction - so "no meaningful regression" could just mean "not looked at" rather than "no change". Which build do you think I should pull? From my reading, the tested candidate is a522d4e1 but the head is acbc90e. Either way, I have a fixed test bed, a measured 0.70 fps noise floor and per-second frame times on a real game rather than a synthetic, so I can tell you whether pacing moved. Keeping my fingers crossed on all this! |
|
Thanks for explaining your setup, that helps; The smooth/playable observation was on the leased HDMI session, not the phone’s own panel. I forgot to make that distinction, oops. Your setup still uses the ordinary Termux:X11 path between Mesa and the X server, with Android handling the external monitor afterward. So the #96 fixes are relevant without root or leasing. I haven’t tested that exact configuration, though, and mirroring may introduce its own timing effects. Your results would be useful for understanding that part. I also realised my description of the builds was confusing:
For the first test, I’d suggest You’re right that some of the changes reach ordinary clients. Avoiding the duplicate outgoing-buffer flush, checking local GPU completion before waiting on the sync file, and fixing Present fence reuse all apply there. The extra back buffer applies specifically to fenced I should also have been clearer about the distinction between those changes and the HDMI presentation bridge. They share some DRI3 code, but the bridge’s worker and image-copy pipeline are explicitly enabled through environment options. Disabling that bridge does not disable #96’s render-fence handling. Your point about uncapped gears is fair, though; That comparison only checked throughput, not pacing. I did separately run synchronized SDL/SurfaceFlinger checks on the PR-only and combined builds. Both were around There are also some shared changes in #101 that can matter even with its bridge disabled:
That leaves room for further testing, but comparing #96 directly with the current #101 head would also remove the five follow-up commits, so it wouldn’t isolate the effect of adding the HDMI changes. Let me know how it goes, fingers crossed too! |
|
Well I ran it and sadly, here's the short version: There was no measurable change on my setup. 91f7e8c (2 runs) 373b340 (1 run) The swap figure is the interesting one though. It's a mean over ~1,650 frames measured inside the app via an LD_PRELOAD shim around SDL_GL_SwapWindow, and my two before-runs agreed to within 4 µs. The after-run sits 55 µs above both - fourteen times that agreement and in the opposite direction to what the changes intended. It's one run against two, so I'm not claiming it - but it's not noise-shaped either. It's worth you knowing why the ceiling is low here: the swap is only about 1.2ms of an 18ms frame on this workload. I've since split that with a forced glFinish before the swap - about 666 µs of it is presentation work, with the GPU already fully drained. My bottleneck is one CPU thread at 90% of a core with the GPU busy 21% at its lowest clock. Roughly 65% of the frame is the game's own x86 code under box64, not OpenGL at all. So there wasn't much for presentation-path work to recover, no matter what it did. One thing that might interest you though: forcing that glFinish cost 7 fps of throughput but narrowed my frame-time spread from 1.9-33.8 ms to 10.6-22.3 ms. Pacing got markedly more even when the pipeline was serialised. It was a single run, so again not a claim - but it points at variable queue depth rather than the presentation path itself. (Oh and the in-game stutter is unchanged, it's still visibly present) Overall, it's not entirely a discouraging result from my side - it's a null on a configuration that nobody had data for, and the fixes are still correctness work regardless. Let me know if you want the raw counters. One correction to my earlier message: I'd said mirroring, but I'm not confident that's what's happening. Mostly the RP6's own panel doesn't show the same image when a monitor is plugged in but occasionally it does, and I haven't worked out what determines it. So treat my configuration as "Android compositor to an external monitor, mirroring state uncertain" rather than confirmed mirroring. |
|
Interesting results indeed. I have moved onto replicating your setup, summarized in a 32 GB image that I'll mount on my device to assess whether OutRun 2 behaves the same here. I use a fully updated trixie image in an arm64 VM machine for now, but I'll be back with actual results and findings. Maybe we can patch some things to improve performance. Your device seems quite cool though! @lfdevs, would you mind testing on your 730 and 830 devices again? I feel like we reached a benchmarkable point right now with the 740 situation. |
|
I'm back and have two things since your last message - one correction and one measurement you might want to know. The correction first: I said roughly 65% of my frame was the game's own x86 code under box64. That was wrong. My instrument measured frame time minus draw time minus swap time and I labelled the remainder as game code, but it also contains every GL call that isn't a draw or a swap - and there are about 4,200 of those per frame. So the residual is partly driver work, not emulation. And now the measurement: Matched scenes, 512 frames each, same game, same build, one driver argument changed. Timed inside the app around the draw call itself: kgsl: 347 draws/frame, 2,960 µs total in draws — 8.52 µs per draw An earlier pair of windows gave 14x; this matched pair gives 11.5x. Stable, not a snapshot artefact. Total frame time is within 1.4 ms between the two, so freedreno isn't slower overall. It pays about 2.7 ms more inside the draw call and gets about 2.0 ms back in the swap. The cost moves rather than disappearing. Also worth knowing for my benchmarking: the RP6 is a Snapdragon 8 Gen 2 with an Adreno 740 - do I know what device you're using for yours? One difference between us might be that I'm in a Debian proot rather than a rooted chroot, so every syscall goes through interception. On the stutter - I have per-frame data now and it's less clean than I thought. Slow frames carry roughly the same draw count as normal ones, so they're stalls rather than workload spikes, but both drivers show them and they have different shapes: zink's are almost entirely in the swap, kgsl's are spread across draw, swap and the rest. Possibly two separate causes and I'll carry on working on this. Let me know if there's anything else you need from my side. |
|
That's quite an interesting result. My device is a Sony Xperia 1V, so my SoC is identical to yours. At the moment I'm trying to figure out some rendering glitches for the game, as it currently looks like this: 
I'll try to get it running with chroot, then I'll try the proot method to gauge whether proot affects the situation linearly. The next step on this topic is to get the game running and attempt to replicate your results, so that we can debug this further on a semi-identical copy. I'll keep you updated, for now it seems like we have all that's needed. Thank you for helping out, this is really interesting! EDIT: Would you mind sharing more details about your lindbergh version and other libraries used? I cannot figure this glitchyness out on my end, and I have a feeling it's related to the issue with the flickering items in HL2. |
3 participants
A Codex assisted patch which attempts to fix reading buffers too early and causing flash-to-black issues in scenarios like Firefox GPU-enabled rendering, Minecraft, reVC and others.
I am not 100% sure what is going on here, but I can confirm this works on my Adreno 740 with KGSL with no significant performance drawbacks (-20% on glxgears compared to unfixed version: 1757 fps original vs 1337 with this patch).
This is accompanied by a fix to Termux:X11 where it does not correctly handle some callback disarming: KiralyCraft/termux-x11@fc534e5
EDIT: I do not completely trust what is going on here, but I can confirm it is reliable on my device. It might be better to rework this for better integration and style adaptation.