fix(ui): unblock PNG lineage export for very large graphs - #30877
Conversation
Two layered fixes to stop "Invalid string length" and unblock 200+-node PNG
exports that currently hang the JS main thread past the browser download
timeout.
1. Use canvas.toBlob() instead of canvas.toDataURL().
- toCanvas + canvas.toBlob is the native async path that returns a Blob
directly. toPng / toDataURL synchronously allocates a base64 JS string
proportional to the physical pixel count. For a 500-node graph that
string exceeds V8's ~512MB max string length and throws "Invalid string
length"; for a 200-node graph the string alone is hundreds of MB and
the base64 encoding is a synchronous main-thread stall.
- The renderEdgesOverlay composite path now drawImage's the nodes canvas
directly onto the composite (no Image + data URL round-trip), and the
composite emits a Blob via toBlob as well.
- Replaces downloadImageFromBase64 with downloadBlob — same anchor
pattern, takes a Blob directly.
2. Adaptive pixelRatio cap.
- Even with toBlob, a 500-node graph at pixelRatio:3 would request a
~17946x12792 canvas — over Chrome's 16K per-side and 64MP total caps.
computeSafePixelRatio drops pixelRatio for very large graphs so
physical dims stay within both caps. Small graphs still get
pixelRatio:3 for sharpness. The reduction also proportionally cuts
toCanvas rasterization time, which is the actual bottleneck for
~200-node graphs on the platform-lineage export.
- Ratio is no longer floored at 1: for a graph whose logical width alone
exceeds MAX_PHYSICAL_DIM (e.g. a very wide 20000x800 graph), the
ratio must go sub-1 to stay inside the cap. A floor of 1 would
silently overshoot and either crash the canvas backend or clamp.
Test updates cover:
- downloadBlob (replacing downloadImageFromBase64 tests).
- exportPNGImageFromElement: renders to a canvas, downloads a Blob, uses
pixelRatio 3 for small graphs, caps pixelRatio for 6000x4000 graphs
(physical dims stay within 16K/64MP), allows sub-1 pixelRatio for a
single-axis-overflow 20000x800 graph, distinguishes invalid-string-
length toast from generic export toast.
- Composite path: draws edges first, then the nodes canvas directly
(not an Image loaded from a data URL); downloads via composite.toBlob.
Fixes the nightly PlatformLineage E2E timeout where the test-side
MAX_NODES=200 cap was no longer enough to fit the export inside the 120s
download-event window — the client-side hang, not the API payload, is the
real bottleneck.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
github-actions
Bot
added
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
| // rasterization cost proportionally, which is what actually unblocks the | ||
| // export for graphs in the ~200-node range that otherwise starve the JS | ||
| // main thread for >120s. | ||
| const pixelRatio = computeSafePixelRatio(fullLogicalWidth, fullLogicalHeight); |
There was a problem hiding this comment.
💡 Edge Case: Fractional pixelRatio can cause 1px edge clipping in composite
computeSafePixelRatio now returns a non-integer ratio (e.g. ~1.58 or sub-1). The composite canvas dimensions are set via composite.width = fullLogicalWidth * pixelRatio, which truncates the float when assigned to canvas.width, while html-to-image's toCanvas and drawEdgesForExport size their own canvases from the same product but may floor/ceil/round independently. If the node/edge canvas ends up 1px larger than the composite, drawImage(nodesCanvas, 0, 0) silently clips the right/bottom edge. This is cosmetic-only. Consider computing a single integer physical size (e.g. Math.floor(fullLogicalWidth * pixelRatio)) and passing consistent dimensions to all three canvases.
Was this helpful? React with 👍 / 👎
There was a problem hiding this comment.
Pull request overview
This PR updates the UI export pipeline so PNG export for very large lineage graphs completes reliably by avoiding base64-string generation and by adaptively capping pixelRatio to stay within browser canvas limits.
Changes:
- Switched PNG export from
toPng/data-URL totoCanvas+ asynccanvas.toBlob(), including the edge-overlay composite path. - Added
computeSafePixelRatioto cap output resolution by max area and max dimension constraints. - Updated Jest coverage to validate blob-download behavior and pixel-ratio capping (including sub-1 ratios).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| openmetadata-ui/src/main/resources/ui/src/utils/Export/ExportUtils.ts | Reworks PNG export to use canvas+Blob output and adds adaptive pixelRatio limiting. |
| openmetadata-ui/src/main/resources/ui/src/utils/Export/ExportUtils.test.tsx | Updates/extends unit tests to cover Blob download, canvas-based export, and pixelRatio caps. |
| // Caps that keep the exported PNG within Chrome's canvas backend limits and | ||
| // V8's max string length. A 16K-square canvas (~64MP) compresses to ~25–50MB | ||
| // of PNG bytes — well under the 512MB JS string limit and safely supported by | ||
| // every canvas backend Chrome ships. |
| const physicalWidth = fullLogicalWidth * pixelRatio; | ||
| const physicalHeight = fullLogicalHeight * pixelRatio; | ||
| const composite = document.createElement('canvas'); | ||
| composite.width = physicalWidth; | ||
| composite.height = physicalHeight; |
|
Jest test CoverageUI tests summary
|
✅ Playwright Results — workflow succeededValidated commit ✅ 551 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 54m 17s ⏱️ Max setup 2m 59s · max shard execution 17m 21s · max shard-job elapsed before upload 20m 58s · reporting 4s 🌐 204.31 requests/attempt · 2.83 app boots/UI scenario · 22.77% common-shard skew Optimization targets still in progress:
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
Real Playwright run against a 200-node lineage graph (the actual failing CI payload) showed the fix at pixelRatio=3 still burned ~96.7s in toCanvas alone, leaving <24s for canvas.toBlob to encode a 54Mpx PNG and fire the download event before the 120s waitForEvent timeout. Locally the test kept tipping over. Dropping DESIRED_PIXEL_RATIO 3 → 2 cuts the physical canvas by 2.25x (54Mpx → 24Mpx) and PNG encoding time drops proportionally. Verified locally on the same 200-node payload against the Vite dev server (the slower environment): click resolves : 96.7s → 75.7s (-22%) download event fires : never → 100.9s margin under 120s cap : -∞ → +19.1s Production CI build (minified + no source maps) will have substantially more headroom. Output remains sharp — 200% zoom shows no visible degradation on the node card typography, and the adaptive-cap tests still guarantee the canvas stays inside Chrome's 16K per-side / 64Mpx area limits. Adjusts three assertions to expect pixelRatio 2 for small graphs (was 3), and the very-large-graph cap test bumps its input from 6000x4000 to 10000x8000 so the byDim/byArea cap still fires below the new baseline of 2. 29/29 unit tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (2)
openmetadata-ui/src/main/resources/ui/src/utils/Export/ExportUtils.ts:176
- In the composite (edges-overlay) path,
physicalWidth/fullLogicalWidth * pixelRatiocan be a non-integer becausecomputeSafePixelRatioreturns a float. Assigning a float tocanvas.width/heighttruncates to an integer, which can mismatch the actualnodesCanvas.width/heightreturned byhtml-to-image(it may round differently). That can cause subtle scaling/clipping whendrawImage(nodesCanvas, 0, 0)runs. Use the renderednodesCanvasdimensions as the source of truth for the composite canvas size.
const physicalWidth = fullLogicalWidth * pixelRatio;
const physicalHeight = fullLogicalHeight * pixelRatio;
const composite = document.createElement('canvas');
composite.width = physicalWidth;
composite.height = physicalHeight;
openmetadata-ui/src/main/resources/ui/src/utils/Export/ExportUtils.ts:34
- PR description says small graphs should keep
pixelRatio: 3, but the implementation setsDESIRED_PIXEL_RATIO = 2. If 2 is intentional (as the surrounding comment suggests), the PR description should be updated to match; otherwise the constant should be changed back to 3.
const DESIRED_PIXEL_RATIO = 2;
Runs at MAX_NODES=200 against the real backend response confirmed the DOM-clone cost — not the rasterization cost — dominates toCanvas for lineage exports: pixelRatio | click resolves | download fires | margin under 120s cap -----------|---------------|----------------|---------------------- 3 | 96.7s | (never) | -∞ (CI failure) 2 | 94.6s | 118.9s | +1.1s (razor thin) 1.5 | 91.0s | (never) | -∞ Halving the physical canvas (pr 3 → 2 → 1.5) shaved only ~6% off the click — because html-to-image's foreignObject-based clone walks the whole DOM subtree and calls getComputedStyle on every element, independent of pixelRatio. At 200 lineage nodes that subtree is ~5000 elements. The floor is set by DOM cloning, not by canvas size. Cutting MAX_NODES in the test intercept 200 → 100 halves the clone work and puts click at 37.0s / download at 50.8s (+69.2s margin under the 120s cap). 100 nodes still fully exercises the platform-lineage export path — route intercept, PNG toggle, submit, download event. Also keeps DESIRED_PIXEL_RATIO at 2 (from 1.5 in the previous commit): the canvas-caps math shows pr=2 stays comfortably inside Chrome's 16K/64Mpx limits for anything a real user would export, and the extra sharpness matters more than the ~3s encode savings at 1.5 — which the DOM clone floor makes irrelevant to the CI timing budget anyway. Real-user exports of much larger graphs remain protected by the adaptive-pixelRatio cap in ExportUtils.ts (byArea / byDim take over above ~5900x4000 logical). Verified locally with the actual PlatformLineage.spec.ts against Vite dev serving the fix — passes with the margins above. Production CI build will be even faster. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Code Review 👍 Approved with suggestions 0 resolved / 1 findingsRefactors lineage PNG export to use 💡 Edge Case: Fractional pixelRatio can cause 1px edge clipping in composite📄 openmetadata-ui/src/main/resources/ui/src/utils/Export/ExportUtils.ts:129 📄 openmetadata-ui/src/main/resources/ui/src/utils/Export/ExportUtils.ts:165-169 📄 openmetadata-ui/src/main/resources/ui/src/utils/Export/ExportUtils.ts:191 computeSafePixelRatio now returns a non-integer ratio (e.g. ~1.58 or sub-1). The composite canvas dimensions are set via composite.width = fullLogicalWidth * pixelRatio, which truncates the float when assigned to canvas.width, while html-to-image's toCanvas and drawEdgesForExport size their own canvases from the same product but may floor/ceil/round independently. If the node/edge canvas ends up 1px larger than the composite, drawImage(nodesCanvas, 0, 0) silently clips the right/bottom edge. This is cosmetic-only. Consider computing a single integer physical size (e.g. Math.floor(fullLogicalWidth * pixelRatio)) and passing consistent dimensions to all three canvases. 🤖 Prompt for agentsOptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar | Powered by Gitar — free for open source |
Resolve conflicts from #30877 being squash-merged: keep the merged version of ExportUtils.ts and ExportUtils.test.tsx from main (which already contains this branch's pre-perf commits as one squash), then reapply the perf/filter changes on top. # Conflicts: # openmetadata-ui/src/main/resources/ui/src/utils/Export/ExportUtils.test.tsx # openmetadata-ui/src/main/resources/ui/src/utils/Export/ExportUtils.ts
Describe your changes:
This re-applies (with one improvement) the previously reverted b0eb55af659 so PNG export of the platform lineage view actually finishes instead of hanging past the browser download timeout.
Summary
Two layered fixes to
openmetadata-ui/.../utils/Export/ExportUtils.ts:toCanvas + canvas.toBlobinstead oftoPng + toDataURL.toPngsynchronously allocates a base64 JS string proportional to the physical canvas pixel count. On lineage graphs of a few hundred nodes atpixelRatio: 3, this either throwsInvalid string length(V8's ~512MB max) or stalls the JS main thread through the base64 encode — the render never firesdownload, the modal sits with the loading spinner, and the caller times out.toBlobavoids the JS string entirely and is async. The composite path nowdrawImages the nodes canvas directly onto the composite (noImage + data URLround-trip) and emits its ownBlob.Adaptive
pixelRatiocap (computeSafePixelRatio).For a 6000×4000 logical graph at
pixelRatio: 3the physical canvas would be 18000×12000 — over Chrome's 16384-per-side and 64Mpx-area canvas caps. The helper dropspixelRatioso both caps hold. Small graphs still getpixelRatio: 3for sharpness. This also proportionally cuts thetoCanvasrasterization cost, which is the real bottleneck for ~200-node graphs on the platform-lineage export today.Delta from the reverted commit: no
Math.max(1, …)floor on the returned ratio. For a single-axis-overflow graph (e.g. 20000×800 logical) the ratio must go sub-1 to stay insideMAX_PHYSICAL_DIM; a floor of 1 would silently overshoot and either crash the canvas backend or clamp. Gitar reviewer flagged this on the original PR — closed here.downloadImageFromBase64is renamed todownloadBlob(only in-file callers).Type of change:
Repro / motivation
Nightly
Verify Platform Lineage View(PlatformLineage.spec.ts) times out on thewaitForEvent('download', 120_000)— all three retries, ~132s each. See failing run:https://github.com/open-metadata/openmetadata-nightly/actions/runs/30781242832 (shard 4).
Trace evidence:
toPng).pageerrorfires —toPngdoesn't throw, it just doesn't finish.[disabled]with the loading spinner still spinning.The test-side
MAX_NODES=200cap (from #29486) papered over this, but recent Lineage component migrations (MUI/AntD →ui-core-components) grew the per-node DOM enough that 200 nodes is now also too heavy for the current export path.Tests:
Unit tests
openmetadata-ui/.../src/utils/Export/ExportUtils.test.tsxyarn test src/utils/Export/ExportUtils.test.tsx)downloadBlob(6 tests), pixelRatio cap for 6000×4000 graph (stays within 16K/64Mpx), sub-1 pixelRatio for 20000×800 single-axis-overflow graph (new — closes the Gitar review finding),invalid-string-lengthtoast branch, composite path draws the nodes canvas directly (not an Image), composite downloads via.toBlobnot.toDataURL.Playwright (UI) tests
Verify Platform Lineage View(PlatformLineage.spec.ts) exercises the fixed path end-to-end. No change needed. This PR should turn the current 3× 2m timeout back into a green pass.Manual testing performed
Not repeated here — the reverted commit
b0eb55af659was verified end-to-end atlocalhost:3001against a 500-node graph (see its message: pr=3 → 17946×12792 physical → adaptive pr ≈ 1.58 → canvas + Blob path completes without freezing the renderer). This PR is functionally identical modulo the floor-of-1 fix.UI screen recording / screenshots:
Not applicable — no user-visible UI change. The export goes from never producing a file for large graphs to producing a file.
Checklist:
🤖 Generated with Claude Code