[RNE Rewrite] feat: Add OCR implementation - #1322
Open
benITo47 wants to merge 31 commits into
Open
Conversation
…TC, rectifyQuad, warpByGrid, rotate)
…essive OCR screen
…OcrOptions.tables) Tables were the odd one out — orientation/dewarp had per-run toggles, tables fired unconditionally when a table region was found. Add RunOcrOptions.tables (default true when the model supports them; a per-run false skips the recognizer), gate the table pass on it, and surface a Table-structure sub-toggle in the demo alongside orientation/dewarp.
…ailing | _ glyphs)
Flat-gray strip padding left a content→pad seam EasyOCR's CRNN read as a
trailing glyph. Add OcrModelOptions.recognizerPadMode ('constant' default,
'cornerMean' = background-matched), thread it through the recognizer, default
EasyOCR to cornerMean. Not a decode bug — the argmax was predicting the seam.
…f composed strip Composing a vertical stack into one horizontal strip fed the recognizer a non-word column with its horizontal-word context, making it hallucinate words (and the vertical-digit phantoms). Recognize each glyph on its own and join top-to-bottom — drops the word bias, gives per-glyph confidence. Cost: one recognizer pass per glyph (fine for the opt-in vertical path).
Vertical reads skip the horizontal drop-score gate because they run lower-confidence. Expose RunOcrOptions.verticalMinConfidence so callers can raise it to filter the low-confidence hallucinations the vertical path emits.
Hardens the pipeline — encodeConstraint variant handling, snapUpDim lattice clamp, span-aware table fill, tagged shape-constraint union, dead compose-mode removal, JSI length guards and demo polish. Also pins the document pipeline to the now-published 0.9.0 model tag.
…chema contract Replaces the removed get_dynamic_dims_/get_enum_shapes_ companion API with validateSpec over model.schema: - detect/recognize input sizes are now per-dimension ConcreteDim domains, so one snap routine covers constant/range/enum and the CoreML enumerated grid needs no cross-dimension shape set. - the recognizer's width-to-CTC-timestep relation is read from the linear runtime constraint the model declares instead of inferred as a width/timestep ratio, which was wrong for EasyOCR's CRNN (512 = 4*127 + 4, not a multiple). - TextBoxExtractor declares the detect output layout it decodes, so the allowed spec covers the outputs instead of wildcarding them. - documentModels validates its four static methods in one spec; repeated symbols now enforce the vocab/hidden/feature agreements the manual checks did. - useOcr resolves its nested models through the single-pass resource download.
…HF repos The four OCR repos now follow the MODEL_SPEC layout the rest of the ecosystem uses — per-backend directories, a root config.json, and <model>[_<size>]_<backend>_<precision>.pte file names — published under v0.10.0, so they share NEXT_VERSION_TAG with the other rewrite models instead of the unprefixed 0.9.0 tag they had. Precision is now part of the path, so each preset spells out what it actually ships rather than hiding it behind a bare backend name.
PP-DocLayoutV3's RT-DETR head is set prediction, so the preset runs it with NMS off — but it still emits nested duplicates, e.g. a region for one line inside the region for the paragraph that contains it. Every region is OCR'd on its own crop, so those duplicates transcribed the same text twice: a page with "Lekcja / Indywidualna" came back as three blocks, one per nesting. IoU suppression cannot fix this (a small box inside a large one has low IoU), so the regions are now merged on containment, as PaddleX's `layout_merge_bboxes_mode: 'large'` does: largest first, drop any region a kept one already covers by >=80% of its own area. Visual regions never suppress, so a caption inside a figure is still read. It also stops recognizing those crops twice.
…very OCR backend The OCR and Model Inspector screens never read the safe-area insets the other eight screens already pad by, so "Run OCR" sat under the navigation bar, and the main menu was a fixed View whose tenth button was clipped by it. The menu scrolls now, still centered while it fits. The OCR screen also filtered its model list by platform instead of disabling what the platform cannot run, so CoreML was invisible on Android. All six variants are listed now, with CoreML disabled off iOS and Vulkan disabled on it, matching every other screen.
Brings the OCR pipeline in line with the conventions that landed on rne-rewrite while this branch was open. - The three OCR-specific native ops were called straight off `rnexecutorchJsi.cv` from inside the task files. add-native-extension requires task-specific ops to be wrapped in `src/extensions/<domain>/utils/<name>.ts` and re-exported from the domain index, the way vadUtils/supertonicUtils/privacyFilterUtils are, so power users can reach them: new `cv/utils/ocrUtils.ts` with typed, documented `extractCraftTextBoxes` / `extractDbnetTextBoxes` / `ctcGreedyDecode`. - Five bare `throw new Error` (geometry, detectors, documentModels, quad ops) now carry codes: INVALID_ARGUMENT for a caller's bad input, INVALID_STATE for calling table recognition on models loaded without a table config. Documented with `@throws` on the entry points that can raise them. - `opts` parameters renamed to `options` (#1358): `makeCraftExtractBoxes`, `makeDbnetExtractBoxes`, `rectifyQuad`. - `OcrModel.ocrOpts` -> `modelOpts`, matching every other task's model type. - Tensor locals carry the `t` prefix the pipeline skill asks for.
Every host function this PR adds to C++ threw raw `jsi::JSError` and was registered as a bare `fnBody`: all three ops in ocr_ops.cpp plus rotate, warpByGrid, crop and rectifyQuad in image_ops.cpp. Both are mandatory rules in the error-handling skill, and the consequence is real: without `error::guarded` the failure reaches JavaScript with no `code`, and the synchronous worklet path VisionCamera frame processors use has no `wrapAsync` fallback to put one back. Usage, shape and argument failures now raise `error::InvalidArgument`; OpenCV and dtype conversion failures raise `error::ExecutionFailed`. All seven registrations are wrapped in `error::guarded`. ocr_ops.cpp gains the `core/error.h` include and the `namespace error` alias at file scope, as in box_ops.cpp.
Per the discussion on #1359: RNE ships the composable building blocks, and document-grade OCR becomes a separate package on top of them. That leaves this pipeline at what the PaddleOCR reference SDK does, detect then recognize, and nothing else. Removed: - documentModels.ts entirely (orientation classification, UVDoc dewarp, SLANet table structure recognition + the autoregressive decode loop and HTML cell fill), plus the PP_HELPERS registry and the SLANet vocab. - The layout stage: the PP-DocLayoutV3 object detector composition, document blocks, region containment merging, and the PP_DOCLAYOUT registry with its label space. - Vertical text: stacked-glyph grouping, per-glyph column reading and the char-level re-detection path, with the RunOcrOptions they were toggled by. - Quad helpers left without callers (splitTallQuad, boundingQuadOf, quadFromBounds). `runOcr(input)` now takes no per-run options and returns `OcrDetection[]` in reading order rather than a result object, since the corrected-frame and blocks fields only existed for the stages above. The demo screen loses its four toggles and the table renderer. Kept: both detectors (CRAFT and DBNet), the fused detect/recognize models for all 9 presets, the get_model_schema contract with the recognizer's linear CTC constraint, reading order, and the custom decode hook. Net -1615 lines. The removed features stay recoverable from the tree linked in #1359 and their exporters remain on the export-scripts branch.
msluszniak
reviewed
Aug 13, 2026
msluszniak
reviewed
Aug 13, 2026
- std::format for every error message in the new image ops - explicit capture list on warpByGrid's grid sampler - size_t/range-for over the quad corner loops, drop the index casts - geometry Box is now BoundingBox<'xyxy'> + angle instead of four loose floats, so rotateBox reuses boundsOfPoints - distance/interpolatePoint move to ops/points, deduping the copies in quad.ts and geometry.ts - drop the unused processTensor added for the removed layout stage - document the decode callback's params and why detectorNorm is set here
msluszniak
force-pushed
the
ocr_dynamic
branch
from
August 13, 2026 08:34
c13e3f5 to
a7823b6
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
- remove rotate, crop and warpByGrid: added for the orientation/dewarp stages that the scope-down deleted, no call sites left in the library or the demo app - remove the CRAFT charLevel branch, unreachable since the per-glyph reader went away (makeCraftExtractBoxes never exposed it) - return the three OCR ops over Float32Array instead of per-element jsi::Array writes, per the boundary convention from #1332 - std::format in ocr_ops.cpp, matching image_ops.cpp - ppOcrV6 mirrors easyOcr's shape, dropping the positional tuple cast - prune the 8 cspell words left behind by the document pipeline - MIN_BOX_SIDE -> MIN_RECOGNIZABLE_SIDE, distinct from DBNet's decode threshold of the same value
The recognizer charsets were inlined in the bundle, so every app importing the model registry carried ~86 KB of CJK tables whether or not it ran OCR (PPOCR_SYMBOLS alone is 18709 entries). They are already published next to every .pte as charset.txt and are byte-identical to the inlined copies, so point `charsetPath` at them and let the resource fetcher resolve it like any other model file: no exporter or fetcher change needed. `modelOpts.charset` stays as an optional inline override for custom exports whose charset is not published; supplying neither raises INVALID_ARGUMENT. A mismatched pair still fails loudly at load, since resolveOcrContract requires charset.length to equal the recognizer vocab. Measured on the computer-vision release bundle: 4,830,700 -> 4,770,548 B.
…hich for quality Its Vulkan detector runs fp16 on the GPU because int8 cannot resize there, so it is several times slower than the int8 XNNPACK one. What it buys is accuracy above the ~800px band the int8 detector is calibrated for, which is the opposite of what picking "the GPU backend" suggests.
The published artifact is now static int8 for detect (fp32 recognize), which is measurably the same quality and ~15% faster end to end on a Galaxy S26 Ultra, at 23.9MB instead of 31.1MB. The precision token is part of the model URL, so the registry has to follow the rename. Also documents that Vulkan is the faster PaddleOCR backend, which holds for a different reason than before: its detector is small enough that the fp16 GPU path wins outright, where EasyOCR's VGG16-backbone CRAFT loses to int8 on CPU.
msluszniak
marked this pull request as ready for review
August 13, 2026 14:26
The OCR task reached for react-native-blob-util directly to read the charset published beside the model. Move that behind readTextFile in the fetcher module, which already owns every other filesystem access, so the task layer depends only on the resource API. The wrapper also stops blob-util's Promise<any> at the boundary.
msluszniak
reviewed
Aug 13, 2026
msluszniak
left a comment
msluszniak
left a comment
Member
There was a problem hiding this comment.
All 15 points from the previous round are resolved. I also checked the new fetch path end to end: all eight EasyOCR charset.txt files, the PP-OCRv6 one and the retagged pp_ocrv6_xnnpack_int8.pte return 200 on the v0.10.0 tag, and the payloads really are JSON arrays of strings. collectRemoteSources picks up charsetPath automatically and download weights progress by byte size, so the sidecar does not skew the progress bar. Typecheck, ESLint and clang-format are clean.
Four small things left, none blocking.
…able one Addresses review on #1322: - The published charset is a JSON array of strings, so name it charset.json. Renamed on both HF repos and re-pointed v0.10.0; better now than after the release, when the URLs are load-bearing. - Failing to READ the charset is the re-fetch-the-asset case that model.cpp already reports as LOAD_FAILED; only malformed content is the caller's argument to fix. Split the try so each maps to its own code. - modelOpts.charset now documents the same indexing convention as charsetPath. - The filename precision tag names the DETECTOR only, so say that rather than leaving pp_ocrv6_xnnpack_int8.pte reading as a contradiction.
Member
|
Two comments for you @barhanc:
|
Member
Yes, let's drop the EasyOCR and make the pipeline PaddleOCR specific, instead of trying to cram two different pipelines into one. |
Member
|
@barhanc advantage of easyOCR comes from better vertical ocr feature. So I will keep export script and the model itself. I just wondering how we would add support for these features without breaking changes in the core package. |
Member
|
We can implement the code for OCR in separate package directly on top of rn-executorch. The TS code will just use our TS API and we can add ocr-specific native code in that lib that can directly include the rn-executorch headers (same way as we include JSI) for all the TensorHostObjects, etc. definitions, so I don't think there is any problem in this regard. |
Member
|
Ok, if we will use our core native code then I think it's fine. |
PP-OCRv6 is faster and reads better on ordinary pages, and without vertical OCR there is nothing EasyOCR does better, so core carries one detector family instead of two. EasyOCR comes back in a dedicated OCR package later. Removes the 8 language presets and, with them, everything only they used: the CRAFT extractor and its native decode, the component-box grouping and de-skew geometry, and the cornerMean strip padding. The published artifacts are untouched, so reintroducing EasyOCR is a matter of restoring this code. Also fixes the postinstall lib map, which still described OCR as CRAFT+CRNN on xnnpack only. PP-OCRv6 ships xnnpack, coreml and vulkan, so an app opting into the ocr feature was not getting the backends its models need.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a unified, OCR and document understanding pipeline to react-native-executorch. Two OCRs - EasyOCR and PP-OCRv6, plus a higher-level document pipeline that orchestrates orientation correction, UVDoc dewarp, PP-DocLayoutV3 region layout, SLANet table-structure recognition, and reading-order assembly into HTML.
Introduces a breaking change?
Type of change
Tested on
Testing instructions
Try OCRing on different images, use different OCR options. See if anything is ill behaved.
Screenshots
Related issues
Checklist
Additional notes