Repository navigation
Conversation
This was referenced Nov 14, 2026
…ts (refs #388 and #315) count() walked the whole result with `for _ in self`: one NativeResult per row, built and thrown away. These tests pin the number it returns (the rows the same query gives through to_list(), at sizes around the 256-row first reply and the 2048-row later replies, for typed and wide rows, aggregates, a Cypher result and a command), the state it leaves the result in (only the rows not yet read are counted, an empty or exhausted result counts 0, a closed result or database raises), the error an engine failure met after the first reply raises, and that no NativeResult is built for a counted row. The last one fails on the parent. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lt (refs #388 and #315) count() read the result with `for _ in self`, so every counted row was a tuple popped from the buffer and a NativeResult built to be thrown away. It now adds len() of the buffer and of each reply the cursor gives, through _fill(into) as to_list() does. The replies are still read and parsed (the library encodes every row), which is what the next commit removes. Checked without the library: the old and the new count return the same number and leave the result in the same state (closed, exhausted, no cursor) for 0, 1, 255, 256, 257, 2303, 2304, 2305 and 5000 rows and for an error met after the first reply; the new one builds 0 NativeResult where the old one built one per row. Not measured: the speed claim waits for a timed A/B. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… call, arcadedb_n_count (refs #388 and #315) After the previous commit count() still reads every reply past the first one as a JSON text (arcadedb_n_next_bin: Java encodes each row, Python parses it) just to take len(). These tests pin the target: a result with rows left in the library is counted with one call of the entry point arcadedb_n_count and no other (the library drains and closes the cursor, so no second close is queued), for a query cursor and for a command cursor; a result whose first reply holds every row, and an exhausted one, make no crossing at all; no reply text is read and no NativeResult is built. They fail on the previous commit (the calls are arcadedb_n_next_bin, and the entry point does not exist). The behaviour tests of the first commit stay the oracle for the number, the state and the errors. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…the library as one long (refs #388 and #315) After the previous commit count() still asked the library for every reply past the first one: Java encoded each row into a JSON text (arcadedb_n_next_bin), Python parsed it into one dict per row, and the dicts were only measured with len(). The new entry point arcadedb_n_count(thread, cursor) drains the cursor in Java (hasNext and next, no row encoded) and closes it, and returns the count as a long; -1 is an error whose text arcadedb_n_last_error holds, built by the same errorJson as the "rerr" of a JSON reply, so the exception count() raises, and the state it leaves the result in, do not change. Like arcadedb_n_tx and arcadedb_n_cmd it returns no reply text: one foreign call, no malloc, no arcadedb_free, no decode. count() asks it only when the cursor still has rows: a result whose first reply holds every row, and an exhausted one, make no crossing. The rows are no longer encoded to be counted, so a row the codec could not encode no longer fails a count(); that was an accident of the old route, not a behaviour. Not measured: the speed and memory claims wait for a timed A/B against 26.11.1.post3. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…w, so its facets survive (refs #388 and #315) Main now appends the facets of a FACETS query to the reply that drains the cursor, and a facets() after the rows are read returns them. arcadedb_n_count drains and closes the cursor in the library and carries no facets, so count() followed by facets() returned None on the rebased branch. A result records at creation whether its statement text holds the word FACETS (any case, a string literal included, so a false positive costs speed only); count() reads such a result with the iterating route of main and uses the new entry for every other result. Tests: count() then facets() equals facets() after to_list() on an engine that has the clause (skipped before 26.12.1), a statement that mentions the word makes no arcadedb_n_count call, and a statement without it still makes the single call. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tae898
force-pushed
the
perf/388-native-count
branch
from
November 20, 2026 04:31
31f7738 to
3574efc
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
On the native wheel,
ResultSet.count()walked the whole result: every reply past the first came back from the library as a JSON text (arcadedb_n_next_bin), Java encoded every row into it, Python parsed it into one dict per row, and aNativeResultwas built for every row, only to be counted and dropped. Acount()of a million-row result was about 500 crossings of 2,048 rows each and a million objects built for nothing.Now the rows already in the result object (the first reply, up to 256) are counted with
len, and the rows still in the library with one new entry point,arcadedb_n_count(thread, cursor), which returns along: the cursor is drained in Java (hasNextandnext, no row encoded) and closed there. A result whose first reply holds every row, and an exhausted one, make no crossing at all. This is the rank 2 item of #388.The entry point is the way
arcadedb_n_txandarcadedb_n_cmdalready return a number: no reply text, so one foreign call with no malloc, noarcadedb_freeand no JSON decode.-1is an error, and its text is read witharcadedb_n_last_error, built by the sameerrorJsonthat builds thererrof a JSON reply. So the exceptioncount()raises and the state it leaves the result in (closed, exhausted, no cursor) are the ones iteration gave before; the tests compare them.Two commits are Python only and need no new library: the second (count
len(batch)of each reply, noNativeResult) is a step of its own. The fourth adds the entry point,lib.countand the call inNativeResultSet.count. The first and the third are the tests, written before the change they test.A row that the codec could not encode used to fail a
count()with a read error; now it is not encoded, so it does not. That was a side effect of the old route, not a behaviour anyone relied on.Related issues
Refs #388 and #315.
Performance changes
Not measured yet, and no speed or memory figure is claimed here. The A/B is registered as follows, for the parent to run.
db.query("sql", "SELECT id, txt FROM R").count(), the full result, on the native wheel (arcadedb).count()call on a freshly run query (the query itself is outside the timed part, or is the same in both arms).NativeResultper row, so the memory figure should move too.26.11.1.post3, ABBA with an A/A control.NativeResultper row; the new code holds nothing per row. It allocates no array and no copy: the library returns a number.count()returns is compared withlen(to_list())of the same query at 0, 1, 255, 256, 257, 2,303, 2,304, 2,305 and 5,000 rows, for plain, typed and wide rows, aggregates, a Cypher result and a command cursor, and the state of the result is compared after it.How I checked it
tests/test_native_count.py(native wheel only, soconftest.pydoes not collect it on the JVM wheel). The first commit holds the number, state and error tests and one that fails on the parent: noNativeResultis built for a counted row. The third commit holds the crossing tests, which fail on the second commit: a result with rows in the library is counted with the single entry callarcadedb_n_count, a command cursor too, no reply text is read, no second close is queued, and a result held in full by its first reply, or exhausted, makes no foreign call.len(to_list())at the edge sizes of the batches (0, 1, 255, 256, 257, 2,303, 2,304, 2,305, 5,000), for typed wide rows (date, datetime,Decimal, bytes, an integer beyond 2^53, a vector, a nested map), aGROUP BYand a projection of@rid, a CypherMATCH, andINSERT INTO B FROM SELECT FROM R, where the count has to run the statement to its end (the rows are inBafter it).next, and after 300, past the first reply), a second count is 0, a result closed early or on a closed database raises the same errors, and a division by zero at row 2,999 raisesReading the result set failed: ...with the result closed and the database still usable.tests/test_resultset_read_errors.pyalready runscount()over the engine errors on both wheels.count()against a fake library for the sizes above, for a lazy command cursor and for an error after the first reply (same number, same final state, noNativeResultbuilt, no read and no close).uvx pre-commit run --fileson the changed files, and bandit as the JVM workflow runs it.CI: to be filled in once the native workflow has run.
Docs
No page under
docs/or the README describes howcount()crosses on the native wheel, and the documented behaviour does not change.Checklist
uvx pre-commit run --files <changed files>passes🤖 Generated with Claude Code