You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Parity: the two wheels return different Python floats from to_json_list, and no cross-wheel golden test exists #314
The two wheels are meant to return the same values. The test suite runs the same tests/ on both, but there is no cross-wheel comparison and no golden file, so a difference stays invisible until a user trips over it. A read-only audit of the tag 26.11.1 (commit 2f28a328be, nothing run) found the following. What is read from the code is marked read, what depends on library behaviour I did not read is marked guess.
Confirmed from the code: to_json_list() returns different Python floats for the same FLOAT or float32 vector value. The native wheel returns the exact float64 widening of the float32 (NativeCodec.java:149-154, 243-261, codec.py:456-480). The JVM wheel passes each float to the engine's JSONObject as a JsonPrimitive (read), and I expect the JSON writer to print the shortest float text such as 0.1 that Python parses as a different double (guess, the writer was not read). It affects scalar FLOAT properties and float[] of any length, not only vectors. On the JVM wheel to_json_list() and to_list() of the same rows disagree, row["vec"] == arr.tolist() is false there and true on native, and a hash or json.dumps of the rows differs. np.allclose and recall are not affected. Only a native-only test pins this (tests/test_native_json_list.py:223-236).
Likely differences, each a guess until a test shows it: to_json_list of DECIMAL (the JVM wheel may give an exact int or number where native gives float(Decimal)); the text of Result.to_json() (native rebuilds it in Python, so key order, separators and float digits may differ); float-list columns in to_columns, to_dataframe and to_arrow (native guesses a float32 array when every value survives float32, the JVM wheel may give a list column); and the last bit of vector distances (the JVM varies between calls, native is repeatable).
Plan, in this order. First one shared test with a fixed table and a committed golden literal, run on both wheels, which shows each difference as a failing assertion (the audit lists 14 cases, starting with the four above). Then fix what the tests show: the float widening in the JVM wheel's RowBatcher.keepNonFinite is a bridge-Java change that fits a wheels-only 26.11.1.post1, after measuring the transport time because the text gets longer. This is part of the work to make both wheels read and write big data well; I will not start before the paper deadline.
Status. The float half is fixed: #320 makes the JVM wheel widen a FLOAT and a float32 float[] to the exact double in to_json_list, and adds tests/test_cross_wheel_golden.py, one fixed table with committed golden values that runs unchanged on both wheels (floats, DECIMAL, a LONG beyond 2^53, DATE, DATETIME, NULL, an embedded list). It is in 26.11.1.post1 and 26.11.1.post2. The test fails on the old JVM code and passes on the new one. It is a behaviour test; no speed was measured.
Not done: the golden test does not yet cover the text of Result.to_json(), float-list columns in to_arrow, vector distances, or an integral DECIMAL in to_json_list. And I have not measured what the longer float text costs the JVM wheel (a float32 now prints about 17 digits instead of about 8); on 26.11.1 the JVM wheel was already the slower one on to_json_list and iter_json_batches of the 128-dimension table (#312). I will measure that on the 26.11.1.post2 pair. The issue stays open for those two items.
Activity
JVM to_json_list: a FLOAT and a float32 float[] arrive as the exact d…
JVM to_json_list: a FLOAT and a float32 float[] arrive as the exact d…
tae898 commentedon Nov 14, 2026
Status. The float half is fixed: #320 makes the JVM wheel widen a
FLOATand a float32float[]to the exact double into_json_list, and addstests/test_cross_wheel_golden.py, one fixed table with committed golden values that runs unchanged on both wheels (floats,DECIMAL, aLONGbeyond 2^53,DATE,DATETIME,NULL, an embedded list). It is in26.11.1.post1and26.11.1.post2. The test fails on the old JVM code and passes on the new one. It is a behaviour test; no speed was measured.Not done: the golden test does not yet cover the text of
Result.to_json(), float-list columns into_arrow, vector distances, or an integralDECIMALinto_json_list. And I have not measured what the longer float text costs the JVM wheel (a float32 now prints about 17 digits instead of about 8); on26.11.1the JVM wheel was already the slower one onto_json_listanditer_json_batchesof the 128-dimension table (#312). I will measure that on the26.11.1.post2pair. The issue stays open for those two items.