Repository navigation
Conversation
|
@FlipperPA, one policy question on the over-width branch in this PR: is truncation intended both at the It may help to give those cases separate regression expectations: preserve the whole row with documented field truncation at the format limit, versus an explicitly undersized caller-declared column. Should the latter still signal the width mismatch, or is the same truncation policy intended there too? We prepared a small synthetic input-boundary fixture using our existing exact-byte CSV intake component, but have not executed ReadStat or Stata, so this is a review question rather than a new reproduction of the dropped-row report. Zero Px Core — human–AI engineering team |
3ae76cb to
f6c4862
Compare
|
@NixEngine Thanks, good catch. Truncation was only meant for the format-limit case (we use read_csv in our use case). I've narrowed it: dta_write_string truncates at a UTF-8 boundary only when the caller passed width 0, which asks the writer for the format's maximum. An explicitly declared width, including exactly 2045, still returns Rebasing onto current dev now rejects string widths above 2045 up front. The CLI sizes CSV columns by their max byte length, so it now passes width 0 for over-wide DTA string columns to get the truncation behavior. Regression tests cover both cases: truncation of 2- and 3-byte characters straddling the limit at default width (117 - 119), and the error for an explicit str10 and an explicit str2045. THANKS AGAIN! |
…n dropping the row A string near Stata's str2045 limit in characters can exceed it in UTF-8 bytes. When the caller passes width 0, asking the writer for the format's maximum, dta_write_string now truncates such a value at the last complete UTF-8 character instead of failing the row. A width the caller chose is still a hard limit and returns READSTAT_ERROR_STRING_VALUE_IS_TOO_LONG. The readstat CLI sizes CSV string columns by their longest value in bytes, which the DTA writer now rejects above 2045. For DTA output it passes width 0 for those columns so the truncation applies. read_csv's DTA module also accepts READSTAT_TYPE_STRING_REF columns. Tests cover truncation of 2- and 3-byte characters that straddle the limit at default width, and the error for values longer than an explicit width, including an explicit width of 2045.
|
@evanmiller Howdy Evan, I hope all is well. The macos test failure on this PR seems to be a timeout error: Any chance we could get eyes on this at some point? We're working on another PR to allow CSV -> SAS7BDAT as well, using similar functionality to STATA/SPSS with the JSON mapping file. |
Fixes #382
This truncates the very rare edge cases that exceed the byte limit, because of heavy UTF-8 usage. This path was chosen because using
strLin STATA would require a significant rewrite to re-count the byte map for the binary file.This will very, very rarely truncate the field, but provide the full row count, rather than silently dropping the entire row.