Skip to content

fix(decoder): normalize delimiter names in decode options - #14

Open
gstarr-cigna wants to merge 1 commit into
ScrapeGraphAI:mainfrom
gstarr-cigna:fix/decoder-delimiter-normalization
Open

fix(decoder): normalize delimiter names in decode options#14
gstarr-cigna wants to merge 1 commit into
ScrapeGraphAI:mainfrom
gstarr-cigna:fix/decoder-delimiter-normalization

Conversation

@gstarr-cigna

Copy link
Copy Markdown

Summary

toon <file>.toon -d (and any caller passing a named delimiter) decodes tabular arrays into garbage:

$ echo 'products[3]{id,name,price}:
  101,Laptop Pro,1299
  102,Magic Mouse,79
  103,USB-C Cable,19' | toon -d
{
  "products": [
    { "id": "101,Laptop Pro,1299", "name": null, "price": null },
    ...

Root cause

The CLI passes the argparse choice strings 'comma' / 'tab' / 'pipe' as default_delimiter, but DecoderOptions stored the value unchanged and used it directly as the row-splitting character. Rows were split on the literal 5-character string "comma", which never matches, so each row came back as one field.

The encoder already normalizes these names (EncoderOptions, toon/encoder.py); the decoder did not.

Fix

Mirror the encoder's normalization in DecoderOptions (toon/decoder.py): map 'comma',, 'tab'\t, 'pipe'|. Character forms continue to work unchanged.

Added test_decode_delimiter_name_normalization covering both named and character delimiter forms. Full suite: 137 passed, 1 skipped.

The CLI passes delimiter names ('comma'/'tab'/'pipe') as
default_delimiter, but DecoderOptions used the value directly as the
split character, so tabular rows were never split on the literal string
'comma'. Normalize names to their character form, matching what
EncoderOptions already does, so CLI decode round-trips correctly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant