Skip to content

fix(data): make FileInstrumentStorage round-trip instrument files correctly#2302

Open
vignesh-nagarajan-vn wants to merge 1 commit into
microsoft:mainfrom
vignesh-nagarajan-vn:fix/instrument-storage-roundtrip
Open

fix(data): make FileInstrumentStorage round-trip instrument files correctly#2302
vignesh-nagarajan-vn wants to merge 1 commit into
microsoft:mainfrom
vignesh-nagarajan-vn:fix/instrument-storage-roundtrip

Conversation

@vignesh-nagarajan-vn

@vignesh-nagarajan-vn vignesh-nagarajan-vn commented Jul 18, 2026

Copy link
Copy Markdown

Two long-standing defects in FileInstrumentStorage broke the instrument file round-trip:

  • _read_instrument let pandas NA-parse the symbol column, so real tickers literally named "NA" or "NULL" were loaded as float NaN and merged into a single bogus instrument, which then leaked out of D.list_instruments. PR Fix the bug of reading string NA as NaN in the function exists_qlib_data. #1736 fixed this same defect in exists_qlib_data (reported in Ticker name "NA" makes the exists_qlib_data function report errors.  #1720), but the storage reader - the path that actually feeds data loading - kept the old behavior.

  • _write_instrument wrote the file twice back-to-back; the second to_csv immediately overwrote the correctly-ordered output with columns in [start, end, symbol] order, which _read_instrument then misparsed (dates became symbols). Present since the original implementation (70fc581); every public mutation (update, __setitem__, __delitem__) went through it.

Fix: disable default NA parsing when reading the symbol column (symbols are opaque identifiers), and write the file once in the documented [symbol, start, end] order, keeping the utf-8 encoding of the removed call.

Both defects are covered by new self-contained tests in tests/storage_tests/test_storage.py (temporary provider, no market data), each verified to fail against the previous code.

Behavior for well-formed data is unchanged: the existing tests/storage_tests pass as-is, D.list_instruments output on qlib_data_simple is hash-identical before/after the change, and black/flake8/pylint/mypy all pass with the CI invocations.

Fixes #1720 (completes the fix that #1736 applied only to exists_qlib_data)

…rectly

Two long-standing defects in FileInstrumentStorage broke the instrument
file round-trip:

- _read_instrument let pandas NA-parse the symbol column, so real
  tickers literally named "NA" or "NULL" were loaded as float NaN and
  merged into a single bogus instrument, which then leaked out of
  D.list_instruments. PR microsoft#1736 fixed this same defect in
  exists_qlib_data (reported in microsoft#1720), but the storage reader - the
  path that actually feeds data loading - kept the old behavior.

- _write_instrument wrote the file twice back-to-back; the second
  to_csv immediately overwrote the correctly-ordered output with
  columns in [start, end, symbol] order, which _read_instrument then
  misparsed (dates became symbols). Present since the original
  implementation (70fc581); every public mutation (update,
  __setitem__, __delitem__) went through it.

Fix: disable default NA parsing when reading the symbol column
(symbols are opaque identifiers), and write the file once in the
documented [symbol, start, end] order, keeping the utf-8 encoding of
the removed call.

Both defects are covered by new self-contained tests in
tests/storage_tests/test_storage.py (temporary provider, no market
data), each verified to fail against the previous code.

Related: microsoft#1720, microsoft#1736

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vignesh-nagarajan-vn

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ticker name "NA" makes the exists_qlib_data function report errors.

1 participant