fix(data): make FileInstrumentStorage round-trip instrument files correctly#2302
Open
vignesh-nagarajan-vn wants to merge 1 commit into
Open
Conversation
…rectly Two long-standing defects in FileInstrumentStorage broke the instrument file round-trip: - _read_instrument let pandas NA-parse the symbol column, so real tickers literally named "NA" or "NULL" were loaded as float NaN and merged into a single bogus instrument, which then leaked out of D.list_instruments. PR microsoft#1736 fixed this same defect in exists_qlib_data (reported in microsoft#1720), but the storage reader - the path that actually feeds data loading - kept the old behavior. - _write_instrument wrote the file twice back-to-back; the second to_csv immediately overwrote the correctly-ordered output with columns in [start, end, symbol] order, which _read_instrument then misparsed (dates became symbols). Present since the original implementation (70fc581); every public mutation (update, __setitem__, __delitem__) went through it. Fix: disable default NA parsing when reading the symbol column (symbols are opaque identifiers), and write the file once in the documented [symbol, start, end] order, keeping the utf-8 encoding of the removed call. Both defects are covered by new self-contained tests in tests/storage_tests/test_storage.py (temporary provider, no market data), each verified to fail against the previous code. Related: microsoft#1720, microsoft#1736 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
@microsoft-github-policy-service agree |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two long-standing defects in
FileInstrumentStoragebroke the instrument file round-trip:_read_instrumentlet pandas NA-parse the symbol column, so real tickers literally named "NA" or "NULL" were loaded as float NaN and merged into a single bogus instrument, which then leaked out ofD.list_instruments. PR Fix the bug of reading string NA as NaN in the function exists_qlib_data. #1736 fixed this same defect inexists_qlib_data(reported in Ticker name "NA" makes the exists_qlib_data function report errors. #1720), but the storage reader - the path that actually feeds data loading - kept the old behavior._write_instrumentwrote the file twice back-to-back; the secondto_csvimmediately overwrote the correctly-ordered output with columns in[start, end, symbol]order, which_read_instrumentthen misparsed (dates became symbols). Present since the original implementation (70fc581); every public mutation (update,__setitem__,__delitem__) went through it.Fix: disable default NA parsing when reading the symbol column (symbols are opaque identifiers), and write the file once in the documented
[symbol, start, end]order, keeping the utf-8 encoding of the removed call.Both defects are covered by new self-contained tests in
tests/storage_tests/test_storage.py(temporary provider, no market data), each verified to fail against the previous code.Behavior for well-formed data is unchanged: the existing
tests/storage_testspass as-is,D.list_instrumentsoutput onqlib_data_simpleis hash-identical before/after the change, and black/flake8/pylint/mypy all pass with the CI invocations.Fixes #1720 (completes the fix that #1736 applied only to
exists_qlib_data)