Skip to content

fix(input): ignore blank lines when reading JSONL files#2423

Open
VectorPeak wants to merge 1 commit into
microsoft:mainfrom
VectorPeak:codex/jsonl-blank-lines
Open

fix(input): ignore blank lines when reading JSONL files#2423
VectorPeak wants to merge 1 commit into
microsoft:mainfrom
VectorPeak:codex/jsonl-blank-lines

Conversation

@VectorPeak

@VectorPeak VectorPeak commented Jul 5, 2026

Copy link
Copy Markdown

Description

Fix JSONL input loading so blank or whitespace-only lines do not cause the entire .jsonl file to be skipped.

JSONLinesFileReader.read_file() currently parses every entry returned by text.splitlines() with json.loads(line). When a .jsonl file contains a blank line between otherwise valid records, json.loads("") raises JSONDecodeError. That exception bubbles up to InputReader._iterate_files(), which logs the error and skips the whole file.

This change filters out blank or whitespace-only lines before parsing JSONL records, while preserving the existing behavior for non-empty malformed JSON lines.

Related Issues

Fixes #2424.

Proposed Changes

  • Ignore blank or whitespace-only lines when reading JSONL input files.
  • Preserve existing parsing behavior for valid non-empty JSONL records.
  • Preserve failure behavior for genuinely malformed non-empty JSONL lines.
  • Add a focused unit test covering valid JSONL records separated by blank lines.

Checklist

  • I have tested these changes locally.
  • I have reviewed the code changes.
  • I have updated the documentation (if necessary).
  • I have added appropriate unit tests (if applicable).

Additional Notes

What Problem This Solves

A JSONL input file may contain accidental blank or whitespace-only lines between otherwise valid records:

{"text":"a"}

{"text":"b"}

Before this change, the blank line was passed directly to json.loads(""), which raises JSONDecodeError. Because InputReader._iterate_files() catches exceptions at the file level, the reader skipped the entire file instead of loading the valid records before and after the blank line.

Evidence

The affected call chain is:

create_input_reader(...)
  -> JSONLinesFileReader.read_file(...)
    -> text.splitlines()
    -> json.loads(line)
       -> JSONDecodeError on blank line
  -> InputReader._iterate_files(...)
    -> catches exception
    -> logs warning
    -> skips the file

This PR only ignores blank or whitespace-only JSONL lines. Malformed non-empty lines still go through json.loads(line) and keep the existing error/skip behavior.

Scope

This change only affects JSONL input handling. It does not change CSV, JSON, text, parquet, MarkItDown, storage behavior, or handling of malformed non-empty JSONL records.

Validation

  • git diff --check
  • uv run ruff format --check packages/graphrag-input/graphrag_input/jsonl.py tests/unit/indexing/input/test_jsonl_loader.py
  • uv run ruff check packages/graphrag-input/graphrag_input/jsonl.py tests/unit/indexing/input/test_jsonl_loader.py
  • uv run --package graphrag-input pytest tests/unit/indexing/input/test_jsonl_loader.py -q - 3 passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSONL reader skips entire file when input contains blank lines

1 participant