Skip to content

feat(environment): stream lines in ReadFileTool to avoid OOM - #7176

Open
ArulJerald wants to merge 1 commit into
google:mainfrom
ArulJerald:feat/stream-read-file-lines-7131
Open

ArulJerald wants to merge 1 commit into
google:mainfrom
ArulJerald:feat/stream-read-file-lines-7131

Conversation

@ArulJerald

Copy link
Copy Markdown

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

  • Closes: feat(environment): stream lines in ReadFileTool to prevent OOM on large files #7131
    Problem:
    In ReadFileTool, reading files previously loaded the entire file contents into memory via self._environment.read_file(path) and split all lines in memory (data_bytes.splitlines()) before slicing by line boundaries. For very large files, buffering the entire file into memory causes significant memory spikes and potential Out Of Memory (OOM) errors.
    Solution:
  • Added read_file_lines(path, start_line, end_line) to BaseEnvironment with a default implementation that falls back to read_file.
  • Overrode read_file_lines in LocalEnvironment using a streaming _sync_read_lines helper that iterates through the file line by line without buffering the whole file in memory.
  • Updated ReadFileTool.run_async to use self._environment.read_file_lines(path, start_line=start, end_line=end_line) to lazily fetch only the requested range and the line count.
  • Added comprehensive unit tests in TestReadFileTool covering empty files, start_line exceeding total lines, and line ordering validation.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.
    Summary of passed pytest results:
tests/unittests/tools/environment/test_read_file_tool.py ......... [100%]
============================== 9 passed in 0.54s ===============================

Manual End-to-End (E2E) Tests:

  • Tested ReadFileTool against empty files, small text files, and files where line bounds are out of range or ordered in reverse. All validations returned expected structured error responses matching previous behavior with reduced memory footprint.

Checklist

  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.

Read lines lazily in LocalEnvironment.read_file_lines instead of buffering
the entire file into memory as bytes before slicing.

Closes: google#7131
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(environment): stream lines in ReadFileTool to prevent OOM on large files

2 participants