Skip to content

csv_splitter_task.md

Latest commit

 

History

History
69 lines (55 loc) · 3.22 KB

csv_splitter_task.md

File metadata and controls

69 lines (55 loc) · 3.22 KB
Raw

CsvSplitterTask

Splits a large CSV file, whose path is given as input, into smaller temporary CSV files, keeping the headers in each of them. Iterates over the chunks until the whole source file has been processed.

Task reference

  • Service: CleverAge\ProcessBundle\Task\File\Csv\CsvSplitterTask
  • Iterable task

Accepted inputs

string: path of the CSV file to split, prefixed by the base_path option if set (same as InputCsvReaderTask).

Possible outputs

string: path of a temporary CSV file (created in the system temporary directory) containing the headers and the next max_lines data lines of the source file (the last file may contain fewer lines).

Options

Code Type Required Default Description
max_lines int 1000 Maximum number of data lines (headers excluded) per produced file, must be greater than 0
base_path string '' Prepended (with a / separator) to the input path. If empty, the input path is used as is
delimiter string ; CSV delimiter (used for both the source and the produced files)
enclosure string " CSV enclosure character
escape string \ CSV escape character
headers array|null null Static list of CSV headers. If null, headers are read from the first line of the source file
mode string rb Source file open mode (see fopen mode parameter)
log_empty_lines bool false Inherited from CsvReaderTask, not used by this task

Examples

  • Split a CSV file into chunks, then read each chunk
# Task configuration level
entry:
  service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask'
  options:
    output: '%kernel.project_dir%/var/data/large_file.csv'
  outputs: [split]
split:
  service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvSplitterTask'
  options:
    delimiter: ';'
    max_lines: 500
  outputs: [read_chunk]
read_chunk:
  service: '@CleverAge\ProcessBundle\Task\File\Csv\InputCsvReaderTask'
  options:
    delimiter: ';'

Notes

  • Lines are copied as is: they are not checked against the headers.
  • Temporary files are not deleted by the task, use FileRemoverTask if needed.
  • Empty lines of the source file are copied and counted like the other lines.
  • No empty file is produced: when the source file has no (more) data line, the task is skipped.