From 09700d2b8c4951da21d474e0974365f4f5e0c672 Mon Sep 17 00:00:00 2001 From: Nicolas Joubert Date: Wed, 30 Sep 2026 09:46:23 +0200 Subject: [PATCH] chore(doc) #12 Add missing documentations: complete reference pages for every Task, complete index and cookbooks. Harmonize and fix existing documentation. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 1 + README.md | 6 +- docs/cookbooks/export_archive_upload.md | 112 ++++++++++++++++++++++++ docs/cookbooks/import_archive.md | 86 ++++++++++++++++++ docs/index.md | 33 +++++-- docs/reference/tasks/_template.md | 33 +++---- docs/reference/tasks/unzip_task.md | 65 +++++++++++--- docs/reference/tasks/zip_task.md | 85 +++++++++++++++--- 8 files changed, 373 insertions(+), 48 deletions(-) create mode 100644 docs/cookbooks/export_archive_upload.md create mode 100644 docs/cookbooks/import_archive.md diff --git a/CHANGELOG.md b/CHANGELOG.md index ca1a657..2f19ab0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ Latest ### Changes * [#10](https://github.com/cleverage/archive-process-bundle/issues/10) Update quality stack: use Rector `withComposerBased()` sets (removed `SYMFONY_64` / `PHPUNIT_100` sets), declare used Symfony packages and PHPUnit range in composer.json, apply quality tools fixes +* [#12](https://github.com/cleverage/archive-process-bundle/issues/12) Add missing documentations: complete reference pages for every Task, complete index and cookbooks. Harmonize and fix existing documentation. v2.0 ------ diff --git a/README.md b/README.md index 000d47f..1e12aa4 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ -CleverAge/UiProcessBundle -======================= +CleverAge/ArchiveProcessBundle +============================== This bundle is a part of the [CleverAge/ProcessBundle](https://github.com/cleverage/process-bundle) project. It provides Archive integration on Process bundle. @@ -12,7 +12,7 @@ For usage documentation, see: ## Support & Contribution -For general support and questions, please use [Github](https://github.com/cleverage/ui-process-bundle/issues). +For general support and questions, please use [Github](https://github.com/cleverage/archive-process-bundle/issues). If you think you found a bug or you have a feature idea to propose, feel free to open an issue after looking at the [contributing](CONTRIBUTING.md) guide. ## License diff --git a/docs/cookbooks/export_archive_upload.md b/docs/cookbooks/export_archive_upload.md new file mode 100644 index 0000000..bea6f32 --- /dev/null +++ b/docs/cookbooks/export_archive_upload.md @@ -0,0 +1,112 @@ +Export, archive and upload a file +================================= + +This recipe describes a typical export flow: read data from a database, write it to a CSV file, zip this file, then +upload the archive to a remote storage (e.g. an SFTP server of a partner). + +It requires, besides this bundle, the +[DoctrineProcessBundle](https://github.com/cleverage/doctrine-process-bundle) and the +[FlysystemProcessBundle](https://github.com/cleverage/flysystem-process-bundle), with two Flysystem storages: + +```yaml +# config/packages/flysystem.yaml +flysystem: + storages: + local.storage: + adapter: 'local' + options: + directory: '%kernel.project_dir%/var/storage/local' + remote.storage: + adapter: 'sftp' + options: + host: '%env(string:SFTP_HOST)%' + port: 22 + username: '%env(string:SFTP_USERNAME)%' + password: '%env(string:SFTP_PASSWORD)%' + root: '%env(string:SFTP_ROOT)%' +``` + +```yaml +clever_age_process: + configurations: + app.export_archive_upload: + description: 'Export the books, zip the file and upload it to the partner SFTP' + help: 'bin/console cleverage:process:execute app.export_archive_upload' + tasks: + read: + service: '@CleverAge\DoctrineProcessBundle\Task\Database\DatabaseReaderTask' + options: + table: 'book' + sql: 'SELECT b.id, b.title FROM book b ORDER BY b.id' + outputs: [write] + + write: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' + options: + file_path: '%kernel.project_dir%/var/exports/books_{date}.csv' + headers: [id, title] + outputs: [to_zip_input] + + to_zip_input: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + wrapper: + wrapper_key: files # '/.../books_20260930.csv' => { files: '/.../books_20260930.csv' } + outputs: [zip] + + zip: + service: '@CleverAge\ArchiveProcessBundle\Task\ZipTask' + options: + filename: '%kernel.project_dir%/var/storage/local/books.zip' # Inside the local.storage root + files_base_path: '%kernel.project_dir%/var/exports' # The archive entry is 'books_20260930.csv' + outputs: [to_storage_path] + + to_storage_path: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + callback: + callback: basename # Path relative to the local.storage root: 'books.zip' + outputs: [upload] + + upload: + service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask' + options: + source_filesystem: 'local.storage' + destination_filesystem: 'remote.storage' + remove_source: true + outputs: [log_upload] + + log_upload: + service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask' + options: + level: info + message: 'Archive uploaded' + context: [input] +``` + +How it works: +- [DatabaseReaderTask](https://github.com/cleverage/doctrine-process-bundle/blob/main/docs/reference/tasks/database_reader_task.md) + is iterable: each row of the query goes to the next task. +- [CsvWriterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/csv_writer_task.md) is + blocking: it writes every row, then outputs the path of the CSV file once all rows have been received. +- As the [ZipTask](../reference/tasks/zip_task.md) only accepts an array input, the + [TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md) + wraps this path under the `files` key with the + [wrapper](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/wrapper_transformer.md) + transformer. +- The [ZipTask](../reference/tasks/zip_task.md) merges this input with its options and creates the archive. As + `files_base_path` is removed from the file path, the CSV file is stored at the root of the archive. The task outputs + the absolute path of the archive. +- The [callback](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/callback_transformer.md) + transformer turns this absolute path into a path relative to the `local.storage` root, as expected by the + [FileFetchTask](https://github.com/cleverage/flysystem-process-bundle/blob/main/docs/reference/tasks/01-FileFetchTask.md), + which copies the archive to `remote.storage` and removes the local one (`remove_source: true`). +- The [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md) logs the + uploaded file name. + +Note that the exported CSV file is kept in `var/exports`. To delete it once archived, add a +[FileRemoverTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/file_remover_task.md) +as a second output of the CsvWriterTask (`outputs: [to_zip_input, remove_csv]`): outputs are processed in order, so the +file is removed after the archive has been created. diff --git a/docs/cookbooks/import_archive.md b/docs/cookbooks/import_archive.md new file mode 100644 index 0000000..671702f --- /dev/null +++ b/docs/cookbooks/import_archive.md @@ -0,0 +1,86 @@ +Import the CSV files of an uploaded archive +========================================== + +This recipe describes how to import data delivered as a zip archive: the archive path is given as the process input +(from the command line or uploaded through the [UI](https://github.com/cleverage/ui-process-bundle)), it is extracted, +then every CSV file it contains is read line by line. + +```yaml +clever_age_process: + configurations: + app.import_archive: + description: 'Import the CSV files of a zip archive' + help: 'bin/console cleverage:process:execute app.import_archive --input=/path/to/archive.zip' + entry_point: prepare # Required to receive the process input + options: + ui: + ui_launch_mode: form + entrypoint_type: file # The uploaded file path is used as input + constraints: + - Collection: + fields: + input: + - File: + mimeTypes: [application/zip] + context: ~ + tasks: + prepare: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + wrapper: + wrapper_key: filename # '/path/to/archive.zip' => { filename: '/path/to/archive.zip' } + outputs: [unzip] + + unzip: + service: '@CleverAge\ArchiveProcessBundle\Task\UnzipTask' + options: + destination: '%kernel.project_dir%/var/imports/archive' + outputs: [browse] + + browse: + service: '@CleverAge\ProcessBundle\Task\File\InputFolderBrowserTask' + options: + name_pattern: '*.csv' + outputs: [read] + + read: + service: '@CleverAge\ProcessBundle\Task\File\Csv\InputCsvReaderTask' + options: + delimiter: ';' + outputs: [count_rows, log_row] + + count_rows: + service: '@CleverAge\ProcessBundle\Task\Reporting\StatCounterTask' + + log_row: + service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask' + options: + level: info + message: 'Imported line' # Replace this task by your own transformation / loading tasks + context: [input] +``` + +How it works: +- The process input is the path of the archive (`--input` option, or the uploaded file when launched from the UI with + `entrypoint_type: file`). As the [UnzipTask](../reference/tasks/unzip_task.md) only accepts an array input, the + [TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md) + wraps it under the `filename` key with the + [wrapper](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/wrapper_transformer.md) + transformer. +- The [UnzipTask](../reference/tasks/unzip_task.md) merges this input with its options (`destination`), extracts the + archive and outputs the destination directory. +- [InputFolderBrowserTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/input_folder_browser_task.md) + is iterable: it outputs, one by one, the path of each CSV file found in the extracted directory (recursively). +- [InputCsvReaderTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/input_csv_reader_task.md) + is iterable too: each line of the current file goes through the following tasks before the next one is read. +- [StatCounterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/stat_counter_task.md) + logs the total count of imported lines at the end of the process, and the + [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md) stands for + your own import tasks. + +Note that the extraction directory is not emptied before extracting: files of a previous archive are browsed again if +they are still present. Use a dedicated directory per execution (e.g. with a +[contextual option](https://github.com/cleverage/process-bundle/blob/main/docs/reference/02-task_definition.md) +`destination: '%kernel.project_dir%/var/imports/{{ import_id }}'` and `-c import_id:"'...'"`), or clean it after the +import. diff --git a/docs/index.md b/docs/index.md index fb2bf9d..b945844 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,6 +1,14 @@ +CleverAge/ArchiveProcessBundle +============================== + +This bundle provides tasks to create and extract zip archives in +[CleverAge/ProcessBundle](https://github.com/cleverage/process-bundle) processes. + ## Prerequisite -CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation. +CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation). + +The PHP [zip extension](https://www.php.net/manual/en/book.zip.php) (`ext-zip`) is required. ## Installation @@ -13,14 +21,27 @@ Open a command console, enter your project directory and install it using compos composer require cleverage/archive-process-bundle ``` -Remember to add the following line to config/bundles.php (not required if Symfony Flex is used) +Remember to add the following line to `config/bundles.php` (not required if Symfony Flex is used): ```php CleverAge\ArchiveProcessBundle\CleverAgeArchiveProcessBundle::class => ['all' => true], ``` -## Reference +## Configuration + +This bundle has no configuration: its tasks are registered as services (public, non-shared) and can be used directly +in your processes by referencing their class name, e.g. `service: '@CleverAge\ArchiveProcessBundle\Task\ZipTask'`. + +Both tasks accept their options either from the task configuration or from their input (an array merged with the +options), which allows computing paths in a previous task. + +## Documentation -- Tasks - - [UnzipTask](reference/tasks/unzip_task.md) - - [ZipTask](reference/tasks/zip_task.md) +- Cookbooks + - [Import the CSV files of an uploaded archive](cookbooks/import_archive.md) + - [Export, archive and upload a file](cookbooks/export_archive_upload.md) +- Reference + - Tasks + - [UnzipTask](reference/tasks/unzip_task.md) + - [ZipTask](reference/tasks/zip_task.md) + - [ProcessBundle documentation](https://github.com/cleverage/process-bundle/blob/main/docs/index.md) diff --git a/docs/reference/tasks/_template.md b/docs/reference/tasks/_template.md index ed1d4a5..919390a 100644 --- a/docs/reference/tasks/_template.md +++ b/docs/reference/tasks/_template.md @@ -1,44 +1,47 @@ TaskName ======== -_Describe main goal an use cases of the task_ +_Describe the main goal and use cases of the task._ Task reference -------------- -* **Service**: `ClassName` +* **Service**: `Fully\Qualified\ClassName` +* **Iterable task** _(only if it implements `IterableTaskInterface`)_ +* **Blocking task** _(only if it implements `BlockingTaskInterface`)_ +* **Flushable task** _(only if it implements `FlushableTaskInterface`)_ Accepted inputs --------------- -_Description of allowed types_ +_Description of allowed types, or "Input is ignored"._ Possible outputs ---------------- -_Description of possible types_ +_Description of possible types._ Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `code` | `type` | **X** _or nothing_ | `default value` _if available_ | _description_ | +| Code | Type | Required | Default | Description | +|--------|--------|:--------:|-----------------|---------------| +| `code` | `type` | **X** | `default value` | _description_ | + +_If the task has no option, replace the table with "This task has no option."._ Examples -------- -_YAML samples and explanations_ - * Example 1 - details - - details - + ```yaml # Task configuration level code: - service: '@service_ref' - options: - a: 1 - b: 2 + service: '@Fully\Qualified\ClassName' + options: + a: 1 + b: 2 + outputs: [next_task] ``` diff --git a/docs/reference/tasks/unzip_task.md b/docs/reference/tasks/unzip_task.md index 14c5263..8634abc 100644 --- a/docs/reference/tasks/unzip_task.md +++ b/docs/reference/tasks/unzip_task.md @@ -1,41 +1,84 @@ UnzipTask -=============== +========= -Unzip a file, requires the destination path in options. +Extracts the whole content of a zip archive into a destination directory, then outputs this directory path. +Typical use case: extract an archive received from a partner (upload, SFTP, ...) before browsing and reading its files. Task reference -------------- -* **Task Service**: `CleverAge\ArchiveProcessBundle\Task\UnzipTask` +* **Service**: `CleverAge\ArchiveProcessBundle\Task\UnzipTask` Accepted inputs --------------- -`array`: inputs are merged with task defined options. +`array` or `null`: the input array is merged with the task options, input values taking precedence over the +configured ones. It may only contain the `filename` and/or `destination` keys (any other key throws an +`UndefinedOptionsException`). A `null` input is handled as an empty array, so the options come from the task +configuration only. + +Any other input type (e.g. a `string` file path) is not supported: convert it into an array first, for instance with a +[TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md) and +the [wrapper](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/wrapper_transformer.md) +transformer (`wrapper_key: filename`). Possible outputs ---------------- -`string`: the destination directory where zip file was extracted. +`string`: the `destination` directory where the archive was extracted. Options ------- -| Code | Type | Required | Default | Description | -|---------------|----------|:--------:|----------|-------------------------------------------------------| -| `filename` | `string` | **X** | | Zip filename to extract | -| `destination` | `string` | **X** | | Destination directory where zip content was extracted | +| Code | Type | Required | Default | Description | +|---------------|----------|:--------:|---------|---------------------------------------------------------------------------------------------------| +| `filename` | `string` | **X** | | Path of the zip archive to extract. It must exist and be readable (`\UnexpectedValueException`) | +| `destination` | `string` | **X** | | Directory where the archive content is extracted | + +Both options are required, but they can be provided either in the task configuration or in the input. +They are validated when the task is executed, not at process initialization. Examples -------- -### Task +* Extract an archive whose paths are set in the options ```yaml # Task configuration level -code: +unzip: service: '@CleverAge\ArchiveProcessBundle\Task\UnzipTask' options: filename: '%kernel.project_dir%/var/data/archive.zip' destination: '%kernel.project_dir%/var/data/unzip_archive' + outputs: [browse] ``` + +* Provide the paths through the input (e.g. from a previous task), then browse the extracted files + +```yaml +# Task configuration level +entry: + service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' + options: + output: + filename: '%kernel.project_dir%/var/data/archive.zip' + destination: '%kernel.project_dir%/var/data/unzip_archive' + outputs: [unzip] +unzip: + service: '@CleverAge\ArchiveProcessBundle\Task\UnzipTask' + outputs: [browse] +browse: + service: '@CleverAge\ProcessBundle\Task\File\InputFolderBrowserTask' +``` + +Notes +----- + +* Underlying method is [ZipArchive::extractTo()](https://www.php.net/manual/en/ziparchive.extractto.php): the whole + archive is extracted, existing files with the same name are overwritten and other files already present in the + destination directory are kept. +* A `\RuntimeException` is thrown if the file cannot be opened as a zip archive. +* Options are resolved (and cached) on the first execution of the task: when the task receives several inputs during + the same process execution (e.g. after an iterable task), the values of the first input are reused for the following + ones. +* See the [Import the CSV files of an uploaded archive](../../cookbooks/import_archive.md) cookbook. diff --git a/docs/reference/tasks/zip_task.md b/docs/reference/tasks/zip_task.md index 1ed4dd7..b70ec5f 100644 --- a/docs/reference/tasks/zip_task.md +++ b/docs/reference/tasks/zip_task.md @@ -1,44 +1,103 @@ ZipTask -=============== +======= -Zip files into a given filename. +Creates a zip archive containing the given files, then outputs the archive path. +Typical use case: archive exported files before sending them to a partner or storing them. Task reference -------------- -* **Task Service**: `CleverAge\ArchiveProcessBundle\Task\ZipTask` +* **Service**: `CleverAge\ArchiveProcessBundle\Task\ZipTask` Accepted inputs --------------- -`array`: inputs are merged with task defined options. +`array` or `null`: the input array is merged with the task options, input values taking precedence over the +configured ones. It may only contain the `filename`, `files` and/or `files_base_path` keys (any other key throws an +`UndefinedOptionsException`). A `null` input is handled as an empty array, so the options come from the task +configuration only. + +Any other input type (e.g. a `string` file path) is not supported: convert it into an array first, for instance with a +[TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md) and +the [wrapper](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/wrapper_transformer.md) +transformer (`wrapper_key: files`). Possible outputs ---------------- -`string`: the zip created filename. +`string`: the `filename` of the created archive. Options ------- -| Code | Type | Required | Default | Description | -|-------------------|---------------------|:---------:|----------|---------------------------------------| -| `filename` | `string` | **X** | | Zip to create filename | -| `files` | `string` or `array` | **X** | | Files to add on archive | -| `files_base_path` | `string` | | '' | Base directory where files to add are | +| Code | Type | Required | Default | Description | +|-------------------|-------------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------| +| `filename` | `string` | **X** | | Path of the zip archive to create. An existing archive is overwritten | +| `files` | `string\|array` | **X** | | Path of the file, or list of paths of the files, to add to the archive. Paths are relative to `files_base_path`, or absolute and starting with `files_base_path` | +| `files_base_path` | `string` | | `''` | Base directory of the files to add. It is removed from the file paths to build the names of the archive entries | + +Options can be provided either in the task configuration or in the input. +They are validated when the task is executed, not at process initialization. Examples -------- -### Task +* Archive files whose paths are set in the options: the archive contains `sample.txt` and `exports/books.csv` ```yaml # Task configuration level -code: +zip: service: '@CleverAge\ArchiveProcessBundle\Task\ZipTask' options: filename: '%kernel.project_dir%/var/data/zip_archive.zip' files: - - '%kernel.project_dir%/var/data/sample.txt' + - '%kernel.project_dir%/var/data/sample.txt' # Absolute path, starting with files_base_path + - 'exports/books.csv' # Path relative to files_base_path files_base_path: '%kernel.project_dir%/var/data' + outputs: [next_task] ``` + +* Archive the file written by a previous task (e.g. a + [CsvWriterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/csv_writer_task.md), + which outputs the path of the written file) + +```yaml +# Task configuration level +write: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' + options: + file_path: '%kernel.project_dir%/var/exports/books_{date}.csv' + outputs: [to_zip_input] +to_zip_input: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + wrapper: + wrapper_key: files # 'path/to/books_20260930.csv' => { files: 'path/to/books_20260930.csv' } + outputs: [zip] +zip: + service: '@CleverAge\ArchiveProcessBundle\Task\ZipTask' + options: + filename: '%kernel.project_dir%/var/exports/books.zip' + files_base_path: '%kernel.project_dir%/var/exports' +``` + +Notes +----- + +* Underlying class is [ZipArchive](https://www.php.net/manual/en/class.ziparchive.php), the archive is opened with the + `ZipArchive::CREATE | ZipArchive::OVERWRITE` flags. A `\RuntimeException` is thrown if it cannot be opened. +* For each file, the entry name is the file path with every occurrence of `files_base_path` removed and leading + directory separators trimmed; the file actually read is `files_base_path` + directory separator + entry name. + Hence: + * all files must be located under `files_base_path`; + * with the default empty `files_base_path`, paths must be absolute (a relative path would be resolved from the + filesystem root), and the entries keep their full path (without the leading `/`) inside the archive. +* Each file must exist and be readable, otherwise an `\UnexpectedValueException` is thrown. Only files can be added: + directories are not browsed. +* Options are resolved (and cached) on the first execution of the task: when the task receives several inputs during + the same process execution (e.g. after an iterable task), the values of the first input are reused for the following + ones. To archive a list of files, aggregate them first (e.g. with an + [AggregateIterableTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/aggregate_iterable_task.md)) + and send them all at once in the `files` key. +* See the [Export, archive and upload a file](../../cookbooks/export_archive_upload.md) cookbook.