chore(doc) #28 Add missing documentations

CHANGELOG.md

@@ -3,6 +3,7 @@ Latest
 
 ### Changes
 * [#26](https://github.com/cleverage/doctrine-process-bundle/issues/26) Update quality stack: use Rector `withComposerBased()` sets (removed `SYMFONY_64` / `PHPUNIT_100` sets), declare used Symfony packages and PHPUnit range in composer.json, apply quality tools fixes
+* [#28](https://github.com/cleverage/doctrine-process-bundle/issues/28) Add missing documentations: complete reference pages for every Task (inherited options, iterable/flushable behaviours, examples, notes), Database to CSV export and CSV to entities import cookbooks. Harmonize index and task template, fix existing documentation.
 
 v3.0
 ------

docs/cookbooks/csv_to_entities_import.md

@@ -0,0 +1,89 @@
+CSV to entities import
+======================
+
+This recipe imports a CSV file into Doctrine entities: invalid lines are rejected, each valid line is denormalized into
+an entity, and entities are written to the database by batches. The entity manager is cleared after each batch to keep
+the memory usage stable, whatever the size of the file.
+
+Source file (`var/data/authors.csv`):
+
+```csv
+firstname;lastname
+Isaac;Asimov
+Ursula;Le Guin
+Mary;Shelley
+```
+
+```yaml
+clever_age_process:
+    configurations:
+        app.csv_to_entities_import:
+            description: 'Import authors from a CSV file'
+            help: 'bin/console cleverage:process:execute app.csv_to_entities_import'
+            tasks:
+                read_csv:
+                    service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvReaderTask'
+                    options:
+                        file_path: '%kernel.project_dir%/var/data/authors.csv'
+                    outputs: [filter_valid]
+
+                filter_valid:
+                    service: '@CleverAge\ProcessBundle\Task\FilterTask'
+                    options:
+                        not_empty:
+                            '[firstname]': ~
+                            '[lastname]': ~
+                    outputs: [denormalize]
+                    error_outputs: [log_rejected]
+
+                log_rejected:
+                    service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask'
+                    options:
+                        level: warning
+                        message: 'Invalid author line'
+
+                denormalize:
+                    service: '@CleverAge\ProcessBundle\Task\Serialization\DenormalizerTask'
+                    options:
+                        class: App\Entity\Author
+                    outputs: [batch_write]
+
+                batch_write:
+                    service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineBatchWriterTask'
+                    options:
+                        batch_count: 200
+                    outputs: [clear, count_batches]
+
+                count_batches:
+                    service: '@CleverAge\ProcessBundle\Task\Reporting\StatCounterTask'
+
+                clear:
+                    service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\ClearEntityManagerTask'
+```
+
+How it works:
+- [CsvReaderTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/csv_reader_task.md) is
+  iterable: each line (an associative array indexed by the headers) goes through the following tasks before the next
+  one is read.
+- [FilterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/filter_task.md) sends lines
+  with an empty `firstname` or `lastname` to its error branch, where the
+  [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md) logs them.
+- [DenormalizerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/denormalizer_task.md)
+  creates a new `App\Entity\Author` from each line, using the Symfony Serializer.
+- [DoctrineBatchWriterTask](../reference/tasks/doctrine_batchwriter_task.md) buffers the entities and skips its
+  outputs until 200 entities are buffered; it then persists and flushes them in one go and outputs the batch. The last
+  incomplete batch is written when the task is flushed, at the end of the file.
+- [ClearEntityManagerTask](../reference/tasks/doctrine_clear_task.md) is only executed after each written batch: it
+  detaches the written entities so that they can be garbage collected.
+- [StatCounterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/stat_counter_task.md)
+  counts the written batches and logs the total at the end of the process.
+
+Going further:
+- To update existing entities instead of always creating new ones, fetch them first (e.g. with a custom task or a
+  transformer), or write raw rows with a SQL upsert statement and the
+  [DatabaseUpdaterTask](../reference/tasks/database_updater_task.md).
+- To validate entities against their mapping constraints before writing them, insert a
+  [ValidatorTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/validator_task.md) between
+  the denormalization and the batch writer.
+- Entities with associations (e.g. `App\Entity\Book` and its `author`) must reference managed entities: as the entity
+  manager is cleared after each batch, fetch the related entities again for each line rather than caching them.

docs/cookbooks/database_to_csv_export.md

@@ -0,0 +1,82 @@
+Database to CSV export
+======================
+
+This recipe exports rows from a database to a CSV file, using a raw SQL query: rows are fetched one at a time, so the
+memory usage stays low even with big tables. The author lastname to export is given in the process context.
+
+```yaml
+clever_age_process:
+    configurations:
+        app.database_to_csv_export:
+            description: 'Export the books of an author to a CSV file'
+            help: "bin/console cleverage:process:execute app.database_to_csv_export -c lastname:\"'King'\""
+            tasks:
+                read_books:
+                    service: '@CleverAge\DoctrineProcessBundle\Task\Database\DatabaseReaderTask'
+                    options:
+                        table: 'book' # Required, even if a custom sql query is used
+                        sql: >
+                            SELECT b.id, b.title, a.firstname, a.lastname
+                            FROM book b
+                            INNER JOIN author a ON a.id = b.author_id
+                            WHERE a.lastname = :lastname
+                            ORDER BY b.title
+                        params:
+                            lastname: '{{ lastname }}'
+                        empty_log_level: notice
+                    outputs: [format]
+
+                format:
+                    service: '@CleverAge\ProcessBundle\Task\TransformerTask'
+                    options:
+                        transformers:
+                            mapping:
+                                mapping:
+                                    id:
+                                        code: '[id]'
+                                    title:
+                                        code: '[title]'
+                                    author:
+                                        code: ['[firstname]', '[lastname]']
+                                        transformers:
+                                            implode:
+                                                separator: ' '
+                    outputs: [write_csv, count_rows]
+
+                count_rows:
+                    service: '@CleverAge\ProcessBundle\Task\Reporting\StatCounterTask'
+
+                write_csv:
+                    service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask'
+                    options:
+                        file_path: '%kernel.project_dir%/var/exports/books_{date_time}.csv'
+                        headers: [id, title, author]
+                    outputs: [log_file]
+
+                log_file:
+                    service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask'
+                    options:
+                        level: info
+                        message: 'Books exported'
+```
+
+How it works:
+- [DatabaseReaderTask](../reference/tasks/database_reader_task.md) is iterable: the query is executed once, then each
+  row (an associative array) goes through the following tasks before the next one is fetched. The `{{ lastname }}`
+  placeholder is replaced by the `lastname` context value and bound as a query parameter (no SQL injection). If no
+  book matches, a `notice` is logged and the process ends without writing any file.
+- The [TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md)
+  builds the CSV line (see the
+  [mapping](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/mapping_transformer.md)
+  and [implode](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/implode_transformer.md)
+  transformers).
+- [StatCounterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/stat_counter_task.md)
+  counts the exported lines and logs the total at the end of the process.
+- [CsvWriterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/csv_writer_task.md) is
+  blocking: it writes each line, and outputs the file path once all the rows have been read, to the
+  [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md).
+
+To export entities instead of raw rows, replace the first task by a
+[DoctrineReaderTask](../reference/tasks/doctrine_reader_task.md) and read the values with property paths
+(e.g. `code: 'author.lastname'`). Note that the DoctrineReaderTask loads all the matching entities in memory: for
+big volumes, prefer the DatabaseReaderTask.

docs/index.md

@@ -19,16 +19,57 @@ Remember to add the following line to config/bundles.php (not required if Symfon
 CleverAge\DoctrineProcessBundle\CleverAgeDoctrineProcessBundle::class => ['all' => true],
 ```
 
-## Reference
-
-- Tasks
-  - [DatabaseReaderTask](reference/tasks/database_reader_task.md)
-  - [DatabaseUpdaterTask](reference/tasks/database_updater_task.md)
-  - [ClearEntityManagerTask](reference/tasks/doctrine_clear_task.md))
-  - [DoctrineBatchWriterTask](reference/tasks/doctrine_batchwriter_task.md)
-  - [DoctrineCleanerTask](reference/tasks/doctrine_cleaner_task.md)
-  - [DoctrineDetacherTask](reference/tasks/doctrine_detacher_task.md)
-  - [DoctrineReaderTask](reference/tasks/doctrine_reader_task.md)
-  - [DoctrineRefresherTask](reference/tasks/doctrine_refresher_task.md)
-  - [DoctrineRemoverTask](reference/tasks/doctrine_remover_task.md)
-  - [DoctrineWriterTask](reference/tasks/doctrine_writer_task.md)
+This bundle relies on [doctrine/doctrine-bundle](https://github.com/doctrine/DoctrineBundle) and
+[doctrine/orm](https://www.doctrine-project.org/projects/orm.html), installed as dependencies: the bundle
+`Doctrine\Bundle\DoctrineBundle\DoctrineBundle` must be enabled as well.
+
+## Configuration
+
+This bundle has no configuration of its own: its tasks use the Doctrine connections and entity managers configured in
+`config/packages/doctrine.yaml`.
+
+```yaml
+# config/packages/doctrine.yaml
+doctrine:
+    dbal:
+        url: '%env(resolve:DATABASE_URL)%'
+    orm:
+        mappings:
+            App:
+                type: attribute
+                dir: '%kernel.project_dir%/src/Entity'
+                prefix: 'App\Entity'
+```
+
+* **Database** tasks ([DatabaseReaderTask](reference/tasks/database_reader_task.md),
+  [DatabaseUpdaterTask](reference/tasks/database_updater_task.md)) run raw SQL queries through Doctrine DBAL. Their
+  `connection` option takes the name of a connection (a key under `doctrine.dbal.connections`); the default connection
+  is used if it is not set.
+* **EntityManager** tasks work with Doctrine ORM entities. Except for the
+  [ClearEntityManagerTask](reference/tasks/doctrine_clear_task.md), which takes the name of an entity manager (a key
+  under `doctrine.orm.entity_managers`) in its `entity_manager` option, they use the entity manager that manages the
+  class of the handled entity.
+
+See the [DoctrineBundle documentation](https://symfony.com/bundles/DoctrineBundle/current/configuration.html) for the
+configuration of multiple connections and entity managers.
+
+## Documentation
+
+- Cookbooks
+    - [Database to CSV export](cookbooks/database_to_csv_export.md)
+    - [CSV to entities import](cookbooks/csv_to_entities_import.md)
+- Reference
+    - Tasks
+        - Database
+            - [DatabaseReaderTask](reference/tasks/database_reader_task.md)
+            - [DatabaseUpdaterTask](reference/tasks/database_updater_task.md)
+        - EntityManager
+            - [ClearEntityManagerTask](reference/tasks/doctrine_clear_task.md)
+            - [DoctrineBatchWriterTask](reference/tasks/doctrine_batchwriter_task.md)
+            - [DoctrineCleanerTask](reference/tasks/doctrine_cleaner_task.md)
+            - [DoctrineDetacherTask](reference/tasks/doctrine_detacher_task.md)
+            - [DoctrineReaderTask](reference/tasks/doctrine_reader_task.md)
+            - [DoctrineRefresherTask](reference/tasks/doctrine_refresher_task.md)
+            - [DoctrineRemoverTask](reference/tasks/doctrine_remover_task.md)
+            - [DoctrineWriterTask](reference/tasks/doctrine_writer_task.md)
+- [CleverAge/ProcessBundle documentation](https://github.com/cleverage/process-bundle/blob/main/docs/index.md)

docs/reference/tasks/_template.md

@@ -1,44 +1,47 @@
 TaskName
 ========
 
-_Describe main goal an use cases of the task_
+_Describe the main goal and use cases of the task._
 
 Task reference
 --------------
 
-* **Service**: `ClassName`
+* **Service**: `Fully\Qualified\ClassName`
+* **Iterable task** _(only if it implements `IterableTaskInterface`)_
+* **Blocking task** _(only if it implements `BlockingTaskInterface`)_
+* **Flushable task** _(only if it implements `FlushableTaskInterface`)_
 
 Accepted inputs
 ---------------
 
-_Description of allowed types_
+_Description of allowed types, or "Input is ignored"._
 
 Possible outputs
 ----------------
 
-_Description of possible types_
+_Description of possible types._
 
 Options
 -------
 
-| Code | Type | Required | Default | Description |
-| ---- | ---- | :------: | ------- | ----------- |
-| `code` | `type` | **X** _or nothing_ | `default value` _if available_ | _description_ |
+| Code   | Type   | Required | Default         | Description   |
+|--------|--------|:--------:|-----------------|---------------|
+| `code` | `type` |  **X**   | `default value` | _description_ |
+
+_If the task has no option, replace the table with "This task has no option."._
 
 Examples
 --------
 
-_YAML samples and explanations_ 
-
 * Example 1
   - details
-  - details
-
+
 ```yaml
 # Task configuration level
 code:
-    service: '@service_ref'
-    options: 
-        a: 1
-        b: 2
+  service: '@Fully\Qualified\ClassName'
+  options:
+    a: 1
+    b: 2
+  outputs: [next_task]
 ```