diff --git a/CHANGELOG.md b/CHANGELOG.md index 889d32b..d822c67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ Latest ### Changes * [#26](https://github.com/cleverage/doctrine-process-bundle/issues/26) Update quality stack: use Rector `withComposerBased()` sets (removed `SYMFONY_64` / `PHPUNIT_100` sets), declare used Symfony packages and PHPUnit range in composer.json, apply quality tools fixes +* [#28](https://github.com/cleverage/doctrine-process-bundle/issues/28) Add missing documentations: complete reference pages for every Task (inherited options, iterable/flushable behaviours, examples, notes), Database to CSV export and CSV to entities import cookbooks. Harmonize index and task template, fix existing documentation. v3.0 ------ diff --git a/docs/cookbooks/csv_to_entities_import.md b/docs/cookbooks/csv_to_entities_import.md new file mode 100644 index 0000000..9cd88c3 --- /dev/null +++ b/docs/cookbooks/csv_to_entities_import.md @@ -0,0 +1,89 @@ +CSV to entities import +====================== + +This recipe imports a CSV file into Doctrine entities: invalid lines are rejected, each valid line is denormalized into +an entity, and entities are written to the database by batches. The entity manager is cleared after each batch to keep +the memory usage stable, whatever the size of the file. + +Source file (`var/data/authors.csv`): + +```csv +firstname;lastname +Isaac;Asimov +Ursula;Le Guin +Mary;Shelley +``` + +```yaml +clever_age_process: + configurations: + app.csv_to_entities_import: + description: 'Import authors from a CSV file' + help: 'bin/console cleverage:process:execute app.csv_to_entities_import' + tasks: + read_csv: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvReaderTask' + options: + file_path: '%kernel.project_dir%/var/data/authors.csv' + outputs: [filter_valid] + + filter_valid: + service: '@CleverAge\ProcessBundle\Task\FilterTask' + options: + not_empty: + '[firstname]': ~ + '[lastname]': ~ + outputs: [denormalize] + error_outputs: [log_rejected] + + log_rejected: + service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask' + options: + level: warning + message: 'Invalid author line' + + denormalize: + service: '@CleverAge\ProcessBundle\Task\Serialization\DenormalizerTask' + options: + class: App\Entity\Author + outputs: [batch_write] + + batch_write: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineBatchWriterTask' + options: + batch_count: 200 + outputs: [clear, count_batches] + + count_batches: + service: '@CleverAge\ProcessBundle\Task\Reporting\StatCounterTask' + + clear: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\ClearEntityManagerTask' +``` + +How it works: +- [CsvReaderTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/csv_reader_task.md) is + iterable: each line (an associative array indexed by the headers) goes through the following tasks before the next + one is read. +- [FilterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/filter_task.md) sends lines + with an empty `firstname` or `lastname` to its error branch, where the + [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md) logs them. +- [DenormalizerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/denormalizer_task.md) + creates a new `App\Entity\Author` from each line, using the Symfony Serializer. +- [DoctrineBatchWriterTask](../reference/tasks/doctrine_batchwriter_task.md) buffers the entities and skips its + outputs until 200 entities are buffered; it then persists and flushes them in one go and outputs the batch. The last + incomplete batch is written when the task is flushed, at the end of the file. +- [ClearEntityManagerTask](../reference/tasks/doctrine_clear_task.md) is only executed after each written batch: it + detaches the written entities so that they can be garbage collected. +- [StatCounterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/stat_counter_task.md) + counts the written batches and logs the total at the end of the process. + +Going further: +- To update existing entities instead of always creating new ones, fetch them first (e.g. with a custom task or a + transformer), or write raw rows with a SQL upsert statement and the + [DatabaseUpdaterTask](../reference/tasks/database_updater_task.md). +- To validate entities against their mapping constraints before writing them, insert a + [ValidatorTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/validator_task.md) between + the denormalization and the batch writer. +- Entities with associations (e.g. `App\Entity\Book` and its `author`) must reference managed entities: as the entity + manager is cleared after each batch, fetch the related entities again for each line rather than caching them. diff --git a/docs/cookbooks/database_to_csv_export.md b/docs/cookbooks/database_to_csv_export.md new file mode 100644 index 0000000..7e51f75 --- /dev/null +++ b/docs/cookbooks/database_to_csv_export.md @@ -0,0 +1,82 @@ +Database to CSV export +====================== + +This recipe exports rows from a database to a CSV file, using a raw SQL query: rows are fetched one at a time, so the +memory usage stays low even with big tables. The author lastname to export is given in the process context. + +```yaml +clever_age_process: + configurations: + app.database_to_csv_export: + description: 'Export the books of an author to a CSV file' + help: "bin/console cleverage:process:execute app.database_to_csv_export -c lastname:\"'King'\"" + tasks: + read_books: + service: '@CleverAge\DoctrineProcessBundle\Task\Database\DatabaseReaderTask' + options: + table: 'book' # Required, even if a custom sql query is used + sql: > + SELECT b.id, b.title, a.firstname, a.lastname + FROM book b + INNER JOIN author a ON a.id = b.author_id + WHERE a.lastname = :lastname + ORDER BY b.title + params: + lastname: '{{ lastname }}' + empty_log_level: notice + outputs: [format] + + format: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + mapping: + mapping: + id: + code: '[id]' + title: + code: '[title]' + author: + code: ['[firstname]', '[lastname]'] + transformers: + implode: + separator: ' ' + outputs: [write_csv, count_rows] + + count_rows: + service: '@CleverAge\ProcessBundle\Task\Reporting\StatCounterTask' + + write_csv: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' + options: + file_path: '%kernel.project_dir%/var/exports/books_{date_time}.csv' + headers: [id, title, author] + outputs: [log_file] + + log_file: + service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask' + options: + level: info + message: 'Books exported' +``` + +How it works: +- [DatabaseReaderTask](../reference/tasks/database_reader_task.md) is iterable: the query is executed once, then each + row (an associative array) goes through the following tasks before the next one is fetched. The `{{ lastname }}` + placeholder is replaced by the `lastname` context value and bound as a query parameter (no SQL injection). If no + book matches, a `notice` is logged and the process ends without writing any file. +- The [TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md) + builds the CSV line (see the + [mapping](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/mapping_transformer.md) + and [implode](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/implode_transformer.md) + transformers). +- [StatCounterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/stat_counter_task.md) + counts the exported lines and logs the total at the end of the process. +- [CsvWriterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/csv_writer_task.md) is + blocking: it writes each line, and outputs the file path once all the rows have been read, to the + [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md). + +To export entities instead of raw rows, replace the first task by a +[DoctrineReaderTask](../reference/tasks/doctrine_reader_task.md) and read the values with property paths +(e.g. `code: 'author.lastname'`). Note that the DoctrineReaderTask loads all the matching entities in memory: for +big volumes, prefer the DatabaseReaderTask. diff --git a/docs/index.md b/docs/index.md index 9c24dd1..f91b391 100644 --- a/docs/index.md +++ b/docs/index.md @@ -19,16 +19,57 @@ Remember to add the following line to config/bundles.php (not required if Symfon CleverAge\DoctrineProcessBundle\CleverAgeDoctrineProcessBundle::class => ['all' => true], ``` -## Reference - -- Tasks - - [DatabaseReaderTask](reference/tasks/database_reader_task.md) - - [DatabaseUpdaterTask](reference/tasks/database_updater_task.md) - - [ClearEntityManagerTask](reference/tasks/doctrine_clear_task.md)) - - [DoctrineBatchWriterTask](reference/tasks/doctrine_batchwriter_task.md) - - [DoctrineCleanerTask](reference/tasks/doctrine_cleaner_task.md) - - [DoctrineDetacherTask](reference/tasks/doctrine_detacher_task.md) - - [DoctrineReaderTask](reference/tasks/doctrine_reader_task.md) - - [DoctrineRefresherTask](reference/tasks/doctrine_refresher_task.md) - - [DoctrineRemoverTask](reference/tasks/doctrine_remover_task.md) - - [DoctrineWriterTask](reference/tasks/doctrine_writer_task.md) +This bundle relies on [doctrine/doctrine-bundle](https://github.com/doctrine/DoctrineBundle) and +[doctrine/orm](https://www.doctrine-project.org/projects/orm.html), installed as dependencies: the bundle +`Doctrine\Bundle\DoctrineBundle\DoctrineBundle` must be enabled as well. + +## Configuration + +This bundle has no configuration of its own: its tasks use the Doctrine connections and entity managers configured in +`config/packages/doctrine.yaml`. + +```yaml +# config/packages/doctrine.yaml +doctrine: + dbal: + url: '%env(resolve:DATABASE_URL)%' + orm: + mappings: + App: + type: attribute + dir: '%kernel.project_dir%/src/Entity' + prefix: 'App\Entity' +``` + +* **Database** tasks ([DatabaseReaderTask](reference/tasks/database_reader_task.md), + [DatabaseUpdaterTask](reference/tasks/database_updater_task.md)) run raw SQL queries through Doctrine DBAL. Their + `connection` option takes the name of a connection (a key under `doctrine.dbal.connections`); the default connection + is used if it is not set. +* **EntityManager** tasks work with Doctrine ORM entities. Except for the + [ClearEntityManagerTask](reference/tasks/doctrine_clear_task.md), which takes the name of an entity manager (a key + under `doctrine.orm.entity_managers`) in its `entity_manager` option, they use the entity manager that manages the + class of the handled entity. + +See the [DoctrineBundle documentation](https://symfony.com/bundles/DoctrineBundle/current/configuration.html) for the +configuration of multiple connections and entity managers. + +## Documentation + +- Cookbooks + - [Database to CSV export](cookbooks/database_to_csv_export.md) + - [CSV to entities import](cookbooks/csv_to_entities_import.md) +- Reference + - Tasks + - Database + - [DatabaseReaderTask](reference/tasks/database_reader_task.md) + - [DatabaseUpdaterTask](reference/tasks/database_updater_task.md) + - EntityManager + - [ClearEntityManagerTask](reference/tasks/doctrine_clear_task.md) + - [DoctrineBatchWriterTask](reference/tasks/doctrine_batchwriter_task.md) + - [DoctrineCleanerTask](reference/tasks/doctrine_cleaner_task.md) + - [DoctrineDetacherTask](reference/tasks/doctrine_detacher_task.md) + - [DoctrineReaderTask](reference/tasks/doctrine_reader_task.md) + - [DoctrineRefresherTask](reference/tasks/doctrine_refresher_task.md) + - [DoctrineRemoverTask](reference/tasks/doctrine_remover_task.md) + - [DoctrineWriterTask](reference/tasks/doctrine_writer_task.md) +- [CleverAge/ProcessBundle documentation](https://github.com/cleverage/process-bundle/blob/main/docs/index.md) diff --git a/docs/reference/tasks/_template.md b/docs/reference/tasks/_template.md index ed1d4a5..919390a 100644 --- a/docs/reference/tasks/_template.md +++ b/docs/reference/tasks/_template.md @@ -1,44 +1,47 @@ TaskName ======== -_Describe main goal an use cases of the task_ +_Describe the main goal and use cases of the task._ Task reference -------------- -* **Service**: `ClassName` +* **Service**: `Fully\Qualified\ClassName` +* **Iterable task** _(only if it implements `IterableTaskInterface`)_ +* **Blocking task** _(only if it implements `BlockingTaskInterface`)_ +* **Flushable task** _(only if it implements `FlushableTaskInterface`)_ Accepted inputs --------------- -_Description of allowed types_ +_Description of allowed types, or "Input is ignored"._ Possible outputs ---------------- -_Description of possible types_ +_Description of possible types._ Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `code` | `type` | **X** _or nothing_ | `default value` _if available_ | _description_ | +| Code | Type | Required | Default | Description | +|--------|--------|:--------:|-----------------|---------------| +| `code` | `type` | **X** | `default value` | _description_ | + +_If the task has no option, replace the table with "This task has no option."._ Examples -------- -_YAML samples and explanations_ - * Example 1 - details - - details - + ```yaml # Task configuration level code: - service: '@service_ref' - options: - a: 1 - b: 2 + service: '@Fully\Qualified\ClassName' + options: + a: 1 + b: 2 + outputs: [next_task] ``` diff --git a/docs/reference/tasks/database_reader_task.md b/docs/reference/tasks/database_reader_task.md index ad77b44..873fe4f 100644 --- a/docs/reference/tasks/database_reader_task.md +++ b/docs/reference/tasks/database_reader_task.md @@ -1,7 +1,9 @@ DatabaseReaderTask ================== -Reads data from a database. +Reads rows from a database using a raw SQL query (Doctrine DBAL, no entity hydration) and outputs them one by one, or +by pages of `paginate` rows. By default, it selects all the columns of the `table`; a custom `sql` query can be given +instead. Useful for fast exports or when there is no Doctrine entity mapped on the data. Task reference -------------- @@ -12,42 +14,100 @@ Task reference Accepted inputs --------------- -`array`or `None`: Input can be used as the query params if needed +Input is ignored, unless `input_as_params` is `true`: the input is then used as the query parameters and must be an +`array` (otherwise an `\UnexpectedValueException` is thrown). Possible outputs ---------------- -`array`: Rows returned by the query. +* `array`: an associative array (column name => value) for each row returned by the query +* `array`: a list of such rows (at most `paginate` rows) if the `paginate` option is set + +If the result set is empty, a log is written with the `empty_log_level` level and the task is skipped. Options ------- -| Code | Type | Required | Default | Description | -|-------------------|--------------------|:--------:|-----------|---------------------------------------------------------------------------| -| `connection` | `string` | | `null` | Doctrine connection (default if not specified) | -| `table` | `string` | **X** | `[]` | Table of the query | -| `sql` | `string` | | `null` | Query to execute (if not specified then: "select tbl.* from `table` tbl") | -| `limit` | `int` or `null` | | `null` | Result max count | -| `offset` | `int` or `null` | | `null` | Result first item offset | -| `paginate` | `int` or `null` | | `null` | Paginate the results | -| `input_as_params` | `bool` | | `false` | Use the input as params | -| `params` | `array` | | `[]` | Query params | -| `types` | `array` | | `[]` | Query params types | -| `empty_log_level` | `string` or `null` | | `warning` | Log level if the result set is empty | - - -Example -------- +| Code | Type | Required | Default | Description | +|-------------------|---------------|:--------:|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `table` | `string` | **X** | | Table to read from when `sql` is not set: the query is `SELECT tbl.* FROM tbl`.
Required even when `sql` is set (its value is then ignored) | +| `connection` | `string\|null` | | `null` | Name of the Doctrine DBAL connection (as defined in `doctrine.dbal.connections`). If `null`, the default connection is used | +| `sql` | `string\|null` | | `null` | Custom SQL query to execute, with optional named (`:name`) or positional (`?`) parameters | +| `limit` | `int\|null` | | `null` | Maximum number of rows. Only used when `sql` is not set | +| `offset` | `int\|null` | | `null` | Index of the first row. Only used when `sql` is not set | +| `paginate` | `int\|null` | | `null` | If set, rows are output by lists of `paginate` rows instead of one by one | +| `input_as_params` | `bool` | | `false` | Use the input as query parameters instead of the `params` option | +| `params` | `array` | | `[]` | Query parameters (ignored if `input_as_params` is `true`) | +| `types` | `array` | | `[]` | Query parameter types, indexed like the parameters (see [DBAL parameter types](https://www.doctrine-project.org/projects/doctrine-dbal/en/current/reference/data-retrieval-and-manipulation.html#list-of-parameters-conversion)) | +| `empty_log_level` | `string` | | `warning` | PSR log level (`Psr\Log\LogLevel` values) used to log an empty result set | + +Examples +-------- + +* Read all the rows of a table ```yaml # Task configuration level -code: +read_books: service: '@CleverAge\DoctrineProcessBundle\Task\Database\DatabaseReaderTask' options: table: 'book' - limit: 10 - offset: 3 - params: - title: "IT" + limit: 100 + offset: 10 empty_log_level: debug -``` \ No newline at end of file + outputs: [next_task] +``` + +* Use a custom query with parameters, and a value coming from the process context + (`bin/console cleverage:process:execute -c lastname:"'King'"`) + +```yaml +# Task configuration level +read_books: + service: '@CleverAge\DoctrineProcessBundle\Task\Database\DatabaseReaderTask' + options: + table: 'book' # Required but not used + sql: > + SELECT b.id, b.title, a.lastname AS author + FROM book b INNER JOIN author a ON a.id = b.author_id + WHERE a.lastname = :lastname + params: + lastname: '{{ lastname }}' + outputs: [next_task] +``` + +* Output rows by pages of 500, using the parameters given by the previous task + +```yaml +# Task configuration level +get_params: + service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' + options: + output: + min_id: 1000 + outputs: [read_books] +read_books: + service: '@CleverAge\DoctrineProcessBundle\Task\Database\DatabaseReaderTask' + options: + table: 'book' + sql: 'SELECT * FROM book WHERE id >= :min_id' + input_as_params: true + types: + min_id: 'integer' + paginate: 500 + outputs: [next_task] +``` + +Notes +----- + +* Options are resolved once per task instance: values coming from the context (`{{ key }}`) are replaced when the + query is first executed. +* The query is executed on the first execution of the task, then rows are fetched from the database one at a time + while the process iterates, so memory usage stays low even with big result sets. The DBAL result is freed when the + process is finalized. +* Array parameters (e.g. for an `IN (:ids)` clause) require an `ArrayParameterType` in `types`, for instance + `ids: !php/enum Doctrine\DBAL\ArrayParameterType::INTEGER` with Doctrine DBAL 4. +* The task is designed to be executed once per process run (typically as the entry point): if it receives a new input + after having iterated over all the rows, that input only resets the task, which is skipped. +* With `paginate`, the row following each full page is currently not output. diff --git a/docs/reference/tasks/database_updater_task.md b/docs/reference/tasks/database_updater_task.md index 29a342a..b7d81fb 100644 --- a/docs/reference/tasks/database_updater_task.md +++ b/docs/reference/tasks/database_updater_task.md @@ -1,7 +1,9 @@ -DatabaseReaderTask -================== +DatabaseUpdaterTask +=================== -Writes data to a database. +Executes a SQL statement (`INSERT`, `UPDATE`, `DELETE`, ...) on a database using Doctrine DBAL, and outputs the number +of affected rows. By default, the input is used as the statement parameters, so the task can be executed once per +received item. Task reference -------------- @@ -11,35 +13,65 @@ Task reference Accepted inputs --------------- -`array`or `None`: Input can be used as the query params if needed +`array`: the statement parameters, when `input_as_params` is `true` (default). Any other type throws an +`\UnexpectedValueException`. + +If `input_as_params` is `false`, input is ignored and the `params` option is used. Possible outputs ---------------- -`int`: Number of rows changed by the query. +`int`: number of rows affected by the statement. Options ------- -| Code | Type | Required | Default | Description | -|-------------------|--------------------|:--------:|-----------|------------------------------------------------| -| `connection` | `string` | | `null` | Doctrine connection (default if not specified) | -| `sql` | `string` | **X** | `null` | Query to execute | -| `input_as_params` | `bool` | | `false` | Use the input as params | -| `params` | `array` | | `[]` | Query params | -| `types` | `array` | | `[]` | Query params types | +| Code | Type | Required | Default | Description | +|-------------------|---------------|:--------:|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `sql` | `string` | **X** | | SQL statement to execute, with optional named (`:name`) or positional (`?`) parameters | +| `connection` | `string\|null` | | `null` | Name of the Doctrine DBAL connection (as defined in `doctrine.dbal.connections`). If `null`, the default connection is used | +| `input_as_params` | `bool` | | `true` | Use the input as statement parameters instead of the `params` option | +| `params` | `array` | | `[]` | Statement parameters (ignored if `input_as_params` is `true`) | +| `types` | `array` | | `[]` | Statement parameter types, indexed like the parameters (see [DBAL parameter types](https://www.doctrine-project.org/projects/doctrine-dbal/en/current/reference/data-retrieval-and-manipulation.html#list-of-parameters-conversion)) | -Example -------- +Examples +-------- + +* Update rows with static parameters, or parameters coming from the process context + (`bin/console cleverage:process:execute -c firstname:"'Stephen'" -c lastname:"'King'"`) ```yaml # Task configuration level -code: +update_authors: service: '@CleverAge\DoctrineProcessBundle\Task\Database\DatabaseUpdaterTask' options: - sql: 'update author set firstname = :firstname, lastname = :lastname' + sql: 'UPDATE author SET firstname = :firstname WHERE lastname = :lastname' input_as_params: false params: - firstname: 'Pascal' - lastname: 'Dupont' -``` \ No newline at end of file + firstname: '{{ firstname }}' + lastname: '{{ lastname }}' + outputs: [next_task] +``` + +* Insert each line of a CSV file (the keys of each line must match the statement parameters) + +```yaml +# Task configuration level +read: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvReaderTask' + options: + file_path: '%kernel.project_dir%/var/data/authors.csv' # Headers: firstname;lastname + outputs: [insert] +insert: + service: '@CleverAge\DoctrineProcessBundle\Task\Database\DatabaseUpdaterTask' + options: + sql: 'INSERT INTO author (firstname, lastname) VALUES (:firstname, :lastname)' +``` + +Notes +----- + +* Options are resolved once per task instance: values coming from the context (`{{ key }}`) are replaced on the first + execution. +* Each execution runs the statement immediately, outside any explicit transaction: for big volumes, consider a + single set-based statement, or the [DoctrineBatchWriterTask](doctrine_batchwriter_task.md) with entities. diff --git a/docs/reference/tasks/doctrine_batchwriter_task.md b/docs/reference/tasks/doctrine_batchwriter_task.md index 6080547..b0b5032 100644 --- a/docs/reference/tasks/doctrine_batchwriter_task.md +++ b/docs/reference/tasks/doctrine_batchwriter_task.md @@ -1,37 +1,76 @@ DoctrineBatchWriterTask ======================= -Writes multiple entities to a database. +Buffers the entities received as input and, every `batch_count` entities, persists them and flushes their entity +manager(s) in one go. Remaining entities are written when the task is flushed, at the end of the upstream iteration. +This is the recommended way to import a large number of entities. Task reference -------------- * **Service**: `CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineBatchWriterTask` +* **Flushable task** Accepted inputs --------------- -`array`: Entities to be persisted in the database +`object`: a Doctrine entity (new or already managed). An object whose class is not managed by any entity manager +throws an `\UnexpectedValueException` when the batch is written. Possible outputs ---------------- -`array`: Batch of the entities persisted to the database +`array`: the list of entities written in the batch (at most `batch_count` entities). The task is skipped when the +input is only buffered, and on flush if there is no remaining entity. Options ------- -| Code | Type | Required | Default | Description | -|---------------|-------|:--------:|---------|-------------| -| `batch_count` | `int` | | `10` | Batch size | +| Code | Type | Required | Default | Description | +|------------------|----------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------| +| `batch_count` | `int` | | `10` | Number of entities to buffer before writing them to the database | +| `entity_manager` | `string\|null` | | `null` | Inherited from the base Doctrine task but not used: the entity manager is the one managing each entity class | -Example -------- +Examples +-------- + +* Write entities by batches of 100, then clear the entity manager after each batch to free memory ```yaml # Task configuration level -code: +denormalize: + service: '@CleverAge\ProcessBundle\Task\Serialization\DenormalizerTask' + options: + class: App\Entity\Author + outputs: [batch_write] +batch_write: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineBatchWriterTask' + options: + batch_count: 100 + outputs: [clear] +clear: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\ClearEntityManagerTask' +``` + +* Get back all the written entities at the end of the process + +```yaml +# Task configuration level +batch_write: service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineBatchWriterTask' options: batch_count: 2 -``` \ No newline at end of file + outputs: [aggregate] +aggregate: + service: '@CleverAge\ProcessBundle\Task\AggregateIterableTask' # Receives one array per batch + outputs: [next_task] +``` + +Notes +----- + +* The batch can contain entities of different classes: each entity is persisted in the entity manager managing its + class, then every involved entity manager is flushed. +* `flush()` writes **all** the pending changes of the entity manager, not only the buffered entities. +* To use the output as a flat list of entities, iterate over it with the + [InputIteratorTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/input_iterator_task.md). diff --git a/docs/reference/tasks/doctrine_cleaner_task.md b/docs/reference/tasks/doctrine_cleaner_task.md index aa32c66..6362f8c 100644 --- a/docs/reference/tasks/doctrine_cleaner_task.md +++ b/docs/reference/tasks/doctrine_cleaner_task.md @@ -1,7 +1,9 @@ DoctrineCleanerTask -==================== +=================== -Clear the entity manager of an entity. +Clears the entity manager that manages the class of the entity received as input: **all** the entities of this +entity manager are detached (not only the input entity). Useful when the entity manager is not the default one, as it +is guessed from the input. Task reference -------------- @@ -11,23 +13,41 @@ Task reference Accepted inputs --------------- -`object`: Entity to be persisted in the database +`object`: a Doctrine entity, used to find its entity manager. A `null` input throws a `\RuntimeException`, and an +object whose class is not managed by any entity manager throws an `\UnexpectedValueException`. Possible outputs ---------------- -None +No output is set. Options ------- -None +| Code | Type | Required | Default | Description | +|------------------|----------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------| +| `entity_manager` | `string\|null` | | `null` | Inherited from the base Doctrine task but not used: the entity manager is the one managing the input's class | -Example -------- +Examples +-------- + +* Clear the entity manager after each processed entity ```yaml # Task configuration level -code: +read_authors: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineReaderTask' + options: + class_name: 'App\Entity\Author' + outputs: [export, clean] +export: + service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask' +clean: service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineCleanerTask' -``` \ No newline at end of file +``` + +Notes +----- + +* Pending changes that have not been flushed are lost. +* To clear an entity manager without any input, or by its name, use the [ClearEntityManagerTask](doctrine_clear_task.md). diff --git a/docs/reference/tasks/doctrine_clear_task.md b/docs/reference/tasks/doctrine_clear_task.md index 8343304..8c847d6 100644 --- a/docs/reference/tasks/doctrine_clear_task.md +++ b/docs/reference/tasks/doctrine_clear_task.md @@ -1,7 +1,7 @@ ClearEntityManagerTask ====================== -Clear the entity manager. +Clears an entity manager: all its managed entities are detached, which frees memory during long imports or exports. Task reference -------------- @@ -11,27 +11,49 @@ Task reference Accepted inputs --------------- -`None` +Input is ignored. Possible outputs ---------------- -`None` +No output is set. Options ------- -| Code | Type | Required | Default | Description | -|------------------|--------------------|:--------:|---------|---------------------------------------------| -| `entity_manager` | `string` or `null` | | `null` | Use another entity manager than the default | +| Code | Type | Required | Default | Description | +|------------------|----------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------------------------| +| `entity_manager` | `string\|null` | | `null` | Name of the entity manager to clear (as defined in `doctrine.orm.entity_managers`). If `null`, the default one is cleared | +Examples +-------- +* Clear the default entity manager after each written batch -Example -------- +```yaml +# Task configuration level +batch_write: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineBatchWriterTask' + options: + batch_count: 100 + outputs: [clear] +clear: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\ClearEntityManagerTask' +``` + +* Clear a specific entity manager ```yaml # Task configuration level -code: +clear: service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\ClearEntityManagerTask' -``` \ No newline at end of file + options: + entity_manager: 'customer' +``` + +Notes +----- + +* Pending changes that have not been flushed are lost. +* Entities read before the clear become detached: they must not be modified and written afterwards without being + fetched again. diff --git a/docs/reference/tasks/doctrine_detacher_task.md b/docs/reference/tasks/doctrine_detacher_task.md index 5fdb9eb..20f6c23 100644 --- a/docs/reference/tasks/doctrine_detacher_task.md +++ b/docs/reference/tasks/doctrine_detacher_task.md @@ -1,7 +1,8 @@ DoctrineDetacherTask ==================== -Detach a Doctrine entity from the entity manager +Detaches the entity received as input from its entity manager: its changes will no longer be tracked nor written by +a subsequent `flush()`, and it can be garbage collected. Task reference -------------- @@ -11,23 +12,42 @@ Task reference Accepted inputs --------------- -`object`: Doctrine managed entity +`object`: a Doctrine entity. A `null` input throws a `\RuntimeException`, and an object whose class is not managed by +any entity manager throws an `\UnexpectedValueException`. Underlying method is Doctrine `EntityManager::detach()`. Possible outputs ---------------- -`None` +No output is set. Options ------- -`None` +| Code | Type | Required | Default | Description | +|------------------|----------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------| +| `entity_manager` | `string\|null` | | `null` | Inherited from the base Doctrine task but not used: the entity manager is the one managing the input's class | -Example -------- +Examples +-------- + +* Detach each entity once it has been exported ```yaml # Task configuration level -code: +read_authors: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineReaderTask' + options: + class_name: 'App\Entity\Author' + outputs: [export, detach] +export: + service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask' +detach: service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineDetacherTask' -``` \ No newline at end of file +``` + +Notes +----- + +* Only the input entity is detached (and its associations configured with `cascade: [detach]`); to detach all the + entities at once, use the [ClearEntityManagerTask](doctrine_clear_task.md) or the + [DoctrineCleanerTask](doctrine_cleaner_task.md). diff --git a/docs/reference/tasks/doctrine_reader_task.md b/docs/reference/tasks/doctrine_reader_task.md index 5e054f8..33c7aa2 100644 --- a/docs/reference/tasks/doctrine_reader_task.md +++ b/docs/reference/tasks/doctrine_reader_task.md @@ -1,50 +1,86 @@ DoctrineReaderTask ================== -Reads Doctrine entity from a repository +Queries Doctrine entities of a given class from their repository, with simple criteria, ordering, limit and offset, and +outputs them one by one. Task reference -------------- * **Service**: `CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineReaderTask` +* **Iterable task** Accepted inputs --------------- -`None` +Input is ignored. Possible outputs ---------------- -`array`: Result set of the entities +`object`: each entity of class `class_name` matching the query. + +If the result set is empty, a log is written with the `empty_log_level` level and the task is skipped. Options ------- +| Code | Type | Required | Default | Description | +|-------------------|---------------|:--------:|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `class_name` | `string` | **X** | | FQCN of the entity to read (e.g. `App\Entity\Author`) | +| `criteria` | `array` | | `[]` | Map of `field: value` conditions, combined with `AND`: a scalar value means `=`, a list of values means `IN (...)` and `null` means `IS NULL`. Field names may only contain letters and digits (`[a-zA-Z0-9]`) | +| `order_by` | `array` | | `[]` | Map of `field: direction` (`asc` or `desc`) | +| `limit` | `int\|null` | | `null` | Maximum number of entities | +| `offset` | `int\|null` | | `null` | Index of the first entity | +| `empty_log_level` | `string` | | `warning` | PSR log level (`Psr\Log\LogLevel` values) used to log an empty result set | +| `entity_manager` | `string\|null` | | `null` | Inherited from the base Doctrine task but not used: the entity manager is the one managing `class_name` | -| Code | Type | Required | Default | Description | -|-------------------|--------------------|:--------:|-----------|------------------------------------------------| -| `class_name` | `string` | **X** | `null` | Name of the class (e.g. : 'App\Entity\Author') | -| `criteria` | `array` | | `[]` | Criteria of the query | -| `order_by` | `array` | | `[]` | Order by of the query | -| `limit` | `int` or `null` | | `null` | Result max count | -| `offset` | `int` or `null` | | `null` | Result first item offset | -| `empty_log_level` | `string` or `null` | | `warning` | Log level if the result set is empty | - +Examples +-------- -Example -------- +* Read authors whose lastname is `King`, ordered by firstname ```yaml # Task configuration level -code: +read_authors: service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineReaderTask' options: class_name: 'App\Entity\Author' criteria: lastname: 'King' order_by: - lastname: 'asc' + firstname: 'asc' limit: 5 offset: 3 -``` \ No newline at end of file + outputs: [next_task] +``` + +* Use an `IN` condition on an association (entity ids), with a value coming from the process context + (`bin/console cleverage:process:execute -c title:"'Dracula'"`) + +```yaml +# Task configuration level +read_books: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineReaderTask' + options: + class_name: 'App\Entity\Book' + criteria: + title: '{{ title }}' + author: [1, 2, 3] + empty_log_level: debug + outputs: [next_task] +``` + +Notes +----- + +* The query is executed on the first execution of the task and **all** the matching entities are loaded in memory + before being output one by one. For big volumes, use `limit`/`offset`, clear the entity manager downstream (see + [ClearEntityManagerTask](doctrine_clear_task.md)) or read raw rows with the + [DatabaseReaderTask](database_reader_task.md). +* Entities stay managed by the entity manager: they can be modified then saved with the + [DoctrineWriterTask](doctrine_writer_task.md). +* The task is designed to be executed once per process run (typically as the entry point): if it receives a new input + after having iterated over all the entities, that input only resets the task, which is skipped. +* For more complex queries, extend `CleverAge\DoctrineProcessBundle\Task\EntityManager\AbstractDoctrineQueryTask` + (which provides the options above and a `getQueryBuilder()` method) or this task. diff --git a/docs/reference/tasks/doctrine_refresher_task.md b/docs/reference/tasks/doctrine_refresher_task.md index d56c428..4d4cb9e 100644 --- a/docs/reference/tasks/doctrine_refresher_task.md +++ b/docs/reference/tasks/doctrine_refresher_task.md @@ -1,7 +1,7 @@ DoctrineRefresherTask ===================== -Refreshes a Doctrine entity from the entity manager +Refreshes the entity received as input from the database, overwriting any unsaved change made to it, then outputs it. Task reference -------------- @@ -11,23 +11,43 @@ Task reference Accepted inputs --------------- -`object`: Doctrine managed entity +`object`: a Doctrine managed entity. A `null` input throws a `\RuntimeException`, and an object whose class is not +managed by any entity manager throws an `\UnexpectedValueException`. Underlying method is Doctrine +`EntityManager::refresh()`, which fails if the entity is not managed. Possible outputs ---------------- -`object`: The refreshed entity +`object`: the refreshed entity. Options ------- -None +| Code | Type | Required | Default | Description | +|------------------|----------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------| +| `entity_manager` | `string\|null` | | `null` | Inherited from the base Doctrine task but not used: the entity manager is the one managing the input's class | -Example -------- +Examples +-------- + +* Modify an entity, then discard the change by refreshing it ```yaml # Task configuration level -code: - service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineRefresherTask' -``` \ No newline at end of file +read_authors: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineReaderTask' + options: + class_name: 'App\Entity\Author' + criteria: + lastname: 'King' + outputs: [modify] +modify: + service: '@CleverAge\ProcessBundle\Task\PropertySetterTask' + options: + values: + firstname: 'Gérard' + outputs: [refresh] +refresh: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineRefresherTask' + outputs: [next_task] +``` diff --git a/docs/reference/tasks/doctrine_remover_task.md b/docs/reference/tasks/doctrine_remover_task.md index 435f126..90f8711 100644 --- a/docs/reference/tasks/doctrine_remover_task.md +++ b/docs/reference/tasks/doctrine_remover_task.md @@ -1,7 +1,7 @@ DoctrineRemoverTask =================== -Removes a Doctrine entity from the entity manager then flushes +Removes the entity received as input and immediately flushes its entity manager, deleting it from the database. Task reference -------------- @@ -11,23 +11,43 @@ Task reference Accepted inputs --------------- -`object`: Doctrine managed entity +`object`: a Doctrine managed entity. An object whose class is not managed by any entity manager throws an +`\UnexpectedValueException`. Possible outputs ---------------- -`None` +No output is set. Options ------- -`None` +| Code | Type | Required | Default | Description | +|------------------|----------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------| +| `entity_manager` | `string\|null` | | `null` | Inherited from the base Doctrine task but not used: the entity manager is the one managing the input's class | -Example -------- +Examples +-------- + +* Delete the books having a given title ```yaml # Task configuration level -code: +read_books: + service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineReaderTask' + options: + class_name: 'App\Entity\Book' + criteria: + title: 'Dracula' + outputs: [remove] +remove: service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineRemoverTask' -``` \ No newline at end of file +``` + +Notes +----- + +* `flush()` writes **all** the pending changes of the entity manager, not only the removal. +* Cascade and `orphanRemoval` rules of the entity mapping apply. To delete many rows at once, a single `DELETE` + statement with the [DatabaseUpdaterTask](database_updater_task.md) is much faster. +* A `null` input is not supported (it throws a `\TypeError`). diff --git a/docs/reference/tasks/doctrine_writer_task.md b/docs/reference/tasks/doctrine_writer_task.md index 5eb42a3..95296ac 100644 --- a/docs/reference/tasks/doctrine_writer_task.md +++ b/docs/reference/tasks/doctrine_writer_task.md @@ -1,7 +1,8 @@ DoctrineWriterTask ================== -Writes a Doctrine entity to the database. +Persists the entity received as input and immediately flushes its entity manager, then outputs the entity. Suited for +low volumes; prefer the [DoctrineBatchWriterTask](doctrine_batchwriter_task.md) for big imports. Task reference -------------- @@ -11,23 +12,46 @@ Task reference Accepted inputs --------------- -`object`: Doctrine managed entity +`object`: a Doctrine entity (new or already managed). A `null` input throws a `\RuntimeException`, and an object whose +class is not managed by any entity manager throws an `\UnexpectedValueException`. Possible outputs ---------------- -`object`: Re-outputs given entity +`object`: the persisted entity (with its generated identifier, if any). Options ------- -`None` +| Code | Type | Required | Default | Description | +|------------------|----------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------| +| `entity_manager` | `string\|null` | | `null` | Inherited from the base Doctrine task but not used: the entity manager is the one managing the input's class | -Example -------- +Examples +-------- + +* Create an entity from an array, then save it ```yaml # Task configuration level -code: +entry: + service: '@CleverAge\ProcessBundle\Task\ConstantOutputTask' + options: + output: + firstname: Isaac + lastname: Asimov + outputs: [denormalize] +denormalize: + service: '@CleverAge\ProcessBundle\Task\Serialization\DenormalizerTask' + options: + class: App\Entity\Author + outputs: [save] +save: service: '@CleverAge\DoctrineProcessBundle\Task\EntityManager\DoctrineWriterTask' -``` \ No newline at end of file + outputs: [next_task] +``` + +Notes +----- + +* `flush()` writes **all** the pending changes of the entity manager, not only the input entity.