Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ Latest

### Changes
* [#17](https://github.com/cleverage/cache-process-bundle/issues/17) Update quality stack: use Rector `withComposerBased()` sets (removed `SYMFONY_64` / `PHPUNIT_100` sets), declare used Symfony packages and PHPUnit range in composer.json, apply quality tools fixes
* [#19](https://github.com/cleverage/cache-process-bundle/issues/19) Add missing documentations: complete Adapter, GetTask & SetTask reference pages, configuration and custom cache tasks guides, cookbooks. Harmonize and fix existing documentation.

v2.0
------
Expand Down
118 changes: 118 additions & 0 deletions docs/cookbooks/cache_warmup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
Warm up a persistent cache from a file
======================================

This recipe loads a reference CSV file into a persistent cache, so that other processes can read its lines by key
without parsing the file again.

First, declare a FrameworkBundle cache pool and wrap it in a cache [adapter](../reference/adapter.md) with the
`catalog` code:

```yaml
# config/packages/cache.yaml
framework:
cache:
pools:
app.cache.catalog:
adapter: cache.adapter.filesystem
default_lifetime: 86400 # One day

# config/services.yaml
services:
app.cleverage_cache_process.adapter.catalog:
class: CleverAge\CacheProcessBundle\Adapter\Adapter
arguments: ['@app.cache.catalog', 'catalog']
tags:
- { name: cleverage.cache.adapter }
```

Then define a process storing each line of the file, and another one reading a line by key:

```yaml
clever_age_process:
configurations:
app.catalog_cache_warmup:
description: 'Store each line of the catalog CSV file in the catalog cache, indexed by sku'
tasks:
read_source:
service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvReaderTask'
options:
file_path: '%kernel.project_dir%/var/data/catalog.csv'
delimiter: ';'
outputs: [filter_valid]

filter_valid:
service: '@CleverAge\ProcessBundle\Task\FilterTask'
options:
not_empty:
'[sku]': ~
outputs: [format]

format:
service: '@CleverAge\ProcessBundle\Task\TransformerTask'
options:
transformers:
mapping:
mapping:
key:
code: '[sku]'
value:
code: '.' # The whole line
outputs: [store, count_rows]

store:
service: '@CleverAge\CacheProcessBundle\Task\SetTask'
error_strategy: skip # A sku which is not a valid cache key is logged and skipped
options:
adapter: 'catalog'
key: '' # Overridden by the input
value: ~ # Overridden by the input

count_rows:
service: '@CleverAge\ProcessBundle\Task\Reporting\StatCounterTask'

app.catalog_cache_read:
description: 'Read a catalog line from the catalog cache'
help: "bin/console cleverage:process:execute app.catalog_cache_read -c sku:\"'ABC-001'\""
tasks:
read:
service: '@CleverAge\CacheProcessBundle\Task\GetTask'
options:
adapter: 'catalog'
key: '{{ sku }}'
outputs: [skip_missing]

skip_missing:
service: '@CleverAge\ProcessBundle\Task\SkipEmptyTask'
outputs: [log]

log:
service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask'
options:
level: info
message: 'Catalog line found in cache'
```

How it works:
- [CsvReaderTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/csv_reader_task.md) is
iterable: each line goes through the following tasks before the next one is read.
- [FilterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/filter_task.md) skips the
lines without `sku`, which cannot be used as cache key.
- The [TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md)
builds a `key` / `value` array with the
[mapping](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/mapping_transformer.md)
transformer (`code: '.'` maps the whole line).
- [SetTask](../reference/tasks/set_task.md) merges this array over its options: `key` and `value` placeholders are
replaced by the values of the current line, which is stored in the `catalog` adapter. With `error_strategy: skip`,
a `sku` containing a PSR-6 reserved character (`{}()/\@:`) is logged and the next line is processed.
- [StatCounterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/stat_counter_task.md)
logs the number of stored lines at the end of the process.
- In the second process, [GetTask](../reference/tasks/get_task.md) reads the key given by the `sku` context value
(see [contextual values](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#contextual-values)).
Since the pool is persistent, the lines stored by the first process are available until they expire.
- A missing key outputs `null`:
[SkipEmptyTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/skip_empty_task.md) stops
the branch, so the [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md)
only logs found lines.

Note that the items expire after the `default_lifetime` of the pool: schedule the warm up process more often than
this lifetime if the other processes must always find the data.
115 changes: 115 additions & 0 deletions docs/cookbooks/share_data_between_branches.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
Share data between process branches
===================================

This recipe uses an in-memory cache to store data in a first branch of a process, then read it in another branch of
the same execution. Nothing is persisted: the values only live during the current PHP process.

First, declare an in-memory cache [adapter](../reference/adapter.md) with the `memory` code:

```php
<?php
// src/Adapter/MemoryAdapter.php

declare(strict_types=1);

namespace App\Adapter;

use CleverAge\CacheProcessBundle\Adapter\Adapter;
use Symfony\Component\Cache\Adapter\ArrayAdapter;

class MemoryAdapter extends Adapter
{
public function __construct()
{
parent::__construct(new ArrayAdapter(), 'memory');
}
}
```

```yaml
# config/services.yaml
services:
app.cleverage_cache_process.adapter.memory:
class: App\Adapter\MemoryAdapter
tags:
- { name: cleverage.cache.adapter }
```

Then define the process:

```yaml
clever_age_process:
configurations:
app.cache_share_data:
description: 'Store items in a first branch, then read some of them in other branches'
tasks:
start:
service: '@CleverAge\ProcessBundle\Task\DummyTask'
outputs: [data, get, get_missing] # Branches are executed in this order

data:
service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask'
options:
output:
- key: 'key1'
column1: value1-1
column2: value2-1
- key: 'key2'
column1: value1-2
column2: value2-2
outputs: [format]

format:
service: '@CleverAge\ProcessBundle\Task\TransformerTask'
options:
transformers:
mapping:
mapping:
key:
code: '[key]'
value:
code: '.'
outputs: [set]

set:
service: '@CleverAge\CacheProcessBundle\Task\SetTask'
options:
adapter: 'memory'
key: '' # Overridden by the input
value: ~ # Overridden by the input

get:
service: '@CleverAge\CacheProcessBundle\Task\GetTask'
options:
adapter: 'memory'
key: 'key2'
outputs: [debug]

get_missing:
service: '@CleverAge\CacheProcessBundle\Task\GetTask'
options:
adapter: 'memory'
key: 'missing'
outputs: [debug]

debug:
service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask'
```

How it works:
- [DummyTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/dummy_task.md) starts three
branches. They are executed one after the other, in the order of `outputs`: the `data` branch is fully processed
before `get` and `get_missing` are executed.
- [ConstantIterableOutputTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/constant_iterable_output_task.md)
outputs each item one by one, the
[TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md)
maps it to a `key` / `value` array, and [SetTask](../reference/tasks/set_task.md) stores it in the `memory` adapter.
- [GetTask](../reference/tasks/get_task.md) `get` outputs the item stored under `key2` (the whole original item,
`key` column included), and `get_missing` outputs `null` since `missing` was never stored.
- [DebugTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/debug_task.md) dumps both
values.

Note that the adapter service is shared: two processes executed in the same PHP process (e.g. with the
[ProcessExecutorTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/process_executor_task.md))
also share the in-memory values. To share data between separate executions, use a persistent pool instead (see
[Warm up a persistent cache from a file](cache_warmup.md)).
63 changes: 62 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Prerequisite

CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation.
CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation).

## Installation

Expand All @@ -19,9 +19,70 @@ Remember to add the following line to config/bundles.php (not required if Symfon
CleverAge\CacheProcessBundle\CleverAgeCacheProcessBundle::class => ['all' => true],
```

## Configuration

The bundle has no semantic configuration. To use the cache tasks, declare at least one cache
[adapter](reference/adapter.md): a service implementing `CleverAge\CacheProcessBundle\Adapter\AdapterInterface`
(usually `CleverAge\CacheProcessBundle\Adapter\Adapter`, wrapping any Symfony cache pool), tagged
`cleverage.cache.adapter`. Its code is then used in the `adapter` option of the tasks.

```yaml
# config/services.yaml
services:
app.cleverage_cache_process.adapter.app:
class: CleverAge\CacheProcessBundle\Adapter\Adapter
arguments: ['@cache.app', 'app']
tags:
- { name: cleverage.cache.adapter }
```

## Custom cache tasks

`CleverAge\CacheProcessBundle\Task\AbstractCacheTask` can be extended to implement other cache operations. It extends
[AbstractConfigurableTask](https://github.com/cleverage/process-bundle/blob/main/docs/03-custom_tasks.md), requires
the `cleverage_cache_process.registry.adapter` service (`AdapterRegistry`) as constructor argument, defines the
required `adapter` and `key` string options, and provides `getMergedOptions()` (options merged with the array input)
and `$this->registry->getAdapter($code)`.

```php
<?php

declare(strict_types=1);

namespace App\Task;

use CleverAge\CacheProcessBundle\Task\AbstractCacheTask;
use CleverAge\ProcessBundle\Model\ProcessState;

class DeleteTask extends AbstractCacheTask
{
public function execute(ProcessState $state): void
{
/** @var array{adapter: string, key: string} $options */
$options = $this->getMergedOptions($state);

$this->registry->getAdapter($options['adapter'])->deleteItem($options['key']);
}
}
```

```yaml
# config/services.yaml
services:
App\Task\DeleteTask:
public: true
shared: false
arguments: ['@cleverage_cache_process.registry.adapter']
```

## Reference

- [Adapter](reference/adapter.md)
- Tasks
- [GetTask](reference/tasks/get_task.md)
- [SetTask](reference/tasks/set_task.md)

## Cookbooks

- [Share data between process branches](cookbooks/share_data_between_branches.md)
- [Warm up a persistent cache from a file](cookbooks/cache_warmup.md)
Loading
Loading