diff --git a/libs/xtm-oaev-sdk/CHANGELOG.md b/libs/xtm-oaev-sdk/CHANGELOG.md index abf7aa5..798e3df 100644 --- a/libs/xtm-oaev-sdk/CHANGELOG.md +++ b/libs/xtm-oaev-sdk/CHANGELOG.md @@ -4,6 +4,100 @@ Every notable change to xtm-oaev-sdk lands here. Every seam change too, with the version of its seam ([seam-changes-are-versioned-contract-changes](docs/tdr/seam-changes-are-versioned-contract-changes.tdr.md)). +## [Unreleased] + +### Added + +- `port_io`: the platform's own payloads, key for key, so injectors and collectors + running without the platform read and write the same JSON. Injector side: + `ExecutableInject` (the queue message, with `ExecutableInjection`, + `ExecutableInjectorContract`, `EndpointTargetOutput`, `AgentOutput`, + `AssetGroupSimple`), `InjectReceptionInput` and `InjectExecutionInput`. Collector + side: `InjectExpectationOutput` (with `InjectExpectationSignature`, + `InjectExpectationResult`), `InjectExpectationUpdateFromCollectorInput`, + `InjectExpectationBulkUpdateInput`, `InjectExpectationTraceInput` and + `InjectExpectationTraceBulkInsertInput`. Each model is written by hand from the + platform DTO it mirrors, names that DTO in its docstring, and ships with a sample + payload. Dates are kept as the platform wrote them, ISO text or epoch numbers. + `PORT_IO_CONTRACT_VERSION` `1.1.0`. +- `daemons`: the dev options common to every daemon. `DevOptions` (seam model) and + `dev_options(argv=None)` expose `--dev`, `--out`, `--record`, `--replay` and the dev + help (`-h` / `--help`, only together with `--dev`); `dev_help()` describes them, and + `dev_argument_parser(description=None, epilog=None)` returns a new parser holding + them, which the injector and collector SDKs extend with their own options. They are + read only under their full name, and flags the daemon does not own are left alone, + since every daemon reads them, dev or not. The port gains + `DaemonRuntimePort.dev_argument_parser`, `dev_options` and `dev_help`. Seam `1.1.0`. +- `daemons`: the dev sink, where a daemon in dev mode writes the calls the platform + would have received. `dev_sink(out=None, secrets=(), keep=(), to_stderr=False)` + checks each call against the payload the platform accepts on its path (a body it + would reject raises `OpenAEVHttpError` with `response_code` 400, and nothing is + written), redacts the secrets (`***`, JSON text redacted as JSON) except in the + `keep` keys, and writes the calls as one JSON list, indented, keys in the payload's + order: rewritten whole after each call in `out`, so the file is valid JSON at any + time, or written once, on close, to stdout or stderr. It also writes the run's notes + and refusals on stderr. `environment_secret_values(environ=None)` lists the values of + the secret-named environment variables. A run can `wait_for(condition, timeout)` a + call another thread records, the `complete` callback of an inject for one. The + paths checked are the injector's execution reception and callback, and the + collector's expectation update, bulk update, trace and bulk traces. Ports + `DevSinkPort` and `DevSinkFactoryPort`. Seam `1.1.0`. +- `common`: `NeedsPlatformError`, the `OpenAEVError` of a call only the platform can + answer, made by a daemon running without it; `call` names the call refused. Seam + `1.1.0`. +- `configuration`: `Configuration.secret_values()` returns the values that must never + be written out, longest first: those of the keys whose name says they hold a secret + (`*_token`, `*_secret`, `*_password`, `*_key`…) and every `SecretStr` of the settings + model, wherever they were read from. Port `ConfigurationResolverPort.secret_values`. + Seam `1.1.0`. +- `contracts`: `ContractValidator` checks inject content against a contract without + the platform, by the platform's own health-check rules (`HealthCheckUtils`, + `InjectModelHelper`): mandatory fields (a text field with a non-blank default counts + as set), mandatory groups and conditionally mandatory fields, plus select choices and + single-value cardinality. `provided` names the fields the inject carries apart from + its content, or maps them to the ids a condition compares. It returns + `ContractViolation` records; the port is `ContractValidationPort`. Its verdicts are + checked against the cases of the platform's `HealthCheckUtilsTest`. Seam `1.1.0`. +- `tape` (new): the call tape. `CallTape(path, mode, secrets=(), daemon=None)` + records the process's outside calls (HTTP through `http.client`, so `requests`, + urllib3 and `urllib.request`; commands through `subprocess.run`) and their answers + to a tape, or replays them offline. A tape is one JSON document (`Tape`), indented, + a header (`tape_version` `1`, `sdk_version`, `daemon`, `recorded_at`) then the calls, + rewritten whole after each call so it is valid JSON at any time. A JSON body is kept + as JSON (`TapedBody.json`), a compressed answer decompressed when readable + (`TapedHttpResponse.decoded_from`) and replayed without its `Content-Encoding`. + Replay matches a call by method, URL and body (JSON by value), answers identical + calls in the order recorded then the last one again, and fails a call the tape does + not hold with `TapeMissError` (a `ConnectionError`), as it fails any other outbound + connection. A tape it cannot read (not JSON, another tape version, a call it does + not model) is refused, saying why. Secrets (the given values, secret-named headers, + parameters and body fields) stay off the tape, and replay matches a call whatever + secret it carries. `lint_tape(path, secrets=())` names each place of a tape holding + what looks like a secret. Port `CallTapePort`. Seam `tape` `1.0.0`. +- `signatures`: `LEGACY_SIGNATURE_TYPES` maps each legacy signature name to its + signature type (`start_time` to `start_date`, `source_ipv4` to + `source_ipv4_address`, `target_hostname` to `target_hostname_address`, and so on: the + pairs of the platform's migration of stored signatures). Seam `1.1.0`. + +### Changed + +- `port_io`: `ExecutionStatus` gains `ASSET_CLEARANCE_RESTRICTED`, a status the + platform accepts. `PORT_IO_CONTRACT_VERSION` `1.1.0`. +- `daemons`: a daemon no longer exits with code 2 on a command-line flag it does not + own. `--dump-config-schema` detection only passes that flag to its parser, so dev + flags and the daemon's own flags can share the command line. Seam `1.1.0`. +- `configuration`: under `--dev`, a settings model no longer needs the platform URL + and token. A last, lowest-priority source fills every `ConfigLoaderOAEV` section the + configuration leaves out with an unreachable platform (`http://platform.openaev.invalid`); + whatever the configuration gives still wins, and without `--dev` both stay required. + Seam `1.1.0`. +- `signatures`: `SignatureManager.build_payload` sends the signature types instead of + the `ExecutionSignature` field names (`start_date`, not `start_time`): the types are + what collectors match on. This applies with and without `--dev`. Seam `1.1.0`. +- `signatures`: `SignatureTypes` reads a legacy name as its signature type + (`SignatureTypes("start_time") is SignatureTypes.SIG_TYPE_START_DATE`), so an + expectation stored with a legacy name still loads. Seam `1.1.0`. + ## [0.1.0] - Unreleased First version. The shared, client-free code of `pyoaev` 3.261014.1 moves out and into diff --git a/libs/xtm-oaev-sdk/README.md b/libs/xtm-oaev-sdk/README.md index 964131f..2742234 100644 --- a/libs/xtm-oaev-sdk/README.md +++ b/libs/xtm-oaev-sdk/README.md @@ -82,6 +82,13 @@ core answering the seam and the public API keeps working. Touch the seam and you have changed a contract: `SEAM_VERSION` bump, changelog entry. The full wiring is in [docs/seam_strategy.md](docs/seam_strategy.md). +## Dev mode + +Run an injector or a collector without the OpenAEV platform: `--dev` reads the platform's own payloads, +writes every call the platform would have received as one readable JSON list, chains an +injector into a collector, and records and replays the outside calls offline. See +[docs/how_to_dev_mode.md](docs/how_to_dev_mode.md). + ## Migrating from pyoaev Every former `pyoaev` path still works, and each one fires a `DeprecationWarning` diff --git a/libs/xtm-oaev-sdk/docs/how_to_dev_mode.md b/libs/xtm-oaev-sdk/docs/how_to_dev_mode.md new file mode 100644 index 0000000..fc9429d --- /dev/null +++ b/libs/xtm-oaev-sdk/docs/how_to_dev_mode.md @@ -0,0 +1,304 @@ +# How to run an injector or a collector without the platform (dev mode) + +This guide is for you if you write or debug an injector or a collector and want to run it +**without an OpenAEV platform**: no registration, no RabbitMQ, no heartbeat, no call to the +platform. The daemon still does its real work (it runs the real tool, queries the real +security product), and everything it would have sent to the platform is written to one +readable JSON file instead. + +Dev mode reads and writes the platform's **own payloads**, key for key. You feed it what +the platform would have sent, and you get back what the platform would have received. + +## TL;DR + +```bash +python -m shodan --dev --help # every dev option +python -m shodan --dev --list-contracts # its contracts +python -m shodan --dev --new-inject "ip enum" > inject.json # a message to edit +# fill what stderr says is still to fill, then: +python -m shodan --dev --inject inject.json --out run.json # run it +python -m my_collector --dev --expectations expectations.json --once --out run.json +python -m shodan --dev --inject inject.json --emit expectations --out injector.json \ + | python -m my_collector --dev --expectations - --once --out collector.json # chain +python -m shodan --dev --inject inject.json --record shodan.tape.json # record its outside calls +python -m shodan --dev --inject inject.json --replay shodan.tape.json # then run offline +``` + +## 1. Before you start + +**The SDKs.** Dev mode lives in `injectors_sdk.OpenAEVInjectorHelper` and +`collectors_sdk.CollectorDaemon`. Daemons that import these names from `pyoaev` get the +very same objects through the deprecation shims, so they get dev mode too, **as long as +their `pyoaev` ships the SDKs**. Check with `pip show pyoaev`: `Requires` must list +`xtm-oaev-sdk`, `injectors-sdk` and `collectors-sdk`. + +**The configuration still loads first**, but under `--dev` it no longer needs the +platform: a `ConfigLoaderOAEV` section the configuration leaves out gets an unreachable +platform (`http://platform.openaev.invalid`). What your configuration gives still wins. +Without `--dev`, the platform URL and token stay required. + +Without `--dev`, nothing changes: the daemon registers, pings and consumes exactly as +before. The one behavior shared by both modes: injectors send their signatures under the +**signature types** collectors match on (`start_date`, `target_ipv4_address`…), and +collectors read a legacy name as its type and drop an unknown type with a warning. + +## 2. The flags + +`--dev --help` (or `-h`) prints the common options and the daemon's own, then exits 0. +An injector shows: + +```text +common options: + --dev run without the platform + -h, --help show this help and exit + --out PATH where the calls the daemon would have sent go, as one JSON + list (default: stdout) + --record PATH record every outside call (HTTP, commands) and its answer + to PATH + --replay PATH answer every outside call from PATH, offline; a call it + does not hold fails + +injector options: + --inject PATH the inject message to run, as the platform publishes it + --timeout SECONDS how long to wait for the inject to complete (default: 600) + --emit expectations also write on stdout the pending expectations collectors + would then get + --dump-contracts write the registration, contracts included, and exit + --list-contracts list the contracts it serves, and exit + --new-inject CONTRACT write an inject message to edit for CONTRACT +``` + +A collector shows the common options and `--expectations PATH` and `--once`. Every daemon +reads these flags, dev or not, so they never act on what is not theirs: they are only read +under their **full name**, `-h` / `--help` only counts together with `--dev`, and any flag +they do not own is left alone. `--dump-config-schema` still works alongside them, and +still wins. + +From Python: `xtm_oaev_sdk.dev_options()` returns the common options (`DevOptions`), +`injectors_sdk.injector_dev_options()` and `collectors_sdk.collector_dev_options()` the +role's (`InjectorDevOptions`, `CollectorDevOptions`), each with its `*_dev_help()`. + +## 3. Run an injector + +### Write the inject message + +```bash +python -m shodan --dev --list-contracts +python -m shodan --dev --new-inject "ip enum" > inject.json +``` + +The contract is named by its id or by any piece of a label, whatever the case. A name +that matches no contract, or several, is refused with exit code 2 and the candidates +listed. On stderr, a guide to each field (type, when it is mandatory, value, choices, where +targets go), then what is still to fill. On stdout, a lean inject message with only the +keys you edit, prefilled the way the platform's inject form fills a new inject. An +`asset-groups` target type is switched to `manual`, else `assets`: only the platform can +resolve the assets of a group. + +### Run it + +```bash +python -m shodan --dev --inject inject.json --out run.json +echo $? # 0 completed, 1 failed or never completed, 2 refused before running +``` + +`--inject` takes the JSON the platform publishes to the injector's queue +(`ExecutableInject`), from a file or `-` for stdin. A lean message is completed the way +the platform fills it: the contract from the injector's own (both contract id paths), an +inject id **derived from the message** (the same message writes the same calls at every +run), a title, and the content defaults under the content you gave. A whole message, +captured from a platform run, is handed over unchanged. + +**Before anything runs**, the message is checked the way the platform checks it, and a +refusal says why on stderr and exits with code 2: it must be an inject message naming a +contract this injector serves, and its content must satisfy the contract by the +platform's own health-check rules (`ContractValidator`; a text field with a non-blank +default counts as set; targets the message carries count as set). + +**The run ends** on the inject's `complete` callback, or after `--timeout` seconds without +one (exit 1, saying so). The exit code comes from the `complete` status: 1 for the +platform's failure family (`ERROR`, `COMMAND_NOT_FOUND`, `TIMEOUT`…), 0 otherwise. + +**Calls only the platform can answer** (a document download, a target search, any manager +the stand-in does not offer) raise `xtm_oaev_sdk.NeedsPlatformError`, naming the call. + +### What it writes + +One JSON list, indented, keys in the platform payload's order, so it reads well and diffs +cleanly. With `--out`, it is rewritten after each call, so the file is valid JSON at any +time; without, it goes to stdout when the run ends. + +```json +[ + { + "method": "POST", + "path": "/api/injects/execution/callback/5dca53ff-…", + "body": { + "execution_message": "Scan done", + "execution_status": "SUCCESS", + "execution_action": "complete" + } + } +] +``` + +Each body is checked against what the platform accepts (`InjectReceptionInput`, +`InjectExecutionInput`). A call the platform would reject is refused the way the platform +answers it: the injector gets an `OpenAEVHttpError` with `response_code` 400, and nothing +is written. `--dump-contracts` writes the registration (`POST /api/injectors`). + +## 4. Run a collector + +```bash +python -m my_collector --dev --expectations expectations.json --once --out run.json +``` + +`--expectations` takes exactly what the platform serves on its expectation endpoints: a +JSON **list** of `InjectExpectationOutput`, from a file or `-` for stdin. Capture a real +one with the collector's own id and token: + +```bash +curl -s -H "Authorization: Bearer $OPENAEV_TOKEN" \ + "$OPENAEV_URL/api/injects/expectations/assets/$COLLECTOR_ID?expiration_time=360" \ + > expectations.json +``` + +It is served through `expectations_assets_for_source`, `detection_expectations_for_source`, +`prevention_expectations_for_source` and `collectors_sdk.expectations_models_for_source`. +An answered expectation leaves the feed, as on the platform. The updates, bulk updates and +traces are written like an injector's calls, checked against +`InjectExpectationUpdateFromCollectorInput`, `InjectExpectationBulkUpdateInput`, +`InjectExpectationTraceInput` and `InjectExpectationTraceBulkInsertInput`. A run without +`--expectations`, or with a file that is not a list of pending expectations, exits 2. +Registration still runs against a local stand-in that hands back ids and writes nothing; +any other manager raises `NeedsPlatformError`. Without `--once`, the cycle runs every +`collector_period` until you stop it. + +## 5. Chain an injector into a collector + +```bash +python -m my_injector --dev --inject inject.json --emit expectations --out injector.json \ + | python -m my_collector --dev --expectations - --once --out collector.json +``` + +`--emit expectations` makes the injector also write, **on stdout**, the pending +expectations the platform would then serve to collectors. Without `--out`, its calls go to +stderr, so stdout always carries the pending expectations alone. The signatures are read +from the execution callbacks the injector posted, the way the platform's +`SignatureOutputProcessor` reads them: + +- the target is the agent (`agent_id` or `agent`), else the asset, else the asset group; + a signature naming none is dropped; +- signatures attach to DETECTION and PREVENTION expectations only, and only to those the + inject content declares; +- a signature reads its type and value from `signature_type`/`type` and + `signature_value`/`value`, keeps scalars with text, and a type/value pair is kept once; +- without `expectation_expiration_time`, an expectation expires after 21600 seconds. + +What the platform would have dropped is noted on stderr (`dev: …`). Dev mode has no +expectation store: every target a signature names is taken as a target of the inject. + +## 6. Record the outside calls, replay them offline + +```bash +python -m shodan --dev --inject inject.json --record shodan.tape.json --out run.json # online, once +python -m shodan --dev --inject inject.json --replay shodan.tape.json --out run.json # offline, any time +``` + +It works for collectors the same way (`--expectations … --once --record splunk.tape.json`). + +| Calls | Through | On replay | +|---|---|---| +| HTTP and HTTPS | `requests`, urllib3, `urllib.request` (so most product SDKs) | answered from the tape; no socket opens | +| commands | `subprocess.run` | exit code, stdout and stderr from the tape; the command does not run | +| anything else | any other outbound connection | **fails** with `TapeMissError` | + +**The tape** is one JSON document, meant to be committed and reviewed in a diff: + +```json +{ + "tape_version": "1", + "sdk_version": "0.1.0", + "daemon": "Shodan", + "recorded_at": "2026-11-10T08:00:00Z", + "calls": [ + { + "kind": "http", + "request": {"method": "GET", "url": "https://api.shodan.io/shodan/host/1.2.3.4?key=***", "headers": [["X-Api-Key", "***"]]}, + "response": {"status": 200, "reason": "OK", "headers": [["Content-Type", "application/json"]], "body": {"json": {"ip_str": "1.2.3.4"}}} + } + ] +} +``` + +A JSON body is kept as JSON (`json`), other text as `text`, binary in `base64`. A +compressed answer is kept decompressed when readable, `decoded_from` naming its encoding, +and replays without its `Content-Encoding`. The tape is rewritten after each call, so it is +valid JSON even after a crash. + +**On replay**, a call is found by its method, URL and body (a JSON body by value, however +it is spaced; a command by its argv and input). Identical calls get their answers in the +order recorded, then the last one again. A call the tape does not hold **fails** with +`TapeMissError` (a `ConnectionError`), naming it. A tape that is not JSON, of another tape +version, or holding a call the SDK does not model is refused, saying why (exit 2). + +**Secrets stay off the tape**: the configuration's and the environment's secret values, +and whatever goes in a secret-named header, parameter or body field. Replay matches a call +whatever secret it carries, so a tape recorded with your key replays with any key, or +none. Before committing a tape, lint it: + +```python +from xtm_oaev_sdk import lint_tape +lint_tape("shodan.tape.json", secrets=["my-real-key"]) # [] when clean +``` + +`--record` and `--replay` together are refused (exit 2). From Python, the same tape is +`xtm_oaev_sdk.CallTape(path, "record" | "replay", secrets=…, daemon=…)`, with `start()` +and `stop()`, or as a context manager. + +## 7. Secrets stay out of the output + +Before anything is written, dev mode replaces with `***` the values of the configuration +keys whose name says they hold a secret (`*_token`, `*_secret`, `*_password`, `*_key`…), +every `SecretStr` of the settings model (`Configuration.secret_values()`), and the values +of environment variables with such names (`environment_secret_values()`). Ids, statuses, +actions and dates are never rewritten. JSON text holding a secret is redacted as JSON. + +## 8. The payload models + +They live in `xtm_oaev_sdk`, are written by hand from the platform DTOs they mirror (each +model's docstring names its DTO source), and ship with a sample payload. What the platform +sends keeps the keys a model does not list; what a daemon sends refuses any key the +platform does not take. Dates are kept as the platform wrote them, ISO text or epoch. + +| Model | Payload | +|---|---| +| `ExecutableInject` | the inject message on the injector queue | +| `InjectReceptionInput` | `POST /api/injects/execution/reception/{inject_id}` | +| `InjectExecutionInput` | `POST /api/injects/execution/callback/{inject_id}` | +| `InjectExpectationOutput` | one pending expectation | +| `InjectExpectationUpdateFromCollectorInput` | `PUT /api/injects/expectations/{expectation_id}` | +| `InjectExpectationBulkUpdateInput` | `PUT /api/injects/expectations/bulk` | +| `InjectExpectationTraceInput` | `POST /api/inject-expectations-traces` | +| `InjectExpectationTraceBulkInsertInput` | `POST /api/inject-expectations-traces/bulk` | + +## 9. When it does not work + +| You see | Why | Do | +|---|---|---| +| connection errors to `openaev_url` or RabbitMQ | no `--dev`: the daemon runs against the platform | add `--dev` | +| `dev: refused: …` and exit code 2 | the message, its contract, its content, the expectations or the tape failed the checks | read the reason; `--new-inject` shows what to fill | +| `NeedsPlatformError: … needs the platform` | a document download, a target search, or a manager the stand-in does not offer | give the targets as `assets`, not `assetGroups` | +| `dev: no complete callback … within … seconds`, exit 1 | the injector never posted `complete` | raise `--timeout`, or fix the injector | +| `TapeMissError: not on the replay tape: …` | the replayed run made a call the recording did not | record again | +| exit code 1, `complete` with `ERROR` | the injector failed for real (tool, API key, network) | read `execution_message` in the output | + +## 10. Known limits + +- **The tape** covers HTTP through `http.client` and commands through `subprocess.run`. + Other clients (`httpx`, `aiohttp`, SMTP, `subprocess.Popen` directly) are not taped: on + replay, their connections fail closed instead of reaching the network. +- **Signatures over 5 MiB** are written as one callback; the platform client splits them + per target. +- **Dates** in the output differ between a run and its replay when the daemon writes the + time it ran; compare those runs dates aside. diff --git a/libs/xtm-oaev-sdk/test/common/test_needs_platform_error.py b/libs/xtm-oaev-sdk/test/common/test_needs_platform_error.py new file mode 100644 index 0000000..78c7ec2 --- /dev/null +++ b/libs/xtm-oaev-sdk/test/common/test_needs_platform_error.py @@ -0,0 +1,44 @@ +"""Tests for the error of a call only the platform can answer. + +Feature: A call that needs the platform + A daemon running without the platform (dev mode) gets a stand-in for the platform + client. A call only the platform can answer fails with an error naming the call, so the + developer knows what dev mode cannot do instead of meeting a bare attribute error. +""" + +from typing import Any + +import xtm_oaev_sdk as sdk + + +def _given_needs_platform_error() -> Any: + """Given the SDK has an error for the calls that need the platform""" + error = getattr(sdk, "NeedsPlatformError", None) + assert error is not None, "Expected xtm_oaev_sdk.NeedsPlatformError" + return error + + +def _when_raising_for(error: Any, call: str) -> Exception: + """When a dev stand-in refuses "" """ + try: + raise error(call) + except Exception as raised: # pylint: disable=broad-except + return raised + + +def test_a_call_that_needs_the_platform_fails_naming_it() -> None: + """It names the call and says dev mode cannot answer it. + + Scenario: A call that needs the platform fails, naming it + Given the SDK has an error for the calls that need the platform + When a dev stand-in refuses "document.download" + Then the error is an OpenAEV error + And it names "document.download" and says it needs the platform, not available in dev mode + """ + error = _given_needs_platform_error() + raised = _when_raising_for(error, "document.download") + assert isinstance(raised, sdk.OpenAEVError) + assert raised.call == "document.download" + assert str(raised) == ( + "document.download needs the platform: not available in dev mode" + ) diff --git a/libs/xtm-oaev-sdk/test/configuration/test_dev_configuration.py b/libs/xtm-oaev-sdk/test/configuration/test_dev_configuration.py new file mode 100644 index 0000000..22413b7 --- /dev/null +++ b/libs/xtm-oaev-sdk/test/configuration/test_dev_configuration.py @@ -0,0 +1,146 @@ +"""Tests for loading a daemon configuration in dev mode. + +Feature: Configuration in dev mode + A daemon started with --dev never reaches the platform, so its configuration must + load without the platform URL and token. Whatever the configuration gives still wins; + without --dev, the platform URL and token stay required. +""" + +import sys +from pathlib import Path +from typing import Any, NamedTuple + +import pytest +import xtm_oaev_sdk as sdk +from pydantic import Field, ValidationError +from pydantic_settings import SettingsConfigDict + +PROGRAM = "openaev-shodan" +PLATFORM_ENV = ("OPENAEV_URL", "OPENAEV_TOKEN", "OPENAEV_TENANT_ID") + + +class Loading(NamedTuple): + settings: Any + error: ValidationError | None + + +class _ShodanSettings(sdk.SettingsLoader): + model_config = SettingsConfigDict(env_nested_delimiter="_", env_nested_max_split=1) + + openaev: sdk.ConfigLoaderOAEV = Field(default_factory=sdk.ConfigLoaderOAEV) + shodan_api_key: str = "dev-key" + + +def _given_no_platform_configuration( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + """Given no configuration file nor environment variable names the platform""" + work = tmp_path / "injector" + work.mkdir() + monkeypatch.chdir(work) + for name in PLATFORM_ENV: + monkeypatch.delenv(name, raising=False) + + +def _given_platform_url_in_environment(monkeypatch: pytest.MonkeyPatch) -> None: + """Given the environment names the platform "https://openaev.example" """ + monkeypatch.setenv("OPENAEV_URL", "https://openaev.example") + monkeypatch.setenv("OPENAEV_TOKEN", "real-token") + + +def _given_process_command_line(monkeypatch: pytest.MonkeyPatch, *args: str) -> None: + """Given the process command line contains "" """ + monkeypatch.setattr(sys, "argv", [PROGRAM, *args]) + + +def _when_loading_settings() -> Loading: + """When the daemon loads its settings""" + try: + return Loading(_ShodanSettings(), None) + except ValidationError as error: + return Loading(None, error) + + +def _then_settings_load(loading: Loading) -> None: + """Then the settings load""" + assert loading.error is None, f"Expected the settings to load: {loading.error}" + + +def _then_settings_are_refused_for(loading: Loading, *fields: str) -> None: + """Then the settings are refused for missing """ + assert loading.error is not None, "Expected the settings to be refused" + failing = {str(error["loc"][-1]) for error in loading.error.errors()} + assert set(fields) <= failing, f"Expected {fields} to fail, got {failing}" + + +def _then_platform_is(loading: Loading, url: str, token: str) -> None: + """Then the platform is with the token """ + assert (str(loading.settings.openaev.url), loading.settings.openaev.token) == ( + url, + token, + ) + + +def _then_platform_is_a_placeholder_never_reached(loading: Loading) -> None: + """Then the platform URL is a placeholder that can never be reached""" + host = loading.settings.openaev.url.host + assert host.endswith(".invalid"), f"Expected an .invalid host, got {host}" + assert loading.settings.openaev.token + + +def test_configuration_loads_without_platform_keys_under_dev( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + """It loads the settings without the platform URL and token under --dev. + + Scenario: The configuration loads without platform keys under --dev + Given no configuration file nor environment variable names the platform + And the process command line contains "--dev" + When the daemon loads its settings + Then the settings load + And the platform URL is a placeholder that can never be reached + """ + _given_no_platform_configuration(monkeypatch, tmp_path) + _given_process_command_line(monkeypatch, "--dev") + loading = _when_loading_settings() + _then_settings_load(loading) + _then_platform_is_a_placeholder_never_reached(loading) + + +def test_configured_platform_keys_still_win_under_dev( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + """It keeps the platform URL and token the configuration gives, dev or not. + + Scenario: Configured platform keys still win under --dev + Given no configuration file nor environment variable names the platform + And the environment names the platform "https://openaev.example" + And the process command line contains "--dev" + When the daemon loads its settings + Then the platform is "https://openaev.example/" with the token "real-token" + """ + _given_no_platform_configuration(monkeypatch, tmp_path) + _given_platform_url_in_environment(monkeypatch) + _given_process_command_line(monkeypatch, "--dev") + loading = _when_loading_settings() + _then_platform_is(loading, "https://openaev.example/", "real-token") + + +# ---- Constraints identified ---- + + +def test_platform_keys_stay_required_without_dev( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + """It still requires the platform URL and token without --dev. + + Scenario: Platform keys stay required without --dev + Given no configuration file nor environment variable names the platform + And the process command line contains "--once" + When the daemon loads its settings + Then the settings are refused for missing "url" and "token" + """ + _given_no_platform_configuration(monkeypatch, tmp_path) + _given_process_command_line(monkeypatch, "--once") + loading = _when_loading_settings() + _then_settings_are_refused_for(loading, "url", "token") diff --git a/libs/xtm-oaev-sdk/test/configuration/test_secret_values.py b/libs/xtm-oaev-sdk/test/configuration/test_secret_values.py new file mode 100644 index 0000000..1fb3596 --- /dev/null +++ b/libs/xtm-oaev-sdk/test/configuration/test_secret_values.py @@ -0,0 +1,139 @@ +"""Tests for the secret values of a configuration: what must never be written out. + +Feature: Secret values of a configuration + A connector running without the platform writes what it would have sent, and its + tools' raw output, to local files. The values its configuration holds as secrets + are listed so they can be redacted from everything written. +""" + +from pathlib import Path +from typing import Any + +import xtm_oaev_sdk as sdk +from pydantic import BaseModel, SecretStr + + +class _Splunk(BaseModel): + password: SecretStr + username: str + + +class _Settings(BaseModel): + splunk: _Splunk + + +def _given_configuration(tmp_path: Path, **kwargs: Any) -> sdk.Configuration: + """Given a connector configuration with: + | key | value | + | openaev_token | platform-token | + | shodan_api_key | shodan-key-1 | + | crowdstrike_client_secret | cs-secret-1 | + | crowdstrike_client_id | client-id-1 | + | collector_period | 60 | + | splunk_token | (unset) | + | nuclei_api_key | abc | + """ + return sdk.Configuration( + config_hints={ + "openaev_token": {"data": "platform-token"}, + "shodan_api_key": {"data": "shodan-key-1"}, + "crowdstrike_client_secret": {"data": "cs-secret-1"}, + "crowdstrike_client_id": {"data": "client-id-1"}, + "collector_period": {"data": 60, "is_number": True}, + "splunk_token": {"data": None}, + "nuclei_api_key": {"data": "abc"}, + }, + config_file_path=str(tmp_path / "missing.yml"), + **kwargs, + ) + + +def _given_settings_model() -> _Settings: + """Given a settings model holding a nested Splunk password "spl-pass-1" and + username "admin" """ + return _Settings(splunk=_Splunk(password="spl-pass-1", username="admin")) + + +def _when_listing_secret_values(configuration: sdk.Configuration) -> list[str]: + """When the configuration lists its secret values""" + return configuration.secret_values() + + +def _then_secrets_are(secrets: list[str], *expected: str) -> None: + """Then the secret values are exactly "" """ + assert set(secrets) == set(expected) + + +def _then_secrets_include(secrets: list[str], value: str) -> None: + """Then "" is a secret value""" + assert value in secrets + + +def _then_secrets_exclude(secrets: list[str], value: str) -> None: + """Then "" is not a secret value""" + assert value not in secrets + + +def _then_longest_secrets_come_first(secrets: list[str]) -> None: + """Then the secret values come longest first""" + lengths = [len(secret) for secret in secrets] + assert lengths == sorted(lengths, reverse=True) + + +def test_values_of_secret_keys_are_secret(tmp_path: Path) -> None: + """It lists the values of the keys whose name says they hold a secret. + + Scenario: Values of secret keys are secret + Given a connector configuration + When the configuration lists its secret values + Then the secret values are exactly "platform-token", "shodan-key-1" and "cs-secret-1" + """ + configuration = _given_configuration(tmp_path) + secrets = _when_listing_secret_values(configuration) + _then_secrets_are(secrets, "platform-token", "shodan-key-1", "cs-secret-1") + + +def test_secret_fields_of_the_settings_model_are_secret(tmp_path: Path) -> None: + """It lists every SecretStr of the settings model, however deep, and nothing else. + + Scenario: Secret fields of the settings model are secret + Given a settings model holding a nested Splunk password and username + And a connector configuration built on that settings model + When the configuration lists its secret values + Then "spl-pass-1" is a secret value + And "admin" is not a secret value + """ + settings = _given_settings_model() + configuration = _given_configuration(tmp_path, config_base_model=settings) + secrets = _when_listing_secret_values(configuration) + _then_secrets_include(secrets, "spl-pass-1") + _then_secrets_exclude(secrets, "admin") + + +def test_longest_secrets_come_first(tmp_path: Path) -> None: + """It lists the longest secrets first, so one containing another is redacted whole. + + Scenario: Longest secrets come first + Given a connector configuration + When the configuration lists its secret values + Then the secret values come longest first + """ + configuration = _given_configuration(tmp_path) + secrets = _when_listing_secret_values(configuration) + _then_longest_secrets_come_first(secrets) + + +# ---- Constraints identified ---- + + +def test_values_too_short_to_be_secret_are_left_out(tmp_path: Path) -> None: + """It leaves out values shorter than 4 characters, which would redact plain words. + + Scenario: Values too short to be secret are left out + Given a connector configuration + When the configuration lists its secret values + Then "abc" is not a secret value + """ + configuration = _given_configuration(tmp_path) + secrets = _when_listing_secret_values(configuration) + _then_secrets_exclude(secrets, "abc") diff --git a/libs/xtm-oaev-sdk/test/contracts/fixtures/health_check_parity.json b/libs/xtm-oaev-sdk/test/contracts/fixtures/health_check_parity.json new file mode 100644 index 0000000..77c530d --- /dev/null +++ b/libs/xtm-oaev-sdk/test/contracts/fixtures/health_check_parity.json @@ -0,0 +1,460 @@ +{ + "description": "Inject content verdicts of the platform's own health-check tests, one case per test: the contract fields as the platform serializes them, the content, the targets given apart from the content, and whether the platform finds the inject ready.", + "platform_source": "openaev-api/src/test/java/io/openaev/healthcheck/HealthCheckUtilsTest.java (openaev-platform, main)", + "cases": [ + { + "source": "HealthCheckUtilsTest.given_an_injector_contract_with_asset_mandatory_and_an_asset_should_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": true + } + ], + "content": {}, + "provided": { + "assets": [ + "assetId" + ], + "asset_groups": [] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_an_injector_contract_with_asset_mandatory_and_no_asset_should_not_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": true + } + ], + "content": {}, + "provided": { + "assets": [], + "asset_groups": [] + }, + "ready": false + }, + { + "source": "HealthCheckUtilsTest.given_an_injector_contract_with_asset_optional_and_an_asset_should_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false + } + ], + "content": {}, + "provided": { + "assets": [ + "assetId" + ], + "asset_groups": [] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_an_injector_contract_with_asset_optional_and_not_asset_should_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false + } + ], + "content": {}, + "provided": { + "assets": [], + "asset_groups": [] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_an_injector_contract_with_mandatory_groups_and_an_element_should_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryGroups": [ + "assets", + "asset_groups" + ] + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false, + "mandatoryGroups": [ + "assets", + "asset_groups" + ] + } + ], + "content": {}, + "provided": { + "assets": [ + "assetId" + ], + "asset_groups": [] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_an_injector_contract_with_mandatory_groups_and_full_elements_should_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryGroups": [ + "assets", + "asset_groups" + ] + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false, + "mandatoryGroups": [ + "assets", + "asset_groups" + ] + } + ], + "content": {}, + "provided": { + "assets": [ + "assetId" + ], + "asset_groups": [ + "assetGroupId" + ] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_an_injector_contract_with_mandatory_groups_and_no_element_should_not_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryGroups": [ + "assets", + "asset_groups" + ] + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false, + "mandatoryGroups": [ + "assets", + "asset_groups" + ] + } + ], + "content": {}, + "provided": { + "assets": [], + "asset_groups": [] + }, + "ready": false + }, + { + "source": "HealthCheckUtilsTest.given_an_injector_contract_with_mandatory_on_condition_and_no_element_should_not_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryConditionFields": [ + "asset_groups" + ] + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false + } + ], + "content": {}, + "provided": { + "assets": [], + "asset_groups": [] + }, + "ready": false + }, + { + "source": "HealthCheckUtilsTest.given_an_injector_contract_with_mandatory_on_condition_and_element_should_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryConditionFields": [ + "asset_groups" + ] + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false + } + ], + "content": {}, + "provided": { + "assets": [ + "assetId" + ], + "asset_groups": [] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_an_injector_contract_with_mandatory_on_condition_and_condition_element_should_not_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryConditionFields": [ + "asset_groups" + ] + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false + } + ], + "content": {}, + "provided": { + "assets": [], + "asset_groups": [ + "assetGroupId" + ] + }, + "ready": false + }, + { + "source": "HealthCheckUtilsTest.given_an_injector_contract_with_mandatory_on_condition_and_all_elements_should_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryConditionFields": [ + "asset_groups" + ] + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false + } + ], + "content": {}, + "provided": { + "assets": [ + "assetId" + ], + "asset_groups": [ + "assetGroupId" + ] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_mandatory_on_condition_with_specific_value_when_condition_matches_should_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryConditionFields": [ + "asset_groups" + ], + "mandatoryConditionValues": { + "asset_groups": "assetGroupId" + } + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false + } + ], + "content": {}, + "provided": { + "assets": [ + "assetId" + ], + "asset_groups": [ + "assetGroupId" + ] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_mandatory_on_condition_with_specific_values_when_condition_matches_should_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryConditionFields": [ + "asset_groups" + ], + "mandatoryConditionValues": { + "asset_groups": [ + "assetGroupId", + "assetGroupId2" + ] + } + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false + } + ], + "content": {}, + "provided": { + "assets": [ + "assetId" + ], + "asset_groups": [ + "assetGroupId2" + ] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_mandatory_on_condition_with_specific_value_when_condition_not_matches_should_not_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryConditionFields": [ + "asset_groups" + ], + "mandatoryConditionValues": { + "asset_groups": "assetGroupId" + } + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false + } + ], + "content": {}, + "provided": { + "assets": [], + "asset_groups": [ + "assetGroupId" + ] + }, + "ready": false + }, + { + "source": "HealthCheckUtilsTest.given_mandatory_on_condition_with_not_specific_value_when_condition_not_matches_should_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryConditionFields": [ + "asset_groups" + ], + "mandatoryConditionValues": { + "asset_groups": "assetGroupId" + } + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false + } + ], + "content": {}, + "provided": { + "assets": [], + "asset_groups": [ + "assetGroupId2" + ] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_mandatory_on_condition_with_not_specific_values_when_condition_not_matches_should_be_ready", + "fields": [ + { + "key": "assets", + "type": "asset", + "mandatory": false, + "mandatoryConditionFields": [ + "asset_groups" + ], + "mandatoryConditionValues": { + "asset_groups": [ + "assetGroupId", + "assetGroupId3" + ] + } + }, + { + "key": "asset_groups", + "type": "asset-group", + "mandatory": false + } + ], + "content": {}, + "provided": { + "assets": [], + "asset_groups": [ + "assetGroupId2" + ] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_text_field_with_default_value_and_no_content_should_be_ready", + "fields": [ + { + "key": "title", + "type": "text", + "mandatory": true, + "defaultValue": "Default title" + } + ], + "content": {}, + "provided": { + "assets": [], + "asset_groups": [] + }, + "ready": true + }, + { + "source": "HealthCheckUtilsTest.given_text_field_without_default_value_and_no_content_should_not_be_ready", + "fields": [ + { + "key": "title", + "type": "text", + "mandatory": true, + "defaultValue": "" + } + ], + "content": {}, + "provided": { + "assets": [], + "asset_groups": [] + }, + "ready": false + } + ] +} diff --git a/libs/xtm-oaev-sdk/test/contracts/test_contract_validator.py b/libs/xtm-oaev-sdk/test/contracts/test_contract_validator.py new file mode 100644 index 0000000..170ae35 --- /dev/null +++ b/libs/xtm-oaev-sdk/test/contracts/test_contract_validator.py @@ -0,0 +1,276 @@ +"""Tests for the local validation of inject content against its contract. + +Feature: Contract validation without the platform + Before an injector runs an inject without the platform, its content is checked + against the injector contract the way the platform checks it: mandatory fields, + mandatory groups and conditionally mandatory fields, plus the select choices and + single-value cardinality the contract declares. +""" + +from typing import Any, Callable + +import pytest +import xtm_oaev_sdk as sdk + + +def _given_contract_validator() -> Callable[..., list[Any]]: + """Given the SDK validates inject content against a contract""" + validator = getattr(sdk, "ContractValidator", None) + assert ( + validator is not None + ), "Expected xtm_oaev_sdk.ContractValidator to exist on public surface" + return validator().validate + + +def _given_scan_contract() -> list[Any]: + """Given a scan contract with: + | field | type | rule | + | targets | text | mandatory | + | assets | asset | in the mandatory group assets, hosts | + | hosts | text | in the mandatory group assets, hosts | + | scan_type | select | choices tcp, udp | + | ports | text | mandatory when scan_type is tcp | + | report_format | select | single value, choices json, xml | + """ + return [ + sdk.ContractText(key="targets", label="Targets", mandatory=True), + sdk.ContractAsset(label="Assets", mandatoryGroups=["assets", "hosts"]), + sdk.ContractText( + key="hosts", label="Hosts", mandatoryGroups=["assets", "hosts"] + ), + sdk.ContractSelect( + key="scan_type", + label="Scan type", + choices={"tcp": "TCP", "udp": "UDP"}, + ), + sdk.ContractText( + key="ports", + label="Ports", + mandatoryConditionFields=["scan_type"], + mandatoryConditionValues={"scan_type": ["tcp"]}, + ), + sdk.ContractSelect( + key="report_format", + label="Report format", + choices={"json": "JSON", "xml": "XML"}, + cardinality=sdk.ContractCardinality.One, + ), + ] + + +def _given_valid_content(**overrides: Any) -> dict[str, Any]: + """Given inject content that fills the scan contract""" + content: dict[str, Any] = { + "targets": "10.0.0.5", + "hosts": "web-01", + "scan_type": "tcp", + "ports": "22,443", + "report_format": "json", + } + content.update(overrides) + return content + + +def _given_content_without(content: dict[str, Any], *keys: str) -> dict[str, Any]: + """Given that content without "" """ + return {key: value for key, value in content.items() if key not in keys} + + +def _when_validating( + validate: Callable[..., list[Any]], + contract: list[Any], + content: dict[str, Any], + provided_keys: tuple[str, ...] = (), +) -> list[Any]: + """When the content is validated against the contract""" + return validate(contract, content, provided_keys) + + +def _then_content_is_valid(violations: list[Any]) -> None: + """Then the content has no violation""" + assert violations == [] + + +def _then_violations_are(violations: list[Any], *expected: tuple[str, str]) -> None: + """Then the violations are exactly " " """ + assert sorted((v.key, v.kind) for v in violations) == sorted(expected) + + +def _then_every_violation_names_its_field(violations: list[Any]) -> None: + """Then every violation message names its field""" + assert violations, "Expected at least one violation" + for violation in violations: + assert violation.key in violation.message + + +def test_content_filling_its_contract_is_valid() -> None: + """It accepts content that satisfies every rule of the contract. + + Scenario: Content filling its contract is valid + Given the SDK validates inject content against a contract + And a scan contract + And inject content that fills the scan contract + When the content is validated against the contract + Then the content has no violation + """ + validate = _given_contract_validator() + contract = _given_scan_contract() + content = _given_valid_content() + violations = _when_validating(validate, contract, content) + _then_content_is_valid(violations) + + +@pytest.mark.parametrize("value", [None, "", [], {}], ids=["absent", "''", "[]", "{}"]) +def test_missing_mandatory_field_is_reported(value: Any) -> None: + """It reports a mandatory field that is absent or empty, as the platform reads it. + + Scenario Outline: A missing mandatory field is reported + Given the SDK validates inject content against a contract + And a scan contract + And inject content that fills the scan contract, with targets + When the content is validated against the contract + Then the violations are exactly "targets mandatory" + + Examples: + | value | + | absent | + | '' | + | [] | + | {} | + """ + validate = _given_contract_validator() + contract = _given_scan_contract() + content = _given_valid_content(targets=value) + violations = _when_validating(validate, contract, content) + _then_violations_are(violations, ("targets", "mandatory")) + + +def test_empty_mandatory_group_reports_every_member() -> None: + """It reports every member of a mandatory group when none of them is set. + + Scenario: An empty mandatory group reports every member + Given the SDK validates inject content against a contract + And a scan contract + And inject content that fills the scan contract + And that content without "hosts" + When the content is validated against the contract + Then the violations are exactly "assets mandatory_group" and "hosts mandatory_group" + """ + validate = _given_contract_validator() + contract = _given_scan_contract() + content = _given_content_without(_given_valid_content(), "hosts") + violations = _when_validating(validate, contract, content) + _then_violations_are( + violations, ("assets", "mandatory_group"), ("hosts", "mandatory_group") + ) + + +def test_conditional_field_is_required_when_its_condition_is_met() -> None: + """It requires a conditional field when its condition field holds a listed value. + + Scenario: A conditional field is required when its condition is met + Given the SDK validates inject content against a contract + And a scan contract + And inject content that fills the scan contract, with scan_type "tcp" + And that content without "ports" + When the content is validated against the contract + Then the violations are exactly "ports mandatory_condition" + """ + validate = _given_contract_validator() + contract = _given_scan_contract() + content = _given_content_without(_given_valid_content(scan_type="tcp"), "ports") + violations = _when_validating(validate, contract, content) + _then_violations_are(violations, ("ports", "mandatory_condition")) + + +def test_conditional_field_is_optional_when_its_condition_is_not_met() -> None: + """It leaves a conditional field optional when its condition value is not selected. + + Scenario: A conditional field is optional when its condition is not met + Given the SDK validates inject content against a contract + And a scan contract + And inject content that fills the scan contract, with scan_type "udp" + And that content without "ports" + When the content is validated against the contract + Then the content has no violation + """ + validate = _given_contract_validator() + contract = _given_scan_contract() + content = _given_content_without(_given_valid_content(scan_type="udp"), "ports") + violations = _when_validating(validate, contract, content) + _then_content_is_valid(violations) + + +def test_select_value_outside_its_choices_is_reported() -> None: + """It reports a select value that is not one of the field's choices. + + Scenario: A select value outside its choices is reported + Given the SDK validates inject content against a contract + And a scan contract + And inject content that fills the scan contract, with scan_type "icmp" + When the content is validated against the contract + Then the violations are exactly "scan_type choice" + """ + validate = _given_contract_validator() + contract = _given_scan_contract() + content = _given_valid_content(scan_type="icmp") + violations = _when_validating(validate, contract, content) + _then_violations_are(violations, ("scan_type", "choice")) + + +def test_several_values_for_a_single_value_field_are_reported() -> None: + """It reports several values given to a field that takes a single one. + + Scenario: Several values for a single-value field are reported + Given the SDK validates inject content against a contract + And a scan contract + And inject content that fills the scan contract, with report_format ["json", "xml"] + When the content is validated against the contract + Then the violations are exactly "report_format cardinality" + """ + validate = _given_contract_validator() + contract = _given_scan_contract() + content = _given_valid_content(report_format=["json", "xml"]) + violations = _when_validating(validate, contract, content) + _then_violations_are(violations, ("report_format", "cardinality")) + + +# ---- Constraints identified ---- + + +def test_targets_given_outside_the_content_count_as_set() -> None: + """It counts a target field as set when the inject's targets are given apart. + + Scenario: Targets given outside the content count as set + Given the SDK validates inject content against a contract + And a scan contract + And inject content that fills the scan contract + And that content without "hosts" + When the content is validated against the contract, with "assets" given apart + Then the content has no violation + """ + validate = _given_contract_validator() + contract = _given_scan_contract() + content = _given_content_without(_given_valid_content(), "hosts") + violations = _when_validating(validate, contract, content, ("assets",)) + _then_content_is_valid(violations) + + +def test_every_violation_names_its_field() -> None: + """It gives each violation a message naming the field. + + Scenario: Every violation names its field + Given the SDK validates inject content against a contract + And a scan contract + And inject content that fills the scan contract, with scan_type "icmp" + And that content without "targets" and "hosts" + When the content is validated against the contract + Then every violation message names its field + """ + validate = _given_contract_validator() + contract = _given_scan_contract() + content = _given_content_without( + _given_valid_content(scan_type="icmp"), "targets", "hosts" + ) + violations = _when_validating(validate, contract, content) + _then_every_violation_names_its_field(violations) diff --git a/libs/xtm-oaev-sdk/test/contracts/test_contract_validator_parity.py b/libs/xtm-oaev-sdk/test/contracts/test_contract_validator_parity.py new file mode 100644 index 0000000..cbca32c --- /dev/null +++ b/libs/xtm-oaev-sdk/test/contracts/test_contract_validator_parity.py @@ -0,0 +1,76 @@ +"""Parity of the contract validator with the platform's health-check. + +Feature: Contract validation verdicts match the platform's + Each case of ``fixtures/health_check_parity.json`` is one test of the platform's own + health-check (``HealthCheckUtilsTest``): the contract fields as the platform serializes + them, the inject content, the targets the inject carries apart from its content, and + whether the platform finds the inject ready. The SDK reaches the same verdict. +""" + +import json +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Callable + +import pytest +import xtm_oaev_sdk as sdk + +PARITY = json.loads( + (Path(__file__).parent / "fixtures" / "health_check_parity.json").read_text( + encoding="utf-8" + ) +)["cases"] + + +def _given_contract_validator() -> Callable[..., list[Any]]: + """Given the SDK validates inject content against a contract""" + return sdk.ContractValidator().validate + + +def _given_platform_fields(case: dict[str, Any]) -> list[Any]: + """Given the contract fields as the platform serializes them""" + return [ + SimpleNamespace( + key=field["key"], + type=field["type"], + mandatory=field.get("mandatory", True), + mandatoryGroups=field.get("mandatoryGroups"), + mandatoryConditionFields=field.get("mandatoryConditionFields"), + mandatoryConditionValues=field.get("mandatoryConditionValues"), + defaultValue=field.get("defaultValue"), + cardinality=field.get("cardinality"), + choices=field.get("choices"), + ) + for field in case["fields"] + ] + + +def _when_validating( + validate: Callable[..., list[Any]], fields: list[Any], case: dict[str, Any] +) -> list[Any]: + """When the content is validated, with the targets the inject carries given apart""" + provided = {key: values for key, values in case["provided"].items() if values} + return validate(fields, case["content"], provided) + + +def _then_ready_is(violations: list[Any], ready: bool) -> None: + """Then the inject is ready exactly when the platform finds it ready""" + assert (violations == []) is ready, f"Expected ready={ready}, got {violations}" + + +@pytest.mark.parametrize( + "case", PARITY, ids=[case["source"].split(".")[-1] for case in PARITY] +) +def test_verdict_matches_the_platform_health_check(case: dict[str, Any]) -> None: + """It finds an inject ready exactly when the platform's health-check does. + + Scenario Outline: The verdict matches the platform health-check + Given the SDK validates inject content against a contract + And the contract fields as the platform serializes them + When the content is validated, with the targets the inject carries given apart + Then the inject is ready exactly when the platform finds it ready + """ + validate = _given_contract_validator() + fields = _given_platform_fields(case) + violations = _when_validating(validate, fields, case) + _then_ready_is(violations, case["ready"]) diff --git a/libs/xtm-oaev-sdk/test/daemons/test_dev_help.py b/libs/xtm-oaev-sdk/test/daemons/test_dev_help.py new file mode 100644 index 0000000..7eb8ea5 --- /dev/null +++ b/libs/xtm-oaev-sdk/test/daemons/test_dev_help.py @@ -0,0 +1,222 @@ +"""Tests for the help of the dev options and the parser the role SDKs extend. + +Feature: Help on the dev options + "--dev --help" (or "--dev -h") asks for the dev options and what each one is for. + Without --dev, -h stays the daemon's own. The SDK only knows the options common to + every daemon: an injector or a collector SDK adds its own options to the dev parser, + and its help then lists the common options next to the role's. +""" + +import argparse +from typing import Any, Callable + +import pytest +import xtm_oaev_sdk as sdk + +COMMON_OPTIONS = ("--dev", "-h", "--out", "--record", "--replay") +ROLE_OPTIONS = ( + "--inject", + "--emit", + "--dump-contracts", + "--list-contracts", + "--new-inject", + "--expectations", + "--once", +) + + +def _given_dev_options_reader() -> Callable[..., Any]: + """Given the SDK exposes the daemon dev options""" + return sdk.dev_options + + +def _given_dev_help() -> Callable[[], str]: + """Given the SDK exposes the help of the dev options""" + helper = getattr(sdk, "dev_help", None) + assert helper is not None, "Expected xtm_oaev_sdk.dev_help on public surface" + return helper + + +def _given_dev_argument_parser() -> Callable[..., argparse.ArgumentParser]: + """Given the SDK exposes the dev parser a role extends""" + builder = getattr(sdk, "dev_argument_parser", None) + assert ( + builder is not None + ), "Expected xtm_oaev_sdk.dev_argument_parser on public surface" + return builder + + +def _given_command_line(*args: str) -> list[str]: + """Given a daemon command line with "" """ + return list(args) + + +def _given_role_parser( + builder: Callable[..., argparse.ArgumentParser], +) -> argparse.ArgumentParser: + """Given a role adds an "injector options" group with --inject to the dev parser""" + parser = builder(description="Injector dev mode.", epilog="Injector exit codes.") + group = parser.add_argument_group("injector options") + group.add_argument("--inject", metavar="PATH", help="the inject message to run") + return parser + + +def _when_reading_dev_options(reader: Callable[..., Any], argv: list[str]) -> Any: + """When the dev options are read""" + return reader(argv) + + +def _when_reading_the_help(helper: Callable[[], str]) -> str: + """When the help of the dev options is read""" + return helper() + + +def _when_parsing(parser: argparse.ArgumentParser, argv: list[str]) -> Any: + """When the role parses the command line""" + args, _ = parser.parse_known_args(argv) + return args + + +def _then_help_is_asked(options: Any, asked: bool) -> None: + """Then the dev help is """ + assert options.help is asked + + +def _then_help_groups(text: str, heading: str, *options: str) -> None: + """Then the help section "" lists exactly """ + section = text.split(f"{heading}:", 1)[1].split("\n\n", 1)[0] + listed = tuple( + token.rstrip(",") + for line in section.splitlines() + for token in line.split() + if token.startswith("-") and line.lstrip().startswith(token) + ) + assert listed == options, f"Expected {options} under {heading}, got {listed}" + + +def _then_help_names_no_role_option(text: str) -> None: + """Then the help names no injector nor collector option""" + named = [option for option in ROLE_OPTIONS if option in text] + assert named == [], f"Expected no role option in the common help, got {named}" + + +def _then_help_says(text: str, *phrases: str) -> None: + """Then the help says "" """ + for phrase in phrases: + assert phrase in text, f"Expected {phrase!r} in {text!r}" + + +def _then_role_option_is(args: Any, value: Any) -> None: + """Then the role's --inject is """ + assert args.inject == value + + +@pytest.mark.parametrize("flag", ["--help", "-h"]) +def test_dev_help_is_asked_with_dev(flag: str) -> None: + """It asks for the dev help when --dev comes with -h or --help. + + Scenario Outline: The dev help is asked with --dev + Given the SDK exposes the daemon dev options + And a daemon command line with "--dev " + When the dev options are read + Then the dev help is True + + Examples: + | flag | + | --help | + | -h | + """ + reader = _given_dev_options_reader() + argv = _given_command_line("--dev", flag) + options = _when_reading_dev_options(reader, argv) + _then_help_is_asked(options, True) + + +def test_dev_help_lists_only_the_common_options() -> None: + """It lists the options common to every daemon, and no role option. + + Scenario: The dev help lists only the common options + Given the SDK exposes the help of the dev options + When the help of the dev options is read + Then the help section "common options" lists exactly --dev, -h, --out, --record and --replay + And the help names no injector nor collector option + """ + helper = _given_dev_help() + text = _when_reading_the_help(helper) + _then_help_groups(text, "common options", *COMMON_OPTIONS) + _then_help_names_no_role_option(text) + + +def test_a_role_extends_the_dev_parser_with_its_own_options() -> None: + """It lets a role add its options and lists them next to the common ones. + + Scenario: A role extends the dev parser with its own options + Given the SDK exposes the dev parser a role extends + And a role adds an "injector options" group with --inject to the dev parser + When the help of the dev options is read + Then the help section "common options" lists exactly --dev, -h, --out, --record and --replay + And the help section "injector options" lists exactly --inject + And the help says "Injector dev mode." and "Injector exit codes." + """ + builder = _given_dev_argument_parser() + parser = _given_role_parser(builder) + text = _when_reading_the_help(parser.format_help) + _then_help_groups(text, "common options", *COMMON_OPTIONS) + _then_help_groups(text, "injector options", "--inject") + _then_help_says(text, "Injector dev mode.", "Injector exit codes.") + + +@pytest.mark.parametrize( + ("argv", "value"), + [ + (["--dev", "--inject", "inject.json"], "inject.json"), + (["--dev", "--inj", "inject.json"], None), + (["--dev", "--nmap-ports", "22"], None), + ], + ids=["full name", "abbreviation", "daemon's own flag"], +) +def test_a_role_parser_reads_its_options_only_under_their_full_name( + argv: list[str], value: Any +) -> None: + """It reads a role option only under its full name and leaves foreign flags alone. + + Scenario Outline: A role parser reads its options only under their full name + Given the SDK exposes the dev parser a role extends + And a role adds an "injector options" group with --inject to the dev parser + When the role parses the command line "" + Then the role's --inject is + + Examples: + | argv | value | + | --dev --inject inject.json | inject.json | + | --dev --inj inject.json | None | + | --dev --nmap-ports 22 | None | + """ + builder = _given_dev_argument_parser() + parser = _given_role_parser(builder) + args = _when_parsing(parser, argv) + _then_role_option_is(args, value) + + +# ---- Constraints identified ---- + + +@pytest.mark.parametrize("flag", ["--help", "-h"]) +def test_help_without_dev_stays_the_daemon_s_own(flag: str) -> None: + """It leaves -h and --help to the daemon when --dev is absent. + + Scenario Outline: Help without --dev stays the daemon's own + Given the SDK exposes the daemon dev options + And a daemon command line with "" + When the dev options are read + Then the dev help is False + + Examples: + | flag | + | --help | + | -h | + """ + reader = _given_dev_options_reader() + argv = _given_command_line(flag) + options = _when_reading_dev_options(reader, argv) + _then_help_is_asked(options, False) diff --git a/libs/xtm-oaev-sdk/test/daemons/test_dev_options.py b/libs/xtm-oaev-sdk/test/daemons/test_dev_options.py new file mode 100644 index 0000000..7322947 --- /dev/null +++ b/libs/xtm-oaev-sdk/test/daemons/test_dev_options.py @@ -0,0 +1,436 @@ +"""Tests for the dev command-line options common to collector and injector daemons. + +Feature: Daemon dev options + Collector and injector daemons can run without the platform ("dev mode"). + Every daemon reads its dev options from its command line, dev or not, + so the dev options must never act on anything that is not theirs. +""" + +import io +import sys +from contextlib import redirect_stdout +from typing import Any, Callable, NamedTuple +from unittest.mock import MagicMock + +import pytest +import xtm_oaev_sdk as sdk + +PROGRAM = "openaev-collector" +SCHEMA = '{"title": "collector configuration"}' + + +class Reading(NamedTuple): + options: Any + exit_code: Any + output: str + + +class Start(NamedTuple): + exit_code: Any + output: str + + +def _given_dev_options_reader() -> Callable[..., Any]: + """Given the SDK exposes the daemon dev options""" + reader = getattr(sdk, "dev_options", None) + assert ( + reader is not None + ), "Expected xtm_oaev_sdk.dev_options to exist on public surface" + return reader + + +def _given_command_line(*args: str) -> list[str]: + """Given a daemon command line with "" """ + return list(args) + + +def _given_command_line_with_every_common_dev_flag() -> list[str]: + """Given a daemon command line with: + | flag | value | + | --dev | | + | --out | run.json | + | --record | shodan.tape.json | + """ + return _given_command_line( + "--dev", "--out", "run.json", "--record", "shodan.tape.json" + ) + + +def _given_process_command_line(monkeypatch: pytest.MonkeyPatch, *args: str) -> None: + """Given the process command line contains "" """ + monkeypatch.setattr(sys, "argv", [PROGRAM, *args]) + + +def _given_configuration() -> sdk.Configuration: + """Given a daemon configuration""" + return sdk.Configuration( + config_hints={ + "log_level": {"data": "info"}, + "name": {"data": "whoami collector"}, + "collector_id": {"data": "whoami-collector"}, + } + ) + + +def _given_configuration_schema( + monkeypatch: pytest.MonkeyPatch, configuration: sdk.Configuration +) -> None: + """Given the daemon configuration has a schema""" + monkeypatch.setattr(configuration, "schema", lambda: SCHEMA) + + +def _given_daemon_with_callback( + configuration: sdk.Configuration, +) -> tuple[sdk.BaseDaemon, MagicMock, MagicMock]: + """Given a daemon with a callback""" + setup, main_loop = MagicMock(), MagicMock() + + class WhoamiCollector(sdk.BaseDaemon): + def _setup(self): + setup() + + def _start_loop(self): + main_loop() + + daemon = WhoamiCollector( + configuration=configuration, + callback=MagicMock(), + logger=MagicMock(), + api_client=MagicMock(), + ) + return daemon, setup, main_loop + + +def _when_reading_dev_options( + reader: Callable[..., Any], argv: list[str] | None +) -> Reading: + """When the dev options are read""" + output = io.StringIO() + try: + with redirect_stdout(output): + options = reader(argv) if argv is not None else reader() + except SystemExit as exit_: + return Reading(None, exit_.code, output.getvalue()) + return Reading(options, None, output.getvalue()) + + +def _when_turning_dev_mode_off(options: Any) -> Exception | None: + """When someone tries to turn dev mode off on those options""" + try: + options.enabled = False + except Exception as err: # pylint: disable=broad-except + return err + return None + + +def _when_starting_daemon(daemon: sdk.BaseDaemon) -> Start: + """When the daemon is started""" + output = io.StringIO() + try: + with redirect_stdout(output): + daemon.start() + except SystemExit as exit_: + return Start(exit_.code, output.getvalue()) + return Start(None, output.getvalue()) + + +def _then_reading_did_not_exit(reading: Reading) -> None: + """Then reading the dev options does not exit""" + assert ( + reading.exit_code is None + ), f"Expected reading the dev options not to exit, exited with {reading.exit_code}" + + +def _then_nothing_was_printed(reading: Reading) -> None: + """Then nothing is printed""" + assert reading.output == "", f"Expected no output, got {reading.output!r}" + + +def _then_dev_mode_is_enabled(options: Any) -> None: + """Then dev mode is enabled""" + assert options.enabled is True + + +def _then_dev_mode_is_disabled(options: Any) -> None: + """Then dev mode is disabled""" + assert options.enabled is False + + +def _then_output_destination_is(options: Any, destination: str | None) -> None: + """Then the output destination is "" """ + assert options.out == destination + + +def _then_tapes_are(options: Any, record: str | None, replay: str | None) -> None: + """Then the tape to record is "" and the tape to replay is "" """ + assert (options.record, options.replay) == (record, replay) + + +def _then_options_hold_only_the_common_fields(options: Any) -> None: + """Then the options hold only the fields common to every daemon""" + assert set(vars(options)) == {"enabled", "out", "record", "replay", "help"} + + +def _then_change_was_refused(error: Exception | None) -> None: + """Then the change is refused""" + assert isinstance( + error, (AttributeError, TypeError, ValueError) + ), f"Expected the change to be refused, got {error!r}" + + +def _then_option_is_unset(options: Any, option: str) -> None: + """Then the