F060: Built-in jq Transform Operation

[pocky]

F060: Built-in jq Transform Operation

User Stories

US1: Apply jq Expression to JSON String Input (P1 - Must Have)

As a workflow author,
I want to apply a jq expression to a JSON string within an operation step,
So that I can filter, extract, and reshape JSON data between workflow steps without shelling out to external tools.

Acceptance Scenarios:

• Given a step with operation: transform.jq and inputs.expression: ".name" and inputs.data: '{"name":"alice","age":30}', when the step executes, then the output contains result: "alice"
• Given a step with operation: transform.jq and inputs.expression: ".items[] | select(.status == \"active\")" and valid JSON array input, when the step executes, then the output contains only the matching objects
• Given a step whose inputs.data references a previous step output via {{states.fetch_api.output}}, when the step executes, then the jq expression is applied to the resolved JSON string

Independent Test: Create a workflow with a single transform.jq operation step using a hardcoded JSON string and verify {{states.step_name.result}} contains the expected filtered value.

US2: Chain Multiple Transformations Across Steps (P1 - Must Have)

As a workflow author,
I want to pipe the jq output of one step as input to another transform step,
So that I can build multi-stage data transformation pipelines within a workflow.

Acceptance Scenarios:

• Given step A outputs result: '[{"id":1},{"id":2}]' and step B uses inputs.data: "{{states.step_a.result}}" with inputs.expression: ".[0].id", when step B executes, then result: "1"
• Given a three-step pipeline (fetch → filter → reshape), when all steps complete, then the final output reflects the composed transformation

Independent Test: Create a two-step workflow where step 1 extracts an array from a JSON object and step 2 selects an element from that array. Verify final output.

US3: Handle Invalid Input and Expression Errors (P2 - Should Have)

As a workflow author,
I want clear error messages when my jq expression is invalid or the input is not valid JSON,
So that I can diagnose and fix data transformation issues quickly.

Acceptance Scenarios:

• Given inputs.data contains non-JSON text (e.g., "not json"), when the step executes, then the operation returns success: false with an error message containing "invalid JSON"
• Given inputs.expression contains a syntax error (e.g., ".foo ||| bar"), when the step executes, then the operation returns success: false with an error message containing "invalid jq expression"
• Given inputs.data is an empty string, when the step executes, then the operation returns success: false with an error describing the empty input

Independent Test: Create a workflow with intentionally invalid JSON input and verify the error output includes a descriptive message. Repeat with an invalid expression.

US4: Produce Compact or Pretty-Printed Output (P3 - Nice to Have)

As a workflow author,
I want to control whether the jq output is compact or pretty-printed,
So that I can optimize for readability in logs or compactness for downstream consumption.

Acceptance Scenarios:

• Given inputs.compact: true (default), when the step executes, then result contains single-line JSON without extra whitespace
• Given inputs.compact: false, when the step executes, then result contains indented, human-readable JSON

Independent Test: Execute the same jq expression with compact: true and compact: false and compare the output formatting.

---

Requirements

Functional Requirements

• FR-001: The system shall provide a transform.jq operation accessible via operation: transform.jq in workflow step definitions.
• FR-002: The operation shall accept a data input (string, required) containing valid JSON and an expression input (string, required) containing a valid jq expression.
• FR-003: The operation shall apply the jq expression to the parsed JSON input and return the result as a JSON string in the result output field.
• FR-004: The operation shall accept an optional compact input (boolean, default true) controlling output formatting.
• FR-005: The operation shall return success: false with a descriptive error string when the input is not valid JSON.
• FR-006: The operation shall return success: false with a descriptive error string when the expression is syntactically invalid.
• FR-007: The operation shall support standard jq features: field access, array indexing, pipe (|), select, map, keys, length, type, to_entries, from_entries, object construction, and array slicing.
• FR-008: The operation shall be registered via CompositeOperationProvider alongside existing github.* and notify.* providers.
• FR-009: The operation shall respect context.Context cancellation for long-running expressions.

Non-Functional Requirements

• NFR-001: Expression evaluation shall complete in < 50ms for inputs up to 1 MB.
• NFR-002: The implementation shall use a pure-Go jq library (no CGO, no external jq binary dependency) to maintain cross-platform portability.
• NFR-003: The operation shall not log or expose raw input data at INFO level to prevent accidental secret leakage; input data shall only appear at DEBUG level.
• NFR-004: The implementation shall follow the established OperationProvider pattern from F054/F056 with infrastructure-internal types (no new domain entities).

---

Success Criteria

• All P1 user stories implemented and tested
• All P2 user stories implemented and tested
• Unit test coverage >= 80%
• No lint errors
• go-arch-lint passes with new infra-transform component
• Documentation updated (plugins.md, workflow-syntax.md, CHANGELOG.md)
• Integration test validates end-to-end YAML workflow execution

---

Key Entities

Entity	Description	Attributes
TransformOperationProvider	Implements ports.OperationProvider for transform.* namespace	operations map, Execute dispatch
JqEvaluator	Applies a jq expression to parsed JSON data	expression string, input any, compact bool
TransformInput	Validated input for transform.jq	data (string), expression (string), compact (bool)

---

Metadata

• Status: backlog
• Version: v0.4.0
• Priority: medium
• Estimation: M

Dependencies

• Blocked by: F057
• Unblocks: none

Clarifications

Section populated during clarify step with resolved ambiguities.

Notes

• Pure-Go jq library candidates: itchyny/gojq (MIT, widely used, full jq compatibility). Evaluate at implementation time.
• Follow F054/F056 pattern: internal/infrastructure/transform/ package with provider.go, operations.go, jq.go, jq_test.go, doc.go.
• Register in CompositeOperationProvider in run.go alongside github and notify providers.
• Single operation initially (transform.jq); namespace transform.* reserves space for future operations (e.g., transform.jsonpath, transform.xpath).
• No external binary dependency — unlike F054's gh CLI fallback pattern, jq evaluation must be fully embedded.