Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions .agents/skills/sensitive-logging-audit/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
---
name: sensitive-logging-audit
description: Audit and fix sensitive-data exposure through Python runtime logging in openai-agents-python. Use when reviewing logging, print, warnings, stderr, traceback, MCP names, model or tool exceptions, redaction flags, or any diagnostic path that may retain user data.
---

# Sensitive Logging Audit

## Objective

Find candidate output sinks, trace their values manually, fix demonstrated leaks at shared runtime boundaries, and prove redaction with adversarial tests.

The collector is only a syntax-based search aid. It does not resolve Python aliases or control flow, certify policy guards, or prove that an absent candidate is safe.

## Workflow

### 1. Establish the review surface

- Work in the current checkout and preserve unrelated changes.
- Read `src/agents/_debug.py`, `src/agents/logger.py`, and the affected callers.
- Treat exception messages, arguments, tracebacks, causes, contexts, notes, names, URLs, and arbitrary values as potentially sensitive.
- Read [the Python redaction validation matrix](references/redaction-validation.md).

Run the collector tests, then collect candidates:

```bash
uv run python .agents/skills/sensitive-logging-audit/scripts/test_inventory.py
uv run python .agents/skills/sensitive-logging-audit/scripts/inventory_logging.py \
--format json --output /tmp/sensitive-logging-candidates.json
```

The report intentionally contains no `policy`, `safe`, or guard classification.

### 2. Supplement the collector with source search

The collector does not follow assignments such as `emit = logger.error`. Search the source directly and inspect aliases, callbacks, wrappers, and reflective dispatch:

```bash
rg -n '\.(debug|info|warning|warn|error|exception|critical|fatal|log)\b' src/agents
rg -n '\b(print|pprint|pp|warn|warn_explicit|write|writelines|print_exc|print_exception)\b' src/agents
rg -n 'DONT_LOG_(MODEL|TOOL)_DATA|log_(model|tool|model_and_tool)_action' src/agents
```

Do not turn collector coverage or a textual guard into a security conclusion. Trace producers and callers.

### 3. Classify manually

Assign each reviewed path one disposition:

- `model`: model requests, responses, Realtime events, or derived values.
- `tool`: tool arguments, outputs, MCP data, tool events, or derived values.
- `model+tool`: either class may reach the sink.
- `operational`: demonstrated to contain only non-sensitive SDK metadata.
- `intentional-output`: explicitly user-facing output rather than diagnostics.
- `uncertain`: source tracing is incomplete.

Record evidence in the audit report. The script does not validate or inherit dispositions.

### 4. Fix runtime boundaries

Before changing runtime behavior, use `$implementation-strategy`.

- Check the relevant `_debug.DONT_LOG_MODEL_DATA` and `_debug.DONT_LOG_TOOL_DATA` flags before formatting or inspecting sensitive values.
- Redact mixed model/tool values when either flag disables data logging.
- In redacted mode, emit a fixed message and omit sensitive `args`, `extra`, and `exc_info`.
- Build diagnostic-only context lazily so redacted mode never reads it.
- Preserve useful diagnostics when sensitive-data logging is explicitly enabled.
- Keep logging failure from changing fallback, cleanup, event, rejection, or cancellation behavior.
- For MCP URLs, remove credentials, query parameters, and fragments in diagnostic mode; never use sanitized names as a substitute for fixed redacted messages.

### 5. Prove caller behavior

Add tests at every changed caller boundary. Inspect the complete `LogRecord`, not only rendered text. Test both redacted policies, diagnostic mode, hostile objects, exception chains, and the caller's observable fallback or cleanup behavior as applicable.

### 6. Re-run and close out

Re-run the collector, the manual searches, focused tests, and applicable repository gates. Use `$code-change-verification` for runtime or test changes and `$pr-draft-summary` when required.

Report candidate counts as search coverage only. Lead with confirmed leaks fixed, retained intentional output, reviewed uncertainty, and verification results. Never report a clean collector result as proof that no sensitive logging path exists.
4 changes: 4 additions & 0 deletions .agents/skills/sensitive-logging-audit/agents/openai.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
interface:
display_name: "Sensitive Logging Audit"
short_description: "Audit and fix sensitive Python logging paths"
default_prompt: "Use $sensitive-logging-audit to inventory, verify, and fix sensitive logging leaks in this repository."
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Python sensitive logging validation

The collector reports syntactic logging and raw-output candidates. It does not resolve aliases, prove receiver types, evaluate guards, classify payloads, or support a completeness claim. Review candidates together with direct source searches and runtime tests.

## Required validation matrix

Test every changed sensitive caller boundary in both redacted and diagnostic modes. Use a unique sentinel for each source and inspect both rendered output and the complete `LogRecord`.

| Case | Model flag | Tool flag | Value | Required assertion |
| --- | --- | --- | --- | --- |
| Model redaction | on | off | `Exception(secret)` | No sentinel or exception object remains in the record |
| Tool redaction | off | on | `Exception(secret)` | No sentinel or exception object remains in the record |
| Both redacted | on | on | model and tool values | Neither sentinel remains anywhere in the record |
| Diagnostic mode | off | off | ordinary exception | Existing diagnostic detail and traceback behavior remain |
| Hostile string | applicable | applicable | object whose `__str__` raises or returns a secret | Logging does not fail or reveal the secret |
| Hostile repr | applicable | applicable | object whose `__repr__` raises or returns a secret | Logging does not fail or reveal the secret |
| Hostile class access | applicable | applicable | exception overriding `__getattribute__` | Redacted logging does not inspect the exception |
| Exception chain | applicable | applicable | `__cause__`, `__context__`, notes, or `ExceptionGroup` containing secrets | No chained secret is attached or rendered |
| Supplemental arguments | applicable | applicable | fixed message plus secret formatting argument | Formatting arguments are omitted in redacted mode |
| Extra payload | applicable | applicable | `extra={"detail": secret}` | Secret `LogRecord` attributes are omitted |
| Traceback payload | applicable | applicable | `exc_info=True` or an exception tuple | `exc_info` and `exc_text` are absent in redacted mode |
| MCP server or tool name | tool | on | path token or custom-name sentinel | Log uses a fixed message and does not read or attach the name |
| URL-derived MCP name | tool | off | URL credentials, query, and fragment | Log retains only scheme, host, port, and path; the runtime value is unchanged |

Also test the observable caller behavior after logging. Redaction is incorrect if it prevents a fallback result, cleanup, event emission, rejection, or cancellation from completing.

## Inspect the full LogRecord

Do not assert only against `caplog.text` or a mock call converted to a string. In redacted mode, inspect at least:

- `record.msg`
- `record.args`
- `record.exc_info`
- `record.exc_text`
- values added through `record.__dict__`
- the final output of a real `logging.Formatter`

The sensitive object itself must not remain attached even when its string representation is absent. A custom handler or exporter may inspect raw record fields.

## Review procedure

1. Run the collector against all of `src/agents`.
2. Run the supplemental `rg` searches from `SKILL.md` and inspect aliases and dynamic dispatch.
3. Review raw output and ambiguous receivers first.
4. Review caught values, `logger.exception`, `exc_info`, `extra`, and formatting arguments.
5. Trace model, tool, Realtime, MCP, session, sandbox, voice, tracing, and cleanup values to their producers.
6. Classify intentional output separately from diagnostics; do not silently exempt `print` or warnings.
7. Add focused tests at every changed caller boundary.
8. Re-run the collector and source searches after the fix.

An empty or unchanged collector report is not proof of safety. Assignment aliases, monkey-patched methods, dynamically installed handlers, non-constant reflection, and arbitrary runtime data flow require manual inspection.

## Audit report expectations

For each confirmed or uncertain path, record:

- The source location and value producer.
- The manual disposition: `model`, `tool`, `model+tool`, `operational`, `intentional-output`, or `uncertain`.
- Concrete evidence for the disposition.
- The fix or reason for retaining the path.
- The caller-level regression test, when behavior changed.

Do not reuse a disposition solely because a fingerprint or call text is unchanged.
Loading