DOC: Executor docs rewrite#1979
Open
rlundeen2 wants to merge 9 commits into
Open
Conversation
- Flatten doc/code/executor into one folder with numbered prefixes - New concise intro distinguishing executor (algorithm) from attack technique - New Single-Turn, Multi-Turn, Attack Configuration, Workflow, Benchmark, and Prompt Generator landing pages - New Scenarios 'Attack Techniques' concept page - Remove violent_durian doc notebook (migrating to a technique separately) - Rewrite internal/external links for the flat layout and update myst.yml TOC Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restructure doc/code/executor/ from one-notebook-per-attack into six grouped notebooks matching the scoring/converters doc style: - 1_single_turn: prompt_sending, role_play, context_compliance, many_shot, skeleton_key, flip (authored fresh, executed) - 2_multi_turn: red_teaming, crescendo, tap, multi_prompt_sending, chunked_request, sequential (executed); barge_in described inline - 3_attack_configuration (renamed from 16) - 4_workflow: xpia_website + xpia_ai_recruiter (merged, outputs preserved) - 5_benchmark: qa + bias (merged, outputs preserved) - 6_promptgen: anecdoctor + fuzzer (merged, outputs preserved) Rewrite 0_executor.md intro with the executor-vs-attack-technique distinction and add a scenarios/0_attack_techniques concept page. Redirect all stale links (blogs, targets, scoring) to the grouped notebooks with section anchors, update myst.yml TOC, and remove the old per-attack notebooks and landing pages. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Freshly ran 1_single_turn, 2_multi_turn, 3_attack_configuration, 5_benchmark, and 6_promptgen end-to-end via jupytext --execute and refreshed their outputs. 4_workflow is left as-is since it needs external infrastructure (Azure Blob Storage / recruiter Docker). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ebook tests Remove the knowledge-graph visualization cell from 6_promptgen, which was the only consumer of networkx (not a PyRIT dependency). The AnecdoctorGenerator output is unchanged; only the optional matplotlib/ networkx rendering is dropped. Update the executor integration test for the flattened, grouped doc folder: scan executor/ directly (it no longer has subfolders) and skip only 4_workflow.ipynb, which needs external infrastructure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Lead 0_executor.md with 'executor = algorithm for interacting with an objective target' (not always an attack), define objective vs adversarial target explicitly, and define single- vs multi-turn by attack turns. Move Sequential out of multi-turn into a new 7_compound notebook; keep Barge-In as a streaming subsection. Refine the attack-technique definition, sweep objective/adversarial terminology through 3-6, and add 7_compound to the TOC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cs-rewrite # Conflicts: # doc/code/executor/attack/skeleton_key_attack.ipynb # doc/code/executor/attack/skeleton_key_attack.py # doc/myst.yml
Clarify in 0_executor.md and 1_single_turn that an attack class earns its place through adaptive, feedback-driven decision-making (branching/backtracking, like graph traversal — Crescendo, TAP), reshapable by swapping primitives. Pure prompt transformations belong as converters; fixed framings as techniques over PromptSendingAttack. Note that several single-turn attacks predate this distinction and remain as classes for compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Enumerate in PyRIT terms what an attack technique can include (0_attack_techniques) - Move objective-vs-adversarial-target detail into Attack Configuration; trim intro - Delete flawed 'fine-grained control' paragraph and the shared-lifecycle section - Replace single/multi split with a linked 'Executor categories' TOC - Generalize and move 'When do you actually need a new executor class?' to page end - Remove redundant initialize_pyrit_async notes across executor pages - Renumber: compound -> 4, workflow -> 5, benchmark -> 6, promptgen -> 7 (update myst.yml TOC, category list, integration test skip, blog links) - Add 'configuring a red teaming attack to generate an image' example Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
+27
to
+29
| PyRIT ships several families of executor. The cleanest way to tell the two main *attack* families | ||
| apart is to **count requests to the objective target**: a single-turn attack sends exactly one; a | ||
| multi-turn attack sends more than one and adapts as it goes. |
Contributor
There was a problem hiding this comment.
I think this is confusing; it reads like there's several types of executors and they are these two families of attacks. I'd add just say :
Suggested change
| PyRIT ships several families of executor. The cleanest way to tell the two main *attack* families | |
| apart is to **count requests to the objective target**: a single-turn attack sends exactly one; a | |
| multi-turn attack sends more than one and adapts as it goes. | |
| PyRIT ships several families of executor. |
and then list the executors
Comment on lines
+39
to
+41
| - **[Attack Configuration](3_attack_configuration.ipynb)** — not an executor itself, but the | ||
| cross-cutting inputs every attack accepts (objective vs. adversarial target, prepended | ||
| conversations, multimodal seeds, next-turn messages, memory labels). |
Contributor
There was a problem hiding this comment.
nit: remove since this isn't an executor (maybe the notebook order should be changed so it's not in the middle of all the executors too)
Comment on lines
+16
to
+17
| - An **executor** (for attacks, an **attack strategy**) is the *algorithm* — e.g. | ||
| `PromptSendingAttack`, `CrescendoAttack`, `TreeOfAttacksWithPruningAttack`. It knows *how* to drive |
Contributor
There was a problem hiding this comment.
i think it's confusing that we are defining executor but only listing attacks. imo attacks vs executor should be the comparison here and we only define attack technique in the scenario md because why introduce a concept here that doesn't pertain
| # ## Translating from Seeds for Attack Parameters | ||
| # | ||
| # Most [attacks](../executor/attack/0_attack.md) make use of several parameters. | ||
| # Most [attacks](../executor/0_executor.md) make use of several parameters. |
Contributor
There was a problem hiding this comment.
should this be ? relatedly should all instances of attack in this notebook be executor ?
Suggested change
| # Most [attacks](../executor/0_executor.md) make use of several parameters. | |
| # Most [executors](../executor/0_executor.md) make use of several parameters. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidated executor docs to be more digestible and structured