feat: policy-grounded comment triage (agent-driven, dry-run first)

[jdwit]

Goal

Let an agent triage the comment backlog against the channel's own written policy, with a preview the human confirms before anything changes.

Two jobs, one loop: get the junk out, and surface the comments that deserve a human (questions, leads, real criticism). A moderation tool that only deletes spam solves half the problem.

The intelligence lives in the agent, not in the CLI. Same architecture as transcript-grounded authoring (#43, PRs #52/#53): no LLM dependency and no API key in ytstudio. The CLI supplies the context (the policy file, the comments as JSON, cheap deterministic signals) and a safe write path; the harness LLM does the judgement.

What already works

comments list --status held -o json plus comments reject <ids> / comments publish <ids> covers the happy path of the moderation loop, and comments reply (#30) covers the response side.

It does not survive contact with a real backlog, though. Three things have to be fixed before the agent loop is safe to run, and they are listed first below for that reason.

The constraint that shapes this feature

commentThreads.list accepts moderationStatus values heldForReview, likelySpam and published only. rejected is not a listable state. After comments reject runs, those comments cannot be enumerated through the API again, and rejecting a comment also hides all of its replies.

There is therefore no way to add undo later: the ids and text have to be written down before the call, or they are gone. A dry-run preview does not cover this on its own. The human approves a list of sixty, one of them was wrong, and without a local record there is nothing to go back to.

This is why the audit log below is in v1 and not in the deferred list.

Prerequisites (do these first)

0a. Verify the preview can resolve ids back to comments

comments reject <ids> receives ids only. To print "author, text, action" the CLI has to resolve them, and no such call exists today; fetch_comments() lists by channel or video only.

commentThreads.list(id=...) is the candidate, with two documented constraints: it cannot be combined with moderationStatus, maxResults or pageToken. Whether it returns comments that are currently heldForReview is undocumented.

One live call settles it, and it is load-bearing for the whole preview premise. If held comments do not come back by id, the fallback is for the agent to pass the text through, which is weaker: the CLI would then be echoing agent-supplied text back to the human instead of the truth from the API. Decide before building the rest.

0b. Paginate fetch_comments()

comments.py:59 caps at min(limit, 100) and never reads nextPageToken, so --limit 500 silently returns 100. "Backlog" triage and cross-video duplicate_of cannot work on one truncated page, which makes this a prerequisite for the signals in scope item 4, not independent work.

0c. Stop silently ignoring --status when --video is set

comments.py:63-69: moderationStatus is only applied in the else branch, so comments list --video X --status held returns published comments and reports nothing. The API reference forbids combining moderationStatus with the id filter only, not with videoId, so this is a self-imposed limitation. Wire it up, or fail loudly the way the --sort relevance check at line 105 already does. Per-video triage is the obvious first use of this feature.

Scope (v1)

1. A moderation policy the agent can load

Straight copy of the existing brand-file plumbing (profiles/<name>/brand.md, load_brand/save_brand in config.py):

• ytstudio profile policy show | edit | set --file <path> -> profiles/<name>/moderation.md
• Free-form markdown, printed verbatim by show: what to reject, what to leave held, what never to touch, which topics always go to the human.

Without this the agent moderates on vibes; with it, every decision is traceable to a line the channel owner wrote. moderation.md governs what happens to a comment; brand.md governs how a reply sounds. They stay separate files.

2. A dry-run gate plus a write-ahead log on comments reject / comments publish

Today both execute immediately. videos update, videos search-replace, upload and the playlist writes all preview by default and only mutate on --execute. Moderation should match:

• Default: print the comments that would be affected (author, text, action) and change nothing.
• --execute: apply.
• --ban keeps needing --execute too; banning stays a human decision. (banAuthor is only valid together with rejected per the API, which the current code already respects.)

Before each API call, append id, author, text, video id, timestamp and the action to a local JSONL in the profile directory. Recovery is then comments publish <ids> read back from that log. Whether re-publishing an already-rejected comment actually works is undocumented and needs the same kind of live spike as 0a; without the log the question cannot even be asked.

Also fix the reporting while in here: _set_moderation_status() (line 140) loops over batches of 50, and on an HttpError handle_api_error re-raises anything that is not a 403, so the loop aborts mid-way with earlier batches already applied. success += len(batch) assumes the whole batch landed. Tolerable for three ids typed by hand, not for an agent-approved batch of 200 where nobody can tell afterwards which comments were rejected. Report applied and unapplied ids per batch.

3. A triage vocabulary the agent reports back in

The agent should not answer "reject or not". It should label, and the label drives the action. v1 taxonomy, documented in the skill so output is consistent across runs:

Label	Default action
junk (link farms, fake giveaways, phishing, crypto/WhatsApp/Telegram bait)	propose reject
impersonation (author name mimics the channel, "message me on Telegram")	propose reject, flag to human
toxic (harassment, slurs, threats)	propose reject
question	leave held, surface to human, candidate for a reply
lead (buying intent, pricing/availability questions, collaboration requests)	leave held, surface to human, never auto-anything
criticism (negative but genuine)	publish, never reject
praise / neutral	publish

The label is deliberately junk and not spam: --status spam already means "YouTube classified this as likelySpam", and the taxonomy value means "the agent judged this". Two judges, two words.

Anything the agent cannot place with confidence stays held and goes in the "needs you" list. The taxonomy is a reporting contract, not a config format; no schema in the CLI in v1.

4. Cheap deterministic signals in comments list

Pulled forward from the deferred list, because bulk triage on raw text alone burns context and misses the obvious cases. Computed in the CLI, no model needed, and dependent on 0b:

• has_link, and whether the link host is off-channel (read from textOriginal)
• duplicate_of: identical or near-identical text posted across multiple videos or repeated by one author
• author_channel_id, already present in the snippet, so repeat offenders are visible across the backlog
• impersonation_suspect: display name closely matches the channel title and author_channel_id is not the owner's

That last one is stated as a pair on purpose. Impersonators normally copy the display name exactly, so a plain edit-distance-to-channel-name check scores them as "not similar" and finds only near-misses. Note also that authorDisplayName now carries the @handle in current API responses, which skews a raw distance calculation.

These go in the -o json output as fields. The agent reads them as evidence; the human reads them in the preview and can sanity-check a bulk reject at a glance.

5. Reply triage as a separate pass

The backlog of unanswered comments is worth more than the spam queue, but it is a different query from the held-review triage: "unanswered" is only meaningful for published comments.

• comments list --unanswered (no reply from the channel owner in the thread), sortable by age.

Cost to be aware of: this needs part=replies plus totalReplyCount, and the replies part returns only a limited set of replies per thread, so threads above that need a follow-up comments.list per thread. Cheap in quota, not free in calls.

The reply itself stays the existing comments reply command, grounded in brand.md and, where relevant, the video transcript. Same dry-run rule: draft, show the human, post on --execute.

6. Document the loop in the skill

New section in skills/ytstudio/SKILL.md:

1. ytstudio profile policy show
2. ytstudio comments list --status held -o json
3. Agent labels each comment against the policy using the v1 taxonomy.
4. ytstudio comments reject <ids> -> preview, show it to the user, grouped by label with the reason per comment.
5. Re-run with --execute after confirmation.
6. Report what was left held and why: questions, leads, and anything uncertain.

Hard rules for the agent, stated in the skill: never ban; when in doubt leave it held; never reject genuine criticism or negative feedback; never reject a comment solely because it is negative about the channel.

Quota, stated in the skill and in docs/api-quota.md: comments.setModerationStatus costs 50 units per call against a 10,000/day default, while commentThreads.list costs 1. Batching up to 50 ids per call (already implemented) is what keeps this viable; a per-id reject loop exhausts the day after 200 comments. An agent needs to be told this explicitly or it will helpfully reject one id at a time.

Release note

Flipping reject and publish to preview-by-default is a behaviour change for anyone scripting 0.4.2 from PyPI. Minor version bump, and call it out in the release notes and the skill.

Explicitly not in v1

• Decision sidecar (comments apply decisions.yaml) with per-comment reasons.
• Transcript-grounded replies to comments.
• Any persisted per-channel classification stats or trend reporting.

Non-goals (design boundaries, not "later")

These are the things this feature deliberately will not become:

• No autonomous mode. No flag, now or later, that posts replies or hides comments without a human seeing the preview. The dry-run gate is the product, not a training-wheels phase.
• No keyword rule engine in the CLI. If/then keyword and sentiment rules belong in moderation.md as prose the agent reads, not as a config DSL ytstudio has to parse, version, and debug.
• No model, no key, no vendor. The CLI ships no LLM SDK. Whichever harness the user runs is the moderator.
• Not a dashboard. No cross-platform inbox, no other networks. This is YouTube, in a terminal, driven by an agent.

Relation to PR #7

PR #7 (interactive moderation TUI, open since February) solves the human-review side of the same problem. It is not needed for v1 - the dry-run preview is the review step. Decide separately whether to close it or keep it as a later interactive reviewer.