feat(03): Server-side safety filters - #45
Open
tpaulshippy wants to merge 12 commits into
Open
Conversation
- SafetyPolicy value object + global floor denylists in services/safety.py - Layered system prompt (preamble + bot prompt + policy suffix) so flags are enforced even with custom advanced-editor prompts - Pre-model input filter and post-model output filter with fixed refusal copy - SafetyEvent audit model (stage/reason/redacted snippet) + admin - Tool filters: web_search pre-query/post-result, flashcard front/back - Optional Bedrock Guardrails check behind BEDROCK_GUARDRAIL_ID, fail-closed Refs docs/roadmap/03-server-side-safety.md
seed_e2e_server_safety creates 'e2e-test-user'/'testpassword123', a profile, and two bots: Safety Demo Bot (custom prompt + flags ON) and Open Flags Bot (flags OFF, global floor still applies). Refs docs/roadmap/03-server-side-safety.md
- policy defaults/flag mapping, layered system prompt incl. bypass attempts - input block without model call, output replacement, global floor with flags off, SafetyEvent stage/reason/redaction assertions - web search: unbound when disabled, blocked query, stripped results - flashcard tools reject unsafe cards without persisting - denylist word-boundary cases; Bedrock guardrail flag + fail-closed Refs docs/roadmap/03-server-side-safety.md
- MarkdownRenderer: confirm dialog showing the domain before Linking.openURL - botSimple/botAdvanced: 'Syft always applies baseline safety' editor copy - botTemplates: document that the client prompt is preview-only now - jest coverage for link confirm/cancel + domain extraction Refs docs/roadmap/03-server-side-safety.md
Baseline 'npm run typecheck' failed before this feature: bare 'global' references in apiClient.test.ts, null-vs-string template_name in Bot fixtures/interface, non-null assertions missing in profiles.test.ts. Refs docs/roadmap/03-server-side-safety.md
- __mocks__/handlers.ts: typed msw json bodies and params - aiModels.test.ts: non-null assertions like profiles.test.ts Refs docs/roadmap/03-server-side-safety.md
Drives the seeded 'Safety Demo Bot': an adult-topic message gets the fixed server refusal (no model call), then a normal homework question still gets a real assistant reply. Header documents seeding + env requirements. Refs docs/roadmap/03-server-side-safety.md
Screenshot now shows the bot editor with safety toggles (Restrict Foul Language, Restrict Adult Topics, Enable Web Search).
There was a problem hiding this comment.
🟡 Changes recommended
There are a few correctness/reliability issues in the new safety/event logging and API base URL handling that should be addressed before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Implements roadmap 03 server-side safety (Issue #55) by moving safety enforcement from client-authored prompt suffixes to a server-owned policy with layered prompts, pre/post model filtering, tool filtering, and audit logging; includes corresponding frontend UX hardening (PIN reauth gating + outbound link confirmation) and expanded test coverage.
Changes:
- Backend: add
SafetyPolicy+ layered system prompts, input/output/tool safety filters, andSafetyEventaudit model wired into chat + web search + flashcard tools. - Frontend: replace local PIN checks with server reauthentication + in-memory parent session, add baseline safety editor notes, and add link confirmation before opening URLs.
- Tests/tooling: add backend safety test suite, frontend unit tests for link confirmation, and a Detox e2e walkthrough with seeded demo data.
File summaries
| File | Description |
|---|---|
| front/e2e/03-server-safety.e2e.js | Detox e2e validating server-side refusals and a normal safe completion path |
| front/components/PinWrapper.tsx | Parent gating via server reauth + in-memory parent session token and keypad UI |
| front/components/NavigationDrawer.tsx | Adds testID for drawer items to support e2e/test automation |
| front/components/MarkdownRenderer.tsx | Adds outbound-link confirmation dialog and URL domain extraction helper |
| front/components/HeaderButtons.tsx | Adds testID for drawer menu button |
| front/components/tests/MarkdownRenderer-test.tsx | Unit tests ensuring links confirm before opening |
| front/app/parent/settings.tsx | Blocks parent settings behind PIN presence + reauth gate; adds testIDs |
| front/app/parent/setPin.tsx | New set/change PIN flow with validation + server call behavior and errors |
| front/app/parent/botSimple.tsx | Adds baseline safety note copy in simple bot editor |
| front/app/parent/botAdvanced.tsx | Adds baseline safety note copy in advanced bot editor |
| front/app/login.tsx | Removes plaintext PIN caching; refreshes hasPin flag after login |
| front/api/pinStorage.ts | Removes legacy plaintext PIN storage; adds hasPin cache + in-memory parent session |
| front/api/botTemplates.ts | Marks client prompt generation as preview-only under server-owned safety |
| front/api/bots.ts | Updates template_name type to allow null |
| front/api/apiClient.ts | Adds X-Parent-Reauth header on unsafe methods when parent session is present |
| front/api/account.ts | Changes account shape to hasPin; adds setPin() API wrapper returning raw response |
| front/tests/api/profiles.test.ts | Tightens typings/assertions to handle nullable response |
| front/tests/api/apiClient.test.ts | Uses globalThis for fetch/XMLHttpRequest mocks |
| front/tests/api/aiModels.test.ts | Tightens typings/assertions to handle nullable response |
| front/mocks/handlers.ts | Adds explicit typing + safer string coercions in MSW handlers |
| back/bots/views/get_chat_response.py | Ensures stored system message uses server-layered prompt |
| back/bots/tests/test_safety.py | Comprehensive backend tests for policy layering, filters, tools, and guardrail behavior |
| back/bots/services/safety.py | New server-owned safety policy, denylists, layered prompts, refusals, logging, Bedrock guardrail integration |
| back/bots/services/chat_agent.py | Adds tool-level safety filters (web search + flashcards) and policy wiring |
| back/bots/models/safety_event.py | New SafetyEvent audit model |
| back/bots/models/chat.py | Enforces pre/post safety filters in get_response(); server-owned system prompt layering |
| back/bots/models/init.py | Exports SafetyEvent from models package |
| back/bots/migrations/0039_safetyevent.py | Migration creating SafetyEvent table |
| back/bots/management/commands/seed_e2e_server_safety.py | Seed command for idempotent e2e safety demo data |
| back/bots/management/commands/init.py | Package init for management commands |
| back/bots/management/init.py | Package init for management module |
| back/bots/admin.py | Registers SafetyEvent in Django admin |
Review details
- Files reviewed: 30/35 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+11
to
+15
| /** | ||
| * Extract the hostname to show in the outbound-link confirm sheet, e.g. | ||
| * "https://docs.example.com/page" -> "example.com". Falls back to the raw | ||
| * URL when it cannot be parsed. | ||
| */ |
Comment on lines
+53
to
+56
| async (entered: string) => { | ||
| setSubmitting(true); | ||
| setMessage(null); | ||
| const attempt = async () => { |
Comment on lines
+108
to
+113
| # Post-model output filter: replace flagged completions before save. | ||
| output_verdict = evaluate_text(response_text, policy, source='OUTPUT') | ||
| if output_verdict.blocked: | ||
| response_text = refusal_for_verdict(output_verdict) | ||
| record_safety_event(stage='output', verdict=output_verdict, chat=self, snippet=response_text) | ||
|
|
Comment on lines
91
to
92
| const url = `${BASE_URL}${endpoint}`; | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap 03: Server-Side Safety
Implements #55.
What changed
SafetyPolicywith denylists, crisis detection, layered system prompt, input/output/tool filters, optional Bedrock Guardrails (fail-closed),SafetyEventmodelEvidence
https://github.com/tpaulshippy/bots/raw/feature/roadmap-03-server-safety/evidence/pr45-safety.mp4