UN-3332 [FIX] Use default profile for export validation when single-pass is enabled#1991
Open
pk-zipstack wants to merge 3 commits into
Open
UN-3332 [FIX] Use default profile for export validation when single-pass is enabled#1991pk-zipstack wants to merge 3 commits into
pk-zipstack wants to merge 3 commits into
Conversation
The Prompt Studio export validation filtered prompt outputs by prompt.profile_manager, but single-pass execution stores outputs against the tool's default profile (see OutputManagerHelper.handle_prompt_output_update) with is_single_pass_extract=True. When the default profile differed from the prompt-level profile FK, the validation lookup missed the rows produced by the single-pass run and incorrectly rejected the export with "Prompt Studio project without prompts cannot be exported." Match the validation lookup to the mode the prompts were actually run in: the tool default profile + is_single_pass_extract=True when single-pass is enabled, the prompt's profile_manager + False otherwise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Regression test for frame_export_json: - single-pass on: filter uses default profile + is_single_pass_extract=True - single-pass off: filter uses prompt.profile_manager + is_single_pass_extract=False - force_export=True bypasses the PromptStudioOutputManager lookup Follows the sys.modules-stub pattern from test_build_index_payload.py to avoid pulling in Django app loading; skips cleanly if imports drift. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
Summary by CodeRabbit
WalkthroughExport validation in frame_export_json now selects an output_profile per prompt based on tool.single_pass_extraction_mode (single-pass => default LLM profile; otherwise the prompt's profile_manager), passes that profile and is_single_pass_extract to PromptStudioOutputManager filter when not force_export, and sources profile-dependent payload fields from the selected output_profile. Regression tests cover single-pass, non-single-pass, and force-export paths. ChangesProfile-aware output validation for single-pass extraction
sequenceDiagram
participant Client
participant PromptStudioRegistryHelper
participant ProfileManager
participant PromptStudioOutputManager
Client->>PromptStudioRegistryHelper: frame_export_json(...)
alt tool.single_pass_extraction_mode == True
PromptStudioRegistryHelper->>ProfileManager: get_default_llm_profile()
else
Note right of PromptStudioRegistryHelper: use prompt.profile_manager
end
PromptStudioRegistryHelper->>PromptStudioOutputManager: objects.filter(tool_id, prompt_id, output_profile, is_single_pass_extract)
PromptStudioOutputManager-->>PromptStudioRegistryHelper: filter result
PromptStudioRegistryHelper-->>Client: exported JSON payloads (populated from output_profile)
🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
| Filename | Overview |
|---|---|
| backend/prompt_studio/prompt_studio_registry_v2/prompt_studio_registry_helper.py | Introduces output_profile to route both the DB validation filter and all per-prompt export fields through the default profile in single-pass mode; logic is correct and consistent with how OutputManagerHelper stores outputs. |
| backend/prompt_studio/prompt_studio_registry_v2/tests/test_frame_export_json.py | New regression tests with thorough mocking; covers single-pass, non-single-pass, and force_export scenarios. Key string constants verified to match JsonSchemaKey values. |
| backend/prompt_studio/prompt_studio_registry_v2/tests/init.py | Empty __init__.py to make the tests directory a Python package; no issues. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[frame_export_json called] --> B{tool.single_pass_extraction_mode?}
B -- Yes --> C[output_profile = default_llm_profile]
B -- No --> D[output_profile = prompt.profile_manager]
C --> E{force_export?}
D --> E
E -- No --> F["PromptStudioOutputManager.filter(\n profile_manager=output_profile,\n is_single_pass_extract=mode\n)"]
F --> G{rows found?}
G -- No --> H[add to invalidated_outputs, skip]
G -- Yes --> I[Build per-prompt export JSON\nfrom output_profile]
E -- Yes --> I
I --> J[append to outputs list]
J --> K{more prompts?}
K -- Yes --> B
K -- No --> L{outputs empty?}
L -- Yes --> M[raise EmptyToolExportError]
L -- No --> N{invalidated_outputs non-empty\n& not force_export?}
N -- Yes --> O[raise InValidCustomToolError]
N -- No --> P[return export_metadata]
Reviews (2): Last reviewed commit: "UN-3332 [FIX] Use default profile for pe..." | Re-trigger Greptile
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@backend/prompt_studio/prompt_studio_registry_v2/tests/test_frame_export_json.py`:
- Around line 142-153: The current module-level try/except sets _IMPORT_ERROR
and skips the whole test suite on any Exception; change the except clause to
only catch ImportError and ModuleNotFoundError (or ImportError subclass) and set
_IMPORT_ERROR in that case, but re-raise any other Exception so unexpected
errors in prompt_studio_registry_helper surface; adjust the handling of
PromptStudioRegistryHelper and _psrh_mod the same way and keep pytestmark using
_IMPORT_ERROR as the skip reason.
- Around line 67-134: The test mutates sys.modules via the helper functions
_install and _install_package (which assign directly into sys.modules), causing
cross-test pollution; update the tests to contain these overrides by saving and
restoring originals or using pytest's monkeypatch: replace direct assignments in
_install/_install_package with monkeypatch.setitem(sys.modules, name, mod) (or
wrap the calls in a fixture that records original sys.modules entries and
restores them in teardown), and ensure any inserted module keys are
removed/restored after the test so other tests do not see the stubbed modules.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4b5fc4f0-0fc1-4771-9bc4-d281a840e861
📒 Files selected for processing (3)
backend/prompt_studio/prompt_studio_registry_v2/prompt_studio_registry_helper.pybackend/prompt_studio/prompt_studio_registry_v2/tests/__init__.pybackend/prompt_studio/prompt_studio_registry_v2/tests/test_frame_export_json.py
…e-pass mode Addresses two review-bot findings on top of the initial validation fix: 1. Greptile: per-prompt export entries (vector_db / embedding / llm / x2text / chunk_size / chunk_overlap / retrieval_strategy / similarity_top_k / section / reindex) still read from prompt.profile_manager even when single-pass mode was on. Same root cause as the validation bug — if the user added a second profile and set it as default before running, the exported tool embedded the stale prompt-level profile's settings instead of the default profile that was actually executed. Lifts the output_profile resolution out of the validation block so it drives both the lookup and the per-prompt payload, eliminating the divergence. 2. CodeRabbit: the test module's import-time `except Exception` masked genuine helper bugs (NameError, TypeError at module load, etc.) behind a "skipped suite" — defeating regression coverage. Narrow to ImportError / ModuleNotFoundError / AttributeError so only true environment/dependency failures skip; everything else surfaces. Tests extended to assert per-prompt JSON content (llm, vector_db, embedding, etc.) matches the expected profile for each mode. Verified the new assertions correctly fail on pre-fix per-prompt logic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
@greptile-apps — addressed in 4ff2b3c. The per-prompt JSON divergence is the same root cause as UN-3332, so I folded it into this PR: lifted Tests extended in the same commit — Re. CodeRabbit's |
Contributor
Test ResultsSummary
Runner Tests - Full Report
SDK1 Tests - Full Report
|
Contributor
|
Looks good — the fix is clean and complete. The A few observations on the final state:
The PR addresses the original validation bug, the follow-up per-prompt JSON divergence, and the test coverage gap. LGTM. Tip: You can customize Greptile's behavior for this repo with |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes UN-3332: Prompt Studio incorrectly rejects tool export with "Prompt Studio project without prompts cannot be exported. Please ensure there is at least one active prompt that has been run before exporting." when single-pass extraction is enabled and the project's default profile differs from the profile attached to the prompt card at creation time.
Root cause
PromptStudioRegistryHelper.frame_export_jsonfilteredPromptStudioOutputManagerrows byprompt.profile_manager(the prompt-card-level FK, frozen at prompt-creation time).Single-pass execution, however, stores its outputs against the tool's default profile with
is_single_pass_extract=True(seeOutputManagerHelper.handle_prompt_output_update). When the user added a second profile and set it as default before running, the validation lookup missed the rows produced by the single-pass run and falsely concluded the prompts hadn't been run.The customer workaround — turn off single-pass and switch the radio button so the prompt-level profile matches the default — confirms the filter mismatch.
Fix
Match the validation lookup to the mode the prompts were actually run in:
tool.single_pass_extraction_modeprofile_manager=is_single_pass_extract=TrueTrueFalseprompt.profile_managerFalseOne-file change in
backend/prompt_studio/prompt_studio_registry_v2/prompt_studio_registry_helper.py.Test plan
backend/prompt_studio/prompt_studio_registry_v2/tests/test_frame_export_json.py— 3 cases covering both modes andforce_export=True. Samesys.modules-stubbing pattern astest_build_index_payload.pyso nopytest-djangodependency.HEAD~1(one assertion fail on wrong profile, oneKeyErroron missing flag) and pass on the fix commit.pytest prompt_studio/ -v.🤖 Generated with Claude Code