IPO-011: one-button IPO Screener dispatchable from the scan button - #109
Merged
Conversation
Groundwork for dispatching the event-driven IPO pipeline through the normal
"Run screener" button instead of building a parallel dispatch path.
- ScreenerDefinition gains requires_candles (default True), read from the
SCREENER metadata dict. Every existing screener keeps its exact behaviour.
- app._execute_screener gates the three candle-only setup steps on that
flag: the Dhan credential check, the universe CSV load, and constructing
DailyDataLoader. An event-driven screener now dispatches with
universe_df=None and data_loader=None.
- The cache payload reports zeroed candle-cache stats instead of crashing
when there is no data loader (caught by the new test, not by review).
- run_scan / _score_results_safely annotations now tell the truth: the
service already tolerated universe_df=None (symbols_scanned becomes NULL).
RANK-002's ScoringContext contract is deliberately untouched - the None is
normalized to an empty frame at that one boundary, which is already the
scorer's documented path to a null final_score.
- The shared progress line is now unit-free ("**{label}** - {n} / {total}
complete.") so stage-based progress reads correctly for every screener.
Tests: event-driven dispatch runs with credentials and universe fakes that
raise if consulted; candle screeners still abort without credentials; the
metadata default is pinned.
Gates: full pytest (1783 passed, coverage 89.74%), ruff, mypy, compileall.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…011) Schema-only step so the collector change lands separately and every commit stays gate-green. - Widen ck_ipo_enrichment_signals_signal_type to accept 'subscription_demand'. CHECK SQL is byte-identical between backend/storage/models.py and the migration so the ORM/Alembic parity test compares cleanly. - Widening is backward compatible: every row valid before stays valid. The downgrade narrows the vocabulary again, so it counts first and refuses rather than destroying rows or failing mid table-rebuild. - The IpoEnrichmentSignalType member and the collector that emits it land in the next commit; a vocabulary the database accepts but nothing yet produces is inert. Tests: new ipo011 test inserts the new value at head (the legacy CHECK would reject it) and asserts the downgrade refusal preserves the row; registered in the contract-policy documentation targets. Both hardcoded table inventories already cover ipo_enrichment_signals, so no table-set change was needed. Gates: full pytest (1784 passed, coverage 89.74%), ruff, mypy, compileall. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the "no subscription source at all" gap using the advisory web layer,
with the blast radius contained by design rather than by convention.
Collector (backend/ipo/sources/enrichment.py):
- New SUBSCRIPTION_DEMAND signal type + fixed query template.
- _parse_subscription_demand reuses the existing GMP clause/distance
discipline via a generalized _near_gmp_value(term_pattern=...): the
multiple must sit in the same clause and within the same character
distance as an explicit QIB/qualified-institutional term. Retail-only or
unanchored "subscribed 5 times" figures parse to None rather than being
misread as institutional demand.
- A parsed reading becomes an ipo_subscriptions row with
source_confidence=LOW, behind two guards:
* it never shadows better evidence - if any non-LOW snapshot exists for
the issue, nothing is written (scoring reads the NEWEST snapshot, so
appending would silently demote official data);
* it never breaks idempotency - an unchanged reading is skipped instead
of appending a row every run, which would churn the scoring
fingerprint and stop any issue ever reporting skipped_unchanged.
Containment (backend/ipo/scoring/caution_flags.py):
- weak_qib_demand_near_close forces Not Recommended outright, so it now
refuses to judge on a low-confidence snapshot and reports NOT_EVALUABLE.
A scraped headline can never reject an issue; the optional ten-point QIB
factor may still consume the same reading.
- CAUTION_FLAGS_VERSION -> ipo-006-flags-v3 so stored verdicts stay
attributable to the exact rule set that produced them. Expect one
re-score of every issue on first deploy.
Tests: parser table (anchored/unanchored/retail/no-number), low-confidence
snapshot round trip, official-evidence shadowing guard, unchanged-reading
idempotency plus a changed reading still recorded, and the containment pair
proving identical numbers differ only by provenance.
Gates: full pytest (1794 passed, coverage 89.79%), ruff, mypy, compileall.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The piece that makes an unattended run able to produce evidence at all, kept as narrow as the guarantee allows. - backend/config/settings.py: get_ipo_auto_approve_high_confidence() reading IPO_AUTO_APPROVE_HIGH_CONFIDENCE, DEFAULT OFF. With the switch unset the shipped behaviour is byte-identical to the reviewed flow - every proposal waits for an administrator. - backend/ipo/agents/auto_approval.py: the policy, and the only place the decision lives. It selects pending HIGH-confidence proposals - the tier where the host already re-resolved every cited value from the hash-verified PDF - and hands them to the existing approve_extraction_proposal. MEDIUM and weaker are skipped and counted. The repository stays identity-agnostic and keeps doing the same strict conversion regardless of caller. - Approvals are attributed to a reserved automation identity (ipo-automation@screener.local) so an autonomous approval is never mistaken for a human attestation in entered_by_email or the audit trail, and each one records EVENT_IPO_PROPOSAL_AUTO_APPROVED. - Failures are counted, never raised: a proposal that cannot convert stays pending for a human, which is the safe direction, and its siblings still process. Tests: default-off touches nothing; only HIGH is approved (MEDIUM/LOW skipped); the automation actor reaches both the approver and the audit payload; one failing proposal neither blocks siblings nor disappears; the env switch drives the default in both directions. Gates: full pytest (1799 passed, coverage 89.80%), ruff, mypy, compileall. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The headline: the whole pipeline now runs from the normal Run screener
button, listed in the dropdown as "IPO Screener".
screeners/ipo_screener.py follows the pattern the two AI screeners already
established - override run(), keep a trivial compute_signal for the ABC -
and re-implements no pipeline stage. It calls the same reviewed
run_ipo_screener() the CLI calls, so the button and the terminal cannot
drift apart, and inherits scan history, run status, provenance receipts,
and the audit trail for free instead of growing a second dispatch path.
- requires_candles: False, so the UI skips Dhan credentials, the universe
CSV, and the data loader (framework support landed earlier in this PR).
- default_params render as the sidebar's "Tune parameters" checkboxes and
map one-to-one onto pipeline stages: run_ingestion, download_documents,
collect_enrichment, draft_ai_extractions, only_active_issues, max_issues.
AI extraction is OFF by default because the button is analyst-accessible
and extraction spends Claude plan credit; max_issues bounds a run that
blocks the Streamlit tab.
- When auto-approval converts proposals during the run, a scoring-only
second pass reflects them immediately, so one press is genuinely one
press rather than two.
- Progress is reported per pipeline STAGE through the shared callback,
which the AI screeners leave frozen.
Result rows: one per IPO issue. symbol is a short synthetic "IPO:{id}" key
because scan_results.symbol is a NOT NULL String(50) with no length guard -
a long company name would fail on PostgreSQL - and the readable name gets
its own column. Unscored issues still emit a row explaining themselves
(awaiting_evidence) rather than vanishing, since a contract-invalid row is
silently dropped from persistence.
Tests: registry metadata, toggle-to-stage wiring, per-stage progress,
issue-selection narrowing, the auto-approval rescore pass, the 50-char
symbol guard with a 200-char company name, and - the important one - every
emitted row is pushed through the real normalize_screener_row the scan
service uses, for both scored and unscored issues.
Gates: full pytest (1809 passed, coverage 89.82%), ruff, mypy, compileall.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- The read-only nav view "IPO screener" becomes "IPO dashboard" so it no longer collides with the new "IPO Screener" dropdown entry. The dashboard stays the detail view (sections, filters, per-issue breakdown receipts); the screener produces the run and the summary table. Orchestration nav tuples updated. - New docs/architecture/ipo-011-one-button-screener.md: the requires_candles framework opt-out, the trust decisions and why each is drawn where it is, the result-row mapping, and - stated plainly - the deviation from the approved plan. - docs/operations.md gains a UI section covering the toggles, the analyst-accessible/paid-AI default, the auto-approval switch, and the price-band caveat. - docs/adding-a-screener.md documents requires_candles for future authors. - AGENTS.md and the architecture README index the new doc. Deviation recorded honestly: the planned price-band-as-cited-fact work is NOT in this PR. It touches _CITED_FACT_SCHEMA_VERSION, the expected-fact map, and approval-time re-resolution - the most safety-critical path in the repo - and deserves its own focused change rather than a rushed pass. Consequence: issues with no price band still resolve to "Insufficient verified data", so an autonomous run will not yet produce positive verdicts for them. The follow-up is fully specified in the new doc. Gates: full pytest (1809 passed, coverage 89.82%), ruff, mypy, compileall, bandit, pip-audit, pre-commit config. constraints/pyproject diff empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Selecting "IPO Screener" crashed the whole page before the user could press
anything:
KeyError: 'ipo_filings'
app.py:681 show_status_panel(selected)
-> universe_status -> universe_file_path -> UNIVERSE_CONFIG[key]
I gated the candle setup inside _execute_screener but missed that main()
renders the candle-data status card earlier, on every rerun, and that its
universe lookup indexes UNIVERSE_CONFIG directly rather than using .get().
show_status_panel and render_universe_table are both candle-data health
checks - Dhan credentials, the universe CSV's symbol count and mtime, and
the daily cache size. None of that applies to a screener that owns its own
data sources, so both are now skipped when requires_candles is False. That
fixes the crash and removes a misleading panel (an IPO run does not need
Dhan credentials, so reporting them as "Missing" was noise).
Audited the rest of the render path for the same shape: every other
selected.universe use is either .get()-based, already inside the
requires_candles branch, or a plain string passed to run_scan/audit. The
chart path early-returns because build_chart is None, so the fundamentals
panel no-ops.
Tests: new tests/test_app_ipo_screener_dispatch.py drives main() with an
event-driven screener and fails if either panel renders, plus the converse
case proving ordinary screeners still render both.
Gates: full pytest (1811 passed, coverage 89.81%), ruff, mypy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Diagnosed from a live run: every category failed with SebiSourceError after
~18s. SEBI's homepage returns 200, so connectivity is fine; the AJAX
endpoint returns HTTP 530 "Unauthorized Request Blocked" - a WAF refusal.
A browser User-Agent is blocked too, so this is not a header nit.
530 sits inside the 5xx range, so the client treated a permanent refusal as
a transient outage and ran the whole backoff ladder: 2+5+10s per category,
three categories, ~54s per run spent retrying a block that can never clear
while continuing to hammer an edge that already said no.
- New SebiBlockedError(SebiSourceError) raised immediately for statuses in
BLOCKED_STATUS_CODES ({530}). The distinct type is the diagnosis: the job
logs only an exception's class name (never its message, because upstream
HTML is untrusted), so an operator now reads error_type=SebiBlockedError
and knows re-running will not help, instead of an ambiguous
SebiSourceError that looks identical to a network outage.
- The retry-exhausted message now names the last status code, so a genuine
outage is also diagnosable. A status code is safe metadata, unlike the
response body.
Measured against the live endpoint: 18.0s -> 0.3s per category.
This does NOT make the fetch succeed. SEBI is deliberately refusing
automated requests to this endpoint; working around a WAF is a policy
decision for the repository owner, not something to slip into a bug fix.
Tests: a 530 fails immediately with no sleeps, one request, response closed,
and the status in the message; a 503 still retries the full ladder and
reports the status it gave up on.
Gates: full pytest (1813 passed, coverage 89.82%), ruff, mypy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two bugs, found by probing the live endpoint rather than reasoning from the
code. SEBI ingestion now actually works.
1) HTTP 530 was a missing Referer, not a bot ban.
SEBI's robots.txt allows this path outright (User-agent: * with an empty
Disallow; only /js and /css are excluded). Isolating the headers showed
the edge rejects the AJAX feed purely for lacking a Referer:
project UA only -> 530
browser UA only -> 530
project UA + Referer -> 200
So the fix is honest rather than a disguise: send the category's own
listing page as the Referer, which is literally where the feed request
originates. The User-Agent stays the identifying
"Streamlit-Scanner-App/IPO-002" - we are not impersonating a browser.
X-Requested-With is included because this genuinely is an XHR endpoint.
2) Pagination silently truncated every scan to 25 rows.
The response splits on "#@#": hidden pagination inputs sit BEFORE the
separator, breadcrumbs after. pagination_soup was built from
"metadata or html", so it parsed the breadcrumbs, found nothing, and fell
back to total_pages=1. The loop then exited after page one. Live SEBI
reports totalpage=16 where we read 1.
Also accept ``name=`` as well as ``id=`` on those inputs; SEBI uses name.
Measured against live SEBI (Jan-Aug 2026 window):
before drhp 25 rhp 25 final_offer 23 (one page each)
after drhp 125 rhp 75 final_offer 40 (spanning months)
This was invisible because the test fixture encoded the OLD response
shape - pagination after "#@#" using id= attributes - so the suite stayed
green while production read a single page. The new fixture mirrors the
real markup exactly.
Tests: the Referer is asserted per category (each cites its own listing
page) alongside the unchanged identifying User-Agent; a live-shape fixture
pins total_pages/next_value parsing; and a three-page walk proves the
fetcher follows pagination instead of stopping at page one.
Gates: full pytest (1816 passed, coverage 89.65%), ruff.
Note on mypy: this branch is clean, but a cold-cache run surfaces 5
PRE-EXISTING errors in backend/indicators.py and
backend/sixty_seven/shortlister.py - files this branch never touches. They
come from local environment drift (pandas 3.0.5 installed against a
pinned pandas==2.3.3 / pandas-stubs==2.3.3.260113), so CI, which installs
with -c constraints.txt, will not see them. Deliberately not "fixed" here:
casts written against pandas 3 semantics could be wrong under the pin.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
Author
|
@codex can you review this? |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8476a45c20
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| progress(step, total, _STAGES[min(step, total - 1)]) | ||
|
|
||
| report(0) | ||
| issue_ids = self._selected_issue_ids(params) |
There was a problem hiding this comment.
When ingestion is enabled and the pre-run snapshot is narrowed—normally because only_active_issues excludes at least one listed issue or the default 25-issue cap applies—this freezes issue_ids before run_ipo_screener performs its filing scan. The job filters its post-ingestion issue list against those old IDs, so every newly discovered filing skips download, enrichment, extraction, and scoring until the operator presses Run a second time, defeating the one-button flow. Perform ingestion before calculating the selected IDs, or otherwise include newly ingested active issues in the same pass.
Useful? React with 👍 / 👎.
Owner
Author
There was a problem hiding this comment.
Confirmed and fixed in 49ad9c7.
Ingestion now runs as its own pass before the selection is computed. The pipeline gained a skip_score flag so that first pass inventories filings without downloading, scraping, or scoring; _selected_issue_ids then reads the snapshot that exists after the SEBI scan, so anything the run discovers is carried into download, enrichment, extraction and scoring in the same press.
Regression guard: test_a_filing_discovered_by_this_run_is_processed_by_this_run stages a pre- and post-ingestion snapshot and asserts the id list reaching the pipeline is [7, 42] — selecting before the scan would have produced [7].
| # Convert freshly drafted, fully verified proposals into evidence when | ||
| # the operator enabled it, then re-score so the button's own run shows | ||
| # the result rather than making the user press it twice. | ||
| approval = auto_approve_ready_proposals() |
There was a problem hiding this comment.
When IPO_AUTO_APPROVE_HIGH_CONFIDENCE is enabled, this call omits issue_id, so it approves every pending HIGH-confidence proposal in the database, including listed issues and issues excluded by only_active_issues or max_issues. An analyst running a capped scan can therefore mutate evidence far outside the requested scope, and the follow-up scoring pass only rescans issue_ids, leaving those out-of-scope approvals with stale verdicts. Limit approval to the IDs selected for this run and rescore the same set.
Useful? React with 👍 / 👎.
Owner
Author
There was a problem hiding this comment.
Confirmed and fixed in 49ad9c7.
auto_approve_ready_proposals gained an issue_ids scope and the screener passes the same selection it gave the pipeline, so a capped or active-only run can no longer convert proposals for issues it never processed and will not rescore. When the run genuinely covers everything the scope is None, which means the same thing on both sides.
Worth noting the blast radius was larger than "stale verdicts": approval also writes the extracted cap price onto the issue row, so an unscoped pass mutated out-of-scope issues, not just their evidence.
Guards: test_auto_approval_is_scoped_to_the_issues_this_run_selected asserts the pass receives {"issue_ids": [7]} when a LISTED issue is filtered out, and test_an_unnarrowed_run_lets_auto_approval_see_the_whole_queue pins the None case.
Comment on lines
+395
to
+397
| value := _near_gmp_value( | ||
| text, _SUBSCRIPTION_MULTIPLE_PATTERN, _QIB_TERM_PATTERN | ||
| ) |
There was a problem hiding this comment.
For headlines containing multiple subscription categories in one clause, the proximity helper returns the first multiple near any QIB token rather than the multiple associated with QIB. For example, Overall subscribed 10x, QIB portion 2x is parsed as a 10x QIB book because punctuation such as commas does not split the clause and both values are within 40 characters of QIB. That incorrect value is persisted as the latest subscription snapshot and feeds the scored QIB factor, potentially changing the verdict; reject ambiguous clauses or explicitly pair the QIB label with its adjacent value.
Useful? React with 👍 / 👎.
Owner
Author
There was a problem hiding this comment.
Confirmed and fixed in 49ad9c7. Reproduced exactly as described before changing anything — Retail 25.6 times, QIB 1.2 times, NII 40 times parsed to 25.6, i.e. a weak 1.2x institutional book recorded as a strong one, which is the dangerous direction.
Two changes rather than one, because splitting on commas alone would still misread QIB and retail 5 times:
- Subscription parsing uses its own clause splitter that treats commas and pipes as boundaries (GMP prose keeps the original splitter — a comma is not a clause end there).
- Within a clause the QIB anchor must be strictly closer to the number than any competing category term (
retail|rii|nii|hni|non-institutional|employee|shareholder|overall|total|anchor). If a competitor is equally close or closer, the clause cannot say who owns the number and it yieldsNone.
This also retired the shared term_pattern parameter I had bolted onto _near_gmp_value; generalising the GMP proximity rule was what made the laxer binding look acceptable in the first place. QIB now has its own _qib_bound_multiple with the stricter rule, and the parametrised table covers the headline cases plus the ambiguous one.
Closes the last blocker to a real verdict. Valuation is a CRITICAL factor,
so an issue with no price band could only ever resolve to "Insufficient
verified data"; nothing in the system could write one.
The cap price now travels the same route as every other number: the model
proposes it with a page citation, the host re-resolves that citation against
the hash-verified PDF bytes, and only an approval writes it to the issue.
Scope decisions, both deliberate:
- Only the CAP price (upper bound) is extracted. It is the single bound
every ratio consumes ("upper price band / computed EPS"), and one field
per line keeps the existing "exactly one semantic field" verification rule
usable - a band line naming a floor and a cap would otherwise be ambiguous
and match neither.
- Issue open/close dates are NOT included. CitedFinancialFact carries a
Decimal, so dates need a cited-date receipt type; that is its own change.
Their absence is harmless: it leaves the near-close QIB caution
NOT_EVALUABLE, which is the safe direction and unchanged from today.
Safety work beyond the plan:
- A price-band line prints BOTH bounds, so plain token matching would bind
the floor as happily as the cap - and that error runs in the unsafe
direction, because a lower price makes the issue look cheaper and inflates
the valuation factor. The host therefore refuses any cap claim that is not
the largest number on the span it cites. Applied in both the table-row and
text-line branches; other fields are untouched.
- The field is optional and paired: a value without its page (or vice versa)
is rejected, and a DRHP - filed before pricing - omits both and stays
fully valid.
Schema: cited-financial-fact/v3 and ipo-010-extractor-v3. v2 proposals stay
approvable so review queues in flight are not invalidated; they simply carry
no issue terms. A v2 payload carrying issue terms is refused outright rather
than approved unbound.
Approval applies the verified cap price to the issue row inside the SAME
transaction as the manual revision and the proposal transition, so a lost
concurrent-review race rolls back all three.
Tests: a dedicated suite covers optional/paired validation, citation
emission, binding to a real band line, and the two refusals that matter -
claiming the floor as the cap, and a number on a line that never names a
price band. The review suite gains an end-to-end proof that an UNPRICED
issue ends up priced only after approval, plus the DRHP case that must stay
unpriced.
Gates: full pytest (1825 passed, coverage 89.65%), ruff.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured coverage has sat at ~89.7% for a while, so an 87% gate had roughly 2.7 points of slack - enough for a sizeable untested addition to land without the gate noticing. 89% keeps about 0.7 points of headroom: normal churn passes, a meaningful block of uncovered code does not. Six live references move together because the repo pins its CI command strings: the workflow, the three assertions in tests/test_supply_chain_policy.py that mirror it, and the docs that print the checklist people actually copy (AGENTS.md, README.md, docs/operations.md, docs/adding-a-screener.md). Untouched on purpose: the --cov-fail-under=84 lines in the handoff docs, the audit register, and the archived plans. Those record what the floor WAS when that work shipped; rewriting them would falsify history. Also drops the now-stale price-band caveats, since the previous commit implemented it: the ADR's "known deviation" section becomes a description of what shipped (cap-only, largest-on-span, optional/paired), operations.md no longer warns that priced issues are unreachable, and the extraction LLD is corrected to extractor-v3 / dual-accepted v2-v3 evidence. Gates: full pytest at the new floor (1825 passed, 89.65%), supply-chain policy test green against the edited workflow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merging origin/main brought in DATA-002's 20260817data002 migration, which branches from the same parent as this branch's IPO-011 migration. Two children of 20260718ipo010 means two Alembic heads, and "alembic upgrade head" refuses to pick one - which is why 17 tests failed immediately after the merge rather than anything being logically wrong with either change. This migration is unreleased, so the honest fix is to move it rather than to add a merge revision: down_revision becomes 20260817data002, and the id is renamed 20260720ipo011 -> 20260820ipo011 so the date prefix still matches the position in the chain. Nothing outside the file referenced the old id. The migration body is byte-identical; only its place in the order changed. Note for anyone with a local dev database stamped at 20260720ipo011: that id no longer exists, so alembic cannot locate it. Delete the throwaway sqlite file and let the bootstrap rebuild it. No deployed environment is affected - this branch has never been merged. Gates on the merged tree: pytest 1915 passed (89.82%), ruff, mypy clean (main's pandas 3.x alignment removed the stubs mismatch), compileall, bandit, pip-audit, and no constraints/pyproject drift against main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ten findings, four of which were mine inverting their own stated guarantee.
SAFETY - the containment rule was backwards
The weak_qib_demand_near_close guard skipped the rule whenever a
low-confidence snapshot existed. But the collector writes a snapshot for
exactly the issues that have NO official evidence - which is precisely when
this caution is meant to fire. So a scraped headline converted "triggered"
into "not_evaluable" and removed the override that forces Not Recommended.
The guarantee was written as "a scrape can never fire this flag"; the real
guarantee is that a scrape cannot move it in EITHER direction. A
low-confidence snapshot is now discarded before any decision, so the answer
is identical to what it would have been had the collector never run. The
test now asserts that equivalence rather than a fixed status.
SAFETY - the wrong category's number was read as QIB demand
A status headline prints every category at once, and the parser bound the
first multiple within 40 characters of the QIB anchor. "Retail 25.6 times,
QIB 1.2 times" therefore recorded a 25.6x institutional book where the truth
was 1.2x - a weak book read as a strong one, in the dangerous direction.
Commas now end a clause, and the QIB anchor must be strictly closer to the
number than any competing category word. Ambiguity yields no reading.
Also: official evidence now outranks a web snapshot at READ time, not only at
write time. An exchange snapshot carries its own publication timestamp, so
recording it after an evening scrape gives it the earlier captured_at, and
ordering by recency alone let the scrape shadow it permanently.
CORRECTNESS - the v2 back-compatibility path never worked
The receipt verifier rebuilds the proposal model by subscripting every known
field, so a genuine pre-IPO-011 payload - which has no price_band_high keys
at all - raised KeyError, was swallowed by the broad except, and surfaced as
"receipts do not match the verified cached PDF". That reads like tampering
and would have made every in-flight v2 review un-approvable, silently voiding
the compatibility this PR advertises. The test fixture had been patched for
this; production had not. Verified by reverting the fix and watching the new
test reproduce the exact error.
CORRECTNESS - the cap price broke sibling facts, and rejected good citations
Registering price_band_high in the shared field-label table added a second
semantic field to every span mentioning a price band - including the standard
"Basis for the Offer Price" rows stating EPS or NAV "in relation to the Price
Band". The "exactly one field per span" rule then rejected them, and EPS is a
core label, so proposals that verified cleanly before this ticket would fail
outright. The cap price is now matched by its own price constructs and is not
in the shared table, so no other field's ambiguity count changes.
The cap guard also demanded the claim be the largest number anywhere on the
span, which rejected real cover-page rows carrying a bid lot ("150 Equity
Shares") beside a two-digit price. It now binds to the upper bound of a band
the span actually prints, or a cap price it names outright - stricter about
what counts as evidence, and free of the false rejection.
SCOPE - a capped run reached outside itself
Issue selection was computed BEFORE ingestion, so a filing discovered by that
very run was filtered out of download, enrichment, extraction and scoring, and
would only be processed on a second button press. Ingestion is now its own
pass (new skip_score on the pipeline) and selection is taken from the snapshot
that exists afterwards.
Auto-approval ran unscoped, converting every pending HIGH proposal in the
database including issues excluded by only_active_issues or max_issues - while
the follow-up scoring pass covered only the selection, leaving those approvals
stale. It now takes the run's own issue set.
ROBUSTNESS
- Approval compared the extracted cap against a stored price-band floor, so a
conflict raises the typed "needs review" error instead of an IntegrityError
escaping as a raw traceback and aborting an unattended run.
- auto_approve_ready_proposals caught only two exception types despite
promising failures are "counted, never raised". Approval touches the
database and the document cache, so an IntegrityError or an OSError reading
a cached PDF aborted the whole screener run after every earlier stage had
succeeded. The catch is now broad, recording only the exception type.
- _expected_cited_facts raised a bare KeyError on a half-written payload
because it runs outside the caller's error conversion.
OPERATIONS - IPO rows poisoned the forward-return queue
Result rows carried a signal_date, and VALID-002 selects every scan result
that has one. Those rows can never resolve the "ipo_filings" universe to
instruments, so every horizon stored PENDING, was re-selected next run, and
consumed the batch budget forever. signal_date is now null - a forward return
is meaningless for an IPO issue - and the evaluation date moved to its own
scored_on column.
QUALITY
- The active-status tuple was duplicated between the screener and the
pipeline; the pipeline's is now public and shared, so the button and the
terminal cannot disagree about what "active" means.
- The selection filter was copy-pasted between the processed set and the
reported set; one helper now serves both.
- The median-of-readings block was duplicated verbatim between the GMP and
subscription parsers.
- price_band_high discarded its normalized value, leaving it the only field
stored with the model's raw spacing.
- The QIB factor's receipt now names a low-confidence web source, matching the
GMP convention, so a reader can tell a scraped headline from an exchange
filing. FACTOR_MODEL_VERSION -> ipo-006-factors-v3.
Gates: pytest 1928 passed (89.79%), ruff, mypy clean, compileall, bandit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the IPO pipeline dispatchable from the ordinary Run screener button as a screener named IPO Screener, alongside the technical-analysis and 67-ka-funda agents.
It re-implements no pipeline stage —
run()calls the samerun_ipo_screener()the CLI calls — so the button and the terminal cannot drift apart, and the run inherits scan history, run status, provenance receipts, and the audit trail from the existing scan lifecycle.Commits (each gate-green in isolation)
requires_candlesframework opt-out — a screener may declare it needs no candle stack; the dispatcher then skips the Dhan credential check, the universe CSV, and theDailyDataLoader. DefaultTrue, so every existing screener is untouched.20260720ipo011— widens the enrichment signal-type CHECK forsubscription_demand. Schema-only, so the collector lands separately.ipo_subscriptionsrows, and the containment rule below.IPO_AUTO_APPROVE_HIGH_CONFIDENCE(default off), HIGH-only, attributed to a reserved automation identity.screeners/ipo_screener.py, toggles → pipeline stages, per-stage progress, one result row per IPO issue.ipo-011design doc, operations/authoring/AGENTS updates.Trust decisions worth reviewing closely
HIGHmeans the host already re-resolved every cited value from the hash-verified PDF;MEDIUMstill waits for a person. With the switch off, behaviour is identical to the reviewed flow.weak_qib_demand_near_closehard caution. That flag forcesNot Recommendedregardless of score, so a scraped headline must not be able to reject an issue — it reportsnot_evaluableinstead. A test pins that identical numbers differ only by provenance.skipped_unchanged).Nonerather than being misread as institutional demand.The plan also specified extracting the price band from the RHP as a cited fact. That is not in this PR. It touches
_CITED_FACT_SCHEMA_VERSION, the expected-fact map, and approval-time re-resolution — the most safety-critical path in the repo — and warrants its own focused change rather than a rushed pass at the end of a large PR.Consequence: an issue with no price band leaves
valuationmissing, and valuation is a critical factor, so those issues still resolve to "Insufficient verified data". The button is fully wired and autonomous, but will not yet produce positive verdicts for unpriced issues. The follow-up is specified indocs/architecture/ipo-011-one-button-screener.md.Verification
normalize_screener_rowthe scan service uses, for both scored and unscored issues.git diff origin/main HEAD -- constraints.txt pyproject.tomlis empty (no new dependencies).🤖 Generated with Claude Code