SmartScraperGraph returns a confident false negative when the answer is on a linked page (no signal that DepthSearchGraph was needed)

[SilverBulletNeo]

Summary

When the answer to a prompt lives on a linked page (a privacy policy, terms, an "our team" page), SmartScraperGraph returns a confident negative rather than any signal that the answer might be elsewhere. The output is indistinguishable from a genuine "this fact is not present on the site."

I know DepthSearchGraph is the right tool for this (per #1008). The issue is that nothing tells you that you needed it — the single-page result looks like a complete answer.

Reproduction

Prompt (a compliance question — does the site forbid transferring contact data?):

Does this page, or a privacy policy it links to, state that email addresses or
personal information will NOT be given, sold, rented, traded or transferred to
third parties? Answer strictly as JSON:
{"no_transfer_notice": true|false, "evidence": "<=25 words"}

SmartScraperGraph(prompt=PROMPT, source=url, config={
    "llm": {"api_key": KEY, "model": "google_genai/gemini-2.5-flash",
            "model_tokens": 1000000},
    "headless": True,
}).run()

Two public sites where the notice is real but sits one link away:

URL	Notice lives at	Actual text	SmartScraperGraph
https://www.covenantcarepractices.com/contact/	/privacy/	"this information will not be shared with any third parties"	no_transfer_notice: False
https://excelsiorhealthcare.com/contact-us/	/privacy-policy/	"does not and will never sell or rent any identifiable information to third party vendors"	no_transfer_notice: False

A ~30-line baseline (fetch page → follow href matching privacy|terms → concatenate → same model, same prompt) returns True with the quoted evidence for both. Across 10 URLs the two approaches were the same speed (4.6 vs 4.9 s/url) and agreed on 6 of 8 comparable pages; both disagreements were this pattern.

Why it's worth a signal rather than just docs

For extraction tasks the failure is a wrong field. For compliance questions the failure has a direction: a false False means "no restriction found," which is precisely the answer that causes someone to act. Silent false negatives are the expensive kind.

Suggestions (any one would help)

1. Have SmartScraperGraph return an explicit "not found on this page" state distinct from a negative finding.
2. When the fetched page contains links whose text/href matches terms in the prompt and the answer is negative, surface a hint that a depth graph may be required.
3. Document on the SmartScraperGraph page that link-dependent questions need DepthSearchGraph, with a worked example — the current docs make the single-page graph the obvious default.

Environment

scrapegraphai installed fresh from PyPI, Python 3.13, macOS, langchain-google-genai, Playwright chromium headless.

Happy to open a PR for (3) if that's the preferred shape.