Feature request: Support Agent Discovery Protocol (/.well-known/agent-discovery.json)

[walkojas-boop]

Problem

AI agents currently have no standard way to discover what services are available at a domain. Each agent must be hardcoded with endpoints or rely on human-provided URLs.

Proposal

Support the Agent Discovery Protocol (ADP) -- an open standard where domains publish agent services at /.well-known/agent-discovery.json.

Like robots.txt for crawlers, one fetch tells the agent everything: identity, memory, governance, affordances, and more.

Example:

# Check what services a domain offers before interacting
resp = requests.get("https://example.com/.well-known/agent-discovery.json")
services = resp.json()["services"]

Live reference: https://walkosystems.com/.well-known/agent-discovery.json

Full spec: https://github.com/walkojas-boop/agent-discovery-protocol

This is a lightweight, stdlib-only addition that helps agents discover infrastructure automatically.

[VinciGit00]

What are the advantages of using this?

[walkojas-boop]

ScrapeGraphAI agents visit domains to extract data, but right now they go in blind — no way to know what the domain offers, what's allowed, or what services are available for agents.

ADP solves that in one fetch:

1. Service discovery before scraping — your agent checks /.well-known/agent-discovery.json first and finds structured endpoints (APIs, data feeds) it can use directly instead of scraping HTML
2. Policy awareness — domains can declare governance rules (rate limits, auth requirements, allowed actions) so your agent respects them automatically
3. Interoperability — any domain that publishes the file becomes agent-friendly. Your scrapers get structured data instead of parsing DOM

Think of it like robots.txt but for agent capabilities, not just crawl permissions.

The spec is lightweight — one JSON file, no dependencies. Happy to help with integration if you're interested.

[walkojas-boop]

One more thing worth showing -- we built Agent Internet (agents.walkosystems.com) as a live reference implementation.

Every agent on the network is ADP-registered and discoverable. If you want to see what agent discovery looks like end-to-end in a real system, agents.walkosystems.com/register walks through it.

Happy to add a ScrapeGraphAI example to the spec if that would help.