Skip to content

llms.txt points at .md mirrors that 404 (the HTML pages are fine) #106

Description

@siliroid

Hi — the .md mirrors that https://docs.speakeasy.com/llms.txt points at are 404ing for a good chunk of the index, while the corresponding HTML pages are all fine.

I was sampling published llms.txt files across ~25 doc sites for an unrelated reason. Speakeasy came out as an outlier, so I went and checked it properly instead of trusting the first number.

Random sample, 15 of the 20 indexed URLs, seeded so it's reproducible — 7 return 404:

404  /md/docs/standalone-mcp/overview.md
404  /md/docs/standalone-mcp/build-server.md
404  /md/docs/standalone-mcp/setting-up-oauth.md
404  /md/docs/standalone-mcp/custom-resources.md
404  /md/docs/standalone-mcp/remote-mcp-servers.md
404  /md/docs/standalone-mcp/cloudflare-deployment.md
404  /md/docs/speakeasy-reference/generation/terraform-config.md

The part I think actually matters: the content isn't missing, only the machine copy is. Strip /md/ and the .md and every one of them is a 200:

/md/docs/standalone-mcp/overview.md                          404
/docs/standalone-mcp/overview                                200

/md/docs/standalone-mcp/build-server.md                      404
/docs/standalone-mcp/build-server                            200

/md/docs/speakeasy-reference/generation/terraform-config.md  404
/docs/speakeasy-reference/generation/terraform-config        200

So the site is healthy and the mirror that llms.txt exists to serve is not. Which means the audience that hits this is the only one that can't tell you about it — a person browsing never touches /md/, and an agent following your index just gets a 404 and moves on. I'd guess that's why it's been sitting there.

Checked before filing, because either would have made this mine rather than yours:

  • Not my user agent. Every non-200 refetched with a normal Chrome UA — zero recovered.
  • Not an unfollowed redirect. redirect: 'follow' throughout, and the HTML equivalents resolve 200 under the identical client.

Repro:

curl -s https://docs.speakeasy.com/llms.txt \
  | grep -oE 'https://[^)]+' \
  | while read u; do printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u"; done \
  | grep -v '^200'

No ask attached — it just seemed like the kind of thing that stays broken precisely because nobody who can see it is looking.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions