The cl0q crawler's bot identity: the User-Agent strings its crawlers send and the
robots.txt user-agent names that address them (cl0q-crawler, cl0q-discovery,
cl0q-favicon, cl0q-favicon-grabber (former name of cl0q-favicon),
cl0q-node, cl0q). Shared by the crawlers and the robots.txt parser
so the name a site owner blocks and the name we send always match.
go get codeberg.org/cl0qsearch/botidRequires Go 1.25+.
go test ./...
go build ./...One file (botid.go), stdlib only. The bot names and User-Agent formats live here
— and only here — so the name a site owner sees in their logs is always the name
they can block in robots.txt. The crawlers take the identity from this package;
it is not configurable per deployment.
botid.MustUserAgent() reads the CL0Q_NODE_ID environment variable (the operator id
assigned at enrollment) and exits if it is unset: a node must never crawl anonymously.
Library users who manage the id themselves should prefer botid.UserAgent(id).
import "codeberg.org/cl0qsearch/botid"
// Generate a User-Agent for your crawler node; always check the error,
// a bad node id is rejected, not silently used.
ua, err := botid.UserAgent("my-node")
if err != nil {
log.Fatal(err)
}
// Mozilla/5.0 (compatible; cl0q-node/1.0; operator=my-node; +https://cl0q.com/bot)
// Check if a robots.txt user-agent name refers to us
botid.IsOurs("cl0q-crawler") // true
botid.IsOurs("googlebot") // falsebotid.MustUserAgent() reads the CL0Q_NODE_ID environment variable (the operator id
assigned at enrollment) and exits if it is unset: a node must never crawl anonymously.
// Exits if CL0Q_NODE_ID is not set
ua := botid.MustUserAgent()Library users who manage the id themselves should prefer botid.UserAgent(id).
Generates a User-Agent string for the given node ID. The node ID must be 2-32
characters: the first character a-z or 0-9, the rest a-z, 0-9, '-' or ''
(a leading '-' or '' is rejected). Returns an error wrapping
ErrInvalidNodeID when the rule fails.
Like UserAgent but reads the node ID from CL0Q_NODE_ID and exits on failure.
For use in crawler binaries, not libraries.
Reports whether the given robots.txt user-agent name addresses a cl0q crawler.
Matches cl0q-crawler, cl0q-discovery, cl0q-favicon, cl0q-favicon-grabber,
cl0q-node, and cl0q (case-insensitive).
None — this is a library with no knobs.
go test ./...
go test -race ./...Pull requests welcome. Please:
- Run
gofmton all changes - Add tests for new functionality
- Keep the User-Agent format stable (sites may parse it)
- robotstxt — the robots.txt parser that matches these names.
- crawler-node — the crawler that sends these User-Agents (vendored as
botid/).
MIT. See LICENSE.