diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index 04894c7..946ce19 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -41,6 +41,10 @@ jobs: go vet ./... go build -o /dev/null ./... + - name: Build the programs of the skill for AI agents + if: matrix.go-version == 'stable' + run: scripts/check-skill.sh + - name: Test zap logger module if: matrix.go-version == 'stable' working-directory: logger/zaplogger diff --git a/scripts/check-skill.sh b/scripts/check-skill.sh new file mode 100755 index 0000000..8789d44 --- /dev/null +++ b/scripts/check-skill.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# Builds the complete programs (code blocks with "package main") of the skill for AI agents against the code of the repository, +# so the skill does not describe the API that does not exist. +set -euo pipefail + +root=$(cd "$(dirname "$0")/.." && pwd) +skill=${1:-$root/skills/tinyconf/SKILL.md} +# SKILL_WORKDIR keeps the built programs to run them by hand +if [ -n "${SKILL_WORKDIR:-}" ]; then + work=$SKILL_WORKDIR + mkdir -p "$work" +else + work=$(mktemp -d) + trap 'rm -rf "$work"' EXIT +fi + +awk -v dir="$work" ' + /^```go$/ { block = ""; inside = 1; next } + /^```$/ && inside { + inside = 0 + if (block ~ /^package main/) { + n++ + path = dir "/p" n + system("mkdir -p " path) + printf "%s", block > (path "/main.go") + close(path "/main.go") + } + next + } + inside { block = block $0 "\n" } +' "$skill" + +count=$(find "$work" -name main.go | wc -l) +if [ "$count" -eq 0 ]; then + echo "no programs in $skill" >&2 + exit 1 +fi +cat > "$work/go.mod" < $root +MOD +cp "$root/go.sum" "$work/go.sum" +cd "$work" +go mod tidy >/dev/null +go vet ./... +go build ./... +echo "$count programs of $(basename "$(dirname "$skill")") skill are built" diff --git a/skills/tinyconf/SKILL.md b/skills/tinyconf/SKILL.md new file mode 100644 index 0000000..8042081 --- /dev/null +++ b/skills/tinyconf/SKILL.md @@ -0,0 +1,261 @@ +--- +name: tinyconf +description: Write Go code that loads configuration with github.com/insei/tinyconf/v2 — config structs with tags filled by a chain of drivers (tag defaults, yaml, json, toml, ini, properties, env, flag, Consul, Vault, etcd), sub configs of packages, reloading with Subscribe and Current, required fields, secrets, value sources, generated config templates and loggers. Use when the code imports github.com/insei/tinyconf, or when a Go service needs configuration from several sources with priorities or reloading without restart. +--- + +# tinyconf v2 + +A config is a Go struct. Every field says where its value lives by tags, drivers fill the struct in the order they are added, **the next driver overrides the previous one** field by field. Full documentation: https://insei.github.io/tinyconf/latest/, as one file for the context: https://insei.github.io/tinyconf/latest/llms-full.txt. + +## Rules + +- The module path is `github.com/insei/tinyconf/v2`: `go get github.com/insei/tinyconf/v2@latest`, the drivers are `github.com/insei/tinyconf/v2/drivers/`. Never import `github.com/insei/tinyconf` without `/v2`, it is v1 with another API. +- `Parse` takes the context: `config.Parse(ctx, conf)`. There is no `ParseContext`, no `Watch` and no `logger.NewZapLogger` in v2. +- `Register` the config (a pointer to struct) before `Parse`. `Parse` accepts the registered config or a field of it with struct type (a sub config), otherwise `tinyconf.ErrNotRegisteredConfig`. +- `Parse` parses all fields and returns all errors as one error, check it with `errors.Is(err, tinyconf.ErrRequired)` and so on. A field gets the value of the last driver that has one, a field without values keeps the value it had before `Parse`. +- A field is read by a driver only if it has the tag of the driver (`yaml:"port"`, `env:"HTTP_PORT"`...). The yaml, json, toml, ini, properties, Consul, Vault and etcd drivers need the tags on the parent structs too: `yaml:"port"` inside `yaml:"http"` is read from `http.port`. The env and flag drivers use the full name in the tag of the field. +- Defaults are the `initial:"..."` tags read by `tag.New("initial")` as the first driver, or values set in the struct before `Parse`. + +| Tag | Meaning | +|---|---| +| `initial:"8080"` | default value, read by the tag driver created with `tag.New("initial")` (any tag name works) | +| `yaml`, `json`, `toml`, `ini`, `properties`, `consul`, `vault`, `etcd` | key of the field inside its parent | +| `env:"HTTP_PORT"`, `flag:"http-port"` | full name of the variable or the flag | +| `required:"true"` | `Parse` returns `tinyconf.ErrRequired` if no driver has a value and the field is zero | +| `hidden:"true"` | the value is masked in the logs and is empty in `GenDoc` | +| `doc:"port of the server"` | description in the templates generated by `GenDoc` | +| `merge:"true"` | slices are appended and maps are merged from all drivers instead of overriding | + +Supported types: strings, numbers, bools, `time.Duration`, `time.Time`, `url.URL`, `encoding.TextUnmarshaler`, slices, arrays, maps, nested structs, pointers. Use `tinyconf.Secret` for passwords and tokens: it prints `******` in `fmt`, JSON and logs, `Value()` returns the value. + +## A service with defaults, a file, the environment and flags + +```go +package main + +import ( + "context" + "fmt" + "os" + "time" + + "github.com/insei/tinyconf/v2" + "github.com/insei/tinyconf/v2/drivers/env" + "github.com/insei/tinyconf/v2/drivers/flag" + "github.com/insei/tinyconf/v2/drivers/tag" + "github.com/insei/tinyconf/v2/drivers/yaml" +) + +type HTTP struct { + Host string `initial:"localhost" yaml:"host" env:"HTTP_HOST" flag:"http-host" doc:"host of the server"` + Port int `initial:"8080" yaml:"port" env:"HTTP_PORT" flag:"http-port" doc:"port of the server"` + Timeout time.Duration `initial:"30s" yaml:"timeout" env:"HTTP_TIMEOUT" doc:"timeout of the requests"` +} + +type Database struct { + DSN string `yaml:"dsn" env:"DB_DSN" required:"true" doc:"address of the database"` + Password tinyconf.Secret `yaml:"password" env:"DB_PASSWORD" doc:"password of the database"` +} + +type Config struct { + HTTP HTTP `yaml:"http"` + Database Database `yaml:"database"` +} + +func main() { + tagDriver, err := tag.New("initial") + if err != nil { + panic(err) + } + // the file is optional: the defaults and the environment are enough + yamlDriver, err := yaml.New("config.yaml", yaml.WithOptional()) + if err != nil { + panic(err) + } + envDriver, err := env.New() // also reads .env files if they exist + if err != nil { + panic(err) + } + flagDriver, err := flag.New() // reads os.Args, unknown flags are ignored + if err != nil { + panic(err) + } + // priority: tags < config.yaml < environment < flags + config, err := tinyconf.New( + tinyconf.WithDriver(tagDriver), + tinyconf.WithDriver(yamlDriver), + tinyconf.WithDriver(envDriver), + tinyconf.WithDriver(flagDriver), + ) + if err != nil { + panic(err) + } + conf := &Config{} + if err = config.Register(conf); err != nil { + panic(err) + } + if err = config.Parse(context.Background(), conf); err != nil { + fmt.Fprintln(os.Stderr, "config:", err) + os.Exit(1) + } + fmt.Printf("%+v\n", *conf) // the password is printed as ****** + + // which driver and which key gave every value + for _, source := range config.Sources(conf) { + fmt.Println(source.Field, source.Driver, source.Source) + } + // the template of the config with the current values and the doc tags: "yaml", "env", "flag"... + fmt.Print(config.GenDoc("env")) +} +``` + +## Packages own their parts: sub configs and reloading + +A package works with its own struct type, not with the whole config. `Parse(ctx, &HTTP{})` parses only this part of the registered config, `tinyconf.Current[HTTP](config)` returns its current version, `tinyconf.Subscribe` calls the handler only when this part changes. Reloading needs a driver that watches its source: files (by the events of the file system), Consul, Vault, etcd; otherwise `Subscribe` returns `tinyconf.ErrNoWatchers`. + +```go +package main + +import ( + "context" + "fmt" + "os" + "os/signal" + + "github.com/insei/tinyconf/v2" + "github.com/insei/tinyconf/v2/drivers/tag" + "github.com/insei/tinyconf/v2/drivers/yaml" +) + +type HTTP struct { + Port int `initial:"8080" yaml:"port"` +} + +type Config struct { + HTTP HTTP `yaml:"http"` +} + +// server is a package that knows only its own config. +type server struct { + config *tinyconf.Manager +} + +func (s *server) handle() { + // Current is an atomic read without allocations, call it on every request; never modify the result + port := tinyconf.Current[HTTP](s.config).Port + fmt.Println("port:", port) +} + +func main() { + tagDriver, err := tag.New("initial") + if err != nil { + panic(err) + } + yamlDriver, err := yaml.New("config.yaml", yaml.WithOptional()) + if err != nil { + panic(err) + } + config, err := tinyconf.New(tinyconf.WithDriver(tagDriver), tinyconf.WithDriver(yamlDriver)) + if err != nil { + panic(err) + } + conf := &Config{} + if err = config.Register(conf); err != nil { + panic(err) + } + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) + defer stop() + if err = config.Parse(ctx, conf); err != nil { + panic(err) + } + + s := &server{config: config} + s.handle() + // called after a reload only if HTTP has changed; a broken file keeps the last good config. + // The subscription ends with ctx, watching stops after the last subscription. + err = tinyconf.Subscribe(ctx, config, func(prev, next *HTTP) { + fmt.Printf("port: %d -> %d\n", prev.Port, next.Port) + }) + if err != nil { + panic(err) + } + <-ctx.Done() +} +``` + +`config.ReloadStatus()` reports the last reload error and the drivers that stopped watching, for health checks. Reloads never modify the config passed to `Parse` and the versions returned by `Current`, they create new versions. + +## The logger configured by the config itself + +Without a logger nothing is logged. `tinyconf.WithLogger(l)` sets it, the bundled ones are `logger.NewFmtLogger(logger.INFO)`, `logger.NewSlogLogger(*slog.Logger)` (Go 1.21+) and `zaplogger.New(*zap.Logger)` from the separate module `github.com/insei/tinyconf/logger/zaplogger`. If the logger is configured by the config, use `WithDeferredLogger`: the records are kept until `SetLogger`, which is safe for concurrent use. + +```go +package main + +import ( + "context" + "log/slog" + "os" + + "github.com/insei/tinyconf/v2" + "github.com/insei/tinyconf/v2/drivers/env" + "github.com/insei/tinyconf/v2/logger" +) + +type Logging struct { + Level string `env:"LOG_LEVEL" doc:"debug, info, warn, error"` +} + +type Config struct { + Logging Logging +} + +func newLogger(conf Logging) tinyconf.Logger { + var level slog.Level + _ = level.UnmarshalText([]byte(conf.Level)) + return logger.NewSlogLogger(slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: level}))) +} + +func main() { + envDriver, err := env.New() + if err != nil { + panic(err) + } + config, err := tinyconf.New(tinyconf.WithDriver(envDriver), tinyconf.WithDeferredLogger()) + if err != nil { + panic(err) + } + conf := &Config{} + if err = config.Register(conf); err != nil { + panic(err) + } + err = config.Parse(context.Background(), conf) + config.SetLogger(newLogger(conf.Logging)) // the records of Parse are written to it + if err != nil { + panic(err) + } +} +``` + +## Drivers + +| Driver | Constructor | Notes | +|---|---|---| +| defaults | `tag.New("initial")` | the first driver | +| yaml, json, toml, ini, properties | `yaml.New("config.yaml", opts...)` | `WithFiles(...)` more files, `WithFS(fsys, ...)`, `WithData(name, []byte)`, `WithOptional()` absent files are skipped, `WithEnvExpansion()` `${VAR}` in the file, `WithWatchInterval(d)`; `yaml.New("", yaml.WithData(...))` without a file | +| env | `env.New(opts...)` | `WithPrefix("APP_")`, `WithAutoKeys()` names from the field paths without tags (`HTTP.MaxConns` → `HTTP_MAX_CONNS`), `WithIndexedKeys()` `SERVERS_0_HOST`, `WithSecretFiles()` `DB_PASSWORD_FILE`, `WithFiles(".env.local")` custom .env files, `WithFiles()` no .env files | +| flag | `flag.New(opts...)` | `WithArgs(args)` instead of `os.Args[1:]`; slices by repeated or comma separated flags: `--host=a --host=b,c` | +| Consul KV | `consul.New("app/config", opts...)` | key per field: `prefix/http/port`; `WithAddress`, `WithToken` (or `CONSUL_HTTP_ADDR`, `CONSUL_HTTP_TOKEN`), `WithDatacenter`; a whole document: `yaml.New("", yaml.WithSource(consul.Document("app/config")))` | +| Vault KV | `vault.New("app", opts...)` | one secret with JSON keys; `WithAddress`, `WithToken`, `WithTokenFile` (or `VAULT_ADDR`, `VAULT_TOKEN`), `WithMount("secret")`, `WithKVv1()`, `WithPaths(...)`, `WithOptional()`; changes by polling | +| etcd | `etcd.New("/app/config", opts...)` | key per field; `WithEndpoints(...)`, `WithAuth(user, password)`; `etcd.Document(key)` like Consul | +| any format | `format.New("hcl", decode, "config.hcl", opts...)` | `decode func([]byte) (map[string]any, error)`, the options of the file drivers, `WithEncoder` for `GenDoc` | + +## Common mistakes + +- Importing `github.com/insei/tinyconf` (v1) or calling `Parse(conf)` without the context. +- `Parse` before `Register`, or a non-pointer config. +- A missing tag on the parent struct for the file and remote drivers: `yaml:"port"` in a struct field without `yaml:"http"` is not read from `http.port`. +- Expecting `env:"PORT"` to get a prefix from the parent struct: env and flag tags are full names (use `env.WithPrefix` or `env.WithAutoKeys` for generated names). +- Modifying the struct returned by `Current`: it is shared by all readers. +- `Subscribe` without a driver that watches (only tag, env and flag drivers): `ErrNoWatchers`. +- Logging the config with a `string` password field: use `tinyconf.Secret` or at least `hidden:"true"` for the logs of tinyconf.