Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ vaultDriver, err := vault.New("app/config", // vault:"password" inside vault:"da
)
```
The address and the token are taken from `VAULT_ADDR` and `VAULT_TOKEN` or set by options, KV version 2 at `secret/` by default.
The driver can log in itself: `vault.WithAppRole(roleID, secretID)` or `vault.WithKubernetes("app", "")` by the service account of the pod.
The secrets are checked for changes every 30 seconds while there are subscriptions.

## etcd
Expand Down
32 changes: 30 additions & 2 deletions docs/drivers/vault.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,9 @@ vaultDriver, err := vault.New("app/config",
| `WithAddress(address)` | `VAULT_ADDR` or `https://127.0.0.1:8200` | URL of Vault |
| `WithToken(token)` | `VAULT_TOKEN` | the token |
| `WithTokenFile(path)` | | the file with the token, it is read before every request |
| `WithAppRole(roleID, secretID)` | | login by AppRole, see [Authentication](#authentication) |
| `WithKubernetes(role, tokenFile)` | | login by the service account of the pod |
| `WithAuthMount(mount)` | `approle`, `kubernetes` | the mount of the auth method |
| `WithNamespace(namespace)` | `VAULT_NAMESPACE` | the namespace of Vault Enterprise |
| `WithMount(mount)` | `secret` | the mount of the KV secrets engine |
| `WithKVv1()` | KV version 2 | read from the KV secrets engine version 1 |
Expand All @@ -70,7 +73,32 @@ The first argument of `New` can be an empty string, if the secrets are set by `W

## Authentication

The driver uses a token. Other methods — Kubernetes, AppRole, cloud IAM — are handled by [Vault Agent](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent): it logs in, renews the token and writes it to the file. Point the driver to this file:
The driver logs in by AppRole or by the service account of a Kubernetes pod, or uses a token.

### AppRole

```go
vaultDriver, err := vault.New("app/config",
vault.WithAppRole(os.Getenv("VAULT_ROLE_ID"), os.Getenv("VAULT_SECRET_ID")),
)
```

### Kubernetes

```go
// the token of the service account from /var/run/secrets/kubernetes.io/serviceaccount/token
vaultDriver, err := vault.New("app/config", vault.WithKubernetes("app", ""))
```

The second argument is the file with the token of the service account, a projected token with the audience of Vault for example. The file is read before every login, so the token rotated by Kubernetes is used.

The auth methods are mounted at `approle` and `kubernetes` by default, another mount is set by `WithAuthMount("k8s-prod")`.

The driver logs in at the first request and keeps the token. The new token is requested after 2/3 of its lease, and when Vault denies the request: the token can be revoked before its lease expires. Wrong credentials are an error of `Parse` that wraps `tinyconf.ErrAccessDenied`. A login method has priority over `WithToken`, `WithTokenFile` and `VAULT_TOKEN`.

### Token

`WithToken` or the `VAULT_TOKEN` variable set the token. Other methods — cloud IAM, LDAP, JWT — are handled by [Vault Agent](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent): it logs in, renews the token and writes it to the file. Point the driver to this file:

```go
vaultDriver, err := vault.New("app/config", vault.WithTokenFile("/vault/secrets/token"))
Expand All @@ -83,7 +111,7 @@ The file is read before every request, so the renewed token is used without rest
| Case | Result |
|---|---|
| the secret is absent | error of `Parse`, unless `WithOptional()` |
| the token can not read the secret | error of `Parse` that wraps `tinyconf.ErrAccessDenied`, logged as `access denied` |
| the token can not read the secret, wrong credentials of the login | error of `Parse` that wraps `tinyconf.ErrAccessDenied`, logged as `access denied` |
| Vault is not available or sealed | error of `Parse` |

The driver is skipped in the `Parse` call with the error, see [Missing files and values](yaml.md#missing-files-and-values).
Expand Down
4 changes: 2 additions & 2 deletions docs/use-cases.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,14 +211,14 @@ Where the value comes from:

```go
// add after the file driver: the secret overrides the file
vault.New("app/database", vault.WithTokenFile("/vault/secrets/token"))
vault.New("app/database", vault.WithKubernetes("app", ""))
```

```go
Password string `yaml:"password" vault:"password" hidden:"true" required:"true"`
```

The token is renewed by Vault Agent, see [vault](drivers/vault.md#authentication).
The driver logs in by the service account of the pod, AppRole and the token of Vault Agent are supported too, see [vault](drivers/vault.md#authentication).

## Containers

Expand Down
2 changes: 1 addition & 1 deletion docs/why-tinyconf.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ Checked against the current documentation of the libraries on 2026-10-16.
## Known limitations

- **Fewer remote stores than koanf.** Consul, Vault and etcd only; others need a [custom driver](drivers/custom.md).
- **Vault: token authentication only.** Kubernetes, AppRole and cloud login are delegated to Vault Agent; changes are detected by polling, because Vault does not notify.
- **Vault: AppRole, Kubernetes and token authentication.** Cloud IAM, LDAP and JWT login are delegated to Vault Agent; changes are detected by polling, because Vault does not notify.
- **No HCL out of the box.** The [format driver](drivers/format.md) accepts any decoder, but the conversion of HCL to a map is up to you.
- **Own flag parser.** It ignores unknown flags, so it lives next to cobra or any CLI framework, but it is not a pflag integration.
- **Subscriptions are package functions,** `tinyconf.Subscribe(ctx, config, ...)`, to keep Go 1.19 support; methods with type parameters need Go 1.27.
Expand Down
195 changes: 172 additions & 23 deletions drivers/vault/client.go
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
package vault

import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
Expand All @@ -13,6 +15,7 @@ import (
"os"
"strconv"
"strings"
"sync"
"time"

"github.com/insei/tinyconf/v2"
Expand All @@ -26,6 +29,8 @@ const (
DefaultMount = "secret"
// DefaultWatchInterval is the interval of checking the secrets for changes, Vault does not notify about them.
DefaultWatchInterval = 30 * time.Second
// KubernetesTokenFile is the token of the service account of the pod, it is used by WithKubernetes by default.
KubernetesTokenFile = "/var/run/secrets/kubernetes.io/serviceaccount/token"
)

// requestTimeout limits the requests to Vault.
Expand All @@ -35,6 +40,8 @@ type options struct {
address string
token string
tokenFile string
login *login
authMount string
namespace string
mount string
kv1 bool
Expand Down Expand Up @@ -69,6 +76,50 @@ func WithTokenFile(path string) Option {
}
}

// login is the auth method that gets the token from Vault.
type login struct {
mount string
// data returns the body of the login request, it is called before every login
data func() (map[string]any, error)
}

// WithAppRole gets the token by the AppRole auth method with the role ID and the secret ID,
// the auth method is mounted at approle by default, it is changed by WithAuthMount.
// The token is requested again before its lease expires. It has priority over the token from the other options.
func WithAppRole(roleID, secretID string) Option {
return func(o *options) {
o.login = &login{mount: "approle", data: func() (map[string]any, error) {
return map[string]any{"role_id": roleID, "secret_id": secretID}, nil
}}
}
}

// WithKubernetes gets the token by the Kubernetes auth method with the role and the token of the service account
// from the tokenFile, KubernetesTokenFile if it is empty. The file is read before every login, so the rotated token is used.
// The auth method is mounted at kubernetes by default, it is changed by WithAuthMount.
// The token of Vault is requested again before its lease expires. It has priority over the token from the other options.
func WithKubernetes(role, tokenFile string) Option {
if tokenFile == "" {
tokenFile = KubernetesTokenFile
}
return func(o *options) {
o.login = &login{mount: "kubernetes", data: func() (map[string]any, error) {
jwt, err := os.ReadFile(tokenFile)
if err != nil {
return nil, fmt.Errorf("failed to read the token of the service account: %w", err)
}
return map[string]any{"role": role, "jwt": strings.TrimSpace(string(jwt))}, nil
}}
}
}

// WithAuthMount sets the mount of the auth method of WithAppRole or WithKubernetes.
func WithAuthMount(mount string) Option {
return func(o *options) {
o.authMount = strings.Trim(mount, "/")
}
}

// WithNamespace sets the namespace of Vault Enterprise, by default it is taken from the VAULT_NAMESPACE variable.
func WithNamespace(namespace string) Option {
return func(o *options) {
Expand Down Expand Up @@ -121,6 +172,11 @@ func WithWatchInterval(interval time.Duration) Option {

type client struct {
options

// token of the login and the time when it should be requested again, zero if it does not expire
mu sync.Mutex
loginToken string
renewAt time.Time
}

func newClient(o options) *client {
Expand All @@ -144,11 +200,17 @@ func newClient(o options) *client {
if o.httpClient == nil {
o.httpClient = http.DefaultClient
}
if o.login != nil && o.authMount != "" {
o.login = &login{mount: o.authMount, data: o.login.data}
}
return &client{options: o}
}

// getToken returns the token, the file with the token is read every time.
func (c *client) getToken() (string, error) {
// getToken returns the token: the token of the login, the file with the token is read every time.
func (c *client) getToken(ctx context.Context) (string, error) {
if c.login != nil {
return c.loginTokenFor(ctx)
}
if c.tokenFile == "" {
return c.token, nil
}
Expand All @@ -159,6 +221,87 @@ func (c *client) getToken() (string, error) {
return strings.TrimSpace(string(data)), nil
}

// loginTokenFor returns the token of the login, the new token is requested after 2/3 of the lease of the previous one.
func (c *client) loginTokenFor(ctx context.Context) (string, error) {
c.mu.Lock()
defer c.mu.Unlock()
if c.loginToken != "" && (c.renewAt.IsZero() || time.Now().Before(c.renewAt)) {
return c.loginToken, nil
}
data, err := c.login.data()
if err != nil {
return "", err
}
// the data is the strings, so it is always encoded
body, _ := json.Marshal(data)
ctx, cancel := context.WithTimeout(ctx, requestTimeout)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.address+"/v1/auth/"+c.login.mount+"/login", bytes.NewReader(body))
if err != nil {
return "", err
}
resp, err := c.do(req)
if err != nil {
return "", fmt.Errorf("failed to login by auth/%s: %w", c.login.mount, err)
}
switch resp.status {
case http.StatusOK:
case http.StatusBadRequest, http.StatusForbidden:
// the wrong role or credentials
return "", fmt.Errorf("%w: failed to login by auth/%s: %s", tinyconf.ErrAccessDenied, c.login.mount, vaultErrors(resp.body))
default:
return "", fmt.Errorf("failed to login by auth/%s: %s: %s", c.login.mount, resp.statusText, vaultErrors(resp.body))
}
var auth struct {
Auth struct {
ClientToken string `json:"client_token"`
LeaseDuration int `json:"lease_duration"`
} `json:"auth"`
}
if err = json.Unmarshal(resp.body, &auth); err != nil || auth.Auth.ClientToken == "" {
return "", fmt.Errorf("failed to login by auth/%s: no token in the response", c.login.mount)
}
c.loginToken, c.renewAt = auth.Auth.ClientToken, time.Time{}
if lease := time.Duration(auth.Auth.LeaseDuration) * time.Second; lease > 0 {
c.renewAt = time.Now().Add(lease * 2 / 3)
}
return c.loginToken, nil
}

// resetLogin forgets the token of the login, the token is requested again by the next request:
// the token can be revoked before its lease expires.
func (c *client) resetLogin(token string) {
c.mu.Lock()
defer c.mu.Unlock()
if c.loginToken == token {
c.loginToken = ""
}
}

// response is the read response of Vault.
type response struct {
status int
statusText string
body []byte
}

// do sends the request with the namespace and reads the response.
func (c *client) do(req *http.Request) (*response, error) {
if c.namespace != "" {
req.Header.Set("X-Vault-Namespace", c.namespace)
}
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := io.ReadAll(io.LimitReader(resp.Body, 10<<20))
if err != nil {
return nil, err
}
return &response{status: resp.StatusCode, statusText: resp.Status, body: body}, nil
}

// secretURL returns the URL of the secret in the KV secrets engine.
func (c *client) secretURL(path string) string {
full := c.mount + "/data/" + path
Expand All @@ -168,43 +311,49 @@ func (c *client) secretURL(path string) string {
return c.address + "/v1/" + (&url.URL{Path: full}).EscapedPath()
}

// read returns the data of the secret and its version: the number of the version for KV version 2,
// the hash of the data for KV version 1. Absent secret is fs.ErrNotExist,
// the token without the permission to read the secret is tinyconf.ErrAccessDenied.
func (c *client) read(ctx context.Context, path string) (map[string]any, string, error) {
token, err := c.getToken()
// readBody requests the secret and returns the token of the request and the body of the response.
func (c *client) readBody(ctx context.Context, path string) (string, []byte, error) {
token, err := c.getToken(ctx)
if err != nil {
return nil, "", err
return "", nil, err
}
ctx, cancel := context.WithTimeout(ctx, requestTimeout)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.secretURL(path), nil)
if err != nil {
return nil, "", err
return token, nil, err
}
if token != "" {
req.Header.Set("X-Vault-Token", token)
}
if c.namespace != "" {
req.Header.Set("X-Vault-Namespace", c.namespace)
}
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, "", err
}
defer resp.Body.Close()
body, err := io.ReadAll(io.LimitReader(resp.Body, 10<<20))
resp, err := c.do(req)
if err != nil {
return nil, "", err
return token, nil, err
}
switch resp.StatusCode {
switch resp.status {
case http.StatusOK:
return token, resp.body, nil
case http.StatusNotFound:
return nil, "", fmt.Errorf("secret %s is not found: %w", path, fs.ErrNotExist)
return token, nil, fmt.Errorf("secret %s is not found: %w", path, fs.ErrNotExist)
case http.StatusForbidden:
return nil, "", fmt.Errorf("%w: failed to read secret %s: %s", tinyconf.ErrAccessDenied, path, vaultErrors(body))
return token, nil, fmt.Errorf("%w: failed to read secret %s: %s", tinyconf.ErrAccessDenied, path, vaultErrors(resp.body))
default:
return nil, "", fmt.Errorf("failed to read secret %s: %s: %s", path, resp.Status, vaultErrors(body))
return token, nil, fmt.Errorf("failed to read secret %s: %s: %s", path, resp.statusText, vaultErrors(resp.body))
}
}

// read returns the data of the secret and its version: the number of the version for KV version 2,
// the hash of the data for KV version 1. Absent secret is fs.ErrNotExist,
// the token without the permission to read the secret is tinyconf.ErrAccessDenied.
// The secret is read again with the new token of the login, if the token is denied.
func (c *client) read(ctx context.Context, path string) (map[string]any, string, error) {
token, body, err := c.readBody(ctx, path)
if c.login != nil && token != "" && errors.Is(err, tinyconf.ErrAccessDenied) {
c.resetLogin(token)
_, body, err = c.readBody(ctx, path)
}
if err != nil {
return nil, "", err
}
if c.kv1 {
var secret struct {
Expand Down
Loading
Loading