`borg2`: Support `env` section in YAML config files

[PhrozenByte]

One of the goals of borg2 has been to reduce the number of dedicated CLI options and instead rely more on env variables. For example, --remote-path and --rsh are about to be removed in favor of BORG_REMOTE_PATH and BORG_RSH (#9952).

While this reduces the CLI surface, it also spreads config across multiple places: some settings are configured via CLI options, while others are configured exclusively via dedicated env variables.

With borg2, jsonargparse was introduced as a new dependency, making it possible to provide CLI options via alternative means:

• via env variables (although the generated names can be somewhat cumbersome, e.g. BORG_CREATE__COMMENT for borg2 create --comment)
• via YAML config files, either specified with --config or loaded from ~/.config/borg/default.yaml by default

Overall, this makes config feel somewhat fragmented. While all settings can now be expressed via env variables, they are not a good fit for persistent and per-job configuration. Config files are the natural alternative. However, the config files currently cannot express settings that are available exclusively via dedicated BORG_* env variables. Allowing config files to define those env variables would make them a complete and consistent alternative for persistent and per-job Borg configuration.

Proposal

Support an optional top-level env section in the YAML config file.

For example:

env:
  BORG_HOSTNAME: my-host
  BORG_RSH: ssh -i /path/to/private/key
  BORG_FILES_CACHE_TTL: 30

log_level: info
show_rc: true

create:
  stats: true
  compression: zstd,6

The values defined in env would be injected into Borg's environment as part of normal argument processing (unless already explicitly set in the calling environment), making them practically behave as if they had been set in the shell.

This would allow users to keep all Borg config in a single declarative file.

Motivation

The YAML config support provided by jsonargparse is a valuable addition to borg2 and has the potential to become a common way of managing Borg configs.

While jsonargparse also makes it possible to provide CLI options via env variables, env variables are not an ideal replacement for persistent application config. They are inherently global to a shell. Users may not want to populate their shell environment with application-specific variables, and setting them only at runtime typically requires wrapper scripts, shell aliases, systemd unit files, cron jobs, or similar tooling. More importantly, Borg is commonly used with multiple independent backup jobs, each requiring different settings, but env variables cannot easily express multiple independent configurations without external scaffolding.

Config files naturally solve these problems. They are easy to version, can be selected on a per-invocation basis via --config, and allow multiple independent backup configurations to coexist cleanly. To fully support this use case, config files should also be able to define settings that are currently only accessible via dedicated env variables.

Supporting an env section would:

• keep config centralized,
• allow complete backup job definitions to live in a single declarative file,
• avoid the need for wrapper scripts whose only purpose is to set env variables,
• make multiple independent backup configurations straightforward to maintain,
• provide a more consistent config experience,
• naturally complement the existing YAML config mechanism.

It would also align well with the goal of reducing CLI-specific options without making config files less expressive.

[ThomasWaldmann]

One of the goals of borg2 has been to reduce the number of dedicated CLI options ...

Not in general, but in case of common options!

The reasoning behind that is:

• common options blow up the docs, they need to get mentioned in the help of every command
• and, especially for rare-usage (but global) options, that is annoying
• sometimes such an option does not even make sense for all commands, just for some/many

Example (borg 1.4.5):

% borg --help
usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] [--debug-topic TOPIC] [-p] [--iec] [--log-json] [--lock-wait SECONDS] [--bypass-lock] [--show-version]
            [--show-rc] [--umask M] [--remote-path PATH] [--remote-ratelimit RATE] [--upload-ratelimit RATE] [--remote-buffer UPLOAD_BUFFER] [--upload-buffer UPLOAD_BUFFER]
            [--consider-part-files] [--debug-profile FILE] [--rsh RSH]
            <command> ...

Borg - Deduplicated Backups

options:
  -V, --version         show version number and exit

Common options:
  -h, --help            show this help message and exit
  --critical            work on log level CRITICAL
  --error               work on log level ERROR
  --warning             work on log level WARNING (default)
  --info, -v, --verbose
                        work on log level INFO
  --debug               enable debug output, work on log level DEBUG
  --debug-topic TOPIC   enable TOPIC debugging (can be specified multiple times). The logger path is borg.debug.<TOPIC> if TOPIC is not fully qualified.
  -p, --progress        show progress information
  --iec                 format using IEC units (1KiB = 1024B)
  --log-json            Output one JSON object per log line instead of formatted text.
  --lock-wait SECONDS   wait at most SECONDS for acquiring a repository/cache lock (default: 1).
  --bypass-lock         Bypass locking mechanism
  --show-version        show/log the borg version
  --show-rc             show/log the return code (rc)
  --umask M             set umask to M (local only, default: 0077)
  --remote-path PATH    use PATH as borg executable on the remote (default: "borg")
  --remote-ratelimit RATE
                        deprecated, use ``--upload-ratelimit`` instead
  --upload-ratelimit RATE
                        set network upload rate limit in kiByte/s (default: 0=unlimited)
  --remote-buffer UPLOAD_BUFFER
                        deprecated, use ``--upload-buffer`` instead
  --upload-buffer UPLOAD_BUFFER
                        set network upload buffer size in MiB. (default: 0=no buffer)
  --consider-part-files
                        treat part files like normal files (e.g. to list/extract them)
  --debug-profile FILE  Write execution profile in Borg format into FILE. For local use a Python-compatible file can be generated by suffixing FILE with ".pyprof".
  --rsh RSH             Use this command to connect to the 'borg serve' process (default: 'ssh')

required arguments:
...

Note that huge "common options" block. Some of course useful and frequently used (like -v or -p), but a lot very special and rarely used stuff also:

  --lock-wait SECONDS   wait at most SECONDS for acquiring a repository/cache lock (default: 1).
  --bypass-lock         Bypass locking mechanism
  --umask M             set umask to M (local only, default: 0077)
  --remote-path PATH    use PATH as borg executable on the remote (default: "borg")
  --remote-ratelimit RATE
                        deprecated, use ``--upload-ratelimit`` instead
  --upload-ratelimit RATE
                        set network upload rate limit in kiByte/s (default: 0=unlimited)
  --remote-buffer UPLOAD_BUFFER
                        deprecated, use ``--upload-buffer`` instead
  --upload-buffer UPLOAD_BUFFER
                        set network upload buffer size in MiB. (default: 0=no buffer)
  --consider-part-files
                        treat part files like normal files (e.g. to list/extract them)
  --debug-profile FILE  Write execution profile in Borg format into FILE. For local use a Python-compatible file can be generated by suffixing FILE with ".pyprof".
  --rsh RSH             Use this command to connect to the 'borg serve' process (default: 'ssh')

Also, even the "usage" display is bloated:

usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] [--debug-topic TOPIC] [-p] [--iec] [--log-json] [--lock-wait SECONDS] [--bypass-lock] [--show-version]
            [--show-rc] [--umask M] [--remote-path PATH] [--remote-ratelimit RATE] [--upload-ratelimit RATE] [--remote-buffer UPLOAD_BUFFER] [--upload-buffer UPLOAD_BUFFER]
            [--consider-part-files] [--debug-profile FILE] [--rsh RSH]
            <command> ...

Most of that stuff is not that interesting for normal usage.

[ThomasWaldmann]

I looked into whether an env section is something jsonargparse gives us, or something we'd be bolting on. Short answer: neither, exactly. jsonargparse has no env section feature — but the proposal doesn't need one. It can be built as a completely ordinary jsonargparse option; the dirty part isn't the parser, it's when borg reads its env vars.

jsonargparse's env support goes one way only

default_env=True / env_prefix="BORG" (src/borg/archiver/__init__.py:266) makes jsonargparse read BORG_* env vars and map them onto options. It never writes to os.environ — grepping the installed 4.49.0 package, the only os.environ[...] assignments are shell-completion internals. There is no concept of a config section that exports variables.

An undeclared env: key in the YAML is simply rejected:

ArgumentError: Option 'env.BORG_RSH' is not accepted before subcommand {create,...}

But the clean implementation is trivial

Declare a real top-level argument and the env: section becomes a plain, validated option value:

parser.add_argument("--env", type=dict[str, str], default={})

I tested this against a parser configured like borg's — the env: block from the issue parses fine, and you get --env '{"A":"1"}' on the CLI plus BORG_ENV='{...}' for free. The only borg-specific code is a few lines of os.environ.setdefault(k, v) after parse_args(). That's application glue, not a parser hack, and setdefault matches jsonargparse's own precedence (real env beats config).

Note that dicts replace rather than merge across sources, so --env on the CLI would discard the config's whole env: block. Worth deciding deliberately.

Where it actually gets dirty: injection happens too late

Three classes of BORG_* vars would silently no-op, including some of the most likely candidates:

• Read at import time — BORG_WORKAROUNDS (src/borg/helpers/__init__.py:70), BORG_EXIT_CODES (src/borg/helpers/errors.py:8), BORG_FUSE_IMPL (src/borg/fuse_impl.py:10), BORG_HOST_ID (src/borg/platform/base.py:328). Unreachable from a config file at any injection point, short of making those reads lazy.
• Read at parser-build time — BORG_LOCK_WAIT is baked into default= at src/borg/archiver/_common.py:535, i.e. before the config is even loaded.
• Consumed by jsonargparse itself — anything of the BORG_CREATE__COMMENT shape is applied during parse_args(), so post-parse injection is invisible to it. (Harmless in practice, since those options can be set directly in the YAML, but it's a confusing asymmetry to document.)

Chicken-and-egg: BORG_CONFIG_DIR / BORG_BASE_DIR / XDG_CONFIG_HOME determine where default.yaml is looked up (src/borg/archiver/__init__.py:266), so they can never come from it.

The example in the issue is fine — BORG_HOSTNAME, BORG_RSH, BORG_FILES_CACHE_TTL are all read late — which is exactly why the failure mode is nasty: it works for the vars in the docs and quietly does nothing for the neighbouring ones.

We could fix the second and third classes by peeking at the config with yaml.safe_load before building the parser, at the cost of duplicating the --config discovery logic and parsing the file twice. Import-time reads would need those globals made lazy regardless.

Two things worth gating

• BORG_PASSPHRASE in a config file puts a passphrase in a plaintext file that users will version-control. BORG_PASSCOMMAND-style indirection is the better story.
• BORG_*_I_KNOW_WHAT_I_AM_DOING / BORG_CHECK_I_KNOW_WHAT_I_AM_DOING in a persistent config permanently pre-answers destructive confirmations.

An explicit allow-list (or at minimum a deny-list plus a documented "these are ignored because they're read too early" list) would turn this from a footgun into a real feature.

[PhrozenByte]

One of the goals of borg2 has been to reduce the number of dedicated CLI options ...

Not in general, but in case of common options!

The reasoning behind that is:

• common options blow up the docs, they need to get mentioned in the help of every command
• and, especially for rare-usage (but global) options, that is annoying
• sometimes such an option does not even make sense for all commands, just for some/many

Thanks for explaining the motivation in more detail. I think the goal itself makes a lot of sense.

However, I had actually assumed a somewhat different motivation behind moving some CLI options to dedicated env variables. With that motivation in mind, I'd like to take one step back and revisit whether env variables are the best mechanism for achieving that goal.

As I mentioned in the original proposal, I think env variables have a few drawbacks when used as persistent application config. My proposal was an attempt to mitigate those drawbacks by making env variables available through config files as well. As you rightly pointed out in your follow-up regarding the implementation details, that approach would also introduce a fair amount of application-specific glue and complexity. If that's the best way forward, I think that complexity would be justified.

However, the more I think about it, the more I wonder whether we're solving a symptom rather than the underlying problem. Looking at the bigger picture might therefore be worthwhile. One thing that struck me is somewhat of a paradox that would not be solved by my proposal either: CLI options are self-discoverable, as users naturally first, and often only, look at --help. Dedicated env variables, however, are not, even though they are equally part of the shell interface. They are typically only discovered by consulting the online docs or the man pages (which might not always be available). As a result, many projects eventually end up listing their supported env variables in the --help output as well, which largely defeats the original purpose.

If I understand correctly, the goal isn't really to reduce the number of configurable settings. Those settings still exist. Rather, the goal is to reduce the amount of rarely used information exposed in the default CLI help by moving it somewhere else. Is that a fair summary?

If so, perhaps it would be worth considering whether that docs problem (with --help in particular) could be solved more directly, without changing the config mechanism itself.

One thing I particularly like about the current jsonargparse integration is that it provides a single option model with multiple frontends: the same option can be supplied on the command line, via YAML, or via the generated env variable support. Introducing (more) dedicated env variables outside that model creates a second, partially overlapping config mechanism with different capabilities and different discoverability. That feels like additional conceptual complexity.

One possible approach might be to hide selected options from the default help output (argparse supports this via help=argparse.SUPPRESS) while keeping them as regular parser options. Those options could still be documented separately (e.g. via borg help, --help-all, or similar), but they would continue to benefit from everything jsonargparse already provides automatically.

If that alone doesn't provide the desired result, there are more ways to streamline the help output without changing the config model. For example, deprecated options could be removed, some options could be consolidated, developer-only options could be documented separately, or the remaining common options could be grouped more narrowly by topic instead of appearing in a single large block.

Regarding the usage: synopsis, it's worth noting that this is largely a consequence of the automatically generated argparse output. Many traditional UNIX tools intentionally keep the synopsis concise rather than enumerating every available option. I'd therefore consider overriding the automatically generated synopsis with a manually defined one if that better reflects the intended user experience.

To summarize, I'm wondering whether the underlying issue is really "we have too many CLI options", or rather "we expose too many rarely used options in the default help output". Those seem like related, but distinct, problems. Solving the latter directly might allow us to keep the config model simpler and more consistent. What do you think?

[ThomasWaldmann]

Reasoning: I focussed primarily on users not using these rare common options.

For the users who need them, there is another problem:

Quite some of them (e.g. --rsh or --remote-path) would need to be given to all borg commands - that is also annoying for manual use as well as for script use (duplication), so that is also implying using env vars only.

Discoverability:

• Guess we can't avoid that completely. Either it's there and annoying or it's not (fully) there.
• But we can have borg help environment or so, like we have for some other stuff already.

[PhrozenByte]

Quite some of them (e.g. --rsh or --remote-path) would need to be given to all borg commands - that is also annoying for manual use as well as for script use (duplication), so that is also implying using env vars only.

I agree that avoiding repetition is a worthwhile goal. To me, though, that seems orthogonal to whether a setting should remain part of the parser's option model. What I'm not yet convinced about is whether that argues for making env vars the only config mechanism available.

I think the key distinction is between the option model and the config frontends. jsonargparse already gives us one option model with multiple frontends (CLI arguments, YAML config, generated env vars). Avoiding repetition doesn't require removing one of those frontends, because they all configure the same underlying option.

In fact, I'd be inclined to steer users more explicitly towards the generated env vars for this class of rarely used options. If they were documented separately rather than in the default --help output, there would also be room to explain when the env vars are the better choice and why. That encourages the preferred workflow without removing the alternative.

Where I see a difference is for users who don't want to or can't rely on env vars. My own setup is an example: I back up my machine to two different remotes (a local NAS and a cloud server) from the same script, each requiring different SSH settings. Today I simply pass --rsh where needed. With env-var-only config I'd instead have to write BORG_RSH="" borg2 …, or manipulate the env around each call. I'm not sure that's really an improvement over borg2 --rsh "" …. This is exactly the kind of "persistent per-job config" I mentioned in my initial proposal, and why I don't think env vars are a good fit as the only config mechanism. This is not specific to --rsh; it is a general limitation of env vars.

More generally, I think env vars are a useful config mechanism and I understand why they help avoid repetition. What I'm struggling to see is why that requires removing the corresponding CLI options. Keeping both mechanisms doesn't make the env-var workflow any less convenient for users who prefer it. Removing the CLI options, however, removes a workflow that some users rely on.

It also seems that this is what makes proposals like my original env: section necessary in the first place. Looking back at it, I now think that proposal was essentially a workaround for restoring a config mechanism that Borg already had naturally through jsonargparse. If the underlying parser options remained available, there would be no need for that additional mechanism at all. I would therefore suggest relying more on jsonargparse, not less.

If the main goal is to avoid overwhelming users with rarely used options in the default help output, perhaps that could be solved directly at the --help / docs level instead. borg help environment sounds like a reasonable place for env vars; similarly, less common CLI options could be documented outside the default --help (e.g. via borg help or a similar mechanism) while remaining part of the same parser model. That would preserve the unified config model provided by jsonargparse without bringing back the clutter in the default help.