Skip to content

CLI crashes with UnicodeEncodeError on Windows when stdout is not a UTF-8 terminal #119

Closed

Description

@tripolskypetr

CLI crashes with UnicodeEncodeError on Windows when stdout is not a UTF-8 terminal

Summary

Every pyne command that prints a non-ASCII glyph through rich/typer crashes with
UnicodeEncodeError on Windows when the process's stdout encoding is a legacy code page
(e.g. cp1251). This happens with the console code page set to UTF-8 (chcp 65001) and with
no special environment variables set — i.e. out of the box for a non-English Windows install.
The CLI is effectively unusable on such systems unless the user manually sets PYTHONUTF8=1
(or PYNE_NO_LOGO=1 + NO_COLOR=1 to suppress the glyphs).

Environment

  • OS: Windows 10 (10.0.19044), non-English system locale
  • Python: 3.12.6
  • pynesys-pynecore: 6.13.5
  • rich: 15.0.0
  • typer: 0.29.1
  • sys.stdout.encoding: cp1251 (note: this stays cp1251 even after chcp 65001,
    because stdout is a redirected pipe, so Python derives its encoding from
    locale.getpreferredencoding(), not from the console code page)

Steps to reproduce

# fresh venv, nothing special in the environment
uv venv --python 3.12
uv pip install "pynesys-pynecore[cli]"

# a workdir with demo data
Set-Content .pyne 'workdir_version = "1.0"'

# any command that prints a unicode glyph through rich/typer
pyne --recreate-demo plugin list

No PYTHONUTF8, PYTHONIOENCODING, PYNE_NO_LOGO, or NO_COLOR are set.

Observed behavior

Crash 1 — pyne plugin list

The plugin tree uses the box-drawing characters └─ (└─):

  Installed plugins:
    ccxt     CCXT     v6.13.5   [provider]
+--------------------- Traceback (most recent call last) ---------------------+
| pynecore\cli\commands\plugin.py:112 in list_plugins                         |
| > 112   secho(f"{' ' * indent}└─ {summary}", dim=True)            |
| typer\_click\termui.py:393 in secho                                         |
| typer\_click\utils.py:275 in echo                                           |
| C:\Python312\Lib\encodings\cp1251.py:19 in encode                           |
+-----------------------------------------------------------------------------+
UnicodeEncodeError: 'charmap' codec can't encode characters in position 13-14:
character maps to <undefined>

Crash 2 — pyne run

The loading spinner uses the check mark ✓ (✓):

+--------------------- Traceback (most recent call last) ---------------------+
| pynecore\cli\commands\run.py:1906 in run                                    |
| > 1906  with Progress(                                                       |
|   1907      SpinnerColumn(finished_text="[green]✓"),                    |
| rich\progress.py:1176 in stop                                               |
| rich\live.py:162 in stop                                                    |
| rich\console.py ... legacy_windows_render(buffer, LegacyWindowsTerm(...))   |
| rich\_win32_console.py:402 in write_text -> self.write(text)                |
| C:\Python312\Lib\encodings\cp1251.py:19 in encode                           |
+-----------------------------------------------------------------------------+
UnicodeEncodeError: 'charmap' codec can't encode character '✓' in position
0: character maps to <undefined>

The ASCII-art Pyne banner also renders a stray replacement byte (Core�), confirming the
stream is being encoded as cp1251.

Expected behavior

pyne commands should run to completion on any Windows locale without the user having to set
environment variables. Non-encodable glyphs should either be emitted as UTF-8 or degrade
gracefully to ASCII.

Root cause

When stdout is not an interactive UTF-8 terminal, Python picks the stream encoding from the
system locale (cp1251 here), not from the console code page. Any glyph outside that code page
(└, ─, ✓, the banner characters) then raises UnicodeEncodeError at write time. The CLI
prints these glyphs unconditionally via rich/typer.

Suggested fixes (any one resolves it)

  1. Force UTF-8 on the CLI's own streams at startup, before any output. For example, early in
    the CLI entry point:
    import sys
    for stream in (sys.stdout, sys.stderr):
        try:
            stream.reconfigure(encoding="utf-8")   # Python 3.7+
        except Exception:
            pass
    or configure the rich Console(file=...) with an explicit UTF-8 writer.
  2. Make rich's output ASCII-safe when the target encoding can't represent the glyphs —
    e.g. pass errors="replace"/an ASCII-only legacy mode, or gate box chars and the ✓ behind
    an encodability check.
  3. Fall back to ASCII substitutes (+--, |, v/OK) when
    sys.stdout.encoding can't encode the chosen glyphs.

Option 1 is the smallest and fixes every command at once.

Workarounds (for users, until fixed)

  • set PYTHONUTF8=1 (forces the interpreter's std streams to UTF-8), or
  • set PYNE_NO_LOGO=1 and set NO_COLOR=1 to suppress the banner and most styled glyphs
    (does not cover the spinner ✓ in pyne run, so PYTHONUTF8=1 is the reliable one).

Notes

  • Reproduced with stdout redirected (a pipe), which is the common case when the CLI is driven by
    another tool or captured. The same locale-based encoding selection applies.

Activity

  1. wallneradam commented on Nov 14, 2026

    @wallneradam
    Contributor

    Fixed in ecb33be: the CLI now switches stdout and stderr to UTF-8 at startup when they use another encoding, keeping their error handlers. The fix will be in the next release (6.13.6).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions