Repository navigation
Description
CLI crashes with UnicodeEncodeError on Windows when stdout is not a UTF-8 terminal
Summary
Every pyne command that prints a non-ASCII glyph through rich/typer crashes with
UnicodeEncodeError on Windows when the process's stdout encoding is a legacy code page
(e.g. cp1251). This happens with the console code page set to UTF-8 (chcp 65001) and with
no special environment variables set — i.e. out of the box for a non-English Windows install.
The CLI is effectively unusable on such systems unless the user manually sets PYTHONUTF8=1
(or PYNE_NO_LOGO=1 + NO_COLOR=1 to suppress the glyphs).
Environment
- OS: Windows 10 (10.0.19044), non-English system locale
- Python: 3.12.6
pynesys-pynecore: 6.13.5rich: 15.0.0typer: 0.29.1sys.stdout.encoding:cp1251(note: this stayscp1251even afterchcp 65001,
because stdout is a redirected pipe, so Python derives its encoding from
locale.getpreferredencoding(), not from the console code page)
Steps to reproduce
# fresh venv, nothing special in the environment
uv venv --python 3.12
uv pip install "pynesys-pynecore[cli]"
# a workdir with demo data
Set-Content .pyne 'workdir_version = "1.0"'
# any command that prints a unicode glyph through rich/typer
pyne --recreate-demo plugin listNo PYTHONUTF8, PYTHONIOENCODING, PYNE_NO_LOGO, or NO_COLOR are set.
Observed behavior
Crash 1 — pyne plugin list
The plugin tree uses the box-drawing characters └─ (└─):
Installed plugins:
ccxt CCXT v6.13.5 [provider]
+--------------------- Traceback (most recent call last) ---------------------+
| pynecore\cli\commands\plugin.py:112 in list_plugins |
| > 112 secho(f"{' ' * indent}└─ {summary}", dim=True) |
| typer\_click\termui.py:393 in secho |
| typer\_click\utils.py:275 in echo |
| C:\Python312\Lib\encodings\cp1251.py:19 in encode |
+-----------------------------------------------------------------------------+
UnicodeEncodeError: 'charmap' codec can't encode characters in position 13-14:
character maps to <undefined>
Crash 2 — pyne run
The loading spinner uses the check mark ✓ (✓):
+--------------------- Traceback (most recent call last) ---------------------+
| pynecore\cli\commands\run.py:1906 in run |
| > 1906 with Progress( |
| 1907 SpinnerColumn(finished_text="[green]✓"), |
| rich\progress.py:1176 in stop |
| rich\live.py:162 in stop |
| rich\console.py ... legacy_windows_render(buffer, LegacyWindowsTerm(...)) |
| rich\_win32_console.py:402 in write_text -> self.write(text) |
| C:\Python312\Lib\encodings\cp1251.py:19 in encode |
+-----------------------------------------------------------------------------+
UnicodeEncodeError: 'charmap' codec can't encode character '✓' in position
0: character maps to <undefined>
The ASCII-art Pyne banner also renders a stray replacement byte (Core�), confirming the
stream is being encoded as cp1251.
Expected behavior
pyne commands should run to completion on any Windows locale without the user having to set
environment variables. Non-encodable glyphs should either be emitted as UTF-8 or degrade
gracefully to ASCII.
Root cause
When stdout is not an interactive UTF-8 terminal, Python picks the stream encoding from the
system locale (cp1251 here), not from the console code page. Any glyph outside that code page
(└, ─, ✓, the banner characters) then raises UnicodeEncodeError at write time. The CLI
prints these glyphs unconditionally via rich/typer.
Suggested fixes (any one resolves it)
- Force UTF-8 on the CLI's own streams at startup, before any output. For example, early in
the CLI entry point:or configure theimport sys for stream in (sys.stdout, sys.stderr): try: stream.reconfigure(encoding="utf-8") # Python 3.7+ except Exception: pass
richConsole(file=...)with an explicit UTF-8 writer. - Make
rich's output ASCII-safe when the target encoding can't represent the glyphs —
e.g. passerrors="replace"/an ASCII-only legacy mode, or gate box chars and the✓behind
an encodability check. - Fall back to ASCII substitutes (
+--,|,v/OK) when
sys.stdout.encodingcan't encode the chosen glyphs.
Option 1 is the smallest and fixes every command at once.
Workarounds (for users, until fixed)
set PYTHONUTF8=1(forces the interpreter's std streams to UTF-8), orset PYNE_NO_LOGO=1andset NO_COLOR=1to suppress the banner and most styled glyphs
(does not cover the spinner✓inpyne run, soPYTHONUTF8=1is the reliable one).
Notes
- Reproduced with stdout redirected (a pipe), which is the common case when the CLI is driven by
another tool or captured. The same locale-based encoding selection applies.
Activity
fix(cli): write stdout and stderr as UTF-8 on every platform (#119)
wallneradam commentedon Nov 14, 2026
Fixed in ecb33be: the CLI now switches stdout and stderr to UTF-8 at startup when they use another encoding, keeping their error handlers. The fix will be in the next release (6.13.6).