Improve support for emojis+cursor w/grapheme clustering #2045
+749
β83
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jquast
changed the title
jquast
force-pushed
the
jq/grapheme-clustering-support
branch
3 times, most recently
from
a244ba5 to
918a8eb
Compare
jquast
changed the title
jquast
changed the title
jquast
changed the title
jquast
force-pushed
the
jq/grapheme-clustering-support
branch
2 times, most recently
from
95e89b0 to
953cfe5
Compare
jquast
commented
Jan 27, 2026
|
|
||
| This offers code coverage without mocks, but using real tty features of the natural "live" call to | ||
| PromptSession() and session.prompt. prompt_toolkit sees a real terminal through use of shared | ||
| pty_accessories module. |
Contributor
Author
There was a problem hiding this comment.
This might show some improved test coverage for this reason
feel free to extend this test to get those previously unreachable lines
| for target in [(40, 120), (10, 40), (3, 10)]: | ||
| _setwinsize(fd, *target) | ||
| time.sleep(0.05) | ||
| assert _get_size(fd) == target |
Contributor
Author
There was a problem hiding this comment.
A more advance version of this test could help verify screen draw reactions and effects of resize on SIGWINCH signal, for example.
| time.sleep(0.2) | ||
| assert ( | ||
| extract_output(read_until_marker(fd, ":END", timeout=10.0)) == grapheme * 5 | ||
| ) |
Contributor
Author
There was a problem hiding this comment.
These tests are simplified but provide pty coverage, the highest level possible exercise, which is nice because it is also necessary to test these changes.
More improvement could be done to also verify screen draw effects of the REPL, these only verify output after sending return key, where repl integration print()'s result.
jquast
force-pushed
the
jq/grapheme-clustering-support
branch
2 times, most recently
from
4d6d2ac to
8aeef93
Compare
**Problem** Test sequence (copy and paste into any REPL/edit area)::
π¨βπ©βπ§ π©ββ€βπ¨ π©βπ»ππΏ β€οΈβ π―π΅π©πͺ cafeΜ ninΜo AΜoΜΜ£ δΈζ!.
Moving the cursor over and around emojis get strange. insertions become
chaotic. Cursor position becomes indeterminate (even negative!), input
result becomes more corrupted with user confusion as draws become
corrupted. This is briefly described in prompt-toolkit#274 by @jonathanslenders:
> Notice that it still requires multiple cursor movements (left/right
arrow) to move across these characters.
**Solution**: Close prompt-toolkit#274 "Handle decomposed unicode characters" (2018)
through careful integration of new functions,
[wcwidth.iter_graphemes](https://wcwidth.readthedocs.io/en/latest/intro.html#iter-graphemes)
and
[wcwidth.grapheme_boundary_before](https://wcwidth.readthedocs.io/en/latest/api.html#wcwidth.grapheme_boundary_before).
getting there, working on a PTY test suite
I don't feel comfortable changing so much code for a large library
without also including more detailed tests -- i keep fixing all errors
with TDD/automatic tests, but when using it interactively, the cursor
position is out of control
jquast
force-pushed
the
jq/grapheme-clustering-support
branch
from
8aeef93 to
aed10aa
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Copy and paste this test sequence to call
prompt_toolkit.prompt('Give me some input: '):Moving the cursor over and around emojis get strange. insertions become chaotic. Cursor position becomes indeterminate (even negative!), input result becomes more corrupted with user confusion as draws become corrupted. This is briefly described in #274 by @jonathanslenders:
wcwidth-integration-before-trimmed.mp4
Solution
Close #274 "Handle decomposed unicode characters" (2018) through careful integration of new functions, wcwidth.iter_graphemes and wcwidth.grapheme_boundary_before.
wcwidth-integration-after.mp4
Details