Skip to content

gh-88336: Fix UnicodeDecodeError with multibyte utf8 characters in ShareableList - #26328

Merged
gpshead merged 4 commits into
python:mainfrom
junnplus:bpo44170-multibyte-fix
Nov 1, 2026
Merged

gh-88336: Fix UnicodeDecodeError with multibyte utf8 characters in ShareableList#26328
gpshead merged 4 commits into
python:mainfrom
junnplus:bpo44170-multibyte-fix

Conversation

@junnplus

@junnplus junnplus commented May 24, 2021 •
edited by serhiy-storchaka
Loading

Copy link
Copy Markdown
Contributor

This PR fix UnicodeDecodeError with multibyte utf8 characters in ShareableList.

>> from multiprocessing.shared_memory import ShareableList
>>> strings = ["Boom 💥 💥 💥"]
>>> ShareableList(strings)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/Jun/.pyenv/versions/3.9.0/lib/python3.9/multiprocessing/shared_memory.py", line 479, in __repr__
    return f'{self.__class__.__name__}({list(self)}, name={self.shm.name!r})'
  File "/Users/Jun/.pyenv/versions/3.9.0/lib/python3.9/multiprocessing/shared_memory.py", line 435, in __getitem__
    v = back_transform(v)
  File "/Users/Jun/.pyenv/versions/3.9.0/lib/python3.9/multiprocessing/shared_memory.py", line 277, in <lambda>
    1: lambda value: value.rstrip(b'\x00').decode(_encoding),  # str
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 15: unexpected end of data

https://bugs.python.org/issue44170

@junnplus
junnplus requested a review from tiran as a code owner May 24, 2021 11:07
@the-knights-who-say-ni

the-knights-who-say-ni commented May 24, 2021

Copy link
Copy Markdown

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

Recognized GitHub username

We couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames:

@junnplus

This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

junnplus
junnplus commented May 24, 2021
View reviewed changes
Comment thread Lib/multiprocessing/shared_memory.py Outdated
if isinstance(value, str) else value)
if len(encoded_value) > allocated_length:
encoded_value = self._encode_value(value)
if len(encoded_value) >= allocated_length:

@junnplus junnplus May 24, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

>>> from multiprocessing.shared_memory import ShareableList
>>> s1 = ShareableList(['1234567'])
>>> s1.format
'8s'
>>> s2 = ShareableList(['12345678'])
>>> s2.format
'16s'
>>> s1[0] = '12345678'  # Is this behavior expected?
>>> s1.format
'8s'
>>> s3 = ShareableList(s1)
>>> s3.format
'16s'

@github-actions

github-actions Bot commented Jun 27, 2021

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Jun 27, 2021
@tcl326 tcl326 mentioned this pull request Feb 21, 2022
Open
@github-actions github-actions Bot removed the stale Stale PR or inactive for long period of time. label Aug 9, 2022
@bedevere-app bedevere-app Bot mentioned this pull request Aug 19, 2024
Closed
@github-actions

github-actions Bot commented Apr 9, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 9, 2026
@github-actions github-actions Bot removed the stale Stale PR or inactive for long period of time. label Aug 11, 2026
@serhiy-storchaka serhiy-storchaka changed the title bpo-44170: Fix UnicodeDecodeError with multibyte utf8 characters in ShareableList gh-88336: Fix UnicodeDecodeError with multibyte utf8 characters in ShareableList Aug 12, 2026
@serhiy-storchaka @claude
Round up the allocated size instead of rejecting the exact fit
9c8d18d 
Allocating one extra alignment block for a value whose encoded size is
a multiple of the alignment made a copy of the list wider than the
original.  Round the size up instead, so that it is the same for the
same value, and keep allowing a value which fills the slot exactly:
rejecting it broke setting an 8-byte string in a slot of an integer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
serhiy-storchaka
serhiy-storchaka approved these changes Aug 14, 2026
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed two off-by-one errors (introduced in the PR and preexisting) and missing NEW's final newline. LGTM now.

@serhiy-storchaka serhiy-storchaka added needs backport to 3.15 bugs and security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.13 only security fixes labels Aug 14, 2026
@gpshead gpshead removed the needs backport to 3.13 only security fixes label Nov 1, 2026
@gpshead
gpshead merged commit 9348e13 into python:main Nov 1, 2026
63 checks passed
@miss-islington-app

miss-islington-app Bot commented Nov 1, 2026

Copy link
Copy Markdown

Thanks @junnplus for the PR, and @gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14, 3.15.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

@bedevere-app

bedevere-app Bot commented Nov 1, 2026

Copy link
Copy Markdown

GH-161116 is a backport of this pull request to the 3.15 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Nov 1, 2026
@bedevere-app

bedevere-app Bot commented Nov 1, 2026

Copy link
Copy Markdown

GH-161117 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Nov 1, 2026
gpshead added a commit that referenced this pull request Nov 1, 2026
@gpshead @junnplus
[3.14] gh-88336: Fix UnicodeDecodeError with multibyte utf8 character…
5f92e38 
…s in ShareableList (GH-26328) (#161117)

* bpo-44170: Fix UnicodeDecodeError with multibyte utf8 characters in ShareableList
* Round up the allocated size instead of rejecting the exact fit

Allocating one extra alignment block for a value whose encoded size is
a multiple of the alignment made a copy of the list wider than the
original.  Round the size up instead, so that it is the same for the
same value, and keep allowing a value which fills the slot exactly:
rejecting it broke setting an 8-byte string in a slot of an integer.

---------



(cherry picked from commit 9348e13)

Co-authored-by: Jun <junnplus@gmail.com>
gpshead added a commit that referenced this pull request Nov 1, 2026
@gpshead @junnplus
[3.15] gh-88336: Fix UnicodeDecodeError with multibyte utf8 character…
ae13f1b 
…s in ShareableList (GH-26328) (#161116)

* bpo-44170: Fix UnicodeDecodeError with multibyte utf8 characters in ShareableList
* Round up the allocated size instead of rejecting the exact fit

Allocating one extra alignment block for a value whose encoded size is
a multiple of the alignment made a copy of the list wider than the
original.  Round the size up instead, so that it is the same for the
same value, and keep allowing a value which fills the slot exactly:
rejecting it broke setting an 8-byte string in a slot of an integer.

---------



(cherry picked from commit 9348e13)

Co-authored-by: Jun <junnplus@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead approved these changes
@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes
@tiran tiran Awaiting requested review from tiran

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@junnplus @the-knights-who-say-ni @gpshead @serhiy-storchaka @ezio-melotti @bedevere-bot