fix: honor encoding= in partition_json and partition_ndjson

[linhongyu510]

Problem

partition_json() and partition_ndjson() read the source with a hard-coded
UTF-8, and neither accepts an encoding parameter:

if filename is not None:
    with open(filename, encoding="utf8") as f:      # hard-coded
        file_text = f.read()

elif file is not None:
    file_content = file.read()
    file_text = file_content if isinstance(file_content, str) else file_content.decode()  # UTF-8

Because both signatures end in **kwargs: Any, an encoding= passed by the
caller — directly, or forwarded by partition() — is swallowed silently. On
main (ee2b3a35):

>>> partition_json(filename="utf16.json", encoding="utf-16")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

The same call succeeds for the sibling text formats, which is where the
inconsistency shows:

partitioner	takes encoding	reads via
partition_text	yes	read_txt_file()
partition_csv	yes	passed to the reader
partition_json	no	open(..., encoding="utf8")
partition_ndjson	no	open(..., encoding="utf8")

UTF-16 JSON is not exotic — it is what several Windows tools emit by default,
and json.dump to a file opened in a non-UTF-8 encoding produces it too.

Fix

Add encoding to both signatures and read through the existing
read_txt_file() helper, exactly as partition_text() does:

if filename is not None:
    _, file_text = read_txt_file(filename=filename, encoding=encoding)

elif file is not None:
    _, file_text = read_txt_file(file=file, encoding=encoding)
    file.seek(0)

read_txt_file() uses the given encoding when one is supplied and falls back to
detect_file_encoding() when it is not, so this also makes a non-UTF-8 document
readable without naming its encoding. The text= path is untouched, and
file.seek(0) is preserved so the detect-then-partition sequence over one handle
still works.

Source change is +11/-8 across the two files; no new helper, no behaviour change
for UTF-8 input.

Verification

$ pytest test_unstructured/partition/test_json.py test_unstructured/partition/test_ndjson.py
35 failed, 75 passed

The 35 failures are pre-existing on main in this environment — they come from
unstructured/nlp/tokenize.py failing to import an optional dependency
(ModuleNotFoundError: No module named 'installer'), unrelated to these files.
I compared the failure sets with git stash: identical before and after, and
the pass count goes 70 → 75, which is exactly the five tests added here.

Five regression tests, following the existing naming style in both files:

• explicit encoding="utf-16" for filename= (both partitioners)
• explicit encoding="utf-16" for file= (both partitioners)
• no encoding given, UTF-16 detected (json)

Rollback proof. Restoring just the two read blocks to the hard-coded form —
leaving the new parameter in place — fails exactly those five and nothing else:

FAILED test_json.py::it_honors_an_explicit_encoding_for_a_non_utf8_file
FAILED test_json.py::it_honors_an_explicit_encoding_for_a_non_utf8_file_like_object
FAILED test_json.py::it_detects_the_encoding_of_a_non_utf8_file_when_none_is_specified
FAILED test_ndjson.py::it_honors_an_explicit_encoding_for_a_non_utf8_file
FAILED test_ndjson.py::it_honors_an_explicit_encoding_for_a_non_utf8_file_like_object
5 failed, 105 deselected

each with UnicodeDecodeError. ruff check on the four changed files is clean,
and a CHANGELOG entry is added under the current 0.27.8-dev0 section.

Notes

This is the parsing side. There are open PRs (#4397, #4398, #4447) addressing
non-UTF-8 crashes during file-type detection in file_utils/filetype.py; I
checked all currently open PRs and none touches partition/json.py or
partition/ndjson.py, so this does not overlap with them.

I kept the scope to honoring the parameter. I did not change what happens when a
document really is undecodable, nor touch the text= path.