Repository navigation
Incomplete message body received when streaming response #303
Description
(This appears to be related to #201, where this exception was added)
I've got some code (below) running in Python 3.8.3 (as packaged in RHEL 8), using httpx 0.17.1. I'm seeing random cases where an exception is thrown claiming that the connection was closed before the complete message body was sent, but I suspect the actual situation is that the connection was closed by the server after it had sent the entire body but before the entire body had been read from the response object.
Code:
async def blob_getter(client: httpx.AsyncClient, blob_dir: pathlib.Path, blob: BlobDetails) -> None:
blob_path = blob_dir / blob.digest
if blob_path.exists():
logger.debug("%s blob %s already present", blob.purpose, blob.name)
return
logger.debug("Getting %s blob %s", blob.purpose, blob.name)
try:
blob_url = blob.job.name + "/blobs/" + blob.digest
async with client.stream("GET", blob_url) as response:
if response.is_error:
logger.error(
"Failed %s blob %s - %s - %s",
blob.purpose,
blob.name,
response.status_code,
response.reason_phrase,
)
return
async with aiofile.async_open(blob_path, "wb") as blob_file:
async for chunk in response.aiter_bytes():
await blob_file.write(chunk)
logger.debug("Completed %s blob %s", blob.purpose, blob.name)
except asyncio.CancelledError:
blob_path.unlink(missing_ok=True)
logger.debug("Cancelled %s blob %s", blob.purpose, blob.name)
except Exception as exc:
blob_path.unlink(missing_ok=True)
logger.error("Failed %s blob %s - %s", blob.purpose, blob.name, str(exc))
raise
I'm seeing exceptions caught by the except Exception clause, and I'm 100% positive that the server did send the full body, but the socket-closure was noticed before the response had been fully ready.
For now, I'm going to catch this particular exception and 'absorb' it so that I can continue reading, but I'm wondering if there's something else I should be doing.
Activity
lovelydinosaur commentedon Apr 19, 2021
Would you be able to reduce this down to an easily reproducible stand-alone example?