Skip to content

Fix memory retained by error responses - #355

Open
theseal wants to merge 1 commit into
masterfrom
jocar-283
Open

theseal wants to merge 1 commit into
masterfrom
jocar-283

Conversation

@theseal

@theseal theseal commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Pyramid keeps an error response's exception on the request. Its traceback reached the frame holding the pipeline result, so each failed request kept the selected metadata tree alive until a full garbage collection.

Run the plumbings in their own function, re-raise HTTP errors without their traceback, and raise 409/500 outside the except clauses. HTTP errors such as 406 from content negotiation are no longer turned into 500.

Closes #283.

Pyramid keeps an error response's exception on the request. Its
traceback reached the frame holding the pipeline result, so each
failed request kept the selected metadata tree alive until a full
garbage collection.

Run the plumbings in their own function, re-raise HTTP errors without
their traceback, and raise 409/500 outside the except clauses. HTTP
errors such as 406 from content negotiation are no longer turned
into 500.

Closes #283.

@kushaldas kushaldas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is okay to merge.
One small gap: an HTTPException can still keep the pipeline result alive
if it was raised while handling another exception.

except exc.HTTPException as ex:
    raise ex.with_traceback(None)

with_traceback(None) only clears the traceback of the HTTPException
itself. Python also attaches the exception that was being handled when it
was raised, as context (or cause with "raise ... from e"). That
inner exception has its own traceback, which goes back to the
_run_plumbings frame, so the result stays alive as in #283:

request.exception
  -> HTTPNotFound            (traceback cleared, fine)
       .__context__
         -> KeyError
              .__traceback__
                -> _run_plumbings frame
                     -> pipeline result   (still alive)

I reproduced this with the same gc.disable()/weakref harness as the new
tests, using a process() that does:

try:
    raise KeyError(...)
except KeyError:
    raise exc.HTTPNotFound()

Result: status 404, result NOT freed. The same harness with a
ResourceException gives 409 and the result is freed, so that path is fine.

pyFF's own code can't hit this today: the only HTTPException under
_run_plumbings is the 406 from _fmt, and it isn't raised inside an except
block. A custom pipe could, though.

Suggested fix:

except exc.HTTPException as ex:
    ex.__context__ = ex.__cause__ = None
    raise ex.with_traceback(None)

("raise ex from None" is not enough: it only hides the context when the
traceback is printed, and context stays set.)

It would also be good to add a test for this chained case, and one for
the 409 path.

Minor, no change needed: a small cycle remains
(request -> exception -> traceback -> process_handler frame -> request),
but those frames hold no pipeline data and the cyclic GC frees it.
Strictly, the code comment about the cycle applies only to the
_run_plumbings frame.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

High Memory Usage

2 participants