Improve Falcon parser

[vytas7]

Let me first say that it was a privilege to meet @sirosen @ EuroPython '26.
As I also replied on LinkedIn to @InessaPawson, the little BoF session on AI impact (that we participated in) was definitely one of the conference's main highlights for me.

IIRC I offered to help to improve Falcon "parser", but now that I've opened the code, it is not as trivial as I thought.

Main things that I'm unsure about (in progress):

• Do we want to DRY URL-encoded forms with Falcon's media handling? Do we need to keep the parse_form_body(...) helper for backwards compatibility, or is it considered private? Do we want to silence Unicode errors as it does?
• How to load media/request body in the async case? Is it properly supported for AIOHTTP?
• Add a recipe how to integrate this in middleware (in possible).

I will create a Draft PR addressing some of the mundane improvements, the rest needs discussion (since I don't have much experience with webargs).

[sirosen]

Meeting everyone at that roundtable was also a highlight for me! I'll be working in the coming weeks on finding ways to keep that conversation going.

@lafrech, @sloria, as a quick explanation, Vytas and I met this past weekend at EuroPython; he's one of the maintainers of Falcon.
I'll be sending some emails when I get home, including ones to each of you, to which will make some sense of this subthread. 😄

IIRC I offered to help to improve Falcon "parser", but now that I've opened the code, it is not as trivial as I thought.

I thought I was going to try to do it and run it past you, but I can't pretend to remember perfectly.
I'll look at the draft PR within the next couple of days and we can discuss details there.

We were preparing to release a v9.0 of webargs in the fall, but it got delayed. That may turn out to be advantageous for us, as it gives us some freedom to make breaking changes as needed.

[sloria]

very cool! looking forward to hearing about it @sirosen . and thanks for offering to help @vytas7

[sirosen]

I finally took some time to start looking through things here! (got swamped with lots of priorities when I came back from EuroPython 😅 )

Also, sorry, this post is quite long because it's hard to condense the information I want to share! 🥲

Item 1: URL-encoded form handling

Do we want to DRY URL-encoded forms with Falcon's media handling? Do we need to keep the parse_form_body(...) helper for backwards compatibility, or is it considered private? Do we want to silence Unicode errors as it does?

I think we should rename it, in our v9.0 release (dev branch). We're not intentionally exposing it, so let's make it _parse_form_body. My conclusion is that we need to keep it. We should realign it with Falcon, which I think means adding the csv=... arg and not much else?

I reach this after realizing that Falcon and webargs have APIs which do not match, and the cost to our users of aligning them outweighs the cost to us of doing some extra work.

webargs is trying to expose the following methods:

• FalconParser.load_querystring: the query parameters as a MultiDictProxy¹
• FalconParser.load_form: the form body as a MultiDictProxy
• FalconParser.load_json_or_form: inherited from core.Parser; wraps load_form() to try JSON and failover to a form body

In Falcon, when a form body is parsed, it is combined with the query params.

FalconParser.load_querystring is currently using req.params, so that seems... wrong. It will include things from a form-post body as well, but we called it "querystring" to match our other parsers.

If we wanted to true up our APIs, we'd make all three of those webargs methods I mentioned -- including the inherited one, override it -- raise NotImplementedError because they don't match Falcon. We'd provide load_params instead, for handling falcon.Request.params. I don't really want to do this: the code we have to copy is not so big, and I think the user-facing breakage is confusing and surprising.

That analysis gives me a second conclusion: FalconParser.load_querystring() should possibly start emitting a warning. I'll file it as an issue for us to assess this during the webargs v9.x lifecycle.² We might deprecate/remove it, or if we're okay with the "lie" in the method name we'll keep it but add a doc warning or similar.

Item 2: async support

How to load media/request body in the async case? Is it properly supported for AIOHTTP?

My first inclination is not to add anything or do anything until a user comes to us with a use-case.

We have a level of orchestration one layer higher which handles choosing sync vs async codepaths. If we define async def load_media where we currently have def load_media, everything would work, but it would break the sync case... which is no good.

We could easily add a separate parser, AsyncFalconParser or a separate "location" in webargs: async def load_media_async. But until I'm talking to a user who wants this, it's hard to be confident about the API.

Item 3: adding a recipe

Add a recipe how to integrate this in middleware (in possible).

I'd welcome such an addition, in docs/framework_support.rst if it's short!
You'll see a minimal Falcon usage example there, and I wouldn't mind a slightly larger one.

If it's much bigger, we can put it in our examples dir, perhaps?

Footnotes

1. MultiDictProxy is part of the webargs problem-space. We wrap dicts and multidicts, to handle get() vs get_list()/get_all() semantics smoothly, when combined with marshmallow schemas. You can mostly ignore and treat it as "special marshmallow-facing dict". ↩

2. I won't file it right away, in case we reach a different conclusion. ↩

[vytas7]

Thanks for the detailed feedback @sirosen!

Just to clarify Falcon's parsing of URL-encoded forms: no, the form is not automatically combined with req.params. This only happens if the deprecated req_options.auto_parse_form_urlencoded option is used. In fact, this option and the code block you referenced will be removed in the next major version (we don't bump these too often though, so maybe in ~3 years).

The canonical way is to simply use req.get_media() to parse the request body, that includes URL-encoded forms too. The URL-encoded form media handler is installed by default.

So what load_querystring does is not incorrect per se.

Regarding these 3 load_* methods in general, is that what defines a webargs "parser" interface?

Edited/added ⬇️

if we are concerned about FalconParser.load_querystring being incorrect when this option is used, we could emit a warning specifically in the case req.options.auto_parse_form_urlencoded is enabled.

In fact, when the user tries to enable this behaviour, Falcon already emits a warning on its own:

>>> import falcon.testing
>>> req = falcon.testing.create_req()
>>> req.options.auto_parse_form_urlencoded
False
>>> req.options.auto_parse_form_urlencoded = True
<...>/falcon/falcon/request.py:2820: DeprecatedWarning: The RequestOptions.auto_parse_form_urlencoded option is deprecated. Please use Request.get_media() to consume the submitted URL-encoded form instead.
  warnings.warn(

[sirosen]

Ah! I even read straight past the "auto" option, but didn't understand what I was seeing.
I need to take some time with this, in that case. I still think there's a mismatch, but it's not quite the one I described.

We list "locations" for getting data: query params, headers, etc. And for the body, we have multiple "locations" which are actually the parsing methods: "json", "form", etc. For the Falcon parser, we have a location named "media", which calls get_media().

One option: we could make load_form check the media type, and then call get_media if that passes. We'd either return our missing sentinel or raise an error if the check fails (I'm not sure offhand). Does that sound like a reasonable way to wrap get_media()?

[vytas7]

One option: we could make load_form check the media type, and then call get_media if that passes. We'd either return our missing sentinel or raise an error if the check fails (I'm not sure offhand). Does that sound like a reasonable way to wrap get_media()?

Aye, that's exactly what I was suggesting.

Just a couple more details, and I can go on to materialize this in my PR:

1. Unlike JSON, Falcon treats an empty body as valid URL-encoded data (deserializes to {}). Is that in line with webargs' behaviour?
2. Regarding other media types passed to load_form(), it's your call 🙂 I think there are two branches here though: one, a multipart form is submitted -- Falcon fully supports parsing it, but unlike other frameworks, the user has to iterate the form (nothing is automatically spooled in memory or temporary files behind your back) -- since this is not supported by webargs yet, we can raise an error; two, the payload is any other non-form media -- here it's your call.