Support for default types in TypedDict

[fellnerse]

See here: python/mypy#6131
What do you think?

[erictraut]

When you say "default types", am I correct in assuming that you want a way to represent that "the TypedDict is allowed to contain additional keys beyond what is specified, but they must be of this type"?

There was a recent discussion about this in the typing sig related to this topic:
https://mail.python.org/archives/list/typing-sig@python.org/thread/66RITIHDQHVTUMJHH2ORSNWZ6DOPM367/#AIBYTWKQHHVP2EYHZULI3MK4M5MB4VZA

[adam-grant-hendry]

+1 for the language server. That’s a good reason.

You're pretty far down the rabbit hole on this one.

Yes, I understand and thank you for your help. I’m just trying to understand things.

Can you please explain what PEP 692 is for then? In what cases should this be used/is it necessary?

[adam-grant-hendry]

@erictraut Thank you again for your help (and Happy Father's Day as well).

From our conversation, I’m inclined to believe you are against PEP 692, and I understand why.

Ironically, part of its motivation states it is useful for:

existing codebases where the need of refactoring the code in order to introduce proper type annotations may be considered not worth the effort. This in turn prevents the project from getting all of the benefits that type hinting can provide.

even though you showed the additional type hinting is a rather less than useful

**kwargs: **KwargsTypedDict

Ultimately, I see now that PEP 692 is a convenience rather than a necessity. It should be up to the purview of the coder to choose whether or not to use it.

To that end, I will recommend an additional note be added to the PEP that recommends against its usage wherever possible as it tends to abstract away useful type hinting information rather than add to it. In all cases, it would seem better to simply type out the information rather than hide it behind a TypedDict.

[erictraut]

I’m inclined to believe you are against PEP 692

On the contrary, I'm supportive of PEP 692. In fact, I implemented support for the mechanism in pyright when Franek was working on the initial draft of the PEP.

The facility described in PEP 692 has utility, but it is applicable only for certain use cases. It is not a replacement for the existing, well-established mechanisms in the type system for describing function signatures with keyword parameters.

PEP 692 is useful in cases where an API accommodates optional keyword-only parameters when no default values are provided for absent keywords. The draft PEP shows some examples. Here's another discussion in the mypy issue tracker that provides some examples where it might be useful. Note that most functions written in Python, even if they use **kwargs in their implementation, do not follow this design pattern. That explains why the mechanism described in PEP 692 is a latecomer to the type system compared to other well-established facilities like specification of positional-only parameters, keyword-only parameters, and overloads. I predict PEP 692 will be used relatively infrequently, but there is clearly demand for it. I hope that it won't be abused.

@adam-grant-hendry, the fact that the draft PEP 692 created so much confusion for you leaves me a bit worried. For reference, are you relatively new to static typing in Python, or have you been using typed Python for a long time? If it confuses other Python developers to the same degree, it has the potential of being a net negative to the type system.

@franekmagiera, perhaps it's worth adding some language in the PEP (and maybe an example) that clarifies where the new mechanism should not be used?

[adam-grant-hendry]

For reference, are you relatively new to static typing in Python, or have you been using typed Python for a long time?

Not terribly new, no.I have over 15 years experience with Python and C++. The concepts of generics and variadics are not foreign to me and I know type hinting was first proposed by Guido back in 2015, so it has been around a while. However, for the sake of argument, I will consider myself relatively new for now.

From a beginner's standpoint, I think your fear that PEP 692 might be abused is warranted on at least three points (hopefully my notes might help you and others with some of the wording in the documentation):

1. I wasn't aware originally that if a function signature implementation uses *args and **kwargs

@staticmethod
def keyClick(*args, **kwargs):
    # Implementation goes here...

that one could type hint it with the actual intended arguments

@staticmethod
def keyClick(widget: QWidget, key: Qt.Key | str, modifier: Qt.KeyboardModifiers = ..., delay: int  = ...) -> None: ...

without using @overload. Furthermore, to use @overload, one needs to have at least 2 type signatures, so I thought I would have to figure out how to type hint using the *args/**kwargs variant anyway, so using @overload wouldn't have made sense for me.

I feel I read PEP 484 very well, but I couldn't gather that that was possible. Furthermore, I still get from pylance the linting errors:

Type of parameter "args" is unknown.
Type of parameter "kwargs" is unknown.

even if my stub is type hinted to the latter, so it would still not pass pre-commit.

2. For those who do want to include type hints in the source code itself, rather than in *.pyi stubs, (unless I'm mistaken) it does seem that one would have to refactor their code in order to type hint:

@staticmethod
def keyClick(*args, **kwargs):
    # Implementation goes here...

would have to change to

@staticmethod
def keyClick(widget: QWidget, key: Qt.Key | str, modifier: Qt.KeyboardModifiers = ..., delay: int  = ...) -> None:
    # Implementation goes here...

which could be a problem for 3rd party libraries if the original maintainers do not want to refactor signatures, but want contributors to develop type hints directly within the code.

3. I'm concerned about the language server hiding type information, as you've shown. This inclines me not to use it in favor of refactoring altogether.

perhaps it's worth adding some language in the PEP (and maybe an example) that clarifies where the new mechanism should not be used?

I am fully in support of this! In addition, it would also be useful if pylance could output a warning or a hint if it detects that it is being used in a way that is not intended by PEP 692.

[franekmagiera]

That's an interesting discussion. I agree that in the example you've been discussing using an "explicit" signature is the best tool for the job.

As you've noticed in point 2 that is not always possible without refactoring. I think PEP 692 would still bring some value - realistically not everyone will want to refactor, in that case I'd much rather see **kwargs: **SomeTypedDictThatIHaveToLookUp than **kwargs: Any (also seems like language servers should be able to present unpacked **kwargs in that case) - also then kwargs variable inside the function could be type checked (and treated as a TypedDict) and that would bring benefits to the maintainers of the "upstream" codebase. Point 2 of your last reply is exactly where I'm coming from.

That said, it is definitely worth including the conclusions from this thread in the draft!

[adam-grant-hendry]

When you say "default types", am I correct in assuming that you want a way to represent that "the TypedDict is allowed to contain additional keys beyond what is specified, but they must be of this type"?

@erictraut This is correct w.r.t. @rra 's original request in python/mypy/6131.

@fellnerse and I seem to be discussing default values.

[spacether]

One can make a dict with an additionalProperty type today in vscode per: https://stackoverflow.com/questions/73558897/in-python-how-can-i-make-a-dict-that-has-type-hints-for-know-keys-and-a-default/73559006?noredirect=1#comment129898346_73559006

[pycaw]

Currently it seems neither TypedDict+Unpack nor ParamSpec nor Protocol support (proper) forwarding of function parameter annotations. They almost do it but then fall short in differents ways. One then needs to resort to using explicit parameter classes.
It would be really great there were a way to define a set of (kw) parameters that could be reused in function definitions with hints, defaults, docs preserved.
Probably not a fair comparision, but now I value std::forward, perfect forwarding and the DRY that was/is possible to achieve with them in C++ much more.