feat: add output token limit utils for chat generators

[sjrl]

Related Issues

• part of https://github.com/deepset-ai/haystack-private/issues/512
• POC branch for the full summarization compactor is here feat: add summarization compactor #12296

Proposed Changes:

Add util method for resolving how to pass down a max token limit from the Summarization Compactor down to the Chat Generator. We need to know the token limit at the Compactor level so we can properly account for how large the expected summary is when computing how much of the previous chat history should be summarized.

The method I chose here is to create a util method that allows us to pass down a max token limit through the generation_kwargs runtime params of the ChatGenerator.

However, I am wondering if a better solution would be to directly expose this as a standardized param name on the ChatGenerators run method instead of needing this util method.

How did you test it?

Added new tests.

Notes for the reviewer

I made this a standalone PR so its easier to discuss whether we like this solution and to reduce the PR size of the eventual summarization compactor.

I'd appreciate your thoughts on whether it makes sense to open issues to update our ChatGenerators to use a standardized param name like max_output_tokens instead of needing to maintain/update this dict.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
haystack-docs	Ignored	Preview	Aug 12, 2026 7:24am

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/generators/chat
utils.py
Project Total

This report was generated by python-coverage-comment-action

[anakin87]

The current idea assumes that Haystack knows how each provider names this parameter.

A potential solution I'd prefer: provide each ChatGenerator with a public field like output_token_limit_key that can be queried: MyChatGenerator.output_token_limit_key.

Another idea, as you also suggest, is to add this param to the run method: more explicit but we would then need to handle conflicts between this param and generation_kwargs; it seems to me that this would require more extensive changes.

What do you think?

[sjrl]

The current idea assumes that Haystack knows how each provider names this parameter.

A potential solution I'd prefer: provide each ChatGenerator with a public field like output_token_limit_key that can be queried: MyChatGenerator.output_token_limit_key.

I think this could work. However, I would want to set this up in such a way that it can be easily extended later to other keys. If this is set up well we could use this for passing down other common generation kwargs in a standardized way like temperature, reasoning level, etc.

Another idea, as you also suggest, is to add this param to the run method: more explicit but we would then need to handle conflicts between this param and generation_kwargs; it seems to me that this would require more extensive changes.

Yeah I agree this would cause a conflict and also I'm not sure if I want to greatly expand the run method signature (thinking of what I mention above about adding more keys in the future).

What do you think?

I agree that I would much prefer having this info stored on the specific implementation itself. Do you think we could take your first idea and make it more extensible?

[anakin87]

I agree that I would much prefer having this info stored on the specific implementation itself. Do you think we could take your first idea and make it more extensible?

I think yes. We could potentially store a dictionary in the ChatGenerator with the mapping: Haystack concept -> Provider-specific key. (Just an idea, haven't tried)

If we have doubts about the implementation and we decide to do this, I'd just make the field(s) private for the moment.

[sjrl]

I agree that I would much prefer having this info stored on the specific implementation itself. Do you think we could take your first idea and make it more extensible?

I think yes. We could potentially store a dictionary in the ChatGenerator with the mapping: Haystack concept -> Provider-specific key. (Just an idea, haven't tried)

If we have doubts about the implementation and we decide to do this, I'd just make the field(s) private for the moment.

Okay sounds good, I'll take a look into this!

[sjrl]

Moving back to draft while looking into a new solution

[sjrl]

Closing as superseded by #12328