[pip] PIP-459: Batch Status Summary and Filtered Listing for Pulsar Functions

[onceMisery]

• Main Issue: #25235

Motivation

See pip/pip-459.md for the full proposal.

Modifications

Added PIP-459 proposal document.

Verifying this change

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

matching-pr

[lhotari]

How would this continuationToken work and what would be the format? The Pulsar Functions Admin API is stateless and subsequent calls could go to different Function Worker instances in a cluster with multiple workers.

[shibd]

Hi @onceMisery,
#25299 (comment)

This design makes sense to me. It can reduce the number of requests needed to fetch function stats. The only trade-off is that each worker needs to first list the functions under the namespace and sort them lexicographically.

1. Should we rename this parameter to something more explicit, such as startAfterFunctionName?

2. We should also clearly document the pagination behavior in the API, similar to how you described it in your comment.

[lhotari]

Add the discussion link here, https://lists.apache.org/thread/k792434hgm237qmtbn4fsd62hdzdt0h7

Suggested change

	* Mailing List discussion thread: https://lists.apache.org/thread/k792434hgm237qmtbn4fsd62hdzdt0h7
	* Mailing List voting thread: TBD

[onceMisery]

Thanks for raising this concern!

You're absolutely right that the Admin API is stateless and requests can hit
different workers. My design addresses this through deterministic, name-based
cursor pagination:

How it works:

• continuationToken is simply the last function name from the previous page
• Each worker independently sorts all function names lexicographically
• The token acts as an exclusive lower bound to find the next page
• No server-side state is required; the token is self-contained

This approach works consistently across workers because the sorting is
deterministic and metadata is replicated.

Trade-off:
If functions are created/deleted during pagination, clients may see duplicates
or miss entries. This is a common limitation of stateless cursor-based pagination
in distributed systems (similar to Kubernetes list pagination or DynamoDB queries).

One improvement I should make:
I also agree the API should return nextContinuationToken explicitly rather than only a bare list, so I can update the response shape to make the paging contract clear.

  {
    "summaries": [...],
    "nextContinuationToken": "func-xyz"  // or null
  }

Would this design work for you? I can update the PR accordingly.

[onceMisery]

Thanks for raising this concern! You're absolutely right that the Admin API is stateless and requests can hit different workers. My design addresses this through deterministic, name-based cursor pagination: How it works: continuationToken is simply the last function name from the previous page Each worker independently sorts all function names lexicographically The token acts as an exclusive lower bound to find the next page No server-side state is required; the token is self-contained This approach works consistently across workers because the sorting is deterministic and metadata is replicated. Trade-off: If functions are created/deleted during pagination, clients may see duplicates or miss entries. This is a common limitation of stateless cursor-based pagination in distributed systems (similar to Kubernetes list pagination or DynamoDB queries). One improvement I should make: Currently the API returns List<FunctionStatusSummary>. I should wrap this in a response object with nextContinuationToken (null when no more data), so clients can easily detect the end of pagination: { "summaries": [...], "nextContinuationToken": "func-xyz" // or null } The name continuationToken might be ambiguous. What do you think of startAfter? for example: public List<FunctionStatusSummary> getFunctionsWithStatus( String tenant, String namespace, Integer limit, String startAfter) Would this design work for you? I can update the PR accordingly. Message ID: ***@***.***>

[shibd]

hi, @onceMisery Thanks for PIP, Have we considered including a bit more information in this response, for example:

- `receivedTotal`
- `processedSuccessfullyTotal`
- `systemExceptionsTotal`
- `userExceptionsTotal`
- `avgProcessLatency`
- `userMetrics`

These values are already aggregate, and they could give operators a more direct view of a function's actual health instead of relying only on RUNNING / STOPPED / PARTIAL / UNKNOWN.

Another option would be to keep the default response lightweight, but add a query parameter to control the level of detail returned by the REST API. That would let us preserve the current "summary" use case while still supporting a more diagnostic view when needed.

[onceMisery]

I think this is a great suggestion. @shibd

[shibd]

Hi @onceMisery,
#25299 (comment)

This design makes sense to me. It can reduce the number of requests needed to fetch function stats. The only trade-off is that each worker needs to first list the functions under the namespace and sort them lexicographically.

1. Should we rename this parameter to something more explicit, such as startAfterFunctionName?

2. We should also clearly document the pagination behavior in the API, similar to how you described it in your comment.

[onceMisery]

@shibd @lhotari
I fully agree with this suggestion:

1. The continuationToken should be renamed to startAfterFunctionName.
2. add a query parameter to control the level of detail returned by the REST API. That would let us preserve the current "summary" use case while still supporting a more diagnostic view when needed.