Skip to content

plugin docs: Stabilize the public plugin authoring API#1959

Merged
rapids-bot[bot] merged 15 commits into
NVIDIA:developfrom
bbednarski9:bbednarski/issue-1952
May 28, 2026
Merged

plugin docs: Stabilize the public plugin authoring API#1959
rapids-bot[bot] merged 15 commits into
NVIDIA:developfrom
bbednarski9:bbednarski/issue-1952

Conversation

@bbednarski9
Copy link
Copy Markdown
Contributor

@bbednarski9 bbednarski9 commented May 18, 2026

Stabilize the public plugin authoring API

Closes #1952.

Summary

This PR defines nat.plugin_api as the stable public import surface for third-party plugin authors and updates the author-facing docs/examples to use that surface instead of deeper implementation modules.

It also adds a third-party plugin package guide for partner-owned repositories, including package naming, project layout, nat.plugins entry points, compatibility expectations, listing requirements, and testing guidance.

The intent is to make the common external plugin contract explicit without redesigning provider-specific capabilities such as web search results, embeddings, or retrieval payloads, and without prematurely promoting every runtime extension point into the stable facade.

What Changed

  • Adds nat.plugin_api as the public facade for plugin authoring APIs:
    • common registration decorators
    • builders and function/function-group authoring types
    • configuration bases and refs for promoted plugin surfaces
    • provider info helpers
    • secret helpers
    • small middleware, memory, and object-store implementation contracts needed by registered components
  • Adds contract tests for the exact nat.plugin_api.__all__ export map so public API drift is intentional.
  • Tracks deferred public API candidates in the contract test, with source modules and rationale, so intentionally unpromoted surfaces are documented and do not silently rot.
  • Adds a public Plugin API docs page that distinguishes stable authoring APIs, trusted-plugin surfaces, deferred candidates, and private implementation modules.
  • Adds third-party plugin packaging docs based on the external repository process: repository ownership, package layout, dependency boundaries, version compatibility, licensing expectations, plugin listing metadata, and contribution checklist.
  • Documents the entry-point policy explicitly: new external packages should use the nat.plugins entry-point group, while nat.components remains supported for backward compatibility.
  • Clarifies that telemetry facade support covers exporter registration and configuration; exporter runtime implementation APIs remain subsystem-specific until promoted deliberately.
  • Updates custom component docs, workflow templates, and examples to import from nat.plugin_api.
  • Keeps docs for deferred extension points on their existing subsystem imports instead of presenting them as stable nat.plugin_api exports.
  • Extends ToolTestRunner with function-group helpers so external packages can test registered function groups and assert group__function names without constructing a full workflow.
  • Fixes stale function-group docs/examples that still referenced legacy group.function names; runtime dot-compatibility remains unchanged.
  • Clarifies that the existing Tavily and Exa integrations are LangChain-backed tools, while external integrations should depend on the framework-agnostic NeMo Agent Toolkit plugin authoring API.

Scope Boundaries

In scope:

  • Public import stability for plugin authors.
  • Documentation of which APIs third-party packages should use.
  • Third-party plugin package guidance for external repository maintainers.
  • Regression tests that protect the public facade.
  • Documented deferred candidates for advanced extension points that should be reviewed separately before promotion.
  • Function-group testing support needed by external packages using the group__function convention.

Out of scope:

  • A shared web search result schema.
  • New capability-specific interfaces for search, embeddings, retrieval, or other managed-provider features.
  • Removing existing legacy imports from implementation modules.
  • Breaking users that still import from older module paths or still publish nat.components entry points.
  • Promoting advanced subsystem extension points that need more review, including front ends, logging methods, registry handlers, optimizer hooks, finetuning components, and TTC strategies.

Web search result fields remain provider-specific in this PR. Third-party packages should use the public plugin authoring API, but their result payloads should stay unconstrained unless a separate capability contract is designed later.

Motivation

External packages such as nemo-agent-toolkit-tavily need a clearer contract for which NeMo Agent Toolkit APIs are public and stable. Today, plugin examples and downstream packages can end up importing from implementation modules such as nat.builder.*, nat.cli.register_workflow, and nat.data_models.*.

This PR creates a conservative facade that external packages can rely on across minor and patch releases, while leaving private implementation modules free to evolve as long as the public contract remains intact.

Installed plugins still execute as trusted Python code in the application environment. This PR stabilizes import paths and authoring contracts; it does not make untrusted plugins safe to install or run.

Testing

  • uv run ruff check packages/nvidia_nat_core/src/nat/plugin_api.py packages/nvidia_nat_core/tests/nat/test_plugin_api.py packages/nvidia_nat_test/src/nat/test/tool_test_runner.py packages/nvidia_nat_core/tests/nat/tools/test_tool_test_runner.py packages/nvidia_nat_langchain/src/nat/plugins/langchain/tools/tavily_internet_search.py packages/nvidia_nat_langchain/src/nat/plugins/langchain/tools/exa_internet_search.py
  • uv run --project packages/nvidia_nat_core --extra test pytest packages/nvidia_nat_core/tests/nat/test_plugin_api.py packages/nvidia_nat_core/tests/nat/tools/test_tool_test_runner.py -q
    • 11 passed
  • uv run ruff check packages/nvidia_nat_core/src/nat/plugin_api.py packages/nvidia_nat_core/tests/nat/test_plugin_api.py docs/source/extend/third-party-plugins.md
  • uv run --project packages/nvidia_nat_core --extra test pytest packages/nvidia_nat_core/tests/nat/test_plugin_api.py packages/nvidia_nat_core/tests/nat/tools/test_tool_test_runner.py -q
    • 14 passed
  • git diff --check
  • python3 ci/scripts/license_diff.py develop > license_diff_output.txt
    • output file is empty
  • python3 ci/scripts/path_checks.py --check-paths-in-files
    • plugin API and third-party plugin docs path issues are cleared; local full run still reports unrelated missing external/lc-deepagents-quickstarts example paths from the checkout

Follow-Ups

  • Migrate downstream external repos, starting with nemo-agent-toolkit-tavily, to import from nat.plugin_api.
  • Defer breaking import removals until external packages have had a release cycle to move to the facade.
  • Review deferred extension points individually before promoting them into nat.plugin_api.
  • Consider future capability contracts only where there is a concrete interoperability need. Web search result payloads are intentionally left provider-specific here.

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
    • Any contribution which contains commits that are not Signed-Off will not be accepted.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

Summary by CodeRabbit

  • Documentation

    • Updated many examples to use unified nat.plugin_api imports; added a Plugin API guide and third‑party plugin guidance; switched function-group/tool naming examples to double‑underscore separators.
  • New Features

    • Introduced nat.plugin_api as the stable public plugin authoring surface.
    • Added function‑group testing support to the test tooling to verify and invoke grouped tools.
  • Tests

    • Added tests enforcing the plugin API export contract and preferred import conventions.
    • Added tests covering function‑group exposure and invocation.

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 18, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds a pinned public plugin-authoring module nat.plugin_api and tests enforcing its export contract and docs import rules; migrates documentation and examples to nat.plugin_api; switches function-group naming to double-underscore (__); and extends test tooling to support building, inspecting, and invoking function groups.

Changes

Plugin API Surface & Contract

Layer / File(s) Summary
Public API module definition
packages/nvidia_nat_core/src/nat/plugin_api.py
New nat.plugin_api module that consolidates and re-exports a fixed set of public symbols including Builder, EvalBuilder, configuration base classes (FunctionBaseConfig, FunctionGroupBaseConfig, LLMBaseConfig, AuthProviderBaseConfig, etc.), provider/info/ref types, middleware/context types, memory/object-store/retriever interfaces and models, secret helpers, and register_* decorators via an explicit __all__.
Public API contract validation
packages/nvidia_nat_core/tests/nat/test_plugin_api.py
Test module that validates the public contract: asserts plugin_api.__all__ contains exactly expected exports (no extras or omissions), each exported name is identity-equal to its declared source, deferred candidates remain unpromoted, docs/templates avoid denied internal direct-import patterns, and the Builder stable/deferred method surface is explicit.
Public plugin API documentation
docs/source/extend/plugin-api.md
New documentation page defining the stable public plugin authoring API surface, the stability/compatibility contract, a "Surface Review" table marking areas Stable vs Deferred, guidance on private implementation modules, and a Function Group contract example showing __-separated function naming.

Documentation & Example Migration to Plugin API

Layer / File(s) Summary
Function documentation migration
docs/source/extend/custom-components/custom-functions/functions.md
Documentation updated to reference nat.plugin_api.register_function, nat.plugin_api.FunctionBaseConfig, and nat.plugin_api.FunctionInfo methods.
Function group documentation migration
docs/source/extend/custom-components/custom-functions/function-groups.md
Guide updated to use nat.plugin_api.FunctionGroupBaseConfig, nat.plugin_api.register_function_group, ToolTestRunner-based testing examples, and __ namespacing examples.
Middleware documentation migration
docs/source/build-workflows/advanced/middleware.md
Middleware guide updated to import middleware configs, base classes, registration decorators, and context types from nat.plugin_api, including API reference entries.
Provider documentation migration
docs/source/extend/custom-components/adding-an-llm-provider.md, adding-an-authentication-provider.md, adding-a-retriever.md, custom-dataset-loader.md, custom-evaluator.md
Provider and retriever docs updated to import provider base configs and registration decorators from nat.plugin_api.
Custom component documentation migration
docs/source/extend/custom-components/memory.md, object-store.md, telemetry-exporters.md, finetuning.md
Custom component docs updated to reference nat.plugin_api base configs, interfaces, models, and register_* helpers.
Additional documentation updates
docs/source/build-workflows/retrievers.md, components/sharing-components.md, extend/custom-components/custom-functions/per-user-functions.md, extend/plugins.md, extend/testing/add-unit-tests-for-tools.md, get-started/tutorials/add-tools-to-a-workflow.md, improve-workflows/evaluate.md, improve-workflows/optimizer.md, run-workflows/existing-agents/langgraph.md, docs/source/index.md
Misc docs updated to prefer nat.plugin_api imports; Plugin API page added to the Extend toctree.
Function group naming convention updates
docs/source/build-workflows/functions-and-function-groups/function-groups.md, docs/source/build-workflows/mcp-client.md
Examples and config snippets switched from dot-separated (.) to double-underscore (__) function-group naming (e.g., math.addmath__add, mcp_tools.tool_amcp_tools__tool_a).
Code examples and configurations
examples/*, examples/A2A/*, examples/dynamo_integration/*, packages/nvidia_nat_core/src/nat/cli/commands/workflow/templates/workflow.py.j2
Example READMEs, YAML configs, and the workflow template updated to use nat.plugin_api imports and __ namespacing for function-group tools.

Function Group Testing Support

Layer / File(s) Summary
ToolTestRunner function group methods
packages/nvidia_nat_test/src/nat/test/tool_test_runner.py
Added _resolve_function_group_tool, test_function_group, and test_function_group_tool to build function groups, discover exposed fully-qualified function names, resolve ambiguities, and invoke group tools with positional or keyword inputs.
Function group test coverage
packages/nvidia_nat_core/tests/nat/tools/test_tool_test_runner.py
New tests register a SimpleCalculator function group and assert exposed tool names (e.g., calculator__add) and invocation correctness via ToolTestRunner helpers.

Deprecation and Future Breaking Changes

Layer / File(s) Summary
Breaking change documentation
packages/nvidia_nat_core/src/nat/builder/workflow_builder.py, per_user_workflow_builder.py, packages/nvidia_nat_eval/src/nat/plugins/eval/runtime/builder.py
Added TODO/deprecation comments documenting planned removal of legacy dot-separator compatibility and the requirement to use the FunctionGroup.SEPARATOR (__) in a future breaking release.
Tool description refinements
packages/nvidia_nat_langchain/src/nat/plugins/langchain/tools/exa_internet_search.py, tavily_internet_search.py, and related example docstrings`
Minor docstring/comment edits clarifying LangChain-backed tools and developer notes on naming convention.

🎯 4 (Complex) | ⏱️ ~75 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 47.06% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: stabilizing the public plugin authoring API with imperative mood and appropriate length.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
…on differed items

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
@bbednarski9 bbednarski9 marked this pull request as ready for review May 19, 2026 05:31
@bbednarski9 bbednarski9 requested a review from a team as a code owner May 19, 2026 05:31
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (4)
packages/nvidia_nat_eval/src/nat/plugins/eval/runtime/builder.py (1)

146-147: 💤 Low value

Optional: Simplify string formatting.

The warning message uses an f-string concatenated with another f-string. This can be simplified to a single f-string for better readability.

♻️ Proposed simplification
             if (fn_name not in self._functions) and (new_fn_name in self._functions):
                 logger.warning(
-                    f"Function `{fn_name}` is deprecated and will be removed in a future release." + \
-                        f"Use `{new_fn_name}` instead."
+                    f"Function `{fn_name}` is deprecated and will be removed in a future release. "
+                    f"Use `{new_fn_name}` instead."
                 )
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nvidia_nat_eval/src/nat/plugins/eval/runtime/builder.py` around
lines 146 - 147, The deprecation warning builds its message by concatenating two
f-strings; replace the concatenation with a single f-string to improve
readability. Update the code that constructs the message using fn_name and
new_fn_name (the string currently written as f"...{fn_name}..." +
f"...{new_fn_name}...") so it becomes one f-string that includes both
placeholders in-line, e.g. a single f"...{fn_name}...{new_fn_name}..."
expression.
docs/source/improve-workflows/optimizer.md (1)

199-200: 💤 Low value

Consider clarifying mixed import sources in the example.

This code example mixes imports from the public API (nat.plugin_api) with imports from internal modules (nat.data_models.optimizable). While optimizer-specific types may be intentionally out of scope for the public plugin API, the mixed import pattern could confuse plugin authors about which APIs are stable.

Consider one of the following:

  • Add a comment explaining that optimizer APIs are not part of the stable plugin surface and require internal imports
  • If OptimizableField, SearchSpace, and OptimizableMixin should be public, migrate them to nat.plugin_api
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/source/improve-workflows/optimizer.md` around lines 199 - 200, The
example mixes public API imports (FunctionBaseConfig) with internal types
(OptimizableField, SearchSpace, OptimizableMixin); clarify this by adding a
short inline comment above the internal imports stating that OptimizableField,
SearchSpace, and OptimizableMixin live in internal modules and are not part of
the stable plugin API (or alternatively move those types into nat.plugin_api if
they are intended to be public), so update the example to either add that
explanatory comment referencing FunctionBaseConfig and the three optimizable
symbols or migrate the three types to nat.plugin_api if public exposure is
desired.
docs/source/run-workflows/existing-agents/langgraph.md (1)

127-127: ⚡ Quick win

Complete the migration to the public API for Builder import.

The example has partially migrated to nat.plugin_api (line 126 for LLMFrameworkEnum) but still imports SyncBuilder from the internal module nat.builder.sync_builder. According to the PR objectives and coding guidelines, Builder is explicitly part of the public plugin API facade and should be imported from nat.plugin_api.

Update the import to:

from nat.plugin_api import Builder  # or SyncBuilder if exported

This ensures consistency within the example and guides users toward the stable public API. As per coding guidelines, Builder imports from nat.plugin_api are the recommended pattern across provider documentation.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/source/run-workflows/existing-agents/langgraph.md` at line 127, The
example currently imports SyncBuilder from the internal module
nat.builder.sync_builder; replace that import with the public API import from
nat.plugin_api (import Builder or SyncBuilder as exported) so the example
consistently uses the public facade (match the existing LLMFrameworkEnum import
from nat.plugin_api), updating any references to SyncBuilder to use the public
Builder symbol if necessary.
packages/nvidia_nat_core/src/nat/plugin_api.py (1)

96-168: ⚡ Quick win

Sort __all__ to satisfy lint and reduce contract churn noise.

RUF022 is already flagging this block. Keeping __all__ sorted will avoid repeated lint noise and make export diffs easier to review.

💡 Minimal fix approach
-__all__ = [
-    "AuthProviderBaseConfig",
-    ...
-    "set_secret_from_env",
-]
+__all__ = sorted([
+    "AuthProviderBaseConfig",
+    ...
+    "set_secret_from_env",
+])
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nvidia_nat_core/src/nat/plugin_api.py` around lines 96 - 168, The
__all__ list in plugin_api.py is unsorted and triggers lint RUF022; reorder the
entries in the __all__ list into a stable alphabetical order (e.g.,
ASCII/lexicographic) so names like "AuthProviderBaseConfig",
"AuthenticationRef", "Builder", ... are sorted; preserve the original quoting,
commas, and list formatting (one entry per line) and keep the existing items (no
additions/removals) to minimize contract churn.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/source/extend/custom-components/custom-functions/functions.md`:
- Line 71: Update the first public mention of the product name on this page:
replace "NeMo Agent Toolkit" with the full product name "NVIDIA NeMo Agent
Toolkit" in the paragraph that introduces registration (the sentence referencing
the nat.plugin_api.register_function decorator), leaving the rest of the text
unchanged.
- Line 197: Fix the grammar typo in the sentence that begins "Its required to
use an async context manager coroutine to register a function…" — change "Its
required" to "It's required" in the functions.md documentation where
nat.plugin_api.register_function is described so the user-facing text reads
"It's required to use an async context manager coroutine to register a
function…".

In `@docs/source/extend/plugin-api.md`:
- Around line 18-21: The page header and first public description use the short
name; update the top-level heading and opening sentence to use the full product
name "NVIDIA NeMo Agent Toolkit" (e.g., change "Public Plugin API" and the
opening line to start with "NVIDIA NeMo Agent Toolkit — Public Plugin API")
while preserving the note about importing from nat.plugin_api and the rest of
the content; ensure the heading and first sentence follow the documentation
naming convention and consistently use the full product name on first mention.
- Around line 74-95: The table block uses slash-form phrases and flagged terms
that break docs CI (Vale and referential path checks); edit the table rows in
the plugin-api section to replace slashes with explicit conjunctions (e.g., "LLM
and embedder" instead of "LLM/embedder"), avoid slash abbreviations (e.g.,
"Trainer and trainer adapter" instead of "Trainer/trainer adapter"), and
rephrase or parenthesize flagged terms like "trusted plugin" and "Deferred"
(e.g., "Stable public (trusted)" or "Deferred — requires review") so that no
path-like tokens or slash-forms remain and the descriptions still convey the
same meaning.

In `@packages/nvidia_nat_core/tests/nat/test_plugin_api.py`:
- Around line 290-303: The test currently uses the public_imports regex which
only matches single-line "from nat.plugin_api import ..." statements so
multiline or parenthesized imports bypass the check; replace the regex-based
approach (public_imports) with an AST-based scan: for each file in files parse
the file with ast.parse, iterate ast.ImportFrom nodes where node.module ==
"nat.plugin_api", collect all alias.name values (handling alias.asname) and then
validate each imported name against EXPECTED_PLUGIN_API_EXPORTS, appending
violations when a non-public symbol is imported; this will correctly handle
multiline and parenthesized imports.

---

Nitpick comments:
In `@docs/source/improve-workflows/optimizer.md`:
- Around line 199-200: The example mixes public API imports (FunctionBaseConfig)
with internal types (OptimizableField, SearchSpace, OptimizableMixin); clarify
this by adding a short inline comment above the internal imports stating that
OptimizableField, SearchSpace, and OptimizableMixin live in internal modules and
are not part of the stable plugin API (or alternatively move those types into
nat.plugin_api if they are intended to be public), so update the example to
either add that explanatory comment referencing FunctionBaseConfig and the three
optimizable symbols or migrate the three types to nat.plugin_api if public
exposure is desired.

In `@docs/source/run-workflows/existing-agents/langgraph.md`:
- Line 127: The example currently imports SyncBuilder from the internal module
nat.builder.sync_builder; replace that import with the public API import from
nat.plugin_api (import Builder or SyncBuilder as exported) so the example
consistently uses the public facade (match the existing LLMFrameworkEnum import
from nat.plugin_api), updating any references to SyncBuilder to use the public
Builder symbol if necessary.

In `@packages/nvidia_nat_core/src/nat/plugin_api.py`:
- Around line 96-168: The __all__ list in plugin_api.py is unsorted and triggers
lint RUF022; reorder the entries in the __all__ list into a stable alphabetical
order (e.g., ASCII/lexicographic) so names like "AuthProviderBaseConfig",
"AuthenticationRef", "Builder", ... are sorted; preserve the original quoting,
commas, and list formatting (one entry per line) and keep the existing items (no
additions/removals) to minimize contract churn.

In `@packages/nvidia_nat_eval/src/nat/plugins/eval/runtime/builder.py`:
- Around line 146-147: The deprecation warning builds its message by
concatenating two f-strings; replace the concatenation with a single f-string to
improve readability. Update the code that constructs the message using fn_name
and new_fn_name (the string currently written as f"...{fn_name}..." +
f"...{new_fn_name}...") so it becomes one f-string that includes both
placeholders in-line, e.g. a single f"...{fn_name}...{new_fn_name}..."
expression.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3b362a3f-fa75-4ac7-af7b-91152d1b9091

📥 Commits

Reviewing files that changed from the base of the PR and between cc8ef19 and 5d50d65.

📒 Files selected for processing (42)
  • docs/source/build-workflows/advanced/middleware.md
  • docs/source/build-workflows/functions-and-function-groups/function-groups.md
  • docs/source/build-workflows/mcp-client.md
  • docs/source/build-workflows/retrievers.md
  • docs/source/components/sharing-components.md
  • docs/source/extend/custom-components/adding-a-retriever.md
  • docs/source/extend/custom-components/adding-an-authentication-provider.md
  • docs/source/extend/custom-components/adding-an-llm-provider.md
  • docs/source/extend/custom-components/custom-dataset-loader.md
  • docs/source/extend/custom-components/custom-evaluator.md
  • docs/source/extend/custom-components/custom-functions/function-groups.md
  • docs/source/extend/custom-components/custom-functions/functions.md
  • docs/source/extend/custom-components/custom-functions/per-user-functions.md
  • docs/source/extend/custom-components/finetuning.md
  • docs/source/extend/custom-components/memory.md
  • docs/source/extend/custom-components/object-store.md
  • docs/source/extend/custom-components/telemetry-exporters.md
  • docs/source/extend/plugin-api.md
  • docs/source/extend/plugins.md
  • docs/source/extend/testing/add-unit-tests-for-tools.md
  • docs/source/get-started/tutorials/add-tools-to-a-workflow.md
  • docs/source/improve-workflows/evaluate.md
  • docs/source/improve-workflows/optimizer.md
  • docs/source/index.md
  • docs/source/run-workflows/existing-agents/langgraph.md
  • examples/A2A/math_assistant_a2a/README.md
  • examples/A2A/math_assistant_a2a_protected/configs/config-client.yml
  • examples/dynamo_integration/README.md
  • examples/dynamo_integration/react_benchmark_agent/README.md
  • examples/dynamo_integration/react_benchmark_agent/src/react_benchmark_agent/DEVELOPER_NOTES.md
  • examples/dynamo_integration/react_benchmark_agent/src/react_benchmark_agent/evaluators/tsq_evaluator.py
  • examples/dynamo_integration/react_benchmark_agent/tests/test_tsq_formula.py
  • packages/nvidia_nat_core/src/nat/builder/per_user_workflow_builder.py
  • packages/nvidia_nat_core/src/nat/builder/workflow_builder.py
  • packages/nvidia_nat_core/src/nat/cli/commands/workflow/templates/workflow.py.j2
  • packages/nvidia_nat_core/src/nat/plugin_api.py
  • packages/nvidia_nat_core/tests/nat/test_plugin_api.py
  • packages/nvidia_nat_core/tests/nat/tools/test_tool_test_runner.py
  • packages/nvidia_nat_eval/src/nat/plugins/eval/runtime/builder.py
  • packages/nvidia_nat_langchain/src/nat/plugins/langchain/tools/exa_internet_search.py
  • packages/nvidia_nat_langchain/src/nat/plugins/langchain/tools/tavily_internet_search.py
  • packages/nvidia_nat_test/src/nat/test/tool_test_runner.py

Comment thread docs/source/extend/custom-components/custom-functions/functions.md Outdated
Comment thread docs/source/extend/custom-components/custom-functions/functions.md Outdated
Comment thread docs/source/extend/plugin-api.md Outdated
Comment thread docs/source/extend/plugin-api.md Outdated
Comment thread packages/nvidia_nat_core/tests/nat/test_plugin_api.py Outdated
@bbednarski9 bbednarski9 added non-breaking Non-breaking change DO NOT MERGE PR should not be merged; see PR for details doc Improvements or additions to documentation feature request New feature or request and removed feature request New feature or request labels May 19, 2026
@bbednarski9 bbednarski9 changed the title improvment: Stabilize the public plugin authoring API plugin docs: Stabilize the public plugin authoring API May 19, 2026
@mnajafian-nv
Copy link
Copy Markdown
Contributor

I did a deeper pass on this. I agree with the direction: nat.plugin_api is the right move for giving Tavily/3P plugins a stable import path and getting them off nat.cli.register_workflow, nat.builder.*, and direct nat.data_models.*.

The scope mostly looks right to me, especially deferring front ends, logging, registry handlers, optimizer hooks, finetuning, and TTC.

I left a few line comments on the places I think would make the public contract sharper before calling it stable: the Builder boundary, telemetry scope, entry point guidance, and one consumer-style contract test. The biggest one is Builder: if the concrete type stays exported, I think the stable subset external plugins can rely on should be explicit in docs/tests.

Comment thread packages/nvidia_nat_core/src/nat/plugin_api.py
Comment thread docs/source/extend/plugin-api.md Outdated
Comment thread packages/nvidia_nat_core/tests/nat/test_plugin_api.py
Comment thread docs/source/extend/plugins.md
@coderabbitai coderabbitai Bot removed the DO NOT MERGE PR should not be merged; see PR for details label May 28, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 28, 2026

Actionable comments posted: 0

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
packages/nvidia_nat_core/tests/nat/test_plugin_api.py (2)

389-406: 💤 Low value

Consider filtering to methods only or clarifying the attribute scope.

The test docstring and set names (STABLE_BUILDER_METHODS, DEFERRED_BUILDER_METHODS) reference "method surface" (line 390), but line 393 checks all non-underscore attributes via vars(Builder), which includes properties, class variables, and other descriptors in addition to methods.

Currently this works if Builder has only methods in its public surface, but if Builder later adds a public property or class variable, the test will fail with a potentially confusing error message (since the sets are named *_METHODS).

♻️ Options for clarity

Option 1: Filter to callables only

-    actual = {name for name in vars(Builder) if not name.startswith("_")}
+    actual = {name for name in vars(Builder) if not name.startswith("_") and callable(getattr(Builder, name))}

Option 2: Rename sets and clarify scope
If the intent is to track all public surface area (not just methods), rename the sets to STABLE_BUILDER_SURFACE / DEFERRED_BUILDER_SURFACE and update the docstring to clarify "public surface" instead of "public method surface".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nvidia_nat_core/tests/nat/test_plugin_api.py` around lines 389 -
406, The test currently uses vars(Builder) to collect public names but the
STABLE_BUILDER_METHODS/DEFERRED_BUILDER_METHODS sets are named for methods, so
update the test to ensure the compared surface matches the intent: either
(preferred) filter the collected names to methods/callables only (e.g., derive
actual = {name for name, val in vars(Builder).items() if not
name.startswith("_") and callable(val)}) so it only compares methods against
STABLE_BUILDER_METHODS and DEFERRED_BUILDER_METHODS, or if you intend to track
all public attributes, rename those sets to
STABLE_BUILDER_SURFACE/DEFERRED_BUILDER_SURFACE and update the docstring to say
“public surface” instead of “public method surface”; adjust references to
Builder, vars(Builder), STABLE_BUILDER_METHODS, and DEFERRED_BUILDER_METHODS
accordingly.

368-371: 💤 Low value

Jinja template handling is fragile for complex templates.

The simple regex substitution jinja_var.sub(r"\1", text) converts {{ var }} to var, which works for the current simple .py.j2 workflow template but will fail silently (via the SyntaxError catch at line 376) for templates containing Jinja control structures ({% if %}, {% for %}, filters, complex expressions).

For the single targeted template this is acceptable, but if additional complex Jinja templates are added to the paths list in the future, violations may go undetected.

♻️ Consider extracting Python code blocks from Jinja templates

A more robust approach would parse Jinja templates to extract only the Python code blocks (between {%- ... %} or as plain Python outside Jinja directives) rather than attempting global text substitution. Alternatively, document that only simple variable-substitution templates are supported and should be added to the paths list.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nvidia_nat_core/tests/nat/test_plugin_api.py` around lines 368 -
371, The current jinja handling uses jinja_var.sub(r"\1", text) when
file_path.name.endswith(".j2"), which will silently fail for complex templates
(caught by the existing SyntaxError handler); replace this fragile regex
approach by parsing the template with Jinja2 (e.g. use
jinja2.Environment().parse(text) and jinja2.meta or walk the AST) to extract
actual Python code blocks/variables to populate snippets, and keep the
SyntaxError handler but rely on the parser to produce accurate snippets for `{%
... %}` and `{{ ... }}` constructs; alternatively, if you prefer the simpler
route, document in tests that only simple variable-substitution `.j2` templates
are supported and restrict paths accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/nvidia_nat_core/tests/nat/test_plugin_api.py`:
- Around line 389-406: The test currently uses vars(Builder) to collect public
names but the STABLE_BUILDER_METHODS/DEFERRED_BUILDER_METHODS sets are named for
methods, so update the test to ensure the compared surface matches the intent:
either (preferred) filter the collected names to methods/callables only (e.g.,
derive actual = {name for name, val in vars(Builder).items() if not
name.startswith("_") and callable(val)}) so it only compares methods against
STABLE_BUILDER_METHODS and DEFERRED_BUILDER_METHODS, or if you intend to track
all public attributes, rename those sets to
STABLE_BUILDER_SURFACE/DEFERRED_BUILDER_SURFACE and update the docstring to say
“public surface” instead of “public method surface”; adjust references to
Builder, vars(Builder), STABLE_BUILDER_METHODS, and DEFERRED_BUILDER_METHODS
accordingly.
- Around line 368-371: The current jinja handling uses jinja_var.sub(r"\1",
text) when file_path.name.endswith(".j2"), which will silently fail for complex
templates (caught by the existing SyntaxError handler); replace this fragile
regex approach by parsing the template with Jinja2 (e.g. use
jinja2.Environment().parse(text) and jinja2.meta or walk the AST) to extract
actual Python code blocks/variables to populate snippets, and keep the
SyntaxError handler but rely on the parser to produce accurate snippets for `{%
... %}` and `{{ ... }}` constructs; alternatively, if you prefer the simpler
route, document in tests that only simple variable-substitution `.j2` templates
are supported and restrict paths accordingly.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: dc37832b-6431-40d1-9fcc-d5b6c64450f5

📥 Commits

Reviewing files that changed from the base of the PR and between 015ecec and 90479d5.

📒 Files selected for processing (5)
  • docs/source/extend/custom-components/custom-functions/functions.md
  • docs/source/extend/plugin-api.md
  • docs/source/extend/plugins.md
  • packages/nvidia_nat_core/src/nat/plugin_api.py
  • packages/nvidia_nat_core/tests/nat/test_plugin_api.py
✅ Files skipped from review due to trivial changes (2)
  • docs/source/extend/plugins.md
  • docs/source/extend/custom-components/custom-functions/functions.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • packages/nvidia_nat_core/src/nat/plugin_api.py
  • docs/source/extend/plugin-api.md

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 28, 2026

Actionable comments posted: 0

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
@bbednarski9 bbednarski9 requested a review from mnajafian-nv May 28, 2026 19:31
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 28, 2026

Actionable comments posted: 0

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Comment thread docs/source/extend/plugin-api.md Outdated
Comment thread docs/source/extend/plugins.md Outdated
Comment thread docs/source/extend/plugin-api.md Outdated
Comment thread docs/source/extend/plugin-api.md Outdated
Copy link
Copy Markdown
Contributor

@mnajafian-nv mnajafian-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Bryan, this is much closer now. The Builder concern and consumer-style tests look addressed to me.

I left a few remaining comments focused on public API contract consistency across the stability table, subsystem guides, and the actual nat.plugin_api facade. I do not think these need to block approval, but I would like them cleaned up before merge so third-party plugin authors get one coherent contract.

Approved from my side assuming those contract-alignment comments are addressed.

… Add retriever, retoutput and document ro the plugin api

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
  - Split middleware into stable basic/function middleware vs provisional dynamic/runtime introspection.
  - Narrowed the builder row to the tested stable method subset.
  - Added evaluator-guide note that only registration/config/EvaluatorInfo are stable facade APIs.
  - Updated nat.plugin_api docstring to include deferred auth builder methods.

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
@bbednarski9
Copy link
Copy Markdown
Contributor Author

/merge

@rapids-bot rapids-bot Bot merged commit 91e5837 into NVIDIA:develop May 28, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc Improvements or additions to documentation non-breaking Non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Public API surface for third-party integrations

2 participants