diff --git a/README.md b/README.md index e15f616..de2b995 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ All packages are extensions to the [UiPath Python SDK](https://github.com/UiPath Build agents using the [LlamaIndex SDK](https://www.llamaindex.ai/): +- [README](packages/uipath-llamaindex/README.md) - [Docs](https://uipath.github.io/uipath-python/llamaindex/quick_start/) - [Samples](packages/uipath-llamaindex/samples/) @@ -23,6 +24,7 @@ Build agents using the [LlamaIndex SDK](https://www.llamaindex.ai/): Build agents using the [OpenAI Agents SDK](https://github.com/openai/openai-agents-python): +- [README](packages/uipath-openai-agents/README.md) - [Docs](https://uipath.github.io/uipath-python/openai-agents/quick_start/) - [Samples](packages/uipath-openai-agents/samples/) diff --git a/packages/uipath-openai-agents/README.md b/packages/uipath-openai-agents/README.md index 293900b..e6ae954 100644 --- a/packages/uipath-openai-agents/README.md +++ b/packages/uipath-openai-agents/README.md @@ -1,6 +1,19 @@ -# UiPath OpenAI Agents SDK +# UiPath OpenAI Agents Python SDK -Build intelligent AI agents with OpenAI's Agents framework and UiPath. +[![PyPI - Version](https://img.shields.io/pypi/v/uipath-openai-agents)](https://pypi.org/project/uipath-openai-agents/) +[![PyPI downloads](https://img.shields.io/pypi/dm/uipath-openai-agents.svg)](https://pypi.org/project/uipath-openai-agents/) +[![Python versions](https://img.shields.io/pypi/pyversions/uipath-openai-agents.svg)](https://pypi.org/project/uipath-openai-agents/) + +A Python SDK that enables developers to build and deploy OpenAI Agents to the UiPath Cloud Platform. It provides programmatic interaction with UiPath Cloud Platform services. + +This package is an extension to the [UiPath Python SDK](https://github.com/UiPath/uipath-python) and implements the [UiPath Runtime Protocol](https://github.com/UiPath/uipath-runtime-python). + +Check out these [sample projects](https://github.com/UiPath/uipath-integrations-python/tree/main/packages/uipath-openai-agents/samples) to see the SDK in action. + +## Requirements + +- Python 3.11 or higher +- UiPath Automation Cloud account ## Installation @@ -8,25 +21,105 @@ Build intelligent AI agents with OpenAI's Agents framework and UiPath. pip install uipath-openai-agents ``` -## Quick Start +using `uv`: + +```bash +uv add uipath-openai-agents +``` + +## Configuration + +### Environment Variables + +Create a `.env` file in your project root with the following variables: + +``` +UIPATH_URL=https://cloud.uipath.com/ACCOUNT_NAME/TENANT_NAME +UIPATH_ACCESS_TOKEN=YOUR_TOKEN_HERE +``` + +## Command Line Interface (CLI) + +The SDK provides a command-line interface for creating, packaging, and deploying OpenAI Agents: + +### Authentication + +```bash +uipath auth +``` + +This command opens a browser for authentication and creates/updates your `.env` file with the proper credentials. + +### Initialize a Project + +```bash +uipath init +``` + +Running `uipath init` will process the agent definitions in the `openai_agents.json` file and create the corresponding `entry-points.json` file needed for deployment. + +For more details on the configuration format, see the [UiPath configuration specifications](https://github.com/UiPath/uipath-python/blob/main/specs/README.md). + +### Debug a Project + +```bash +uipath run AGENT [INPUT] +``` + +Executes the agent with the provided JSON input arguments. + +### Package a Project + +```bash +uipath pack +``` + +Packages your project into a `.nupkg` file that can be deployed to UiPath. + +**Note:** Your `pyproject.toml` must include: + +- A description field (avoid characters: &, <, >, ", ', ;) +- Author information + +Example: + +```toml +description = "Your package description" +authors = [{name = "Your Name", email = "your.email@example.com"}] +``` + +### Publish a Package + +```bash +uipath publish +``` + +Publishes the most recently created package to your UiPath Orchestrator. + +## Project Structure + +To properly use the CLI for packaging and publishing, your project should include: -See the [main repository documentation](../../docs/) for getting started guides and examples. +- A `pyproject.toml` file with project metadata +- A `openai_agents.json` file with your agent definitions (e.g., `"agents": {"agent": "main.py:agent"}`) +- A `entry-points.json` file (generated by `uipath init`) +- A `bindings.json` file (generated by `uipath init`) to configure resource overrides +- Any Python files needed for your automation -## Features +## Development -- **OpenAI Agents Integration**: Build agents using OpenAI's native Agents framework -- **Agent Orchestration**: Multi-agent coordination and communication -- **State Management**: Persistent agent state with SQLite sessions -- **UiPath Integration**: Seamless integration with UiPath runtime and tooling +### Developer Tools -## Status +Check out [uipath-dev](https://github.com/uipath/uipath-dev-python) - an interactive terminal application for building, testing, and debugging UiPath Python runtimes, agents, and automation scripts. -⚠️ **Early Development**: This package is in early development (v0.1.0). APIs may change as the OpenAI Agents framework evolves. +### Setting Up a Development Environment -## Documentation +Please read our [contribution guidelines](https://github.com/UiPath/uipath-integrations-python/packages/uipath-openai-agents/blob/main/CONTRIBUTING.md) before submitting a pull request. -Full documentation is available in the [main repository](https://github.com/UiPath/uipath-llamaindex-python). +### Special Thanks -## License +A huge thank-you to the open-source community and the maintainers of the libraries that make this project possible: -See [LICENSE](../../LICENSE) in the repository root. +- [OpenAI](https://github.com/openai/openai-python) for providing a powerful framework for building AI agents. +- [OpenInference](https://github.com/Arize-ai/openinference) for observability and instrumentation support. +- [Pydantic](https://github.com/pydantic/pydantic) for reliable, typed configuration and validation. diff --git a/packages/uipath-openai-agents/docs/quick_start.md b/packages/uipath-openai-agents/docs/quick_start.md index dc26b8c..13410db 100644 --- a/packages/uipath-openai-agents/docs/quick_start.md +++ b/packages/uipath-openai-agents/docs/quick_start.md @@ -106,7 +106,7 @@ Generate your first UiPath OpenAI agent: ✓ Created 'pyproject.toml' file. 🔧 Please ensure to define OPENAI_API_KEY in your .env file. 💡 Initialize project: uipath init -💡 Run agent: uipath run agent '{"message": "Hello"}' +💡 Run agent: uipath run agent '{"messages": "Hello"}' ``` This command creates the following files: @@ -173,7 +173,7 @@ Execute the agent with a sample input: ```shell -> uipath run agent '{"message": "Hello"}' +> uipath run agent '{"messages": "Hello"}' {'response': 'Hello! How can I help you today?', 'agent_used': 'main'} ✓ Successful execution. ``` @@ -185,19 +185,19 @@ Depending on the shell you are using, it may be necessary to escape the input js /// tab | Bash/ZSH/PowerShell ```console -uipath run agent '{"message": "Hello"}' +uipath run agent '{"messages": "Hello"}' ``` /// /// tab | Windows CMD ```console -uipath run agent "{""message"": ""Hello""}" +uipath run agent "{""messages"": ""Hello""}" ``` /// /// tab | Windows PowerShell ```console -uipath run agent '{\"message\":\"Hello\"}' +uipath run agent '{\"messages\":\"Hello\"}' ``` /// @@ -215,7 +215,7 @@ The `run` command can also take a .json file as an input. You can create a file ```json { - "message": "Hello" + "messages": "Hello" } ``` @@ -275,7 +275,7 @@ Set the environment variables using the provided link. ```shell -> uipath invoke agent '{"message": "Hello"}' +> uipath invoke agent '{"messages": "Hello"}' ⠴ Loading configuration ... ⠴ Starting job ... ✨ Job started successfully! diff --git a/packages/uipath-openai-agents/pyproject.toml b/packages/uipath-openai-agents/pyproject.toml index df2c464..2ca503c 100644 --- a/packages/uipath-openai-agents/pyproject.toml +++ b/packages/uipath-openai-agents/pyproject.toml @@ -70,7 +70,8 @@ plugins = [ "pydantic.mypy" ] exclude = [ - "samples/.*" + "samples/.*", + "testcases/.*" ] follow_imports = "silent" warn_redundant_casts = true diff --git a/packages/uipath-openai-agents/samples/agent-as-tools/input.json b/packages/uipath-openai-agents/samples/agent-as-tools/input.json index 3cbe087..e276566 100644 --- a/packages/uipath-openai-agents/samples/agent-as-tools/input.json +++ b/packages/uipath-openai-agents/samples/agent-as-tools/input.json @@ -1,3 +1,3 @@ { - "message": "Tell me a joke" + "messages": "Tell me a joke" } \ No newline at end of file diff --git a/packages/uipath-openai-agents/samples/agent-as-tools/main.py b/packages/uipath-openai-agents/samples/agent-as-tools/main.py index 02d533a..c0a52fe 100644 --- a/packages/uipath-openai-agents/samples/agent-as-tools/main.py +++ b/packages/uipath-openai-agents/samples/agent-as-tools/main.py @@ -3,6 +3,7 @@ from pydantic import BaseModel, Field from uipath_openai_agents.chat import UiPathChatOpenAI +from uipath_openai_agents.chat.supported_models import OpenAIModels """ This example shows the agents-as-tools pattern adapted for UiPath coded agents. @@ -32,7 +33,7 @@ def main() -> Agent: """Configure UiPath OpenAI client and return the orchestrator agent.""" # Configure UiPath OpenAI client for agent execution # This routes all OpenAI API calls through UiPath's LLM Gateway - MODEL = "gpt-4o-2024-11-20" + MODEL = OpenAIModels.gpt_5_1_2025_11_13 uipath_openai_client = UiPathChatOpenAI(model_name=MODEL) _openai_shared.set_default_openai_client(uipath_openai_client.async_client) diff --git a/packages/uipath-openai-agents/samples/rag-assistant/.agent/CLI_REFERENCE.md b/packages/uipath-openai-agents/samples/rag-assistant/.agent/CLI_REFERENCE.md index c8031c0..f33b409 100644 --- a/packages/uipath-openai-agents/samples/rag-assistant/.agent/CLI_REFERENCE.md +++ b/packages/uipath-openai-agents/samples/rag-assistant/.agent/CLI_REFERENCE.md @@ -59,8 +59,10 @@ uv run uipath init --infer-bindings | `--input-file` | value | `Sentinel.UNSET` | Alias for '-f/--file' arguments | | `--output-file` | value | `Sentinel.UNSET` | File path where the output will be written | | `--trace-file` | value | `Sentinel.UNSET` | File path where the trace spans will be written (JSON Lines format) | +| `--state-file` | value | `Sentinel.UNSET` | File path where the state file is stored for persisting execution state. If not provided, a temporary file will be used. | | `--debug` | flag | false | Enable debugging with debugpy. The process will wait for a debugger to attach. | | `--debug-port` | value | `5678` | Port for the debug server (default: 5678) | +| `--keep-state-file` | flag | false | Keep the temporary state file even when not resuming and no job id is provided | **Usage Examples:** @@ -102,6 +104,7 @@ uv run uipath run --resume trace_file: File path where traces will be written in JSONL format max_llm_concurrency: Maximum concurrent LLM requests input_overrides: Input field overrides mapping (direct field override with deep merge) + resume: Resume execution from a previous suspended state **Arguments:** @@ -124,6 +127,7 @@ uv run uipath run --resume | `--model-settings-id` | value | `"default"` | Model settings ID from evaluation set to override agent settings (default: 'default') | | `--trace-file` | value | `Sentinel.UNSET` | File path where traces will be written in JSONL format | | `--max-llm-concurrency` | value | `20` | Maximum concurrent LLM requests (default: 20) | +| `--resume` | flag | false | Resume execution from a previous suspended state | **Usage Examples:** diff --git a/packages/uipath-openai-agents/samples/rag-assistant/.agent/SDK_REFERENCE.md b/packages/uipath-openai-agents/samples/rag-assistant/.agent/SDK_REFERENCE.md index 8373939..637a26d 100644 --- a/packages/uipath-openai-agents/samples/rag-assistant/.agent/SDK_REFERENCE.md +++ b/packages/uipath-openai-agents/samples/rag-assistant/.agent/SDK_REFERENCE.md @@ -16,6 +16,25 @@ sdk = UiPath() sdk = UiPath(base_url="https://cloud.uipath.com/...", secret="your_token") ``` +### Agenthub + +Agenthub service + +```python +# Fetch available models from LLM Gateway discovery endpoint. +sdk.agenthub.get_available_llm_models(headers: dict[str, Any] | None=None) -> list[uipath.platform.agenthub.agenthub.LlmModel] + +# Asynchronously fetch available models from LLM Gateway discovery endpoint. +sdk.agenthub.get_available_llm_models_async(headers: dict[str, Any] | None=None) -> list[uipath.platform.agenthub.agenthub.LlmModel] + +# Start a system agent job. +sdk.agenthub.invoke_system_agent(agent_name: str, entrypoint: str, input_arguments: dict[str, Any] | None=None, folder_key: str | None=None, folder_path: str | None=None, headers: dict[str, Any] | None=None) -> str + +# Asynchronously start a system agent and return the job. +sdk.agenthub.invoke_system_agent_async(agent_name: str, entrypoint: str, input_arguments: dict[str, Any] | None=None, folder_key: str | None=None, folder_path: str | None=None, headers: dict[str, Any] | None=None) -> str + +``` + ### Api Client Api Client service @@ -31,6 +50,12 @@ service = sdk.api_client Assets service ```python +# List assets using OData API with offset-based pagination. +sdk.assets.list(folder_path: Optional[str]=None, folder_key: Optional[str]=None, filter: Optional[str]=None, orderby: Optional[str]=None, skip: int=0, top: int=100) -> uipath.platform.common.paging.PagedResult[uipath.platform.orchestrator.assets.Asset] + +# Asynchronously list assets using OData API with offset-based pagination. +sdk.assets.list_async(folder_path: Optional[str]=None, folder_key: Optional[str]=None, filter: Optional[str]=None, orderby: Optional[str]=None, skip: int=0, top: int=100) -> uipath.platform.common.paging.PagedResult[uipath.platform.orchestrator.assets.Asset] + # Retrieve an asset by its name. sdk.assets.retrieve(name: str, folder_key: Optional[str]=None, folder_path: Optional[str]=None) -> uipath.platform.orchestrator.assets.UserAsset | uipath.platform.orchestrator.assets.Asset @@ -505,7 +530,7 @@ Llm service ```python # Generate chat completions using UiPath's normalized LLM Gateway API. -sdk.llm.chat_completions(messages: list[dict[str, str]] | list[tuple[str, str]], model: str="gpt-4o-mini-2024-07-18", max_tokens: int=4096, temperature: float=0, n: int=1, frequency_penalty: float=0, presence_penalty: float=0, top_p: float | None=1, top_k: int | None=None, tools: list[uipath.platform.chat.llm_gateway.ToolDefinition] | None=None, tool_choice: Union[uipath.platform.chat.llm_gateway.AutoToolChoice, uipath.platform.chat.llm_gateway.RequiredToolChoice, uipath.platform.chat.llm_gateway.SpecificToolChoice, Literal['auto', 'none'], NoneType]=None, response_format: dict[str, Any] | type[pydantic.main.BaseModel] | None=None, api_version: str="2024-08-01-preview") +sdk.llm.chat_completions(messages: list[dict[str, str]] | list[tuple[str, str]], model: str="gpt-4.1-mini-2025-04-14", max_tokens: int=4096, temperature: float=0, n: int=1, frequency_penalty: float=0, presence_penalty: float=0, top_p: float | None=1, top_k: int | None=None, tools: list[uipath.platform.chat.llm_gateway.ToolDefinition] | None=None, tool_choice: Union[uipath.platform.chat.llm_gateway.AutoToolChoice, uipath.platform.chat.llm_gateway.RequiredToolChoice, uipath.platform.chat.llm_gateway.SpecificToolChoice, Literal['auto', 'none'], NoneType]=None, response_format: dict[str, Any] | type[pydantic.main.BaseModel] | None=None, api_version: str="2024-08-01-preview") ``` @@ -515,7 +540,7 @@ Llm Openai service ```python # Generate chat completions using UiPath's LLM Gateway service. -sdk.llm_openai.chat_completions(messages: list[dict[str, str]], model: str="gpt-4o-mini-2024-07-18", max_tokens: int=4096, temperature: float=0, response_format: dict[str, Any] | type[pydantic.main.BaseModel] | None=None, api_version: str="2024-10-21") +sdk.llm_openai.chat_completions(messages: list[dict[str, str]], model: str="gpt-4.1-mini-2025-04-14", max_tokens: int=4096, temperature: float=0, response_format: dict[str, Any] | type[pydantic.main.BaseModel] | None=None, api_version: str="2024-10-21") # Generate text embeddings using UiPath's LLM Gateway service. sdk.llm_openai.embeddings(input: str, embedding_model: str="text-embedding-ada-002", openai_api_version: str="2024-10-21") diff --git a/packages/uipath-openai-agents/samples/rag-assistant/.claude/commands/eval.md b/packages/uipath-openai-agents/samples/rag-assistant/.claude/commands/eval.md new file mode 100644 index 0000000..15fa04f --- /dev/null +++ b/packages/uipath-openai-agents/samples/rag-assistant/.claude/commands/eval.md @@ -0,0 +1,287 @@ +--- +allowed-tools: Bash, Read, Write, Edit, Glob +description: Create and run agent evaluations +--- + +I'll help you create and run evaluations for your UiPath agent. + +## Step 1: Check project setup + +Let me check your project structure: + +!ls -la evaluations/ entry-points.json 2>/dev/null || echo "NEEDS_SETUP" + +# Check if schemas might be stale (main.py newer than entry-points.json) +!if [ -f main.py ] && [ -f entry-points.json ] && [ main.py -nt entry-points.json ]; then echo "SCHEMAS_MAY_BE_STALE"; fi + +### If NEEDS_SETUP + +If `entry-points.json` doesn't exist, initialize the project first: + +!uv run uipath init + +Then re-run this skill. + +### If SCHEMAS_MAY_BE_STALE + +Your `main.py` is newer than `entry-points.json`. Refresh schemas: + +!uv run uipath init --no-agents-md-override + +## Step 2: What would you like to do? + +1. **Create new eval set** - Set up evaluations from scratch +2. **Add test case** - Add a test to existing eval set +3. **Run evaluations** - Execute tests and see results +4. **Analyze failures** - Debug failing tests + +--- + +## Creating an Eval Set + +First, create the directory structure: + +!mkdir -p evaluations/eval-sets evaluations/evaluators + +Read the agent's Input/Output schema from entry-points.json to understand the data types. + +### Evaluator Selection Guide + +| If your output is... | Use this evaluator | evaluatorTypeId | +|---------------------|-------------------|-----------------| +| Exact string/number | `ExactMatchEvaluator` | `uipath-exact-match` | +| Contains key phrases | `ContainsEvaluator` | `uipath-contains` | +| Semantically correct | `LLMJudgeOutputEvaluator` | `uipath-llm-judge-output-semantic-similarity` | +| JSON with numbers | `JsonSimilarityEvaluator` | `uipath-json-similarity` | + +### Step 1: Create Evaluator Config Files + +**Each evaluator needs a JSON config file** in `evaluations/evaluators/`. + +**ExactMatchEvaluator** (`evaluations/evaluators/exact-match.json`): +```json +{ + "version": "1.0", + "id": "ExactMatchEvaluator", + "name": "ExactMatchEvaluator", + "description": "Checks for exact output match", + "evaluatorTypeId": "uipath-exact-match", + "evaluatorConfig": { + "name": "ExactMatchEvaluator", + "targetOutputKey": "*" + } +} +``` + +**LLMJudgeOutputEvaluator** (`evaluations/evaluators/llm-judge-output.json`): +```json +{ + "version": "1.0", + "id": "LLMJudgeOutputEvaluator", + "name": "LLMJudgeOutputEvaluator", + "description": "Uses LLM to judge semantic similarity", + "evaluatorTypeId": "uipath-llm-judge-output-semantic-similarity", + "evaluatorConfig": { + "name": "LLMJudgeOutputEvaluator", + "model": "gpt-4o-mini-2024-07-18" + } +} +``` + +**JsonSimilarityEvaluator** (`evaluations/evaluators/json-similarity.json`): +```json +{ + "version": "1.0", + "id": "JsonSimilarityEvaluator", + "name": "JsonSimilarityEvaluator", + "description": "Compares JSON structures", + "evaluatorTypeId": "uipath-json-similarity", + "evaluatorConfig": { + "name": "JsonSimilarityEvaluator", + "targetOutputKey": "*" + } +} +``` + +**ContainsEvaluator** (`evaluations/evaluators/contains.json`): +```json +{ + "version": "1.0", + "id": "ContainsEvaluator", + "name": "ContainsEvaluator", + "description": "Checks if output contains text", + "evaluatorTypeId": "uipath-contains", + "evaluatorConfig": { + "name": "ContainsEvaluator" + } +} +``` + +### Step 2: Create Eval Set + +**Eval Set Template** (`evaluations/eval-sets/default.json`): +```json +{ + "version": "1.0", + "id": "default-eval-set", + "name": "Default Evaluation Set", + "evaluatorRefs": ["ExactMatchEvaluator"], + "evaluations": [ + { + "id": "test-1", + "name": "Test description", + "inputs": { + "field": "value" + }, + "evaluationCriterias": { + "ExactMatchEvaluator": { + "expectedOutput": { + "result": "expected value" + } + } + } + } + ] +} +``` + +**Important notes:** +- `evaluatorRefs` must list ALL evaluators used in any test case +- Each evaluator in `evaluatorRefs` needs a matching JSON config in `evaluations/evaluators/` +- `evaluationCriterias` keys must match entries in `evaluatorRefs` +- Use `expectedOutput` for most evaluators +- LLM evaluators need `model` in their config. Available models are defined in the SDK's `ChatModels` class (`uipath.platform.chat.ChatModels`): + - `gpt-4o-mini-2024-07-18` (recommended for cost-efficiency) + - `gpt-4o-2024-08-06` (higher quality, higher cost) + - `o3-mini-2025-01-31` (latest reasoning model) + - Model availability varies by region and tenant configuration + - Check your UiPath Automation Cloud portal under AI Trust Layer for available models in your region + +--- + +## Adding a Test Case + +When adding a test to an existing eval set: + +1. Read the existing eval set +2. Check which evaluators are in `evaluatorRefs` +3. Add the new test to `evaluations` array +4. If using a new evaluator, add it to `evaluatorRefs` + +### Test Case Template + +```json +{ + "id": "test-{n}", + "name": "Description of what this tests", + "inputs": { }, + "evaluationCriterias": { + "EvaluatorName": { + "expectedOutput": { } + } + } +} +``` + +--- + +## Running Evaluations + +First, read entry-points.json to get the entrypoint name (e.g., `main`): + +!uv run uipath eval main evaluations/eval-sets/default.json --output-file eval-results.json + +**Note:** Replace `main` with your actual entrypoint from entry-points.json. + +### Analyze Results + +After running, read `eval-results.json` and show: +- Pass/fail summary table +- For failures: expected vs actual output +- Suggestions for fixing or changing evaluators + +### Results Format + +```json +{ + "evaluationSetResults": [{ + "evaluationRunResults": [ + { + "evaluationId": "test-1", + "evaluatorId": "ExactMatchEvaluator", + "result": { "score": 1.0 }, + "errorMessage": null + } + ] + }] +} +``` + +- Score 1.0 = PASS +- Score < 1.0 = FAIL (show expected vs actual) +- errorMessage present = ERROR (show message) + +--- + +## Evaluator Reference + +### Deterministic Evaluators + +**ExactMatchEvaluator** - Exact output matching +```json +"ExactMatchEvaluator": { + "expectedOutput": { "result": "exact value" } +} +``` + +**ContainsEvaluator** - Output contains substring +```json +"ContainsEvaluator": { + "searchText": "must contain this" +} +``` + +**JsonSimilarityEvaluator** - JSON comparison with tolerance +```json +"JsonSimilarityEvaluator": { + "expectedOutput": { "value": 10.0 } +} +``` + +### LLM-Based Evaluators + +**LLMJudgeOutputEvaluator** - Semantic correctness +```json +"LLMJudgeOutputEvaluator": { + "expectedOutput": { "summary": "Expected semantic meaning" } +} +``` + +**LLMJudgeTrajectoryEvaluator** - Validate agent reasoning +```json +"LLMJudgeTrajectoryEvaluator": { + "expectedAgentBehavior": "The agent should first fetch data, then process it" +} +``` + +--- + +## Common Issues + +### "No evaluations found" +- Check `evaluations/eval-sets/` directory exists +- Verify JSON file is valid + +### Evaluator not found +- Each evaluator needs a JSON config file in `evaluations/evaluators/` +- Config file must have correct `evaluatorTypeId` (see templates above) +- Config file must have `name` field at root level +- LLM evaluators need `model` in `evaluatorConfig` + +### Evaluator skipped +- Ensure evaluator is listed in root `evaluatorRefs` array +- Check evaluator config file exists in `evaluations/evaluators/` + +### Schema mismatch +- Run `uv run uipath init --no-agents-md-override` to refresh schemas +- Check `entry-points.json` matches your Input/Output models diff --git a/packages/uipath-openai-agents/samples/rag-assistant/.claude/commands/new-agent.md b/packages/uipath-openai-agents/samples/rag-assistant/.claude/commands/new-agent.md new file mode 100644 index 0000000..b1d0518 --- /dev/null +++ b/packages/uipath-openai-agents/samples/rag-assistant/.claude/commands/new-agent.md @@ -0,0 +1,103 @@ +--- +allowed-tools: Bash, Read, Write, Edit, Glob +description: Create a new UiPath coded agent from a description +--- + +I'll help you create a new UiPath coded agent. + +## Step 1: Check existing project + +Let me check if this is an existing UiPath project: + +!ls uipath.json main.py 2>/dev/null || echo "NEW_PROJECT" + +## Step 2: Gather requirements + +**What should this agent do?** + +Please describe: + +- What inputs it needs (e.g., "a file path and bucket name") +- What it should accomplish (e.g., "process CSV data") +- What outputs it should return (e.g., "total count and status") + +I'll generate the agent structure based on your description. + +## Step 3: Generate agent + +After you describe the agent, I will: + +1. Create `main.py` with Input/Output Pydantic models and `async def main()` +2. Add entrypoint to `uipath.json` under `"functions": {"agent_name": "main.py:main"}` +3. Run `uv run uipath init --no-agents-md-override` to generate schemas + +**Template structure** (from .agent/REQUIRED_STRUCTURE.md): + +```python +from pydantic import BaseModel +from uipath.platform import UiPath + +class Input(BaseModel): + """Input fields for the agent.""" + # Fields based on your description + pass + + +class Output(BaseModel): + """Output fields returned by the agent.""" + # Fields based on your description + pass + + +async def main(input: Input) -> Output: + """Main entry point for the agent. + + Args: + input: The input data for the agent. + + Returns: + The output data from the agent. + """ + + uipath = UiPath() + + # TODO: Implement agent logic + return Output() +``` + +**Important notes:** + +- Use `async def main` - many SDK methods are async +- Initialize `UiPath()` inside the function, not at module level +- After creating main.py, add entrypoint to `uipath.json` under `"functions"` + +## Step 4: Update entry-point schemas + +After creating main.py, regenerate the schemas: + +!uv run uipath init --no-agents-md-override + +## Step 5: Verify + +Quick test to verify the setup: + +!uv run uipath run main '{}' 2>&1 | head -30 + +## Summary + +Once complete, you'll have: + +| File | Purpose | +| ------------------- | ----------------------------------- | +| `main.py` | Agent code with Input/Output models | +| `uipath.json` | Project configuration | +| `entry-points.json` | Entry point schemas | +| `bindings.json` | Resource bindings | +| `.agent/` | SDK and CLI reference docs | + +**Next steps:** + +1. Implement your logic in `main()` +2. Test: `uv run uipath run main '{"field": "value"}'` +3. Create `eval_set.json` for evaluations +4. Evaluate: `uv run uipath eval` diff --git a/packages/uipath-openai-agents/samples/rag-assistant/.env.example b/packages/uipath-openai-agents/samples/rag-assistant/.env.example new file mode 100644 index 0000000..4ed32aa --- /dev/null +++ b/packages/uipath-openai-agents/samples/rag-assistant/.env.example @@ -0,0 +1,3 @@ +# OpenAI API Key +# Get your API key from https://platform.openai.com/api-keys +OPENAI_API_KEY=sk-... diff --git a/packages/uipath-openai-agents/samples/rag-assistant/CLAUDE.md b/packages/uipath-openai-agents/samples/rag-assistant/CLAUDE.md index 43c994c..eef4bd2 100644 --- a/packages/uipath-openai-agents/samples/rag-assistant/CLAUDE.md +++ b/packages/uipath-openai-agents/samples/rag-assistant/CLAUDE.md @@ -1 +1 @@ -@AGENTS.md +@AGENTS.md \ No newline at end of file diff --git a/packages/uipath-openai-agents/samples/rag-assistant/entry-points.json b/packages/uipath-openai-agents/samples/rag-assistant/entry-points.json index a3d7a31..251fa31 100644 --- a/packages/uipath-openai-agents/samples/rag-assistant/entry-points.json +++ b/packages/uipath-openai-agents/samples/rag-assistant/entry-points.json @@ -4,12 +4,12 @@ "entryPoints": [ { "filePath": "agent", - "uniqueId": "31b69d3d-4ae1-4caa-be64-b2fe368590a7", + "uniqueId": "6d0e928f-cf8e-465e-9b63-cb972e231e19", "type": "agent", "input": { "type": "object", "properties": { - "message": { + "messages": { "anyOf": [ { "type": "string" @@ -21,12 +21,12 @@ } } ], - "title": "Message", - "description": "User message(s) to send to the agent" + "title": "Messages", + "description": "User messages to send to the agent" } }, "required": [ - "message" + "messages" ] }, "output": { diff --git a/packages/uipath-openai-agents/samples/rag-assistant/input.json b/packages/uipath-openai-agents/samples/rag-assistant/input.json index 4bb1e69..65686d1 100644 --- a/packages/uipath-openai-agents/samples/rag-assistant/input.json +++ b/packages/uipath-openai-agents/samples/rag-assistant/input.json @@ -1,3 +1,3 @@ { - "message": "Who is the best ai llm?" + "messages": "What is the answer to life, the universe, and everything?" } \ No newline at end of file diff --git a/packages/uipath-openai-agents/samples/rag-assistant/main.py b/packages/uipath-openai-agents/samples/rag-assistant/main.py index 3c969f2..4d944cb 100644 --- a/packages/uipath-openai-agents/samples/rag-assistant/main.py +++ b/packages/uipath-openai-agents/samples/rag-assistant/main.py @@ -1,27 +1,20 @@ """RAG Assistant sample - Basic Agent with Chat. -This sample demonstrates a basic OpenAI agent using the Agents SDK framework with UiPath integration. +This sample demonstrates a basic OpenAI agent using the Agents SDK framework with direct OpenAI client. Features: - OpenAI Agents SDK integration -- UiPath tracing with OpenTelemetry +- Direct OpenAI API client usage - Type-safe input/output with Pydantic models - Streaming responses support """ from agents import Agent -from agents.models import _openai_shared - -from uipath_openai_agents.chat import UiPathChatOpenAI def main() -> Agent: - """Configure UiPath OpenAI client and return the assistant agent.""" - # Configure UiPath OpenAI client for agent execution - # This routes all OpenAI API calls through UiPath's LLM Gateway - MODEL = "gpt-4o-2024-11-20" - uipath_openai_client = UiPathChatOpenAI(model_name=MODEL) - _openai_shared.set_default_openai_client(uipath_openai_client.async_client) + """Return the assistant agent.""" + MODEL = "gpt-5.1-2025-11-13" # Define the assistant agent assistant_agent = Agent( @@ -38,4 +31,3 @@ def main() -> Agent: ) return assistant_agent - return assistant_agent diff --git a/packages/uipath-openai-agents/samples/triage-agent/README.md b/packages/uipath-openai-agents/samples/triage-agent/README.md index f639484..fdfb302 100644 --- a/packages/uipath-openai-agents/samples/triage-agent/README.md +++ b/packages/uipath-openai-agents/samples/triage-agent/README.md @@ -29,13 +29,13 @@ In this pattern: ```bash # Spanish message -uipath run main '{"message": "Hola, ¿cómo estás?"}' +uipath run main '{"messages": "Hola, ¿cómo estás?"}' # French message -uipath run main '{"message": "Bonjour, comment allez-vous?"}' +uipath run main '{"messages": "Bonjour, comment allez-vous?"}' # English message -uipath run main '{"message": "Hello, how are you?"}' +uipath run main '{"messages": "Hello, how are you?"}' ``` ### Configure in openai_agents.json @@ -53,7 +53,7 @@ uipath run main '{"message": "Hello, how are you?"}' ### Input ```python class Input(BaseModel): - message: str # User message in any language + messages: str # User message in any language ``` ### Output @@ -66,7 +66,7 @@ class Output(BaseModel): ## Example Execution ```bash -$ uipath run main '{"message": "Hola, ¿cómo estás?"}' +$ uipath run main '{"messages": "Hola, ¿cómo estás?"}' Processing message: Hola, ¿cómo estás? ¡Hola! Estoy muy bien, gracias. ¿Y tú, cómo estás? diff --git a/packages/uipath-openai-agents/samples/triage-agent/entry-points.json b/packages/uipath-openai-agents/samples/triage-agent/entry-points.json index bb0969f..0c34f69 100644 --- a/packages/uipath-openai-agents/samples/triage-agent/entry-points.json +++ b/packages/uipath-openai-agents/samples/triage-agent/entry-points.json @@ -9,7 +9,7 @@ "input": { "type": "object", "properties": { - "message": { + "messages": { "anyOf": [ { "type": "string" @@ -21,12 +21,12 @@ } } ], - "title": "Message", - "description": "User message(s) to send to the agent" + "title": "Messages", + "description": "User messages to send to the agent" } }, "required": [ - "message" + "messages" ] }, "output": { diff --git a/packages/uipath-openai-agents/samples/triage-agent/input.json b/packages/uipath-openai-agents/samples/triage-agent/input.json index b29e37c..d6f78c3 100644 --- a/packages/uipath-openai-agents/samples/triage-agent/input.json +++ b/packages/uipath-openai-agents/samples/triage-agent/input.json @@ -1,3 +1,3 @@ { - "message": "une blague." + "messages": "une blague." } diff --git a/packages/uipath-openai-agents/samples/triage-agent/main.py b/packages/uipath-openai-agents/samples/triage-agent/main.py index 97aa187..c7721e9 100644 --- a/packages/uipath-openai-agents/samples/triage-agent/main.py +++ b/packages/uipath-openai-agents/samples/triage-agent/main.py @@ -3,6 +3,7 @@ from pydantic import BaseModel from uipath_openai_agents.chat import UiPathChatOpenAI +from uipath_openai_agents.chat.supported_models import OpenAIModels """ This example shows the handoffs/routing pattern adapted for UiPath coded agents. @@ -24,7 +25,7 @@ def main() -> Agent: """Configure UiPath OpenAI client and return the triage agent.""" # Configure UiPath OpenAI client for agent execution # This routes all OpenAI API calls through UiPath's LLM Gateway - MODEL = "gpt-4o-2024-11-20" + MODEL = OpenAIModels.gpt_5_1_2025_11_13 uipath_openai_client = UiPathChatOpenAI(model_name=MODEL) _openai_shared.set_default_openai_client(uipath_openai_client.async_client) diff --git a/packages/uipath-openai-agents/src/uipath_openai_agents/__init__.py b/packages/uipath-openai-agents/src/uipath_openai_agents/__init__.py index 123fb62..8d27080 100644 --- a/packages/uipath-openai-agents/src/uipath_openai_agents/__init__.py +++ b/packages/uipath-openai-agents/src/uipath_openai_agents/__init__.py @@ -1,7 +1,28 @@ -"""UiPath OpenAI Agents SDK.""" +""" +UiPath OpenAI Agents SDK. + +NOTE: This module uses lazy imports via __getattr__ to avoid loading heavy +dependencies (openai SDK) at import time. This significantly improves CLI +startup performance. + +Do NOT add eager imports like: + from .chat import UiPathChatOpenAI # BAD - loads openai SDK immediately + +Instead, all exports are loaded on-demand when first accessed. +""" + + +def __getattr__(name): + if name == "UiPathChatOpenAI": + from .chat import UiPathChatOpenAI + + return UiPathChatOpenAI + if name == "register_middleware": + from .middlewares import register_middleware + + return register_middleware + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") -from .chat import UiPathChatOpenAI -from .middlewares import register_middleware __version__ = "0.1.0" __all__ = ["register_middleware", "UiPathChatOpenAI"] diff --git a/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/_templates/AGENTS.md.template b/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/_templates/AGENTS.md.template index 5ffabe2..abde426 100644 --- a/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/_templates/AGENTS.md.template +++ b/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/_templates/AGENTS.md.template @@ -18,4 +18,4 @@ This documentation is split into multiple files for efficient context loading. L 3. **@.agent/CLI_REFERENCE.md** - CLI commands documentation - **When to load:** Working with `uipath init`, `uipath run agent`, or deployment commands - - **Contains:** Command syntax for OpenAI agents, options, input formats (`{"message": "..."}` or `{"messages": [...]}`), debug mode, usage examples + - **Contains:** Command syntax for OpenAI agents, options, input formats (`{"messages": "..."}` or `{"messages": [...]}`), debug mode, usage examples diff --git a/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/_templates/main.py.template b/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/_templates/main.py.template index a50c8ce..48ba5ae 100644 --- a/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/_templates/main.py.template +++ b/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/_templates/main.py.template @@ -1,7 +1,11 @@ -from agents import Agent -from openai import OpenAI +from agents import Agent, function_tool +from agents.models import _openai_shared +from uipath_openai_agents.chat import UiPathChatOpenAI +from uipath_openai_agents.chat.supported_models import OpenAIModels + +@function_tool def get_weather(location: str) -> str: """Get the current weather for a location. @@ -12,17 +16,19 @@ def get_weather(location: str) -> str: Weather information for the location """ # This is a mock implementation - return f"The weather in {location} is sunny and 72F" + return f"The weather in {location} is sunny and 32 degrees Celsius." + +MODEL = OpenAIModels.gpt_5_1_2025_11_13 -# Initialize the OpenAI client -client = OpenAI() +# Initialize the UiPath OpenAI client +uipath_openai_client = UiPathChatOpenAI(model_name=MODEL) +_openai_shared.set_default_openai_client(uipath_openai_client.async_client) # Create an agent with tools agent = Agent( name="weather_agent", instructions="You are a helpful weather assistant. Use the get_weather tool to provide weather information.", - model="gpt-4o-mini", + model=MODEL, tools=[get_weather], - client=client, ) diff --git a/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/cli_new.py b/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/cli_new.py index f9ac2c7..639ab62 100644 --- a/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/cli_new.py +++ b/packages/uipath-openai-agents/src/uipath_openai_agents/_cli/cli_new.py @@ -39,8 +39,7 @@ def generate_pyproject(target_directory, project_name): description = "{project_name}" authors = [{{ name = "John Doe", email = "john.doe@myemail.com" }}] dependencies = [ - "uipath-openai-agents>=0.1.0", - "openai>=1.0.0" + "uipath-openai-agents>=0.0.1, <0.1.0", ] requires-python = ">=3.11" """ @@ -63,11 +62,8 @@ def openai_agents_new_middleware(name: str) -> MiddlewareResult: console.success("Created 'AGENTS.md' file.") generate_pyproject(directory, name) console.success("Created 'pyproject.toml' file.") - console.config( - f""" Please ensure to define {click.style("OPENAI_API_KEY", fg="bright_yellow")} in your .env file. """ - ) init_command = """uipath init""" - run_command = """uipath run agent '{"message": "What is the weather in San Francisco?"}'""" + run_command = """uipath run agent '{"messages": "What is the weather in San Francisco?"}'""" console.hint( f""" Initialize project: {click.style(init_command, fg="cyan")}""" ) diff --git a/packages/uipath-openai-agents/src/uipath_openai_agents/chat/__init__.py b/packages/uipath-openai-agents/src/uipath_openai_agents/chat/__init__.py index 6553005..3e5abab 100644 --- a/packages/uipath-openai-agents/src/uipath_openai_agents/chat/__init__.py +++ b/packages/uipath-openai-agents/src/uipath_openai_agents/chat/__init__.py @@ -1,5 +1,32 @@ -"""UiPath OpenAI Chat models.""" +""" +UiPath OpenAI Chat module. -from .openai import UiPathChatOpenAI +NOTE: This module uses lazy imports via __getattr__ to avoid loading heavy +dependencies (openai SDK, httpx) at import time. This significantly +improves CLI startup performance. -__all__ = ["UiPathChatOpenAI"] +Do NOT add eager imports like: + from .openai import UiPathChatOpenAI # BAD - loads openai SDK immediately + +Instead, all exports are loaded on-demand when first accessed. +""" + + +def __getattr__(name): + if name == "UiPathChatOpenAI": + from .openai import UiPathChatOpenAI + + return UiPathChatOpenAI + if name in ("OpenAIModels", "GeminiModels", "BedrockModels"): + from . import supported_models + + return getattr(supported_models, name) + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") + + +__all__ = [ + "UiPathChatOpenAI", + "OpenAIModels", + "GeminiModels", + "BedrockModels", +] diff --git a/packages/uipath-openai-agents/src/uipath_openai_agents/chat/supported_models.py b/packages/uipath-openai-agents/src/uipath_openai_agents/chat/supported_models.py index 6ad9039..226be09 100644 --- a/packages/uipath-openai-agents/src/uipath_openai_agents/chat/supported_models.py +++ b/packages/uipath-openai-agents/src/uipath_openai_agents/chat/supported_models.py @@ -1,78 +1,60 @@ -"""Supported OpenAI model definitions for UiPath LLM Gateway.""" +"""Supported model definitions for UiPath LLM Gateway. +OpenAI Agents SDK supports other LLM providers (Bedrock, Gemini, etc.) +through LiteLLM integration, so we provide model definitions for all supported providers. +""" -class OpenAIModels: - """OpenAI model names supported by UiPath LLM Gateway. +from enum import StrEnum - These are specific model versions required by UiPath. - Generic names like "gpt-4o" are not supported - use specific versions. - """ - # GPT-4o Models (recommended) - gpt_4o_2024_11_20 = "gpt-4o-2024-11-20" - gpt_4o_2024_08_06 = "gpt-4o-2024-08-06" +class OpenAIModels(StrEnum): + """Supported OpenAI model identifiers.""" + + # GPT-4o models gpt_4o_2024_05_13 = "gpt-4o-2024-05-13" + gpt_4o_2024_08_06 = "gpt-4o-2024-08-06" + gpt_4o_2024_11_20 = "gpt-4o-2024-11-20" gpt_4o_mini_2024_07_18 = "gpt-4o-mini-2024-07-18" - # GPT-4.1 Models + # GPT-4.1 models gpt_4_1_2025_04_14 = "gpt-4.1-2025-04-14" gpt_4_1_mini_2025_04_14 = "gpt-4.1-mini-2025-04-14" gpt_4_1_nano_2025_04_14 = "gpt-4.1-nano-2025-04-14" - # GPT-4 Models - gpt_4 = "gpt-4" - gpt_4_32k = "gpt-4-32k" - gpt_4_turbo_2024_04_09 = "gpt-4-turbo-2024-04-09" - gpt_4_1106_preview = "gpt-4-1106-Preview" - gpt_4_vision_preview = "gpt-4-vision-preview" - - # GPT-3.5 Models - gpt_35_turbo = "gpt-35-turbo" - gpt_35_turbo_0125 = "gpt-35-turbo-0125" - gpt_35_turbo_1106 = "gpt-35-turbo-1106" - gpt_35_turbo_16k = "gpt-35-turbo-16k" - - # GPT-5 Models + # GPT-5 models gpt_5_2025_08_07 = "gpt-5-2025-08-07" gpt_5_chat_2025_08_07 = "gpt-5-chat-2025-08-07" gpt_5_mini_2025_08_07 = "gpt-5-mini-2025-08-07" gpt_5_nano_2025_08_07 = "gpt-5-nano-2025-08-07" + + # GPT-5.1 models gpt_5_1_2025_11_13 = "gpt-5.1-2025-11-13" + + # GPT-5.2 models gpt_5_2_2025_12_11 = "gpt-5.2-2025-12-11" - # o3 Models - o3_mini_2025_01_31 = "o3-mini-2025-01-31" - - # Other Models - computer_use_preview_2025_03_11 = "computer-use-preview-2025-03-11" - text_davinci_003 = "text-davinci-003" - - # Embedding Models - text_embedding_3_large = "text-embedding-3-large" - text_embedding_3_large_community_ecs = "text-embedding-3-large-community-ecs" - text_embedding_ada_002 = "text-embedding-ada-002" - - # Model aliases - maps generic names to specific versions - MODEL_ALIASES = { - # Map gpt-4.1 variants to gpt-4o (most capable available model) - "gpt-4.1": gpt_4o_2024_11_20, - "gpt-4.1-mini": gpt_4o_mini_2024_07_18, - "gpt-4.1-nano": gpt_4o_mini_2024_07_18, - "gpt-4.1-2025-04-14": gpt_4o_2024_11_20, # Map invalid model to valid one - "gpt-4.1-mini-2025-04-14": gpt_4o_mini_2024_07_18, - "gpt-4.1-nano-2025-04-14": gpt_4o_mini_2024_07_18, - # Generic model mappings - "gpt-4o": gpt_4o_2024_11_20, - "gpt-4o-mini": gpt_4o_mini_2024_07_18, - "gpt-5": gpt_5_2025_08_07, - "gpt-5-mini": gpt_5_mini_2025_08_07, - "gpt-5-nano": gpt_5_nano_2025_08_07, - "gpt-5.1": gpt_5_1_2025_11_13, - "gpt-5.2": gpt_5_2_2025_12_11, - "o3-mini": o3_mini_2025_01_31, - } - - @classmethod - def normalize_model_name(cls, model_name: str) -> str: - """Normalize a model name to UiPath-specific version.""" - return cls.MODEL_ALIASES.get(model_name, model_name) + +class GeminiModels(StrEnum): + """Supported Google Gemini model identifiers.""" + + # Gemini 2 models + gemini_2_5_pro = "gemini-2.5-pro" + gemini_2_5_flash = "gemini-2.5-flash" + gemini_2_0_flash_001 = "gemini-2.0-flash-001" + + # Gemini 3 models + gemini_3_pro_preview = "gemini-3-pro-preview" + + +class BedrockModels(StrEnum): + """Supported AWS Bedrock model identifiers.""" + + # Claude 3.7 models + anthropic_claude_3_7_sonnet = "anthropic.claude-3-7-sonnet-20250219-v1:0" + + # Claude 4 models + anthropic_claude_sonnet_4 = "anthropic.claude-sonnet-4-20250514-v1:0" + + # Claude 4.5 models + anthropic_claude_sonnet_4_5 = "anthropic.claude-sonnet-4-5-20250929-v1:0" + anthropic_claude_haiku_4_5 = "anthropic.claude-haiku-4-5-20251001-v1:0" diff --git a/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/__init__.py b/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/__init__.py index c1acb4e..9aab7d5 100644 --- a/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/__init__.py +++ b/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/__init__.py @@ -1,4 +1,10 @@ -"""UiPath OpenAI Agents Runtime.""" +""" +UiPath OpenAI Agents Runtime. + +NOTE: This module uses lazy imports for most exports to avoid loading heavy +dependencies (openai SDK) at import time. However, the runtime factory +registration must happen eagerly for CLI discovery. +""" from uipath.runtime import ( UiPathRuntimeContext, @@ -6,13 +12,6 @@ UiPathRuntimeFactoryRegistry, ) -from uipath_openai_agents.runtime.factory import UiPathOpenAIAgentRuntimeFactory -from uipath_openai_agents.runtime.runtime import UiPathOpenAIAgentRuntime -from uipath_openai_agents.runtime.schema import ( - get_agent_schema, - get_entrypoints_schema, -) - def register_runtime_factory() -> None: """Register the OpenAI Agents factory. Called automatically via entry point.""" @@ -20,6 +19,11 @@ def register_runtime_factory() -> None: def create_factory( context: UiPathRuntimeContext | None = None, ) -> UiPathRuntimeFactoryProtocol: + # Import lazily when factory is actually created + from uipath_openai_agents.runtime.factory import ( + UiPathOpenAIAgentRuntimeFactory, + ) + return UiPathOpenAIAgentRuntimeFactory( context=context if context else UiPathRuntimeContext(), ) @@ -29,8 +33,30 @@ def create_factory( ) +# Register factory eagerly (required for CLI discovery) register_runtime_factory() + +def __getattr__(name): + if name == "get_entrypoints_schema": + from .schema import get_entrypoints_schema + + return get_entrypoints_schema + if name == "get_agent_schema": + from .schema import get_agent_schema + + return get_agent_schema + if name == "UiPathOpenAIAgentRuntimeFactory": + from .factory import UiPathOpenAIAgentRuntimeFactory + + return UiPathOpenAIAgentRuntimeFactory + if name == "UiPathOpenAIAgentRuntime": + from .runtime import UiPathOpenAIAgentRuntime + + return UiPathOpenAIAgentRuntime + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") + + __all__ = [ "register_runtime_factory", "get_entrypoints_schema", diff --git a/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/runtime.py b/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/runtime.py index 0cde361..7a67556 100644 --- a/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/runtime.py +++ b/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/runtime.py @@ -219,10 +219,20 @@ def _convert_stream_event_to_runtime_event( return None def _prepare_agent_input(self, input: dict[str, Any] | None) -> str | list[Any]: - """Prepare agent input from UiPath input dictionary.""" - if input and "messages" in input and isinstance(input["messages"], list): - return input.get("messages", []) - return input.get("message", "") if input else "" + """ + Prepare agent input from UiPath input dictionary. + + """ + if not input: + return "" + + messages = input.get("messages", "") + + if isinstance(messages, (str, list)): + return messages + + # Fallback to empty string for unexpected types + return "" def _serialize_message(self, message: Any) -> dict[str, Any]: """ diff --git a/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/schema.py b/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/schema.py index 03483f6..28c91e2 100644 --- a/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/schema.py +++ b/packages/uipath-openai-agents/src/uipath_openai_agents/runtime/schema.py @@ -63,7 +63,7 @@ def get_entrypoints_schema(agent: Agent) -> dict[str, Any]: schema["input"] = { "type": "object", "properties": { - "message": { + "messages": { "anyOf": [ {"type": "string"}, { @@ -71,11 +71,11 @@ def get_entrypoints_schema(agent: Agent) -> dict[str, Any]: "items": {"type": "object"}, }, ], - "title": "Message", - "description": "User message(s) to send to the agent", + "title": "Messages", + "description": "User messages to send to the agent", } }, - "required": ["message"], + "required": ["messages"], } # Extract output schema - Agent's output_type (native OpenAI Agents pattern) diff --git a/packages/uipath-openai-agents/testcases/common/__init__.py b/packages/uipath-openai-agents/testcases/common/__init__.py new file mode 100644 index 0000000..56c3a88 --- /dev/null +++ b/packages/uipath-openai-agents/testcases/common/__init__.py @@ -0,0 +1 @@ +"""Common utilities for OpenAI Agents testcases.""" diff --git a/packages/uipath-openai-agents/testcases/common/trace_assert.py b/packages/uipath-openai-agents/testcases/common/trace_assert.py new file mode 100644 index 0000000..b14aa63 --- /dev/null +++ b/packages/uipath-openai-agents/testcases/common/trace_assert.py @@ -0,0 +1,143 @@ +""" +Simple trace assertion - just check that expected spans exist with required attributes. +Much simpler than the tree-based approach. +""" + +import json +from typing import Any + + +def load_traces(traces_file: str) -> list[dict[str, Any]]: + """Load traces from a JSONL file.""" + traces = [] + with open(traces_file, "r", encoding="utf-8") as f: + for line in f: + if line.strip(): + traces.append(json.loads(line)) + return traces + + +def load_expected_traces(expected_file: str) -> list[dict[str, Any]]: + """Load expected trace definitions from a JSON file.""" + with open(expected_file, "r", encoding="utf-8") as f: + data = json.load(f) + return data.get("required_spans", []) + + +def get_attributes(span: dict[str, Any]) -> dict[str, Any]: + """ + Parse attributes from a span. + Supports both formats: + - Old format: 'Attributes' as a JSON string + - New format: 'attributes' as a dict + """ + # New format: attributes is already a dict + if "attributes" in span and isinstance(span["attributes"], dict): + return span["attributes"] + # Old format: Attributes is a JSON string + attributes_str = span.get("Attributes", "{}") + try: + return json.loads(attributes_str) + except json.JSONDecodeError: + return {} + + +def matches_value(expected_value: Any, actual_value: Any) -> bool: + """ + Check if an actual value matches the expected value. + Supports: + - List of possible values: ["value1", "value2"] + - Wildcard: "*" (any value accepted) + - Exact match: "value" + """ + # Wildcard - accept any value + if expected_value == "*": + return True + # List of possible values + if isinstance(expected_value, list): + return actual_value in expected_value + # Exact match + return expected_value == actual_value + + +def matches_expected(span: dict[str, Any], expected: dict[str, Any]) -> bool: + """ + Check if a span matches the expected definition. + Supports both formats: + - Old format: 'Name', 'SpanType' fields + - New format: 'name', 'attributes.span_type' fields + """ + # Check name - can be a string or list of possible names + expected_name = expected.get("name") + # Support both old format (Name) and new format (name) + actual_name = span.get("name") or span.get("Name") + if isinstance(expected_name, list): + if actual_name not in expected_name: + return False + elif expected_name != actual_name: + return False + # Check span type if specified + if "span_type" in expected: + # Old format: SpanType field + # New format: attributes.span_type field + actual_span_type = span.get("SpanType") + if not actual_span_type: + actual_attrs = get_attributes(span) + actual_span_type = actual_attrs.get("span_type") + if actual_span_type != expected["span_type"]: + return False + # Check attributes if specified + if "attributes" in expected: + actual_attrs = get_attributes(span) + for key, expected_value in expected["attributes"].items(): + if key not in actual_attrs: + return False + # Use flexible value matching + if not matches_value(expected_value, actual_attrs[key]): + return False + return True + + +def assert_traces(traces_file: str, expected_file: str) -> None: + """ + Assert that all expected traces exist in the traces file. + Args: + traces_file: Path to the traces.jsonl file + expected_file: Path to the expected_traces.json file + Raises: + AssertionError: If any expected trace is not found + """ + traces = load_traces(traces_file) + expected_spans = load_expected_traces(expected_file) + print(f"Loaded {len(traces)} traces from {traces_file}") + print(f"Checking {len(expected_spans)} expected spans...") + missing_spans = [] + for expected in expected_spans: + # Find a matching span + found = False + name = expected["name"] + # Handle both string and list of names + name_str = name if isinstance(name, str) else f"[{' | '.join(name)}]" + + for span in traces: + if matches_expected(span, expected): + found = True + print(f"✓ Found span: {name_str}") + break + if not found: + missing_spans.append(name_str) + print(f"✗ Missing span: {name_str}") + + print("Traces file content:") + with open(traces_file, "r", encoding="utf-8") as f: + print(f.read()) + if missing_spans: + print(f"\n=== Dumping raw traces from {traces_file} ===") + with open(traces_file, "r", encoding="utf-8") as f: + print(f.read()) + print("\n=== End of traces dump ===\n") + raise AssertionError( + f"Missing expected spans: {', '.join(missing_spans)}\n" + f"Expected {len(expected_spans)} spans, found {len(expected_spans) - len(missing_spans)}" + ) + print(f"\n✓ All {len(expected_spans)} expected spans found!") diff --git a/packages/uipath-openai-agents/testcases/common/validate_output.sh b/packages/uipath-openai-agents/testcases/common/validate_output.sh index 71ff190..1a11857 100755 --- a/packages/uipath-openai-agents/testcases/common/validate_output.sh +++ b/packages/uipath-openai-agents/testcases/common/validate_output.sh @@ -22,14 +22,18 @@ debug_print_uipath_output() { fi } -validate_output() { - echo "Printing output file for validation..." - debug_print_uipath_output - - echo "Validating output..." - python src/assert.py || { echo "Validation failed!"; exit 1; } - - echo "Testcase completed successfully." +# Run assertions from the testcase's src directory +run_assertions() { + echo "Running assertions..." + if [ -f "src/assert.py" ]; then + # Use the Python from the virtual environment + # Prepend the common directory to the python path so it can be resolved + PYTHONPATH="../common:$PYTHONPATH" python src/assert.py + else + echo "assert.py not found in src directory!" + exit 1 + fi } -validate_output +debug_print_uipath_output +run_assertions diff --git a/packages/uipath-openai-agents/testcases/init-flow/README.md b/packages/uipath-openai-agents/testcases/init-flow/README.md new file mode 100644 index 0000000..409419c --- /dev/null +++ b/packages/uipath-openai-agents/testcases/init-flow/README.md @@ -0,0 +1,50 @@ +# OpenAI Agents Integration Test - Init Flow + +This testcase validates the complete init-flow for OpenAI Agents SDK integration with UiPath. + +## What it tests + +1. **Project Setup**: Creates a new UiPath agent project using `uipath new agent` +2. **Initialization**: Runs `uipath init` to generate configuration files +3. **Packaging**: Packs the agent into a NuGet package +4. **Deployment Execution**: Runs the agent with UiPath platform integration +5. **Local Execution**: Runs the agent locally with tracing enabled +6. **Output Validation**: Validates the agent output structure and status +7. **Trace Validation**: Verifies that expected OpenTelemetry spans are generated + +## Files + +- **pyproject.toml**: Project dependencies and configuration +- **input.json**: Test input for the agent +- **run.sh**: Main test script that executes all steps +- **expected_traces.json**: Expected OpenTelemetry spans for validation +- **src/assert.py**: Assertion script that validates outputs and traces + +## Expected Traces + +The test validates that the following spans are generated: + +- **Agent workflow**: OpenAI Agents SDK top-level agent span (AGENT kind) +- **response**: OpenAI Responses API call span (LLM kind) - note that OpenAI Agents SDK uses the Responses API, not ChatCompletion + +## Running the test + +```bash +cd testcases/init-flow +bash run.sh +``` + +The test requires: +- `CLIENT_ID`: UiPath OAuth client ID +- `CLIENT_SECRET`: UiPath OAuth client secret +- `BASE_URL`: UiPath platform base URL + +## Success Criteria + +The test passes if: + +1. NuGet package (.nupkg) is created successfully +2. Agent executes without errors (status: "successful") +3. Output contains the expected "result" field +4. Local run output contains "Successful execution." +5. All expected OpenTelemetry spans are present in traces.jsonl diff --git a/packages/uipath-openai-agents/testcases/init-flow/expected_traces.json b/packages/uipath-openai-agents/testcases/init-flow/expected_traces.json new file mode 100644 index 0000000..be8ebd2 --- /dev/null +++ b/packages/uipath-openai-agents/testcases/init-flow/expected_traces.json @@ -0,0 +1,18 @@ +{ + "description": "OpenAI Agents SDK integration - checks key spans exist", + "required_spans": [ + { + "name": "Agent workflow", + "attributes": { + "openinference.span.kind": "AGENT" + } + }, + { + "name": "response", + "attributes": { + "openinference.span.kind": "LLM", + "llm.system": "openai" + } + } + ] +} diff --git a/packages/uipath-openai-agents/testcases/init-flow/input.json b/packages/uipath-openai-agents/testcases/init-flow/input.json new file mode 100644 index 0000000..c10632c --- /dev/null +++ b/packages/uipath-openai-agents/testcases/init-flow/input.json @@ -0,0 +1,3 @@ +{ + "messages": "Hello, how are you?" +} diff --git a/packages/uipath-openai-agents/testcases/init-flow/pyproject.toml b/packages/uipath-openai-agents/testcases/init-flow/pyproject.toml new file mode 100644 index 0000000..c8e2a64 --- /dev/null +++ b/packages/uipath-openai-agents/testcases/init-flow/pyproject.toml @@ -0,0 +1,14 @@ +[project] +name = "agent" +version = "0.0.1" +description = "OpenAI Agents integration test" +authors = [{ name = "John Doe", email = "john.doe@myemail.com" }] +dependencies = [ + "openai>=1.0.0", + "openai-agents>=0.6.5", + "uipath-openai-agents", +] +requires-python = ">=3.11" + +[tool.uv.sources] +uipath-openai-agents = { path = "../../", editable = true } diff --git a/packages/uipath-openai-agents/testcases/init-flow/run.sh b/packages/uipath-openai-agents/testcases/init-flow/run.sh new file mode 100755 index 0000000..3cbb733 --- /dev/null +++ b/packages/uipath-openai-agents/testcases/init-flow/run.sh @@ -0,0 +1,32 @@ +#!/bin/bash +set -e + +echo "Syncing dependencies..." +uv sync + +echo "Backing up pyproject.toml..." +cp pyproject.toml pyproject-overwrite.toml + +echo "Creating new UiPath agent..." +uv run uipath new agent + +# uipath new overwrites pyproject.toml, so we need to copy it back +echo "Restoring pyproject.toml..." +cp pyproject-overwrite.toml pyproject.toml +uv sync + +echo "Authenticating with UiPath..." +uv run uipath auth --client-id="$CLIENT_ID" --client-secret="$CLIENT_SECRET" --base-url="$BASE_URL" + +echo "Initializing UiPath..." +uv run uipath init + +echo "Packing agent..." +uv run uipath pack + +echo "Input from input.json file" +uv run uipath run agent --file input.json + +echo "Running agent again with empty UIPATH_JOB_KEY..." +export UIPATH_JOB_KEY="" +uv run uipath run agent --trace-file .uipath/traces.jsonl --file input.json >> local_run_output.log diff --git a/packages/uipath-openai-agents/testcases/init-flow/src/assert.py b/packages/uipath-openai-agents/testcases/init-flow/src/assert.py new file mode 100644 index 0000000..7b12ca1 --- /dev/null +++ b/packages/uipath-openai-agents/testcases/init-flow/src/assert.py @@ -0,0 +1,65 @@ +import json +import os + +from trace_assert import assert_traces + +print("Checking init-flow output...") + +# Check NuGet package +uipath_dir = ".uipath" +assert os.path.exists(uipath_dir), "NuGet package directory (.uipath) not found" + +nupkg_files = [f for f in os.listdir(uipath_dir) if f.endswith(".nupkg")] +assert nupkg_files, "NuGet package file (.nupkg) not found in .uipath directory" + +print(f"NuGet package found: {nupkg_files[0]}") + +# Check agent output file +output_file = "__uipath/output.json" +assert os.path.isfile(output_file), "Agent output file not found" + +print("Agent output file found") + +# Check status and required fields +with open(output_file, "r", encoding="utf-8") as f: + output_data = json.load(f) + +# Check status +status = output_data.get("status") +assert status == "successful", f"Agent execution failed with status: {status}" + +print("Agent execution status: successful") + +# Check required fields for OpenAI agent +assert "output" in output_data, "Missing 'output' field in agent response" + +output_content = output_data["output"] +assert "result" in output_content, "Missing 'result' field in output" + +result = output_content["result"] +assert result and isinstance(result, (str, dict, list)), ( + "Result field is empty or invalid type" +) + +print(f"Result field validated: {type(result).__name__}") + +# Check local run output +with open("local_run_output.log", "r", encoding="utf-8") as f: + local_run_output = f.read() + +# Check if response contains 'Successful execution.' +assert "Successful execution." in local_run_output, ( + f"Response does not contain 'Successful execution.'. Actual response: {local_run_output}" +) + +print("Local run output validated") + +# Check traces +with open(".uipath/traces.jsonl", "r", encoding="utf-8") as f: + local_run_traces = f.read() + print(f"Traces generated: {len(local_run_traces)} bytes") + +# Simple trace assertions - just check that expected spans exist +assert_traces(".uipath/traces.jsonl", "expected_traces.json") + +print("All validations passed successfully!") diff --git a/packages/uipath-openai-agents/testcases/triage-agent/input.json b/packages/uipath-openai-agents/testcases/triage-agent/input.json index 5d8e6c8..aceda16 100644 --- a/packages/uipath-openai-agents/testcases/triage-agent/input.json +++ b/packages/uipath-openai-agents/testcases/triage-agent/input.json @@ -1,3 +1,3 @@ { - "message": "Hola, ¿cómo estás?" + "messages": "Hola, ¿cómo estás?" } diff --git a/packages/uipath-openai-agents/tests/cli/test_init.py b/packages/uipath-openai-agents/tests/cli/test_init.py index a13143a..0690ec4 100644 --- a/packages/uipath-openai-agents/tests/cli/test_init.py +++ b/packages/uipath-openai-agents/tests/cli/test_init.py @@ -4,7 +4,7 @@ import os from click.testing import CliRunner -from uipath._cli.cli_init import init +from uipath._cli import cli class TestInit: @@ -30,7 +30,7 @@ def test_init_basic_config_generation( with open("openai_agents.json", "w") as f: f.write(openai_agents_config) - result = runner.invoke(init) + result = runner.invoke(cli, ["init"]) assert result.exit_code == 0 assert os.path.exists("entry-points.json") @@ -54,9 +54,9 @@ def test_init_basic_config_generation( assert isinstance(input_schema["properties"], dict) assert isinstance(input_schema["required"], list) - # OpenAI agents use default message input - assert "message" in input_schema["properties"] - assert "message" in input_schema["required"] + # OpenAI agents use default messages input + assert "messages" in input_schema["properties"] + assert "messages" in input_schema["required"] # Verify output schema (default since no output_type specified) assert "output" in entry @@ -87,7 +87,7 @@ def test_init_translation_agent_config_generation( with open("openai_agents.json", "w") as f: f.write(openai_agents_config) - result = runner.invoke(init) + result = runner.invoke(cli, ["init"]) assert result.exit_code == 0 assert os.path.exists("entry-points.json") @@ -112,8 +112,8 @@ def test_init_translation_agent_config_generation( input_schema = translation_entry["input"] assert input_schema["type"] == "object" assert "properties" in input_schema - assert "message" in input_schema["properties"] - assert "message" in input_schema["required"] + assert "messages" in input_schema["properties"] + assert "messages" in input_schema["required"] # Verify output schema from agent's output_type assert "output" in translation_entry diff --git a/packages/uipath-openai-agents/tests/cli/test_run.py b/packages/uipath-openai-agents/tests/cli/test_run.py index 5e612f3..5f240c3 100644 --- a/packages/uipath-openai-agents/tests/cli/test_run.py +++ b/packages/uipath-openai-agents/tests/cli/test_run.py @@ -22,7 +22,7 @@ def test_run_basic_agent_invocation( ) -> None: """Test basic agent invocation structure (without actual OpenAI API calls).""" input_file_name = "input.json" - input_json_content = '{"message": "Hello, agent!"}' + input_json_content = '{"messages": "Hello, agent!"}' with runner.isolated_filesystem(temp_dir=temp_dir): # create input file @@ -54,7 +54,7 @@ def test_run_with_invalid_agent_name( ) -> None: """Test run command with non-existent agent name.""" input_file_name = "input.json" - input_json_content = '{"message": "test"}' + input_json_content = '{"messages": "test"}' with runner.isolated_filesystem(temp_dir=temp_dir): input_file_path = os.path.join(temp_dir, input_file_name) @@ -83,7 +83,7 @@ def test_run_without_config_file( ) -> None: """Test run command without openai_agents.json file.""" input_file_name = "input.json" - input_json_content = '{"message": "test"}' + input_json_content = '{"messages": "test"}' with runner.isolated_filesystem(temp_dir=temp_dir): input_file_path = os.path.join(temp_dir, input_file_name) @@ -109,7 +109,7 @@ def test_run_with_malformed_input_json( ) -> None: """Test run command with malformed JSON input.""" input_file_name = "input.json" - input_json_content = '{"message": invalid json}' # Malformed + input_json_content = '{"messages": invalid json}' # Malformed with runner.isolated_filesystem(temp_dir=temp_dir): input_file_path = os.path.join(temp_dir, input_file_name) diff --git a/packages/uipath-openai-agents/tests/test_agent_as_tools_schema.py b/packages/uipath-openai-agents/tests/test_agent_as_tools_schema.py index bcc6516..42d70eb 100644 --- a/packages/uipath-openai-agents/tests/test_agent_as_tools_schema.py +++ b/packages/uipath-openai-agents/tests/test_agent_as_tools_schema.py @@ -33,17 +33,17 @@ def test_agent_as_tools_input_schema(): # OpenAI Agents use messages as input (not custom types) input_props = schema["input"]["properties"] - assert "message" in input_props + assert "messages" in input_props - # Verify message field accepts string or array - assert "anyOf" in input_props["message"] - types = [t.get("type") for t in input_props["message"]["anyOf"]] + # Verify messages field accepts string or array + assert "anyOf" in input_props["messages"] + types = [t.get("type") for t in input_props["messages"]["anyOf"]] assert "string" in types assert "array" in types # Check required fields assert "required" in schema["input"] - assert "message" in schema["input"]["required"] + assert "messages" in schema["input"]["required"] def test_agent_as_tools_output_schema(): @@ -86,7 +86,7 @@ def test_agent_as_tools_schema_metadata(): # Input uses default messages format (no custom title/description) assert "input" in schema assert "properties" in schema["input"] - assert "message" in schema["input"]["properties"] + assert "messages" in schema["input"]["properties"] # Check output metadata from agent's output_type assert "title" in schema["output"] diff --git a/packages/uipath-openai-agents/tests/test_integration.py b/packages/uipath-openai-agents/tests/test_integration.py index 4bfecc6..76c3eed 100644 --- a/packages/uipath-openai-agents/tests/test_integration.py +++ b/packages/uipath-openai-agents/tests/test_integration.py @@ -56,7 +56,7 @@ def test_schema_extraction_with_new_serialization(): # Verify input schema (messages format) assert "input" in schema - assert "message" in schema["input"]["properties"] + assert "messages" in schema["input"]["properties"] # Verify output schema (from agent's output_type) assert "output" in schema diff --git a/packages/uipath-openai-agents/tests/test_schema_inference.py b/packages/uipath-openai-agents/tests/test_schema_inference.py index 17e572e..1db51f3 100644 --- a/packages/uipath-openai-agents/tests/test_schema_inference.py +++ b/packages/uipath-openai-agents/tests/test_schema_inference.py @@ -35,9 +35,9 @@ def test_schema_inference_from_agent_output_type(): # Check input schema - should be default messages format assert "input" in schema assert "properties" in schema["input"] - assert "message" in schema["input"]["properties"] + assert "messages" in schema["input"]["properties"] assert "required" in schema["input"] - assert "message" in schema["input"]["required"] + assert "messages" in schema["input"]["required"] # Check output schema - extracted from agent's output_type assert "output" in schema @@ -60,9 +60,9 @@ def test_schema_fallback_without_types(): """Test that schemas fall back to defaults when no types are provided.""" schema = get_entrypoints_schema(test_agent) - # Should use default message-based input schema + # Should use default messages-based input schema assert "input" in schema - assert "message" in schema["input"]["properties"] + assert "messages" in schema["input"]["properties"] # Should fall back to default result-based output assert "output" in schema @@ -73,9 +73,9 @@ def test_schema_with_plain_agent(): """Test schema extraction with a plain agent.""" schema = get_entrypoints_schema(test_agent) - # Should use default message input + # Should use default messages input assert "input" in schema - assert "message" in schema["input"]["properties"] + assert "messages" in schema["input"]["properties"] # Should use default result output assert "output" in schema