feat: evalhub CLI server & mcp Lifecycle managements with sdk>=0.4.3#91
Conversation
…ub#81, eval-hub#84) Add documentation for the new `evalhub server run|start|stop|status` commands, `server_config_file` config key, `--unfold` flag on `config get`, and `config unset`. Convert CLI guide from .md to .mdx for Since component support. Update MCP installation docs with CLI-managed `mcp_config_file` configuration flow, `evalhub mcp` subcommands, and breaking change notice for the old `evalhub mcp` stdio command removed in 0.4.3. Co-Authored-By: Claude <noreply@anthropic.com>
Replace eval-hub-server --local --configdir with evalhub server start/stop workflow. Add inline server config YAML, BYOF provider template with MLFLOW_TRACKING_URI, sample job.yaml, and OCI credentials guidance. Remove deprecated --local flag references, job.json schema, and environment variables section. Co-Authored-By: Claude <noreply@anthropic.com>
Bump Since tags to 0.4.3 eval-hub-sdk, reorder tutorial steps to download adapter before installing dependencies, pin SDK to >=0.4.3, add MLFLOW_TRACKING_URI to tutorial provider YAML, update cluster mode multi-tenancy description, and clean up next steps. Co-Authored-By: Claude <noreply@anthropic.com>
…output Replace cat heredoc with file-creation instruction for consistency, add PROVIDER_ID/BENCHMARK_ID/JOB_ID shell variables from CLI output to use in subsequent commands, and update example results with actual evalhub output. Pin minimum SDK version in local-mode guide. Co-Authored-By: Claude <noreply@anthropic.com>
gnaulak-redhat
changed the title
|
Warning Review limit reached
Next review available in: 45 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughDocumentation across the CLI guide, local-mode guides, and MCP installation guide is updated to describe new CLI-managed ChangesCLI-managed server/MCP documentation update
Estimated code review effort: 2 (Simple) | ~12 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
src/content/docs/guides/local-mode-tutorial.mdx (2)
34-52: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winVersion pin inconsistency between tutorial and local-mode guide.
The tutorial pins
eval-hub-sdk[server,cli]>=0.4.3(line 52) while the Local Mode guide at line 25 installseval-hub-sdk[server,cli]without a version constraint. Since theSincemarker in both files declares0.4.3, consider aligning the install commands so users following either guide get a compatible version.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/content/docs/guides/local-mode-tutorial.mdx` around lines 34 - 52, The install commands for eval-hub-sdk are inconsistent across the local-mode tutorial and the Local Mode guide, which can lead to users installing different versions. Update the install step in the tutorial section that uses uv pip install so it matches the version constraint used elsewhere, aligning with the Since 0.4.3 marker. Keep the wording and command structure consistent around the install instructions in the local-mode guide content.
121-152: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueIn-memory SQLite URL may need clarification for persistence.
The database URL
file::eval_hub:?mode=memory&cache=shareduses an in-memory SQLite database. While this is fine for a tutorial, users following along may lose all evaluation data when the server stops. Consider adding a brief note that this is ephemeral and that a file-based URL should be used for persistent storage, or that the tutorial intentionally uses in-memory for simplicity.This is consistent with the same URL in
local-mode.mdxline 36, so both guides share the same behavior.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/content/docs/guides/local-mode-tutorial.mdx` around lines 121 - 152, Clarify that the `database.url` value using `file::eval_hub:?mode=memory&cache=shared` in the EvalHub server setup is ephemeral and will not persist data after shutdown. Update the `local-mode-tutorial.mdx` step around the `my-config.yaml` example to briefly note that this in-memory SQLite configuration is intentional for the tutorial, and point readers to using a file-based SQLite URL if they want persistent storage. Keep the note close to the `service`, `database`, and `mlflow` config example so it is easy to find and consistent with the same URL used in `local-mode.mdx`.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/content/docs/guides/cli.mdx`:
- Around line 73-79: Clarify the CLI docs text around the `--unfold` and
`--unmask` flags so their interaction is unambiguous: the current wording
contradicts the example by calling them mutually exclusive while showing them
used together. Rewrite the description in the `config get` section to explicitly
state when each flag is allowed alone and what happens when they are combined,
using the same flag names and the `server_config_file` example to make the
behavior clear.
---
Nitpick comments:
In `@src/content/docs/guides/local-mode-tutorial.mdx`:
- Around line 34-52: The install commands for eval-hub-sdk are inconsistent
across the local-mode tutorial and the Local Mode guide, which can lead to users
installing different versions. Update the install step in the tutorial section
that uses uv pip install so it matches the version constraint used elsewhere,
aligning with the Since 0.4.3 marker. Keep the wording and command structure
consistent around the install instructions in the local-mode guide content.
- Around line 121-152: Clarify that the `database.url` value using
`file::eval_hub:?mode=memory&cache=shared` in the EvalHub server setup is
ephemeral and will not persist data after shutdown. Update the
`local-mode-tutorial.mdx` step around the `my-config.yaml` example to briefly
note that this in-memory SQLite configuration is intentional for the tutorial,
and point readers to using a file-based SQLite URL if they want persistent
storage. Keep the note close to the `service`, `database`, and `mlflow` config
example so it is easy to find and consistent with the same URL used in
`local-mode.mdx`.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: cfa36100-c0c0-4244-abd6-aa5c487efe9b
📒 Files selected for processing (4)
src/content/docs/guides/cli.mdxsrc/content/docs/guides/local-mode-tutorial.mdxsrc/content/docs/guides/local-mode.mdxsrc/content/docs/mcp/installation.mdx
Remove contradictory "mutually exclusive" language — the flags can be used independently or combined. Co-Authored-By: Claude <noreply@anthropic.com>
2 participants
What and why
Changes: evalhub CLI updates
evalhub serverto manage local evalhub serverevalhub mcpto replace the python based with Go basedeval-hub-sdkdirectly . It will hide eval-hub-server from end user now. And simplified the onboarding of config and provider.Closes # #81 #84
Type
Testing
Breaking changes
No
Summary by CodeRabbit