Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 46 additions & 9 deletions apps/web/src/content/docs/docs/evaluation/running-evals.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -390,9 +390,11 @@ agentv eval --strict evals/my-eval.yaml

## Config File Defaults

Set default execution options so you don't have to pass them on every CLI invocation. Both `.agentv/config.yaml` and `agentv.config.ts` are supported.
Set default execution options so you don't have to pass them on every CLI invocation. Project-local `.agentv/config.yaml`, home/global `$AGENTV_HOME/config.yaml` (or `~/.agentv/config.yaml`), and `agentv.config.ts` are supported.

### YAML config (`.agentv/config.yaml`)
Project-local YAML config takes precedence over home/global YAML config. AgentV uses the first config file it finds; it does not merge project and global YAML files.

### YAML config (`.agentv/config.yaml` or `$AGENTV_HOME/config.yaml`)

```yaml
execution:
Expand Down Expand Up @@ -423,35 +425,70 @@ export default defineConfig({

The `{timestamp}` placeholder is replaced with an ISO-like timestamp (e.g., `2026-03-05T14-30-00-000Z`) at execution time.

**Precedence:** CLI flags > `.agentv/config.yaml` > `agentv.config.ts` > built-in defaults.
**Precedence:** CLI flags > project-local `.agentv/config.yaml` > home/global `$AGENTV_HOME/config.yaml` (or `~/.agentv/config.yaml`) > `agentv.config.ts` > built-in defaults.

## Environment Variables

### AGENTV_HOME

Override the data directory for heavy runtime artifacts — workspaces, workspace pool, subagents, trace state, git cache, and downloaded dependencies. Lightweight config and cache files (`version-check.json`, `last-config.json`, `projects.yaml`) always stay in `~/.agentv` regardless of this setting.
Override AgentV's lightweight home/config directory. This directory stores files such as `config.yaml`, `projects.yaml`, `version-check.json`, `last-config.json`, and managed helper binaries.

```bash
# Linux/macOS
export AGENTV_HOME=/data/agentv
export AGENTV_HOME=/config/agentv

# Windows (PowerShell)
$env:AGENTV_HOME = "D:\agentv"
$env:AGENTV_HOME = "D:\agentv-config"

# Windows (CMD)
set AGENTV_HOME=D:\agentv
set AGENTV_HOME=D:\agentv-config
```

When set, AgentV logs `Using AGENTV_HOME: <path>` on startup to confirm the override is active.
When unset, AgentV uses `~/.agentv`.

### AGENTV_DATA_DIR

Override the heavy runtime data directory for workspaces, workspace pool, subagents, trace state, git caches, downloaded dependencies, and results repository clones. If `AGENTV_DATA_DIR` is unset, AgentV stores heavy data in `AGENTV_HOME` (or `~/.agentv`) for backward compatibility.

```bash
# Linux/macOS
export AGENTV_HOME=/config/agentv
export AGENTV_DATA_DIR=/data/agentv

# Windows (PowerShell)
$env:AGENTV_HOME = "D:\agentv-config"
$env:AGENTV_DATA_DIR = "E:\agentv-data"

# Windows (CMD)
set AGENTV_HOME=D:\agentv-config
set AGENTV_DATA_DIR=E:\agentv-data
```

:::tip[Windows long paths]
If you use a custom `AGENTV_HOME` on Windows for large monorepo workspaces, enable long path support:
If you use a custom `AGENTV_DATA_DIR` on Windows for large monorepo workspaces, enable long path support:
```powershell
git config --system core.longpaths true
```
Or set the registry key: `HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled = 1`
:::

### Docker directories

Keep the container user's `HOME`, AgentV config home, and AgentV heavy data directory separate so config files and large runtime artifacts can be mounted independently:

```bash
docker run --rm \
--user "$(id -u):$(id -g)" \
-e HOME=/home/agentv \
-e AGENTV_HOME=/home/agentv/.agentv \
-e AGENTV_DATA_DIR=/data/agentv \
-v agentv-home:/home/agentv/.agentv \
-v agentv-data:/data/agentv \
-v "$PWD:/workspace" \
-w /workspace \
agentv
```

## All Options

Run `agentv eval --help` for the full list of options including workers, timeouts, output formats, and trace dumping.
4 changes: 3 additions & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@ services:
environment:
- PORT=${PORT:-3117}
- HOME=/home/agentv
- AGENTV_HOME=/home/agentv/.agentv/data
- AGENTV_HOME=/home/agentv/.agentv
- AGENTV_DATA_DIR=/data/agentv
- ANTHROPIC_API_KEY
volumes:
- ${AGENTV_HOME_DIR:-./.agentv-docker}:/home/agentv/.agentv
- ${AGENTV_DATA_DIR_HOST:-./.agentv-data-docker}:/data/agentv
- ${AGENTV_PROJECTS_DIR:-./examples}:/data/projects/agentv-examples
- ${AGENTV_RESULTS_DIR:-./results}:/data/results/agentv-evalresults
command:
Expand Down
110 changes: 60 additions & 50 deletions packages/core/src/evaluation/loaders/config-loader.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { readFile } from 'node:fs/promises';
import path from 'node:path';

import { getAgentvConfigDir } from '../../paths.js';
import { interpolateEnv } from '../interpolation.js';
import type {
EvalTargetRef,
Expand Down Expand Up @@ -59,8 +60,12 @@ export type AgentVConfig = {
};

/**
* Load optional .agentv/config.yaml configuration file.
* Searches from eval file directory up to repo root.
* Load optional AgentV YAML configuration.
*
* Project-local `.agentv/config.yaml` files are searched from the eval file
* directory up to the repo root. If no project-local config is found, AgentV
* falls back to the home/global config at `${AGENTV_HOME:-~/.agentv}/config.yaml`.
* The first valid file wins; there is intentionally no cross-file merge.
*/
export async function loadConfig(
evalFilePath: string,
Expand All @@ -75,56 +80,61 @@ export async function loadConfig(
continue;
}

try {
const rawConfig = await readFile(configPath, 'utf8');
const parsed = interpolateEnv(parseYamlValue(rawConfig), process.env) as unknown;

if (!isJsonObject(parsed)) {
logWarning(`Invalid .agentv/config.yaml format at ${configPath}`);
continue;
}

const config = parsed as AgentVConfig;

const requiredVersion = (parsed as Record<string, unknown>).required_version;
if (requiredVersion !== undefined && typeof requiredVersion !== 'string') {
logWarning(`Invalid required_version in ${configPath}, expected string`);
continue;
}

const evalPatterns = (config as Record<string, unknown>).eval_patterns;
if (evalPatterns !== undefined && !Array.isArray(evalPatterns)) {
logWarning(`Invalid eval_patterns in ${configPath}, expected array`);
continue;
}

if (Array.isArray(evalPatterns) && !evalPatterns.every((p) => typeof p === 'string')) {
logWarning(`Invalid eval_patterns in ${configPath}, all entries must be strings`);
continue;
}

const executionDefaults = parseExecutionDefaults(
(parsed as Record<string, unknown>).execution,
configPath,
);
const results = parseResultsConfig((parsed as Record<string, unknown>).results, configPath);
const hooks = parseHooksConfig((parsed as Record<string, unknown>).hooks, configPath);

return {
required_version: requiredVersion as string | undefined,
eval_patterns: evalPatterns as readonly string[] | undefined,
execution: executionDefaults,
results,
...(hooks && { hooks }),
};
} catch (error) {
logWarning(
`Could not read .agentv/config.yaml at ${configPath}: ${(error as Error).message}`,
);
}
const config = await readConfigFile(configPath);
if (config) return config;
}

return null;
const globalConfigPath = path.join(getAgentvConfigDir(), 'config.yaml');
return (await fileExists(globalConfigPath)) ? readConfigFile(globalConfigPath) : null;
}

async function readConfigFile(configPath: string): Promise<AgentVConfig | null> {
try {
const rawConfig = await readFile(configPath, 'utf8');
const parsed = interpolateEnv(parseYamlValue(rawConfig), process.env) as unknown;

if (!isJsonObject(parsed)) {
logWarning(`Invalid config.yaml format at ${configPath}`);
return null;
}

const config = parsed as AgentVConfig;

const requiredVersion = (parsed as Record<string, unknown>).required_version;
if (requiredVersion !== undefined && typeof requiredVersion !== 'string') {
logWarning(`Invalid required_version in ${configPath}, expected string`);
return null;
}

const evalPatterns = (config as Record<string, unknown>).eval_patterns;
if (evalPatterns !== undefined && !Array.isArray(evalPatterns)) {
logWarning(`Invalid eval_patterns in ${configPath}, expected array`);
return null;
}

if (Array.isArray(evalPatterns) && !evalPatterns.every((p) => typeof p === 'string')) {
logWarning(`Invalid eval_patterns in ${configPath}, all entries must be strings`);
return null;
}

const executionDefaults = parseExecutionDefaults(
(parsed as Record<string, unknown>).execution,
configPath,
);
const results = parseResultsConfig((parsed as Record<string, unknown>).results, configPath);
const hooks = parseHooksConfig((parsed as Record<string, unknown>).hooks, configPath);

return {
required_version: requiredVersion as string | undefined,
eval_patterns: evalPatterns as readonly string[] | undefined,
execution: executionDefaults,
results,
...(hooks && { hooks }),
};
} catch (error) {
logWarning(`Could not read config.yaml at ${configPath}: ${(error as Error).message}`);
return null;
}
}

/**
Expand Down
4 changes: 2 additions & 2 deletions packages/core/src/evaluation/providers/pi-coding-agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import path from 'node:path';
import { createInterface } from 'node:readline';
import { fileURLToPath, pathToFileURL } from 'node:url';

import { getAgentvHome } from '../../paths.js';
import { getAgentvDataDir } from '../../paths.js';
import { recordPiLogEntry } from './pi-log-tracker.js';
import {
normalizeAzureSdkBaseUrl,
Expand Down Expand Up @@ -80,7 +80,7 @@ function findAgentvRoot(): string {
}

function findManagedSdkInstallRoot(): string {
return path.join(getAgentvHome(), 'deps', 'pi-sdk');
return path.join(getAgentvDataDir(), 'deps', 'pi-sdk');
}

function resolveGlobalNpmRoot(): string | undefined {
Expand Down
6 changes: 3 additions & 3 deletions packages/core/src/evaluation/results-repo.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import os from 'node:os';
import path from 'node:path';
import { promisify } from 'node:util';

import { getAgentvHome } from '../paths.js';
import { getAgentvDataDir } from '../paths.js';
import type { ResultsConfig } from './loaders/config-loader.js';

const execFileAsync = promisify(execFile);
Expand Down Expand Up @@ -82,7 +82,7 @@ export function normalizeResultsConfig(config: ResultsConfig): Required<ResultsC
const repo = config.repo.trim();
const resolvedPath = config.path
? expandHome(config.path.trim())
: path.join(getAgentvHome(), 'results', sanitizeRepoSlug(repo));
: path.join(getAgentvDataDir(), 'results', sanitizeRepoSlug(repo));
return {
mode: 'github',
repo,
Expand All @@ -100,7 +100,7 @@ export function resolveResultsRepoUrl(repo: string): string {
}

export function getResultsRepoLocalPaths(repo: string): ResultsRepoLocalPaths {
const rootDir = path.join(getAgentvHome(), 'cache', 'results-repo', sanitizeRepoSlug(repo));
const rootDir = path.join(getAgentvDataDir(), 'cache', 'results-repo', sanitizeRepoSlug(repo));
return {
rootDir,
repoDir: path.join(rootDir, 'repo'),
Expand Down
1 change: 1 addition & 0 deletions packages/core/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ export {
export {
getAgentvConfigDir,
getAgentvHome,
getAgentvDataDir,
getWorkspacesRoot,
getSubagentsRoot,
getTraceStateRoot,
Expand Down
55 changes: 30 additions & 25 deletions packages/core/src/paths.ts
Original file line number Diff line number Diff line change
@@ -1,51 +1,56 @@
import os from 'node:os';
import path from 'node:path';

let logged = false;
function readEnvPath(name: string): string | undefined {
const value = process.env[name];
if (!value || value === 'undefined') return undefined;
return value;
}

/**
* The default config directory (~/.agentv). Always resolves to the user's home
* directory regardless of AGENTV_HOME. Used for lightweight, machine-local files
* like version-check.json, last-config.json, and benchmarks.yaml.
* AgentV's lightweight home/config directory. Stores machine-local config files
* such as config.yaml, projects.yaml, version-check.json, last-config.json, and
* managed helper binaries. AGENTV_HOME relocates only this config/home surface.
*/
export function getAgentvConfigDir(): string {
return path.join(os.homedir(), '.agentv');
return readEnvPath('AGENTV_HOME') ?? path.join(os.homedir(), '.agentv');
}

/**
* The data root for heavy/large artifacts (workspaces, workspace-pool, subagents,
* trace-state, cache, deps). Respects AGENTV_HOME override so users can relocate
* bulky data to a different drive. Falls back to ~/.agentv when unset.
* Backward-compatible alias for AgentV's home/config directory.
* Prefer getAgentvConfigDir() for lightweight config files and
* getAgentvDataDir() for heavy runtime data.
*/
export function getAgentvHome(): string {
const envHome = process.env.AGENTV_HOME;
if (envHome && envHome !== 'undefined') {
if (!logged) {
logged = true;
console.log(`Using AGENTV_HOME: ${envHome}`);
}
return envHome;
}
return path.join(os.homedir(), '.agentv');
return getAgentvConfigDir();
}

/**
* AgentV's heavy runtime data directory. Stores workspaces, workspace pool,
* subagents, trace state, caches, downloaded dependencies, and results clones.
* AGENTV_DATA_DIR can separate this large data from AGENTV_HOME; when unset it
* falls back to AGENTV_HOME (or ~/.agentv) so existing AGENTV_HOME users keep
* their runtime data in the same location.
*/
export function getAgentvDataDir(): string {
return readEnvPath('AGENTV_DATA_DIR') ?? getAgentvConfigDir();
}

export function getWorkspacesRoot(): string {
return path.join(getAgentvHome(), 'workspaces');
return path.join(getAgentvDataDir(), 'workspaces');
}

export function getSubagentsRoot(): string {
return path.join(getAgentvHome(), 'subagents');
return path.join(getAgentvDataDir(), 'subagents');
}

export function getTraceStateRoot(): string {
return path.join(getAgentvHome(), 'trace-state');
return path.join(getAgentvDataDir(), 'trace-state');
}

export function getWorkspacePoolRoot(): string {
return path.join(getAgentvHome(), 'workspace-pool');
return path.join(getAgentvDataDir(), 'workspace-pool');
}

/** @internal Reset logged flag for testing. */
export function _resetLoggedForTesting(): void {
logged = false;
}
/** @internal Retained for older tests that reset path-helper state. */
export function _resetLoggedForTesting(): void {}
Loading
Loading