Skip to content

cli: drop-in flags, Allure reports and Maestro Cloud aliases (TB-374) #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jochen-testingbot merged 1 commit into from
Sep 2, 2026
Merged

cli: drop-in flags, Allure reports and Maestro Cloud aliases (TB-374) #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ testingbot maestro <app> <flows...> [options]
| `--groups <names>` | Tag the test session with one or more groups (comma-separated). Groups appear on the test in the TestingBot dashboard |
| `--include-tags <tags>` | Only run flows with these tags (comma-separated) |
| `--exclude-tags <tags>` | Exclude flows with these tags (comma-separated) |
| `--exclude-flows <paths>` | Flow files, directories or glob patterns to leave out of the run (comma-separated, repeatable). An excluded flow that another flow still invokes via `runFlow` is bundled as a subflow but never runs on its own |
| `-e, --env <KEY=VALUE>` | Environment variable for flows (can be repeated) |
| `--config <path>` | Path to a custom Maestro config file (default: config.yaml in project root) |
| `--maestro-version <version>` | Maestro version to use (e.g., "2.0.10") |
Expand Down Expand Up @@ -148,7 +149,7 @@ testingbot maestro <app> <flows...> [options]
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
| `--json-file` | Write results as JSON to a file (default: `<appId>_testingbot.json` in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
| `--json-file-name <path>` | Custom path for the JSON results file (requires `--json-file`) |
| `--report <format>` | Download report after completion: html or junit |
| `--report <format>` | Download report after completion: `html`, `html-detailed`, `junit` or `allure` |
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
| `--download-artifacts [mode]` | Download test artifacts (logs, screenshots, video). Mode: `all` (default) or `failed` |
| `--artifacts-output-dir <path>` | Directory to save artifacts zip (defaults to current directory) |
Expand All @@ -167,10 +168,17 @@ testingbot maestro <app> <flows...> [options]

| Option | Description |
|--------|-------------|
| `--branch <name>` | Git branch this test run was built from |
| `--commit-sha <sha>` | Git commit SHA associated with this test run |
| `--pull-request-id <id>` | Pull request ID this test run originated from |
| `--pr-url <url>` | Pull request URL this test run originated from |
| `--repo-name <name>` | Repository name (e.g., GitHub repo slug) |
| `--repo-owner <owner>` | Repository owner (e.g., GitHub organization or username) |
| `-m, --metadata <KEY=VALUE>` | Free-form metadata attached to the run and shown in the dashboard (repeatable, e.g. `-m team=mobile -m env=staging`) |

**Allure reports:** `--report allure` converts each run's results into Allure result files under `<report-output-dir>/allure-results/`, one JSON per flow with its steps, status and failure details. Render them with `allure serve <report-output-dir>/allure-results` (requires the [Allure CLI](https://allurereport.org/docs/install/)). Results from several runs or shards accumulate in the same directory.

**Migrating from Maestro Cloud:** the `maestro cloud` spelling of common flags is accepted as hidden aliases, so an existing command line runs unchanged: `--app-file`, `--flows <a,b>`, `--apiKey`, `--device-model iPhone-17-Pro`, `--device-os iOS-18-2` / `android-34`, `--format JUNIT|HTML`, `--output <file>` (its directory becomes `--report-output-dir`) and `--test-suite-name`. The canonical flag wins when both are given.

**Examples:**

Expand Down Expand Up @@ -357,7 +365,7 @@ Exit code is `0` while the project is still running (JSON `outcome: "running"`),

| Option | Description |
|--------|-------------|
| `--report <format>` | Download report: `html`, `html-detailed` or `junit` |
| `--report <format>` | Download report: `html`, `html-detailed`, `junit` or `allure` |
| `--report-output-dir <path>` | Directory to save reports (required with `--report`) |
| `--download-artifacts [mode]` | Download logs, screenshots and video. Mode: `all` (default) or `failed` |
| `--artifacts-output-dir <path>` | Directory to save the artifacts zip (defaults to current directory) |
Expand Down
233 changes: 221 additions & 12 deletions src/cli.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { Command, InvalidArgumentError } from 'commander';
import { Command, InvalidArgumentError, Option } from 'commander';
import logger, { enableDebugLogging } from './logger';
import Auth from './auth';
import Espresso from './providers/espresso';
Expand All @@ -23,6 +23,8 @@ import MaestroOptions, {
import Maestro from './providers/maestro';
import Login from './providers/login';
import Credentials from './models/credentials';
import path from 'node:path';
import type { RunMetadata } from './models/maestro_options';
import TestingBotError from './models/testingbot_error';
import { redirectLogsToStderr } from './logger';
import {
Expand Down Expand Up @@ -88,6 +90,147 @@ function parseProjectId(value: string): number {
return id;
}

/**
* Commander accumulator for repeatable flags. Returns a new array each time
* and takes no default, so nothing is shared between parses (a default `[]`
* would be mutated in place and leak values across invocations).
*/
function collectRepeatable(value: string, acc: string[] | undefined): string[] {
return [...(acc ?? []), value];
}

/** Like collectRepeatable, but each value may also be comma-separated. */
function collectCommaSeparated(
value: string,
acc: string[] | undefined,
): string[] {
const parts = value
.split(',')
.map((part) => part.trim())
.filter(Boolean);
return [...(acc ?? []), ...parts];
}

/** Parses `KEY=VALUE` entries; empty keys or a missing `=` are rejected. */
function parseKeyValues(
entries: string[] | undefined,
flag: string,
): Record<string, string> | undefined {
if (!entries || entries.length === 0) return undefined;
const result: Record<string, string> = {};
for (const entry of entries) {
const eq = entry.indexOf('=');
if (eq <= 0) {
throw new TestingBotError(
`Invalid ${flag} entry "${entry}": expected KEY=VALUE.`,
);
}
result[entry.slice(0, eq).trim()] = entry.slice(eq + 1);
}
return result;
}

/** Drops unset fields; returns undefined when nothing is set. */
function buildRunMetadata(fields: RunMetadata): RunMetadata | undefined {
const metadata: Record<string, unknown> = {};
for (const [key, value] of Object.entries(fields)) {
if (value !== undefined && value !== '') metadata[key] = value;
}
return Object.keys(metadata).length > 0
? (metadata as RunMetadata)
: undefined;
}

/** Android API level → OS version, for `--device-os android-34`. */
const ANDROID_API_TO_VERSION: Record<string, string> = {
'26': '8.0',
'27': '8.1',
'28': '9',
'29': '10',
'30': '11',
'31': '12',
'32': '12',
'33': '13',
'34': '14',
'35': '15',
'36': '16',
};

interface MaestroCloudAliasArgs {
appFile?: string;
flows?: string;
deviceModel?: string;
deviceOs?: string;
format?: string;
output?: string;
testSuiteName?: string;
app?: string;
device?: string;
platform?: 'Android' | 'iOS';
deviceVersion?: string;
report?: ReportFormat;
reportOutputDir?: string;
name?: string;
}

/**
* Maestro Cloud spells several flags differently (`maestro cloud --app-file
* app.zip --flows ./flows --device-os iOS-18-2 --format JUNIT`). These hidden
* aliases let such a command line run unchanged. Canonical flags win when
* both are given. Returns extra flow paths from `--flows`.
*/
function applyMaestroCloudAliases(args: MaestroCloudAliasArgs): string[] {
if (args.appFile && !args.app) args.app = args.appFile;

const extraFlows = (args.flows ?? '')
.split(',')
.map((f) => f.trim())
.filter(Boolean);

if (args.deviceModel && !args.device) {
// Maestro Cloud model ids are dash/underscore-joined: iPhone-17-Pro, pixel_7
args.device = args.deviceModel.replace(/[-_]+/g, ' ');
}

if (args.deviceOs) {
const match = /^(ios|android)[-_]?(.*)$/i.exec(args.deviceOs.trim());
if (!match) {
throw new TestingBotError(
`Invalid --device-os "${args.deviceOs}": expected iOS-<major>[-<minor>] or android-<api level>.`,
);
}
const isIos = match[1].toLowerCase() === 'ios';
if (!args.platform) args.platform = isIos ? 'iOS' : 'Android';
if (!args.deviceVersion && match[2]) {
const raw = match[2].replace(/-/g, '.');
args.deviceVersion = isIos ? raw : (ANDROID_API_TO_VERSION[raw] ?? raw);
}
}

if (args.format && !args.report) {
const format = args.format.toLowerCase();
if (format === 'junit' || format === 'html') {
args.report = format;
} else if (format !== 'noop') {
throw new TestingBotError(
`Invalid --format "${args.format}": expected JUNIT, HTML or NOOP.`,
);
}
}

if (args.output && !args.reportOutputDir) {
args.reportOutputDir = path.dirname(path.resolve(args.output));
if (args.report) {
logger.warn(
`--output is mapped to --report-output-dir ${args.reportOutputDir}; reports are named report_run_<id>.<ext>`,
);
}
}

if (args.testSuiteName && !args.name) args.name = args.testSuiteName;
return extraFlows;
}

/** Emits JSON output (if requested) and sets the exit code for a finished run. */
async function finishCommand(
output: JsonOutput,
Expand Down Expand Up @@ -445,6 +588,11 @@ program
'Exclude flows with these tags (comma-separated).',
(val) => val.split(',').map((t) => t.trim()),
)
.option(
'--exclude-flows <paths>',
'Flow files, directories or globs to leave out (comma-separated, repeatable).',
collectCommaSeparated,
)
// Environment variables
.option(
'-e, --env <KEY=VALUE>',
Expand Down Expand Up @@ -489,7 +637,7 @@ program
// Report options
.option(
'--report <format>',
'Download test report after completion: html, html-detailed, or junit.',
'Download test report after completion: html, html-detailed, junit, or allure.',
(val) => val.toLowerCase() as ReportFormat,
)
.option(
Expand All @@ -516,7 +664,17 @@ program
(val) => parseInt(val, 10),
)
// CI/CD metadata
.option('--branch <name>', 'Git branch this upload was built from.')
.option('--commit-sha <sha>', 'The commit SHA of this upload.')
.option(
'--pr-url <url>',
'URL of the pull request this upload originated from.',
)
.option(
'-m, --metadata <KEY=VALUE>',
'Arbitrary metadata to attach to the run, shown in the dashboard (repeatable).',
collectRepeatable,
)
.option(
'--pull-request-id <id>',
'The ID of the pull request this upload originated from.',
Expand All @@ -543,10 +701,59 @@ program
'--json-file-name <path>',
'Custom path for the JSON results file (requires --json-file).',
)
.addOption(
new Option(
'--app-file <path>',
'Alias of --app (Maestro Cloud compatibility).',
).hideHelp(),
)
.addOption(
new Option(
'--flows <paths>',
'Comma-separated flow paths (Maestro Cloud compatibility).',
).hideHelp(),
)
.addOption(
new Option(
'--apiKey <key>',
'Alias of --api-key (Maestro Cloud compatibility).',
).hideHelp(),
)
.addOption(
new Option(
'--device-model <model>',
'Alias of --device, e.g. iPhone-17-Pro (Maestro Cloud compatibility).',
).hideHelp(),
)
.addOption(
new Option(
'--device-os <os>',
'Platform and OS version, e.g. iOS-18-2 or android-34 (Maestro Cloud compatibility).',
).hideHelp(),
)
.addOption(
new Option(
'--format <format>',
'Alias of --report: JUNIT or HTML (Maestro Cloud compatibility).',
).hideHelp(),
)
.addOption(
new Option(
'--output <path>',
'Report file path; its directory becomes --report-output-dir (Maestro Cloud compatibility).',
).hideHelp(),
)
.addOption(
new Option(
'--test-suite-name <name>',
'Alias of --name (Maestro Cloud compatibility).',
).hideHelp(),
)
.action(async (appFileArg, flowsArgs, args) => {
let jsonOptions: JsonOutputOptions | undefined;
try {
jsonOptions = jsonOptionsFrom(args);
const aliasFlows = applyMaestroCloudAliases(args);
let app: string;
let flows: string[];

Expand All @@ -561,6 +768,7 @@ program
app = appFileArg;
flows = flowsArgs || [];
}
flows = [...flows, ...aliasFlows];

const missing: string[] = [];
if (!app && args.appBinaryId == null)
Expand Down Expand Up @@ -612,19 +820,20 @@ program
}
}

const metadata =
args.commitSha || args.pullRequestId || args.repoName || args.repoOwner
? {
commitSha: args.commitSha,
pullRequestId: args.pullRequestId,
repoName: args.repoName,
repoOwner: args.repoOwner,
}
: undefined;
const metadata = buildRunMetadata({
commitSha: args.commitSha,
pullRequestId: args.pullRequestId,
pullRequestUrl: args.prUrl,
repoName: args.repoName,
repoOwner: args.repoOwner,
branch: args.branch,
custom: parseKeyValues(args.metadata, '--metadata'),
});

const options = new MaestroOptions(app, flows, args.device, {
includeTags: args.includeTags,
excludeTags: args.excludeTags,
excludeFlows: args.excludeFlows,
platformName: args.platform,
version: args.deviceVersion,
name: args.name,
Expand Down Expand Up @@ -1003,7 +1212,7 @@ withFlags(
)
.option(
'--report <format>',
'Download test report: html, html-detailed, or junit.',
'Download test report: html, html-detailed, junit, or allure.',
(val) => val.toLowerCase() as ReportFormat,
)
.option(
Expand Down
Loading
Loading