diff --git a/History.md b/History.md index 9447bfd..5be5f45 100644 --- a/History.md +++ b/History.md @@ -1,3 +1,20 @@ +### v4.0.0 + - [#46](https://github.com/darkguy2008/parallelshell/issues/46) Add + `-p, --prefix` to label every output line with its command and + `-l, --label ` to name a command, with colored labels like + `docker compose`. + - [#22](https://github.com/darkguy2008/parallelshell/issues/22) Stop the + commands behind `npm run` too: on Unix each command runs in its own process + group and parallelshell stops the whole group. + - [#28](https://github.com/darkguy2008/parallelshell/issues/28) Run commands + without `exec`, so `export`, `cd` and `a && b` work. + - Stop background processes a command left behind and wait for every stopped + process before exiting. Ctrl+Z and `fg` pause and resume commands. Commands + are stopped even if parallelshell itself is killed. + - Breaking: a failing command, `--timeout` and closed output now stop the + others with SIGTERM instead of SIGINT. Commands can't read from the terminal + anymore (use askpass or credential helpers for passwords). + ### v3.2.1 - [#31](https://github.com/darkguy2008/parallelshell/issues/31) Report a missing npm once instead of also marking the script as successful. diff --git a/README.md b/README.md index a29750c..0cdcf05 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,9 @@ ## Parallel Shell This is a super simple npm module to run shell commands in parallel. All -processes will share the same stdout/stderr, and if any command exits with a -non-zero exit status, the rest are stopped and the exit code carries through. +processes will share the same stdout/stderr, optionally with every line labelled, +and if any command exits with a non-zero exit status, the rest are stopped and +the exit code carries through. ### Version compatibility notes @@ -69,6 +70,8 @@ Available options: -w, --wait will not close sibling processes on error -t, --timeout stop remaining commands after the deadline -n, --npm run matching npm scripts from package.json +-p, --prefix prefix each output line with its command's label +-l, --label label the next command, implies --prefix ``` Use `-n` (or `--npm`) before each npm script name or @@ -94,4 +97,69 @@ parallelshell --timeout 10 "node server.js" "node request.js" On timeout parallelshell exits with code 124, or with an earlier failure's code under `--wait`. Commands are stopped the same way as on failure, so on Unix a -command that ignores SIGINT can outlive the deadline. +command that ignores SIGTERM can outlive the deadline. + +### Prefixed output + +Use `-p` (or `--prefix`) to start every line with the command it came from, like +`docker compose` does. Put `-l` (or `--label`) before a command to name it, which +turns on `--prefix` by itself: + +```bash +parallelshell -l api "node server.js" -l web "npm run watch" +``` + +``` +api | listening on port 3000 +web | compiled in 120ms +``` + +npm scripts are labelled with their name, other commands with their text +shortened to 10 characters. A label names one command, so it can't go before an +`-n` pattern matching several scripts. Labels are padded to the same width, each +command gets its own color and stderr lines stay on stderr. A line that isn't +finished yet shows up right away. If another command prints in the meantime, it +carries on in a new line with its label. + +Colors are used when the output is a terminal. `NO_COLOR` turns them off and +`FORCE_COLOR` turns them on (`FORCE_COLOR=0` turns them off). + +To label lines parallelshell reads each command's output through a pipe, so +commands no longer see a terminal: + +* When the labels are colored, commands get `FORCE_COLOR` and `CLICOLOR_FORCE` + (unless you set them) so chalk based tools, npm, jest or macOS `ls` stay + colored. Tools that ignore both, like git or cargo, need their own flag such as + `--color=always`. Everything a command starts sees these variables too, so + output written to a file can end up with color codes: set `FORCE_COLOR=0` to + avoid that. +* Progress bars, spinners and interactive modes that need a terminal are turned + off by the tools themselves. +* Python and Ruby buffer their output when it isn't a terminal, so it can show up + late. Set `PYTHONUNBUFFERED=1` for Python and `$stdout.sync = true` in Ruby. + +### Stopping commands + +On Unix every command runs in its own process group and parallelshell stops the +whole group. Ctrl+C, SIGTERM and SIGHUP are passed on as they are. A failing +command, the `--timeout` deadline or closed output stop the others with SIGTERM. +Background processes a command left behind get SIGTERM too. That reaches the +commands behind `npm run` as well. Commands can use any shell syntax, like +`export PORT=3000 && cd api && npm start`. + +* parallelshell exits once everything it stopped is gone, even when a command's + shell ends before the processes it started. Only daemons that close every + inherited file descriptor aren't waited for. +* Ctrl+Z pauses every command and `fg` resumes them. +* If parallelshell is killed with SIGKILL, a small helper stops the commands. +* When every command finishes by itself, background processes they started keep + running. +* Commands can't read from the terminal, so password prompts fail right away + instead of hanging. Use the non-interactive options instead: `sudo -A` with + `SUDO_ASKPASS` for sudo. `SSH_ASKPASS` with `SSH_ASKPASS_REQUIRE=force` or + ssh-agent for ssh. `GIT_ASKPASS` or a credential helper for git. + +On Windows parallelshell stops each command's process tree with `taskkill`. A +process started with `start /b` by a command that already exited is outside that +tree and keeps running. With `--prefix` parallelshell stops waiting for its +output. diff --git a/index.js b/index.js index d238649..e1b3b34 100755 --- a/index.js +++ b/index.js @@ -7,15 +7,44 @@ var path = require('path'); var signals = require('constants'); var SIGNAL_EXIT_CODE_BASE = 128; var FORWARDED_SIGNALS = ['SIGINT', 'SIGTERM', 'SIGHUP']; +var STOP_SIGNAL = 'SIGTERM'; +var UNSIGNALABLE_GROUP_ERRORS = ['ESRCH', 'EPERM']; +var CLOSED_READER_ERRORS = ['EPIPE', 'ECONNRESET']; +var LIFETIME_FD = 3; +var LABEL_USAGE = '--label requires a name followed by a command'; var WINDOWS_CONTROL_C_EXIT = 0xC000013A; var TIMEOUT_EXIT_CODE = 124; var MILLISECONDS_PER_SECOND = 1000; var MAX_TIMEOUT_MS = Math.pow(2, 31) - 1; var SCRIPT_PATTERN_OPTIONS = { dot: true, nocomment: true, nonegate: true, allowWindowsEscape: true }; +var CONCURRENTLY_PREFIX_LENGTH = 10; +var LABEL_ELLIPSIS = '..'; +var PREFIX_SEPARATOR = ' | '; +var DOCKER_COMPOSE_COLORS = ['36', '33', '32', '35', '34', '36;1', '33;1', '32;1', '35;1', '34;1']; +var BASIC_COLOR_LEVEL = '1'; +var FORCE_COLOR_ENABLED = ['', BASIC_COLOR_LEVEL, '2', '3', 'true']; +var FORCE_COLOR_LEVELS = { 4: '1', 8: '2', 24: '3' }; +var LINE_ENDINGS = /(\r\n|\r|\n)/; +var CARRIAGE_RETURN = '\r'; +var WATCHDOG_ADD = 'add'; +var WATCHDOG_FORGET = 'forget'; +var WATCHDOG_RELEASE = 'release'; +var WATCHDOG_SCRIPT = [ + "trap '' INT TERM HUP", + 'groups=', + 'while read -r action group; do', + ' case $action in', + ' ' + WATCHDOG_ADD + ') groups="$groups $group" ;;', + ' ' + WATCHDOG_FORGET + ') kept=; for tracked in $groups; do [ "$tracked" = "$group" ] || kept="$kept $tracked"; done; groups=$kept ;;', + ' ' + WATCHDOG_RELEASE + ') exit ;;', + ' esac', + 'done', + 'for group in $groups; do kill -' + STOP_SIGNAL.slice('SIG'.length) + ' -$group; done' +].join('\n'); var WINDOWS = process.platform === 'win32'; -var commandPrefix = WINDOWS ? '' : 'exec '; -var children, args, wait, cmds, verbose, timeout, timer, scripts, i ,len; +var children, args, wait, cmds, verbose, timeout, timer, scripts, prefix, label, stopping, stopSignal, watchdog, i ,len; +var groups = []; cmds = []; args = process.argv.slice(2); for (i = 0, len = args.length; i < len; i++) { @@ -23,23 +52,35 @@ for (i = 0, len = args.length; i < len; i++) { switch (args[i]) { case '-n': case '--npm': + var pattern = args[++i]; + var names; try { - matchingScripts(args[++i]).forEach(function (name) { - cmds.push({ cmd: 'npm run -- ' + name, script: name }); - }); + names = matchingScripts(pattern); } catch (error) { - console.error('--npm: ' + error.message); - process.exit(1); + usageError('--npm: ' + error.message); } + if (label !== undefined && names.length > 1) usageError('--label names one command but ' + JSON.stringify(pattern) + ' matches ' + names.length + ' npm scripts'); + names.forEach(function (name) { + cmds.push({ cmd: 'npm run -- ' + name, script: name, label: takeLabel(name) }); + }); break; case '-t': case '--timeout': timeout = Number(args[++i]); if (!(timeout > 0 && timeout * MILLISECONDS_PER_SECOND <= MAX_TIMEOUT_MS)) { - console.error('--timeout requires positive seconds no greater than ' + MAX_TIMEOUT_MS / MILLISECONDS_PER_SECOND); - process.exit(1); + usageError('--timeout requires positive seconds no greater than ' + MAX_TIMEOUT_MS / MILLISECONDS_PER_SECOND); } break; + case '-p': + case '--prefix': + prefix = true; + break; + case '-l': + case '--label': + if (label !== undefined || !args[i + 1] || args[i + 1][0] === '-') usageError(LABEL_USAGE); + label = args[++i]; + prefix = true; + break; case '-w': case '--wait': wait = true; @@ -55,13 +96,37 @@ for (i = 0, len = args.length; i < len; i++) { console.log('-w, --wait will not close sibling processes on error'); console.log('-t, --timeout stop remaining commands after the deadline'); console.log('-n, --npm run matching npm scripts from package.json'); + console.log('-p, --prefix prefix each output line with its command\'s label'); + console.log('-l, --label label the next command, implies --prefix'); process.exit(); break; } } else { - cmds.push({ cmd: commandPrefix + args[i] }); + cmds.push({ cmd: args[i], label: takeLabel(shorten(args[i])) }); } } +if (label !== undefined) usageError(LABEL_USAGE); + +function usageError (message) { + console.error(message); + process.exit(1); +} + +function rethrowUnless (codes, error) { + if (codes.indexOf(error.code) === -1) throw error; +} + +function takeLabel (fallback) { + var taken = label === undefined ? fallback : label; + label = undefined; + return taken; +} + +function shorten (text) { + if (text.length <= CONCURRENTLY_PREFIX_LENGTH) return text; + var kept = CONCURRENTLY_PREFIX_LENGTH - LABEL_ELLIPSIS.length; + return text.slice(0, Math.ceil(kept / 2)) + LABEL_ELLIPSIS + text.slice(text.length - Math.floor(kept / 2)); +} function matchingScripts (pattern) { if (!pattern || pattern[0] === '-') throw new Error('requires a script name or pattern'); @@ -75,17 +140,96 @@ function matchingScripts (pattern) { return matches; } +function colorsEnabled (stream) { + if (process.env.FORCE_COLOR !== undefined) return FORCE_COLOR_ENABLED.indexOf(process.env.FORCE_COLOR) !== -1; + return !process.env.NO_COLOR && Boolean(stream.isTTY) && (!stream.hasColors || stream.hasColors()); +} + +function childEnvironment () { + var env = Object.assign({}, process.env); + if (env.FORCE_COLOR === undefined && !env.NO_COLOR && outputs.stdout.colored) { + env.FORCE_COLOR = process.stdout.getColorDepth ? FORCE_COLOR_LEVELS[process.stdout.getColorDepth()] : BASIC_COLOR_LEVEL; + if (env.CLICOLOR_FORCE === undefined) env.CLICOLOR_FORCE = BASIC_COLOR_LEVEL; + } + return env; +} + +function output (stream) { + var target = { stream: stream, colored: colorsEnabled(stream), writer: null, lineOpen: false, paused: [], broken: false }; + stream.on('error', function (error) { + rethrowUnless(CLOSED_READER_ERRORS, error); + target.broken = true; + process.exitCode = process.exitCode || SIGNAL_EXIT_CODE_BASE + signals.SIGPIPE; + close(); + }); + return target; +} + +var outputs = { stdout: output(process.stdout), stderr: output(process.stderr) }; + +function labelFor (cmd, index, target) { + var text = cmd.label + ' '.repeat(labelWidth - cmd.label.length) + PREFIX_SEPARATOR; + return target.colored ? '\x1b[' + DOCKER_COMPOSE_COLORS[index % DOCKER_COMPOSE_COLORS.length] + 'm' + text + '\x1b[0m' : text; +} + +function write (target, writer, linePrefix, text, source) { + if (!text || target.broken) return; + var result = ''; + if (target.lineOpen && target.writer !== writer) { + result = '\n'; + target.lineOpen = false; + } + text.split(LINE_ENDINGS).forEach(function (piece, position) { + if (!piece) return; + var ending = position % 2 === 1; + if (!target.lineOpen && piece !== CARRIAGE_RETURN) result += linePrefix; + result += piece; + target.lineOpen = !ending; + }); + target.writer = writer; + if (!target.stream.write(result) && source) { + source.pause(); + if (target.paused.push(source) === 1) target.stream.once('drain', function () { + target.paused.splice(0).forEach(function (paused) { paused.resume(); }); + }); + } +} + +function relay (source, target, writer, linePrefix) { + var pendingReturn = ''; + source.setEncoding('utf8'); + source.on('data', function (text) { + text = pendingReturn + text; + pendingReturn = text[text.length - 1] === CARRIAGE_RETURN ? CARRIAGE_RETURN : ''; + write(target, writer, linePrefix, text.slice(0, text.length - pendingReturn.length), source); + }).on('end', function () { + write(target, writer, linePrefix, pendingReturn, source); + }); +} + +function say (target, message) { + write(target, null, '', message + '\n'); +} + +function isClosed (child) { + return child.closed; +} + +function hasExited (child) { + return child.exitCode !== null || child.signalCode !== null; +} + function isRunning (child) { - return child.exitCode === null && child.signalCode === null; + return !isClosed(child) && !hasExited(child); } function childClose (code, signal) { code = signal ? SIGNAL_EXIT_CODE_BASE + signals[signal] : code; if (verbose) { if (code > 0) { - console.error('`' + this.cmd + '` failed with exit code ' + code); + say(outputs.stderr, '`' + this.cmd + '` failed with exit code ' + code); } else { - console.log('`' + this.cmd + '` ended successfully'); + say(outputs.stdout, '`' + this.cmd + '` ended successfully'); } } if (code > 0) { @@ -97,84 +241,177 @@ function childClose (code, signal) { status(); } +function childClosed () { + this.closed = true; + if (groups.indexOf(this) !== -1) forgetGroup(this); + settle(); +} + +function settle () { + if (!children.every(isClosed)) return; + if (stopping) { + exit(stopSignal); + } else { + release(function () {}); + } +} + function status () { if (verbose) { var i, len; - console.log('\n'); - console.log('### Status ###'); + say(outputs.stdout, '\n'); + say(outputs.stdout, '### Status ###'); for (i = 0, len = children.length; i < len; i++) { - if (isRunning(children[i])) { - console.log('`' + children[i].cmd + '` is still running'); + if (!isClosed(children[i])) { + say(outputs.stdout, '`' + children[i].cmd + '` is still running'); } else if (children[i].exitCode !== 0) { - console.log('`' + children[i].cmd + '` errored'); + say(outputs.stdout, '`' + children[i].cmd + '` errored'); } else { - console.log('`' + children[i].cmd + '` finished'); + say(outputs.stdout, '`' + children[i].cmd + '` finished'); } } - console.log('\n'); + say(outputs.stdout, '\n'); } } function close (signal) { clearTimeout(timer); - var running = children.filter(isRunning); - var remaining = running.length; - if (remaining === 0) return exit(signal); - running.forEach(function (child) { - child.removeAllListeners('close'); - if (verbose) console.log('`' + child.cmd + '` will now be closed'); - child.on('close', function () { - remaining--; - if (remaining === 0) exit(signal); + if (signal) stopSignal = signal; + if (!stopping) { + stopping = true; + children.forEach(function (child) { + child.removeListener('close', childClose); + if (verbose && !isClosed(child)) say(outputs.stdout, '`' + child.cmd + '` will now be closed'); }); - }); - stop(running, signal || 'SIGINT'); + } + stop(signal || STOP_SIGNAL); + settle(); } -function stop (running, signal) { +function stop (signal) { if (WINDOWS) { - spawn(path.join(process.env.SystemRoot, 'System32', 'taskkill.exe'), running.reduce(function (taskkillArgs, child) { + var running = children.filter(isRunning); + if (running.length) spawn(path.join(process.env.SystemRoot, 'System32', 'taskkill.exe'), running.reduce(function (taskkillArgs, child) { return taskkillArgs.concat('/PID', String(child.pid)); }, ['/T', '/F']), { stdio: 'ignore' }); + children.filter(function (child) { return !isClosed(child) && hasExited(child) && child.stdout; }).forEach(function (child) { + child.stdout.destroy(); + child.stderr.destroy(); + }); } else { - running.forEach(function (child) { - child.kill(signal); + groups.forEach(function (child) { + signalGroup(child, hasExited(child) ? STOP_SIGNAL : signal); }); } } -function exit (signal) { - if (signal && WINDOWS) { - process.exit(WINDOWS_CONTROL_C_EXIT); - } else if (signal) { - process.removeAllListeners(signal); - process.kill(process.pid, signal); - } else { - process.exit(); +function signalGroup (child, signal) { + try { + process.kill(-child.pid, signal); + } catch (error) { + rethrowUnless(UNSIGNALABLE_GROUP_ERRORS, error); } } +function forgetGroup (child) { + try { + process.kill(-child.pid, 0); + } catch (error) { + rethrowUnless(UNSIGNALABLE_GROUP_ERRORS, error); + if (error.code === 'EPERM') return; + groups.splice(groups.indexOf(child), 1); + watchdog.stdin.write(WATCHDOG_FORGET + ' ' + child.pid + '\n'); + } +} + +function release (callback) { + if (!watchdog || hasExited(watchdog)) return callback(); + watchdog.ref(); + watchdog.once('exit', callback); + if (watchdog.stdin.writable) watchdog.stdin.end(WATCHDOG_RELEASE + '\n'); +} + +function flush (target, callback) { + if (target.broken) return callback(); + target.stream.write('', callback); +} + +function exit (signal) { + release(function () { + flush(outputs.stdout, function () { + flush(outputs.stderr, function () { + if (signal && WINDOWS) { + process.exit(WINDOWS_CONTROL_C_EXIT); + } else if (signal) { + process.removeAllListeners(signal); + process.kill(process.pid, signal); + } else { + process.exit(); + } + }); + }); + }); +} + FORWARDED_SIGNALS.forEach(function (signal) { process.once(signal, function () { close(signal); }); }); -children = cmds.map(function (cmd) { - var stdio = ['pipe', process.stdout, process.stderr]; +if (!WINDOWS && cmds.length) { + watchdog = spawn(WATCHDOG_SCRIPT, { shell: true, stdio: ['pipe', 'ignore', 'ignore'] }); + watchdog.unref(); + watchdog.stdin.on('error', function (error) { + rethrowUnless(CLOSED_READER_ERRORS, error); + }); + process.on('SIGTSTP', function () { + groups.forEach(function (child) { signalGroup(child, 'SIGSTOP'); }); + process.kill(process.pid, 'SIGSTOP'); + }); + process.on('SIGCONT', function () { + groups.forEach(function (child) { signalGroup(child, 'SIGCONT'); }); + }); +} + +var env = prefix ? childEnvironment() : process.env; +var labelWidth = cmds.reduce(function (longest, cmd) { return Math.max(longest, cmd.label.length); }, 0); +children = cmds.map(function (cmd, index) { + var options = { + stdio: (prefix ? ['pipe', 'pipe', 'pipe'] : ['pipe', process.stdout, process.stderr]).concat(WINDOWS ? [] : ['pipe']), + detached: !WINDOWS, + env: env, + shell: cmd.script === undefined + }; var child = cmd.script === undefined - ? spawn(cmd.cmd, { shell: true, stdio: stdio }) - : require('cross-spawn')('npm', ['run', '--', cmd.script], { stdio: stdio }); + ? spawn(cmd.cmd, options) + : require('cross-spawn')('npm', ['run', '--', cmd.script], options); child.on('error', function (error) { - console.error(error.message); + say(outputs.stderr, error.message); this.removeListener('close', childClose); childClose.call(this, 1); - }).on('close', childClose); + }).on('close', childClosed).on('close', childClose); child.cmd = cmd.cmd; + if (prefix) { + relay(child.stdout, outputs.stdout, child, labelFor(cmd, index, outputs.stdout)); + relay(child.stderr, outputs.stderr, child, labelFor(cmd, index, outputs.stderr)); + } + if (watchdog && child.pid) { + groups.push(child); + watchdog.stdin.write(WATCHDOG_ADD + ' ' + child.pid + '\n'); + child.stdio[LIFETIME_FD].resume(); + child.on('exit', function () { + if (stopping) { + signalGroup(this, STOP_SIGNAL); + } else { + this.stdio[LIFETIME_FD].destroy(); + } + }); + } return child; }); if (timeout) { timer = setTimeout(function () { - console.error('parallelshell timed out after ' + timeout + ' seconds'); + say(outputs.stderr, 'parallelshell timed out after ' + timeout + ' seconds'); process.exitCode = process.exitCode || TIMEOUT_EXIT_CODE; close(); }, timeout * MILLISECONDS_PER_SECOND); diff --git a/package.json b/package.json index cc7c111..e574054 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "parallelshell", - "version": "3.2.1", + "version": "4.0.0", "description": "Invoke multiple commands, running in parallel", "homepage": "https://github.com/darkguy2008/parallelshell", "repository": { diff --git a/test/fixtures/staged-output.js b/test/fixtures/staged-output.js new file mode 100644 index 0000000..648080e --- /dev/null +++ b/test/fixtures/staged-output.js @@ -0,0 +1,15 @@ +'use strict'; +var whenTriggered = require('./trigger'); + +var stages = process.argv.slice(2); + +function nextStage () { + if (!stages.length) return; + var text = Buffer.from(stages[1], 'hex').toString(); + whenTriggered(stages[0], function () { + process.stdout.write(text, nextStage); + }); + stages = stages.slice(2); +} + +nextStage(); diff --git a/test/fixtures/trigger.js b/test/fixtures/trigger.js new file mode 100644 index 0000000..ecb509a --- /dev/null +++ b/test/fixtures/trigger.js @@ -0,0 +1,14 @@ +'use strict'; +var fs = require('fs'); +var path = require('path'); + +module.exports = function whenTriggered (trigger, callback) { + var watcher = fs.watch(path.dirname(trigger), check); + function check () { + if (!watcher || !fs.existsSync(trigger)) return; + watcher.close(); + watcher = null; + callback(); + } + check(); +}; diff --git a/test/fixtures/waiting.js b/test/fixtures/waiting.js index caf12fd..107ab89 100644 --- a/test/fixtures/waiting.js +++ b/test/fixtures/waiting.js @@ -1,17 +1,13 @@ 'use strict'; -var fs = require('fs'); -var path = require('path'); +var whenTriggered = require('./trigger'); var trigger = process.argv[2]; var code = Number(process.argv[3]); +var ignoredSignal = process.argv[4]; -function exitIfTriggered () { - if (fs.existsSync(trigger)) { - console.log('done'); - process.exit(code); - } -} - -fs.watch(path.dirname(trigger), exitIfTriggered); +if (ignoredSignal) process.on(ignoredSignal, function () {}); console.log('ready ' + process.pid); -exitIfTriggered(); +whenTriggered(trigger, function () { + console.log('done'); + process.exit(code); +}); diff --git a/test/index.coffee b/test/index.coffee index 541dd24..227677a 100644 --- a/test/index.coffee +++ b/test/index.coffee @@ -32,6 +32,15 @@ TIMEOUT_EXIT_CODE = 124 COMPLETION_TIMEOUT_SECONDS = "60" FRACTIONAL_TIMEOUT_SECONDS = "0.001" PACKAGE_FILES = ["package.json", "npm-debug.log"] +COLOR_VARIABLES = ["FORCE_COLOR", "NO_COLOR", "CLICOLOR_FORCE"] +IGNORED_SIGNAL = "SIGINT" +SIGPIPE_EXIT_CODE = SIGNAL_EXIT_CODE_BASE + signals.SIGPIPE +MANY_COMMANDS = 11 +FAILURE_OUTPUT_LINES = 2000 +POLL_INTERVAL_MS = 20 +MAX_POLLS = 250 +STOPPED_STATE = "T" +ZOMBIE_STATE = "Z" fixture = (name, args...) -> [process.execPath, path.join(FIXTURES_DIR, name)].concat(args).join " " exitProcess = (code) -> fixture "exit.js", code @@ -39,6 +48,7 @@ failingProcess = exitProcess 1 succeedingProcess = exitProcess 0 printCwdProcess = "#{process.execPath} -p \"process.cwd()\"" printEnvProcess = "#{process.execPath} -p process.env.#{ENV_NAME}" +evalProcess = (source) -> "#{process.execPath} -e \"#{source}\"" usageInfo = """ -h, --help output usage information @@ -46,6 +56,8 @@ usageInfo = """ -w, --wait will not close sibling processes on error -t, --timeout stop remaining commands after the deadline -n, --npm run matching npm scripts from package.json +-p, --prefix prefix each output line with its command's label +-l, --label label the next command, implies --prefix """ + "\n" spawned = [] @@ -59,9 +71,44 @@ newDirectory = -> newTrigger = -> path.join newDirectory(), TRIGGER_NAME +project = (scripts) -> + directory = newDirectory() + fs.writeFileSync path.join(directory, "package.json"), JSON.stringify {name: "parallelshell-fixture", version: "1.0.0", scripts} + directory + release = (trigger) -> fs.writeFileSync trigger, "" -waitingProcess = (trigger = newTrigger(), code = 0) -> fixture "waiting.js", trigger, code +waitingProcess = (trigger = newTrigger(), code = 0, ignoredSignal = "") -> fixture "waiting.js", trigger, code, ignoredSignal + +hex = (text) -> Buffer.from(text).toString "hex" + +stagedProcess = (stages...) -> + fixture "staged-output.js", [].concat(([trigger, hex text] for [trigger, text] in stages)...)... + +colorEnv = (overrides = {}) -> + env = {} + env[name] = value for name, value of process.env when name not in COLOR_VARIABLES + Object.assign env, overrides + +waitUntil = (predicate) -> + new Promise (resolve, reject) -> + polls = 0 + poll = -> + return resolve() if predicate() + polls++ + return reject new Error "condition not met after #{MAX_POLLS} polls" if polls >= MAX_POLLS + setTimeout poll, POLL_INTERVAL_MS + poll() + +processState = (pid) -> + procStat = "/proc/#{pid}/stat" + try + if fs.existsSync "/proc" + fs.readFileSync(procStat, "utf8").split(") ").pop().charAt 0 + else + childProcess.execFileSync("ps", ["-o", "stat=", "-p", String pid], {stdio: ["ignore", "pipe", "ignore"]}).toString().trim().charAt 0 + catch + "" track = (ps) -> ps.output = "" @@ -92,8 +139,8 @@ waitForOutput = (ps, predicate) -> readyPids = (ps) -> outputLines(ps) - .filter (line) -> line.indexOf(READY_PREFIX) == 0 - .map (line) -> Number line.slice READY_PREFIX.length + .filter (line) -> line.indexOf(READY_PREFIX) != -1 + .map (line) -> Number line.slice line.indexOf(READY_PREFIX) + READY_PREFIX.length waitForReady = (ps, count) -> waitForOutput(ps, -> readyPids(ps).length >= count).then -> readyPids ps @@ -104,9 +151,15 @@ hasLineEndingWith = (suffix) -> (ps) -> isAlive = (pid) -> try process.kill pid, 0 - true catch error - error.code != "ESRCH" + return error.code != "ESRCH" + WINDOWS or processState(pid) not in [ZOMBIE_STATE, ""] + +processExited = (ps) -> ps.exitCode != null or ps.signalCode != null + +exitedProcess = (ps) -> + new Promise (resolve) -> + if processExited ps then resolve() else ps.on "exit", resolve doneCount = (ps) -> outputLines(ps).filter((line) -> line == DONE_LINE).length @@ -120,9 +173,9 @@ shouldRejectBeforeLaunch = (ps, message) -> afterEach -> for ps in spawned.splice(0) if WINDOWS - childProcess.spawnSync "taskkill", ["/T", "/F", "/PID", String ps.pid] if ps.exitCode == null and ps.signalCode == null - else if isAlive -ps.pid - process.kill -ps.pid, "SIGKILL" + childProcess.spawnSync "taskkill", ["/T", "/F", "/PID", String ps.pid] unless processExited ps + else unless processExited ps + ps.kill "SIGKILL" for directory in directories.splice(0) for name in [TRIGGER_NAME].concat PACKAGE_FILES file = path.join directory, name @@ -134,11 +187,6 @@ describe "parallelshell", -> literalNames = ["literal*", "#hash", "!bang", "two words", "quote\"name", "amp&name", "percent%PATH%"] scriptNames = ["build:js", "build:css", "build:html", "test:js", ".hidden", "nested/build/js"].concat literalNames - project = (scripts) -> - directory = newDirectory() - fs.writeFileSync path.join(directory, "package.json"), JSON.stringify {name: "parallelshell-fixture", version: "1.0.0", scripts} - directory - npmProject = -> scripts = {} for name in scriptNames @@ -232,6 +280,105 @@ describe "parallelshell", -> it "should reject a missing package.json before launching commands", -> shouldRejectBeforeLaunch spawnParallelshellWith({cwd: newDirectory()}, "echo ordinary", "-n", "build"), "--npm" + describe "prefixed output", -> + buildScripts = {"build:js": "echo built", "build:css": "echo styled"} + spawnPrefixed = (args...) -> spawnParallelshellWith {env: colorEnv()}, args... + printBoth = evalProcess "console.log('out'); console.error('err')" + + ["-p", "--prefix"].forEach (flag) -> + it "should label lines with shortened command text with #{flag}", -> + ps = spawnPrefixed flag, "echo ordinary", "echo short" + ps.exited.then (result) -> + result.code.should.equal 0 + outputLines(ps).should.include.members ["echo..nary | ordinary", "echo short | short"] + + ["-l", "--label"].forEach (flag) -> + it "should pad labels, keep stderr on stderr and imply --prefix with #{flag}", -> + ps = spawnPrefixed flag, "api", printBoth, flag, "worker", "echo second" + ps.exited.then (result) -> + result.code.should.equal 0 + outputLines(ps).should.include.members ["api | out", "worker | second"] + ps.errorOutput.split(LINE_BREAK).should.include "api | err" + ps.output.should.not.contain "err" + + it "should label npm scripts with their names", -> + directory = project buildScripts + ps = spawnParallelshellWith {cwd: directory, env: colorEnv()}, "-p", "-n", "build:*", "-l", "custom", "-n", "build:js" + ps.exited.then (result) -> + result.code.should.equal 0 + outputLines(ps).should.include.members ["build:js | built", "build:css | styled", "custom | built"] + + [ + ["FORCE_COLOR=1", {FORCE_COLOR: "1"}, true] + ["FORCE_COLOR=1 over NO_COLOR=1", {FORCE_COLOR: "1", NO_COLOR: "1"}, true] + ["FORCE_COLOR=0", {FORCE_COLOR: "0"}, false] + ["NO_COLOR=1", {NO_COLOR: "1"}, false] + ["a pipe", {}, false] + ].forEach ([description, overrides, colored]) -> + it "should #{if colored then "" else "not "}color labels with #{description}", -> + ps = spawnParallelshellWith {env: colorEnv overrides}, "-l", "a", "echo first", "-l", "b", "echo second" + ps.exited.then -> + lines = outputLines ps + if colored + lines.should.include.members ["\u001b[36ma | \u001b[0mfirst", "\u001b[33mb | \u001b[0msecond"] + else + lines.should.include.members ["a | first", "b | second"] + + it "should finish another command's partial line before interleaving", -> + triggers = (newTrigger() for index in [0..2]) + ps = spawnPrefixed "-l", "a", stagedProcess([triggers[0], "part"], [triggers[2], "ial\n"]), "-l", "b", stagedProcess([triggers[1], "line\n"]) + release triggers[0] + waitForOutput(ps, -> ps.output == "a | part").then -> + release triggers[1] + waitForOutput ps, -> ps.output.endsWith "b | line\n" + .then -> + release triggers[2] + ps.exited + .then (result) -> + result.code.should.equal 0 + ps.output.should.equal "a | part\nb | line\na | ial\n" + + it "should prefix after carriage returns and keep split CRLF line endings", -> + triggers = [newTrigger(), newTrigger()] + ps = spawnPrefixed "-l", "a", stagedProcess([triggers[0], "10%\r20%\r"], [triggers[1], "\ndone\r\n"]) + release triggers[0] + waitForOutput(ps, -> ps.output == "a | 10%\ra | 20%").then -> + release triggers[1] + ps.exited + .then (result) -> + result.code.should.equal 0 + ps.output.should.equal "a | 10%\ra | 20%\r\na | done\r\n" + + it "should relay many commands without listener warnings", -> + ps = spawnPrefixed "-p", ("echo #{index}" for index in [1..MANY_COMMANDS])... + ps.exited.then (result) -> + result.code.should.equal 0 + ps.errorOutput.should.equal "" + outputLines(ps).filter((line) -> line.indexOf(" | ") != -1).length.should.equal MANY_COMMANDS + + it "should relay all of a failing command's output before exiting", -> + failing = evalProcess "for (var line = 0; line < #{FAILURE_OUTPUT_LINES}; line++) console.error('line ' + line); process.exitCode = #{FAILURE_EXIT_CODE}" + ps = spawnPrefixed "-l", "f", failing, waitingProcess() + ps.exited.then (result) -> + result.code.should.equal FAILURE_EXIT_CODE + ps.errorOutput.split(LINE_BREAK).filter((line) -> line.indexOf("f") == 0).length.should.equal FAILURE_OUTPUT_LINES + + onPosix "should stop its commands and exit with the SIGPIPE status when its output closes", -> + ps = spawnPrefixed "-p", "yes", waitingProcess() + waitForOutput(ps, -> ps.output.length > 0).then -> + ps.stdout.destroy() + ps.exited + .then (result) -> + result.code.should.equal SIGPIPE_EXIT_CODE + + [["-l"], ["-l", "-w", "echo ordinary"], ["-l", "a"], ["-l", "a", "-l", "b", "echo ordinary"]].forEach (args) -> + it "should reject #{JSON.stringify args} before launching any command", -> + shouldRejectBeforeLaunch spawnPrefixed(["echo ordinary"].concat(args)...), "--label requires a name followed by a command" + + it "should reject a label for a pattern matching several npm scripts", -> + directory = project buildScripts + shouldRejectBeforeLaunch spawnParallelshellWith({cwd: directory}, "echo ordinary", "-l", "all", "-n", "build:*"), "--label names one command" + it "should stop running commands at the deadline", -> ps = spawnParallelshell "--timeout", TIMEOUT_SECONDS, succeedingProcess, waitingProcess(), waitingProcess() waitForReady(ps, 2).then (pids) -> @@ -413,7 +560,7 @@ describe "parallelshell", -> result.code.should.equal SIGNAL_EXIT_CODE_BASE + signals.SIGINT it "should run a command containing double quotes as written", -> - ps = spawnParallelshell "#{process.execPath} -e \"console.log('#{QUOTED_TEXT}')\"" + ps = spawnParallelshell evalProcess "console.log('#{QUOTED_TEXT}')" ps.exited.then (result) -> result.code.should.equal 0 outputLines(ps)[0].should.equal QUOTED_TEXT @@ -427,6 +574,77 @@ describe "parallelshell", -> ps.errorOutput.should.equal "" result.code.should.equal 0 + onPosix "should run compound commands and stop them when a sibling fails", -> + trigger = newTrigger() + compound = "export #{ENV_NAME}=#{ENV_VALUE} && cd \"#{FIXTURES_DIR}\" && #{printEnvProcess} && #{printCwdProcess} && #{waitingProcess()}" + ps = spawnParallelshell compound, waitingProcess(trigger, FAILURE_EXIT_CODE) + waitForReady(ps, 2).then (pids) -> + release trigger + ps.exited.then (result) -> + result.code.should.equal FAILURE_EXIT_CODE + outputLines(ps).should.include.members [ENV_VALUE, FIXTURES_DIR] + pids.filter(isAlive).should.be.empty + + it "should stop an npm script's command when a sibling fails", -> + trigger = newTrigger() + ps = spawnParallelshellWith {cwd: project({serve: waitingProcess()})}, "-n", "serve", waitingProcess(trigger, FAILURE_EXIT_CODE) + waitForReady(ps, 2).then (pids) -> + release trigger + ps.exited.then (result) -> + result.code.should.equal FAILURE_EXIT_CODE + pids.filter(isAlive).should.be.empty + + [[], ["--prefix"]].forEach (options) -> + onPosix "should stop a background process a command left running when a sibling fails#{if options.length then " with " + options else ""}", -> + trigger = newTrigger() + ps = spawnParallelshell options.concat(["#{waitingProcess(newTrigger(), 0, IGNORED_SIGNAL)} &", waitingProcess(trigger, FAILURE_EXIT_CODE)])... + waitForReady(ps, 2).then (pids) -> + release trigger + ps.exited.then (result) -> + result.code.should.equal FAILURE_EXIT_CODE + waitUntil -> pids.filter(isAlive).length == 0 + + onPosix "should leave a background process running when every command finishes", -> + trigger = newTrigger() + ps = spawnParallelshell "#{waitingProcess(trigger)} &" + waitForReady(ps, 1).then ([pid]) -> + exitedProcess(ps).then -> + ps.exitCode.should.equal 0 + isAlive(pid).should.be.true + release trigger + waitForOutput ps, -> doneCount(ps) == 1 + + onPosix "should stop its commands when it is killed with SIGKILL", -> + ps = spawnParallelshell waitingProcess(), waitingProcess() + waitForReady(ps, 2).then (pids) -> + ps.kill "SIGKILL" + waitUntil -> pids.filter(isAlive).length == 0 + + onPosix "should stop and resume its commands with SIGTSTP and SIGCONT", -> + trigger = newTrigger() + ps = spawnParallelshell waitingProcess(trigger) + waitForReady(ps, 1).then ([pid]) -> + ps.kill "SIGTSTP" + waitUntil(-> processState(pid) == STOPPED_STATE and processState(ps.pid) == STOPPED_STATE).then -> + ps.kill "SIGCONT" + waitUntil -> processState(pid) != STOPPED_STATE and processState(ps.pid) != STOPPED_STATE + .then -> + release trigger + ps.exited + .then (result) -> + result.code.should.equal 0 + + onWindows "should finish with --prefix when a command's background process keeps its output open", -> + backgroundTrigger = newTrigger() + trigger = newTrigger() + ps = spawnParallelshell "--prefix", "start /b \"\" #{waitingProcess(backgroundTrigger)}", waitingProcess(trigger, FAILURE_EXIT_CODE) + waitForReady(ps, 2).then (pids) -> + release trigger + exitedProcess(ps).then -> + ps.exitCode.should.equal FAILURE_EXIT_CODE + release backgroundTrigger + waitUntil -> pids.filter(isAlive).length == 0 + onWindows "should stop its children and exit with the Ctrl+C status on Ctrl+C", -> ps = track childProcess.spawn "powershell", POWERSHELL_ARGS.concat ["-File", CTRL_C_HELPER, process.execPath, PARALLELSHELL_PATH, waitingProcess(), waitingProcess()] waitForReady(ps, 2).then (pids) ->