Skip to content

Distinguish proxied tool exits from lstk errors in command telemetry - #499

Draft
joe4dev wants to merge 2 commits into
mainfrom
devx-1004-distinguish-proxied-tool-exits-from-lstk-errors-in-command
Draft

Distinguish proxied tool exits from lstk errors in command telemetry#499
joe4dev wants to merge 2 commits into
mainfrom
devx-1004-distinguish-proxied-tool-exits-from-lstk-errors-in-command

Conversation

@joe4dev

@joe4dev joe4dev commented Sep 9, 2026

Copy link
Copy Markdown
Member

Motivation

lstk aws fails for two very different reasons: an lstk preflight error, or the AWS CLI itself rejecting the user's command. Telemetry recorded both identically, so the error dashboard ranks users' own CLI mistakes as lstk errors — that is how "exit status 252" became the no. 1 lstk error (DEVX-1003).

Solution

lstk_command events gain result.proxy_error, true when the failure came from a wrapped tool the user asked for. It describes the error's origin, not whether a tool was proxied — a successful lstk aws s3 ls is false.

The discriminator is not an *exec.ExitError in the chain, as the ticket proposed: lstk shells out for its own purposes too, and those exits are lstk's failures. Instead each proxy exec site declares ownership via proc.MarkUserToolExit. Forgetting it on a future proxy only loses a data point; marking one of lstk's own execs would hide an lstk bug behind the user's name.

That forced one split: azurecli.Exec served both the lstk az passthrough and lstk's own setup azure/interception calls. It is now Exec (the user's, marks) and Run (lstk's, does not) over a shared execAz.

proxy_error is deliberately not omitempty, so absence means only "emitted before this field existed".

The user's tool failed → proxy_error: true

Command exit_code Why
lstk aws s3 lss 252 AWS CLI usage error — not lstk's fault
lstk aws s3 ls against a service error the CLI's Emulator rejected it, lstk only forwarded
lstk terraform apply on a bad config terraform's terraform rejected the user's HCL
lstk <extension> … exiting non-zero the tool's Extension's own outcome

The exit code is always the wrapped tool's own, unchanged.

lstk failed → proxy_error: false

Command exit_code Why
lstk aws s3 ls, Docker down 1 runtime not healthy — never reached the tool
lstk aws s3 ls, emulator not running 1 lstk preflight
lstk aws s3 ls, success 0 No failure to attribute

lstk shells out, but the failure is still lstk's → proxy_error: false

These are the cases the rejected *exec.ExitError discriminator would have got wrong. The user never typed these invocations and cannot fix them by changing their command line.

Command Shells out to Resulting error_msg
lstk setup azure az cloud list could not list Azure clouds: az […]: exit status 1
lstk az stop-interception az cloud show could not determine the active Azure cloud: az […]: exit status 1
lstk update (Homebrew install) brew upgrade update failed: exit status 1
lstk terraform init (S3 backend) aws s3api create-bucket creating state bucket …: exit status N

Note the contrast: lstk aws with the AWS CLI exiting 252 gives error='exit status 252' and proxy_error: true, while lstk update with brew exiting 1 gives error='update failed: exit status 1' and proxy_error: false. Both carry an *exec.ExitError and both read "exit status N" — only the call site knows which is the user's.

For the analytics side

  • The field is proxy_error, not proxied as the ticket specifies. proxied reads as "this invocation was proxied", which is false for a successful lstk aws s3 ls that the field would still mark false. Renaming needs a nod from whoever builds the dashboard panel.
  • A signal-terminated tool (Ctrl-C on lstk terraform apply, CI timeout) records proxy_error: true, exit_code: -1. Not an lstk error, but not a usage error either — exclude exit_code == -1.
  • The error-rate denominator (proxy invocations including successes) is not derivable from proxy_error alone, since the field is failure-only per the ticket. It still needs the proxy command-name list.

Also

The integration env inherited an ambient LOCALSTACK_DISABLE_EVENTS=1, silently disabling every telemetry assertion. 43 integration tests that fail locally for developers who export it now pass. Separate commit.

Docs

Nothing user-facing. No command, flag, output, env var, or documented behavior changes — this adds a field to the internal lstk_command analytics payload, which no file under docs/ describes. The consumer is the analytics dashboard, not the CLI user.

Review

Human review advised — it changes what the analytics error ranking means, and the azurecli split touches lstk setup azure and interception, whose integration tests skip without a working Azure CLI.

Related

Resolves DEVX-1004

@joe4dev joe4dev added semver: patch docs: skip Pull request does not require documentation changes labels Sep 9, 2026
@joe4dev
joe4dev force-pushed the devx-1004-distinguish-proxied-tool-exits-from-lstk-errors-in-command branch 2 times, most recently from 483514f to e2f5cec Compare September 10, 2026 12:48
Co-Authored-By: Claude <noreply@anthropic.com>
@joe4dev
joe4dev force-pushed the devx-1004-distinguish-proxied-tool-exits-from-lstk-errors-in-command branch from e2f5cec to 8a19bc0 Compare September 10, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs: skip Pull request does not require documentation changes semver: patch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant