Keep the end of long output, and pass the command line to _helpless - #246
Merged
Merged
Conversation
The typescript was capped with head -n 1024, keeping the start of the output. Errors are usually at the end (tracebacks, make: *** Error, segfaults), so a program that printed a lot and then failed lost its error before any helper or _helpless saw it. Now keep the first 64 lines (where the command line is echoed and found) plus the last 1024, with a marker for what was omitted. _helpless now also receives the command line as a second argument, so whatever explains the output (in cs50/codespace, the CS50 Duck via cs50.ai) can see what was run. The default _helpless ignores it; output stays the first argument, so the codespace's empty-output check is unaffected.
Nothing exercised _help50 itself: the head/tail cap and the new command-line argument were only checked by hand under a pty. Drive _help50 directly in the image instead, with a fabricated typescript and a fake _helpless, and assert that a command printing 3000 lines then an error yields the command line as the second argument and, as the first, output that starts at the program's first line, carries the omission marker, drops the middle, and ends with the error. Also check that a failed command with no output yields an empty first argument, which is what the codespace's empty-output check depends on. The check fails against the previous help50.sh.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two improvements to what reaches the CS50 Duck when no helper matches, from testing on the canary codespace (cs50/codespace#196).
Keep the end of long output. The typescript was capped with
head -n 1024, i.e. the start of the output. Errors are usually at the end: tracebacks,make: *** Error, segfaults. A program that printed a few thousand lines and then crashed lost its error entirely, so neither the helpers nor_helplessever saw it. Now the first 64 lines are kept (the echoed command line lives there and is still found) plus the last 1024, with a[... N lines omitted ...]marker between them. Short output is unchanged.Pass the command line to
_helpless. cs50.ai's/api/v1/helpwraps whatever it receives in a code block and asks the model to explain it, but until now it only got the output, not what the student typed._helplessnow receives the command line as a second argument (_helpless OUTPUT CMD). The default_helplessignores it; output stays$1, so cs50/codespace's "empty output, nothing to explain" check is unaffected. The companion change that uses it is cs50/codespace#199.The hook contract is now documented above the default definitions in
help50.sh.Testing.
tests/smoke.shnow drives_help50directly in the image with a fabricated typescript and a fake_helpless: a command that prints 3000 lines and then fails yields./slowas the second argument and, as the first, output that starts at the program's first line, contains[... 1914 lines omitted ...], no longer contains the middle, and ends with the error; a failed command with no output yields an empty first argument. The check fails against the previoushelp50.sh. Also verified under a pty on the canary codespace:cat nothereyields$ cat nothere+ its error in the codespace payload.