Router/Extension - Large echo fails / hangs on deployed AWS env

[huntharo]

Motivations

• Big files should not be slower than small files on an /echo test
• Big files work fine on /echo-slow (which reads into memory before sending response bytes)

Acceptance Criteria

• Confirm that response bytes can be written before request is finished
• Add configurable header to enable debug logging - implement handling in Router and Extension
• Add access log to the demo-app
• Add access log to the extension
• Build - Router is not deploying updates when image changes #260
• Add access log to the router
• Add config option to disable periodic metrics printing in router
• Try a Node.js test server behind an ALB and hosted on ECS - See if the issue can be replicated - Yes the issue is able to be replicated without the Router
• Identify source of the hang - It's a problem with the AWS ALB
• Fix the issue - Will report to AWS
• Export NLB URL / ports to the PR and as step outputs
• Add smoke check tests for the NLB ports to GHA
• Add echo tests?

Current Understanding - 2024-01-11

• The problem is reproducible with AWS ALB and ECS Task using Node.js echo request
• The problem is intermittent but happens most of the time (> 60%)
• Will report to AWS

Test Tool Receives only 1.62 MB of Response

Lambda Extension only Received 1.62 MB of Request

Setup

wget -H "Accept: image/jpeg" https://github.com/pwrdrvr/lambda-dispatch/blob/main/src/demo-app/public/silly-test-image.jpg?raw=true -O silly-test-image.jpg

wget https://hey-release.s3.us-east-2.amazonaws.com/hey_linux_amd64
chmod +x hey_linux_amd64

wget https://github.com/hatoo/oha/releases/download/v1.5.0/oha-linux-amd64
chmod +x oha-linux-amd64

Special Test Tools

node src/demo-app/echo-upload-cli.js https://lambdadispatch-pr-257.ghpublic.pwrdrvr.com/echo ~/.cargo/bin/oha

node src/demo-app/echo-cli.js https://lambdadispatch-pr-257.ghpublic.pwrdrvr.com/echo 

✅ Test Case - CloudShell - Small File - Works

• Upload: 91 MB/sec
• Download: 91 MB/sec
• File Size: 165 KB

TOKIO_WORKER_THREADS=1 ./oha-linux-amd64 -c 10 -z 10s -D ./silly-test-image.jpg -T image/jpeg -m POST https://lambdadispatch-pr-257.ghpublic.pwrdrvr.com/echo

./hey_linux_amd64 -m POST -c 20 -z 10s -h2 -T image/jpeg -D ./silly-test-image.jpg https://lambdadispatch-pr-257.ghpublic.pwrdrvr.com/echo

❌ Test Case - CloudShell - Large File - Duplex- Takes 20 Seconds

• Upload:
• Download:
• File Size: 9.0 MB

TOKIO_WORKER_THREADS=1 ./oha-linux-amd64 -c 1 -n 1 -D ./oha-linux-amd64 -T application/octet-stream -m POST https://lambdadispatch-pr-257.ghpublic.pwrdrvr.com/echo

./hey_linux_amd64 -m POST -c 1 -n 1 -h2 -T "application/octet-stream" -D ./hey_linux_amd64 https://lambdadispatch-pr-257.ghpublic.pwrdrvr.com/echo

✅ Test Case - CloudShell - Large File - Simplex - Takes 0.25 Seconds

• Upload: 89.9 MB/sec
• Download: 89.9 MB/sec
• File Size: 9.0 MB

TOKIO_WORKER_THREADS=1 ./oha-linux-amd64 -c 2 -z 10s -D ./oha-linux-amd64 -T application/octet-stream -m POST https://lambdadispatch-pr-2
57.ghpublic.pwrdrvr.com/echo-slow