Fix: AE-1371: poll for jobs while tasks are running

[jhcipar]

Modified from #457

• JobScaler.run_jobs now tracks the active handlers in a set, adds a short wait timeout, and idles with asyncio.sleep(0.1) when there is nothing to do so the loop no longer busy-spins
• Added tests/test_serverless/test_rp_scale.py, a new async test suite that exercises queue draining, concurrency limits, shutdown behavior, and an end-to-end “fetch jobs then process them” workflow using a patched JobScaler.

[Copilot]

Pull request overview

This PR fixes a busy-wait issue in the job processing loop by implementing proper idle handling and concurrent task tracking. The changes ensure the worker efficiently polls for new jobs while existing tasks are running, preventing CPU waste when idle.

Key Changes:

• Modified JobScaler.run_jobs to use a set for tracking tasks, added a timeout to asyncio.wait, and implemented idle sleep to prevent busy-spinning
• Added comprehensive async test suite covering queue draining, concurrency limits, shutdown behavior, and end-to-end job processing workflows

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
runpod/serverless/modules/rp_scale.py	Changed task tracking from list to set, added timeout to asyncio.wait, and implemented idle sleep to eliminate busy-waiting
tests/test_serverless/test_rp_scale.py	New comprehensive test suite covering concurrency, queue management, shutdown behavior, and job processing workflows

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The await asyncio.sleep(0) at line 248 is redundant when the else branch already performs await asyncio.sleep(0.1). This creates unnecessary event loop yields. Consider removing line 248 or restructuring to avoid the double sleep in the idle case.