[UX] Show No quota and No balance when provisioning fails

[peterschmidt85]

Offers correctly show No quota / No balance, but fleet/job provisioning reports No offers / No capacity. In dstack Factory (including Sky), quota or billing restrictions therefore look like provider capacity problems.

Steps to reproduce

1. On an affected Factory/Sky server, use a project with a dstack backend, positive owner balance, no reusable instances, and available Nebius offers matching cpu: 8.. and gpu: H100.

2. Set the project's CPU quota to 4. Matching offers should now show no_quota. Save fleet.yml and run dstack apply -f fleet.yml:

   type: fleet
   name: quota-repro
   nodes: 0..1
   backends: [nebius]
   resources:
     cpu: 8..
     gpu: H100

3. Save task.yml and run dstack apply -f task.yml, confirming submission:

   type: task
   name: quota-task
   fleets: [quota-repro]
   retry: false
   resources:
     cpu: 8..
     gpu: H100
   commands:
     - echo hello

4. Inspect the failed job. To reproduce fleet-instance provisioning, apply a second fleet with the same resources, a different name, and nodes: 1; inspect its terminated instance.

5. Repeat with sufficient quota and zero owner balance to exercise no_balance. Use another supported GPU if H100 offers are unavailable.

Actual behaviour

Object	Reported failure
Offer	No quota / No balance
Fleet instance	No offers
Job/run	No capacity

Expected behaviour

Report No quota / No balance when no usable offers remain and all matching rejected offers share that reason. Try available alternatives; keep the generic failure for mixed reasons. Preserve the existing lifecycle statuses and retry policy.

For example:

No quota
All matching offers are blocked by quota

Notes

Root cause: create_cloud_instance() and _provision_new_capacity() request offers with exclude_not_available=True. get_backend_offers() drops NO_QUOTA / NO_BALANCE offers before those callers choose a termination reason, leaving their generic no-offers/no-capacity paths.

Fix:

1. Add an optional on_unavailable_offer callback to get_backend_offers(). Before discarding an offer, get_offers_by_requirements() records its availability in UnavailableOffers. Reuse _OfferFilter and block-divisibility checks so only matching offers contribute; keep only distinct reasons, without copying or retaining offers.
2. In both provisioning functions, select a specific failure only when there are no available offers and the recorded reasons are exactly {NO_QUOTA} or {NO_BALANCE}. Fleet instances use the new InstanceTerminationReason.NO_QUOTA or existing NO_BALANCE. Jobs return new FAILED_TO_START_DUE_TO_NO_QUOTA / FAILED_TO_START_DUE_TO_NO_BALANCE through _FailedNewCapacityProvisioning.reason instead of hardcoding no capacity.
3. Map the new job reasons to JobStatus.FAILED, RetryEvent.NO_CAPACITY, and the no quota / no balance status messages. Include them in the run's retrying display and the existing superseded-submission cleanup DELETE predicate.
4. Include quota/balance failures in cluster-master failure handling so pending siblings terminate with MASTER_FAILED. Omit the CLI runner-log hint for these failures because no runner started.

Performance: this implementation took about 17% longer to process the same jobs in a local stress test: 2.14 s before vs. 2.51 s after (160 jobs, 10,000 blocked offers per job, 80 concurrent workers; provider responses simulated). The extra CPU work can delay other server work under load. There are no additional DB queries or cloud API calls, and offer classification holds no DB connection, but this does not mean zero server latency impact.

Deployment: during a rolling update, old and new servers share the DB. If a new server saves one of the new job failure reasons, an old server fails when reading that job because it does not recognize the value. Safe rolling updates need two releases: first teach every server to read and handle these reasons without saving them; then start saving them. This draft must be split into those two releases before deployment.

[r4victor]

Deployment: during a rolling update, old and new servers share the DB. If a new server saves one of the new job failure reasons, an old server fails when reading that job because it does not recognize the value. Safe rolling updates need two releases: first teach every server to read and handle these reasons without saving them; then start saving them. This draft must be split into those two releases before deployment.

Is this addressed?