Fix broker stuck in SYNCHRONIZING on DB error during rollback

[kathap]

When a service broker update job fails, a database connection failure during state rollback could leave the broker permanently stuck in SYNCHRONIZING state with a FAILED job. This PR adds comprehensive error handling and a recovery hook to ensure brokers are reverted to their previous state even when database outages occur during job failure.

Changes:

a) app/jobs/v3/services/update_broker_job.rb:

• Add recover_from_failure hook to revert broker state when job transitions to FAILED
• Refactor rescue block to use rollback_broker_state and destroy_update_request helper methods with graceful error handling
• Add conditional WHERE clause (state: SYNCHRONIZING) to prevent overwriting newer broker states

b) app/jobs/pollable_job_wrapper.rb:

• Invoke recover_from_failure hook in failure method using Enqueuer.unwrap_job
• Log recovery failures without raising to prevent cascading errors

c) spec/unit/jobs/v3/services/update_broker_job_spec.rb:

• Add test for database disconnect during state rollback
• Add comprehensive tests for recover_from_failure method covering normal recovery, idempotency, state protection, and error handling

d) spec/unit/jobs/pollable_job_wrapper_spec.rb:

• Add integration test verifying recovery hook is called on job failure

• A short explanation of the proposed change:

This PR adds multi-layered error handling for broker state management during job failures:

1. Immediate rollback: Best-effort state revert in the job's rescue block (extracted to rollback_broker_state helper)
2. Failure hook: recover_from_failure method called when job exhausts retries and transitions to FAILED
3. Conditional updates: WHERE clause ensures only SYNCHRONIZING brokers are reverted, protecting against overwriting newer states

• An explanation of the use cases your change solves

This change solves a critical production issue where service brokers become permanently stuck in SYNCHRONIZING state. This occurs when:

1. A service broker update job is executing and encounters an error (e.g., invalid catalog, validation failure)
2. The job's error handling attempts to revert the broker state back to its previous state (e.g., AVAILABLE)
3. A database connection failure occurs during this state revert operation
4. The job exhausts retries while the database remains unavailable
5. Without this fix: The broker remains stuck in SYNCHRONIZING state with a FAILED job, requiring manual intervention
6. With this fix: When the job transitions to FAILED, the recover_from_failure hook is invoked by PollableJobWrapper, which attempts to revert the broker state. The conditional WHERE
   clause prevents overwriting any newer state that may have been set, ensuring safe recovery.

• I have reviewed the contributing guide

• I have viewed, signed, and submitted the Contributor License Agreement

• I have made this pull request to the main branch

• I have run all the unit tests using bundle exec rake

• I have run CF Acceptance Tests

[johha]

LGTM