Add support for Swarming preprocess queue and task scheduling

[IvanBM18]

Overview

This change adds support for scheduling tasks to the new Swarming backend. Because Swarming uses a different execution model and to be able to later account for backpressure it requires its own separate preprocess queue and a much lower default target size (5) to prevent unbounded task queuing.

By refactoring the cron scheduling logic, we can now simultaneously feed both the Swarming and Batch environments at their respective ideal rates.

Changes

• Add a distinct SWARMING_PREPROCESS_TARGET_SIZE_DEFAULT set to 5 to support the Swarming backend's task capacity needs.
• Define Swarming-specific queue mappings (SWARMING_QUEUES).
• Refactor BaseFuzzTaskScheduler into an abstract base class with a generic _schedule_fuzz_tasks method to support multiple backends.
• Update ChromeFuzzTaskScheduler to independently schedule Swarming tasks (_schedule_swarming_fuzz_tasks) alongside standard Batch tasks.
• Update schedule_fuzz_test.py to match the updated scheduler class signatures.

TODO

• src/clusterfuzz/_internal/base/feature_flags.py: Update this value based off dev & stage metrics and tests.

[IvanBM18]

The Swarming pool has a hard limit of 25 (LINUX) bots running 1 task each.

• At average 1 hour per fuzzing/swarming task, those 25 bots can finish 4 to 5 tasks every 10 minutes (the interval the cron job runs)
• Because the 2,000(in prod) preprocess tworkers instantly process the preprocess queue, the target size acts as an injection rate more than a buffer.

So, Injecting 5 tasks every 10 minutes matches the expected Swarming rate, preventing an infinitely growing backlog of stale tasks. This is still the default value, the real value is managed trough a feature flag, we will later tweak this feature flag based on metrics & how swarming handled this workload, so that we have a more acqurate value

[fernandofloresg]

lgtm just had one question

[fernandofloresg]

is this still true? if this is going to master then is going to stage and prod ?

[IvanBM18]

Yes this is still true, right now no swarming related code executes outside of dev, we have a ton of featureFlags in place for this reason. So when this changes reach stage/prod they are going to be safe.