[WIP] feat: support LAION large TopK benchmarks

[jamesgao-jpg]

Related to #826.

Summary

• Parameterize Performance768D100M to select LAION query and ground-truth artifacts for K through 1,000,000.
• Keep wide ground truth path-based and read one Arrow/NumPy row at a time instead of materializing Python integer lists.
• Replace quadratic recall/NDCG work with O(K) scoring and report recall_at across supported depth cutoffs.
• Add serial/concurrent p50 latency and propagate K-aware results through REST and frontend consumers.
• Add first-class response payload profiles to every vector PerformanceCase, with separate IDs-only and vector runs.
• Expose both return scenarios in the frontend for Milvus and Zilliz Cloud, reject unsupported backend/profile combinations before dataset preparation or loading, and preserve payload identity in frontend, REST, and leaderboard results.

Metrics

• Each return scenario reports independent P99 latency, QPS, and recall.
• qps remains the highest successful QPS observed among the configured concurrency levels.
• VDBBench does not attempt to discover a backend concurrency limit.

Compatibility

• Existing LAION runs with K up to 1,000 continue using the original 1,000-query artifacts.
• K from 1,001 through 100,000 uses the hosted 200-query top-100K ground truth.
• K from 100,001 through 1,000,000 uses the hosted 200-query top-1M ground truth.
• Filtered LAION runs above K=1,000 and LAION K above 1,000,000 fail before database initialization.
• IDs-only remains the default payload profile, and legacy custom_case.payload_profile input remains supported.
• Existing result files load with defaults for the new K, latency, recall, and payload fields.

Verification

• make lint
• make unittest
• 108 focused dataset, runner, CLI, backend-translation, result-compatibility, REST, and frontend tests
• IDs-only and vector CLI examples validated with --dry-run
• Impact-map rescan and validation passed
• Exact DCO sign-off verified for every outgoing commit

Scope Notes

• Backend-specific large-TopK collection setup is not included. The target database or pre-created collection must already permit the requested K; VDBBench forwards K unchanged.
• No concurrency-limit discovery is included.
• Milvus request translation is covered by a mocked client test; no live Milvus or Zilliz Cloud benchmark probe was run in this development environment.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jamesgao-jpg
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[frankleaf]

#836

Please take a look at this PR to see if it's needed.