From 729cf8ab644f092c5530f448a6eb9cc3e6c9f670 Mon Sep 17 00:00:00 2001 From: Louie Lu Date: Wed, 19 Aug 2026 20:30:00 -0500 Subject: [PATCH] Leaderboard: Paix2 #1 (#164), Sqwish update (#177), Divyam #3 (#175), nadir-caliper (#173), BARouter update (#180), AsiaInfo (#181) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Paix2 (#164): new πŸ₯‡ arena 77.63 / acc 79.69% / $0.27 - Sqwish update (#177): πŸ₯ˆ 75.27 -> 76.21 (sqwish-fvcta8 pool) - Divyam (#175): new πŸ₯‰ arena 75.85 / acc 78.59% - nadir-caliper (#173): 74.55, supersedes Nadir-Tumbler (73.44) - BARouter update (#180): 67.09 -> 73.79 - AsiaInfo (#181): arena 65.87 (high-cost 2-model) All audited: 8400 unique rows, real tokens, faithfulness gaps within trusted band. Co-Authored-By: Claude Opus 4.8 --- README.md | 57 ++++++++++++++++++++------------------- leaderboard_manifest.yaml | 42 ++++++++++++++++++++++++----- 2 files changed, 66 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index 0f3ff67c..65af0e14 100644 --- a/README.md +++ b/README.md @@ -37,33 +37,36 @@ For more details, please see our [website](https://routeworks.github.io/leaderbo | Rank | Router | Affiliation | Acc-Cost Arena | Accuracy | Cost/1K Queries | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | |------|--------------------|-----------------------------|--------|----------|---------|-----------------|--------------|----------------|---------|------------| -| πŸ₯‡ | [Cross-Router]() | πŸ‘€ [@JiaHg](https://github.com/JiaHg) | 75.75 | 78.14 | $0.40 | 17.66 | 45.49 | 90.31 | β€” | 67.14 | -| πŸ₯ˆ | [Sqwish Router](https://www.sqwish.ai/) | πŸ‘€ [@namitha-sqwish](https://github.com/namitha-sqwish) | 75.27 | 76.40 | $0.18 | 7.41 | 25.10 | 90.47 | β€” | 100.00 | -| πŸ₯‰ | [vLLM‑SR](https://vllm-semantic-router.com/) [[Code]](https://github.com/vllm-project/semantic-router) [[HF]](https://huggingface.co/llm-semantic-router) | πŸŽ“ vLLM SR Team | 74.86 | 77.18 | $0.42 | 16.81 | 25.10 | 89.37 | β€” | 67.62 | -| 4 | [AgentForge Router]() | πŸ‘€ [@YangY-Z](https://github.com/YangY-Z) | 74.13 | 74.72 | $0.13 | 17.84 | 52.47 | 98.68 | β€” | 40.48 | -| 5 | [Nadir-Tumbler]() | πŸ‘€ [@doramirdor](https://github.com/doramirdor) | 73.44 | 75.34 | $0.37 | β€” | β€” | β€” | β€” | 66.43 | -| 6 | [Weave Router](https://workweave.ai) | πŸŽ“ Weave | 72.82 | 76.32 | $0.94 | β€” | β€” | β€” | β€” | 100.00 | -| 7 | [Nadir Router](https://github.com/NadirRouter/NadirClaw) | πŸŽ“ NadirRouter | 72.29 | 75.01 | $0.68 | β€” | β€” | β€” | β€” | 25.48 | -| 8 | [OrcaRouter‑Adaptive](https://www.orcarouter.ai/) [[Code]](https://github.com/Continuum-AI-Corp/OrcaRouter-Lite) [[Paper]](https://arxiv.org/abs/2605.30736) [[X]](https://x.com/orcarouter) | πŸŽ“ [Continuum AI](https://www.continuum01.ai/) | 72.08 | 75.54 | $1.00 | β€” | β€” | β€” | β€” | 22.62 | -| 9 | [Hybrid Router]() | πŸ‘€ [@mikemao27](https://github.com/mikemao27) | 72.08 | 71.38 | $0.04 | 89.87 | 94.19 | 92.81 | β€” | 96.67 | -| 10 | [R2-Router](https://arxiv.org/abs/2602.02823/) | πŸŽ“ UCF | 71.60 | 71.23 | $0.06 | 24.51 | 48.70 | 99.85 | β€” | 45.71 | -| 11 | [LLM Router](https://github.com/ypollak2/llm-router) [[PyPI]](https://pypi.org/project/llm-routing/) | πŸ‘€ [@ypollak2](https://github.com/ypollak2) | 71.26 | 72.05 | $0.20 | 18.01 | 20.46 | 89.13 | β€” | 30.00 | -| 12 | [chuzom-solo-v32]() | πŸ‘€ [@ypollak2](https://github.com/ypollak2) | 70.61 | 70.59 | $0.10 | β€” | β€” | β€” | β€” | 100.00 | -| 13 | [Azure-Model-Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | πŸ’Ό Microsoft | 70.42 | 72.94 | $0.73 | β€” | β€” | β€” | β€” | 71.43 | -| 14 | [Auto Router]() | πŸ‘€ [@cxf2015](https://github.com/cxf2015) | 70.05 | 70.17 | $0.12 | 37.58 | 40.02 | 86.04 | β€” | 49.52 | -| 15 | [Lynkr]() | πŸ‘€ [@vishalveerareddy123](https://github.com/vishalveerareddy123) | 67.65 | 68.41 | $0.29 | 10.97 | 16.08 | 84.48 | β€” | 92.38 | -| 16 | [BARouter]() | πŸ‘€ [@zulk2002](https://github.com/zulk2002) | 67.09 | 68.80 | $0.63 | 58.56 | 59.03 | 87.91 | β€” | 52.38 | -| 17 | [MIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[Code]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.89 | 66.88 | $0.15 | 3.44 | 19.62 | 78.18 | 27.03 | 61.19 | -| 18 | [NIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[Code]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.12 | 66.34 | $0.21 | 3.83 | 14.04 | 77.88 | 10.42 | 49.29 | -| 19 | [GPT‑5](https://openai.com/index/introducing-gpt-5/) | πŸ’Ό OpenAI | 64.32 | 73.96 | $10.02 | β€” | β€” | β€” | β€” | β€” | -| 20 | [CARROT](https://arxiv.org/abs/2502.03261) [[Code]](https://github.com/somerstep/CARROT) [[HF]](https://huggingface.co/CARROT-LLM-Routing) | πŸŽ“ UMich | 63.87 | 67.21 | $2.06 | 2.68 | 6.77 | 78.63 | 1.50 | 89.05 | -| 21 | [Chayan](https://huggingface.co/adaptive-classifier/chayan) [[HF]](https://huggingface.co/adaptive-classifier/chayan) | πŸŽ“ Adaptive Classifier | 63.83 | 64.89 | $0.56 | 43.03 | 43.75 | 88.74 | β€” | β€” | -| 22 | [RouterBench‑MLP](https://arxiv.org/pdf/2403.12031) [[Code]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Martian | 57.56 | 61.62 | $4.83 | 13.39 | 24.45 | 83.32 | 90.91 | 80.00 | -| 23 | [NotDiamond](https://www.notdiamond.ai/) | πŸ’Ό NotDiamond | 57.29 | 60.83 | $4.10 | 1.55 | 2.14 | 76.81 | β€” | 55.91 | -| 24 | [GraphRouter](https://arxiv.org/abs/2410.03834) [[Code]](https://github.com/ulab-uiuc/GraphRouter) | πŸŽ“ UIUC | 57.22 | 57.00 | $0.34 | 4.73 | 38.33 | 74.25 | 2.70 | 94.29 | -| 25 | [RouterBench‑KNN](https://arxiv.org/pdf/2403.12031) [[Code]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Martian | 55.48 | 58.69 | $4.27 | 13.09 | 25.49 | 78.77 | 1.33 | 83.33 | -| 26 | [RouteLLM](https://arxiv.org/abs/2406.18665) [[Code]](https://github.com/lm-sys/RouteLLM) [[HF]](https://huggingface.co/routellm) | πŸŽ“ Berkeley | 48.07 | 47.04 | $0.27 | 99.72 | 99.63 | 68.76 | 0.40 | 100.00 | -| 27 | [RouterDC](https://arxiv.org/abs/2409.19886) [[Code]](https://github.com/shuhao02/RouterDC) | πŸŽ“ SUSTech | 33.75 | 32.01 | $0.07 | 39.84 | 73.00 | 49.05 | 10.75 | 85.24 | +| πŸ₯‡ | [Paix2]() | πŸ‘€ [@xufan866](https://github.com/xufan866) | 77.63 | 79.69 | $0.27 | 89.67 | 62.92 | 100.00 | β€” | 77.86 | +| πŸ₯ˆ | [Sqwish Router](https://www.sqwish.ai/) | πŸ‘€ [@namitha-sqwish](https://github.com/namitha-sqwish) | 76.21 | 79.76 | $0.70 | 9.04 | 23.49 | 94.07 | β€” | 51.67 | +| πŸ₯‰ | [Divyam]() | πŸ‘€ [@samikd](https://github.com/samikd) | 75.85 | 78.59 | $0.48 | 4.93 | 16.03 | 93.43 | β€” | 98.33 | +| 4 | [Cross-Router]() | πŸ‘€ [@JiaHg](https://github.com/JiaHg) | 75.75 | 78.14 | $0.40 | 17.66 | 45.49 | 90.31 | β€” | 67.14 | +| 5 | [vLLM‑SR](https://vllm-semantic-router.com/) [[Code]](https://github.com/vllm-project/semantic-router) [[HF]](https://huggingface.co/llm-semantic-router) | πŸŽ“ vLLM SR Team | 74.86 | 77.18 | $0.42 | 16.81 | 25.10 | 89.37 | β€” | 67.62 | +| 6 | [nadir-caliper]() | πŸ‘€ [@doramirdor](https://github.com/doramirdor) | 74.55 | 75.84 | $0.22 | β€” | β€” | β€” | β€” | 79.76 | +| 7 | [AgentForge Router]() | πŸ‘€ [@YangY-Z](https://github.com/YangY-Z) | 74.13 | 74.72 | $0.13 | 17.84 | 52.47 | 98.68 | β€” | 40.48 | +| 8 | [BARouter]() | πŸ‘€ [@zulk2002](https://github.com/zulk2002) | 73.79 | 75.72 | $0.36 | 64.41 | 67.13 | 93.84 | β€” | 68.81 | +| 9 | [Weave Router](https://workweave.ai) | πŸŽ“ Weave | 72.82 | 76.32 | $0.94 | β€” | β€” | β€” | β€” | 100.00 | +| 10 | [Nadir Router](https://github.com/NadirRouter/NadirClaw) | πŸŽ“ NadirRouter | 72.29 | 75.01 | $0.68 | β€” | β€” | β€” | β€” | 25.48 | +| 11 | [OrcaRouter‑Adaptive](https://www.orcarouter.ai/) [[Code]](https://github.com/Continuum-AI-Corp/OrcaRouter-Lite) [[Paper]](https://arxiv.org/abs/2605.30736) [[X]](https://x.com/orcarouter) | πŸŽ“ [Continuum AI](https://www.continuum01.ai/) | 72.08 | 75.54 | $1.00 | β€” | β€” | β€” | β€” | 22.62 | +| 12 | [Hybrid Router]() | πŸ‘€ [@mikemao27](https://github.com/mikemao27) | 72.08 | 71.38 | $0.04 | 89.87 | 94.19 | 92.81 | β€” | 96.67 | +| 13 | [R2-Router](https://arxiv.org/abs/2602.02823/) | πŸŽ“ UCF | 71.60 | 71.23 | $0.06 | 24.51 | 48.70 | 99.85 | β€” | 45.71 | +| 14 | [LLM Router](https://github.com/ypollak2/llm-router) [[PyPI]](https://pypi.org/project/llm-routing/) | πŸ‘€ [@ypollak2](https://github.com/ypollak2) | 71.26 | 72.05 | $0.20 | 18.01 | 20.46 | 89.13 | β€” | 30.00 | +| 15 | [chuzom-solo-v32]() | πŸ‘€ [@ypollak2](https://github.com/ypollak2) | 70.61 | 70.59 | $0.10 | β€” | β€” | β€” | β€” | 100.00 | +| 16 | [Azure-Model-Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | πŸ’Ό Microsoft | 70.42 | 72.94 | $0.73 | β€” | β€” | β€” | β€” | 71.43 | +| 17 | [Auto Router]() | πŸ‘€ [@cxf2015](https://github.com/cxf2015) | 70.05 | 70.17 | $0.12 | 37.58 | 40.02 | 86.04 | β€” | 49.52 | +| 18 | [Lynkr]() | πŸ‘€ [@vishalveerareddy123](https://github.com/vishalveerareddy123) | 67.65 | 68.41 | $0.29 | 10.97 | 16.08 | 84.48 | β€” | 92.38 | +| 19 | [MIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[Code]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.89 | 66.88 | $0.15 | 3.44 | 19.62 | 78.18 | 27.03 | 61.19 | +| 20 | [NIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[Code]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.12 | 66.34 | $0.21 | 3.83 | 14.04 | 77.88 | 10.42 | 49.29 | +| 21 | [AsiaInfo-Router]() | πŸ‘€ [@Uncle-LL](https://github.com/Uncle-LL) | 65.87 | 75.20 | $8.54 | β€” | β€” | β€” | β€” | 69.52 | +| 22 | [GPT‑5](https://openai.com/index/introducing-gpt-5/) | πŸ’Ό OpenAI | 64.32 | 73.96 | $10.02 | β€” | β€” | β€” | β€” | β€” | +| 23 | [CARROT](https://arxiv.org/abs/2502.03261) [[Code]](https://github.com/somerstep/CARROT) [[HF]](https://huggingface.co/CARROT-LLM-Routing) | πŸŽ“ UMich | 63.87 | 67.21 | $2.06 | 2.68 | 6.77 | 78.63 | 1.50 | 89.05 | +| 24 | [Chayan](https://huggingface.co/adaptive-classifier/chayan) [[HF]](https://huggingface.co/adaptive-classifier/chayan) | πŸŽ“ Adaptive Classifier | 63.83 | 64.89 | $0.56 | 43.03 | 43.75 | 88.74 | β€” | β€” | +| 25 | [RouterBench‑MLP](https://arxiv.org/pdf/2403.12031) [[Code]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Martian | 57.56 | 61.62 | $4.83 | 13.39 | 24.45 | 83.32 | 90.91 | 80.00 | +| 26 | [NotDiamond](https://www.notdiamond.ai/) | πŸ’Ό NotDiamond | 57.29 | 60.83 | $4.10 | 1.55 | 2.14 | 76.81 | β€” | 55.91 | +| 27 | [GraphRouter](https://arxiv.org/abs/2410.03834) [[Code]](https://github.com/ulab-uiuc/GraphRouter) | πŸŽ“ UIUC | 57.22 | 57.00 | $0.34 | 4.73 | 38.33 | 74.25 | 2.70 | 94.29 | +| 28 | [RouterBench‑KNN](https://arxiv.org/pdf/2403.12031) [[Code]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Martian | 55.48 | 58.69 | $4.27 | 13.09 | 25.49 | 78.77 | 1.33 | 83.33 | +| 29 | [RouteLLM](https://arxiv.org/abs/2406.18665) [[Code]](https://github.com/lm-sys/RouteLLM) [[HF]](https://huggingface.co/routellm) | πŸŽ“ Berkeley | 48.07 | 47.04 | $0.27 | 99.72 | 99.63 | 68.76 | 0.40 | 100.00 | +| 30 | [RouterDC](https://arxiv.org/abs/2409.19886) [[Code]](https://github.com/shuhao02/RouterDC) | πŸŽ“ SUSTech | 33.75 | 32.01 | $0.07 | 39.84 | 73.00 | 49.05 | 10.75 | 85.24 | πŸŽ“ Open-sourceβ€ƒβ€ƒπŸ’Ό Closed-source  diff --git a/leaderboard_manifest.yaml b/leaderboard_manifest.yaml index 6f15d8c7..7e1bfe52 100644 --- a/leaderboard_manifest.yaml +++ b/leaderboard_manifest.yaml @@ -23,7 +23,7 @@ routers: # --- Routers submitted to RouterArena (derived data regenerated) --- - readme_name: "Sqwish Router" website_name: "Sqwish Router" - prediction: "sqwish-router" + prediction: "sqwish-fvcta8" category_key: "sqwish-router" flip_key: "sqwish-router" website: @@ -126,11 +126,14 @@ routers: affiliation: "@ypollak2" github_url: "https://github.com/ypollak2" - - readme_name: "Nadir-Tumbler" - website_name: "Nadir-Tumbler" - prediction: "nadir-tumbler" - category_key: "nadir-tumbler" - flip_key: "nadir-tumbler" + # nadir-caliper (#173) supersedes nadir-tumbler (#159); regenerated from the + # true-grok-4.3-priced prediction file. Website's old "Nadir-Tumbler" row is + # replaced on the next sync. + - readme_name: "nadir-caliper" + website_name: "nadir-caliper" + prediction: "nadir-caliper" + category_key: "nadir-caliper" + flip_key: "nadir-caliper" website: affiliation: "@doramirdor" github_url: "https://github.com/doramirdor" @@ -162,6 +165,33 @@ routers: affiliation: "@vishalveerareddy123" github_url: "https://github.com/vishalveerareddy123" + - readme_name: "Paix2" + website_name: "Paix2" + prediction: "Paix2-router" + category_key: "paix2" + flip_key: "paix2" + website: + affiliation: "@xufan866" + github_url: "https://github.com/xufan866" + + - readme_name: "Divyam" + website_name: "Divyam" + prediction: "divyam-router" + category_key: "divyam" + flip_key: "divyam" + website: + affiliation: "@samikd" + github_url: "https://github.com/samikd" + + - readme_name: "AsiaInfo-Router" + website_name: "AsiaInfo-Router" + prediction: "AsiaInfo-Router" + category_key: "asiainfo" + flip_key: "asiainfo" + website: + affiliation: "@Uncle-LL" + github_url: "https://github.com/Uncle-LL" + # --- Externally-evaluated baselines (headline from README; derived data on # the website is preserved as-is) --- - readme_name: "MIRT-BERT"