Include Bloom Taxonomy & Dewey Labels into public dataset?

[xXmrVictorXx]

Greetings, dear developers of RouterArena.
I have read your paper, and discovered that you constructed the dataset using Dewey Decimal & Bloom Taxonomy to determine the domains & difficulties of the queries. However, in the public dataset on HuggingFace, there are no columns about the Dewey or Bloom classification.
I wonder if you can add these information to the public dataset? Thank you for your contribution.

[jiarong0907]

Hi @xXmrVictorXx, thanks for your interest in using our dataset.

So far, the dataset has major Dewey Decimal Classification label for each query. I wonder whether we could provide the subdomain label as well? @yl231.

For the Bloom Taxonomy, we could potentially provide the label as well. However, we found that using the LLM-as-judges generated Bloom Taxonomy to determine difficulties is not very reliable.

We now use empricial results to determinie difficulty levels. For each query, we count how many of the 42 models (used by existing routers) answer it correctly and group queries into three bands---hard ($\leq$4/42), medium (5–19/42), and easy ($\geq$20/42).

The current dataset's difficulty labels have been updated with the empricial results. We will update the arXiv paper very soon.

[yl231]

Hi @xXmrVictorXx and @jiarong0907,

Our updated paper was submitted to ArXiv today, and it should be available within a few days.

The current HF dataset contains the Category and Domain columns (corresponding to the Dewey Decimal Classification you mentioned). The difficulty is now empirically calculated depending on the number of LLMs that answer that query correctly, as @jiarong0907 mentioned.

For the Bloom Taxonomy, we found it less reliable than empirical results for difficulty labels. So we have switched to empirical results as the difficulty label, and we used Bloom's taxonomy solely to ensure comprehensive coverage of cognitive skills required in the dataset.

[jiarong0907]

Oh, @yl231 is right, It has the category and domain columns.