Add spelling benchmark for character-level evaluation

[zherendong]

Summary

Adds a synthetic benchmark to evaluate character-level understanding in language models, specifically designed to test the effectiveness of spelling bee embeddings.

Task Types

Task	Format	Example Output
Count	The number of times the letter A occurs in banana is	3
Index	Q: What is the third letter of the word 'banana'? A:	n
Reverse	cat reversed is	tac

Results

Evaluated on 816m parameter models (764m non-embedding):

Task	Baseline	+Spelling Bee	Delta
Count	12.6%	27.4%	+14.8%
Index	10.3%	16.2%	+5.9%
Reverse	0.0%	0.0%	+0.0%
Overall	11.2%	21.4%	+10.2%

Key findings:

• Spelling bee embeddings nearly double overall accuracy on spelling tasks
• Count task shows largest improvement (+14.8%), likely because counting requires aggregating across all character positions
• Index task improves +5.9%, demonstrating position-encoded character access
• Reverse task: 0% for both models (outputs forward spelling instead of reversed)

Implementation

Dataset (n=5000)

• Distribution: 2,450 count / 2,450 index / 100 reverse
• Word sources: 50/50 from Google 10k common words and 238k English word list
• Filtering: 4-10 character words, palindromes excluded from reverse task
• Tokenization metadata: num_tokens, is_single_token for stratified analysis

Evaluation

• Few-shot prompting with 3 examples per task type
• Integration with lm-evaluation-harness via spelling_bee.yaml
• Generation-based evaluation with exact match (case-insensitive)

Files

File	Description
spelling_benchmark/generate_data.py	Dataset generation with task synthesis
spelling_benchmark/spelling_bee.yaml	lm-eval harness task configuration
spelling_benchmark/spelling_bee.jsonl	Generated benchmark (5000 samples)
spelling_benchmark/analyze_results.py	Post-evaluation analysis with stratified metrics
spelling_benchmark/README.md	Documentation
eval_main.py	Added --max_samples_log parameter for full sample logging

Usage

from spelling_benchmark.generate_data import generate_dataset

generate_dataset(
    output_path='spelling_benchmark/spelling_bee.jsonl',
    num_samples=5000,
    seed=42,
    reverse_samples=100
)

python eval_main.py \
  --checkpoint_paths checkpoints/baseline.pt checkpoints/spelling_bee.pt \
  --tasks spelling_bee \
  --max_samples_log 0

Test plan

• Unit tests pass (pytest spelling_benchmark/spelling_benchmark_test.py)
• Evaluation runs successfully on baseline and spelling bee models
• Results are reproducible (exact match on repeated runs with same seed)