Labeling

Labeling is an agent-first CLI for reproducible LLM-as-rater data labeling workflows. It manages datasets that need structured labels, gold-standard comparison, rule baselines, multi-rater aggregation, metrics, exports, and reports. The agent helps the user turn a vague labeling goal into a labeling.yaml spec, generate EDSL jobs, run low-cost samples, inspect quality, record full runs, aggregate label sources, and export final labels with an audit trail.

Install

Install from GitHub:

pip install "git+https://github.com/expectedparrot/labeling.git"

For local development:

git clone https://github.com/expectedparrot/labeling.git
cd labeling
python -m pip install -e ".[dev]"
pytest -q

The CLI is available as either labeling or python -m labeling.

Read the worked HTML tutorial for an end-to-end labeling project and the explicit Jobs.ep → ep run → Results.ep handoff.

Give this to your coding agent

Copy this block into Codex or another coding agent from the repository that contains the dataset you want to label:

Help me build and run an auditable LLM-assisted data-labeling study with
Labeling and EDSL.

Install the current Labeling repository and its dependencies:

python -m pip install --upgrade \
  "research-labeling @ git+https://github.com/expectedparrot/labeling.git"

Verify that `labeling --help` and `ep --help` both work. Then run
`labeling docs show getting-started` and treat the bundled guidance as the
source of truth.

Help me identify the input CSV, stable row-ID column, prompt-visible columns,
label questions, answer options, ambiguity policy, model configuration, and any
available human or gold labels. Prevent target leakage and preserve raw source
columns. Initialize the project, edit labeling.yaml, inspect the dataset, and
validate the spec.

Use Labeling to build portable EDSL Jobs artifacts. Labeling must never execute
model inference itself: `ep` is exclusively responsible for running Jobs.ep and
writing Results.ep. Inspect each generated job and show me its model, scenario
count, question count, output path, and cost risk before execution.

Always start with a small sample. Do not run any `ep run` command that may incur
model cost until I explicitly approve it. After the sample run, record the
Results.ep artifact with Labeling, preview representative labels and edge cases,
and revise the rubric if needed. Obtain separate approval before the full run.

After the full run, preserve the audit trail under .labeling/, evaluate against
gold labels or inter-rater disagreement when available, export the final labeled
CSV, and generate a report. Run `labeling status` after each completed phase and
follow its returned next actions. Never edit .labeling/ records by hand.

Start here

labeling docs show getting-started
labeling init <project-dir> --csv <dataset.csv> --id-column <id-column>
labeling status

When to use this

• The user has rows of data and needs structured labels from one or more LLM raters.
• The labeling task can be expressed as yes/no, multiple choice, list, free text, or numeric questions.
• The user needs sampling, preview, gold labels, metrics, aggregation, or reproducible exports.
• The agent should stage the job with sample approval before running at scale.

When this is a stretch (and how to adapt)

• The categories are not clear yet. Use bewley on a small sample to develop a codebook, then encode it in labeling.yaml.
• The user has no dataset, only documents or transcripts. Build a row-level dataset first, or use bewley directly for qualitative coding.
• The task requires expert judgment. Use labeling to organize model labels, but include gold labels and report model-vs-expert metrics.
• The model output is open-ended. Use free-text questions for exploration, then convert stable categories into multiple-choice questions for scalable labeling.
• The user wants a calibrated persona population. Use manning after labels define target distributions.

Decision rule for the calling agent

Before dispatching to labeling, confirm:

1. There is a tabular dataset or a clear plan to create one row per item.
2. Each row can be labeled by one or more explicit questions.
3. The user needs reproducibility, sampling, metrics, aggregation, or export.
4. The agent can review a sample before approving a full run.

If yes to the first three, labeling is the right method.

Inputs and elicitation

Dataset

What it is: the CSV or row-oriented data file to label, with stable row IDs and columns used by the rater prompt.

How the agent elicits this:

• Ask where the dataset lives, what each row represents, and which column uniquely identifies rows.
• Ask which columns the model should see and which should be hidden to prevent leakage.
• Ask whether rows need cleaning, deduplication, or filtering before labeling.

Default to suggest: preserve a stable ID column, include only task-relevant text/context columns, and keep raw source columns unchanged.

Fallback: if no ID exists, create a deterministic row ID before initializing the labeling project.

Labeling questions and rubric

What it is: the questions the LLM rater answers for each row, including type, options, instructions, and examples.

How the agent elicits this:

• Ask what decision each label will support.
• Ask whether the label is binary, categorical, multi-label, numeric, or free text.
• Ask for inclusion/exclusion rules and borderline examples.
• Keep each question focused; split multi-part judgments into separate questions.

Default to suggest: start with one primary label question and one optional rationale/free-text field for sample review.

Fallback: if the rubric is vague, run a small exploratory sample and revise the spec before generating a full job.

Rater configuration

What it is: model(s), prompt style, number of raters or label sources, and any keyword-rule baselines.

How the agent elicits this:

• Ask whether the user wants one model, multiple models, or comparison against human/gold labels.
• Ask whether cost, latency, or accuracy is the binding constraint.
• Suggest a cheap sample run before full-scale labeling.

Default to suggest: one strong model for the first sample; add a second model only when disagreement/robustness matters.

Fallback: use keyword rules as a transparent baseline when model calls are expensive or the label is simple.

Quality checks and aggregation

What it is: sample review, gold labels, metrics, multi-rater aggregation, and export rules.

How the agent elicits this:

• Ask whether gold labels exist and which rows they cover.
• Ask what metric matters: accuracy, precision, recall, F1, or qualitative acceptability.
• Ask how to resolve multiple label sources: majority vote, authoritative source, or manual review.

Default to suggest: sample 20-50 rows, preview labels, revise the rubric, then run full labeling and export.

Fallback: if no gold exists, report inter-rater agreement or disagreement patterns rather than accuracy.

Outputs

labeling produces:

• labeling.yaml — project spec with dataset, questions, models, and output settings.
• .labeling/ — append-only audit state for samples, plans, runs, imports, gold labels, rules, aggregates, metrics, exports, and reports.
• Portable EDSL Jobs.ep artifacts under an edsl_jobs/ directory.
• Sample manifests and run plans for staged approval.
• Imported label results, previews, gold comparisons, rule baselines, aggregate labels, metrics, final exports, and report artifacts.

Workflow

Canonical sequence:

1. labeling init — create project state and baseline spec.
2. Edit labeling.yaml with dataset path, ID column, columns used, questions, and model(s).
3. labeling inspect — profile the dataset and spec context.
4. labeling validate — catch missing files, bad question specs, or invalid model settings.
5. labeling generate — create the portable full-dataset Jobs.ep artifact.
6. labeling sample — select a low-cost sample.
7. labeling plan sample --latest — get the exact command and run plan for the latest sample.
8. Run the returned ep run ... command outside Labeling when approved.
9. labeling record run --kind sample --results data/results.sample.ep — register and normalize the EDSL results as a first-class label source.
10. labeling preview — review sample quality with the user.
11. Approve accepted sample labels with labeling approve sample; revise the spec instead when needed.
12. labeling plan full --reuse-approved-sample — build Jobs only for rows not already approved.
13. Optional: labeling gold, labeling rule, labeling metrics, and labeling aggregate.
14. labeling export and labeling report context — create final data and a bounded handoff for the report agent.

Run labeling next at any point to recover source coverage, readiness, blockers, and exact next actions.

Worked examples

Building a policy-comment classifier

User: "I need to label public comments as supporting or opposing the rule."

Agent: "Labeling fits. I need the CSV path, the row ID column, and the text column. For the label, I suggest a multiple-choice question with support, oppose, mixed, and unclear, plus a short rationale for sample review. Do you have any gold labels?"

User: "CSV has comment_id and comment_text. No gold labels yet."

Agent: "I’ll initialize the project, define that rubric, sample 30 comments, and preview the model labels before any full run."

labeling init comments-labeling --csv data/comments.csv --id-column comment_id --columns-used comment_text
labeling inspect
labeling validate
labeling generate
labeling sample --n 30 --strategy random
labeling plan sample --latest

Output: a validated spec, generated EDSL job, sample manifest, and run plan for approval.

After running the plan command from the generated output, record the EDSL results:

labeling record run --kind sample --results data/results.sample.ep
labeling preview latest --human

record run loads Results.ep directly and returns a stable source ID such as sample_run or full_run. Metrics and exports accept that ID immediately; no ep results export and re-import round trip is needed.

Comparing model labels to gold data

labeling labels import model_a results/model_a_labels.csv --id-column comment_id --label-columns topic
labeling gold import data/gold_labels.csv --id-column comment_id --label-columns topic
labeling metrics classification --reference gold --candidate model_a --question topic
labeling preview latest --human
labeling export --final-label-source model_a --output outputs/final_labels.csv

Output: imported labels, classification metrics against gold labels, preview rows, and a final labeled CSV.

Aggregating multiple model raters

labeling labels import gpt4o results/gpt4o.csv --id-column row_id --label-columns issue_category,urgency
labeling labels import claude results/claude.csv --id-column row_id --label-columns issue_category,urgency
labeling aggregate create --name majority --sources gpt4o claude --method majority
labeling export --final-label-source majority --output outputs/majority_labels.csv

Output: an aggregate label source that can be exported or evaluated.

Use labeling labels import for CSV-style model, human, rule, or aggregate label files. Record EDSL Results.ep artifacts with labeling record run --kind sample --results ... or --kind full --results ...; do not pass them to labels import.

Quick command reference

For full options, run labeling <subcommand> --help.

Command	Purpose
labeling init	Create project state and initial spec.
labeling inspect	Profile dataset/spec context.
labeling validate	Validate the project spec and inputs.
labeling generate	Generate EDSL labeling job files.
labeling sample	Create reproducible sample sets.
labeling plan	Show run plan and commands for generated jobs.
labeling record run	Record executed run metadata.
labeling approve sample	Approve reviewed sample labels for provenance-safe reuse.
labeling labels import	Import model or human label results.
labeling preview	Inspect labeled rows before scaling or exporting.
labeling gold ...	Manage gold-standard labels.
labeling rule ...	Create and run keyword-rule baselines.
labeling aggregate create	Combine multiple label sources.
labeling metrics classification	Compute classification metrics.
labeling export	Export final labeled datasets.
labeling status / labeling next	Show source coverage, readiness, blockers, and exact next actions.
labeling report context	Produce bounded JSON evidence for an external report agent.
labeling report template	Produce a noncanonical Markdown handoff template.
labeling report package	Optionally render a package-owned analytical report.
labeling docs	Read built-in guidance.

Common pitfalls

• Running the full dataset before sample review can waste money and lock in a bad rubric.
• Letting the model see target/leakage columns can inflate apparent quality.
• Multi-label tasks should not be forced into single-choice questions unless categories are mutually exclusive.
• Gold metrics are only meaningful if gold labels use the same rubric and row universe.
• Aggregation can hide systematic disagreement; inspect disagreements before treating majority vote as truth.

Cross-references

• Upstream: bewley develops qualitative codebooks; dewey can organize source documents before row-level labeling.
• Downstream: messick validates model-mediated results; manning can use label distributions as calibration targets.
• Reporting: gutenberg, tufte, and herndon create report, figure, and workbook deliverables from labels.

State contract

.labeling/ is the append-only audit store for project events, samples, generated jobs, run records, imported labels, gold labels, rule baselines, aggregates, metrics, exports, and reports. labeling.yaml is the editable project spec. Generated EDSL Jobs artifacts and exported CSVs are derived artifacts; preserve the audit store when rerunning or revising labels.

JSON output and error codes

labeling commands use structured responses for agent workflows. Treat validation errors, missing dataset columns, invalid question specs, unknown label sources, missing gold labels, and export conflicts as recoverable project-state issues: fix the spec or import state, rerun labeling validate or labeling status, then continue.

License

Labeling is released under the MIT License.