Turn a folder of raw images into a clean, structured-JSON caption dataset for fine-tuning Ideogram 4 (or any model trained on its caption schema). Fully autonomous, resumable, and schema-validated end to end.
Datagram4 runs a five-stage pipeline that detects every subject and text
block in an image, describes each one with a vision-language model,
transcribes the on-image text, and assembles a caption that passes
Ideogram 4's own CaptionVerifier. The output is ready to feed straight
into a training loop.
Ideogram 4 was trained on dense, structured JSON captions, not plain-text prompts. Fine-tuning it (or a derivative) requires a dataset in exactly that distribution, and hand-writing hundreds of compliant captions is impractical. Datagram4 automates the whole construction:
- Open-vocabulary detection finds objects and text from a free-form prompt.
- A dedicated text detector catches small, dense text that the object detector misses.
- A vision LLM writes literal, visually-grounded descriptions and reads stylized typography accurately.
- The result is deduplicated, normalized, and validated against the reference schema before it ever touches disk.
The pipeline is intentionally conservative about data quality: descriptions that the LLM refuses to write are detected and handled, LaTeX artifacts in transcribed text are normalized, and duplicate reads of the same wordmark are collapsed so a downstream model never learns to render the same text twice.
data/raw_images/
|
v
+----------------------+ Grounding DINO -> objects + general text (boxes)
| Stage 1: DETECT | EasyOCR -> small/dense text GD misses (boxes)
| | SAM 2 -> refine each box into a tight mask
+----------+-----------+
| elements: [{label, bbox, score, source}]
v
+----------------------+ Qwen3-VL (or any OpenAI-compatible VLM)
| Stage 2: DESCRIBE | - describe_global: whole-image HLD + background
| | - describe_element: per-crop description
| | - transcribe text (lifted into the `text` field)
+----------+-----------+
| per-element desc + transcribed text
v
+----------------------+ Format-B caption assembly
| Stage 3: ASSEMBLE | - fold style into HLD/background prose
| | - dedup duplicate/fragmented text elements
| | - normalize markup artifacts
+----------+-----------+
| caption dict
v
+----------------------+ ideogram4 CaptionVerifier
| Stage 4: VALIDATE | - schema / key-order / bbox-range checks
| + EXPORT | - export (image, json) pairs to data/dataset/
+----------------------+
Each image produces a paired image.webp + image.json matching the
training-caption schema:
{
"high_level_description": "A studio product photograph of a Falcon Cycles Model 8 ...",
"compositional_deconstruction": {
"background": "Solid dark navy blue studio background ...",
"elements": [
{"type": "obj", "bbox": [267,107,650,888], "desc": "Falcon Cycles Model 8 road bicycle, large and centered ..."},
{"type": "text", "bbox": [43,292,93,706], "text": "FALCON CYCLES", "desc": "Small, centered, wide-spaced sans-serif font in white."}
]
}
}Bounding boxes are [y_min, x_min, y_max, x_max] normalized to 0-1000,
origin top-left.
Three sample images and their generated captions are included in this repo
under data/raw_images/ (inputs) and
data/dataset/ (outputs). They span the cases the pipeline
is built for: a product photograph, a scene with stylized neon typography,
and a graphic-design layout. The annotated previews below show the final
detected boxes: blue for objects, orange for text.
Input: data/raw_images/image1.webp |
Caption: data/dataset/image1.json
A studio shot of a cream-colored bicycle against a dark blue backdrop. The
pipeline segments the bicycle as one object and recovers every text block on
the layout, including small footer type that the object detector initially
missed (HAND BUILT - CHICAGO), found by the dedicated text detector:
{"type": "text", "bbox": [963,730,974,959], "text": "HAND BUILT - CHICAGO",
"desc": "Large, bold, uppercase, sans-serif font in white, centered within the crop."}
Input: data/raw_images/image2.webp |
Caption: data/dataset/image2.json
A vintage cinema marquee at twilight. Stylized, backlit text that dedicated
OCR engines routinely misread is transcribed accurately by the vision LLM:
IDEOGRAM 4.0 / OUT NOW, CINEMA, and the price tag .25c. The text-dedup
step collapses the marquee, which the detector reads at several granularities,
into a single element so a downstream model does not learn to render it twice.
Input: data/raw_images/image3.webp |
Caption: data/dataset/image3.json
A retro album cover combining a photograph with bold graphic overlays. The
caption separates the photographic subject from the typographic and graphic
elements, and preserves non-ASCII characters literally (SIDE A - SUMMER 86 - 45 MIN uses the rendered middle dot, not an escaped codepoint).
To show exactly what each stage produces, the repo includes the intermediate
output for image1 (the bicycle) under
examples/stage_results/. Each file is the real
data structure that stage passes to the next, with a _description field
explaining what it represents.
| File | Stage | What it contains |
|---|---|---|
1_detect.json |
DETECT + SEGMENT | Raw boxes from Grounding DINO + EasyOCR + SAM2, with label, score, and normalized bbox per element |
2_describe.json |
DESCRIBE | The vision LLM's whole-image read (global_info) plus per-crop descriptions and the EasyOCR hint |
3_assemble_validate.json |
ASSEMBLE + VALIDATE | The final exported caption after text-lifting, dedup, normalization, and schema validation |
Following the data from 1_detect.json through to 3_assemble_validate.json
is the fastest way to understand the pipeline: boxes become descriptions,
descriptions become the schema-valid caption. Compare the same bicycle
element across all three files to see how a raw detection turns into a
final desc.
- Python 3.10+
- An Apple Silicon Mac (MPS) or an NVIDIA CUDA GPU
- An OpenAI-compatible vision LLM endpoint (vLLM, ollama, llama.cpp /
Unsloth Studio, etc.). Tested with
Qwen3-VL-8B-Instruct.
git clone https://github.com/halilugur/datagram4.git
cd datagram4
# 1. Configure (copy the example, never edit secrets into the real config)
cp config.example.yaml config.yaml
# then edit config.yaml to point at your LLM endpoint and model
# 2. Export your credentials (config.yaml reads these via ${VAR})
export LLM_BASE_URL="http://localhost:8000/v1"
export LLM_API_KEY="your-key-here"
# 3. Add source images
mkdir -p data/raw_images && cp /path/to/your/images/*.webp data/raw_images/
# 4. Run (installs deps + downloads weights on first run)
./run.shDataset pairs land in data/dataset/. Resumable: re-running ./run.sh
skips images already finished.
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python run_pipeline.py --config config.yamlAll settings live in config.yaml (gitignored — copy config.example.yaml).
Secrets are never hardcoded: reference environment variables with the
${VAR} or ${VAR:-fallback} syntax and the pipeline resolves them at
startup.
Key knobs:
| Section | Setting | Purpose |
|---|---|---|
detection |
box_threshold |
Recall lever; lower finds more boxes (more noise) |
detection |
text_row_merge_overlap |
Merges EasyOCR fragments on the same text line |
detection |
text_dedup_overlap / text_dedup_containment |
Drops duplicate text reads (e.g. FALCON + FALCON CYCLES) |
describer_llm |
model |
Any OpenAI-compatible vision model |
validation |
drop_on_verifier_error |
Hard-fail vs. warn-and-keep on schema errors |
inspect_image.py renders a read-only, multi-panel visualization of every
stage an image passed through, drawn directly from the saved manifest:
python inspect_image.py image1 # render + open the panels
python inspect_image.py --all # every imageIt shows, left to right: every detected box (color-coded by detector), the per-element descriptions, and the final exported caption. The tool only reads persisted data; it does not run the pipeline or call any model.
The manifest tracks each image's progress, so runs are resumable by default. Run flags:
| Command | Behavior |
|---|---|
./run.sh |
Resumable; skips images already at done. |
./run.sh --retry-failed |
Resets only failed images to pending; keeps finished work. Use after fixing a transient error (e.g. the LLM server was down). |
./run.sh --reset |
Clears manifest + output for a fully fresh run. Use after changing prompts, thresholds, or models. |
./run.sh detect / describe / assemble / validate |
Run a single stage. |
Individual stages:
python run_pipeline.py --stage detect # detection + segmentation only
python run_pipeline.py --stage describe # descriptions + transcription onlydatagram4/
run_pipeline.py # orchestrator + stage functions
inspect_image.py # read-only journey visualizer
run.sh # one-command setup + run wrapper
config.example.yaml # committed template (placeholders only)
config.yaml # your real config (gitignored)
pipeline/
detect_segment.py # Grounding DINO + EasyOCR + SAM 2 detection
describe.py # VLM descriptions + OCR + env-var resolution
assemble.py # caption schema assembly
validate.py # CaptionVerifier gate
manifest.py # resumable progress tracker
data/ # raw images, manifest, dataset (gitignored)
checkpoints/ # SAM 2 weights (gitignored, downloaded by run.sh)
| Role | Model | Loaded via |
|---|---|---|
| Object / general detection | Grounding DINO (tiny) | HuggingFace transformers |
| Text-region detection | EasyOCR | easyocr |
| Mask refinement | SAM 2 (hiera-large) | sam-2 |
| Description + transcription | Qwen3-VL-8B-Instruct (or any compatible VLM) | OpenAI-compatible API |
| Schema validation | Ideogram 4 CaptionVerifier |
ideogram4 |
The describer is swappable without code changes: point describer_llm.model
at any OpenAI-compatible vision endpoint. For reasoning-capable models
(Gemma-4 thinking, Qwen3-VL Thinking), the pipeline disables thinking per
request so descriptions return directly.
Apache License 2.0. See LICENSE.
Datagram4 is an independent tool and is not affiliated with or endorsed by Ideogram AI. "Ideogram" is a trademark of its respective owner. This project consumes Ideogram 4's public schema and verifier for compatibility only.