Blog: Getting structured data out of images with Granite Vision 4.1

[planetf1]

What this is showing off

Vision models return prose. The point of this post is that they don't have to.

The blog demonstrates Mellea's extraction pattern — pass a format= Pydantic model to
m.instruct(), get a typed Python object back instead of a string. No JSON prompt
engineering, no json.JSONDecodeError handlers, no post-processing regex. The return type
is the contract, and constrained decoding enforces it.

It then builds up two layers of validation on top:

• requirements= — plain-English semantic constraints (date format, positive totals).
  The model retries with the failed requirement injected into the repair prompt.
• IVR validation_fn — programmatic arithmetic check (line items × quantities = subtotal).
  The failure reason gets fed back into the repair prompt verbatim.

The receipt image is synthetic (PIL-generated) with a thermal-printer smudge over part of
the subtotal, giving the validation layers something realistic to catch.

Strategy fix (latest commit): Both validation sections now use RepairTemplateStrategy
instead of RejectionSamplingStrategy. RejectionSamplingStrategy.repair() returns the
unchanged action/context — same prompt, no feedback. RepairTemplateStrategy builds a
repair prompt with the failed requirement or ValidationResult.reason injected, which is
what the surrounding blog prose describes.

---

Status: Draft — scenario still being refined

The Mellea API usage and code structure are stable. The receipt scenario is still being
iterated. Detection is reliable: the date format requirement consistently detects 22/03/2026
and the arithmetic check confirms extractions are correct. Repair of the date format issue
at 4b scale is not guaranteed (the blog's conclusion now says this explicitly). Receipt values
may change before publication.

---

Model availability — why this is blocked

The blog is written for Ollama, which is the right default for a local-first post. The
problem: Ollama requires GGUF format, and Granite Vision 4.1 is only available as full
bfloat16 safetensors on Hugging Face right now (~8 GB download, not a 4-bit quantized GGUF
like you'd get from ollama pull).

Ollama cannot load safetensors directly — it needs IBM to publish a GGUF to the Ollama
library (or a community conversion to appear). Until then, the testing path is mlx-vlm on
Apple Silicon or vLLM, both of which can serve safetensors directly.

Watch https://ollama.com/library for granite-vision-4.1. When it lands: remove the
editorial note, verify ollama pull granite-vision-4.1 works, flip to ready.

---

Reviewing now

Follow the editorial note at the top of the post. Short version:

1. Set up a clean environment and start the model server:

mkdir granite-vision-test && cd granite-vision-test
uv init --bare --python 3.12
uv add mlx-vlm mellea pillow
uv run python -m mlx_vlm.server --model ibm-granite/granite-vision-4.1-4b

Model downloads ~8 GB on first run (full bfloat16 safetensors — larger than an Ollama pull).
Serves at http://localhost:8080/v1.

2. In each code snippet, swap the session setup from:

m = start_session(model_id="granite-vision-4.1")

to:

m = MelleaSession(OpenAIBackend("ibm-granite/granite-vision-4.1-4b",
                                base_url="http://localhost:8080/v1", api_key="mlx"))

---

Test plan

• npm run dev — confirm post renders at /blogs/granite-vision-structured-extraction
• Receipt image displays correctly (smudge on subtotal visible)
• Code syntax highlighting looks right on all blocks
• Run the code against mlx-vlm per the instructions above

🤖 Generated with Claude Code

[psschwei]

Don't have a strong opinion here, but assuming it takes a while to get the vision model into Ollama should we consider using vllm in the blog instead?

[ajbozarth]

I'll walk through the blog and try it out myself when I have bandwidth, but to start heres a small review from Claude:

---

Code checks out against current mellea source — APIs, imports, signatures, and the RepairTemplateStrategy switch all verify. Front matter and asset are good. Snippet syntax checks pass; live execution skipped (model not in Ollama yet, per the editorial note). de-llmify score 1.

A few inline notes below. Pre-publish blockers (editorial note removal, Ollama availability) are already tracked in the PR description.

[ajbozarth]

Consider a one-line caption after this block flagging it as a representative example, not a verbatim response — so a reader running the snippet and getting different prose doesn't think they've broken something.

[ajbozarth]

Light de-llmify nit — the "isn't X — it's Y" construction is a Tier 1 phrase tell. Doing real semantic work here, so optional, but if you want to drop it: commit to the stronger half.

Suggested change

	wiring the check programmatically isn't that repair always succeeds — it's that a silent wrong
	The point of wiring the check programmatically is that a silent wrong answer is no longer possible. Repair success is a separate question.

[ajbozarth]

The detection-vs-repair point in paragraph 3 is the genuinely new framing here; paragraphs 1, 2, and 4 recap ground already covered (and the backend portability paragraph overlaps with "Swapping backends" above). Consider tightening to lead with the detection-vs-repair point and dropping the recaps. Take or leave.