Document processing (IDP)

IDP (intelligent document processing) pulls structured data out of documents whose format you don’t control: equipment records, invoices from different suppliers, letters, certificates. It is not OCR. OCR tells you which letters are there; IDP tells you which of those texts is the serial number.

The difference is where the value comes from:

A template finds the value by position — “the serial number is in the top-right box”. It works until the next hospital’s format shows up.
IDP finds it by meaning — “this is a serial number”. The format can change and it still works.

That’s why IDP pays off when the format varies and you don’t control it. If every document comes out of the same system and always looks the same, a template is cheaper and faster: don’t use IDP.

When it pays off

Both have to be true:

The format varies across suppliers, hospitals, clients or years, and you can’t force them onto yours.
The volume justifies automating. Ten documents get typed in; two thousand don’t.

It doesn’t pay off if the document is always identical (use a template), if the value already exists in an API or database (ask there), or if errors are intolerable and nobody will review. IDP doesn’t remove human review: it shrinks it and points it at the right place.

The pieces

Each does one thing, and all three are needed:

Piece	What it does	Why
Reading	Pulls the text out, with each word’s position	A digital PDF is read for free. A scan goes to Azure Document Intelligence, the only route that reads handwriting
Extraction	A model reads that text and returns your project’s fields	This is what understands meaning and survives format changes
Grounding	Looks for each extracted value among the document’s words	This is where measured confidence and each field’s box come from

Two guarantees, and they’re not the same

This is the one people conflate, and conflating it is expensive:

Classifying answers “is this document the type I think it is?”
Grounding answers “is this value written in the document?”

Neither covers the other. Hand an invoice to an equipment-record extractor and it will find a number that looks like a serial number, locate it in the document, and come out with high confidence. Grounding didn’t fail: the value is there. What failed is that the document wasn’t the right one.

That’s why you classify before extracting, and discard what doesn’t belong.

The extraction project

The extraction project is the Zoan Cloud resource per document type. It is the one thing you configure to support a new type — no code, no training, no labelling examples — and it carries everything the pipeline needs:

The canonical schema — the list of fields you deliver, the contract that never changes for the type.
N format profiles — one per layout the type arrives in, each overriding the canonical per field.
A router — which profile a given document goes to, by its printed format code.
The config — the model, the reading mode, the OCR endpoint and credentials, the storage backend and retention, the review thresholds.

You edit it in the portal, under Extraction projects, and the playbook references it by name:

IDP Extract   project = "device-record"   → res

That is the real reason it is not a project file: the person who knows which fields matter in a device record is a biomedical engineer, not whoever writes playbooks. With the schema inside the deployed playbook, that person cannot touch a threshold without opening the Designer and redeploying. Here they change a field and that’s it.

It is also where the industry landed: UiPath moved the taxonomy out of the workflow into a versioned cloud project — its modern activities “don’t require setting the taxonomy of Document Types” — and Automation Anywhere keeps it in the Control Room as a learning instance.

The canonical schema

The canonical schema is the list of what you want out, and it is what the rest of the tenant consumes — the same field names regardless of which format the document arrived in:

The document type and a description — the classifier comes from this, with nothing extra to write.
Each field: name, type (string, number, integer, boolean, date), description and its confidence threshold.
Tables: a table field with its columns.

Each field’s description is what does the work. It isn’t documentation: it’s what the model reads to decide which value is which. “Serial number” works worse than “the manufacturer’s serial number, printed on the equipment’s plate; not to be confused with the hospital’s inventory code”.

The per-field threshold is a business decision, not a technical one. A misread serial number on a medical device is a serious problem: high threshold. The manufacturer’s name, less so: lower. A field below its threshold doesn’t fail — it goes to review.

Drafting the schema from a sample

Writing the schema by hand is tedious, so the editor can propose it: in the Canonical schema tab, upload a sample document (PDF; an Excel goes through excel-to-pdf) and say what you want extracted. The model proposes the fields — name, type, description-as-instruction, threshold — and every proposal is verified against the document: the example value is located among the words and you see its box on the page (a scan goes through Azure DI with the project’s config). Whatever is not found is flagged unverified and unchecked by default — the draft never looks more confident than it is.

The text the model sees has every line numbered, and it is asked to cite the line each value came from. The box is drawn on that line, not on the first occurrence of the literal in the document: without this, a value that repeats — a manufacture date that reappears as a service date — would always point at the first one and mislead. And when two identical values share the same row (two equal dates, one under “Start date” and one under “End date”), they are told apart by the printed label right before each one: every field carries its own label just ahead, and that decides which box is which. When a value could not be confirmed that way and also repeats in the document, its box is drawn in dashed amber: it is honest to say “this appears several times and I couldn’t be sure which one” instead of flagging one at random. If the literal is unique, the box is correct regardless and shows in green.

You tick what you want and bring it to the grid two ways: Replace schema — what you’d expect from generating: it defines the schema with the ticked fields (if you already had hand-made fields, it asks before overwriting them) — or Merge, which updates same-named fields and adds the new ones without touching the rest. Neither one piles up duplicates when you regenerate. Then you correct it (the nuances — criticality thresholds, disambiguations — remain yours) and save. Nothing publishes itself.

The same exists for profiles: in the Formats tab, upload a document of the new format and the generator extracts with the canonical as-is, verifies field by field, and proposes overrides only for what failed — plus the detected printed format code for the router. A check tells you how many fields the canonical already covered; if it covers them all, you may not need a profile.

These generators’ calls are design usage: they count in the project’s consumption (and the operator’s cost) just like extraction calls — an invisible cost is an underestimated cost.

Format profiles and the router

The same type arrives in several formats — a different hospital, a different year, a different template — and each paints the fields somewhere else. Instead of one taxonomy per format (which drifts, because the delivered contract is then different per format), a project has one canonical schema and N profiles. A profile overrides the canonical per field: a tighter description for that layout, a different threshold, extra table columns, or marking a field not applicable here. Because profiles use the canonical field names, the output is always canonical — the mapping is identity.

The router picks the profile. Each profile declares a routerKey — the printed format code of that layout (F-GOP-007, FOR-GBM-001), literal or a regex, with a priority when several could match. IDP Extract reads the code off the document and routes; the model is the fallback when no code matches; a document that matches nothing goes to quarantine, never to a wrong profile.

Gold sets and measurement

A project can hold a gold set: a handful of documents labelled by hand with their true values. It is not for training the model — it is for measuring it.

Labelling the gold happens in the project’s Measurement (gold) tab. You annotate the correct values field by field: scalars in a form, and tables in a per-cell editor —a grid with the schema’s real columns, where you add one row per row the table has in the document and type each cell’s value. Ten to twenty documents per format is usually enough for a reliable signal.

Running a measurement is a button in that same tab. Zoan Cloud compares the gold against the already-persisted extraction of those documents (the one IDP Extract left behind), per field —exact / normalised / critical, and per cell for tables— and stores a report: accuracy per field, per profile, per model. It is server-side: it does not re-extract or pay Azure/LLM again, and it measures against the published schema (the same one the runtime uses), not the draft you are editing. So to measure a schema change, publish it first.

The requirement is that each gold is linked to its document (by document_id) and that document has already been extracted; otherwise the report tells you with a clear error.

That is what lets you answer “is a small model accurate enough on my documents?” or “did this change to a description help?” with a number instead of a hunch — the honest way to pick thresholds and models before running two thousand documents.

Two loops close the improvement cycle:

Publishing measures on its own. When you publish a project version, the gold measurement runs automatically (if extracted gold exists) and the dashboard shows the delta against the previous run — nobody publishes a schema that got worse without seeing it.
Most-corrected fields. Every correction a reviewer makes while validating is recorded, and the dashboard aggregates them: the field people fix over and over points at the schema description to rewrite. Rewrite, publish, measure.

Scope: environment or global

Same as credentials:

Global — applies across the tenant.
Environment — only there, and it wins over the global one with the same name.

That is what lets you try a change where it doesn’t hurt before rolling it out.

The escape hatch: inline resolve JSON

project also accepts the resolved project inline as JSON — canonical schema, profiles and router in one object — instead of a name. That is for CI and for the CLI, which have no Zoan Cloud session and cannot resolve a name. In a real run from Zoan Cloud you always pass the name.

Publish and version

The runtime extracts against the project’s published version, not against what you’re editing. When you click Publish, a version is frozen; from there, editing the schema or the profiles is a draft that doesn’t affect in-flight extractions until you publish again. The editor shows the state: Published, Unpublished changes or Not published. So you can iterate the schema safely without breaking what’s running.

Auto-accept and QA sampling

The project’s governance decides what needs human eyes:

A document with fields below threshold ALWAYS goes to review, whatever the policy says.
With auto-accept off (the default), clean documents also pass through a person before flowing — the prudent mode until you calibrate with the gold set.
With auto-accept on, clean documents auto-validate and flow on their own; QA sampling (a configurable fraction) still sends some clean ones to the inbox, tagged QA, to watch quality without blocking the business (the sampled document already flowed).

In the inbox, every document says why it is there: fields below threshold, project policy, or QA sampling.

Model escalation

If you configure an escalation model, when extraction leaves more than 10% of the fields below threshold it retries once with that (more capable) model and the result with fewer review fields wins. The reviewer’s minute is IDP’s dominant cost: a second call worth cents that avoids a review is a good trade. Usage of both calls is recorded.

The OCR cache for bundles

When an uncertain bundle goes through the classification station, the playbook can upload it with its reading already done (the document parameter of Documents — Send Bundle to Classification). When the confirmed segments are extracted later, Take Confirmed Bundles returns that reading and IDP Extract slices it per segment — the same paper is never paid to Azure twice.

Output queue: the hand-off to the downstream bot

In the project’s governance you can name an output queue. If you set it, extracted documents enter that queue as work for the bot that processes them downstream — the clean ones at extraction, the ones that need review at validation — always with their final values. Empty = no queue: the data stays in the store for whoever consumes it. The queue is a business decision, not something IDP imposes.

Bundles: one file, several documents

A client rarely hands you one document per file. A single PDF arrives with the device record, its maintenance reports, calibrations and annexes stuck together. IDP Classify handles that: it classifies each page — its type and whether it starts a new document — and splits the file into segments, each with its real type. Boundaries come from a type change, from a start the model detects (letterhead, a new heading, a page-1 field) or from the printed “page 1 of N” mark.

The playbook iterates the segments and extracts each one against its own project, by its page range. A single-document file comes out as one segment, so the normal case is unchanged.

When the split isn’t clear: the classification station

Splitting several documents of the same type stuck together with no marker is ambiguous. IDP Classify flags those cases with splitNeedsReview. Instead of extracting blindly, the bundle goes to the classification station: a person sees the pages, adjusts the boundaries and types, and confirms; only then is each segment extracted. It’s the conditional gate that makes the split come out right no matter how the document arrives. See the tutorial.

Configuration

The model credential

Create it in the environment with a single property, named exactly value, holding the API key.

Field	Value
Name	`anthropic-key`
Property → key	`value`
Property → value	the API key
Sensitive	checked

The property name matters: with a single property named value, Zoan Cloud returns the key as a plain string. With any other name — or with several properties — it returns an object, and then the playbook has to index it: = credential("my-credential")["api-key"].

Getting it wrong doesn’t fail silently: the activity tells you the credential does not contain the API key.

Azure Document Intelligence (scans only)

Needed only if you’ll process scanned or handwritten documents. A PDF with a text layer is read without leaving the machine and without cost.

Configure your resource’s endpoint and a credential with its key. With mode = auto the activity decides on its own: if the document has text it reads it directly; if not, it sends it to Azure. Both routes return the same thing, so nothing downstream changes.

The dpi, in both places

If you’re going to upload the document for review, the pages are rendered with PDF To Images. The dpi must match the one you gave IDP Load Document: field boxes come back in pixels at that dpi and are drawn on those images with no conversion.

With mismatched dpi nothing fails: the boxes simply land in the wrong place, and nobody tells you.

Review

A processed document ends up in one of these states:

Not extracted — uploaded but not extracted. That’s a real state: uploading and extracting are two different steps.
Pending review — extracted and not signed off by anyone.
Validated — a person approved it.

Pending doesn’t mean “IDP was unsure”. It means nobody has looked at it yet. A document with all 44 fields clean still needs someone to sign it; the flagged-field count only tells you how much work is inside.

In the validation station the document sits on the left with each field’s box on top of it, and the work panel on the right. Review is a guided flow, not a list to scan:

The station walks you from one pending field to the next: Enter accepts the value and moves on, Shift+Enter goes back, and every field has its ✓ “it’s fine” button. A progress bar tells you how much is left.
The sync goes both ways: clicking a field jumps to its page, centres its box and dims the rest; clicking a box on the document takes you to its field and focuses it.
Already-verified fields collapse to a single line — with 45 fields and 3 to review, you see 3.
Tables are edited at full page width (a table is wide by nature), with a sticky header and the cells under review highlighted; the panel keeps one chip per table with its pending cells.
The image has zoom (+/−/fit) and the page thumbnails show how many pending fields each page holds.

You fix what needs fixing and sign.

A corrected value loses its review mark — a person looked at it — but everything else is kept: boxes, confidences and reasons. That’s the traceability of where each value came from, and dropping it on validation would leave the document unable to explain itself.

Retention

Uploaded documents expire. Each environment has a default retention, and each upload can ask for its own.

This isn’t an infrastructure detail. Storage costs cents; keeping too much is legal liability. A clinical document or a payslip still sitting there three years after it served its purpose is a risk, not an archive. It’s worth deciding deliberately: maybe 90 days for what’s already validated and years for what a regulation demands, in the same batch.

A sweeper deletes what has expired — the file, its pages and its record — without you having to remember.

Isolation

Documents never mix between tenants. Each lives under its tenant’s prefix in storage, and the database enforces it at the row level: even if a query dropped its filter, it returns nothing from another tenant.

Neither the tenant nor the environment travel in the upload request: Zoan Cloud resolves them from the authenticated execution. A playbook cannot upload to the wrong tenant, by accident or on purpose.

Costs

You pay for three things, and only one is expensive:

	Cost
Reading a digital PDF	Free — it never leaves the machine
Reading a scan (Azure DI)	Per page
Classifying	One short call per document
Extracting	The heavy one: the whole document goes in, JSON comes out

Every AI activity can leave its usage — model, input and output tokens — in a variable. Store it in the item’s result and you get the real cost per document, queryable by API, with nothing estimated.

The project’s Usage tab aggregates all of this in units: pages processed (the unit that applies to service billing), documents, tokens per model and OCR pages. The provider cost in dollars does not appear in the tenant portal — it belongs to the platform operator and lives in their portal (client detail → IDP Usage), computed from billing.provider_pricing prices ($/million tokens per model, $/page for Azure DI), editable without redeploying. A model with usage but no configured price is flagged as unpriced — the cost shown is a floor, never a misleading $0.

Two ways to cut it a lot: a small model is just as accurate on structured documents (worth measuring with your own documents before paying for a big one), and a backfill isn’t in a hurry — if you can wait, batch processing costs half.

A limit worth knowing

A real schema doesn’t fit in one call. The API rejects schemas above roughly 20 fields, so IDP Build Schema returns several schemas and the playbook loops over that list.

This isn’t an implementation detail you can ignore: it’s why the playbook has a loop, and why extractions accumulate into a list before grounding. The document is sent once per batch, and that’s where most of the cost is.

Next steps

IDP activities — what each node does and the full playbook
Work queues — the pattern for processing batches with retries
Credential management — how they’re stored and resolved