IDP — Extract

Type: idp-extract  ·  Package: IDP Activities v1.20.4  ·  Output: object

Since: v1.6.0

Extracts everything an extraction project asks for from a document, routes to the profile of the layout in front of it and looks each value up on the page, so you know where each value came from and how much confidence it deserves.

It is the normal IDP path: one activity. Internally it loads the document if needed, routes to the profile, extracts every field in a single call to the model, grounds and stores the result in Zoan Cloud — but that is its business, not yours.

Idempotent: it does not pay twice

IDP — Extract stores its result by reference (by default, the file name). If that document was already extracted, it reuses what was stored and does not call Azure or the model again — a retry after a downstream failure costs nothing. It also stores the document and the extraction, so you don’t need Documents — Upload afterwards.

IDP — Extract   document = doc
                project  = "supplier-invoice"
                → output: res

The project carries the configuration, not the playbook

The activity asks for an extraction project and everything else comes from it. The project —edited in the portal, in Document processing— carries:

• the canonical schema (the fields you deliver, the contract),
• N format profiles, one per layout, each with its adjustments over the canonical,
• a router that picks the profile by the format’s printed code,
• the config: model, escalation model, reading mode, OCR endpoint and the OCR and model credentials.

So whoever knows which fields matter configures them in the portal and does not touch the playbook. The playbook only says which project and where the document comes from.

How it works

1. Resolves the project in Zoan Cloud (or parses the inline JSON, for CI).
2. If there is a reference and the document was already extracted, returns what was stored (with reused = true) without calling anything else.
3. Uses the document you pass or reads path with IDP — Load Document and the project’s config. If you pass document and a pages range such as "4-7", it slices that reading to the range instead of re-reading.
4. Routes to the format profile and extracts every field in one call to the model. If maskPii is on, the document text goes out with personal data masked and the real values come back in the result. If the answer is not valid JSON, it asks for one cheap repair; if that fails too, it fails.
5. Locates each value with IDP — Ground, applies model escalation when it applies and sends to review tables with identical consecutive rows.
6. If there is a path, a reference and Zoan Cloud, stores the file and the extraction and returns its documentId.

Required parameters

Parameter	Type	Default	Description
project	string	—	The name of an extraction project in the environment. It carries the canonical, the profiles, the model, the reading mode and the OCR endpoint. For CI it also accepts an inline project-resolve JSON.

Optional parameters

Parameter	Type	Default	Description
path	string	—	Path to the document (PDF, PNG, JPEG or TIFF), if you don’t pass document. You need path or document.
document	Dictionary	—	A document that IDP — Classify or IDP — Load Document already read. If you pass it, the file is not read again — with a scan, re-reading it means paying Azure twice.
reference	string	file name of path	Stable key for the document (idempotency): if it was already extracted, the result is reused without calling Azure or the model. Pass one per segment when you split a bundle. It comes back in the output as reference. Since 1.14.0.
sourceGroupId	string	—	Only when you split a bundle: the id (for example, the original file’s reference) that groups the documents that came out of the same file. Together with pages, it marks this document as a segment: Zoan Cloud stores its page range and shows only those pages. Since 1.15.0.
dpi	int	200	The dpi the boxes come back in. Range: 36 to 1200. Use the same one you give PDF To Images if you are going to show the document, or the boxes land in the wrong place. Ignored if you pass document.
pages	string	— (all)	Which pages to read, such as "1-3". With document, an "A-B" range slices the given reading; with sourceGroupId, an "A-B" range is stored as the segment’s range.
maxRetries	int	5	Retries on a transient provider failure (0 to 10). A bulk run is thousands of calls: the odd 429 or 5xx is a certainty, not a risk.
maskPii	bool	true	Replaces the personal data in the document text with placeholders before sending it to the model, and restores the real values in the result. Since 1.19.0.

Set in the extraction project

The credentials are not shown in the activity panel: they are set in the Lectura y modelo (Reading and model) tab of the extraction project in Zoan Cloud, together with the model, the escalation model, the reading mode and the Azure DI endpoint.

Project field	Parameter	What it controls
Model credential	credential	Your own key for the model provider. Empty, the AI included in your plan.
Azure DI credential	azureCredential	Scans only: the key of your Azure Document Intelligence resource. With the endpoint and credential empty, the reading included in your plan.

Give it the document, not the path

If you come from IDP — Classify, use document = doc. The file is read only once. Keep in mind that, without path, the activity does not store the document in Zoan Cloud: also pass path and reference if you want it stored and idempotent.

It routes to the format profile

The same document type arrives in several layouts, and each layout puts the fields somewhere else. The project has one profile per layout; IDP — Extract reads the printed code on the document (for example F-COM-007), picks the matching profile and merges its adjustments over the canonical. The output always comes out with canonical names — the mapping is identity.

A layout that matches no profile goes to review

If the project has profiles and the document matches none of them, it is extracted with the project’s general schema, the log warns about it and the result comes out with needsReview = true and formatMatched = false, even if the fields came out clean: an unknown layout is never accepted on its own. A project without profiles always extracts with its general schema, without flagging anything.

Model escalation

If the project sets an escalation model, when more than 10% of the fields fall below their threshold the activity retries once with that model and keeps the result that leaves fewer fields to review. An extra call worth cents that saves minutes of a reviewer’s time. The usage of both calls is added up in usage.

It does not classify

It extracts what it is asked for, always. Checking that the document is the right one is your decision, made beforehand with IDP — Classify.

It is not a whim: an activity that decides on its own not to do its job hides the control flow, and it also prevents routing between several projects — exactly what you need when the queue brings mixed documents.

No credentials: the AI and the reading included in your plan

Credentials are optional. If neither the activity nor the project has them and your Zoan Cloud plan includes AI and IDP, the model and the scan reading go through Zoan’s accounts via Zoan Cloud: the robot works without creating keys, and the usage (tokens and pages) shows up in AI Usage and counts toward what your plan includes. With your own credentials, calls go straight to your providers and Zoan Cloud only records the usage.

With the included reading, a document that does not fit in the page quota left is rejected before it is read, with the figures, and you never get a partial reading presented as complete (see IDP — Load Document).

Output

Field	Type	What it is
values	Dictionary	The data, clean: { number: "FV-1234", items: [...] }. It is what the rest of the playbook consumes.
fields	List	One entry per field, with its box, its confidence and why it needs review (the shape is in IDP — Ground). It is what the validation station draws.
needsReview	bool	Whether any field fell below its threshold, a table repeats rows, or the layout matched no profile.
reviewFields	List	Names of the fields to review.
documentType	string	The type the project declares.
format	string	The profile it was routed to (e.g. F-COM-007). Useful to trace and show which layout was used.
formatMatched	bool	Only present, as false, when the project has profiles and none matched.
source	string	text or azure-di.
dpi	int	The dpi of the boxes.
pageCount	int	Pages read.
pages	List	Each page’s dimensions (page, width, height, text), in pixels at the boxes’ dpi.
reference	string	The key it was stored under (idempotency). null if you passed document without path or reference.
documentId	string	The id of the document stored in Zoan Cloud. Only when it was stored (there is a path, a reference and Zoan Cloud) or reused.
reused	bool	Only present, as true, when the result was reused from an earlier extraction.
usage	Dictionary	The document’s usage: { model, inputTokens, outputTokens, calls }.

Usage comes in tokens, not dollars

Extraction asks for every field in one call, so the document is read once. usage carries that call’s tokens (plus the repair or escalation call, if any). It is in tokens and not dollars on purpose: the price per model changes, so whoever consumes it works out the cost with the current rate. The usage also shows up in AI Usage.

Confidence is not asked of the model

A model grading itself says “95%” with the same certainty when it is right as when it makes something up. That confidence is not calibrated.

The one here comes from looking for the literal among the document’s words. If it is there, there is confidence and a box. If it isn’t, the value cannot be verified and the field goes to review. It is a mechanical check, not an opinion.

Example

Extract a document from a queue and close the item with the data:

IDP — Extract   path      = content["path"]
                reference = item.Reference
                project   = "supplier-invoice"
                → output: res

If   condition = res["needsReview"]
  Log   message = "To review: " + toString(res["reviewFields"])

Queue Set Transaction Status   item = item   status = "successful"
                               output = { "documentId": res["documentId"],
                                          "values": res["values"] }

Extract each segment of a bundle without re-reading it:

For Each   items = doc["segments"]   itemVariable = segment
  IDP — Extract   document      = doc
                  path          = file
                  project       = segment["type"]
                  pages         = segment["pageStart"] + "-" + segment["pageEnd"]
                  reference     = file + "#" + segment["pageStart"]
                  sourceGroupId = file
                  → output: res

Common errors

Message	Cause and fix
'project' is required.	The project name is missing.
Give it a 'path' to read, or a 'document' that IDP Classify already read.	Neither path nor document was passed.
Extraction project '…' does not exist in this environment. Create it in the portal.	The name matches no project in the environment.
'…' is a project name but there is no Zoan Automation runtime to resolve it. …	No Zoan Cloud in the run (e.g. from the command line). Run it from Zoan Cloud or the Designer, or pass the project JSON.
This extraction project has no fields in its schema yet. Add them in Zoan Cloud before extracting.	The project is empty. Set up its canonical schema and publish it.
The model did not return valid JSON for the extraction, even after a repair attempt.	Unrecoverable model answer. Retry with a Retry; if it keeps happening, review the field descriptions.
Zoan Cloud rejected the document lookup (…): … / Zoan Cloud rejected the document persist (…): … / Zoan Cloud rejected the extraction persist (…): …	Zoan Cloud did not accept the lookup or the save. The detail carries the code; retry.
The document has N pages and M are left in your plan's quota this month. … / The reading asks for N pages and M are left in your plan's quota this month. …	The document or range does not fit in the page quota. All reading messages in IDP — Load Document.
The AI usage included in your plan for this month has been reached. Use your own credential or upgrade the plan.	The included AI usage is spent. Set your own credential on the project or upgrade the plan.
Choose a credential or turn on the AI included in your plan.	Neither the project nor the activity has a credential and your plan does not include AI.
The model hit the 'maxTokens' limit before finishing its response. …	The document has too many fields or rows for one answer. Split the document by pages.

Related activities

• IDP — Classify — which type it is, before extracting
• IDP — Ground — standalone grounding, if you build the pipeline by hand
• Documents — Upload — upload a document by hand for review