IDP — Build Schema

IDP — Build Schema

Type: idp-build-schema  ·  Package: IDP Activities v1.20.4  ·  Output: List<object>

Since: v1.1.0

Turns a taxonomy into the JSON Schemas that AI — Extract consumes. For each field it asks for three things: the interpreted value, the literal text from the document it comes from and the line number where it appears — which is what IDP — Ground then uses to locate it.

The normal flow no longer uses it

IDP — Extract extracts every field in a single call (without a strict grammar), so it does not split the schema into batches or use this activity. IDP — Build Schema stays as a low-level piece for anyone building an extraction pipeline by hand.

How it works

It validates the taxonomy and splits it into batches: a model with guaranteed output compiles the schema before generating, and a schema that is too large gets rejected. Each batch holds at most maxFields cells: a scalar field counts 1 and a table counts one per column. A table is never split across batches, because its columns have to go together to return complete rows.

The log says how many fields (and cells) there are and how many batches they ended up in. If a table on its own has more columns than maxFields, it still goes in its own batch and the log warns about it: if the model rejects that schema, that is why.

Required parameters

Parameter	Type	Default	Description
taxonomy	string | Dictionary	—	The taxonomy: the path to a .json, the JSON as text, or the object. It describes a type of document, not a layout.

Optional parameters

Parameter	Type	Default	Description
maxFields	int	7	How many cells go in each batch (positive integer). A table counts once per column and is never split. Raising it cuts the number of calls, but the model rejects a schema that gets too large: change it only if you can re-test the taxonomy. Since 1.5.0.

Output

Returns a List<object>: one JSON Schema per batch, each ready to hand as-is to AI — Extract. A small taxonomy returns a one-element list.

Each schema has a "type": "object" root with one property per field in the batch. Each field asks for { value, valueRaw, line }: line always (-1 if the field does not appear), while value/valueRaw are left out when the field is not there, instead of being made up. A date field is asked for as text in date format, and a boolean is read as a checkbox (☑ checked, ☐ empty). A table asks for a list of rows with that same shape per cell. Pass all the answers, in a list, to IDP — Ground: that is where they are merged.

The taxonomy

It describes a type of document, not a template. That is where the value is: the visual layer changes from one issuer to another, but the semantics do not. A single supplier-invoice.json works for every supplier.

{
  "documentType": "supplier-invoice",
  "description": "A supplier's purchase invoice. The layout varies between issuers.",
  "fields": [
    {
      "name": "number",
      "type": "string",
      "description": "Invoice number, exactly as printed",
      "minConfidence": 0.95
    },
    {
      "name": "supplier",
      "type": "string",
      "description": "Legal name of whoever issues the invoice",
      "minConfidence": 0.6
    }
  ]
}

Key	Required	What it is
documentType	yes	Identifier of the document type.
description	no	What this document is. Passed to the model as context.
fields	yes	List of fields; it cannot be empty or repeat names.
fields[].name	yes	Name of the field in the result.
fields[].type	no	string (default), number, integer, boolean, date or table.
fields[].description	yes	What this field is.
fields[].minConfidence	no	Confidence required to accept it, between 0 and 1. Default 0.8.
fields[].fields	only in table	The table’s columns.

Tables: fields that repeat

A field of type table is a field that repeats: one row per occurrence in the document. It is what lets you extract an invoice’s line items, a parts list, or anything with N entries.

{
  "name": "items",
  "type": "table",
  "description": "The invoice's line items",
  "fields": [
    { "name": "description", "type": "string", "description": "Description of the product or service", "minConfidence": 0.7 },
    { "name": "quantity",    "type": "number", "description": "Quantity billed",                      "minConfidence": 0.9 },
    { "name": "amount",      "type": "number", "description": "Line total",                           "minConfidence": 0.9 }
  ]
}

Columns are declared just like a normal field, with their own description and their own minConfidence — because a quantity and a description are not read with the same reliability. The table itself has no minConfidence: it is not verified as a whole, it is verified cell by cell.

No tables inside tables

Columns can only be scalars (string, number, integer, boolean, date). A nested table does not exist in these documents, and allowing it would complicate the schema, the grounding and the review screen for nothing.

An empty table is not an error

If the document has no such rows, the result is an empty list and it is not flagged for review: “this invoice has no credit notes” is a real and frequent state, and flagging it would send every document in the batch to a person.

The description is not documentation — it is the prompt

The model reads each field’s description to decide what to put in it. That is why it is required. Write it as you would explain it to someone seeing the document for the first time: “invoice number, exactly as printed” works much better than “number”.

Thresholds are per field

minConfidence is the minimum confidence to accept the field without a person looking at it. The higher, the stricter.

It is per field on purpose. An invoice or serial number is alphanumeric with no linguistic redundancy: it is exactly where OCR fails, and where getting it wrong is costly — so near-certainty is required (0.95). A company name is always read well and context corrects it on its own, so demanding the same would only send to review work that doesn’t need it (0.6).

A single threshold for the whole document sends everything to review and kills the ROI.

Example

The pipeline by hand, with one AI — Extract per batch:

IDP — Load Document   path = docPath   → output: doc
IDP — Build Schema    taxonomy = asset("taxonomies/supplier-invoice.json")   → output: schemas
Create List           → output: answers
Foreach   items = schemas   itemVariable = schema
  AI — Extract   prompt = doc["text"]
                 schema = schema
                 → output: partial
  Add to List    list = answers   item = partial
IDP — Ground          document   = doc
                      extraction = answers
                      taxonomy   = asset("taxonomies/supplier-invoice.json")
                      → output: res

Common errors

Message	Cause and fix
'maxFields' must be a positive integer. Received: …	Invalid maxFields value.
'taxonomy' is required. / 'taxonomy' is not valid JSON: … / 'taxonomy' must be a JSON object, but it is …	The taxonomy is missing or not a JSON object. If you pass a path, check that the file exists.
The taxonomy must declare a 'documentType'. / The taxonomy must declare a 'fields' array. / The taxonomy's 'fields' array is empty; there is nothing to extract.	A required taxonomy key is missing.
The taxonomy declares the field '…' more than once.	Repeated field names (case-insensitive).
Every field in the taxonomy must have a 'name'. / Field '…' has no 'description'. The model reads it to decide what belongs in this field, so it is not optional.	A field without a name or a description.
Field '…' has an unknown type '…'. Valid types: string, number, integer, boolean, date, table. / Column '…' has type '…'. A table's columns must be one of: string, number, integer, boolean, date.	Invalid type (or a table inside a table).
Field '…': 'minConfidence' must be between 0 and 1. Received: …	Threshold out of range or not a number.
Field '…' is a table, so it must declare a 'fields' array with its columns. / Table '…' declares no columns. / Field '…' declares 'fields' but its type is '…'. Only a field of type 'table' has columns.	Wrong table structure.

Related activities

• IDP — Ground — the next step: locate and validate what was extracted.
• AI — Extract — the model call that uses each schema.
• IDP — Extract — the whole pipeline in one activity.