Documents are how businesses tell each other things. A supplier does not send an API call announcing what it billed; it sends a PDF. A candidate does not populate your HRIS; they email a resume. A regulator does not query your database; it issues a notice. Between every two organizations sits a layer of documents, and someone has to turn those documents into data the receiving system can act on.
For most of the last thirty years that translation was done by people, occasionally assisted by template-based scanning software that worked as long as nothing about the layout changed. Intelligent document processing is the category that replaced it. The distinguishing claim is not that IDP reads text — that problem was solved decades ago — but that it understands what the text means, decides how confident it is, and hands back structured records rather than a wall of characters.
This guide defines the term precisely, walks through the pipeline stage by stage, explains the document taxonomy that determines how hard any given document is, and covers the concept that governs everything in production: the confidence threshold. Understanding that last piece is what separates teams who get real straight-through processing from teams who end up reviewing every document anyway.
The Definition
Intelligent document processing is the automated conversion of business documents into validated, structured data using a combination of optical character recognition, machine learning, natural language understanding, and business rule validation.
Three words in that definition carry the weight.
Automated means the pipeline runs without a person initiating each document. Files arrive, get processed, and land in a downstream system on their own.
Validated means the extracted data has been checked against something — arithmetic, a master record, a business rule, a second document — before it is trusted. Extraction without validation is just a faster way to get bad data into your systems.
Structured means the output is typed fields with known names, not text. An invoice becomes a record with a supplier ID, an invoice number, a date, a currency, a tax amount, and a set of line items, each with its own quantity, unit price, and general ledger coding.
An intelligent document processing platform is therefore not a reading tool. It is a translation layer between the document world and the record world.
The Five-Stage Pipeline
Nearly every IDP implementation, regardless of vendor, moves documents through the same five stages. Knowing them makes it much easier to diagnose where a deployment is underperforming.
Stage 1: Ingest
Documents enter from monitored mailboxes, scanner output folders, supplier portals, mobile captures, API uploads, or shared drives. The ingest stage is responsible for more than collection. It de-duplicates, so a supplier who resends the same invoice does not create a second liability. It splits multi-document PDFs at the correct page boundaries, which matters enormously when a scanning clerk feeds forty pages containing eleven separate documents. And it normalizes the file — deskewing, correcting rotation and contrast, converting formats — so downstream stages see something consistent.
Ingest is the stage most often underspecified and most often the source of production surprises. The volume of genuinely bad input, from photographs taken at an angle to faxes of faxes, is always higher than the pilot suggested.
Stage 2: Classify
Before anything can be extracted, the system must know what it is looking at. Classification assigns a document type: invoice, credit note, purchase order, delivery note, signed agreement, bank statement, identity document, tax certificate.
This matters because extraction models are type-specific. The field set for a credit note is not the field set for an invoice, and treating one as the other produces a plausible-looking record with the sign inverted. Good classifiers also detect the unknown case and route it rather than forcing a bad match — a document type the system has never seen should land in a review queue, not be scored as a 51% invoice.
Stage 3: Extract
The stage everyone thinks of as IDP. Modern extraction locates fields by meaning and context rather than by fixed coordinates, which is why it works on a new supplier's layout without a configuration project.
Extraction operates at two levels. Header extraction pulls the document-level fields — number, date, total, currency, counterparty, payment terms. Line-level extraction pulls the repeating table rows, which is substantially harder because tables break across pages, contain merged cells, include subtotals interleaved with items, and vary wildly in column ordering. Many deployments that report good accuracy on headers are quietly struggling at line level, which is where the value in matching and coding actually lives.
Extraction also handles unstructured content: clause identification in contracts, obligation dates in notices, named entities in correspondence. Here the output is less a field than a located assertion with a page and paragraph reference.
Stage 4: Validate
Extraction produces candidate values. Validation decides whether to believe them.
Four validation types do most of the work. Arithmetic checks confirm that line items sum to the subtotal and that tax and total reconcile. Format checks confirm dates parse, tax IDs match jurisdictional patterns, and IBANs pass checksum. Master data lookups confirm the supplier exists, the bank details match those on file rather than a recently changed set, and the cost center is active. Cross-document checks confirm the document agrees with its counterpart — the purchase order, the delivery note, the contract.
Validation is also where duplicate detection belongs at the semantic level: the same invoice number from the same supplier for the same amount is a duplicate even if the file bytes differ.
Stage 5: Export
The final stage posts a clean record into the system that will act on it — an ERP, an accounting ledger, an HRIS, a case management system — along with a link back to the source document.
Export should be idempotent and should carry provenance. Every field in the posted record ought to be traceable to a coordinate on a page, because that is the trail an auditor will ask for. Retention rules attach here too, which is the point where document processing and HR document management compliance obligations converge: a processed document is still a record, and it still has a legally defined lifespan.
The Document Taxonomy That Predicts Difficulty
Not all documents are equally hard, and the taxonomy is the single best predictor of how a project will go.
Structured documents have a fixed layout that never varies — a tax form, an internal application, a standardized claim form. Field positions are stable. These are close to solved; even older template systems handled them.
Semi-structured documents contain the same information every time in a different arrangement every time. Invoices, purchase orders, delivery notes, bank statements, remittance advices. Every supplier's invoice has a total; no two put it in the same place. This is the category IDP exists for, it represents the bulk of business document volume, and it is why the earliest and clearest returns tend to appear in accounts payable automation.
Unstructured documents have no reliable field set at all — contracts, correspondence, legal notices, medical narratives. The task shifts from extraction to comprehension: find the termination clause, identify the notice period, determine whether an obligation was triggered. Accuracy here is judged differently, usually on recall of relevant passages rather than field-level correctness.
Confidence Thresholds and Straight-Through Processing
This is the concept that determines whether an IDP deployment saves labor or relocates it.
Every extracted field carries a confidence score. A threshold defines the level above which the system commits the value without review. Set it at 99% and almost everything routes to a human, which means you have bought a faster data entry tool. Set it at 70% and errors flow into the ledger unchallenged.
The right approach is not a single global number. Thresholds should vary by field consequence. Getting a supplier's bank account wrong is catastrophic and irreversible; getting a purchase order reference wrong is annoying and self-correcting at the matching stage. High-consequence fields deserve high thresholds and sometimes mandatory review regardless of score. Low-consequence fields can run loose.
The metric that summarizes all of this is the straight-through processing rate: the share of documents that complete the full pipeline without any human touch. A mature deployment on semi-structured documents typically lands somewhere between 70% and 90%, with the remainder split between genuinely ambiguous documents and low-confidence fields on otherwise clean ones. A stalled deployment sits far lower, and the diagnosis is almost always at the ingest or validation stage rather than the extraction model everyone blames.
One more distinction is worth holding onto. Field accuracy — the percentage of individual fields extracted correctly — always looks better than document accuracy, the percentage of documents with every field correct. A model at 97% field accuracy across twenty fields per document produces a fully correct document only about half the time. When a vendor quotes accuracy, ask which one they mean.
If you are evaluating whether document processing belongs in a specific workflow, start by sampling a hundred real documents from that workflow and sorting them into the three structural categories above. That single exercise tells you more about feasibility, cost, and realistic straight-through rates than any product demonstration will.
