General-purpose document AI products are designed for the documents that most businesses deal with: invoices, contracts, receipts, medical forms, financial statements. These documents have characteristics that make them tractable: they are usually generated digitally, they use consistent fonts, they follow well-known layouts, and they are scanned (if at all) under controlled conditions with decent equipment.
Government permit and violation records violate most of these assumptions. They were designed for human clerks, not for machines. They were printed and handwritten by many different people over many decades. They were scanned as an afterthought, often to preserve documents rather than to make them queryable. A pipeline designed for invoices will produce poor results on a county permit archive. Understanding why, and what the alternative design looks like, is the subject of this article.
The template explosion problem
A county permit archive does not contain one document format. It contains the accumulated output of a planning department that may have used different application forms for different permit types, updated those forms at each building code cycle, and sometimes used different forms for different geographic districts or zoning categories. A county with 30 years of records may have eight or ten distinct permit application templates, each with slightly different field layouts, field names, and form proportions.
General-purpose document AI handles this by training on large volumes of diverse documents and learning to extract fields by their semantic content rather than their position. This works reasonably well for documents where the content is syntactically distinct: an invoice total looks different from a line item description, and the AI can learn to distinguish them. It works poorly for government forms where multiple templates use the same field labels but in different positions. "Parcel APN:" appears on every permit form. Its position on the page is what distinguishes one form template from another. A model that does not know which template it is looking at cannot reliably locate the APN field.
The design decision that addresses this is template classification as a first step, before field extraction. The pipeline identifies which template generation a document belongs to (based on visual features: page aspect ratio, presence or absence of specific structural elements, column layout) and then applies the field extraction schema for that specific template. This is fundamentally different from a single-model approach that treats all documents the same way. It requires investing in template mapping for each county's archive before processing begins, but it pays off in extraction accuracy on documents that differ structurally from the majority.
Mixed handwritten and printed content
County permit forms are printed templates with handwritten fill-ins. The template text (field labels, section headings, instructions) is printed. The field values (parcel APN, applicant name, permit type, dates, inspector comments) are handwritten or typed on a typewriter, sometimes in ink, sometimes in pencil, sometimes in multiple ink colors if the form was updated by multiple people.
Standard OCR engines produce different accuracy on printed text versus handwritten text. Printed text OCR accuracy on a clear scan is high. Handwritten OCR accuracy depends heavily on legibility and varies substantially between clerks and eras. A permit form from the 1990s filled out in careful block capitals by a clerk who followed a consistent style will produce better handwritten OCR results than one from the 1980s filled out with a running hand in faded ink.
The design decision here involves separate OCR handling for printed regions versus handwritten regions. Layout analysis identifies whether a form cell is likely to contain printed text, typed text, or handwritten text, and routes it to the appropriate OCR path or to a confidence threshold that reflects the expected accuracy for that content type. A handwritten APN field that is read with 65% character-level confidence should be flagged differently than a printed field label read with 98% confidence. Treating both the same way produces misleadingly uniform confidence scores that mask where the real extraction risk sits.
Multi-page records with split fields
A building permit application is typically more than one page. It may include a cover sheet with the permit number and parcel information, an application detail sheet with project description and applicant information, an attached plan review checklist, and a fee calculation sheet. The permit number appears on the cover sheet. The project description appears on the application detail. The inspector's comments appear on the plan review checklist. All of these belong to the same permit record and need to be extracted as a unit.
This is a document bundling problem, and it is distinct from the field extraction problem. Before any field can be extracted, the pipeline needs to know which pages belong together as a single permit record, and which pages are the start of a new record. In a well-organized archive, this might be signaled by a physical separator or by a consistent page numbering scheme. In practice, county archives were scanned in batch, with page separators present only for some eras and some document types. A permit file that was originally a stapled set of pages may have been scanned with the pages in order but no separator indicating where one permit file ends and the next begins.
Boundary detection between permit records is one of the harder problems in government document pipeline design. The approaches include visual boundary detection (detecting page layout changes that indicate a new document type), APN-based grouping (pages with the same APN are grouped together, pages with a new APN start a new record), and document structure detection (identifying pages that look like cover sheets versus continuation sheets). Each approach has failure modes. APN-based grouping fails when a parcel has multiple permits in the archive and the pages are interleaved. Visual boundary detection fails when two successive permit applications used the same form template. In practice, a combination of approaches with human review for uncertain bundling decisions is more reliable than any single method.
Confidence scoring: what it measures and what it does not
Every OCR engine produces a confidence score per character or per word. These scores reflect the engine's internal estimate of how likely its transcription is to match the source text. High confidence means the character image was clear and the engine's top hypothesis matches well. Low confidence means either that the image is ambiguous, or that the top hypothesis is not a strong match for any familiar character pattern.
What confidence scores do not measure is whether the extracted field value is semantically correct for its context. An APN field might be read with high character-level confidence but produce a value that does not match any known parcel APN format. A date field might be read clearly as "13/04/1997," which is a valid ISO date in day-month-year format but invalid if the county used month-day-year convention. These are not OCR errors. They are interpretation errors that require additional validation beyond the raw OCR confidence.
A well-designed extraction pipeline adds a validation layer after OCR confidence scoring. For each field type, there are validity rules: APN format must match the county's known format pattern; dates must parse as valid calendar dates; permit type codes must fall within a known vocabulary. Fields that pass OCR confidence thresholds but fail validation rules are flagged separately from fields that fail OCR confidence. The two categories of failure require different human review actions: a low-OCR-confidence field needs the human to look at the original image and re-transcribe; a validation failure may be a transcription error or may be a legitimate unusual value, and the human needs to understand which.
Human review queue design
Any extraction pipeline for government documents will produce records that require human review before they are published to the output layer. The question is not whether to have a review queue, but how to structure it so that reviewers can work efficiently and so that the review decisions feed back into improving the pipeline.
Review queue design involves two main decisions: what to route to review, and how to present the review task. For routing, we use a combination of OCR confidence thresholds, validation failures, and field-level importance. A low-confidence permit number field routes to review because permit number is a primary key for record identity. A low-confidence field that carries secondary information routes to review only if confidence is very low, because the cost of routing everything is that reviewers spend time on records where the extraction is close enough to be acceptable.
For review presentation, the reviewer needs to see the original document image side by side with the extracted field values, with the extracted values highlighted in their source position on the document image. This spatial alignment between the extraction result and the source image is what allows a reviewer to correct a misread quickly. A reviewer looking at an extracted APN of "07O43OO21O" (OCR misread of "0" as "O") can immediately see the error when the source image shows the field clearly. Without the spatial alignment, the reviewer has to manually locate the field in the document, which slows review substantially and introduces additional errors.
What general-purpose document AI gets right and where it breaks down
We want to be direct about this: general-purpose document AI is genuinely capable on certain government document tasks, and we use it for components where it performs well. Classification of document types (is this a permit, a violation notice, or an inspection report?) is a task where models trained on diverse document types perform acceptably even without county-specific training. Initial field extraction on high-quality scans of common form templates can produce usable results as a first pass that reduces human review volume.
Where general-purpose models break down is on the combination of challenges that define the hardest government document cases: low-resolution scans, unusual form templates with no close analogues in training data, multi-page bundled records with ambiguous boundaries, and handwritten fields in degraded condition. These are not edge cases in a county permit archive. They are common. A pipeline designed for the median case of the general-purpose model will fail on a predictable and substantial fraction of real government document archives.
The design decisions described here, template classification, separate handling for handwritten content, multi-page bundling, field-level validation, and structured human review, are the components that close the gap between what general-purpose models produce and what a county GIS layer actually needs. None of these decisions are exotic. They are the result of working directly with scanned government archives and iterating on what breaks. The architecture is specific to the problem, not to a particular AI technique.