Processing Pipeline Overview

DocAssure uses a sophisticated multi-stage processing pipeline that combines traditional OCR, AI vision analysis, and mathematical verification to ensure the highest accuracy for extracted data.

Processing Stages

1. Upload

MinIO Storage

2. OCR

Python Tesseract

3. Doc Intelligence

Advanced Vision

4. Verification

4-Level System

5. Index

Search Engine

Stage 1: Document Upload

Documents can be uploaded through multiple channels, all of which use the same processing pipeline:

Web Interface - Manual upload through the Documents page
Watch Folders - Automatic processing from monitored directories
API - Programmatic upload from external systems
Bulk Upload - ZIP files with multiple documents

Supported Formats

PDF, DOCX, DOC, XLSX, XLS, PPTX, PPT, TXT, CSV, ZIP (50MB limit per file)

Stage 2: OCR Text Extraction

The Python processor extracts text using Tesseract OCR with quality detection:

Standard OCR

Tesseract 5.x with LSTM engine
Multi-language support
Image preprocessing (deskew, denoise)
Searchable PDF generation

Quality Detection

Confidence scoring (0-100%)
Character confusion detection (O/0, 1/I/l)
Fragmented text detection
Auto-trigger Vision OCR if quality < 40%

Stage 3: Document Intelligence

Every page is analyzed by AI vision models to detect and extract visual content:

Visual Content Detection

Tables

Financial tables, data grids, comparison matrices

Charts & Graphs

Bar charts, pie charts, line graphs, trend analysis

Diagrams

Org charts, flowcharts, process diagrams, structures

Financial Statement Detection

When financial statements are detected (balance sheets, income statements, cash flow), the system triggers structured extraction with IFRS taxonomy mapping:

Statement Type Detection - Automatically identifies balance sheet, income statement, cash flow, or equity changes
Multi-Year Column Detection - Identifies columns like "2024", "2023 Restated", ensures extraction from current year only
IFRS Taxonomy Mapping - Maps extracted labels to standard IFRS elements (ifrs-full:Assets, ifrs-full:Equity, etc.)
Currency Detection - Identifies reporting currency and scale (millions, billions)

Stage 4: Four-Level Multi-Level Verification

Financial data undergoes rigorous verification through four independent levels to ensure accuracy:

Level 1: Dual-Pass Vision Extraction

Values are extracted twice using different image crops and prompts:

Smart table region detection using row/column density analysis
Two independent extraction passes
Fuzzy label matching with keyword extraction
Values must match within 2% tolerance

Level 2: OCR-Vision Cross-Reference

Vision-extracted values are verified against OCR text:

Extract all numbers from OCR text (13,000+ typically)
Match vision values to OCR numbers
Exact match, close match (within tolerance), or not found
Fusion confidence score calculated

Level 3: Confidence Thresholds

Each extracted value must meet minimum confidence:

Model confidence from vision extraction
OCR fusion confidence bonus
Dual-pass agreement bonus
Minimum 60% threshold for inclusion

Level 4: Equation Verification

Financial equations are mathematically verified:

Balance Sheet Equations:

Assets = Liabilities + Equity (fundamental equation)
Assets = Current Assets + Non-current Assets
Liabilities = Current + Non-current Liabilities
Equity and Liabilities = Assets

Income Statement Equations:

Gross Profit = Revenue - Cost of Sales
Operating Profit = Gross Profit - Operating Expenses
Net Income = Revenue - Total Expenses - Tax

Verification Certification Levels

VERIFIED

All 4 levels pass, equations balance

HIGH

3+ levels pass, minor imbalances

MEDIUM

2 levels pass, some verification

LOW

Insufficient verification

Automatic Corrections

The system automatically detects and corrects common extraction errors:

Balance Sheet Misclassification

Detects when "Total equity and liabilities" is incorrectly labeled as "Total liabilities":

Pattern: Liabilities ≈ Assets (should be L + E = A)
Auto-reclassifies to EquityAndLiabilities
Derives correct Liabilities = E&L - Equity

Multi-Year Column Mixing

Prevents mixing values from different reporting periods:

Detects column headers (2024, 2023 Restated, etc.)
Validates all values from same column
Warns if values mixed from different years

Data Reliability Levels in AI Responses

When you ask questions through the AI Consulting feature, responses indicate data reliability:

VERIFIED FINANCIAL DATA

IFRS-mapped, equation-verified - HIGHEST reliability

VISUAL ELEMENTS

Document Intelligence extracted from images - HIGH reliability

DOCUMENT TEXT

OCR/extracted text - MODERATE reliability

File Status Indicators

Every file shows a small coloured dot so you can tell, at a glance, whether it has finished going through the pipeline above and is ready for AI chat and search. This matters most after a cloud import, which uploads the whole batch first and only then runs each file through OCR and indexing — you'll see the dots start amber and settle to green as processing completes.

Ready

Fully ingested — text has been extracted and indexed into the search store, so AI chat and search work on this file. Green means AI-ready, not merely uploaded.

Processing

Uploaded and moving through OCR and indexing. AI answers scoped to this file are held back until it finishes (usually a few minutes), so you never get an answer from a document the system hasn't read yet.

Needs attention

Processing failed, or the file couldn't be read after several automatic retries (for example an image-only scan with no extractable text). Re-upload a clearer copy, or contact support if it persists.

No indicator

Folders and non-document files (video, audio, archives) don't go through document ingestion, so they show no status dot.

The list refreshes on its own while any file is still processing, so the dots update without you having to pull to refresh.

Document Processing & AI Verification