Most PDF question-answering systems fail when documents contain tables, headers, and visual structure because they flatten everything into plain text, losing critical context. A new approach combining layout-aware parsing with multimodal retrieval shows how separating parsing, retrieval, and reasoning tasks can dramatically improve accuracy on complex documents like medication factsheets and medical records .

Why Vision Language Models Alone Aren't Enough for PDFs?

At first glance, using a vision language model (VLM) like GPT-4V or Gemini Vision to read PDFs seems like the obvious solution. These models can see and understand images, so why not just convert each page to an image and let the VLM handle it? The problem is that VLMs treat every page as a reasoning challenge rather than a parsing problem .

VLM inference is expensive per page, degrades significantly over long documents, and struggles with systematic evaluation. A medication side-effects factsheet illustrates the core issue: when the word "depression" appears both as a reason for prescribing a drug and as a side effect of another, a VLM might focus on the wrong section of the page during retrieval. The model has to solve the parsing problem and the reasoning problem simultaneously, which wastes computational resources and introduces errors .

The traditional text-extraction pipeline has its own problems. Extract text, chunk it, embed it, retrieve relevant passages. For simple lookups, this works. But once a question depends on document structure or multimodal context, the system breaks down. When a parser flattens a table into a text stream, column identity is lost, and no amount of prompt engineering recovers it .

How Can You Build a Better PDF Question-Answering System?

The solution is to separate three distinct concerns that most systems conflate: parsing (extracting structured content), retrieval (surfacing relevant chunks), and reasoning (synthesizing answers). LlamaIndex's LiteParse framework, released in March 2026, demonstrates this approach by using layout-aware parsing that preserves document structure without relying on expensive VLM calls .

• Selective OCR: Native PDF text extraction is the default path, with optical character recognition (OCR) only triggering on pages with no extractable text or garbled characters. Born-digital PDFs skip the OCR processing entirely, saving time and cost.
• Spatial text preservation: Instead of converting tables to Markdown (which breaks on merged cells and irregular grids), LiteParse projects extracted text onto a virtual character grid that preserves the visual layout of the original page, allowing language models to read spatially-formatted tables as they would in training data.
• Multimodal fallback: The system renders high-resolution page images alongside text extraction, enabling agents to escalate visually complex pages to a vision model only when necessary, without reprocessing the entire document.
• Local execution: No cloud dependencies or API keys required, making the approach suitable for teams handling sensitive documents or deploying in constrained environments.

This architecture produces two outputs per page: structured text with spatial metadata and a high-resolution screenshot. LanceDB, a multimodal database, stores both the text chunks, embedding vectors, and raw image bytes in a single table row . This matters because the image is versioned alongside the structured metadata in the same row, eliminating drift between what the retrieval layer returns and what the source document actually contains.

What Real-World Challenges Does This Solve?

A two-page medication factsheet from MedStar Visiting Nurse Association illustrates the practical value of this approach. The document maps 11 medication categories to their generic names, brand names, and common side effects across three columns. While structurally straightforward, it presents several challenges that break traditional systems :

• Synonym mismatch: The PDF uses terms like "Queasiness or Throwing Up" and "Helps With Inflammation," while users ask about "nausea" and "anti-inflammatory drugs." Exact term matching fails; the system needs semantic bridging between user language and document language.
• Column disambiguation: "Queasiness" appears both as a reason for medicine and as a side effect of pain relief drugs. "Throwing up" shows up in five separate side effect lists and one category name. The system must distinguish between these roles based on column position, not string matching alone.
• Near-duplicate categories: "Lowers Blood Pressure" and "Lowers Blood Pressure and Heart Rate" are distinct categories with different drugs and different side effects. A retrieval system that conflates them produces incorrect answers, particularly on negation questions like "Is headache a side effect of Losartan's category?"
• Category-level side effects: Side effects are listed per medication category, not per individual drug. The system must not fabricate per-drug distinctions that the source document does not make.

By separating parsing from reasoning, the agent receives a deterministic, structured substrate to reason on top of rather than through. This work is faster, cheaper, and reproducible compared to traditional OCR-plus-agent flows where the agent has to infer layout, repair extraction errors, and reconstruct structure on every query .

The hybrid approach also supports both vector search and hybrid search (vector plus full-text) out of the box, which matters for document question-answering where exact term matching and semantic similarity serve different query types. At query time, a single fetch can return the text chunk and its embedding for fast similarity search, or include the image bytes when the agent needs visual context, without orchestrating across multiple storage backends .

This architecture represents a meaningful shift in how teams should approach document intelligence. Rather than treating every PDF as a vision problem or a text problem, the most effective systems recognize that different parts of the pipeline require different tools. Parsing benefits from layout awareness. Retrieval benefits from multimodal storage. Reasoning benefits from clean, structured input. Separating these concerns unlocks better accuracy, lower costs, and more transparent debugging for production document question-answering systems.