Building Document Q&A with Any LLM: File Handling, Prompting, and Guardrails

The Document Intelligence demo on this site lets an authenticated user upload a file and ask questions about it. It is small, but it illustrates the engineering choices that matter for any document-Q&A feature — and it is deliberately LLM-agnostic.

The model is a swappable component

The application talks to a thin "answer from document" interface. Behind it you can run a local model via Ollama (for example Llama 3) for privacy and zero per-token cost, or a cloud LLM for maximum capability. This deployment happens to use Google Gemini, but switching backends is a configuration change, not a rewrite.

// One interface, many backends
interface LLM { answer(question: string, doc: DocPart): Promise<string> }

const llm: LLM =
  process.env.LLM === "ollama" ? new OllamaLLM("llama3")
                               : new CloudLLM(process.env.MODEL!);

Handle files by type

PDFs are sent to the model natively as inline binary data — the model parses layout and text directly, so there is no brittle PDF-to-text step. Text-based files (txt, md, csv, json, log, xml, html, yaml) are read as UTF-8 and passed as text, truncated to a token-safe length. Each path plays to the model's strengths.

Prompt for grounding, not eloquence

The system instruction tells the model to answer only from the supplied document and to reply with a fixed phrase when the answer is absent. This single constraint is what prevents the assistant from confidently answering from general knowledge when the document is silent.

Guardrails are not optional

• Authentication (password + email OTP) and per-user authorization
• File-type and 10MB size limits with signature checks
• Rate limiting per user
• In-memory storage that expires after an hour
• The model credentials live only on the server, never in the browser

Why it matters

The same shape scales up: swap in-memory storage for a vector store, add reranking and citations, and you have an enterprise knowledge assistant. The discipline — typed file handling, grounded prompting, hard guardrails, and a swappable model — stays exactly the same.