Case study · Demo
AI invoice checker demo: 6 of 6 planted errors caught, 0 false alarms
A working demo of our Microsoft 365 document flow. It reads invoice PDFs, checks the numbers, and sends every doubtful invoice to a person before anything lands in Excel as approved.
This is a demo on synthetic invoices, not a client project. We generated all 13 invoices ourselves. The vendors are Microsoft sample names (Contoso, Fabrikam, Northwind, Tailspin, Woodgrove, Adventure Works). The IBANs pass the checksum but belong to nobody. Results on real invoices need a sample of real documents and their own accuracy report.
The test batch
- 13 invoice PDFs in 4 layouts.
- 6 invoices carry one planted error each. 7 are clean.
- One clean invoice (inv-13) uses a layout the rule-based reader has never seen, on purpose.
How the checker works
- It reads the text layer of each PDF.
- Two readers extract the fields: an LLM with one JSON-only prompt, and a rule set over known labels.
- 11 business checks run: line math, lines add up to the subtotal, VAT equals rate times subtotal, subtotal plus VAT equals total, VAT ID format per country, a legal VAT rate for that country, IBAN mod-97, dates parse, due date after issue date, no issue date in the future, and no duplicate vendor plus invoice number.
- Each field gets a confidence score. Both readers agree: high. They disagree, or a check fails on that field: low.
- Only invoices with no failed check and high confidence on every required field are approved without a person. Everything else goes to a review queue in Excel.
- The reviewer approves, corrects, or rejects each row. A correction goes through every check again before it is accepted.
Results
| Mode | Fields correct | Planted errors caught | False alarms | Clean, auto-approved |
|---|---|---|---|---|
| Rules only | 94.9% | 6 / 6 | 1 | 2 / 7 |
| LLM only | 100% | 6 / 6 | 0 | 0 / 7 (by design) |
| LLM + rules | 100% | 6 / 6 | 0 | 6 / 7 |
The 7th clean invoice is inv-13. Only the LLM could read its layout, so one reader alone did not count as enough, and it went to a person. That is the intended behavior: one unverified read never approves an invoice on its own.
Why each planted error stopped
| File | Check | Why it stopped |
|---|---|---|
| inv-07 | Total mismatch | 1,890.00 + 378.00 = 2,268.00, invoice says 2,278.00 |
| inv-08 | VAT ID format | NL85374629B01 has 8 digits, needs 9 |
| inv-09 | IBAN checksum | Fails the mod-97 check |
| inv-10 | Dates | Due date is before the issue date |
| inv-11 | VAT rate | 17% is not a German VAT rate |
| inv-12 | Duplicate | Same vendor and invoice number as inv-03 |
What this demo does not prove
- The rule-based reader was written for these layouts. Its accuracy does not carry over to your invoices.
- Scanned PDFs need OCR first. In Microsoft 365, AI Builder or Azure Document Intelligence does that step.
- The demo used a free hosted model. A client build uses your own model endpoint (Azure OpenAI or AI Builder), so documents stay in your tenant.
- The Power Automate flow definition for this pipeline is written but has not yet run in a client tenant. Expect small fixes on the first import.
What a client build adds
The fixed-price Microsoft 365 document flow runs this logic in your tenant: a SharePoint or Outlook trigger, approval in Teams and Outlook, rows in Excel Online, and error alerts to your admin. Before you pay the second half, we run it on 30 of your real documents and give you the accuracy report.
Want this on your own invoices?
Book a scoping call. We agree the fields, the checks, and the acceptance tests in writing. $2,000, 5 working days.