diff --git a/pageindex/flash/README.md b/pageindex/flash/README.md index 675c2acec..762b870dd 100644 --- a/pageindex/flash/README.md +++ b/pageindex/flash/README.md @@ -38,6 +38,37 @@ missing, non-PDF, encrypted, empty, or unreadable file. Page indexes are 1-based. `nodes` nests the same shape recursively. +## Benchmark + +Nine PDFs, each run end to end with tree optimization: PDF parse, layout +outline, merge, LLM expand, then a summary for every node. + +![Time against document length](assets/time_vs_pages.png) + +![Cost against document length](assets/cost_vs_pages.png) + +Both scale close to linearly with length, at 218 s and $0.85 per 1,000 pages. +Two qualifiers. Cost follows node count a little more closely than page count, +$0.0007 to $0.0016 per node, so a densely structured document costs more than +its length suggests. And wall clock flattens past roughly 700 pages, where +summary concurrency rather than length becomes the limit. + +| Document | Pages | Input tokens | Output tokens | +|---|---:|---:|---:| +| Bitcoin whitepaper | 9 | 8,715 | 4,673 | +| Attention Is All You Need | 15 | 26,805 | 10,183 | +| KIMI K3 | 47 | 85,704 | 35,217 | +| DeepSeek-R1 | 86 | 68,398 | 26,351 | +| Situational Awareness | 165 | 115,130 | 54,347 | +| Federal Reserve 2023 report | 222 | 280,975 | 136,982 | +| 9/11 Commission Report | 585 | 720,624 | 200,202 | +| Pattern Recognition and Machine Learning | 758 | 857,983 | 277,675 | +| Machine Learning: A Probabilistic Perspective | 1,098 | 1,587,265 | 646,958 | +| **Total** | **2,985** | **3,751,599** | **1,392,588** | + +Measured with `gpt-5.6-luna` at $0.20 / $1.20 per million input / output tokens, +priced cold with no prompt-cache discount. + ## Limits - Scanned PDFs without embedded text are not supported. diff --git a/pageindex/flash/assets/cost_vs_pages.png b/pageindex/flash/assets/cost_vs_pages.png new file mode 100644 index 000000000..177139de2 Binary files /dev/null and b/pageindex/flash/assets/cost_vs_pages.png differ diff --git a/pageindex/flash/assets/time_vs_pages.png b/pageindex/flash/assets/time_vs_pages.png new file mode 100644 index 000000000..b2b7d27ab Binary files /dev/null and b/pageindex/flash/assets/time_vs_pages.png differ