feat: node-level text/summary versioning with deferred regeneration

examples/README.md

@@ -0,0 +1,31 @@
+# Incremental Markdown Update Demo
+
+`incremental_update_demo.py` — index a Markdown doc, then incrementally update
+it so only changed sections are re-summarized.
+
+Set an API key first (e.g. `export OPENAI_API_KEY=...`) and configure the model
+in `pageindex/config.yaml`.
+
+```bash
+python examples/incremental_update_demo.py
+```
+
+## How it works
+
+- `client.get_doc_id_by_path(path)` returns the `doc_id` already indexed for a
+  file path, or `None`.
+- First run for a path → `client.index(path)` builds the tree fresh.
+- Later runs → `client.update(doc_id)` re-summarizes **only** the sections whose
+  content hash changed; unchanged sections reuse their cached summary. No diff →
+  `{"status": "unchanged"}` (zero LLM work).
+
+Re-indexing the same file path reuses its `doc_id` and overwrites the same
+workspace JSON instead of creating a duplicate document.
+
+The script copies the sample (`documents/sample.md`) into a stable workspace
+path so re-runs reuse the same `doc_id`.
+
+## Workspace
+
+Indexed documents persist under `examples/workspace/` as `<doc_id>.json` plus a
+`_meta.json` index. Generated files there are throwaway test artifacts.

examples/documents/sample.md

@@ -0,0 +1,102 @@
+# PageIndex Overview
+
+PageIndex turns long documents into a navigable tree of sections, each with a
+summary, so agents can reason over structure instead of flat chunks. This
+sample document is used by the incremental update demo, and its sections are
+deliberately long enough that section summaries are generated by a model rather
+than passed through as raw text, which is what makes the incremental behaviour
+observable when only part of the document is edited.
+
+## 1. What PageIndex Does
+
+PageIndex parses a PDF or Markdown file into a hierarchical structure of nodes.
+Each node holds a title, its own text span, and a generated summary. The tree
+lets a retrieval agent walk from the document root down to the exact section
+that answers a question, without embedding every chunk into a vector store and
+without relying on nearest-neighbour similarity to decide relevance.
+
+The practical consequence is that retrieval becomes an act of navigation rather
+than an act of matching. An agent reads the root description, decides which
+branch is plausible, reads that branch's summary, and descends. At every step
+the decision is legible: there is a title and a summary that explain why the
+branch was taken. When the agent lands on a leaf it has the full text of that
+section, not a windowed fragment that may have been cut mid-argument.
+
+This matters most for documents where meaning depends on position. A clause in
+a contract, a subsection of a policy manual, or a numbered requirement in a
+regulatory filing all derive part of their meaning from where they sit in the
+document. Flat chunking discards that placement. A tree preserves it, and the
+path from root to leaf is itself a piece of evidence the agent can cite.
+
+## 2. Indexing
+
+Indexing builds the tree once. For Markdown, headings define the hierarchy
+directly: each heading opens a node, and the heading level determines where
+that node attaches to its parent. For PDFs, the table of contents and the page
+layout are used instead, with a series of checks that verify the extracted
+table of contents actually corresponds to the physical pages of the document.
+
+Every section is then summarized, and the whole document is given a short
+description derived from the structure. Summarization is conditional: a section
+whose text falls below a token threshold is stored verbatim, on the grounds
+that a summary of a short passage costs a model call and returns something no
+more useful than the passage itself. Longer sections are sent to the model.
+
+The result is persisted in a workspace directory as JSON, keyed by a document
+identifier. Alongside the tree, the record stores a hash of the whole file and
+a map of per-section hashes. Those hashes are what make the next run cheap:
+they are the record of what the tree was built from, so a later run can compare
+against them instead of re-deriving the tree from scratch to find out whether
+anything moved.
+
+## 3. Incremental Update
+
+When a document changes, PageIndex avoids rebuilding everything. The update
+path applies two gates in sequence, and each gate that passes eliminates a
+larger amount of work than the one before it.
+
+The first gate is the file hash. If the hash of the file's current contents
+matches the hash recorded at index time, nothing in the document has changed,
+the update returns immediately with a status of unchanged, and no model call is
+made at all. This is the common case for a scheduled re-ingest over a corpus
+where most documents are static between runs.
+
+The second gate is the section diff. The file is re-parsed into sections, each
+section is hashed, and the new hash map is compared against the stored one.
+That comparison yields three sets: sections that are new, sections that were
+removed, and sections whose text changed in place. The changed and added
+sections are marked dirty, and the ancestors of each dirty section are added to
+the set as well, on the assumption that a parent's roll-up may be affected by
+what happened underneath it.
+
+Everything in that set is re-summarized. Everything outside it reuses the
+summary already stored on the previous tree. A two-page revision to a five
+hundred page manual therefore costs a handful of model calls rather than a
+full rebuild, and the cost scales with the size of the edit rather than with
+the size of the document.
+
+## 4. Vectorless Retrieval
+
+Because the tree carries summaries at every level, an agent can retrieve by
+traversing the structure instead of doing nearest-neighbour search over
+embeddings. There is no index to build beyond the tree itself, no embedding
+model to keep consistent between ingest and query time, and no drift when the
+embedding model is upgraded underneath a corpus that was embedded with an
+older version.
+
+It also keeps retrieval explainable. A vector search returns a ranked list with
+a similarity score, and the score is not an explanation: it does not say why
+one passage outranked another, and it cannot be audited after the fact. A
+traversal returns a path, and the path is an explanation: this section, inside
+this chapter, inside this document, chosen because its summary matched what was
+asked. For compliance and audit workflows that difference is the point.
+
+## Appendix: Key Methods
+
+`client.index(path)` builds the tree for a document that has not been seen
+before and returns its document identifier. `client.update(doc_id)` refreshes
+an existing tree incrementally, applying the two gates described above and
+returning a dictionary describing which sections were updated, added, or
+deleted. `client.get_doc_id_by_path(path)` resolves an existing document by its
+source path, so that re-ingesting the same file finds the tree that already
+exists instead of minting a second identifier and orphaning the first.

examples/incremental_update_demo.py

@@ -0,0 +1,59 @@
+"""
+Incremental Markdown Update with PageIndex - Demo
+
+Shows how PageIndexClient resolves a document by file path: the first run
+indexes it fresh; later runs find the same doc_id and call update(), which
+re-summarizes only the sections whose content changed.
+
+Flow:
+  - First run for a path → index() builds the tree fresh.
+  - Later runs → same doc_id is found, update() runs; with no content change
+    it reports "unchanged" (zero LLM work).
+
+The source document (documents/sample.md) is copied into the workspace under a
+stable path, so re-running the demo reuses the same doc_id. An API key is
+required to generate section summaries.
+
+Run:
+  python examples/incremental_update_demo.py
+"""
+import shutil
+from pathlib import Path
+
+from pageindex import PageIndexClient
+
+SOURCE_MD = Path(__file__).parent / "documents" / "sample.md"
+
+
+def ingest_or_update(client, doc_path):
+    """Index the doc if new, otherwise incrementally update it."""
+    doc_id = client.get_doc_id_by_path(str(doc_path))
+    if doc_id:
+        result = client.update(doc_id)
+        if result.get("status") == "unchanged":
+            print(f"\n[{doc_path.name}] Loaded from cache (unchanged): {doc_id}")
+        else:
+            print(f"\n[{doc_path.name}] Incremental update done: {result}")
+    else:
+        doc_id = client.index(str(doc_path))
+        print(f"\n[{doc_path.name}] Indexed fresh. doc_id: {doc_id}")
+    return doc_id
+
+
+def main():
+    workspace = Path(__file__).parent / "workspace"
+    client = PageIndexClient(workspace=str(workspace))
+
+    # Stable copy inside the workspace so re-runs reuse the same doc_id.
+    workspace.mkdir(parents=True, exist_ok=True)
+    md_path = workspace / SOURCE_MD.name
+    shutil.copy(SOURCE_MD, md_path)  
+
+    print("== Ingest or update ==")
+    doc_id = ingest_or_update(client, md_path)
+
+
+
+
+if __name__ == "__main__":
+    main()