diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 000000000..d2a44119c --- /dev/null +++ b/examples/README.md @@ -0,0 +1,31 @@ +# Incremental Markdown Update Demo + +`incremental_update_demo.py` — index a Markdown doc, then incrementally update +it so only changed sections are re-summarized. + +Set an API key first (e.g. `export OPENAI_API_KEY=...`) and configure the model +in `pageindex/config.yaml`. + +```bash +python examples/incremental_update_demo.py +``` + +## How it works + +- `client.get_doc_id_by_path(path)` returns the `doc_id` already indexed for a + file path, or `None`. +- First run for a path → `client.index(path)` builds the tree fresh. +- Later runs → `client.update(doc_id)` re-summarizes **only** the sections whose + content hash changed; unchanged sections reuse their cached summary. No diff → + `{"status": "unchanged"}` (zero LLM work). + +Re-indexing the same file path reuses its `doc_id` and overwrites the same +workspace JSON instead of creating a duplicate document. + +The script copies the sample (`documents/sample.md`) into a stable workspace +path so re-runs reuse the same `doc_id`. + +## Workspace + +Indexed documents persist under `examples/workspace/` as `.json` plus a +`_meta.json` index. Generated files there are throwaway test artifacts. diff --git a/examples/documents/sample.md b/examples/documents/sample.md new file mode 100644 index 000000000..75154a8c9 --- /dev/null +++ b/examples/documents/sample.md @@ -0,0 +1,102 @@ +# PageIndex Overview + +PageIndex turns long documents into a navigable tree of sections, each with a +summary, so agents can reason over structure instead of flat chunks. This +sample document is used by the incremental update demo, and its sections are +deliberately long enough that section summaries are generated by a model rather +than passed through as raw text, which is what makes the incremental behaviour +observable when only part of the document is edited. + +## 1. What PageIndex Does + +PageIndex parses a PDF or Markdown file into a hierarchical structure of nodes. +Each node holds a title, its own text span, and a generated summary. The tree +lets a retrieval agent walk from the document root down to the exact section +that answers a question, without embedding every chunk into a vector store and +without relying on nearest-neighbour similarity to decide relevance. + +The practical consequence is that retrieval becomes an act of navigation rather +than an act of matching. An agent reads the root description, decides which +branch is plausible, reads that branch's summary, and descends. At every step +the decision is legible: there is a title and a summary that explain why the +branch was taken. When the agent lands on a leaf it has the full text of that +section, not a windowed fragment that may have been cut mid-argument. + +This matters most for documents where meaning depends on position. A clause in +a contract, a subsection of a policy manual, or a numbered requirement in a +regulatory filing all derive part of their meaning from where they sit in the +document. Flat chunking discards that placement. A tree preserves it, and the +path from root to leaf is itself a piece of evidence the agent can cite. + +## 2. Indexing + +Indexing builds the tree once. For Markdown, headings define the hierarchy +directly: each heading opens a node, and the heading level determines where +that node attaches to its parent. For PDFs, the table of contents and the page +layout are used instead, with a series of checks that verify the extracted +table of contents actually corresponds to the physical pages of the document. + +Every section is then summarized, and the whole document is given a short +description derived from the structure. Summarization is conditional: a section +whose text falls below a token threshold is stored verbatim, on the grounds +that a summary of a short passage costs a model call and returns something no +more useful than the passage itself. Longer sections are sent to the model. + +The result is persisted in a workspace directory as JSON, keyed by a document +identifier. Alongside the tree, the record stores a hash of the whole file and +a map of per-section hashes. Those hashes are what make the next run cheap: +they are the record of what the tree was built from, so a later run can compare +against them instead of re-deriving the tree from scratch to find out whether +anything moved. + +## 3. Incremental Update + +When a document changes, PageIndex avoids rebuilding everything. The update +path applies two gates in sequence, and each gate that passes eliminates a +larger amount of work than the one before it. + +The first gate is the file hash. If the hash of the file's current contents +matches the hash recorded at index time, nothing in the document has changed, +the update returns immediately with a status of unchanged, and no model call is +made at all. This is the common case for a scheduled re-ingest over a corpus +where most documents are static between runs. + +The second gate is the section diff. The file is re-parsed into sections, each +section is hashed, and the new hash map is compared against the stored one. +That comparison yields three sets: sections that are new, sections that were +removed, and sections whose text changed in place. The changed and added +sections are marked dirty, and the ancestors of each dirty section are added to +the set as well, on the assumption that a parent's roll-up may be affected by +what happened underneath it. + +Everything in that set is re-summarized. Everything outside it reuses the +summary already stored on the previous tree. A two-page revision to a five +hundred page manual therefore costs a handful of model calls rather than a +full rebuild, and the cost scales with the size of the edit rather than with +the size of the document. + +## 4. Vectorless Retrieval + +Because the tree carries summaries at every level, an agent can retrieve by +traversing the structure instead of doing nearest-neighbour search over +embeddings. There is no index to build beyond the tree itself, no embedding +model to keep consistent between ingest and query time, and no drift when the +embedding model is upgraded underneath a corpus that was embedded with an +older version. + +It also keeps retrieval explainable. A vector search returns a ranked list with +a similarity score, and the score is not an explanation: it does not say why +one passage outranked another, and it cannot be audited after the fact. A +traversal returns a path, and the path is an explanation: this section, inside +this chapter, inside this document, chosen because its summary matched what was +asked. For compliance and audit workflows that difference is the point. + +## Appendix: Key Methods + +`client.index(path)` builds the tree for a document that has not been seen +before and returns its document identifier. `client.update(doc_id)` refreshes +an existing tree incrementally, applying the two gates described above and +returning a dictionary describing which sections were updated, added, or +deleted. `client.get_doc_id_by_path(path)` resolves an existing document by its +source path, so that re-ingesting the same file finds the tree that already +exists instead of minting a second identifier and orphaning the first. diff --git a/examples/incremental_update_demo.py b/examples/incremental_update_demo.py new file mode 100644 index 000000000..07751d009 --- /dev/null +++ b/examples/incremental_update_demo.py @@ -0,0 +1,59 @@ +""" +Incremental Markdown Update with PageIndex - Demo + +Shows how PageIndexClient resolves a document by file path: the first run +indexes it fresh; later runs find the same doc_id and call update(), which +re-summarizes only the sections whose content changed. + +Flow: + - First run for a path → index() builds the tree fresh. + - Later runs → same doc_id is found, update() runs; with no content change + it reports "unchanged" (zero LLM work). + +The source document (documents/sample.md) is copied into the workspace under a +stable path, so re-running the demo reuses the same doc_id. An API key is +required to generate section summaries. + +Run: + python examples/incremental_update_demo.py +""" +import shutil +from pathlib import Path + +from pageindex import PageIndexClient + +SOURCE_MD = Path(__file__).parent / "documents" / "sample.md" + + +def ingest_or_update(client, doc_path): + """Index the doc if new, otherwise incrementally update it.""" + doc_id = client.get_doc_id_by_path(str(doc_path)) + if doc_id: + result = client.update(doc_id) + if result.get("status") == "unchanged": + print(f"\n[{doc_path.name}] Loaded from cache (unchanged): {doc_id}") + else: + print(f"\n[{doc_path.name}] Incremental update done: {result}") + else: + doc_id = client.index(str(doc_path)) + print(f"\n[{doc_path.name}] Indexed fresh. doc_id: {doc_id}") + return doc_id + + +def main(): + workspace = Path(__file__).parent / "workspace" + client = PageIndexClient(workspace=str(workspace)) + + # Stable copy inside the workspace so re-runs reuse the same doc_id. + workspace.mkdir(parents=True, exist_ok=True) + md_path = workspace / SOURCE_MD.name + shutil.copy(SOURCE_MD, md_path) + + print("== Ingest or update ==") + doc_id = ingest_or_update(client, md_path) + + + + +if __name__ == "__main__": + main() diff --git a/pageindex/client.py b/pageindex/client.py index 894dab181..91f4d0dca 100644 --- a/pageindex/client.py +++ b/pageindex/client.py @@ -8,9 +8,24 @@ import PyPDF2 from .page_index import page_index -from .page_index_md import md_to_tree +from .page_index_md import ( + md_to_tree, + extract_nodes_from_markdown, + extract_node_text_content, + get_node_summary, + build_tree_from_nodes, + split_summary_fields, +) from .retrieve import get_document, get_document_structure, get_page_content -from .utils import ConfigLoader, remove_fields +from .utils import ( + ConfigLoader, + remove_fields, + hash_text, + compute_section_hashes, + walk_with_paths, + write_node_id, + format_structure, +) META_INDEX = "_meta.json" @@ -60,7 +75,9 @@ def index(self, file_path: str, mode: str = "auto") -> str: if not os.path.exists(file_path): raise FileNotFoundError(f"File not found: {file_path}") - doc_id = str(uuid.uuid4()) + # Re-indexing the same file path reuses its doc_id (overwrites in place) + # instead of creating a duplicate document/JSON. + doc_id = self.get_doc_id_by_path(file_path) or str(uuid.uuid4()) ext = os.path.splitext(file_path)[1].lower() is_pdf = ext == '.pdf' @@ -112,6 +129,33 @@ def index(self, file_path: str, mode: str = "auto") -> str: result = pool.submit(asyncio.run, coro).result() except RuntimeError: result = asyncio.run(coro) + # Compute hashes from the raw file to enable incremental update(). + _md_content = open(file_path, encoding='utf-8').read() + _node_list, _md_lines = extract_nodes_from_markdown(_md_content) + + _flat_nodes = extract_node_text_content(_node_list, _md_lines) + _new_hashes = compute_section_hashes(_flat_nodes) + # Re-indexing reuses the doc_id, so versions must carry forward: + # a reader holding version N must never see it go backwards. + if doc_id in self.documents and self.workspace: + self._ensure_doc_loaded(doc_id) + _prev = self.documents.get(doc_id, {}) + _old_versions = { + p: n.get('text_version', 1) + for p, n in walk_with_paths(_prev.get('structure') or []) + } + _old_hashes = _prev.get('section_hashes') or {} + for _p, _n in walk_with_paths(result['structure']): + _old_tv = _old_versions.get(_p) + if _old_tv is None: + _tv = 1 + elif _old_hashes.get(_p) != _new_hashes.get(_p): + _tv = _old_tv + 1 + else: + _tv = _old_tv + # index() regenerates every summary, so nothing is left stale. + _n['text_version'] = _tv + _n['summary_version'] = _tv self.documents[doc_id] = { 'id': doc_id, 'type': 'md', @@ -120,6 +164,8 @@ def index(self, file_path: str, mode: str = "auto") -> str: 'doc_description': result.get('doc_description', ''), 'line_count': result.get('line_count', 0), 'structure': result['structure'], + 'file_hash': hash_text(_md_content), + 'section_hashes': _new_hashes, } else: raise ValueError(f"Unsupported file format for: {file_path}") @@ -205,6 +251,14 @@ def _load_workspace(self): doc['path'] = str((self.workspace / doc['path']).resolve()) self.documents[doc_id] = doc + def get_doc_id_by_path(self, file_path: str) -> str | None: + """Return the doc_id already indexed for this file path, or None.""" + file_path = os.path.abspath(os.path.expanduser(file_path)) + return next( + (did for did, d in self.documents.items() if d.get('path') == file_path), + None, + ) + def _ensure_doc_loaded(self, doc_id: str): """Load full document JSON on demand (structure, pages, etc.).""" doc = self.documents.get(doc_id) @@ -216,15 +270,161 @@ def _ensure_doc_loaded(self, doc_id: str): doc['structure'] = full.get('structure', []) if full.get('pages'): doc['pages'] = full['pages'] + if full.get('section_hashes'): + doc['section_hashes'] = full['section_hashes'] + if full.get('file_hash'): + doc['file_hash'] = full['file_hash'] + + def update(self, doc_id: str) -> dict: + """Incrementally update an indexed MD document. + + Re-summarizes only sections whose own text changed (plus their + ancestors, whose roll-up may be affected); unchanged sections reuse + their cached summary. Returns a status dict describing the change set. + """ + self._ensure_doc_loaded(doc_id) + doc = self.documents.get(doc_id) + if not doc: + raise ValueError(f"Unknown doc_id: {doc_id}") + if doc.get('type') != 'md': + raise ValueError("update() only supports MD documents") + + file_path = doc['path'] + content = open(file_path, encoding='utf-8').read() + + # Gate 1: file-level hash — skip entirely if nothing changed. + new_file_hash = hash_text(content) + if new_file_hash == doc.get('file_hash'): + return {"status": "unchanged"} + + # Gate 2: section-level diff. + node_list, md_lines = extract_nodes_from_markdown(content) + new_nodes = extract_node_text_content(node_list, md_lines) + new_hashes = compute_section_hashes(new_nodes) + old_hashes = doc.get('section_hashes') or {} + + new_keys = set(new_hashes) + old_keys = set(old_hashes) + added = new_keys - old_keys + deleted = old_keys - new_keys + changed = {p for p in new_keys & old_keys if new_hashes[p] != old_hashes[p]} + + dirty = changed | added + + # Carry summaries and versions forward from the old tree. + old_by_path = dict(walk_with_paths(doc.get('structure', []))) + old_summary_map = { + p: n.get('summary') or n.get('prefix_summary', '') + for p, n in old_by_path.items() + } + + # No LLM work here. Bump text_version on dirty sections and leave + # summary_version behind, marking the summary stale; regeneration is + # deferred to read time (see _reconcile_summaries). Repeated updates + # between two reads therefore cost one regeneration, not one each. + for node in new_nodes: + path = node['title_path'] + old = old_by_path.get(path) + node['summary'] = old_summary_map.get(path, '') + if old is None: + # Newly added: no summary yet, so it starts out stale. + node['text_version'] = 1 + node['summary_version'] = 0 + else: + old_tv = old.get('text_version', 1) + node['text_version'] = old_tv + 1 if path in dirty else old_tv + node['summary_version'] = old.get('summary_version', old_tv) + + # Rebuild the tree with fresh node ids. + new_structure = build_tree_from_nodes(new_nodes) + split_summary_fields(new_structure) + write_node_id(new_structure) + new_structure = format_structure( + new_structure, + order=['title', 'node_id', 'line_num', 'text_version', 'summary_version', + 'summary', 'prefix_summary', 'text', 'nodes'], + ) + + doc['structure'] = new_structure + doc['file_hash'] = new_file_hash + doc['section_hashes'] = new_hashes + doc['line_count'] = content.count('\n') + 1 + + if self.workspace: + tmp = self.workspace / f"{doc_id}.tmp" + save_doc = dict(doc) + save_doc['structure'] = new_structure + with open(tmp, "w", encoding="utf-8") as f: + json.dump(save_doc, f, ensure_ascii=False, indent=2) + os.replace(tmp, self.workspace / f"{doc_id}.json") + self._save_meta(doc_id, self._make_meta_entry(doc)) + + return { + "status": "updated", + "updated": sorted(changed), + "added": sorted(added), + "deleted": sorted(deleted), + } def get_document(self, doc_id: str) -> str: """Return document metadata JSON.""" return get_document(self.documents, doc_id) + def _reconcile_summaries(self, doc_id: str) -> int: + """Regenerate summaries whose text has moved on, and return how many. + + A node is stale when summary_version != text_version. update() only + bumps text_version, so this is where deferred regeneration is paid -- + on the first read after an edit, batched across every stale node. + """ + if self.workspace: + self._ensure_doc_loaded(doc_id) + doc = self.documents.get(doc_id) + if not doc or doc.get('type') != 'md': + return 0 + + stale = [ + n for _, n in walk_with_paths(doc.get('structure', [])) + if n.get('summary_version') != n.get('text_version') + ] + if not stale: + return 0 + + async def _run(): + return await asyncio.gather(*( + get_node_summary(n, summary_token_threshold=200, model=self.model) + for n in stale + )) + + try: + asyncio.get_running_loop() + with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool: + summaries = pool.submit(asyncio.run, _run()).result() + except RuntimeError: + summaries = asyncio.run(_run()) + + for node, summary in zip(stale, summaries): + key = 'prefix_summary' if node.get('nodes') else 'summary' + node.pop('prefix_summary' if key == 'summary' else 'summary', None) + node[key] = summary + node['summary_version'] = node['text_version'] + + if self.workspace: + # _save_doc evicts structure from memory for lazy reload; pull it + # back so callers see the tree we just reconciled, not an empty one. + self._save_doc(doc_id) + self._ensure_doc_loaded(doc_id) + return len(stale) + def get_document_structure(self, doc_id: str) -> str: - """Return document tree structure JSON (without text fields).""" + """Return document tree structure JSON (without text fields). + + Stale summaries are regenerated first, so a reader never sees a + summary that describes text the document no longer contains. + """ if self.workspace: self._ensure_doc_loaded(doc_id) + self._reconcile_summaries(doc_id) return get_document_structure(self.documents, doc_id) def get_page_content(self, doc_id: str, pages: str) -> str: diff --git a/pageindex/page_index_md.py b/pageindex/page_index_md.py index 5a5971690..df7238848 100644 --- a/pageindex/page_index_md.py +++ b/pageindex/page_index_md.py @@ -29,6 +29,19 @@ async def generate_summaries_for_structure_md(structure, summary_token_threshold return structure +def split_summary_fields(tree_nodes): + """Apply the same leaf/parent convention as generate_summaries_for_structure_md. + + For callers that attach `summary` to a flat node list before the tree is + built, so parent nodes end up with `prefix_summary` as index() produces. + """ + for node in tree_nodes: + if node.get('nodes'): + node['prefix_summary'] = node.pop('summary', '') + split_summary_fields(node['nodes']) + return tree_nodes + + def extract_nodes_from_markdown(markdown_content): header_pattern = r'^(#{1,6})\s+(.+)$' code_block_pattern = r'^```' @@ -75,7 +88,19 @@ def extract_node_text_content(node_list, markdown_lines): 'level': len(header_match.group(1)) } all_nodes.append(processed_node) - + + # Build title_path per node using a level-keyed ancestor stack. + # Enables stable section identity across edits (incremental update). + ancestor_stack = {} + for node in all_nodes: + level = node['level'] + for l in list(ancestor_stack.keys()): + if l >= level: + del ancestor_stack[l] + parts = [ancestor_stack[l] for l in sorted(ancestor_stack)] + [node['title']] + node['title_path'] = ' > '.join(parts) + ancestor_stack[level] = node['title'] + for i, node in enumerate(all_nodes): start_line = node['line_num'] - 1 if i + 1 < len(all_nodes): @@ -205,6 +230,11 @@ def build_tree_from_nodes(node_list): 'line_num': node['line_num'], 'nodes': [] } + # Callers that summarize or version nodes before building the tree + # (e.g. incremental update) would otherwise have those fields dropped. + for field in ('summary', 'text_version', 'summary_version'): + if field in node: + tree_node[field] = node[field] node_counter += 1 while stack and stack[-1][1] >= current_level: diff --git a/pageindex/utils.py b/pageindex/utils.py index f00ccf3a7..224f7323d 100644 --- a/pageindex/utils.py +++ b/pageindex/utils.py @@ -5,6 +5,7 @@ from datetime import datetime import time import json +import hashlib import PyPDF2 import copy import asyncio @@ -708,3 +709,35 @@ def print_wrapped(text, width=100): for line in text.splitlines(): print(textwrap.fill(line, width=width)) + +# --------------------------------------------------------------------------- +# Incremental update helpers +# --------------------------------------------------------------------------- + +def hash_text(text: str) -> str: + return hashlib.sha256(text.encode()).hexdigest() + + +def compute_section_hashes(node_list: list) -> dict: + """Build {title_path: sha256_of_own_text} from a flat node list.""" + return {node["title_path"]: hash_text(node.get("text", "")) for node in node_list} + + +def walk_with_paths(nodes, prefix=""): + """Yield (title_path, node) for every node in a tree. + + The persisted tree stores no title_path; this reconstructs it with the + same ' > ' join used by extract_node_text_content, so tree nodes can be + matched against the flat node list and section_hashes keys. + """ + for node in nodes: + path = f"{prefix} > {node['title']}" if prefix else node["title"] + yield path, node + yield from walk_with_paths(node.get("nodes", []), path) + + +def find_ancestors(title_path: str) -> list: + """Return ancestor title paths from root to immediate parent.""" + parts = title_path.split(" > ") + return [" > ".join(parts[:i]) for i in range(1, len(parts))] + diff --git a/tests/test_incremental_update.py b/tests/test_incremental_update.py new file mode 100644 index 000000000..ef00233a1 --- /dev/null +++ b/tests/test_incremental_update.py @@ -0,0 +1,114 @@ +"""Tests for incremental MD update (section-hash diff). + +Covers the deterministic layer only — section identity, hashing, diff +classification and ancestor expansion — so it runs without an API key. + +Run: python tests/test_incremental_update.py +""" +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from pageindex.page_index_md import ( + extract_nodes_from_markdown, + extract_node_text_content, + build_tree_from_nodes, + split_summary_fields, +) +from pageindex.utils import compute_section_hashes, find_ancestors + + +def _hashes(md): + node_list, lines = extract_nodes_from_markdown(md) + nodes = extract_node_text_content(node_list, lines) + return compute_section_hashes(nodes) + + +def _diff(old, new): + old_k, new_k = set(old), set(new) + added = new_k - old_k + deleted = old_k - new_k + changed = {p for p in old_k & new_k if old[p] != new[p]} + return added, deleted, changed + + +def test_title_path_is_hierarchical(): + md = "# Root\nintro\n## A\nalpha\n### A1\nsub\n## B\nbeta\n" + node_list, lines = extract_nodes_from_markdown(md) + nodes = extract_node_text_content(node_list, lines) + paths = [n["title_path"] for n in nodes] + assert paths == ["Root", "Root > A", "Root > A > A1", "Root > B"], paths + + +def test_unchanged_doc_has_identical_hashes(): + md = "# Root\nintro\n## A\nalpha\n## B\nbeta\n" + assert _hashes(md) == _hashes(md) + + +def test_changed_added_deleted_classification(): + v1 = "# Root\nintro\n## A\nalpha\n## B\nbeta\n" + v2 = "# Root\nintro\n## A\nalpha CHANGED\n## C\ngamma\n" + added, deleted, changed = _diff(_hashes(v1), _hashes(v2)) + assert changed == {"Root > A"}, changed + assert added == {"Root > C"}, added + assert deleted == {"Root > B"}, deleted + + +def test_ancestors_expand_to_root(): + assert find_ancestors("Root > A > A1") == ["Root", "Root > A"] + assert find_ancestors("Root") == [] + + +def test_dirty_set_includes_ancestors(): + v1 = "# Root\nintro\n## A\nalpha\n### A1\nsub\n" + v2 = "# Root\nintro\n## A\nalpha\n### A1\nsub CHANGED\n" + _, _, changed = _diff(_hashes(v1), _hashes(v2)) + to_summarize = set(changed) + for p in changed: + to_summarize.update(find_ancestors(p)) + assert to_summarize == {"Root", "Root > A", "Root > A > A1"}, to_summarize + + +def test_build_tree_preserves_summaries(): + """update() attaches summaries before building the tree; they must survive.""" + md = "# Root\nintro\n## A\nalpha\n## B\nbeta\n" + node_list, lines = extract_nodes_from_markdown(md) + nodes = extract_node_text_content(node_list, lines) + for n in nodes: + n["summary"] = f"S:{n['title_path']}" + + tree = split_summary_fields(build_tree_from_nodes(nodes)) + root = tree[0] + # Parents carry prefix_summary, leaves carry summary — as index() produces. + assert root["prefix_summary"] == "S:Root", root + assert "summary" not in root, root + assert [c["summary"] for c in root["nodes"]] == ["S:Root > A", "S:Root > B"] + + +def test_tree_walk_paths_match_flat_title_paths(): + """The cached-summary lookup keys the old tree by walking it; those paths + must equal the title_paths the new flat node list is keyed by.""" + md = "# Root\nintro\n## A\nalpha\n### A1\nsub\n## B\nbeta\n" + node_list, lines = extract_nodes_from_markdown(md) + nodes = extract_node_text_content(node_list, lines) + tree = build_tree_from_nodes(nodes) + + walked = [] + + def collect(ns, prefix=""): + for n in ns: + path = f"{prefix} > {n['title']}" if prefix else n["title"] + walked.append(path) + collect(n.get("nodes", []), path) + + collect(tree) + assert walked == [n["title_path"] for n in nodes], walked + + +if __name__ == "__main__": + fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + for fn in fns: + fn() + print(f"ok {fn.__name__}") + print(f"\n{len(fns)} passed") diff --git a/tests/test_summary_versioning.py b/tests/test_summary_versioning.py new file mode 100644 index 000000000..7d1b41422 --- /dev/null +++ b/tests/test_summary_versioning.py @@ -0,0 +1,213 @@ +"""Tests for node-level text/summary versioning and deferred regeneration. + +The summarizer is stubbed, so these run without an API key and without +spending anything. Covers: version stamping, staleness detection, batching +across repeated updates, monotonicity across re-index, and the invariant +that a reconciled read leaves nothing stale. + +Run: python tests/test_summary_versioning.py +""" +import json +import shutil +import sys +import tempfile +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +import pageindex.client as client_mod +from pageindex import PageIndexClient +from pageindex.utils import walk_with_paths + +V1 = "# Root\nroot intro\n## A\nalpha text\n## B\nbeta text\n" + +CALLS = [] + + +async def _fake_summary(node, summary_token_threshold=200, model=None): + """Deterministic stand-in for get_node_summary — records what it is asked to do.""" + CALLS.append(node.get("title")) + return f"SUMMARY({node.get('title')})@{node.get('text_version')}" + + +class _Harness: + """Client with summarization stubbed out at every call site.""" + + def __enter__(self): + self.dir = Path(tempfile.mkdtemp()) + self._orig_client = client_mod.get_node_summary + client_mod.get_node_summary = _fake_summary + # index() summarizes via md_to_tree -> generate_summaries_for_structure_md, + # and also generates a doc description. Both would hit the network. + import pageindex.page_index_md as md_mod + self._orig_md = md_mod.get_node_summary + self._orig_desc = md_mod.generate_doc_description + md_mod.get_node_summary = _fake_summary + md_mod.generate_doc_description = lambda structure, model=None: "DESC" + self._md_mod = md_mod + CALLS.clear() + return self + + def __exit__(self, *a): + client_mod.get_node_summary = self._orig_client + self._md_mod.get_node_summary = self._orig_md + self._md_mod.generate_doc_description = self._orig_desc + shutil.rmtree(self.dir, ignore_errors=True) + + def client(self): + return PageIndexClient(workspace=str(self.dir)) + + def write(self, text): + p = self.dir / "d.md" + p.write_text(text, encoding="utf-8") + return str(p) + + +def _versions(c, doc_id): + c._ensure_doc_loaded(doc_id) + return { + p: (n.get("text_version"), n.get("summary_version")) + for p, n in walk_with_paths(c.documents[doc_id]["structure"]) + } + + +def _stale(c, doc_id): + return {p for p, (tv, sv) in _versions(c, doc_id).items() if tv != sv} + + +def test_index_stamps_versions_and_nothing_is_stale(): + with _Harness() as h: + c = h.client() + doc_id = c.index(h.write(V1)) + vs = _versions(c, doc_id) + assert vs, "structure should not be empty" + assert all(v == (1, 1) for v in vs.values()), vs + assert _stale(c, doc_id) == set() + + +def test_update_bumps_text_version_only_for_changed_nodes(): + with _Harness() as h: + c = h.client() + path = h.write(V1) + doc_id = c.index(path) + Path(path).write_text(V1.replace("alpha text", "alpha CHANGED"), encoding="utf-8") + + CALLS.clear() + c.update(doc_id) + + assert CALLS == [], f"update() must not summarize, but called {CALLS}" + vs = _versions(c, doc_id) + assert vs["Root > A"] == (2, 1), vs # changed -> stale + assert vs["Root > B"] == (1, 1), vs # untouched + assert vs["Root"] == (1, 1), vs # no propagation to parent + assert _stale(c, doc_id) == {"Root > A"} + + +def test_repeated_updates_batch_into_one_regeneration(): + with _Harness() as h: + c = h.client() + path = h.write(V1) + doc_id = c.index(path) + + for i in range(3): + Path(path).write_text(V1.replace("alpha text", f"alpha v{i}"), encoding="utf-8") + c.update(doc_id) + + vs = _versions(c, doc_id) + assert vs["Root > A"] == (4, 1), vs # 3 edits -> 3 bumps, summary still at 1 + + CALLS.clear() + n = c._reconcile_summaries(doc_id) + assert n == 1, n + assert CALLS == ["A"], CALLS # one call, not three + assert _stale(c, doc_id) == set() + + +def test_reconcile_is_idempotent(): + with _Harness() as h: + c = h.client() + path = h.write(V1) + doc_id = c.index(path) + Path(path).write_text(V1.replace("beta text", "beta CHANGED"), encoding="utf-8") + c.update(doc_id) + + assert c._reconcile_summaries(doc_id) == 1 + CALLS.clear() + assert c._reconcile_summaries(doc_id) == 0, "second read must regenerate nothing" + assert CALLS == [] + + +def test_added_node_starts_stale_and_gets_a_summary(): + with _Harness() as h: + c = h.client() + path = h.write(V1) + doc_id = c.index(path) + Path(path).write_text(V1 + "## C\ngamma text\n", encoding="utf-8") + c.update(doc_id) + + vs = _versions(c, doc_id) + assert vs["Root > C"] == (1, 0), vs # no summary yet + assert "Root > C" in _stale(c, doc_id) + + c._reconcile_summaries(doc_id) + c._ensure_doc_loaded(doc_id) + node = dict(walk_with_paths(c.documents[doc_id]["structure"]))["Root > C"] + assert node["summary"] == "SUMMARY(C)@1", node + assert _stale(c, doc_id) == set() + + +def test_structure_is_not_empty_after_reconciling_read(): + """_save_doc evicts structure for lazy reload; a reconciling read must not + hand the agent an empty tree.""" + with _Harness() as h: + c = h.client() + path = h.write(V1) + doc_id = c.index(path) + Path(path).write_text(V1.replace("alpha text", "alpha CHANGED"), encoding="utf-8") + c.update(doc_id) + assert _stale(c, doc_id), "precondition: something must be stale" + + payload = json.loads(c.get_document_structure(doc_id)) + assert payload, f"reconciling read returned empty structure: {payload!r}" + titles = [t for t, _ in walk_with_paths(payload)] + assert "Root > A" in titles, titles + + +def test_reindex_does_not_regress_versions(): + """Re-indexing reuses the doc_id; a reader at version N must not see it drop.""" + with _Harness() as h: + c = h.client() + path = h.write(V1) + doc_id = c.index(path) + Path(path).write_text(V1.replace("alpha text", "alpha CHANGED"), encoding="utf-8") + c.update(doc_id) + before = _versions(c, doc_id)["Root > A"][0] + assert before == 2 + + # Full re-index of the same path. + same_id = c.index(path) + assert same_id == doc_id + after = _versions(c, doc_id)["Root > A"] + assert after[0] >= before, f"version regressed: {before} -> {after[0]}" + assert after[0] == after[1], "re-index regenerates everything, so nothing is stale" + + +def test_reindex_after_edit_bumps_changed_node(): + with _Harness() as h: + c = h.client() + path = h.write(V1) + doc_id = c.index(path) + Path(path).write_text(V1.replace("alpha text", "alpha CHANGED"), encoding="utf-8") + c.index(path) + + vs = _versions(c, doc_id) + assert vs["Root > A"] == (2, 2), vs # changed -> bumped, summary fresh + assert vs["Root > B"] == (1, 1), vs # untouched -> held + + +if __name__ == "__main__": + fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + for fn in fns: + fn() + print(f"ok {fn.__name__}") + print(f"\n{len(fns)} passed")