diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 548c23c1ee..bae41c94c2 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -35,6 +35,7 @@ Use the full suite by default. Use targeted filters only while iterating locally - FTS optimization recommendation coverage keeps the shared evaluator exact at one write below, at, and one write above the 25-write threshold. Status, explain, optimize dry-run, optimize execution, and vacuum maintenance guidance must expose the same `recommended`, `action`, `reason`, `threshold_writes`, `observed_writes`, and `state`; stale batches, known WAL-stale snapshots, forward-incompatible schema stamps, and unavailable legacy counters/page snapshots suppress the recommendation, query-only status performs no source writes, and execution uses the focused counter/page/forward-contract/freshness snapshot instead of full status scans. A hot-WAL fixture opened through an explicit `immutable=1` URI must prove that status, standalone optimize dry-run, and the `index --optimize` dry-run alias preserve the same stale recommendation. A WAL or freelist state of `unknown` cannot select the optimize command, and a successful optimize reports the reset counter afterward. - Rebuild reclaim coverage creates a real high-freelist incremental-auto-vacuum database, runs CLI rebuild with a concurrently open reader, and verifies row integrity, before/after logical database-size reduction, ratio reduction below the shared threshold, reclaimed pages/bytes, immediate JSON, persisted `last_index_run.rebuild_reclaim`, and truthful explicit-vacuum metrics. Keep focused cases for below-threshold no-op, injected reclaim failure after commit, and interrupted rebuild recovery; a maintenance failure must preserve a usable committed database and stable bounded telemetry. - Full-scan CLI and MCP no-op coverage treats one repository-wide reusable-stat snapshot read and one folded-readiness verification as performance contracts. Keep assertions for one snapshot read, one stat lookup per candidate, one folded verification, and no content load for unchanged files when changing incremental indexing. +- Full-scan pre-write boundary coverage pins the phase order to `before_write`, write start, then `before_readiness`. The first-barrier drift theory must cover both incremental and rebuild requests and preserve readiness, indexed rows, C# source evidence, FTS recovery, and incomplete-batch markers. Keep C# preflight, target selection, dirty-byte FTS choice, final C# stat revalidation, scan-input validation, and loaded-workspace file-stat validation on the same reference-owned preparation state; no database repair or outer bulk-write/transaction guard may begin before those validations succeed. - Reference-graph refresh coverage treats graph-neutral indexing as a performance contract across incremental full scan, scoped `--files` update, and MCP indexing. Keep zero-refresh assertions for new and modified source files without symbols/references, plus a single batched refresh assertion when existing or new graph identity rows change. A healthy incremental generation must restrict identity/candidate/recursion work to transaction-committed dirty files, old and new `(language, folded name)` dependencies, and their old/new reciprocal edges; retain C#/Python language-transition and unchanged-target parity with a subsequent full refresh, rolled-back file batches, cancellation/retry, orphan-candidate cleanup, and the controlled 4,100-of-4,100 broad-scope fallback. Fresh/rebuild runs, missing identity contracts, and dirty sets of at least 4,096 references covering at least 50% of the graph must keep the full-refresh path. Query-plan coverage must keep all four scoped update phases and all ten candidate inserts on dirty-table-driven reference primary-key seeks, keep C# instantiate grouping on lookup names plus `idx_symbols_name_folded`, and prove that a sub-4,096 dirty set does not count the whole reference table without an explicit diagnostic hook. - Index-generation completeness coverage uses a table-driven full, symbols-only, max-file-byte, max-symbol, and max-reference matrix, with extractor failure kept separate because it uses a mutable hook. Assert that index-command JSON, immediate status, and workspace health expose identical index/graph booleans and reason arrays where available; MCP cap coverage must match the persisted status snapshot as well. Remove the additive completeness metadata in healthy and capped fixtures to preserve legacy fallback coverage, and clear issue readiness before a scoped capped update to prove current omission evidence survives degraded prior metadata. Lowering or raising the file-size policy must reprocess unchanged files in CLI and MCP indexing so a prior `file_too_large` issue cannot be reused. Structured remediation must distinguish symbols-only / missing-graph causes from reference safety caps and must not label an incomplete index as fold-only. Human output must identify incomplete generations instead of printing a complete summary. - Reference-identity refresh coverage treats a stable graph rebuild as a physical-write performance contract. Keep NULL-safe changed-row predicates for source identity, the four-column target-resolution tuple, self-reference, and mutual-recursion updates; trigger audits must remain at zero on a stable rerun, repair each corrupted phase once, and prove a later-phase failure rolls back earlier identity writes. SQLite `changes()` must continue to report the final mutual-recursion phase. @@ -52,6 +53,7 @@ Use the full suite by default. Use targeted filters only while iterating locally - `TransactionScope_RollbackClearsActiveTransactionBeforeDisposeReleasesGate` blocks rollback immediately before terminal-state publication, requires the detached active-transaction reference to be null while concurrent disposal and a successor still wait, then proves the successor transaction remains enlisted and commits after ownership transfers. - TypeScript augmentation grouping coverage seeds 5,000 singleton interfaces plus one two-declaration merge. Keep singleton groups as one stored declaration index and materialize declaration-index lists only for merged groups; preserve first-seen group and declaration order without restoring LINQ `GroupBy` allocation. - TypeScript augmentation graph-finalization coverage treats one mutual-recursion refresh as a physical-pass performance contract across incremental full scan, scoped update, and MCP indexing. A planned augmentation rebuild must replace the earlier graph pass and still refresh an empty result when it inherited that pass or deleted synthetic edges; marker-only validation with no inserted, deleted, or deferred graph work must restamp readiness with zero graph passes. Fresh or authoritative rebuild scans without TypeScript must use the normal graph pass and stamp augmentation readiness without rebuilding. Keep a late immutable-input drift case that skips augmentation, executes exactly one fallback refresh before partial readiness, leaves the augmentation marker stale, and succeeds with one augmentation-owned refresh on retry. Preserve the ordinary non-TypeScript graph tests, reference-index lifecycle, and the deferred `reference_graph` memory-sample attribution. +- Direct TypeScript augmentation transaction fixtures must prove that scoped deletion tracks an old edge whose other declaration disappeared outside the graph scope, leaving no augmentation candidate or deleted-only hotspot, and that cancellation during owned deferred-hotspot completion restores reference IDs and rows, candidates, hotspots, and readiness markers exactly. - Incremental TypeScript augmentation coverage tracks both old and new interface names across full scan, scoped update, and MCP indexing. Rebuild only declarations sharing those exact names, delete stale augmentation rows for removed names, and batch name predicates below SQLite's parameter budget; retain full-rebuild fallback for fresh/rebuild runs, broad dirty-name sets, project-root or contract-version changes, and an upfront forced extractor refresh. A runtime JavaScript/TypeScript configuration refresh must track every refreshed file's old and new names, with the adaptive broad-set fallback remaining authoritative. Full-fallback and symbols-only paths must keep rollback-safe readiness-only tracking without materializing interface names. Coverage must keep 1,000 untouched singleton interfaces outside the candidate set, cross the name-batch boundary with 1,001 requested names, switch a 5,001-name request back to the full path, preserve unrelated merged references and module classification from non-dirty indexed interface names when disk fallback is unavailable, detect stale-file purge plus a persisted TypeScript-to-non-TypeScript language transition before replacement in all three indexing paths, clear readiness at most once after the latest rollback and skip further checks while that clear is durable, and interrupt synchronous SQLite work plus roll back augmentation rows on cancellation. - Index-finalization readiness coverage keeps reference-cap reads valid inside an active writer transaction, preserves an unavailable last-run cap snapshot when scoped updates inherit a missing IssuesReady flag, and checks mixed C#/VB partial hotspot-family rows with one grouped reader initialization. Preserve both language results and the degraded-readiness gate when changing readiness SQL; do not replace them with wall-clock thresholds. - CI runs the test project through `tests/CodeIndex.Tests/CodeIndex.Tests.runsettings`, enables VSTest blame crash and hang collection, applies a 75-minute session timeout plus 60-second xUnit long-running diagnostics, and reruns the suite once after an initial failure. If the retry passes, CI uploads `TestResults/flaky-retry.txt` with the TRX and blame artifacts so the run is treated as suspect instead of silently trusted. @@ -186,8 +188,10 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result Cross-language taxonomy coverage requires every declared symbol and reference kind to be unique and accepted by the exact Ordinal lookup, while null, empty, whitespace-only, case variants, trailing-space variants, and unknown values remain rejected. Keep the writer's unknown symbol/reference and container-kind diagnostics, schema/catalog parity, and pattern-sidecar invalid-kind rejection in the same focused validation set. `DbSchemaConstraintTests` runs the public-array mutation regression in the non-parallel SQLite-sensitive collection and restores both arrays in `finally`. It must prove that writer validation and newly generated symbol/reference CHECK clauses continue to use the same private canonical taxonomy while the compatibility arrays are visibly mutated. Do not put a timing assertion in this test class. When a static lookup replay is useful, keep its Release harness temporary, feed the same persisted kind/count distribution to the legacy and candidate lookups, alternate execution order, and remove it before committing. The normal empty-database full-index A/B remains authoritative for the user-visible performance decision. -- `DatabaseTests.cs`, `DatabasePermissionPolicyTests.cs`, `DbReader*Tests.cs` - SQLite schema, write paths, migrations, and query behavior. DbReader coverage is split by query family, including search, SQL qualified-name handling, file dependencies, impact, and symbol-query suites, while shared seeded fixture state remains on the root `DbReaderTests` part. +- `DatabaseTests.cs`, `DatabasePermissionPolicyTests.cs`, `DbReader*Tests.cs`, `CSharpBaseListParserTests.cs` + SQLite schema, write paths, migrations, and query behavior. DbReader coverage is split by query family, including search, SQL qualified-name handling, file dependencies, impact, and symbol-query suites, while shared seeded fixture state remains on the root `DbReaderTests` part. File-dependency list and cycle-candidate coverage keep pre-cancellation ahead of command construction and execution; cycle-candidate coverage also pins reverse include/exclude path scope to the target side. + `DbReaderCSharpUsingCatalogTests` keeps reader-scoped C# using-catalog behavior explicit: file-scoped namespaces extend through EOF, nested namespace imports stay within the innermost inclusive scope, repeated active path/line lookups reuse the same cached set, and local/global alias shadowing preserves resolvable chains while terminating cycles. + `CSharpBaseListParserTests` keeps the SQLite-free base-list grammar matrix together, including generic constraints, nested generic/tuple/array syntax, alias qualifiers, primary/base constructors, declaration terminators, and the distinct reader type-reference and metadata head-identifier projections. Maintenance-lookup coverage keeps the `files(checksum)`, `files(path COLLATE NOCASE)`, and `file_issues(file_id, kind)` indexes aligned with their predicates and requires `EXPLAIN QUERY PLAN` index `SEARCH` operations for checksum purge, ASCII case-alias lookup, reusable-stat issue probes, and directory/stem rename candidates. The case-alias fixture uses changed content so checksum cannot hide a missing path lookup, and managed validation remains authoritative because SQLite `NOCASE` is only an ASCII prefilter, not a Unicode casing contract. Keep wildcard-bearing, extensionless, and near-match stem semantics in one fixture. Scoped-cleanup planning coverage unions checksum and exact same-directory/stem candidates into ascending deduplicated ID snapshots, merges overlapping plans, proves apply does not absorb matching rows added after planning, and rereads planned IDs immediately before apply so a deleted C# contract that reappears is deferred until a clean retry. The grouped C# fixture gives many targets one common checksum/stem and requires candidate-reader work to grow with unique keys plus returned rows rather than target-count squared; it also asserts that C# pre-workspace planning never deletes matching non-C# rows. Reference-resolution lookup coverage keeps the exact fresh-schema and read-migration index sets aligned for file/name, retained folded/NOCASE composite prefixes, resolved reverse edges, and the partial unresolved-mutual path. Assert the absence of all six retired single-prefix indexes plus the former all-row mutual index after migration and reopen, exact column order/collation/partial predicates, and `EXPLAIN QUERY PLAN` `SEARCH` operations for both full and scoped mutual SQL. Preserve fold-ready and legacy-fallback CLI/MCP exact-query signals, and rerun the same-name/different-container plus partial-class-over-same-file ranking fixtures when these indexes change. Repository-metadata graph coverage indexes TOML local paths and application-manifest assembly dependencies through the normal writer, then requires explicit `project_reference` / `dependency` caller queries so capability advertising cannot drift from persisted graph behavior. @@ -199,6 +203,9 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result `DatabasePermissionPolicyTests.cs` injects a file-mode provider so unsupported and access-denied Unix mode operations remain deterministic and cross-platform while proving both best-effort diagnostics and strict remediation failures. `DbSchemaConstraintTests.cs` also locks schema constraints to `SymbolKindCatalog` and required file foreign keys so DB readiness checks fail when code enums and SQLite CHECK clauses drift. Hotspot ranking fixtures should use the smallest counts that cross each ranking threshold; for structural-rank tests, keep one side just above the raw-reference comparison and the other just above the symbol-count threshold instead of scaling both far beyond the boundary. + Hotspot candidate-filter alignment is one public database-behavior contract across site, grouped, and file list/count queries. Keep visibility include/exclude filters before ambiguity cardinality, path include/exclude and test suppression after cardinality, authoritative family identity intact, and all six result/count shapes aligned. The structural SQL fixture separately pins the shared candidate plan to its explicit ten-column projection and forbids `SELECT *`. + Unused-candidate query coverage must keep the SQL-aware and non-SQL paths aligned on the shared 17-column projection, provisional bucket ordering, scope filters, and parameter binding. Retain one mixed C#/SQL public-behavior fixture that proves SQL reference resolution, C# projection and null fallbacks, list/count agreement, and detailed bucket counts together. A separate mixed partial-family fixture must force the resolver-aware and non-SQL routes with an SQL scope probe plus a visibility-filtered SQL control, then require result/count/detailed-count parity while the shared pre-page predicate suppresses a private member used from a peer part; keep same-file C# suppression and enum-collision scope in their focused fixtures. + Unused-symbol classification coverage must preserve surface-tag insertion order and contract-domain precedence when one symbol overlaps generated, contract-member, configuration, exception-metadata, public, and test signals. Keep those surface predicates evaluated once into the shared analysis, retain private/non-public early returns, and assert aligned reason tags, domain tags, and aggregate domain counts. Checkpoint listing and retention fixtures should exceed the relevant directory cap once and exceed the inspected-file cap on only one checkpoint; multiplying both caps together adds filesystem work without increasing boundary coverage. Checkpoint and restore-backup cleanup dry-run tests must assert both exact deleted/retained paths and the continued existence of every candidate directory. Restore dry-run tests should inject destination free space through `DbCommandRunner.AvailableFreeSpaceForTesting`, verify manifest and regular-file path diagnostics independently, and prove that no restore staging or backup directory is created. WAL checkpoint contention fixtures must establish a real read transaction before a later writer commit, then lower `busy_timeout` to keep the assertion bounded without fixed sleeps. Permission/error-path coverage may throw from `DbContext.WalCheckpointTruncateExecutedForTesting`; restore the previous `AsyncLocal` value in `finally` so parallel test state cannot leak. - `ConcurrencyTests.cs` @@ -216,8 +223,9 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result End-to-end upgrade path: seeds a pre-column legacy DB, opens it through `TryMigrateForRead`, and exercises the read paths that touch nullable symbol ordinals (outline, symbol search, nearby, unused, analyze bundle) to lock in the real-world failure mode behind #58 / #49. Migration ownership coverage must also distinguish caller-owned transactions from owned transactions, propagate unrelated `BEGIN` errors, preserve populated foreign-key rows across rebuilds, and verify rollback, foreign-key restoration, and successful retry after injected failures. - `IndexCommandRunner*Tests.cs`, `QueryCommandRunner*Tests.cs`, `ProgramCliTests.cs`, `InstallScriptTests.cs` YAML `outline` integration coverage must assert nested sequence paths, mapping-parent depth, unchanged source ranges and symbol counts, deterministic sorting, and cursor replay across a page boundary. + Search-core orchestration coverage must keep validation ahead of database execution, preserve list-recipes > named-query > recipe > plain route precedence and the recipe-field-before-conflict / plain-conflict-before-field aggregation error order, keep count-mode zero/nonzero payloads aligned, retain strict-not-found exit semantics, and order profiled NDJSON as result records, profile metadata, then the terminal record. CLI parsing, command execution, and installer behavior. Index command coverage is split by run mode or feature area, and query command coverage is split by command family with partial test classes so shared console and fixture helpers stay centralized. Keep repeated query-result fixtures, such as overlapping chunk content used by multiple search deduplication tests, in narrow class-level helpers instead of duplicating local builders. `ProgramCliTests.cs` covers top-level entrypoint behavior that must be exercised through a subprocess, while `InstallScriptTests.cs` runs focused bash snippets against `install.sh` in library mode to lock in release-installer regressions without performing real network installs. Installer bundle-generation tests must also verify that `install.sh` is marked generated while every canonical `install_modules/` source remains unmarked. - Authoritative C# scoped-update coverage treats bounded parallel extraction as a correctness and resource contract. Keep the `2 * workers` window bound, fixed-worker reuse, whole-window extraction barrier, target-ordered single-writer persistence, and mixed-language serial boundary in the same suite. Required fallbacks cover parallelism one, active symbol filters, content-load seams, post-extraction hooks, non-authoritative or undersized snapshots, ambiguous nullable language reuse, and speculative probe exceptions returning to the serial per-file boundary. Serial/parallel parity must compare summaries, files/chunks/all semantic symbol and reference columns, normalized reference-line and candidate identities, hotspot aggregates, issues, readiness/user-version metadata, and last-run readable-byte counters across normal, generated, cap, and oversized inputs. Retain all three file-stat barriers. Cancellation before persistence, after validated load, and at extraction completion must preserve the expected committed prefix, batch ownership, and derived readiness. Global watchdog and direct-fatal tests must assert active phase, bounded return, and terminal side effects; source-negative ordering tests must cover both cap-discarded confirmed evidence and an earlier lexical contract candidate still blocked in symbol extraction. Abnormal-window fixtures release every injected block and wait for the snapshotted all-workers-stopped seam before restoring static hooks or deleting the project root. + Authoritative C# scoped-update coverage treats bounded parallel extraction as a correctness and resource contract. Keep the `2 * workers` window bound, fixed-worker reuse, whole-window extraction barrier, target-ordered single-writer persistence, and mixed-language serial boundary in the same suite. Required fallbacks cover parallelism one, active symbol filters, content-load seams, post-extraction hooks, non-authoritative or undersized snapshots, ambiguous nullable language reuse, and speculative probe exceptions returning to the serial per-file boundary. Keep serial target disposition in its exact missing/delete, path-filter, indexability/language-probe, unsupported-purge, hardlink, stat-reuse, load/revalidate, reuse-cleanup, persistence, and catch order. Serial/parallel parity must compare summaries, files/chunks/all semantic symbol and reference columns, normalized reference-line and candidate identities, hotspot aggregates, issues, batch-marker cleanup, readiness/user-version metadata, and last-run readable-byte counters across normal, generated, cap, and oversized inputs. Retain all three file-stat barriers. The ordered worker-failure fixture keeps a failed authoritative target, a normal changed target, an unchanged but explicitly targeted authoritative target, and an oversized target in one four-file window; pin the exact persistence-event and commit-hook order, isolated failure phase, incomplete reason, metadata demotion, and separately observed post-update migration-marker cleanup. Cancellation before persistence, after validated load, and at extraction completion must preserve the expected committed prefix, batch ownership, and derived readiness. Global watchdog and direct-fatal tests must assert active phase, bounded return, and terminal side effects; source-negative ordering tests must cover both cap-discarded confirmed evidence and an earlier lexical contract candidate still blocked in symbol extraction. Abnormal-window fixtures release every injected block and wait for the snapshotted all-workers-stopped seam before restoring static hooks or deleting the project root. `JsonEnvelopeWrapperTests.cs` owns global JSON-envelope token-role coverage. Keep parser-accepted separated and inline query values, option ordering, the `--` end-of-options form, missing-value boundaries for other value-bearing options, and a genuine `--json-envelope` output request together so preprocessing cannot consume command data, suppress a structured error envelope, or place its injected `--json` after the positional boundary. Status-explain coverage must enumerate the source-generated `StatusResult` serializer properties and require every serialized top-level key to succeed without opening a database. Keep major readiness/trust/extension/maintenance/cap-hit metadata assertions, dotted-member resolution, bounded valid unknown candidates, and redaction of hostile field input in the same focused fixture so serialization and explainability cannot drift independently. Keep every structured explain response key registered for status `--fields` projection, and exercise the response through a bounded JSON projection that proves the outer envelope also omits runtime and path metadata. Safety-recipe semantic coverage must keep safe and unsafe controls in separate indexed files: assert that `Regex.Escape` / `Regex.Unescape` and literal `UseShellExecute=false` are absent, while matching, source-defined, or unresolved Regex members, literal `true`, and propagated shell-policy values remain with classification evidence. Include alias trivia, alias-like comments and strings, a qualified BCL call in a file with a bare `BoundedRegex` alias, a line break before literal `false`, and computed continuations after block and line comments. Also retain a bare receiver from a legacy index without current reference identity, add enough safe helpers to saturate the normal result envelope, keep count output aligned with full JSON, and retain a separate `new Regex` construction positive. @@ -241,7 +249,7 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result Doctor full-inventory coverage keeps composed filter selection, filtered summary counts, exact UTF-8 byte-budget boundaries, and structured overflow errors together in `ProgramRunnerTests`; license JSON remains a subprocess contract in `ProgramCliTests` so immediate-command dispatch and the published field names are both exercised. Ctags export JSON coverage reuses one seeded database for default and `--include-generated` variants, asserts the fixed skip-reason keys sum to `skipped_count`, and keeps the missing-`files.generated` degradation in a separate legacy-schema fixture. Dry-run JSON coverage for ambiguous `.h` files locks the bounded `language_detections` entries and their stable source/confidence codes without mutating the index. - Batch `--json-summary` coverage keeps side-effect-free dispatch plus successful JSON, single-row NDJSON, and JSON-looking text projection in one seeded fixture when the assertions stay distinct; include case-insensitive format values, command-specific JSON formats, and alias-injected JSON defaults so batch classification cannot drift from child parsers. Serialized-output exhaustion remains a separate boundary fixture because of its large escaped payload. Assert `result` / `results` versus raw `stdout` presence explicitly, compare `output_chars` with the actual captured stream length, and cover repeated malformed lines up to the input cap so neither syntax sniffing nor envelope overhead can bypass the transport bounds. Typed-failure coverage reuses one mixed scalar/array/schema/policy/child-failure input across serial and parallel modes, asserts stable ordered categories, and keeps timeout, caller cancellation, and explicit `--include-raw-streams` behavior in focused fixtures. Cancellation coverage includes pre-cancelled setup, cancellation after an earlier failure and after parallel item preparation, and a blocking stdin reader that proves both prompt wake-up and preservation of an in-flight line for the next batch invocation. + Batch `--json-summary` coverage keeps side-effect-free dispatch plus successful JSON, single-row NDJSON, and JSON-looking text projection in one seeded fixture when the assertions stay distinct; include case-insensitive format values, command-specific JSON formats, and alias-injected JSON defaults so batch classification cannot drift from child parsers. Serialized-output exhaustion remains a separate boundary fixture because of its large escaped payload. Assert `result` / `results` versus raw `stdout` presence explicitly, compare `output_chars` with the actual captured stream length, and cover repeated malformed lines up to the input cap so neither syntax sniffing nor envelope overhead can bypass the transport bounds. Typed-failure coverage reuses one mixed scalar/array/schema/policy/child-failure input across serial and parallel modes, asserts stable ordered categories, and keeps timeout, caller cancellation, and explicit `--include-raw-streams` behavior in focused fixtures. Shared line-preparation/accounting coverage compares mixed blank, valid, malformed, and unsupported input across serial and parallel modes, including ordered records, first failure, counters, limits, and output characters. Output-limit drain coverage gates two active workers without sleeps, proves the batch remains pending until the held worker completes, keeps an unpublished input-limit record out of committed counters, verifies console restoration after the join, and replays only accepted-but-undispatched nonblank input records in source order before pump-buffered future input with fresh line numbers and counters on the next invocation; blocked-input preservation remains in the focused cancellation fixture. Cancellation coverage includes pre-cancelled setup, cancellation after an earlier failure and after parallel item preparation, and a blocking stdin reader that proves both prompt wake-up and preservation of an in-flight line for the next batch invocation. Argument-validation variants that only differ by invalid scalar input share one database fixture and iterate within a fact when no per-case state or discovery identity is required. Positional `files` glob coverage shares one indexed-file fixture and iterates `*`, `?`, and recursive `**` patterns in a fact, matching the exact tokens that a quoted shell argument passes to the CLI. Excerpt focus coverage reuses one indexed fixture for line-only leading-window behavior, the focus-length dependency, and focus-column range validation; zero and non-numeric focus-column values share one indexed Markdown fixture. @@ -260,6 +268,7 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result Keep long multiline declaration/assignment continuations, declaration types split from their constructors, fully qualified local declaration types, generic-wrapper expression-bodied return types whose audited type occupies a non-terminal argument, and generic return types split before the audited type as positive JSON trust-boundary controls so formatting, qualification, and wrapper shape cannot change annotation binding. Keep overlapping child-query substrings on one call as a negative reuse control so only the first lexical match consumes the annotation. MCP schema-origin coverage keeps identical audit phrases in `McpToolCatalog.cs` top-level tool descriptions, concatenated description segments, nested schema-property prose, and executable C# in one indexed fixture; assert explicit `schema_description` search metadata and the recipe's JSON, SARIF, and issue-draft outputs so origin filtering cannot drift across projections. Recipe row-selection coverage reuses one multi-file, multi-chunk fixture across aggregate JSON, compact JSON, NDJSON, and issue-draft source metadata. Assert emitted/matched/omitted counts, `selection_reason` / `selection_omitted_count`, first-per-file path uniqueness, selector-preserving replay commands, suppressed raw cursors when a later limit truncates selected rows, and rejection of incoming cursors with either selector. A separate candidate-window fixture must exceed the default low-limit fetch envelope and prove that `--sample ` observes at least its requested candidate target; validate rejected selectors for non-row recipe shapes without opening a database. + Recipe query-driver parity coverage must share one indexed fixture across full JSON, compact JSON, count-summary JSON, and file aggregation, retaining the intended match while suppressing semantic-safe operations, bounded-alias files, and comment-only origins. Assert matching freshness counts in row/count projections and keep classified child failures invalid in freshness while omitting their result DTOs. Unused default-suppression row, JSON count, summary-only, and text count envelopes, including the `--all` count control, share one unused-symbol fixture. Unused default-suppressed and `--all` JSON cursor pagination share one unused-symbol fixture. Unused full and compact `--by-bucket` JSON envelopes share one taxonomy fixture. @@ -434,6 +443,8 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result cover multi-frame remainder reuse, CRLF split across a 4 KiB read boundary, Unicode bytes, an unterminated final frame, stable EOF, and rejection at one byte over the negotiated cap. They protect direct worker-response deserialization without constructing decoded JSON strings. - `IndexCommandRunnerTests`, `FileIndexerTests`, and `PerformanceTests` also cover `CSharpStaticInterfacePrepass` text, raw-byte, chunked raw-token, and streaming file contract probes. Stable candidate reads must authorize and open each file once, keep raw-negative reads to one bounded pass, and rewind that same handle only for a raw-positive full decode. A detected in-place or atomic-replacement mutation must discard that snapshot and reauthorize/reopen once so the prepass cannot diverge from the main indexing pass. Preserve UTF-8 / UTF-16, NUL rejection, growth, cancellation, and lexical-boundary behavior. The 576 KiB semantic-negative/positive allocation guard runs each probe 12 times and stays below 4 KiB of current-thread allocation so a whole-content mask cannot return. - The parallel C# static-interface full-scan fixture uses 64 implementation files and treats one workspace lookup build as a performance contract. Keep the contract lookup attached to the immutable prepass snapshot across CLI full scan, scoped update, and MCP indexing; do not rebuild it once per C# file. +- `Run_FullScan_PostPrepassCsharpContractLeavesReadinessPartialUntilCleanRetry` keeps the full-scan extraction state monotonic across fresh, rebuild, and incremental-existing routes. Its ordered Python-before-C# fixture must prove that the earlier raw chunk persists and remains searchable through both standard and trigram FTS after the later C# workspace snapshot drifts, with exactly one bulk-load optimization. +- `Run_FullScan_FatalParallelResultKeepsWorkerResourcesAliveUntilPeersStop` blocks one C# symbols worker while a peer reports a fatal extraction stall. Keep the command return prompt, assert that neither worker completion nor artifact-cache clearing occurs before the blocked peer is released, and wait for both cleanup signals before restoring process-wide hooks or deleting the fixture. - `CSharpPrepassSymbolArtifactCacheTests`, `FileIndexerTests`, and the CLI/MCP fresh-index fixtures protect bounded prepass artifact reuse. Keep deep-clone independence for generic admission; production owned-list admission must retain list/symbol identity only after successful atomic publication, leave rejected or cancelled input caller-owned, and release workspace fallback symbols only after both lookup snapshots are materialized. Preserve lookup parity and mutation isolation, take-once checksum matching, mismatch consumption, atomic file/symbol/estimated-byte caps, cancellation without partial admission, and non-admission of partial symbols after any bounded-regex timeout. Encoding theories must compare UTF-8, UTF-16 LE/BE, and invalid-UTF-8 prepass checksums with the authoritative loader. Integration coverage must prove reuse only for an empty non-rebuild full index, ordinary extraction for rebuild/symbols-only/existing/incomplete-or-stall paths, authoritative main-read mutation fallback, unchanged post hooks and family/kind processing, and cache clearing before graph work. - `SymbolExtractorRequiredLiteralGateTests` keeps built-in required-literal gating deterministic and output-preserving. It pins 400 audited Tier A patterns across 51 case-sensitive languages, compares @@ -455,6 +466,7 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result - `PreparedCommandCacheTests.DbWriter_WithCache_CSharpStaticInterfaceContractQueriesReuseCacheAndOneWorkspaceRead` keeps persisted C# contract-member candidates and the pending-path contract flag on one `files(lang)` → `symbols(file_id, kind)` row pass. Derive both results from that reader; do not restore a second all-contract query for excluded paths. Interface declarations may be loaded only for exact retained contract container names through bounded `symbols(name)` batches, and those dynamic tail shapes must stay out of the prepared cache. `DatabaseTests.CSharpContractWorkspaceQueries_UseFileKindThenBoundedInterfaceNamePlans` pins both index plans, while `LoadCSharpContractWorkspace_MaterializesOnlyCandidatesAndMatchingInterfaces` requires negative/LIKE-decoy-only reads to execute no interface phase and keeps unrelated plain interfaces out of managed materialization. `PreparedCommandCacheTests.DbWriter_CSharpStaticInterfaceContractMemberPreflightsAreExactBatchedAndCancellable` also splits a 503-path language lookup at 500 parameters, returns only persisted C# paths plus sorted purge-plan IDs/bytes, keeps both SQL tail shapes out of the prepared cache, and stops before the second batch when cancellation is requested. - `IndexCommandRunnerTests.Run_UpdateFiles_CsharpContractPreflightAvoidsRedundantWorkspacePasses` keeps a false source-evidence marker authoritative, leaves the repository-wide persisted-member preflight count at zero, and defers candidate reads plus workspace materialization to the single expanded C# pass. Its plain-interface, newly-added-contract, and persisted-contract phases pin the respective one scoped pass, discovery-plus-expanded passes, and one expanded pass. The paired full-scan/MCP known-evidence no-op fixtures require zero source loads, raw prepasses, persisted C# symbol loads, and lookup builds after the initial true or false snapshot. The shortcut must be rejected when explicit `index_completeness=complete` or GraphReady is absent, when symbols-only/filter/version/root/hotspot contracts drift, or when a persisted C# path changes language. Their final-stat mutation fixtures require one complete raw prepass plus every C# content load, and the explicit/shared-DB root-switch fixtures reject cross-root stat reuse. Legacy full-scan checkpoint coverage requires graph-bearing and symbols-only retries alike to rescan every directory and proves full scans neither read nor write HEAD-only resume state. Successful and stable-snapshot partial runs delete a seeded legacy checkpoint after the first barrier, while delete failure remains a warning. Listing-bound snapshot fixtures pin one baseline read per directory and no scan-end whole-tree validation. Full, expanded-Update, and MCP consumer fixtures pin exactly two scan-input validations on successful paths: immediately before index-data mutation and immediately before readiness. First-barrier failure must preserve domain rows, trust, evidence, purge state, and FTS recovery state after schema compatibility initialization; final-barrier failure must stay partial and must not stamp readiness. Ignored-directory churn remains clean. + MCP C# pre-write coverage runs the first snapshot-barrier preservation fixture with both ordinary and rebuild requests, so rebuild destruction and interrupted-FTS recovery cannot precede the barrier. A C# stat drift injected at `before_write` must suppress purge, preserve prior C# checksums and implicit-implementation references, publish unknown source evidence, and report only the `csharp_prepass` failure phase. The fresh-rebuild fixture also keeps static-interface evidence and its implicit reference authoritative without admitting C# prepass symbol artifacts. Plain-interface purge fixtures keep the sorted file-ID preflight exact, cancellation-aware, and bounded to 500 SQLite parameters per uncached batch. Transition-path preflight coverage places the only real member in the second path batch, requires a path-first `files(path)` plus `symbols(file_id, kind)` index plan, rejects keyword-boundary/casing decoys, interrupts on cancellation, and proves the dynamic commands never enter the prepared cache. One-sided same-stem/checksum rename and positive/unknown-evidence `--changed-between` outside-range fixtures require the immutable cleanup plan to remove stale contract rows before reference regeneration without sweeping unrelated missing non-C# rows; `Run_UpdateFiles_ChangedExistingRetainedTargetPreplansMatchingCsharpAlias` proves an already-indexed, stat-changed retained target also plans a checksum-matching C# alias before workspace construction, while unchanged targets reuse persisted checksums without an extra content read. Binary/oversized skip-record drift fixtures mutate after `BuildSkippedFileRecord` and require the nested cleanup/upsert transaction plus batch marker to roll back before retry. `Run_UpdateMode_WithChangedBetween_PurgesMissingIndexedPathOutsideDiff_4056` separately retains the historical all-language reconciliation under authoritative false evidence. `Run_UpdateMode_WithChangedBetween_CleanupPathReappearingAfterScanIsDeferredUntilRetry` restores a deleted contract during discovery, while `Run_UpdateCommits_ExactCleanupPathReappearingAfterSnapshotBarrierPreservesPriorRow` restores an exact Git-deleted target after the final snapshot barrier; both require a partial result that preserves the prior row and implicit reference. Fatal expanded discovery and post-prepass change/delete fixtures likewise preserve prior C# rows and implicit references, leave source evidence unknown, and converge on a later clean retry; unrelated Python/JavaScript cleanup and unchanged scoped targets still proceed under deferred C# mutation. The parallel symbol-cap fixture requires a contract first observed after prepass to leave source evidence unknown and force a later complete repair. - Case-folded cleanup coverage treats SQLite `NOCASE` and managed folding as candidate filters only: leaf-case, ancestor-case, and Unicode-folded paths that resolve to distinct file identities must preserve both exact rows, while a true case-only alias may remove the old spelling without a checksum read. An unrelated hardlink is an alias only when its path and identity match the same retained case-fold bucket; `ScopedFileCleanupReappearance_FoldBucketsDoNotCrossMatchTargetIdentities` prevents a path match from one target and an identity match from another from being combined. Git name-status coverage uses a real NUL-delimited rename whose old/new paths contain a tab and newline and asserts both commit and range helpers return the exact unquoted strings. - Project-marker budget integration coverage overrides the directory budget to its smallest boundary and enumerates one child; do not materialize the production 8,192-directory cap merely to prove warning propagation. @@ -541,7 +553,7 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result - C# constructor base-target coverage shares generic, interface-list, nested-generic, multiline, and constrained layouts in one fixture and asserts each rewritten edge by its unique terminal type. - C# multiline member attribution shares expression-bodied method and property layouts, Allman/same-line braces, and intervening block comments in one fixture, using unique member containers for assertions. - Python `isinstance`/`issubclass` runtime type checks share single-type and tuple forms in one module fixture, with function containers retaining case-level assertions. -- Python class-header type references share single-base, multiple-base, and metaclass forms in one module fixture, distinguished by class container. +- Python class-header type references share single-base, multiple-base, and metaclass forms in one module fixture, distinguished by class container; a dedicated capped-output fixture also locks multiline class-header remapping after the reference limit is reached. - Python annotation coverage shares direct and generic return, parameter, and local-variable forms in one module fixture, with function containers preserving each assertion. - Python semantic-type symbol coverage shares PEP 695 aliases, `TypeAlias`, `NewType`, `TypeVar`, `ParamSpec`, `TypeVarTuple`, and real imports in one module fixture, asserting dedicated kinds and import-filter exclusion; typing-factory reference coverage keeps unique target types in its shared module fixture. - Python advanced-typing coverage shares multiline/commented `TypeVar`, `ParamSpec`, callable annotations, variadic tuple unpacking, and literal unions in one logical-header fixture with unique type names. @@ -585,7 +597,8 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result - Go function-like symbol coverage shares qualified receiver containers and branch labels in one file fixture while excluding switch keywords. - Consolidated Python/Go extractor fixtures retain exact symbol/reference cardinality, including complete per-container result sets, and source-line assertions when those were part of the original regression contract. - Assembly reference coverage shares direct call/branch forms and tab-separated decorated indirect-target negatives in one exact-cardinality fixture. -- Solidity reference coverage shares inheritance whitespace variants, library/modifier/event/interface edges, and comment/string negatives in one fixture. +- Solidity reference coverage shares inheritance whitespace variants, library/modifier/event/interface edges, and comment/string negatives in one fixture. Its detailed early-return fixture lowers the definition and container safety limits together and asserts that both diagnostics survive before Solidity-specific extraction returns. +- Python mutual-recursion finalization coverage keeps a complete two-edge cycle ahead of a capped later call and asserts that the retained cycle is still marked when the reference list is already full. - Terraform reference coverage shares resource/module/data traversals and raw `var`/`local` object references in one exact-per-name fixture. - Language masker copy-on-write coverage pairs unchanged-array reuse and required-clone behavior within the same Lua and Solidity test methods. - COBOL target-statement coverage places SQL/CICS, report, sort, queue, file, literal, and external-call variants in one program and checks exact grouped edge counts. @@ -718,7 +731,7 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result JVM graph fixtures cover demand-driven Java type/module/method-reference, Kotlin type/infix/constructor, Scala contextual, and Gradle/Groovy DSL matches. Preserve dense-line ordering when adding cap-aware exits. Python graph fixtures cover streamed decorator arguments, annotations, runtime type checks, typing factories, dataclass/framework integrations, and dynamic imports. `BoundedRegexTests.EnumerateMatches_InstanceRegex_StartsAtRequestedOffset` fixes the no-prefix-rescan contract used by decorator arguments, while `EnumerateMatches_InstanceRightToLeftRegex_PreservesDefaultStartAndOrder` preserves instance-regex default direction. Direct instance-enumerator coverage in `BoundedRegexTests` also fixes lazy invalid-`startAt` validation, zero-length progress, `\G` continuation, explicit right-to-left start positions, one terminal timeout diagnostic after a successful prefix, early disposal without suffix work, unsupported `Reset`, and intentional LINQ compatibility. - PHP, Ruby, R, and Perl graph fixtures cover streamed attributes/docblocks/types, DSL command targets, namespace/member/resource references, and arrow calls. Keep nested token/type enumeration cap-aware. + PHP, Ruby, R, and Perl graph fixtures cover streamed attributes/docblocks/types, DSL command targets, namespace/member/resource references, and arrow calls. Keep nested token/type enumeration cap-aware. `ReferenceExtractorTests.Extract_PhpDocblockMethod_ReturnPrecedesParametersAtReferenceLimit` fixes PHP docblock preamble ordering and container state: an `@method` return type consumes the remaining reference slot before its parameter types while retaining the enclosing-class container snapshot. Secondary-language graph fixtures cover streamed Fortran, Visual Basic, F#, Pascal, Objective-C, Haskell, Elixir, Smalltalk, Lua, Dart, Razor, JSON, JavaScript, GitHub Actions, and C++ compound-requirement matches. `BoundedRegexTests.EnumerateMatches_StaticPatternCustomTimeout_ReturnsEmpty` and `EnumerateMatches_StaticPatternCustomTimeout_StopsAfterConsumerBreak` preserve explicit-timeout failure and early-disposal behavior. `PerformanceTests.ReferenceExtraction_BoundedDenseFSharpPipeline_StopsAtCapacity` fixes the bounded-list contract across line-phase handoffs and action-based call emitters: a 4,000-stage F# pipeline capped at one reference must not enumerate the unused stages. `PerformanceTests.ReferenceMatchEnumeration_BoundedListDoesNotRequestMatchAfterCapacity` and `ReferenceMatchEnumeration_ConcreteRegexDoesNotRequestSuffixAfterCapacity` prove that generic and concrete wrappers do not call the underlying enumerator's next `MoveNext()` after the cap is filled. `ReferenceMatchEnumeration_BelowCapacity_DoesNotAllocateWrapperEnumerators` keeps 10,000 ordinary generic below-cap scans free of wrapper-enumerator heap allocations. The blocking `net8.0` tests `BoundedRegexEnumeration_DirectNoMatchScansDoNotAllocateEnumerators` and `ReferenceMatchEnumeration_PrefilledCapacityDoesNotAllocateOrLookAhead` each run 10,000 concrete scans with a budget below 1,024 allocated bytes, covering both direct no-match enumeration and the already-full cap path. `PerformanceTests.ReferenceExtraction_PrologCallFreeRules_AvoidsPerLineLists` keeps 8,000 call-free Prolog rules from allocating empty goal lists or copying populated directive lists; its `net8.0` allocation budget is blocking. @@ -1088,6 +1101,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" - FTS optimization recommendation coverage は、25 write threshold の1つ下、ちょうど、1つ上で shared evaluator の境界を固定します。status、explain、optimize dry-run、optimize execution、vacuum maintenance guidance は同じ `recommended`、`action`、`reason`、`threshold_writes`、`observed_writes`、`state` を公開し、stale batch、既知の WAL-stale snapshot、forward-incompatible な schema stamp、利用できない legacy counter / page snapshot は recommendation を抑止します。query-only status は source に書き込まず、execution は full status scan ではなく counter / page / forward-contract / freshness に限定した snapshot を使います。hot WAL fixture を明示的な `immutable=1` URI で開き、status、standalone optimize dry-run、`index --optimize` dry-run alias が同じ stale recommendation を保持することも証明します。WAL または freelist の state が `unknown` の場合は optimize command を選択せず、成功した optimize は reset 後の counter を返す必要があります。 - rebuild reclaim coverage は、実際に high-freelist となった incremental-auto-vacuum database を作成し、reader connection を開いたまま CLI rebuild を実行して、row integrity、before / after の logical database size 縮小、共通 threshold 未満への ratio 低下、回収 page / byte、即時 JSON、保存された `last_index_run.rebuild_reclaim`、明示的 vacuum の正確な metrics を検証します。below-threshold no-op、commit 後の reclaim failure 注入、interrupted rebuild recovery の focused case も維持してください。maintenance failure が発生しても利用可能な commit 済み database と stable で上限付きの telemetry を保持する必要があります。 - full-scan CLI と MCP の no-op coverage は、リポジトリ全体の reusable-stat snapshot read と folded-readiness verification がそれぞれ 1 回であることを performance contract とします。incremental indexing を変更するときは、snapshot read が 1 回、候補ごとの stat lookup が 1 回、folded verification が 1 回、unchanged file の content load が 0 回という assertion を維持してください。 +- full-scan の write 前境界 coverage は、phase 順序を `before_write`、write 開始、`before_readiness` に固定します。最初の barrier で drift する theory は incremental / rebuild の両方を対象にし、readiness、indexed row、C# source evidence、FTS recovery、incomplete-batch marker を保持してください。C# preflight、target selection、dirty-byte FTS 判定、最終 C# stat 再検証、scan-input 検証、読み込み済み workspace の file-stat 検証は、同じ reference-owned preparation state 上で実行します。これらが成功する前に database repair や outer bulk-write / transaction guard を開始してはいけません。 - reference-graph refresh coverage は、incremental full scan、scoped `--files` update、MCP indexing を横断する graph-neutral indexing を performance contract とします。symbol/reference を持たない新規・変更 source file では refresh 0 回を維持し、既存または新規の graph identity 行が変化する場合は batch 全体で refresh 1 回を assertion してください。健全な incremental generation では identity / candidate / recursion 処理を transaction commit 済みの dirty file、旧・新の `(language, folded name)` 依存、旧・新の逆辺に限定します。C# / Python の言語遷移、未変更targetを参照する新規callerと後続full refreshのparity、rollback file batch、cancel後retry、孤立candidate cleanup、4,100件中4,100件をdirtyにする制御broad-scope fallbackを維持してください。fresh/rebuild、identity契約欠落、または4,096件以上かつgraphの50%以上を占めるdirty集合ではfull-refresh経路を維持します。query-plan coverageでは、scoped updateの4 phaseとcandidate INSERT 10本をdirty table起点のreference主キーseekに保ち、C# instantiate groupingをlookup nameと`idx_symbols_name_folded`起点にし、明示的なdiagnostic hookがない4,096件未満のdirty集合ではreference table全件COUNTを行わないことを検証してください。 - index generation の completeness coverage は full、symbols-only、max-file-byte、max-symbol、max-reference を table-driven matrix で検証し、mutable hook を使う extractor failure は別 case に保ちます。index command JSON、直後の status、workspace health で、利用可能な index/graph の boolean と reason array が完全に一致すること、MCP の cap case も persisted status snapshot と一致することを assertion してください。healthy / capped fixture から additive completeness metadata を削除して legacy fallback coverage を維持し、scoped capped update の前に issue readiness を clear して、prior metadata が degraded でも今回の omission evidence が失われないことを検証します。file-size policy を下げた場合も上げた場合も、CLI / MCP indexing は unchanged file を再処理し、以前の `file_too_large` issue を再利用してはいけません。structured remediation は symbols-only / missing-graph 原因と reference safety cap を区別し、incomplete index を fold-only と表示しない必要があります。human output は complete summary ではなく incomplete generation を明示する必要があります。 - reference identity refresh coverage は、安定graphの再構築を物理writeのperformance contractとします。source identity、target resolutionの4列tuple、self reference、mutual recursionの更新にはNULL-safeなchanged-row predicateを維持し、安定rerunのtrigger auditは0、各corrupt phaseのrepairは1回、後段phaseの失敗で先行identity writeもrollbackされることを検証してください。SQLite `changes()` は引き続き最後のmutual-recursion phaseを表します。 @@ -1105,6 +1119,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" - `TransactionScope_RollbackClearsActiveTransactionBeforeDisposeReleasesGate` は rollback の terminal-state 公開直前を停止し、detach 済み active-transaction reference が null の間も並行 Dispose と後続 owner が待機すること、その後の owner 移譲後に後続 transaction が enlist 状態を維持して commit できることを要求します。 - TypeScript augmentation grouping coverage は、5,000個の singleton interface と2宣言からなる1個の merge を投入します。singleton group は1個の declaration index だけを保持し、declaration-index list は merged group に限って materializeしてください。LINQ `GroupBy` allocation を戻さず、first-seen group と declaration の順序を維持します。 - TypeScript augmentation の graph-finalization coverage は、incremental full scan、scoped update、MCP indexing を横断して mutual-recursion refresh が物理的に1回であることを performance contract とします。予定された augmentation rebuild は先行 graph pass を置き換え、空の結果でもそのpassを引き継いだ場合または合成edgeを削除した場合はrefreshしなければなりません。edgeの挿入・削除も遅延graph workもないmarker検証だけなら、graph pass 0回でreadinessをrestampしてください。TypeScript のない fresh または authoritative rebuild scan は通常 graph pass を使い、augmentation rebuild なしで readiness を stamp します。late immutable-input drift では augmentation をskipし、partial readiness 前に fallback refresh を厳密に1回実行して augmentation marker を stale のまま残し、retry は augmentation 所有のrefresh 1回で成功することを維持してください。通常の非TypeScript graph test、reference-index lifecycle、遅延した `reference_graph` memory sample の帰属も保持します。 +- TypeScript augmentation transaction の direct fixture では、graph scope 外で片側宣言が消えた旧edgeを scoped delete が追跡して augmentation candidate と deleted-only hotspot を残さないこと、および owned deferred-hotspot completion 中の cancellation が reference ID / row、candidate、hotspot、readiness marker を完全に復元することを証明してください。 - incremental TypeScript augmentation coverage は、full scan、scoped update、MCP indexing を横断して変更前・変更後の interface 名を追跡します。その完全一致名を共有する宣言だけを再構築し、削除済み名の stale augmentation 行を消し、SQLite parameter budget 未満で name predicate をbatch化してください。fresh/rebuild、広範なdirty-name集合、project rootまたはcontract version変更、開始時点での強制extractor refreshでは全量fallbackを維持します。実行中に判明するJavaScript/TypeScript設定refreshは、refresh対象全fileの変更前後の名前を追跡し、広範囲集合ではadaptive fallbackをauthoritativeにします。full-fallbackとsymbols-only pathではinterface名をmaterializeせず、rollback-safeなreadiness-only trackingを維持します。1,000個の未変更singleton interfaceがcandidate外であること、1,001 requested namesでname batch境界を越えること、5,001-name requestが全量pathへ戻ること、無関係なmerged referenceとdisk fallback不能時に非dirty名が示すmodule分類を維持すること、stale-file purgeおよび3つのindexing pathすべてで置換前に永続化済みTypeScript→非TypeScript言語遷移を検知すること、直近のrollback後にreadinessを最大1回clearしそのclearが永続化している間は追加checkを省くこと、cancel時に同期SQLite処理をinterruptしてaugmentation行をrollbackすることをcoverageに含めます。 - index-finalization readiness coverage は active writer transaction 中の reference-cap read、IssuesReady flag を欠いた scoped update が unavailable な last-run cap snapshot を維持すること、C# / VB の partial hotspot-family rows を1回の grouped reader initialization で検証します。readiness SQL を変更するときは両言語の結果と degraded-readiness gate を維持し、wall-clock threshold へ置き換えないでください。 - CI は `tests/CodeIndex.Tests/CodeIndex.Tests.runsettings` 経由でテストプロジェクトを実行し、VSTest の blame crash / hang 収集、75分のセッションタイムアウト、60秒の xUnit long-running 診断を有効にします。初回失敗時は suite を1回だけ再実行し、再実行で成功した場合は TRX / blame artifact と一緒に `TestResults/flaky-retry.txt` を upload して、その実行を疑わしい flaky run として扱います。 @@ -1244,8 +1259,10 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" 全言語共通の taxonomy coverage では、宣言済みの全 symbol / reference kind が重複せず、完全一致の Ordinal lookup で受理されることを必須とします。null、空文字、空白のみ、case 違い、末尾空白、未知の値は引き続き拒否してください。writer の未知 symbol/reference kind と container kind の診断、schema/catalog parity、pattern sidecar の invalid-kind 拒否も同じ focused validation set で維持します。 `DbSchemaConstraintTests` の公開 array mutation 回帰は、並列実行しない SQLite-sensitive collection で実行し、`finally` で両 array を復元します。互換 array の変更が実際に見える間も、writer validation と新規生成する symbol / reference CHECK 句が同じ private canonical taxonomy を使い続けることを必須とします。 この test class に timing assertion を追加してはいけません。static lookup replay が有用な場合は Release harness を一時的なものに限定し、同一の保存済み kind/count 分布を旧来経路と候補経路へ流し、実行順を交互にしたうえで commit 前に削除します。ユーザーに見える性能の採否は、通常の空 database full-index A/B を authoritative としてください。 -- `DatabaseTests.cs`、`DatabasePermissionPolicyTests.cs`、`DbReader*Tests.cs` - SQLite スキーマ、書き込み経路、マイグレーション、クエリ挙動のテスト。DbReader のカバレッジは search、SQL qualified name、file dependency、impact、symbol query などの query family ごとの partial suite に分割し、共有の seed 済み fixture 状態は root 側の `DbReaderTests` に残します。 +- `DatabaseTests.cs`、`DatabasePermissionPolicyTests.cs`、`DbReader*Tests.cs`、`CSharpBaseListParserTests.cs` + SQLite スキーマ、書き込み経路、マイグレーション、クエリ挙動のテスト。DbReader のカバレッジは search、SQL qualified name、file dependency、impact、symbol query などの query family ごとの partial suite に分割し、共有の seed 済み fixture 状態は root 側の `DbReaderTests` に残します。file dependency list / cycle candidate の coverage では command 構築・実行より前の pre-cancellation を固定し、cycle candidate の coverage では reverse include/exclude path scope が target 側へ適用されることも固定します。 + `DbReaderCSharpUsingCatalogTests` では、reader scope の C# using catalog 契約を明示的に固定します。file-scoped namespace は EOF まで有効、nested namespace import は inclusive な最内 scope の外へ漏れず、同じ active path/line の反復 lookup は同一の cache set を再利用し、local/global alias の shadowing は解決可能な chain を保ちながら cycle で終了することを維持してください。 + `CSharpBaseListParserTests` には SQLite を使わない base-list 構文 matrix をまとめ、generic constraint、nested generic / tuple / array、alias qualifier、primary / base constructor、宣言終端、および reader の type-reference 投影と metadata の head-identifier 投影の差を固定します。 maintenance lookup の coverage では `files(checksum)`、`files(path COLLATE NOCASE)`、`file_issues(file_id, kind)` の index を predicate と同期させ、checksum purge、ASCII case-alias lookup、再利用 stat の issue probe、directory/stem rename 候補が `EXPLAIN QUERY PLAN` で index `SEARCH` を使うことを必須とします。case-alias fixture は checksum で path lookup の欠落が隠れないよう content も変更し、SQLite `NOCASE` は Unicode casing contract ではなく ASCII prefilter にすぎないため managed 検証を authoritative に保ちます。wildcard を含む stem、拡張子なし、近似 stem の意味論は1つの fixture にまとめます。scoped cleanup planning の coverage では checksum と正確な同一 directory/stem の候補を昇順・重複排除済み ID snapshot に統合し、重複 plan の merge、plan 後に追加された一致 row を apply が取り込まないこと、apply 直前に planned ID を再読込して再出現した C# contract を clean retry まで延期することを固定します。grouped C# fixture は多数の target に共通 checksum/stem を与え、candidate reader の処理量が target 数の二乗ではなく unique key 数と返却 row 数に比例すること、および C# workspace 前の plan が一致する non-C# row を削除しないことを要求します。 参照解決 lookup の coverage では fresh schema / read migration の正確な index setを、file/name、保持したfolded/NOCASE composite prefix、解決済み逆辺、未解決mutual用partial pathまで同期させます。migrationと再open後に退役したsingle-prefix 6本と旧全row mutual indexが存在しないこと、列順・collation・partial predicateを厳密に検証し、full/scoped mutual SQLの両方で`EXPLAIN QUERY PLAN`のindex `SEARCH`を必須とします。fold-ready/legacy fallback双方のCLI/MCP exact-query signalを維持し、これらのindexを変える際は同名・別containerとpartial classが同一file候補より優先される既存ranking fixtureも再実行してください。 repository metadata の graph coverage では TOML の local path と application manifest の assembly dependency を通常 writer 経由で index し、明示的な `project_reference` / `dependency` caller query を必須にして capability 広告と永続化済み graph のずれを防いでください。 @@ -1257,6 +1274,9 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" `DatabasePermissionPolicyTests.cs` は file-mode provider を注入し、unsupported / access-denied な Unix mode 操作を deterministic かつ cross-platform に保ちながら、best-effort diagnostic と strict remediation failure の両方を検証します。 `DbSchemaConstraintTests.cs` は DB readiness check が code enum と SQLite CHECK 句の drift を検出できるよう、schema constraint と `SymbolKindCatalog`、必須 file foreign key の同期も固定します。 hotspot ranking fixture は各 ranking threshold を跨ぐ最小 count を使ってください。structural-rank test では、raw reference 比較をわずかに超える側と symbol-count threshold をわずかに超える側を用意し、境界から大きく離れた件数まで膨らませないでください。 + hotspot candidate filter の整合性は、site / grouped / file の list / count 全6 query に共通する public database behavior 契約です。visibility include / exclude は ambiguity cardinality より前、path include / exclude と test suppression は cardinality より後に適用し、authoritative family identity と全6形状の結果・件数を一致させてください。structural SQL fixture は別途、共通 candidate plan の明示的な10列 projection と `SELECT *` 禁止を固定します。 + unused candidate query の coverage では、SQL-aware / non-SQL 経路で共通の17列 projection、provisional bucket 順、scope filter、parameter binding を一致させてください。mixed C#/SQL の public behavior fixture 1つで SQL reference resolution、C# projection と null fallback、list/count の一致、詳細 bucket count をまとめて固定します。別の mixed partial-family fixture では、SQL scope probe と visibility filter で除外される SQL control によって resolver-aware / non-SQL の両経路を強制し、peer part から使用される private member を共有の page 前 predicate が抑制した状態で、result / count / detailed-count の一致を必須にしてください。same-file C# suppression と enum collision scope は焦点を絞った既存 fixture で維持します。 + unused symbol classification の coverage では、1つの symbol が generated、contract member、configuration、exception metadata、public、test の各 signal に重なる場合も、surface tag の追加順と contract domain の優先順位を維持してください。これらの surface predicate は共有 analysis へ1度だけ評価し、private / non-public の early return を保ったまま、reason tag、domain tag、domain 集計件数の一致を assertion します。 checkpoint listing / retention fixture は対象の directory cap を 1 件だけ超え、inspected-file cap は 1 checkpoint だけで超えてください。両方の cap を掛け合わせても boundary coverage は増えず、filesystem work だけが増えます。checkpoint / restore-backup cleanup の dry-run test では、削除予定・保持予定の正確な path と、全候補 directory が引き続き存在することの両方を検証してください。restore dry-run test は `DbCommandRunner.AvailableFreeSpaceForTesting` で destination free space を注入し、manifest と regular-file path の diagnostic を独立に検証し、restore staging directory や backup directory が作られないことを固定します。 WAL checkpoint contention fixture は後続 writer commit より先に実際の read transaction を確立し、固定 sleep を使わず assertion を bounded にするため `busy_timeout` を短くします。permission / error path の coverage では `DbContext.WalCheckpointTruncateExecutedForTesting` から exception を throw できますが、parallel test state を漏らさないよう、以前の `AsyncLocal` 値を `finally` で復元してください。 - `ConcurrencyTests.cs` @@ -1273,8 +1293,9 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" エンドツーエンドのアップグレード経路: カラム追加前のレガシー DB を用意し、`TryMigrateForRead` 経由で開いてから NULL になりうるシンボル列を触る read path（outline、シンボル検索、近傍、unused、analyze バンドル）を一通り叩き、#58 / #49 の実機失敗モードを固定する。migration ownership の coverage では、caller-owned transaction と cdidx-owned transaction を区別し、無関係な `BEGIN` error を伝播させ、foreign key が設定された既存 row を rebuild 後も保持し、注入 failure 後の rollback、foreign key 復元、retry 成功まで検証する。 - `IndexCommandRunner*Tests.cs`、`QueryCommandRunner*Tests.cs`、`ProgramCliTests.cs`、`InstallScriptTests.cs` YAML `outline` の integration coverage では、入れ子 sequence path、mapping parent の depth、不変な source range と symbol count、決定的な sort、page 境界を越える cursor replay を検証してください。 + search core の orchestration coverage では、validation が database execution より先に行われること、list-recipes > named-query > recipe > plain の route precedence と recipe では field error が conflict より先・plain では conflict が field error より先になる aggregation error 順、count mode の zero/nonzero payload、strict-not-found の exit semantics、profile 付き NDJSON が result record、profile metadata、terminal record の順になることを維持してください。 CLI の引数解析、コマンド実行、installer 挙動のテスト。Index command coverage は run mode または機能領域ごとの partial suite に分割し、Query command coverage は command family ごとの partial test class に分割して、共有 console / fixture helper は一箇所に保ちます。`ProgramCliTests.cs` はグローバル引数の解釈や完全な CLI 起動フローのように subprocess 経由で確認すべき Program エントリポイント挙動を扱い、`InstallScriptTests.cs` は `install.sh` を library mode で source した bash snippet を実行して、実ネットワーク install を行わずに release installer の回帰を固定する。installer bundle 生成テストでは、`install.sh` が generated と判定される一方、canonical な `install_modules/` source はすべて unmarked のままであることも検証してください。 - authoritative C# scoped update の coverage では、bounded parallel extraction を correctness と resource の契約として扱います。`2 * workers` の window 上限、固定 worker の再利用、window 全体の extraction barrier、target 順の single-writer persistence、mixed-language の serial 境界を同じ suite で維持してください。必須 fallback は parallelism 1、active symbol filter、content-load seam、post-extraction hook、non-authoritative / target 不足 snapshot、ambiguous な nullable language reuse、speculative probe 例外から serial per-file boundary への復帰を含みます。serial / parallel parity は normal、generated、cap、oversized input を横断し、summary、file / chunk、symbol / reference の全 semantic column、正規化した reference-line / candidate identity、hotspot aggregate、issue、readiness / user-version metadata、last-run readable-byte counter を比較してください。3段階の file-stat barrier も維持します。persistence 前、validated load 後、extraction completion 時の cancellation は、想定 committed prefix、batch ownership、derived readiness を固定します。global watchdog と direct fatal は active phase、bounded return、terminal side effect を検証し、source-negative ordering は cap で payload から除かれた confirmed evidence と、symbol extraction 中に block した earlier lexical contract candidate の両方を含めます。異常 window の fixture は注入 block をすべて解放し、snapshot 済み all-workers-stopped seam を待ってから static hook の復元と project root の削除を行ってください。 + authoritative C# scoped update の coverage では、bounded parallel extraction を correctness と resource の契約として扱います。`2 * workers` の window 上限、固定 worker の再利用、window 全体の extraction barrier、target 順の single-writer persistence、mixed-language の serial 境界を同じ suite で維持してください。必須 fallback は parallelism 1、active symbol filter、content-load seam、post-extraction hook、non-authoritative / target 不足 snapshot、ambiguous な nullable language reuse、speculative probe 例外から serial per-file boundary への復帰を含みます。serial target disposition は missing/delete、path filter、indexability / language probe、unsupported purge、hardlink、stat reuse、load / revalidate、reuse cleanup、persistence、catch の厳密な順序を維持してください。serial / parallel parity は normal、generated、cap、oversized input を横断し、summary、file / chunk、symbol / reference の全 semantic column、正規化した reference-line / candidate identity、hotspot aggregate、issue、batch marker cleanup、readiness / user-version metadata、last-run readable-byte counter を比較してください。3段階の file-stat barrier も維持します。順序付き worker-failure fixture では、失敗する authoritative target、通常の変更 target、明示対象だが変更なしの authoritative target、oversized target を4-file windowにまとめ、persistence event と commit hook の厳密な順序、分離された failure phase、incomplete reason、metadata demotion、update後に独立観測する migration marker cleanupを固定してください。persistence 前、validated load 後、extraction completion 時の cancellation は、想定 committed prefix、batch ownership、derived readiness を固定します。global watchdog と direct fatal は active phase、bounded return、terminal side effect を検証し、source-negative ordering は cap で payload から除かれた confirmed evidence と、symbol extraction 中に block した earlier lexical contract candidate の両方を含めます。異常 window の fixture は注入 block をすべて解放し、snapshot 済み all-workers-stopped seam を待ってから static hook の復元と project root の削除を行ってください。 `JsonEnvelopeWrapperTests.cs` は global JSON envelope の token role coverage を担当します。前処理が command data を消費したり、構造化 error envelope を抑止したり、補う `--json` を positional 境界より後ろへ置いたりしないよう、parser が受理する分離形式と inline 形式の query 値、option 順序、`--` end-of-options 形式、他の value-bearing option の missing-value 境界、実際の `--json-envelope` 出力要求を同じ fixture で維持してください。 status explain の coverage は source-generated `StatusResult` serializer property を列挙し、database を開かずに serialized top-level key がすべて成功することを必須にします。主要な readiness / trust / extension / maintenance / cap-hit metadata、dot 区切り member resolution、unknown key に対する上限付きの有効な candidate、悪意ある field input の redaction を同じ focused fixture に置き、serialization と explainability が独立して drift しないようにしてください。structured explain response の全 key を status の `--fields` projection に登録し、outer envelope からも runtime / path metadata が省略されることを上限付き JSON projection で固定してください。 safety recipe の意味論 coverage では、安全側と危険側の control を別々の indexed file に置いてください。`Regex.Escape` / `Regex.Unescape` と literal `UseShellExecute=false` が除外され、matching、source-defined、または解決不能な Regex member、literal `true`、伝播された shell-policy 値が classification evidence 付きで残ることを検証します。alias の trivia、alias に見える comment / string、bare `BoundedRegex` alias と完全修飾 BCL call が同居する file、literal `false` の前の改行、block / line comment の後に続く計算式を含めます。また、現行 reference identity を持たない legacy index の bare receiver を残すこと、通常の result envelope を埋める数の safe helper、count 出力と full JSON の一致、別の `new Regex` construction 正例も維持してください。 @@ -1298,7 +1319,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" doctor full-inventory coverage では、合成 filter の選択、filtered summary 件数、UTF-8 byte budget の exact boundary、structured overflow error を `ProgramRunnerTests` にまとめます。license JSON は `ProgramCliTests` の subprocess contract として、immediate-command dispatch と公開 field 名を同時に検証します。 ctags export JSON coverage は1つの seeded database を既定と `--include-generated` variant で再利用し、固定された skip-reason key の合計が `skipped_count` と一致することを検証します。`files.generated` がない場合の縮退は別の legacy-schema fixture に保ってください。 曖昧な `.h` に対する dry-run JSON coverage は、index を変更せず、上限付き `language_detections` entry と安定した判定元・信頼度 code を固定します。 - batch `--json-summary` coverage は、assertion を明確に保てる場合、副作用なし dispatch と成功時の JSON / 1 row NDJSON / JSON に見える text projection を1つの seeded fixture で共有し、case-insensitive な format 値、command 固有の JSON format、alias が内部追加する JSON default も含めて child parser と batch classification の drift を防いでください。serialized output の枯渇は escape を多く含む大きな payload の境界 fixture として分離します。test 側で `result` / `results` と raw `stdout` の有無を明示し、`output_chars` を実際に捕捉した stream 長と比較し、input cap までの malformed line 反復も検証して、syntax sniffing や envelope overhead が transport 上限を迂回しないようにしてください。typed failure coverage は scalar / array / schema / policy / child failure を混在させた1つの input を serial / parallel mode で共有し、安定した順序と category を検証してください。timeout、caller cancellation、明示的な `--include-raw-streams` の挙動は焦点を絞った fixture に分離します。cancellation coverage には事前 cancel 済み setup、先行 failure 後と parallel item 準備後の cancellation、blocking stdin reader を含め、型付き summary、迅速な wait 解除、同時に読み取った line が次の batch invocation 用に保持されることを決定的に検証します。 + batch `--json-summary` coverage は、assertion を明確に保てる場合、副作用なし dispatch と成功時の JSON / 1 row NDJSON / JSON に見える text projection を1つの seeded fixture で共有し、case-insensitive な format 値、command 固有の JSON format、alias が内部追加する JSON default も含めて child parser と batch classification の drift を防いでください。serialized output の枯渇は escape を多く含む大きな payload の境界 fixture として分離します。test 側で `result` / `results` と raw `stdout` の有無を明示し、`output_chars` を実際に捕捉した stream 長と比較し、input cap までの malformed line 反復も検証して、syntax sniffing や envelope overhead が transport 上限を迂回しないようにしてください。typed failure coverage は scalar / array / schema / policy / child failure を混在させた1つの input を serial / parallel mode で共有し、安定した順序と category を検証してください。timeout、caller cancellation、明示的な `--include-raw-streams` の挙動は焦点を絞った fixture に分離します。共有 line preparation / accounting の coverage では blank、valid、malformed、unsupported input の混在を serial / parallel mode で比較し、record 順、first failure、counter、limit、output character 数を固定します。output-limit drain coverage は sleep なしの gate で active worker 2つを保持し、一方の worker が完了するまで batch が pending のままであること、未配送の input-limit record が確定済み counter に混入しないこと、join 後に console が復元されること、受理済みでも未 dispatch の nonblank input record だけが source 順で pump buffer 内の後続入力より前に戻され、次の invocation では line number と counter が新しく付け直されることを検証します。blocked input の保持は focused cancellation fixture に分離してください。cancellation coverage には事前 cancel 済み setup、先行 failure 後と parallel item 準備後の cancellation、blocking stdin reader を含め、型付き summary、迅速な wait 解除、同時に読み取った line が次の batch invocation 用に保持されることを決定的に検証します。 invalid scalar input だけが異なる argument-validation variant は、case ごとの state や discovery identity が不要なら1つの database fixture を共有し、fact 内で反復してください。 `files` の positional glob coverage は1つの indexed-file fixture を共有し、quote された shell 引数が CLI に渡す token と同じ `*`、`?`、recursive `**` pattern を fact 内で反復してください。 excerpt の focus coverage は、line-only 時の先頭側 window、focus-length の依存関係、focus-column の範囲検証を1つの indexed fixture で共有してください。focus-column の zero / non-numeric value も1つの indexed Markdown fixture を再利用してください。 @@ -1317,6 +1338,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" 長い複数行へ継続する宣言 / 代入、constructor と別行に分割された宣言型、完全修飾された local 宣言型、audit 対象型が末尾以外の型引数にある generic wrapper を持つ expression-bodied な戻り型、および audit 対象型より前で改行された generic 戻り型を JSON trust-boundary の positive control とし、formatting、修飾、wrapper の形によって注釈の束縛が変わらないことを維持してください。同じ呼び出しで重なる child-query substring は negative reuse control とし、最初の lexical match だけが注釈を消費することを維持してください。 MCP schema-origin coverage では、同一の audit phrase を `McpToolCatalog.cs` の top-level tool description、連結された description segment、nested schema property の prose、実行可能な C# に置いた1つの indexed fixture を共有し、明示的な `schema_description` 検索 metadata と recipe の JSON、SARIF、issue-draft 出力を検証して、projection 間で origin filter が drift しないようにしてください。 recipe row-selection coverage は aggregate JSON、compact JSON、NDJSON、issue-draft の source metadata で1つの multi-file / multi-chunk fixture を共有します。emitted / matched / omitted count、`selection_reason` / `selection_omitted_count`、first-per-file の path uniqueness、selector を保持する replay command、後続 limit が選択済み row を truncate する場合の raw cursor 抑止、両 selector と受け取った cursor の併用拒否を検証してください。別の candidate-window fixture では既定の low-limit fetch envelope を超え、`--sample ` が少なくとも要求 candidate 数を観測することを証明し、row を持たない recipe shape での selector 拒否は database を開かずに確認してください。 + recipe query-driver parity coverage は full JSON、compact JSON、count-summary JSON、file aggregation で1つの indexed fixture を共有し、意図した match を保持しつつ、semantic-safe な操作、bounded alias の file、comment-only origin を抑制してください。row / count projection の freshness match count を検証し、分類済み child failure は freshness では invalid のまま、result DTO からは省略されることを確認してください。 unused default-suppressionのrow、JSON count、summary-only、text count envelopeは、`--all` count controlも含めて1つのunused-symbol fixtureを共有してください。 unusedのdefault-suppressed JSON cursor paginationと`--all` JSON cursor paginationは1つのunused-symbol fixtureを共有してください。 unusedのfull JSONとcompact `--by-bucket` JSON envelopeは1つのtaxonomy fixtureを共有してください。 @@ -1491,6 +1513,8 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" multi-frame remainder の再利用、4 KiB read 境界をまたぐ CRLF、Unicode byte、改行なし最終 frame、安定した EOF、合意済み上限を1 byte 超えた時点での拒否を検証します。decode 済み JSON string を作らず worker response を直接 deserialize する経路を固定します。 - `IndexCommandRunnerTests`、`FileIndexerTests`、`PerformanceTests` は `CSharpStaticInterfacePrepass` のテキスト判定、raw-byte、chunked raw-token、streaming file 契約 probe も扱います。安定した候補読み取りは各 file を1回だけ認可・openし、raw-negative は bounded pass 1回に留め、raw-positive の full decode だけ同じ handle を rewind してください。in-place mutation または atomic replacement を検知した場合は snapshot を破棄し、prepass と main indexing pass が乖離しないよう1回だけ再認可・再openします。UTF-8 / UTF-16、NUL 拒否、growth、cancellation、lexical boundary を維持してください。576 KiB の semantic-negative/positive allocation guard は各 probe を12回実行して current-thread allocation を4 KiB未満に保ち、content 全体 mask の再導入を防ぎます。 - parallel C# static-interface full-scan fixture は64個のimplementation fileを使い、workspace lookup buildが1回であることをperformance contractとします。CLI full scan、scoped update、MCP indexingを横断してcontract lookupをimmutable prepass snapshotに保持し、C# fileごとの再構築を戻さないでください。 +- `Run_FullScan_PostPrepassCsharpContractLeavesReadinessPartialUntilCleanRetry` は fresh、rebuild、incremental-existing の各 route で full-scan extraction state が単調に維持されることを固定します。Python を C# より先に処理する fixture で、後段の C# workspace snapshot drift 後も先行 raw chunk が保存され、standard / trigram FTS の両方から検索でき、bulk-load optimization が厳密に1回であることを証明してください。 +- `Run_FullScan_FatalParallelResultKeepsWorkerResourcesAliveUntilPeersStop` は一方の C# symbols worker を block し、peer に fatal extraction stall を返させます。command が速やかに戻ること、block 中の peer を release する前に worker completion と artifact-cache clear のどちらも起きないことを assertion し、process-wide hook の復元や fixture 削除の前に両 cleanup signal を待ってください。 - `CSharpPrepassSymbolArtifactCacheTests`、`FileIndexerTests`、CLI/MCP の fresh-index fixture は bounded prepass artifact reuse を固定します。汎用 admission の deep-clone 独立性を維持し、production の owned-list admission は原子的な publish 成功後だけ list / symbol identity を保持し、reject または cancel された input は caller-owned のままにしてください。2種類の lookup snapshot を materialize した後だけ workspace fallback symbol を解放し、lookup parity と mutation isolation を保ちます。checksum 一致時の take-once、不一致時の消費、file / symbol / estimated-byte cap の原子性、partial admission を残さない cancellation、bounded-regex timeout 後の partial symbol をadmitしない契約も維持してください。encoding theory は UTF-8、UTF-16 LE/BE、不正 UTF-8 の prepass checksum を authoritative loader と比較します。integration coverage では空 database の非 rebuild full index だけが再利用し、rebuild / symbols-only / existing / incomplete-or-stall 経路は通常 extraction、main read 中の mutation は checksum fallback、post hook と family/kind 処理は従来どおり、graph 開始前に cache が clear されることを証明してください。 - `SymbolExtractorRequiredLiteralGateTests` は built-in required-literal gate の決定性と output 不変性を固定します。51 の case-sensitive 言語にまたがる監査済み Tier A pattern 400件、出力順を @@ -1512,6 +1536,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" - `PreparedCommandCacheTests.DbWriter_WithCache_CSharpStaticInterfaceContractQueriesReuseCacheAndOneWorkspaceRead` は、永続化済み C# contract member 候補と pending-path contract flag を1回の `files(lang)` → `symbols(file_id, kind)` row passで得ることを固定します。両方を同じ reader から導出し、除外 path 判定のための2回目の全 contract query を戻さないでください。interface 宣言は厳密検証後に保持した contract container 名だけを bounded な `symbols(name)` batch で取得でき、その dynamic tail shape は prepared cache に残してはいけません。`DatabaseTests.CSharpContractWorkspaceQueries_UseFileKindThenBoundedInterfaceNamePlans` は両方の index plan を固定し、`LoadCSharpContractWorkspace_MaterializesOnlyCandidatesAndMatchingInterfaces` は negative / LIKE decoy だけの読込で interface phase が0回となり、無関係な通常 interface を managed materialization へ入れないことを要求します。`PreparedCommandCacheTests.DbWriter_CSharpStaticInterfaceContractMemberPreflightsAreExactBatchedAndCancellable` は503件のpath言語lookupを500 parameterで分割し、永続C# pathと昇順のpurge-plan ID/byteだけを返し、両方のSQL tail shapeをprepared cache外に保ち、1つ目のbatch後のcancellationで2つ目を実行しないことも固定します。 - `IndexCommandRunnerTests.Run_UpdateFiles_CsharpContractPreflightAvoidsRedundantWorkspacePasses` は false の source-evidence marker を authoritative に保ち、repository 全体の persisted-member preflight 回数を0に固定して、candidate read と workspace materialization を1回の expanded C# passへ委譲します。plain-interface、新規contract追加、永続化済みcontractの各phaseで、それぞれscoped 1回、discovery＋expanded、expanded 1回を固定します。対になるfull-scan/MCPのknown-evidence no-op fixtureでは初回のtrueまたはfalse snapshot以降、source load、raw prepass、永続C# symbol load、lookup buildをすべて0に固定します。明示的な `index_completeness=complete` または GraphReady がない場合、symbols-only/filter/version/root/hotspot contract が変わった場合、永続 C# path が別言語へ変わった場合はshortcutを拒否しなければなりません。final-stat mutation fixtureではcomplete raw prepass 1回と全C# content loadを要求し、explicit/shared DBのroot-switch fixtureではcross-root stat reuseを拒否します。 legacy full-scan checkpoint fixtureはgraph-bearing / symbols-onlyの両方で全directoryを再scanし、HEAD-only resume stateをfull scanが作成も参照もしないことを固定します。seedした旧checkpointはfirst barrier後にsuccess pathとstable-snapshot partial pathの両方で削除し、delete failureだけをwarningにします。listingに結び付いたsnapshot fixtureは各directoryのbaseline取得が1回、scan終端の全件validationが0回であることを固定します。Full / expanded Update / MCPのconsumer fixtureはsuccessful pathのscan-input validationをfirst-write直前とreadiness直前の2回に固定し、first barrier failureではschema compatibility initialization後のdomain row・trust・evidence・purge・FTS recovery stateが不変、final barrier failureではpartialかつreadiness非stampとなることを要求します。ignored directoryのchurnはcleanのままです。 + MCP C# のpre-write coverageは通常requestとrebuild requestの両方でfirst snapshot-barrierの保持fixtureを実行し、rebuild破棄と中断FTS recoveryがbarrierより前に起きないことを固定します。`before_write`で注入したC# stat driftではpurgeを抑止し、旧C# checksumとimplicit-implementation referenceを保持し、source evidenceをunknownにして、failure phaseを`csharp_prepass`だけに限定します。fresh rebuild fixtureでもC# prepass symbol artifactを取り込まず、static-interface evidenceとimplicit referenceをauthoritativeに保ちます。 plain-interface purge fixtureではsorted file-ID preflightを正確かつcancellation-awareに保ち、prepared cacheへ残さないSQLite parameter 500件以内のbatchに固定します。transition-path preflight の coverage は実memberを2つ目のpath batchだけに置き、path-first の `files(path)` と `symbols(file_id, kind)` のindex plan、keyword境界・大小文字decoyの拒否、cancellationによる中断、dynamic commandがprepared cacheへ入らないことを要求します。one-sided の同一stem/checksum rename と positive/unknown evidence の `--changed-between` 範囲外 fixture はimmutable cleanup planがreference再生成前にstale contract rowを消し、無関係なmissing non-C# rowを全掃除しないことを要求します。`Run_UpdateFiles_ChangedExistingRetainedTargetPreplansMatchingCsharpAlias` は既にindex済みでstat-changedのretained targetもchecksum一致のC# aliasをworkspace構築前にplanし、未変更targetは追加content readなしで永続checksumを再利用することを固定します。binary/oversized skip-record drift fixtureは`BuildSkippedFileRecord`後にmutationし、retry前にnested cleanup/upsert transactionとbatch markerの両方がrollbackされることを要求します。`Run_UpdateMode_WithChangedBetween_PurgesMissingIndexedPathOutsideDiff_4056` は authoritative false evidence のときだけ従来の全言語 reconciliation を維持することを別途固定します。`Run_UpdateMode_WithChangedBetween_CleanupPathReappearingAfterScanIsDeferredUntilRetry` はdiscovery中に削除済みcontractを復元し、`Run_UpdateCommits_ExactCleanupPathReappearingAfterSnapshotBarrierPreservesPriorRow` はfinal snapshot barrier後にexactなGit削除targetを復元します。どちらも旧rowとimplicit referenceを保持するpartial resultを要求します。expanded discovery のfatal errorとprepass後のchange/delete fixtureも旧C# rowとimplicit referenceを保持し、source evidenceをunknownにして、後続のclean retryで収束させます。C# mutation延期中も無関係なPython/JavaScript cleanupと未変更scoped targetは進行します。parallel symbol-cap fixtureではprepass後に初めて見つかったcontractをsource evidence unknownとして、後続のcomplete repairを強制します。 - case-folded cleanup coverageではSQLite `NOCASE` とmanaged foldingを候補filterだけに使い、leaf case・ancestor case・Unicode foldが一致してもfile identityが異なるpathは両方のexact rowを保持し、真のcase-only aliasだけがchecksum readなしで旧spellingを削除できることを固定します。無関係なhardlinkをaliasとするにはpathとidentityの両方が同じretained case-fold bucketに一致しなければならず、`ScopedFileCleanupReappearance_FoldBucketsDoNotCrossMatchTargetIdentities` は別target由来のpath一致とidentity一致の合成を防ぎます。Git name-status coverageはold/new pathにtabと改行を含む実NUL区切りrenameを使い、commit/range helperの両方がunquotedな正確な文字列を返すことを検証します。 - project-marker budget の integration coverage は directory budget を最小境界に override し、child を 1 件だけ列挙します。warning 伝播の検証だけのために本番の 8,192-directory cap を実体化しないでください。shared discovery の coverage は1回の directory enumeration から C#、VB、F#、MSBuild の全 fingerprint が得られることを検証し、family-scope coverage は scan 後の ancestor filesystem probe が0回、fingerprint budget 枯渇後も complete な scope snapshot を利用可能、不完全 discovery では live fallback を維持することを固定してください。 @@ -1596,7 +1621,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" - C# constructor の base-target coverage は generic、interface list、nested generic、multiline、constraint 付き layout を1つの fixture で共有し、一意な末尾型ごとに書き換え後の edge を検証します。 - C# multiline member 帰属テストは expression-bodied method/property、Allman/same-line brace、途中の block comment を1つの fixture で共有し、一意な member container ごとに検証します。 - Python の `isinstance`/`issubclass` runtime type check は単一型と tuple 形式を1つの module fixture で共有し、function container ごとの検証を維持します。 -- Python class header の type reference は単一 base、複数 base、metaclass 形式を1つの module fixture で共有し、class container で区別します。 +- Python class header の type reference は単一 base、複数 base、metaclass 形式を1つの module fixture で共有して class container で区別し、専用の capped-output fixture で reference limit 到達後の multiline class-header remap も固定します。 - Python annotation coverage は direct/generic の return、parameter、local variable 形式を1つの module fixture で共有し、function container ごとの検証を維持します。 - Python semantic-type symbol coverage は PEP 695 alias、`TypeAlias`、`NewType`、`TypeVar`、`ParamSpec`、`TypeVarTuple`、実際の import を1つの module fixture で共有し、専用 kind と import filter からの除外を検証します。typing factory の reference coverage は一意な target type を持つ共有 module fixture を維持します。 - Python advanced typing coverage は multiline/comment 付き `TypeVar`、`ParamSpec`、callable annotation、variadic tuple unpack、literal union を一意な型名を持つ1つの logical-header fixture で共有します。 @@ -1640,7 +1665,8 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" - Go function-like symbol coverage は qualified receiver container と branch label を1つの file fixture で共有し、switch keyword を除外します。 - 統合した Python/Go extractor fixture でも、container ごとの完全な result set を含む symbol/reference の厳密な件数と、元の回帰契約に含まれていた source line の assertion を維持します。 - Assembly reference coverage は direct call/branch と tab 区切り decorated indirect-target の negative case を、厳密な件数を持つ1つの fixture で共有します。 -- Solidity reference coverage は inheritance whitespace variant、library/modifier/event/interface edge、comment/string negative を1つの fixture で共有します。 +- Solidity reference coverage は inheritance whitespace variant、library/modifier/event/interface edge、comment/string negative を1つの fixture で共有します。詳細 early-return fixture では definition と container の safety limit を同時に下げ、Solidity 固有抽出から戻る前に両方の diagnostic が維持されることを検証します。 +- Python mutual-recursion finalization coverage は、上限で除外される後続 call より前に完全な2-edge cycle を置き、reference list が既に満杯でも保持済み cycle が mutual として印付けされることを検証します。 - Terraform reference coverage は resource/module/data traversal と raw `var`/`local` object reference を、name ごとの厳密な件数を持つ1つの fixture で共有します。 - Language masker の copy-on-write coverage は、変更不要時の配列再利用と mask 必要時の clone を Lua/Solidity それぞれ同じテストメソッド内で検証します。 - COBOL target-statement coverage は SQL/CICS、report、sort、queue、file、literal、external-call variant を1つの program に配置し、group ごとの edge 件数を厳密に検証します。 @@ -1773,7 +1799,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" JVM graph fixture は Java type / module / method-reference、Kotlin type / infix / constructor、Scala contextual、Gradle / Groovy DSL の demand-driven match を検証します。cap-aware exit を追加するときも dense-line の順序を維持してください。 Python graph fixture は decorator argument、annotation、runtime type check、typing factory、dataclass / framework integration、dynamic import の逐次走査を検証します。`BoundedRegexTests.EnumerateMatches_InstanceRegex_StartsAtRequestedOffset` は decorator argument が prefix を再走査しない契約を固定し、`EnumerateMatches_InstanceRightToLeftRegex_PreservesDefaultStartAndOrder` は instance regex の既定方向を維持します。 `BoundedRegexTests` の direct instance-enumerator coverage はさらに、invalid `startAt` の遅延検証、zero-length の進行、`\G` continuation、right-to-left の明示開始位置、prefix 成功後に1回だけ記録して終了する timeout、suffix work を行わない早期 Dispose、未対応の `Reset`、意図的な LINQ 互換性を固定します。 - PHP、Ruby、R、Perl の graph fixture は attribute / docblock / type、DSL command target、namespace / member / resource reference、arrow call の逐次走査を検証します。nested token / type enumeration も cap-aware のままにしてください。 + PHP、Ruby、R、Perl の graph fixture は attribute / docblock / type、DSL command target、namespace / member / resource reference、arrow call の逐次走査を検証します。nested token / type enumeration も cap-aware のままにしてください。`ReferenceExtractorTests.Extract_PhpDocblockMethod_ReturnPrecedesParametersAtReferenceLimit` は PHP docblock preamble の順序と container state を固定し、`@method` の return type が parameter type より先に残り1枠を消費しつつ、enclosing class の container snapshot を保持することを検証します。 secondary-language graph fixture は Fortran、Visual Basic、F#、Pascal、Objective-C、Haskell、Elixir、Smalltalk、Lua、Dart、Razor、JSON、JavaScript、GitHub Actions、C++ compound requirement の逐次走査を検証します。`BoundedRegexTests.EnumerateMatches_StaticPatternCustomTimeout_ReturnsEmpty` と `EnumerateMatches_StaticPatternCustomTimeout_StopsAfterConsumerBreak` は、明示 timeout の失敗時挙動と早期破棄を固定します。 `PerformanceTests.ReferenceExtraction_BoundedDenseFSharpPipeline_StopsAtCapacity` は、行 phase の引き継ぎと action-based call emitter をまたぐ bounded-list 契約を固定します。上限1の4,000段 F# pipeline は未使用の段を列挙してはなりません。`PerformanceTests.ReferenceMatchEnumeration_BoundedListDoesNotRequestMatchAfterCapacity` と `ReferenceMatchEnumeration_ConcreteRegexDoesNotRequestSuffixAfterCapacity` は、generic / concrete wrapper が上限到達後に下位 enumerator の次の `MoveNext()` を呼ばないことを固定します。`ReferenceMatchEnumeration_BelowCapacity_DoesNotAllocateWrapperEnumerators` は通常の generic 上限未到達 scan 10,000回で wrapper enumerator の heap allocation がないことを固定します。blocking な `net8.0` test `BoundedRegexEnumeration_DirectNoMatchScansDoNotAllocateEnumerators` と `ReferenceMatchEnumeration_PrefilledCapacityDoesNotAllocateOrLookAhead` は、direct no-match と既に満杯の cap 経路をそれぞれ10,000回走査し、allocation を1,024 bytes未満に固定します。 `PerformanceTests.ReferenceExtraction_PrologCallFreeRules_AvoidsPerLineLists` は8,000件の call-free Prolog rule で empty goal list と populated directive list の copy を防ぎます。`net8.0` の allocation budget は blocking です。 diff --git a/changelog.d/unreleased/+batch-output-limit-replay.fixed.md b/changelog.d/unreleased/+batch-output-limit-replay.fixed.md new file mode 100644 index 0000000000..db31b2247a --- /dev/null +++ b/changelog.d/unreleased/+batch-output-limit-replay.fixed.md @@ -0,0 +1,14 @@ +--- +category: fixed +affected: + - src/CodeIndex/Cli/QueryCommandRunner.Batch.cs + - src/CodeIndex/Cli/QueryCommandRunner.BatchParallelExecution.cs +--- + +## English + +- **Parallel batch output limits no longer discard accepted, undispatched input records** — when `batch --parallel` reaches `--max-output-chars`, accepted nonblank input records that were not dispatched are returned to the shared bounded input pump in source order. The next batch invocation reparses those records with fresh line numbers and counters, ahead of input the pump already buffered, while commands that started remain owned by the first invocation. + +## 日本語 + +- **parallel batch の出力上限で、受理済みかつ未 dispatch の input record が失われなくなりました** — `batch --parallel` が `--max-output-chars` に達した場合、受理済みでも未 dispatch の nonblank input record を source 順のまま共有の bounded input pump に戻します。次の batch invocation は、すでに pump が buffer した入力より先にそれらの record を再 parse し、新しい line number と counter を割り当てます。開始済みの command は最初の invocation の所有のままです。 diff --git a/changelog.d/unreleased/+csharp-base-list-parser.fixed.md b/changelog.d/unreleased/+csharp-base-list-parser.fixed.md new file mode 100644 index 0000000000..ef5bef604e --- /dev/null +++ b/changelog.d/unreleased/+csharp-base-list-parser.fixed.md @@ -0,0 +1,16 @@ +--- +category: fixed +affected: + - src/CodeIndex/Database/CSharpBaseListParser.cs + - src/CodeIndex/Database/DbContext.ConnectionFunctions.cs + - src/CodeIndex/Database/DbReader.CSharpResolution.cs + - src/CodeIndex/Database/DbWriter.CSharpMetadataTargets.cs +--- + +## English + +- **C# inheritance resolution now shares one grammar-aware base-list parser** — exact caller and metadata-target resolution no longer confuse generic `where` constraints with base types, and nested generics, tuples, arrays, alias qualifiers, and primary/base-constructor syntax are split consistently. + +## 日本語 + +- **C# 継承解決が grammar-aware な base-list parser を共有するようになりました** — exact caller と metadata-target の解決で generic `where` constraint を基底型と誤認せず、nested generic、tuple、array、alias qualifier、primary / base constructor 構文を一貫して分割します。 diff --git a/changelog.d/unreleased/+full-scan-extraction-state.fixed.md b/changelog.d/unreleased/+full-scan-extraction-state.fixed.md new file mode 100644 index 0000000000..5891f6f867 --- /dev/null +++ b/changelog.d/unreleased/+full-scan-extraction-state.fixed.md @@ -0,0 +1,15 @@ +--- +category: fixed +affected: + - src/CodeIndex/Cli/IndexCommandRunner.FullScan.ExtractionPipeline.cs + - src/CodeIndex/Cli/IndexCommandRunner.FullScan.ResultConsumer.cs + - tests/CodeIndex.Tests/IndexCommandRunnerFullScanTests.cs +--- + +## English + +- **Full-scan extraction now preserves shared state and worker-resource ownership** — a later C# workspace drift no longer drops text-index mutations already committed for earlier files, and a fatal parallel worker result defers shared queue, cache, hook, and cancellation cleanup until every peer worker has stopped. + +## 日本語 + +- **full-scan extraction が共有 state と worker resource の ownership を保持するようになりました** — 後段の C# workspace drift が先行 file で commit 済みの text-index mutation を失わず、parallel worker の fatal result では全 peer worker の停止まで共有 queue、cache、hook、cancellation の cleanup を延期します。 diff --git a/changelog.d/unreleased/+unused-legacy-schema.fixed.md b/changelog.d/unreleased/+unused-legacy-schema.fixed.md new file mode 100644 index 0000000000..94d0123f5d --- /dev/null +++ b/changelog.d/unreleased/+unused-legacy-schema.fixed.md @@ -0,0 +1,15 @@ +--- +category: fixed +affected: + - src/CodeIndex/Database/DbReader.cs + - src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.Candidates.cs + - src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.cs +--- + +## English + +- **C# unused-symbol queries now remain safe on read-only legacy databases** — partial-type use filtering is applied before paging without issuing per-candidate database queries, while databases whose legacy chunk or symbol tables lack the required columns retain the existing degraded fallback instead of failing during SQL preparation. + +## 日本語 + +- **C# の unused-symbol query が read-only legacy database でも安全に動作するようになりました** — partial type の使用判定を candidate ごとの database query なしで paging 前に適用し、必要な column が legacy chunk / symbol table にない database では SQL prepare で失敗せず、従来の degraded fallback を維持します。 diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FileByteTracking.cs b/src/CodeIndex/Cli/IndexCommandRunner.FileByteTracking.cs index 6f32888341..8ed6eb069c 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.FileByteTracking.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.FileByteTracking.cs @@ -93,94 +93,113 @@ or NotSupportedException } } - private static bool ShouldUseFullScanFtsBulkLoad( - bool rebuild, - bool startedWithNoIndexedFiles, - int extractionWorkItemCount, - FilePurgePlan staleFilePurgePlan, - bool scanHadErrors, - ReadableFileByteTracker readableFileBytes, - ReusableIndexedFileStatsSnapshot reusableIndexedFileStats, - IReadOnlyList fileTargets, - IReadOnlyList? extractionFileIndexes, - Action throwIfCancelled) + private sealed partial class FullScanPreWriteSession { - if (rebuild || startedWithNoIndexedFiles) - return true; - if (extractionWorkItemCount == 0 && staleFilePurgePlan.Count == 0) - return false; - - var dirtyBytes = staleFilePurgePlan.DeletedBytes; - var persistedSizeExcessBytes = 0L; - var byteEstimateComplete = !scanHadErrors - && staleFilePurgePlan.ByteEstimateComplete - && readableFileBytes.EstimateComplete; - - void AddDirtyFileBytes(int fileIndex) + internal void DecideFtsBulkLoad() { - throwIfCancelled(); - try + var request = Request; + var options = request.Core.Options; + var baseline = request.Baseline; + var runtime = request.Runtime; + var scan = State.Scan; + var csharp = State.CSharp; + var selection = State.Selection; + if (options.Rebuild || baseline.StartedWithNoIndexedFiles) { - var target = fileTargets[fileIndex]; - var info = new FileInfo(target.FilePath); - if (!info.Exists || info.Length < 0) + selection.UseFtsBulkLoad = true; + return; + } + + if (selection.ExtractionWorkItemCount == 0 + && scan.StaleFilePurgePlan.Count == 0) + { + selection.UseFtsBulkLoad = false; + return; + } + + var dirtyBytes = scan.StaleFilePurgePlan.DeletedBytes; + var persistedSizeExcessBytes = 0L; + var byteEstimateComplete = !baseline.ScanHadErrors + && scan.StaleFilePurgePlan.ByteEstimateComplete + && selection.ReadableFileBytes.EstimateComplete; + + void AddDirtyFileBytes(int fileIndex) + { + ThrowIfFullScanCancelled(); + try + { + var target = runtime.FileTargets[fileIndex]; + var info = new FileInfo(target.FilePath); + if (!info.Exists || info.Length < 0) + { + byteEstimateComplete = false; + return; + } + + selection.ReadableFileBytes.Remember( + fileIndex, + info.Length); + var persistedSize = csharp.ReusableIndexedFileStats! + .GetPersistedSize(target.IndexPath); + if (!FtsBulkLoadTriggerGuard.TryAccumulateDirtyFileBytes( + dirtyBytes, + persistedSizeExcessBytes, + info.Length, + persistedSize, + out dirtyBytes, + out persistedSizeExcessBytes)) + { + byteEstimateComplete = false; + } + } + catch (Exception ex) when ( + ex is IOException + or UnauthorizedAccessException + or NotSupportedException + or ArgumentException) { byteEstimateComplete = false; - return; } + } - readableFileBytes.Remember(fileIndex, info.Length); - var persistedSize = reusableIndexedFileStats.GetPersistedSize(target.IndexPath); - if (!FtsBulkLoadTriggerGuard.TryAccumulateDirtyFileBytes( - dirtyBytes, - persistedSizeExcessBytes, - info.Length, - persistedSize, - out dirtyBytes, - out persistedSizeExcessBytes)) + if (selection.ExtractionFileIndexes != null) + { + foreach (var fileIndex in selection.ExtractionFileIndexes) + AddDirtyFileBytes(fileIndex); + } + else + { + for (var fileIndex = 0; + fileIndex < runtime.FileTargets.Length; + fileIndex++) { - byteEstimateComplete = false; + AddDirtyFileBytes(fileIndex); } } - catch (Exception ex) when ( - ex is IOException - or UnauthorizedAccessException - or NotSupportedException - or ArgumentException) + + byteEstimateComplete &= + selection.ReadableFileBytes.EstimateComplete; + var totalBytes = selection.ReadableFileBytes.KnownBytes; + if (!selection.ReadableFileBytes.EstimateComplete + || totalBytes + > long.MaxValue - scan.StaleFilePurgePlan.DeletedBytes) { byteEstimateComplete = false; } - } + else + { + totalBytes += scan.StaleFilePurgePlan.DeletedBytes; + } - if (extractionFileIndexes != null) - { - foreach (var fileIndex in extractionFileIndexes) - AddDirtyFileBytes(fileIndex); - } - else - { - for (var fileIndex = 0; fileIndex < fileTargets.Count; fileIndex++) - AddDirtyFileBytes(fileIndex); - } + if (totalBytes > long.MaxValue - persistedSizeExcessBytes) + byteEstimateComplete = false; + else + totalBytes += persistedSizeExcessBytes; - byteEstimateComplete &= readableFileBytes.EstimateComplete; - var totalBytes = readableFileBytes.KnownBytes; - if (!readableFileBytes.EstimateComplete - || totalBytes > long.MaxValue - staleFilePurgePlan.DeletedBytes) - { - byteEstimateComplete = false; - } - else - { - totalBytes += staleFilePurgePlan.DeletedBytes; + selection.UseFtsBulkLoad = byteEstimateComplete + && FtsBulkLoadTriggerGuard.ShouldUseForDirtyBytes( + dirtyBytes, + totalBytes); } - - if (totalBytes > long.MaxValue - persistedSizeExcessBytes) - byteEstimateComplete = false; - else - totalBytes += persistedSizeExcessBytes; - - return byteEstimateComplete - && FtsBulkLoadTriggerGuard.ShouldUseForDirtyBytes(dirtyBytes, totalBytes); } } diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.CSharpFinalRevalidation.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.CSharpFinalRevalidation.cs index a637c8e146..b7c25beba6 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.CSharpFinalRevalidation.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.CSharpFinalRevalidation.cs @@ -5,247 +5,165 @@ namespace CodeIndex.Cli; public static partial class IndexCommandRunner { - private sealed class FullScanCSharpFinalRevalidationContext + private sealed partial class FullScanPreWriteSession { - internal required DbWriter Writer { get; init; } - internal required FileIndexer Indexer { get; init; } - internal required IndexCommandOptions Options { get; init; } - internal required string ProjectRoot { get; init; } - internal required FullScanFileTarget[] FileTargets { get; init; } - internal required IReadOnlyList - CSharpPrepassTargets - { get; init; } - internal required FilePurgePlan StaleFilePurgePlan { get; init; } - internal required bool StartedWithNoIndexedFiles { get; init; } - internal required bool CSharpIndexedProjectRootCompatible + internal void RevalidateFinalCSharpNoOp() { - get; - init; - } - - internal required int ExtractionParallelism { get; init; } - internal required bool? PriorCSharpStaticInterfaceSourceEvidence - { - get; - init; - } - - internal required ReusableIndexedFileStatsSnapshot - ReusableIndexedFileStats - { get; init; } - internal List? ExtractionFileIndexes { get; init; } - internal required int ExtractionWorkItemCount { get; init; } - internal required bool UseFtsBulkLoad { get; init; } - internal required CSharpStaticInterfaceWorkspaceSymbols CSharpWorkspace - { - get; - init; - } + var csharp = State.CSharp; + var selection = State.Selection; + if (!csharp.PreservePriorPositiveSourceNoOp + || (selection.ExtractionWorkItemCount == 0 + && State.Scan.StaleFilePurgePlan.Count == 0)) + { + return; + } - internal Dictionary? - CSharpWorkspaceFileSnapshots - { get; init; } - internal required bool ForceFullCSharpRefreshFromInvalidatedNoOp - { - get; - init; + // The dirty-byte pass can be long on a mixed-language monorepo. Revalidate C# + // once more at the final read-only boundary, then undo tentative stat skips + // and promote every affected C# target if any source changed. + // mixed-language dirty-byte pass後の最終read-only境界でC#を再statする。 + FullScanCSharpFinalStatRevalidationForTesting?.Invoke(); + var invalidatedCSharpFileIndexes = + FindInvalidatedFinalCSharpTargets(); + if (invalidatedCSharpFileIndexes.Count == 0) + return; + + RebuildFinalFullScanCSharpWorkspace( + invalidatedCSharpFileIndexes); } - internal required bool PreservePriorPositiveCSharpSourceNoOp + private List FindInvalidatedFinalCSharpTargets() { - get; - init; - } - - internal required bool CSharpSourceEvidenceForStamp { get; init; } - internal required bool CSharpSourceEvidenceComplete { get; init; } - internal required CancellationToken CancellationToken { get; init; } - internal required Action - DeferCSharpMutationsForIncompleteWorkspace - { get; init; } - internal required Func - IsExistingCSharpSymbolPathNowNonCSharp - { get; init; } - } - - private sealed record FullScanCSharpFinalRevalidationResult( - List? ExtractionFileIndexes, - int ExtractionWorkItemCount, - bool UseFtsBulkLoad, - CSharpStaticInterfaceWorkspaceSymbols CSharpWorkspace, - Dictionary? - CSharpWorkspaceFileSnapshots, - bool ForceFullCSharpRefreshFromInvalidatedNoOp, - bool PreservePriorPositiveCSharpSourceNoOp, - bool CSharpSourceEvidenceForStamp, - bool CSharpSourceEvidenceComplete, - int PromotedCSharpTargetCount, - bool PromotedAllCSharpTargets); - - private static FullScanCSharpFinalRevalidationResult - RevalidateFinalFullScanCSharpNoOp( - FullScanCSharpFinalRevalidationContext context) - { - if (!context.PreservePriorPositiveCSharpSourceNoOp - || (context.ExtractionWorkItemCount == 0 - && context.StaleFilePurgePlan.Count == 0)) - { - return BuildUnchangedFinalCSharpRevalidationResult(context); - } - - // The dirty-byte pass can be long on a mixed-language monorepo. Revalidate C# - // once more at the final read-only boundary, then undo tentative stat skips - // and promote every affected C# target if any source changed. - // mixed-language dirty-byte pass後の最終read-only境界でC#を再statする。 - FullScanCSharpFinalStatRevalidationForTesting?.Invoke(); - var invalidatedCSharpFileIndexes = FindInvalidatedFinalCSharpTargets( - context); - if (invalidatedCSharpFileIndexes.Count == 0) - return BuildUnchangedFinalCSharpRevalidationResult(context); - - return RebuildFinalFullScanCSharpWorkspace( - context, - invalidatedCSharpFileIndexes); - } - - private static List FindInvalidatedFinalCSharpTargets( - FullScanCSharpFinalRevalidationContext context) - { - var invalidatedCSharpFileIndexes = new List(); - for (var fileIndex = 0; - fileIndex < context.FileTargets.Length; - fileIndex++) - { - var target = context.FileTargets[fileIndex]; - if (target.Language != "csharp") - continue; - - context.CancellationToken.ThrowIfCancellationRequested(); - if (IndexedFileStatReuse.TryGetReusableUnchangedFile( - context.ReusableIndexedFileStats, - target.FilePath, - target.IndexPath, - target.Language, - target.GeneratedExtractionSuppressed) == null) + var runtime = Request.Runtime; + var invalidatedCSharpFileIndexes = new List(); + for (var fileIndex = 0; + fileIndex < runtime.FileTargets.Length; + fileIndex++) { - invalidatedCSharpFileIndexes.Add(fileIndex); + var target = runtime.FileTargets[fileIndex]; + if (target.Language != "csharp") + continue; + + runtime.CancellationToken.ThrowIfCancellationRequested(); + if (IndexedFileStatReuse.TryGetReusableUnchangedFile( + State.CSharp.ReusableIndexedFileStats!, + target.FilePath, + target.IndexPath, + target.Language, + target.GeneratedExtractionSuppressed) == null) + { + invalidatedCSharpFileIndexes.Add(fileIndex); + } } - } - return invalidatedCSharpFileIndexes; - } + return invalidatedCSharpFileIndexes; + } - private static FullScanCSharpFinalRevalidationResult - RebuildFinalFullScanCSharpWorkspace( - FullScanCSharpFinalRevalidationContext context, + private void RebuildFinalFullScanCSharpWorkspace( IReadOnlyList invalidatedCSharpFileIndexes) - { - var workspace = BuildStableFullScanCSharpWorkspace( - context.ProjectRoot, - context.CSharpPrepassTargets, - out var workspaceFileSnapshots, - () => CSharpStaticInterfacePrepass.BuildWorkspaceSymbols( - context.Writer, - context.Indexer, - context.CSharpPrepassTargets, - includeExistingSymbols: - context.CSharpIndexedProjectRootCompatible - && !context.Options.Rebuild - && !context.StartedWithNoIndexedFiles, - canReuseExistingSymbolsWithoutRead: null, - parallelism: context.ExtractionParallelism, - excludedExistingFileIds: - context.StaleFilePurgePlan.FileIds, - isExistingSymbolPathExcluded: - context.IsExistingCSharpSymbolPathNowNonCSharp, - patternConfigsAlreadyLoaded: true, - cancellationToken: context.CancellationToken), - context.CancellationToken); - if (!workspace.SourceContractEvidenceComplete) { - var incompleteSourcePaths = workspace.IncompleteSourcePaths; - context.DeferCSharpMutationsForIncompleteWorkspace(workspace); - return new FullScanCSharpFinalRevalidationResult( - context.ExtractionFileIndexes, - context.ExtractionWorkItemCount, - UseFtsBulkLoad: false, - new CSharpStaticInterfaceWorkspaceSymbols( - [], - false, - SourceContractEvidenceComplete: false, - IncompleteSourcePaths: incompleteSourcePaths), - workspaceFileSnapshots, - context.ForceFullCSharpRefreshFromInvalidatedNoOp, - PreservePriorPositiveCSharpSourceNoOp: false, - CSharpSourceEvidenceForStamp: false, - CSharpSourceEvidenceComplete: false, - PromotedCSharpTargetCount: 0, - PromotedAllCSharpTargets: false); - } + var request = Request; + var core = request.Core; + var baseline = request.Baseline; + var contracts = request.Contracts; + var runtime = request.Runtime; + var scan = State.Scan; + var csharp = State.CSharp; + var selection = State.Selection; + var workspace = BuildStableFullScanCSharpWorkspace( + core.ProjectRoot, + runtime.CSharpPrepassTargets, + out var workspaceFileSnapshots, + () => CSharpStaticInterfacePrepass.BuildWorkspaceSymbols( + core.Writer, + core.Indexer, + runtime.CSharpPrepassTargets, + includeExistingSymbols: + contracts.CSharpIndexedProjectRootCompatible + && !core.Options.Rebuild + && !baseline.StartedWithNoIndexedFiles, + canReuseExistingSymbolsWithoutRead: null, + parallelism: runtime.ExtractionParallelism, + excludedExistingFileIds: + scan.StaleFilePurgePlan.FileIds, + isExistingSymbolPathExcluded: + IsExistingCSharpSymbolPathNowNonCSharp, + patternConfigsAlreadyLoaded: true, + cancellationToken: runtime.CancellationToken), + runtime.CancellationToken); + if (!workspace.SourceContractEvidenceComplete) + { + var incompleteSourcePaths = workspace.IncompleteSourcePaths; + DeferCSharpMutationsForIncompleteWorkspace(workspace); + selection.UseFtsBulkLoad = false; + csharp.Workspace = + new CSharpStaticInterfaceWorkspaceSymbols( + [], + false, + SourceContractEvidenceComplete: false, + IncompleteSourcePaths: incompleteSourcePaths); + csharp.WorkspaceFileSnapshots = workspaceFileSnapshots; + csharp.PreservePriorPositiveSourceNoOp = false; + csharp.Evidence.ForStamp = false; + csharp.Evidence.Complete = false; + return; + } - var requiresFullCSharpRefresh = - context.PriorCSharpStaticInterfaceSourceEvidence == true - || workspace.HasStaticInterfaceContracts; - IReadOnlyList csharpFileIndexesToRefresh; - if (requiresFullCSharpRefresh) - { - workspace = workspace with + var requiresFullCSharpRefresh = + baseline.PriorCSharpStaticInterfaceSourceEvidence == true + || workspace.HasStaticInterfaceContracts; + IReadOnlyList csharpFileIndexesToRefresh; + if (requiresFullCSharpRefresh) { - HasStaticInterfaceContracts = true, - }; - var allCSharpFileIndexes = - new List(context.CSharpPrepassTargets.Count); - for (var fileIndex = 0; - fileIndex < context.FileTargets.Length; - fileIndex++) + workspace = workspace with + { + HasStaticInterfaceContracts = true, + }; + var allCSharpFileIndexes = + new List(runtime.CSharpPrepassTargets.Count); + for (var fileIndex = 0; + fileIndex < runtime.FileTargets.Length; + fileIndex++) + { + if (runtime.FileTargets[fileIndex].Language == "csharp") + allCSharpFileIndexes.Add(fileIndex); + } + + csharpFileIndexesToRefresh = allCSharpFileIndexes; + } + else { - if (context.FileTargets[fileIndex].Language == "csharp") - allCSharpFileIndexes.Add(fileIndex); + // A previously authoritative negative workspace only needs the + // stat-invalidated files when the raw fallback is still negative. + // prior negative のraw fallbackもnegativeなら変更fileだけを更新する。 + csharpFileIndexesToRefresh = invalidatedCSharpFileIndexes; } - csharpFileIndexesToRefresh = allCSharpFileIndexes; - } - else - { - // A previously authoritative negative workspace only needs the - // stat-invalidated files when the raw fallback is still negative. - // prior negative のraw fallbackもnegativeなら変更fileだけを更新する。 - csharpFileIndexesToRefresh = invalidatedCSharpFileIndexes; - } + var extractionFileIndexes = selection.ExtractionFileIndexes + ?? new List(csharpFileIndexesToRefresh.Count); + foreach (var fileIndex in csharpFileIndexesToRefresh) + extractionFileIndexes.Add(fileIndex); + extractionFileIndexes.Sort(); + selection.ExtractionFileIndexes = extractionFileIndexes; + selection.ExtractionWorkItemCount = extractionFileIndexes.Count; + selection.UseFtsBulkLoad = false; + selection.Skipped -= csharpFileIndexesToRefresh.Count; + selection.Processed -= csharpFileIndexesToRefresh.Count; + if (csharpFileIndexesToRefresh.Count + == runtime.CSharpPrepassTargets.Count) + { + selection.SkippedSymbolExtractorLanguages?.Remove("csharp"); + selection.ReusedHotspotFamilyLanguages?.Remove("csharp"); + } - var extractionFileIndexes = context.ExtractionFileIndexes - ?? new List(csharpFileIndexesToRefresh.Count); - foreach (var fileIndex in csharpFileIndexesToRefresh) - extractionFileIndexes.Add(fileIndex); - extractionFileIndexes.Sort(); - return new FullScanCSharpFinalRevalidationResult( - extractionFileIndexes, - extractionFileIndexes.Count, - UseFtsBulkLoad: false, - workspace, - workspaceFileSnapshots, - requiresFullCSharpRefresh, - PreservePriorPositiveCSharpSourceNoOp: false, - workspace.HasSourceStaticInterfaceContracts, - CSharpSourceEvidenceComplete: true, - csharpFileIndexesToRefresh.Count, - csharpFileIndexesToRefresh.Count - == context.CSharpPrepassTargets.Count); + csharp.Workspace = workspace; + csharp.WorkspaceFileSnapshots = workspaceFileSnapshots; + csharp.ForceFullRefreshFromInvalidatedNoOp = + requiresFullCSharpRefresh; + csharp.PreservePriorPositiveSourceNoOp = false; + csharp.Evidence.ForStamp = + workspace.HasSourceStaticInterfaceContracts; + csharp.Evidence.Complete = true; + } } - - private static FullScanCSharpFinalRevalidationResult - BuildUnchangedFinalCSharpRevalidationResult( - FullScanCSharpFinalRevalidationContext context) - => new( - context.ExtractionFileIndexes, - context.ExtractionWorkItemCount, - context.UseFtsBulkLoad, - context.CSharpWorkspace, - context.CSharpWorkspaceFileSnapshots, - context.ForceFullCSharpRefreshFromInvalidatedNoOp, - context.PreservePriorPositiveCSharpSourceNoOp, - context.CSharpSourceEvidenceForStamp, - context.CSharpSourceEvidenceComplete, - PromotedCSharpTargetCount: 0, - PromotedAllCSharpTargets: false); } diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.CSharpPreflight.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.CSharpPreflight.cs index a02960a217..2af16d949e 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.CSharpPreflight.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.CSharpPreflight.cs @@ -5,125 +5,116 @@ namespace CodeIndex.Cli; public static partial class IndexCommandRunner { - private sealed class FullScanCSharpPreflightContext + private sealed partial class FullScanPreWriteSession { - internal required DbWriter Writer { get; init; } - internal required FileIndexer Indexer { get; init; } - internal required IndexCommandOptions Options { get; init; } - internal required string ProjectRoot { get; init; } - internal required FullScanFileTarget[] FileTargets { get; init; } - internal required IReadOnlyList CSharpPrepassTargets { get; init; } - internal required int CSharpPrepassCapacity { get; init; } - internal required FilePurgePlan StaleFilePurgePlan { get; init; } - internal required bool StartedWithNoIndexedFiles { get; init; } - internal required bool PriorIndexComplete { get; init; } - internal required int PriorReadiness { get; init; } - internal required bool ScanHadErrors { get; init; } - internal required bool ForceExtractorRefresh { get; init; } - internal required bool PriorSymbolsOnlyGraphOmitted { get; init; } - internal required bool SymbolKindFilterMatchesPrior { get; init; } - internal required bool CSharpSymbolNameContractMatchesCurrent { get; init; } - internal required bool CSharpIndexedProjectRootCompatible { get; init; } - internal required bool CSharpHotspotTrustMatchesCurrent { get; init; } - internal required bool RequiresConservativeCSharpSourceRefresh { get; init; } - internal required bool HadCSharpStaticInterfaceContractsBeforePurge { get; init; } - internal required bool? PriorCSharpStaticInterfaceSourceEvidence { get; init; } - internal required bool ProjectRootWritten { get; init; } - internal required int ExtractionParallelism { get; init; } - internal required int FilesCount { get; init; } - internal required string ActualMode { get; init; } - internal required CancellationToken CancellationToken { get; init; } - internal required Func IsExistingCSharpSymbolPathNowNonCSharp { get; init; } - internal required Func GetDeferCSharpMutationsForIncompleteScan { get; init; } - internal required Func GetPurged { get; init; } - internal required Action DeferCSharpMutationsForIncompleteWorkspace { get; init; } - } - - private sealed record FullScanCSharpPreflightResult( - ReusableIndexedFileStatsSnapshot? ReusableIndexedFileStats, - Dictionary? CSharpPrepassStatReuse, - Dictionary? CSharpWorkspaceFileSnapshots, - CSharpStaticInterfaceWorkspaceSymbols CSharpWorkspace, - CSharpPrepassSymbolArtifactCache? CSharpPrepassSymbolArtifacts, - bool ForceFullCSharpRefreshFromInvalidatedNoOp, - bool PreservePriorPositiveCSharpSourceNoOp, - bool CSharpSourceEvidenceForStamp, - bool CSharpSourceEvidenceComplete); - - private static FullScanCSharpPreflightResult PrepareFullScanCSharpWorkspace( - FullScanCSharpPreflightContext context) - { - var writer = context.Writer; - var options = context.Options; - var cancellationToken = context.CancellationToken; - HashSet? retainedPathsForReuse = null; - if (!options.Rebuild - && !context.StartedWithNoIndexedFiles - && context.StaleFilePurgePlan.RemainingFileCount - - context.FileTargets.LongLength - > context.FileTargets.LongLength) + internal void PrepareCSharpWorkspace() { - retainedPathsForReuse = new HashSet( - context.FileTargets.Length, - StringComparer.Ordinal); - foreach (var target in context.FileTargets) - retainedPathsForReuse.Add(target.IndexPath); - } + var request = Request; + var core = request.Core; + var baseline = request.Baseline; + var contracts = request.Contracts; + var runtime = request.Runtime; + var scan = State.Scan; + var csharp = State.CSharp; + var writer = core.Writer; + var options = core.Options; + var cancellationToken = runtime.CancellationToken; + HashSet? retainedPathsForReuse = null; + if (!options.Rebuild + && !baseline.StartedWithNoIndexedFiles + && scan.StaleFilePurgePlan.RemainingFileCount + - runtime.FileTargets.LongLength + > runtime.FileTargets.LongLength) + { + retainedPathsForReuse = new HashSet( + runtime.FileTargets.Length, + StringComparer.Ordinal); + foreach (var target in runtime.FileTargets) + retainedPathsForReuse.Add(target.IndexPath); + } - var csharpPositiveNoOpPolicyCandidate = !options.SymbolsOnly - && context.PriorCSharpStaticInterfaceSourceEvidence is not null - && context.PriorIndexComplete - && (context.PriorReadiness & DbContext.GraphReadyFlag) != 0 - && !context.ScanHadErrors - && !context.HadCSharpStaticInterfaceContractsBeforePurge - && !context.ForceExtractorRefresh - && !context.PriorSymbolsOnlyGraphOmitted - && context.SymbolKindFilterMatchesPrior - && context.CSharpSymbolNameContractMatchesCurrent - && context.CSharpIndexedProjectRootCompatible - && context.CSharpHotspotTrustMatchesCurrent - && context.CSharpPrepassTargets.Count > 0; - var hasCSharpLanguageTransitions = false; - void ObservePersistedCSharpPath(string indexPath) - { - if (!hasCSharpLanguageTransitions - && context.IsExistingCSharpSymbolPathNowNonCSharp(indexPath)) + var csharpPositiveNoOpPolicyCandidate = !options.SymbolsOnly + && baseline.PriorCSharpStaticInterfaceSourceEvidence is not null + && baseline.PriorIndexComplete + && (baseline.PriorReadiness & DbContext.GraphReadyFlag) != 0 + && !baseline.ScanHadErrors + && !scan.HadCSharpStaticInterfaceContractsBeforePurge + && !contracts.ForceExtractorRefresh + && !baseline.PriorSymbolsOnlyGraphOmitted + && contracts.SymbolKindFilterMatchesPrior + && contracts.CSharpSymbolNameContractMatchesCurrent + && contracts.CSharpIndexedProjectRootCompatible + && contracts.CSharpHotspotTrustMatchesCurrent + && runtime.CSharpPrepassTargets.Count > 0; + var hasCSharpLanguageTransitions = false; + void ObservePersistedCSharpPath(string indexPath) { - hasCSharpLanguageTransitions = true; + if (!hasCSharpLanguageTransitions + && IsExistingCSharpSymbolPathNowNonCSharp(indexPath)) + { + hasCSharpLanguageTransitions = true; + } } - } - var reusableIndexedFileStats = - !options.Rebuild && !context.StartedWithNoIndexedFiles - ? writer.LoadReusableIndexedFileStats( - options.MaxSymbolsPerFile, - options.MaxReferencesPerFile, - cancellationToken, - context.FileTargets.Length, - retainedPathsForReuse, - context.StaleFilePurgePlan.FileIds, - csharpPositiveNoOpPolicyCandidate - ? ObservePersistedCSharpPath - : null, - maxFileSizeBytes: - options.MaxFileSizeBytes ?? FileIndexer.DefaultMaxFileSizeBytes) - : null; - Dictionary? - csharpPrepassStatReuse = null; - var priorPositiveCSharpSourceNoOpCandidate = - csharpPositiveNoOpPolicyCandidate - && !hasCSharpLanguageTransitions; - var allCSharpPrepassTargetsReusable = false; - if (priorPositiveCSharpSourceNoOpCandidate) - { - allCSharpPrepassTargetsReusable = true; - csharpPrepassStatReuse = - new Dictionary( - context.CSharpPrepassCapacity, - StringComparer.Ordinal); - foreach (var target in context.CSharpPrepassTargets) + var reusableIndexedFileStats = + !options.Rebuild && !baseline.StartedWithNoIndexedFiles + ? writer.LoadReusableIndexedFileStats( + options.MaxSymbolsPerFile, + options.MaxReferencesPerFile, + cancellationToken, + runtime.FileTargets.Length, + retainedPathsForReuse, + scan.StaleFilePurgePlan.FileIds, + csharpPositiveNoOpPolicyCandidate + ? ObservePersistedCSharpPath + : null, + maxFileSizeBytes: + options.MaxFileSizeBytes ?? FileIndexer.DefaultMaxFileSizeBytes) + : null; + Dictionary? + csharpPrepassStatReuse = null; + var priorPositiveCSharpSourceNoOpCandidate = + csharpPositiveNoOpPolicyCandidate + && !hasCSharpLanguageTransitions; + var allCSharpPrepassTargetsReusable = false; + if (priorPositiveCSharpSourceNoOpCandidate) + { + allCSharpPrepassTargetsReusable = true; + csharpPrepassStatReuse = + new Dictionary( + runtime.CSharpPrepassCapacity, + StringComparer.Ordinal); + foreach (var target in runtime.CSharpPrepassTargets) + { + cancellationToken.ThrowIfCancellationRequested(); + var existingFile = + IndexedFileStatReuse.TryGetReusableUnchangedFile( + reusableIndexedFileStats!, + target.FilePath, + target.IndexPath, + target.Language, + target.GeneratedExtractionSuppressed); + csharpPrepassStatReuse[target.IndexPath] = existingFile; + allCSharpPrepassTargetsReusable &= existingFile != null; + } + } + + bool CanReuseCSharpPrepassTargetWithoutRead( + CSharpStaticInterfacePrepass.FileTarget target) { - cancellationToken.ThrowIfCancellationRequested(); + if (contracts.ForceExtractorRefresh + || options.Rebuild + || baseline.StartedWithNoIndexedFiles + || !baseline.ProjectRootWritten + || (contracts.RequiresConservativeCSharpSourceRefresh + && !priorPositiveCSharpSourceNoOpCandidate) + || !contracts.SymbolKindFilterMatchesPrior + || !contracts.CSharpSymbolNameContractMatchesCurrent + || target.Language != "csharp") + { + return false; + } + var existingFile = IndexedFileStatReuse.TryGetReusableUnchangedFile( reusableIndexedFileStats!, @@ -131,217 +122,263 @@ void ObservePersistedCSharpPath(string indexPath) target.IndexPath, target.Language, target.GeneratedExtractionSuppressed); - csharpPrepassStatReuse[target.IndexPath] = existingFile; - allCSharpPrepassTargetsReusable &= existingFile != null; + if (existingFile == null) + allCSharpPrepassTargetsReusable = false; + (csharpPrepassStatReuse ??= + new Dictionary( + runtime.CSharpPrepassCapacity, + StringComparer.Ordinal))[target.IndexPath] = existingFile; + return existingFile != null; } - } - bool CanReuseCSharpPrepassTargetWithoutRead( - CSharpStaticInterfacePrepass.FileTarget target) - { - if (context.ForceExtractorRefresh - || options.Rebuild - || context.StartedWithNoIndexedFiles - || !context.ProjectRootWritten - || (context.RequiresConservativeCSharpSourceRefresh - && !priorPositiveCSharpSourceNoOpCandidate) - || !context.SymbolKindFilterMatchesPrior - || !context.CSharpSymbolNameContractMatchesCurrent - || target.Language != "csharp") + Dictionary? + csharpWorkspaceFileSnapshots = null; + CSharpStaticInterfaceWorkspaceSymbols csharpWorkspace; + var forceFullCSharpRefreshFromInvalidatedNoOp = false; + var csharpWorkspaceMaterialized = + !options.SymbolsOnly + && !scan.DeferCSharpMutationsForIncompleteScan + && runtime.CSharpPrepassTargets.Count > 0 + && !(priorPositiveCSharpSourceNoOpCandidate + && allCSharpPrepassTargetsReusable); + var csharpPrepassSymbolArtifacts = CSharpPrepassSymbolArtifactCache + .CreateForFreshBuiltInExtraction( + csharpWorkspaceMaterialized + && baseline.StartedWithNoIndexedFiles + && !options.Rebuild + && !options.SymbolsOnly + && IndexExtractionStallTimeoutForTesting == null); + if (options.SymbolsOnly + || scan.DeferCSharpMutationsForIncompleteScan) + { + csharpWorkspace = + new CSharpStaticInterfaceWorkspaceSymbols([], false); + } + else { - return false; + csharpWorkspace = BuildFullScanCSharpWorkspaceWithHeartbeat( + priorPositiveCSharpSourceNoOpCandidate, + allCSharpPrepassTargetsReusable, + CanReuseCSharpPrepassTargetWithoutRead, + csharpPrepassSymbolArtifacts, + out csharpWorkspaceFileSnapshots); + forceFullCSharpRefreshFromInvalidatedNoOp = + csharpWorkspaceMaterialized + && (baseline.PriorCSharpStaticInterfaceSourceEvidence == true + || csharpWorkspace.HasStaticInterfaceContracts + || csharpWorkspace + .RequiresMemberReadReferenceRefresh); } - var existingFile = - IndexedFileStatReuse.TryGetReusableUnchangedFile( - reusableIndexedFileStats!, - target.FilePath, - target.IndexPath, - target.Language, - target.GeneratedExtractionSuppressed); - if (existingFile == null) - allCSharpPrepassTargetsReusable = false; - (csharpPrepassStatReuse ??= - new Dictionary( - context.CSharpPrepassCapacity, - StringComparer.Ordinal))[target.IndexPath] = existingFile; - return existingFile != null; - } + if (!options.SymbolsOnly + && !csharpWorkspace.SourceContractEvidenceComplete) + { + csharpPrepassSymbolArtifacts?.Clear(); + csharpPrepassSymbolArtifacts = null; + var incompleteSourcePaths = + csharpWorkspace.IncompleteSourcePaths; + DeferCSharpMutationsForIncompleteWorkspace( + csharpWorkspace); + csharpWorkspace = new CSharpStaticInterfaceWorkspaceSymbols( + [], + false, + SourceContractEvidenceComplete: false, + IncompleteSourcePaths: incompleteSourcePaths); + } - Dictionary? - csharpWorkspaceFileSnapshots = null; - CSharpStaticInterfaceWorkspaceSymbols csharpWorkspace; - var forceFullCSharpRefreshFromInvalidatedNoOp = false; - var csharpWorkspaceMaterialized = - !options.SymbolsOnly - && !context.GetDeferCSharpMutationsForIncompleteScan() - && context.CSharpPrepassTargets.Count > 0 - && !(priorPositiveCSharpSourceNoOpCandidate - && allCSharpPrepassTargetsReusable); - var csharpPrepassSymbolArtifacts = CSharpPrepassSymbolArtifactCache - .CreateForFreshBuiltInExtraction( - csharpWorkspaceMaterialized - && context.StartedWithNoIndexedFiles - && !options.Rebuild - && !options.SymbolsOnly - && IndexExtractionStallTimeoutForTesting == null); - if (options.SymbolsOnly - || context.GetDeferCSharpMutationsForIncompleteScan()) - { - csharpWorkspace = - new CSharpStaticInterfaceWorkspaceSymbols([], false); - } - else - { - csharpWorkspace = BuildFullScanCSharpWorkspaceWithHeartbeat( - context, - priorPositiveCSharpSourceNoOpCandidate, - allCSharpPrepassTargetsReusable, - CanReuseCSharpPrepassTargetWithoutRead, - csharpPrepassSymbolArtifacts, - out csharpWorkspaceFileSnapshots); - forceFullCSharpRefreshFromInvalidatedNoOp = - csharpWorkspaceMaterialized - && (context.PriorCSharpStaticInterfaceSourceEvidence == true - || csharpWorkspace.HasStaticInterfaceContracts - || csharpWorkspace - .RequiresMemberReadReferenceRefresh); + var preservePriorPositiveCSharpSourceNoOp = + priorPositiveCSharpSourceNoOpCandidate + && allCSharpPrepassTargetsReusable + && !scan.DeferCSharpMutationsForIncompleteScan; + var csharpSourceEvidenceForStamp = + preservePriorPositiveCSharpSourceNoOp + ? baseline.PriorCSharpStaticInterfaceSourceEvidence == true + : csharpWorkspace.HasSourceStaticInterfaceContracts; + var csharpSourceEvidenceComplete = + preservePriorPositiveCSharpSourceNoOp + || csharpWorkspace.SourceContractEvidenceComplete; + if (preservePriorPositiveCSharpSourceNoOp) + { + csharpWorkspace = + csharpWorkspace with { HasStaticInterfaceContracts = false }; + } + if (!options.SymbolsOnly + && !scan.DeferCSharpMutationsForIncompleteScan + && !preservePriorPositiveCSharpSourceNoOp + && (forceFullCSharpRefreshFromInvalidatedNoOp + || contracts.RequiresConservativeCSharpSourceRefresh + || !csharpSourceEvidenceComplete + || (scan.Purged > 0 + && scan + .HadCSharpStaticInterfaceContractsBeforePurge))) + { + csharpWorkspace = + csharpWorkspace with { HasStaticInterfaceContracts = true }; + } + + csharp.ReusableIndexedFileStats = reusableIndexedFileStats; + csharp.CSharpPrepassStatReuse = csharpPrepassStatReuse; + csharp.WorkspaceFileSnapshots = csharpWorkspaceFileSnapshots; + csharp.Workspace = csharpWorkspace; + csharp.PrepassSymbolArtifacts = csharpPrepassSymbolArtifacts; + csharp.ForceFullRefreshFromInvalidatedNoOp = + forceFullCSharpRefreshFromInvalidatedNoOp; + csharp.PreservePriorPositiveSourceNoOp = + preservePriorPositiveCSharpSourceNoOp; + csharp.Evidence.ForStamp = csharpSourceEvidenceForStamp; + csharp.Evidence.Complete = csharpSourceEvidenceComplete; } - if (!options.SymbolsOnly - && !csharpWorkspace.SourceContractEvidenceComplete) + private CSharpStaticInterfaceWorkspaceSymbols + BuildFullScanCSharpWorkspaceWithHeartbeat( + bool priorPositiveCSharpSourceNoOpCandidate, + bool allCSharpPrepassTargetsReusable, + Func + canReuseCSharpPrepassTargetWithoutRead, + CSharpPrepassSymbolArtifactCache? symbolArtifactCache, + out Dictionary? + csharpWorkspaceFileSnapshots) { - csharpPrepassSymbolArtifacts?.Clear(); - csharpPrepassSymbolArtifacts = null; - var incompleteSourcePaths = - csharpWorkspace.IncompleteSourcePaths; - context.DeferCSharpMutationsForIncompleteWorkspace( - csharpWorkspace); - csharpWorkspace = new CSharpStaticInterfaceWorkspaceSymbols( - [], - false, - SourceContractEvidenceComplete: false, - IncompleteSourcePaths: incompleteSourcePaths); + var request = Request; + var core = request.Core; + var baseline = request.Baseline; + var contracts = request.Contracts; + var runtime = request.Runtime; + var scan = State.Scan; + WriteFullScanJsonLiveness( + core.Options, + "preparing C# workspace symbols..."); + var activeCSharpWorkspaceFiles = + new string?[runtime.CSharpPrepassTargets.Count]; + var heartbeat = StartFullScanJsonPhaseHeartbeat( + core.Options, + "preparing C# workspace symbols", + () => GetActiveCSharpPrepassPath( + activeCSharpWorkspaceFiles)); + try + { + if (runtime.CSharpPrepassTargets.Count == 0 + || (priorPositiveCSharpSourceNoOpCandidate + && allCSharpPrepassTargetsReusable)) + { + csharpWorkspaceFileSnapshots = null; + return new CSharpStaticInterfaceWorkspaceSymbols([], false); + } + + return BuildStableFullScanCSharpWorkspace( + core.ProjectRoot, + runtime.CSharpPrepassTargets, + out csharpWorkspaceFileSnapshots, + () => CSharpStaticInterfacePrepass.BuildWorkspaceSymbols( + core.Writer, + core.Indexer, + runtime.CSharpPrepassTargets, + includeExistingSymbols: + contracts.CSharpIndexedProjectRootCompatible + && !core.Options.Rebuild + && !baseline.StartedWithNoIndexedFiles, + canReuseExistingSymbolsWithoutRead: + priorPositiveCSharpSourceNoOpCandidate + ? null + : canReuseCSharpPrepassTargetWithoutRead, + reportCandidateFile: (candidateIndex, path) => + SetActiveCSharpPrepassPath( + activeCSharpWorkspaceFiles, + candidateIndex, + path), + parallelism: runtime.ExtractionParallelism, + excludedExistingFileIds: + scan.StaleFilePurgePlan.FileIds, + isExistingSymbolPathExcluded: + IsExistingCSharpSymbolPathNowNonCSharp, + patternConfigsAlreadyLoaded: true, + cancellationToken: runtime.CancellationToken, + symbolArtifactCache: symbolArtifactCache), + runtime.CancellationToken); + } + catch (OperationCanceledException) when ( + runtime.CancellationToken.IsCancellationRequested) + { + throw new IndexInterruptedException( + 0, + runtime.FilesCount, + runtime.ActualMode); + } + finally + { + Array.Clear(activeCSharpWorkspaceFiles); + StopFullScanJsonPhaseHeartbeat(heartbeat); + } } - var preservePriorPositiveCSharpSourceNoOp = - priorPositiveCSharpSourceNoOpCandidate - && allCSharpPrepassTargetsReusable - && !context.GetDeferCSharpMutationsForIncompleteScan(); - var csharpSourceEvidenceForStamp = - preservePriorPositiveCSharpSourceNoOp - ? context.PriorCSharpStaticInterfaceSourceEvidence == true - : csharpWorkspace.HasSourceStaticInterfaceContracts; - var csharpSourceEvidenceComplete = - preservePriorPositiveCSharpSourceNoOp - || csharpWorkspace.SourceContractEvidenceComplete; - if (preservePriorPositiveCSharpSourceNoOp) + internal bool IsExistingCSharpSymbolPathNowNonCSharp(string indexPath) { - csharpWorkspace = - csharpWorkspace with { HasStaticInterfaceContracts = false }; + var core = Request.Core; + var currentPath = Path.Combine( + core.ProjectRoot, + FileIndexer.NormalizeRelativePathForCurrentPlatform(indexPath)); + return Request.Runtime.FileLanguages.TryGetValue( + currentPath, + out var currentLanguage) + && currentLanguage != "csharp"; } - if (!options.SymbolsOnly - && !context.GetDeferCSharpMutationsForIncompleteScan() - && !preservePriorPositiveCSharpSourceNoOp - && (forceFullCSharpRefreshFromInvalidatedNoOp - || context.RequiresConservativeCSharpSourceRefresh - || !csharpSourceEvidenceComplete - || (context.GetPurged() > 0 - && context - .HadCSharpStaticInterfaceContractsBeforePurge))) + + private void DeferCSharpMutationsForIncompleteWorkspace( + CSharpStaticInterfaceWorkspaceSymbols workspace) { - csharpWorkspace = - csharpWorkspace with { HasStaticInterfaceContracts = true }; - } + if (workspace.SourceContractEvidenceComplete) + return; - return new FullScanCSharpPreflightResult( - reusableIndexedFileStats, - csharpPrepassStatReuse, - csharpWorkspaceFileSnapshots, - csharpWorkspace, - csharpPrepassSymbolArtifacts, - forceFullCSharpRefreshFromInvalidatedNoOp, - preservePriorPositiveCSharpSourceNoOp, - csharpSourceEvidenceForStamp, - csharpSourceEvidenceComplete); - } + var scan = State.Scan; + scan.DeferCSharpMutationsForIncompleteScan = true; + scan.StaleFilePurgePlan = FilePurgePlan.Empty; + scan.Purged = 0; + scan.FtsMutated = false; + scan.HadCSharpStaticInterfaceContractsBeforePurge = false; - private static CSharpStaticInterfaceWorkspaceSymbols - BuildFullScanCSharpWorkspaceWithHeartbeat( - FullScanCSharpPreflightContext context, - bool priorPositiveCSharpSourceNoOpCandidate, - bool allCSharpPrepassTargetsReusable, - Func - canReuseCSharpPrepassTargetWithoutRead, - CSharpPrepassSymbolArtifactCache? symbolArtifactCache, - out Dictionary? - csharpWorkspaceFileSnapshots) - { - WriteFullScanJsonLiveness( - context.Options, - "preparing C# workspace symbols..."); - var activeCSharpWorkspaceFiles = - new string?[context.CSharpPrepassTargets.Count]; - var heartbeat = StartFullScanJsonPhaseHeartbeat( - context.Options, - "preparing C# workspace symbols", - () => GetActiveCSharpPrepassPath( - activeCSharpWorkspaceFiles)); - try - { - if (context.CSharpPrepassTargets.Count == 0 - || (priorPositiveCSharpSourceNoOpCandidate - && allCSharpPrepassTargetsReusable)) + var incompletePaths = workspace.IncompleteSourcePaths; + if (incompletePaths == null || incompletePaths.Count == 0) { - csharpWorkspaceFileSnapshots = null; - return new CSharpStaticInterfaceWorkspaceSymbols([], false); + RecordCSharpWorkspaceFailure( + "", + "csharp_prepass", + new IOException( + "C# static-interface workspace preflight could not read a source file.")); + return; } - return BuildStableFullScanCSharpWorkspace( - context.ProjectRoot, - context.CSharpPrepassTargets, - out csharpWorkspaceFileSnapshots, - () => CSharpStaticInterfacePrepass.BuildWorkspaceSymbols( - context.Writer, - context.Indexer, - context.CSharpPrepassTargets, - includeExistingSymbols: - context.CSharpIndexedProjectRootCompatible - && !context.Options.Rebuild - && !context.StartedWithNoIndexedFiles, - canReuseExistingSymbolsWithoutRead: - priorPositiveCSharpSourceNoOpCandidate - ? null - : canReuseCSharpPrepassTargetWithoutRead, - reportCandidateFile: (candidateIndex, path) => - SetActiveCSharpPrepassPath( - activeCSharpWorkspaceFiles, - candidateIndex, - path), - parallelism: context.ExtractionParallelism, - excludedExistingFileIds: - context.StaleFilePurgePlan.FileIds, - isExistingSymbolPathExcluded: - context - .IsExistingCSharpSymbolPathNowNonCSharp, - patternConfigsAlreadyLoaded: true, - cancellationToken: context.CancellationToken, - symbolArtifactCache: symbolArtifactCache), - context.CancellationToken); - } - catch (OperationCanceledException) when ( - context.CancellationToken.IsCancellationRequested) - { - throw new IndexInterruptedException( - 0, - context.FilesCount, - context.ActualMode); + foreach (var path in incompletePaths.Take(PartialIndexFileErrorLimit)) + { + RecordCSharpWorkspaceFailure( + path, + "csharp_prepass", + new IOException( + "C# static-interface workspace preflight could not read this source file.")); + } } - finally + + private void RecordCSharpWorkspaceFailure( + string path, + string phase, + Exception exception) { - Array.Clear(activeCSharpWorkspaceFiles); - StopFullScanJsonPhaseHeartbeat(heartbeat); + var diagnostics = State.Diagnostics; + path = string.IsNullOrWhiteSpace(path) ? "" : path; + if (!diagnostics.ReportedCSharpWorkspaceFailures.Add( + $"{phase}\n{path}")) + { + return; + } + + diagnostics.Errors++; + diagnostics.ErrorList.Add( + new CliJsonMessage(path, FormatIndexFileException(exception))); + if (diagnostics.FileErrorList.Count < PartialIndexFileErrorLimit) + { + diagnostics.FileErrorList.Add( + BuildIndexFileError(path, phase, exception)); + } } } } diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ExtractionPipeline.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ExtractionPipeline.cs index fdce894107..cb8fe31f6f 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ExtractionPipeline.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ExtractionPipeline.cs @@ -10,113 +10,271 @@ namespace CodeIndex.Cli; public static partial class IndexCommandRunner { - private sealed class FullScanExtractionPipelineContext + private readonly record struct FullScanExtractionCore( + DbWriter Writer, + FileIndexer Indexer, + IndexCommandOptions Options, + string ProjectRoot, + FullScanFileTarget[] FileTargets, + ReadableFileByteTracker ReadableFileBytes, + IndexProgressReporter IndexProgress, + FullScanProgressSession FullScanProgress); + + private readonly record struct FullScanExtractionWork( + List? FileIndexes, + int ItemCount, + int Parallelism, + int FilesCount, + bool ForceExtractorRefresh, + DbWriter.AuthoritativeFreshFoldRowsClaim? AuthoritativeFreshFoldRowsClaim, + CancellationToken CancellationToken, + string ActualMode); + + private readonly record struct FullScanExtractionContracts( + bool PriorSymbolsOnlyGraphOmitted, + bool SymbolKindFilterMatchesPrior, + bool CSharpIndexedProjectRootCompatible, + bool CSharpSymbolNameContractMatchesCurrent, + bool SqlGraphContractMatchesCurrent, + bool HdlGraphContractMatchesCurrent, + bool StartedWithNoIndexedFiles); + + private readonly record struct FullScanExtractionReuse( + bool JavaScriptTypeScriptRefreshRequired, + IReadOnlyDictionary HotspotFamilyTrustMatchesCurrent); + + private struct FullScanExtractionRefreshState + { + internal bool FtsMutated { get; set; } + internal bool MutualRecursionRefreshNeeded { get; set; } + internal bool CSharpMetadataTargetsNeedRefresh { get; set; } + internal int SymbolsDroppedByKindFilter { get; set; } + internal HashSet? ReusedHotspotFamilyLanguages { get; set; } + internal HashSet? SkippedSymbolExtractorLanguages { get; set; } + internal HashSet IndexedSymbolExtractorLanguages { get; set; } + } + + private struct FullScanExtractionCounts + { + internal int Processed { get; set; } + internal int Skipped { get; set; } + internal int Warnings { get; set; } + internal int Errors { get; set; } + internal long ExtractedFiles { get; set; } + internal long ExtractedChunks { get; set; } + internal long ExtractedSymbols { get; set; } + internal long ExtractedReferences { get; set; } + } + + private struct FullScanExtractionPersistenceCounts { - internal required DbWriter Writer { get; init; } - internal required FileIndexer Indexer { get; init; } - internal required IndexCommandOptions Options { get; init; } - internal required string ProjectRoot { get; init; } - internal required FullScanFileTarget[] FileTargets { get; init; } - internal List? ExtractionFileIndexes { get; init; } - internal required int ExtractionWorkItemCount { get; init; } - internal required int ExtractionParallelism { get; init; } - internal required int FilesCount { get; init; } - internal required bool ForceExtractorRefresh { get; init; } - internal required bool StartedWithNoIndexedFiles { get; init; } + internal long PersistedFiles { get; set; } + internal long PersistedChunks { get; set; } + internal long PersistedSymbols { get; set; } + internal long PersistedReferences { get; set; } + internal long FreshFiles { get; set; } + internal long FreshChunks { get; set; } + internal long FreshSymbols { get; set; } + internal long FreshReferences { get; set; } + } + + private readonly record struct FullScanExtractionRequest( + FullScanExtractionCore Core, + FullScanExtractionWork Work, + FullScanExtractionContracts Contracts, + FullScanExtractionReuse Reuse); + + private sealed class FullScanExtractionState + { + internal required FullScanPreWriteState PreWrite { get; init; } + internal FullScanExtractionRefreshState Refresh; + internal FullScanExtractionCounts Counts; + internal FullScanExtractionPersistenceCounts PersistenceCounts; + } + + private readonly record struct FullScanExtractionExternalOperations( + Action RequireTypeScriptAugmentationRefresh, + Action WriteProjectRootOnce); + + private sealed partial class FullScanExtractionSession + { + internal required FullScanExtractionRequest Request { get; init; } + internal required FullScanExtractionState State { get; init; } + internal required FullScanExtractionExternalOperations External { get; init; } + internal FullScanExtractionWorkerResources? Lifetime { get; set; } + + private FullScanExtractionCore Core => Request.Core; + private FullScanExtractionWork Work => Request.Work; + private FullScanExtractionContracts Contracts => Request.Contracts; + private FullScanExtractionReuse Reuse => Request.Reuse; + internal FullScanPreWriteState PreWriteState => State.PreWrite; + internal DbWriter Writer => Core.Writer; + internal FileIndexer Indexer => Core.Indexer; + internal IndexCommandOptions Options => Core.Options; + internal string ProjectRoot => Core.ProjectRoot; + internal FullScanFileTarget[] FileTargets => Core.FileTargets; + internal ReadableFileByteTracker ReadableFileBytes => Core.ReadableFileBytes; + internal IndexProgressReporter IndexProgress => Core.IndexProgress; + internal FullScanProgressSession FullScanProgress => Core.FullScanProgress; + internal List? ExtractionFileIndexes => Work.FileIndexes; + internal int ExtractionWorkItemCount => Work.ItemCount; + internal int ExtractionParallelism => Work.Parallelism; + internal int FilesCount => Work.FilesCount; + internal bool ForceExtractorRefresh => Work.ForceExtractorRefresh; + internal bool StartedWithNoIndexedFiles => Contracts.StartedWithNoIndexedFiles; internal DbWriter.AuthoritativeFreshFoldRowsClaim? AuthoritativeFreshFoldRowsClaim + => Work.AuthoritativeFreshFoldRowsClaim; + internal CancellationToken CancellationToken => Work.CancellationToken; + internal bool PriorSymbolsOnlyGraphOmitted => Contracts.PriorSymbolsOnlyGraphOmitted; + internal bool SymbolKindFilterMatchesPrior => Contracts.SymbolKindFilterMatchesPrior; + internal bool CSharpIndexedProjectRootCompatible + => Contracts.CSharpIndexedProjectRootCompatible; + internal bool CSharpSymbolNameContractMatchesCurrent + => Contracts.CSharpSymbolNameContractMatchesCurrent; + internal bool SqlGraphContractMatchesCurrent => Contracts.SqlGraphContractMatchesCurrent; + internal bool HdlGraphContractMatchesCurrent => Contracts.HdlGraphContractMatchesCurrent; + internal int ProcessedCount { - get; - init; + get => PreWriteState.Selection.Processed; + set => PreWriteState.Selection.Processed = value; } - internal required bool PriorSymbolsOnlyGraphOmitted { get; init; } - internal required bool SymbolKindFilterMatchesPrior { get; init; } - internal required bool CSharpIndexedProjectRootCompatible + internal bool IndexProgressVisible { - get; - init; + get => FullScanProgress.IndexProgressVisible; + set => FullScanProgress.IndexProgressVisible = value; } - - internal required bool CSharpSymbolNameContractMatchesCurrent + internal ActiveExtractionPhase?[] ActiveExtractionPhases { - get; - init; + get => FullScanProgress.ActiveExtractionPhases; + set => FullScanProgress.ActiveExtractionPhases = value; } - - internal required bool SqlGraphContractMatchesCurrent { get; init; } - internal required bool HdlGraphContractMatchesCurrent { get; init; } - internal required ReadableFileByteTracker ReadableFileBytes + internal string? CurrentJsonIndexFile { - get; - init; + get => FullScanProgress.CurrentJsonIndexFile; + set => FullScanProgress.CurrentJsonIndexFile = value; } + internal bool DeferCSharpMutationsForIncompleteScan + => PreWriteState.Scan.DeferCSharpMutationsForIncompleteScan; + internal CSharpStaticInterfaceWorkspaceSymbols CSharpWorkspace + => PreWriteState.CSharp.Workspace; + internal CSharpPrepassSymbolArtifactCache? CSharpPrepassSymbolArtifacts + => PreWriteState.CSharp.PrepassSymbolArtifacts; + internal Dictionary? + CSharpWorkspaceFileSnapshots + => PreWriteState.CSharp.WorkspaceFileSnapshots; + internal Action RequireTypeScriptAugmentationRefresh + => External.RequireTypeScriptAugmentationRefresh; + internal Action WriteProjectRootOnce => External.WriteProjectRootOnce; + internal int Processed { get => State.Counts.Processed; set => State.Counts.Processed = value; } + internal int Skipped { get => State.Counts.Skipped; set => State.Counts.Skipped = value; } + internal int Warnings { get => State.Counts.Warnings; set => State.Counts.Warnings = value; } + internal int ErrorsAdded { get => State.Counts.Errors; set => State.Counts.Errors = value; } + internal long ExtractedFiles { get => State.Counts.ExtractedFiles; set => State.Counts.ExtractedFiles = value; } + internal long ExtractedChunks { get => State.Counts.ExtractedChunks; set => State.Counts.ExtractedChunks = value; } + internal long ExtractedSymbols { get => State.Counts.ExtractedSymbols; set => State.Counts.ExtractedSymbols = value; } + internal long ExtractedReferences { get => State.Counts.ExtractedReferences; set => State.Counts.ExtractedReferences = value; } + internal bool FtsMutated { get => State.Refresh.FtsMutated; set => State.Refresh.FtsMutated = value; } + internal bool MutualRecursionRefreshNeeded { get => State.Refresh.MutualRecursionRefreshNeeded; set => State.Refresh.MutualRecursionRefreshNeeded = value; } + internal bool CSharpMetadataTargetsNeedRefresh { get => State.Refresh.CSharpMetadataTargetsNeedRefresh; set => State.Refresh.CSharpMetadataTargetsNeedRefresh = value; } + internal int SymbolsDroppedByKindFilter { get => State.Refresh.SymbolsDroppedByKindFilter; set => State.Refresh.SymbolsDroppedByKindFilter = value; } + internal HashSet? ReusedHotspotFamilyLanguages { get => State.Refresh.ReusedHotspotFamilyLanguages; set => State.Refresh.ReusedHotspotFamilyLanguages = value; } + internal HashSet? SkippedSymbolExtractorLanguages { get => State.Refresh.SkippedSymbolExtractorLanguages; set => State.Refresh.SkippedSymbolExtractorLanguages = value; } + internal HashSet IndexedSymbolExtractorLanguages => State.Refresh.IndexedSymbolExtractorLanguages; + internal List ErrorList => PreWriteState.Diagnostics.ErrorList; + internal List FileErrorList => PreWriteState.Diagnostics.FileErrorList; + internal List WarningList => PreWriteState.Diagnostics.WarningList; + + internal bool TargetRequiresJavaScriptTypeScriptRefresh( + string? language, + string indexPath) + => Reuse.JavaScriptTypeScriptRefreshRequired + && (IsJavaScriptTypeScriptLanguage(language) + || IsJavaScriptTypeScriptConfigPath(indexPath)); - internal required IndexProgressReporter IndexProgress { get; init; } - internal required FullScanProgressSession FullScanProgress { get; init; } - internal required CancellationToken CancellationToken { get; init; } - internal required Func GetProcessedCount { get; init; } - internal required Action PublishProcessedCount { get; init; } - internal required Action ThrowIfFullScanCancelled + internal bool AllowReuseWithCurrentHotspotFamilyTrust(string? language) + => IndexCommandRunner.AllowReuseWithCurrentHotspotFamilyTrust( + language, + Reuse.HotspotFamilyTrustMatchesCurrent); + + internal void InsertIssuesForIndexedFile( + long fileId, + IReadOnlyList issues) { - get; - init; + if (StartedWithNoIndexedFiles) + Writer.InsertIssuesForNewFile(fileId, issues); + else + Writer.InsertIssues(fileId, issues); } - internal required Action SetIndexProgressVisible { get; init; } - internal required Action - SetActiveExtractionPhases - { get; init; } - internal required Action SetCurrentJsonIndexFile { get; init; } - internal required Func GetCurrentJsonIndexFile { get; init; } - internal required Func - GetDeferCSharpMutationsForIncompleteScan - { get; init; } - internal required Func GetFtsMutated { get; init; } - internal required Func - GetCSharpWorkspace - { get; init; } - internal required Func - GetCSharpPrepassSymbolArtifacts - { get; init; } - internal required Func?> - GetCSharpWorkspaceFileSnapshots - { get; init; } - internal required Action - DeferCSharpMutationsForLoadedSnapshotDrift - { get; init; } - internal required Func - TargetRequiresJavaScriptTypeScriptRefresh - { get; init; } - internal required Func - AllowReuseWithCurrentHotspotFamilyTrust - { get; init; } - internal required Action RequireTypeScriptAugmentationRefresh + internal void CountFreshInsertedRows( + int chunkCount, + int symbolCount, + int referenceCount) { - get; - init; + State.PersistenceCounts.PersistedFiles++; + State.PersistenceCounts.PersistedChunks += chunkCount; + State.PersistenceCounts.PersistedSymbols += symbolCount; + State.PersistenceCounts.PersistedReferences += referenceCount; + if (!StartedWithNoIndexedFiles) + return; + + State.PersistenceCounts.FreshFiles++; + State.PersistenceCounts.FreshChunks += chunkCount; + State.PersistenceCounts.FreshSymbols += symbolCount; + State.PersistenceCounts.FreshReferences += referenceCount; } - internal required Action WriteProjectRootOnce { get; init; } - internal required Action> - InsertIssuesForIndexedFile - { get; init; } - internal required Action CountFreshInsertedRows + internal void ThrowIfFullScanCancelled(int filesProcessed, int? filesTotal) { - get; - init; + if (!CancellationToken.IsCancellationRequested) + return; + + throw new IndexInterruptedException( + filesProcessed, + filesTotal, + Work.ActualMode); } - internal required FullScanExtractionConsumerState ConsumerState + internal void DeferCSharpMutationsForLoadedSnapshotDrift(string path) { - get; - init; + Lifetime?.TakeCSharpArtifacts()?.Clear(); + var scan = PreWriteState.Scan; + var csharp = PreWriteState.CSharp; + csharp.PrepassSymbolArtifacts = null; + path = FormatCSharpWorkspaceSnapshotPath(ProjectRoot, path); + scan.DeferCSharpMutationsForIncompleteScan = true; + csharp.PreservePriorPositiveSourceNoOp = false; + csharp.Evidence.ForStamp = false; + csharp.Evidence.Complete = false; + csharp.WorkspaceFileSnapshots = null; + csharp.Workspace = new CSharpStaticInterfaceWorkspaceSymbols( + [], + HasStaticInterfaceContracts: true, + SourceContractEvidenceComplete: false, + IncompleteSourcePaths: [path]); + Writer.SetCSharpStaticInterfaceSourceEvidence(null); + + var diagnostics = PreWriteState.Diagnostics; + const string phase = "csharp_workspace_validation"; + if (!diagnostics.ReportedCSharpWorkspaceFailures.Add( + $"{phase}\n{path}")) + { + return; + } + + var exception = new IOException( + "A C# source changed after workspace preflight; rerun indexing to refresh the complete C# graph."); + diagnostics.Errors++; + diagnostics.ErrorList.Add( + new CliJsonMessage(path, FormatIndexFileException(exception))); + if (diagnostics.FileErrorList.Count < PartialIndexFileErrorLimit) + { + diagnostics.FileErrorList.Add( + BuildIndexFileError(path, phase, exception)); + } } } - private sealed record FullScanExtractionPipelineResult( - PostExtractionHookRunner? PostExtractionHooks, - FullScanExtractionConsumerState? ConsumerState); - private readonly record struct FullScanExtractionScheduling( bool Parallelize, string? Reason); @@ -125,56 +283,143 @@ private readonly record struct FullScanExtractionTailCandidate( int WorkOrdinal, long? Length); + private sealed class FullScanExtractionWorkerResources( + CSharpPrepassSymbolArtifactCache? csharpPrepassSymbolArtifacts, + PostExtractionHookRunner postExtractionHooks) + { + private int disposed; + private Task workerCompletion = Task.CompletedTask; + private CSharpPrepassSymbolArtifactCache? csharpArtifacts = + csharpPrepassSymbolArtifacts; + + internal BlockingCollection? Results { get; private set; } + internal CancellationTokenSource? Cancellation { get; private set; } + internal LazyDisposable? SymbolExtractionWorker + { get; private set; } + + internal void AttachResults(BlockingCollection results) + => Results = results; + + internal void AttachCancellation(CancellationTokenSource cancellation) + => Cancellation = cancellation; + + internal void AttachSymbolExtractionWorker( + LazyDisposable symbolExtractionWorker) + => SymbolExtractionWorker = symbolExtractionWorker; + + internal CSharpPrepassSymbolArtifactCache? TakeCSharpArtifacts() + => Interlocked.Exchange(ref csharpArtifacts, null); + + internal void AttachWorkers(Task[] workers) + { + workerCompletion = Task.WhenAll(workers).ContinueWith( + task => + { + _ = task.Exception; + try + { + Results?.CompleteAdding(); + } + catch (ObjectDisposedException) + { + } + }, + CancellationToken.None, + TaskContinuationOptions.ExecuteSynchronously, + TaskScheduler.Default); + } + + internal void DisposeNowOrWhenWorkersStop() + { + var completion = workerCompletion; + if (completion.IsCompleted) + { + _ = completion.Exception; + DisposeResources(); + return; + } + + _ = completion.ContinueWith( + static (task, state) => + { + _ = task.Exception; + ((FullScanExtractionWorkerResources)state!).DisposeResources(); + }, + this, + CancellationToken.None, + TaskContinuationOptions.ExecuteSynchronously, + TaskScheduler.Default); + } + + private void DisposeResources() + { + if (Interlocked.Exchange(ref disposed, 1) != 0) + return; + + try { TakeCSharpArtifacts()?.Clear(); } catch { } + try { postExtractionHooks.Dispose(); } catch { } + try { SymbolExtractionWorker?.Dispose(); } catch { } + try { Results?.Dispose(); } catch { } + try { Cancellation?.Dispose(); } catch { } + try { FullScanExtractionWorkersStoppedForTesting?.Invoke(); } catch { } + } + } + private const int FullScanExtractionTailWorkerWaves = 4; internal const int MaxFullScanExtractionTailProbeCount = 64; - private static FullScanExtractionPipelineResult + private static PostExtractionHookRunner? RunFullScanExtractionPipeline( - FullScanExtractionPipelineContext context) + FullScanExtractionSession context) { if (context.ExtractionWorkItemCount == 0) { - context.GetCSharpPrepassSymbolArtifacts()?.Clear(); + context.CSharpPrepassSymbolArtifacts?.Clear(); + context.PreWriteState.CSharp.PrepassSymbolArtifacts = null; FullScanExtractionSchedulingForTesting?.Invoke(false, null); - return new FullScanExtractionPipelineResult(null, null); + return null; } + var csharpArtifacts = context.CSharpPrepassSymbolArtifacts; var postExtractionHooks = PostExtractionHookRunner.DiscoverDefault( context.Options.MaxFileSizeBytes, maxSymbolCount: context.Options.MaxSymbolsPerFile + 1, maxReferenceCount: context.Options.MaxReferencesPerFile + 1); - if (postExtractionHooks.HasHooks) - context.AuthoritativeFreshFoldRowsClaim?.Invalidate(); - var scheduling = ResolveFullScanExtractionScheduling( - context, + var workerResources = new FullScanExtractionWorkerResources( + csharpArtifacts, postExtractionHooks); - FullScanExtractionSchedulingForTesting?.Invoke( - scheduling.Parallelize, - scheduling.Reason); - context.FullScanProgress.EnsureIndexingActivityVisible(); - context.FullScanProgress.StartJsonHeartbeatIfNeeded(); + context.Lifetime = workerResources; try { - var consumerState = ExecuteFullScanExtractionPipeline( + if (postExtractionHooks.HasHooks) + context.AuthoritativeFreshFoldRowsClaim?.Invalidate(); + var scheduling = ResolveFullScanExtractionScheduling( context, + postExtractionHooks); + FullScanExtractionSchedulingForTesting?.Invoke( + scheduling.Parallelize, + scheduling.Reason); + context.FullScanProgress.EnsureIndexingActivityVisible(); + context.FullScanProgress.StartJsonHeartbeatIfNeeded(); + ExecuteFullScanExtractionPipeline( + context, + workerResources, postExtractionHooks, scheduling.Parallelize); - return new FullScanExtractionPipelineResult( - postExtractionHooks, - consumerState); + return postExtractionHooks; } finally { - context.GetCSharpPrepassSymbolArtifacts()?.Clear(); - context.SetCurrentJsonIndexFile(null); + workerResources.DisposeNowOrWhenWorkersStop(); + context.Lifetime = null; + context.CurrentJsonIndexFile = null; context.FullScanProgress.StopJsonHeartbeat(); - postExtractionHooks.Dispose(); } } private static FullScanExtractionScheduling ResolveFullScanExtractionScheduling( - FullScanExtractionPipelineContext context, + FullScanExtractionSession context, PostExtractionHookRunner postExtractionHooks) { var parallelize = !context.Options.SymbolKindFilter.IsActive @@ -281,9 +526,9 @@ internal static int ResolveFullScanExtractionFileIndex( return info.Exists ? info.Length : null; } - private static FullScanExtractionConsumerState - ExecuteFullScanExtractionPipeline( - FullScanExtractionPipelineContext context, + private static void ExecuteFullScanExtractionPipeline( + FullScanExtractionSession context, + FullScanExtractionWorkerResources workerResources, PostExtractionHookRunner postExtractionHooks, bool parallelizeExtraction) { @@ -294,179 +539,92 @@ private static FullScanExtractionConsumerState context.ExtractionWorkItemCount); var activeExtractionPhases = new ActiveExtractionPhase?[extractionWorkerCount]; - context.SetActiveExtractionPhases(activeExtractionPhases); + context.ActiveExtractionPhases = activeExtractionPhases; var extractionQueueCapacity = parallelizeExtraction ? Math.Max(1, extractionWorkerCount * 2) : 1; FullScanExtractionQueueCapacityForTesting?.Invoke( extractionQueueCapacity); - using var extractionResults = + var extractionResults = new BlockingCollection( extractionQueueCapacity); - using var extractionStallCts = + workerResources.AttachResults(extractionResults); + var extractionStallCts = CancellationTokenSource.CreateLinkedTokenSource( context.CancellationToken); - using var mainSymbolExtractionWorker = + workerResources.AttachCancellation(extractionStallCts); + var mainSymbolExtractionWorker = new LazyDisposable( () => new SymbolExtractionWorkerClient( context.Options.MaxFileSizeBytes)); - var extractionTailSchedule = parallelizeExtraction - ? BuildFullScanExtractionTailSchedule( - context.ExtractionWorkItemCount, - extractionWorkerCount, - context.Indexer.MaxFileSizeBytes, - workOrdinal => ReadFullScanExtractionFileLength( - context.FileTargets[ - ResolveFullScanExtractionFileIndex( - context.ExtractionFileIndexes, - workOrdinal)].FilePath), - context.CancellationToken) - : []; - var workers = StartFullScanExtractionWorkers( - new FullScanExtractionWorkerContext - { - Indexer = context.Indexer, - Options = context.Options, - ProjectRoot = context.ProjectRoot, - FileTargets = context.FileTargets, - ExtractionFileIndexes = - context.ExtractionFileIndexes, - ExtractionWorkItemCount = + workerResources.AttachSymbolExtractionWorker(mainSymbolExtractionWorker); + try + { + var extractionTailSchedule = parallelizeExtraction + ? BuildFullScanExtractionTailSchedule( context.ExtractionWorkItemCount, - ExtractionWorkerCount = extractionWorkerCount, - ParallelizeExtraction = parallelizeExtraction, - ExtractionTailSchedule = extractionTailSchedule, - CSharpWorkspace = context.GetCSharpWorkspace(), - CSharpPrepassSymbolArtifacts = - context.GetCSharpPrepassSymbolArtifacts(), - CSharpWorkspaceFileSnapshots = - context.GetCSharpWorkspaceFileSnapshots(), - PostExtractionHooks = postExtractionHooks, - ActiveExtractionPhases = activeExtractionPhases, - ExtractionResults = extractionResults, - ExtractionCancellationToken = - extractionStallCts.Token, - CancellationToken = context.CancellationToken, - }); - CompleteFullScanExtractionQueueWhenWorkersFinish( - workers, - extractionResults); - - var processedBeforeExtraction = context.GetProcessedCount(); - var consumerContext = CreateFullScanExtractionConsumerContext( - context, - postExtractionHooks, - mainSymbolExtractionWorker.Value, - extractionResults, - extractionStallCts, - workers, - activeExtractionPhases, - processedBeforeExtraction); - var consumerState = - ConsumeFullScanExtractionResults(consumerContext); - context.PublishProcessedCount( - processedBeforeExtraction + consumerState.Processed); - return consumerState; + extractionWorkerCount, + context.Indexer.MaxFileSizeBytes, + workOrdinal => ReadFullScanExtractionFileLength( + context.FileTargets[ + ResolveFullScanExtractionFileIndex( + context.ExtractionFileIndexes, + workOrdinal)].FilePath), + context.CancellationToken) + : []; + var workers = StartFullScanExtractionWorkers( + new FullScanExtractionWorkerContext + { + Indexer = context.Indexer, + Options = context.Options, + ProjectRoot = context.ProjectRoot, + FileTargets = context.FileTargets, + ExtractionFileIndexes = context.ExtractionFileIndexes, + ExtractionWorkItemCount = context.ExtractionWorkItemCount, + ExtractionWorkerCount = extractionWorkerCount, + ParallelizeExtraction = parallelizeExtraction, + ExtractionTailSchedule = extractionTailSchedule, + CSharpWorkspace = context.CSharpWorkspace, + CSharpPrepassSymbolArtifacts = context.CSharpPrepassSymbolArtifacts, + CSharpWorkspaceFileSnapshots = context.CSharpWorkspaceFileSnapshots, + PostExtractionHooks = postExtractionHooks, + ActiveExtractionPhases = activeExtractionPhases, + ExtractionResults = extractionResults, + ExtractionCancellationToken = extractionStallCts.Token, + CancellationToken = context.CancellationToken, + }, + workerResources.AttachWorkers); + var processedBeforeExtraction = context.ProcessedCount; + var consumerResources = new FullScanExtractionConsumerResources( + postExtractionHooks, + mainSymbolExtractionWorker.Value, + extractionResults, + workers, + IndexExtractionStallTimeoutForTesting?.Invoke() + ?? IndexExtractionStallTimeout, + activeExtractionPhases, + extractionStallCts, + processedBeforeExtraction); + context.ConsumeFullScanExtractionResults(in consumerResources); + context.ProcessedCount = processedBeforeExtraction + context.State.Counts.Processed; + } + catch + { + try { extractionStallCts.Cancel(); } catch { } + throw; + } } private static void PrepareFullScanExtractionProgress( - FullScanExtractionPipelineContext context) + FullScanExtractionSession context) { if (context.Options.Json || context.Options.Quiet) return; context.IndexProgress.Pause(); - context.SetIndexProgressVisible(true); + context.IndexProgressVisible = true; ConsoleUi.PrintProgress(0, context.FilesCount); } - private static void CompleteFullScanExtractionQueueWhenWorkersFinish( - Task[] workers, - BlockingCollection extractionResults) - { - _ = Task.WhenAll(workers).ContinueWith( - _ => extractionResults.CompleteAdding(), - CancellationToken.None, - TaskContinuationOptions.ExecuteSynchronously, - TaskScheduler.Default); - } - - private static FullScanExtractionConsumerContext - CreateFullScanExtractionConsumerContext( - FullScanExtractionPipelineContext context, - PostExtractionHookRunner postExtractionHooks, - SymbolExtractionWorkerClient symbolExtractionWorker, - BlockingCollection extractionResults, - CancellationTokenSource extractionStallCts, - Task[] workers, - ActiveExtractionPhase?[] activeExtractionPhases, - int processedBeforeExtraction) - => new() - { - Writer = context.Writer, - Indexer = context.Indexer, - Options = context.Options, - ProjectRoot = context.ProjectRoot, - FileTargets = context.FileTargets, - FilesCount = context.FilesCount, - ProcessedBeforeExtraction = processedBeforeExtraction, - ForceExtractorRefresh = context.ForceExtractorRefresh, - StartedWithNoIndexedFiles = - context.StartedWithNoIndexedFiles, - PriorSymbolsOnlyGraphOmitted = - context.PriorSymbolsOnlyGraphOmitted, - SymbolKindFilterMatchesPrior = - context.SymbolKindFilterMatchesPrior, - CSharpIndexedProjectRootCompatible = - context.CSharpIndexedProjectRootCompatible, - CSharpSymbolNameContractMatchesCurrent = - context.CSharpSymbolNameContractMatchesCurrent, - SqlGraphContractMatchesCurrent = - context.SqlGraphContractMatchesCurrent, - HdlGraphContractMatchesCurrent = - context.HdlGraphContractMatchesCurrent, - ReadableFileBytes = context.ReadableFileBytes, - PostExtractionHooks = postExtractionHooks, - SymbolExtractionWorker = symbolExtractionWorker, - IndexProgress = context.IndexProgress, - ExtractionResults = extractionResults, - Workers = workers, - ExtractionStallTimeout = - IndexExtractionStallTimeoutForTesting?.Invoke() - ?? IndexExtractionStallTimeout, - ActiveExtractionPhases = activeExtractionPhases, - CancellationToken = context.CancellationToken, - CancelExtraction = extractionStallCts.Cancel, - EnsureIndexingActivityVisible = - context.FullScanProgress.EnsureIndexingActivityVisible, - ReportJsonIndexProgressIfNeeded = - context.FullScanProgress.ReportJsonIndexProgressIfNeeded, - ThrowIfFullScanCancelled = - context.ThrowIfFullScanCancelled, - PublishProcessedCount = context.PublishProcessedCount, - SetCurrentJsonIndexFile = context.SetCurrentJsonIndexFile, - GetCurrentJsonIndexFile = context.GetCurrentJsonIndexFile, - GetDeferCSharpMutationsForIncompleteScan = - context.GetDeferCSharpMutationsForIncompleteScan, - GetFtsMutated = context.GetFtsMutated, - GetCSharpWorkspace = context.GetCSharpWorkspace, - GetCSharpPrepassSymbolArtifacts = - context.GetCSharpPrepassSymbolArtifacts, - GetCSharpWorkspaceFileSnapshots = - context.GetCSharpWorkspaceFileSnapshots, - DeferCSharpMutationsForLoadedSnapshotDrift = - context.DeferCSharpMutationsForLoadedSnapshotDrift, - TargetRequiresJavaScriptTypeScriptRefresh = - context.TargetRequiresJavaScriptTypeScriptRefresh, - AllowReuseWithCurrentHotspotFamilyTrust = - context.AllowReuseWithCurrentHotspotFamilyTrust, - RequireTypeScriptAugmentationRefresh = - context.RequireTypeScriptAugmentationRefresh, - WriteProjectRootOnce = context.WriteProjectRootOnce, - InsertIssuesForIndexedFile = - context.InsertIssuesForIndexedFile, - CountFreshInsertedRows = context.CountFreshInsertedRows, - State = context.ConsumerState, - }; } diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ExtractionWorkers.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ExtractionWorkers.cs index dcb4b521db..a5cf071b86 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ExtractionWorkers.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ExtractionWorkers.cs @@ -30,7 +30,8 @@ private sealed class FullScanExtractionWorkerContext } private static Task[] StartFullScanExtractionWorkers( - FullScanExtractionWorkerContext context) + FullScanExtractionWorkerContext context, + Action attachWorkers) { var indexer = context.Indexer; var options = context.Options; @@ -50,8 +51,14 @@ private static Task[] StartFullScanExtractionWorkers( var extractionCancellationToken = context.ExtractionCancellationToken; var cancellationToken = context.CancellationToken; var nextExtractionIndex = -1; - var workers = Enumerable.Range(0, extractionWorkerCount) - .Select(workerIndex => Task.Factory.StartNew(() => + var workers = new Task[extractionWorkerCount]; + var startedWorkerCount = 0; + try + { + for (var workerIndex = 0; workerIndex < extractionWorkerCount; workerIndex++) + { + var capturedWorkerIndex = workerIndex; + workers[workerIndex] = Task.Factory.StartNew(() => { using var workerSymbolExtractionWorker = new LazyDisposable( () => new SymbolExtractionWorkerClient(options.MaxFileSizeBytes)); @@ -75,7 +82,7 @@ private static Task[] StartFullScanExtractionWorkers( var displayRelativePath = target.DisplayRelativePath; try { - Volatile.Write(ref activeExtractionPhases[workerIndex], new(displayRelativePath, "reading")); + Volatile.Write(ref activeExtractionPhases[capturedWorkerIndex], new(displayRelativePath, "reading")); FullScanFileContentLoadForTesting?.Invoke(displayRelativePath); var loaded = indexer.BuildLoadedRecordWithRawBytes( filePath, @@ -120,13 +127,13 @@ private static Task[] StartFullScanExtractionWorkers( : null; if (parallelizeExtraction) { - Volatile.Write(ref activeExtractionPhases[workerIndex], new(record.Path, "chunking")); + Volatile.Write(ref activeExtractionPhases[capturedWorkerIndex], new(record.Path, "chunking")); chunks = ChunkSplitter.SplitNormalized(0, content, loaded.Facts); if (generatedSuppressionIssue != null) { symbols = []; references = []; - Volatile.Write(ref activeExtractionPhases[workerIndex], new(record.Path, "validating")); + Volatile.Write(ref activeExtractionPhases[capturedWorkerIndex], new(record.Path, "validating")); issues = AppendIssueIfMissing( FileIndexer.ValidateContent(record.Path, rawBytes, content, record.Lang, loaded.Inspection, loaded.Facts), generatedSuppressionIssue); @@ -146,7 +153,7 @@ private static Task[] StartFullScanExtractionWorkers( extractionCancellationToken); continue; } - Volatile.Write(ref activeExtractionPhases[workerIndex], new(record.Path, "symbols")); + Volatile.Write(ref activeExtractionPhases[capturedWorkerIndex], new(record.Path, "symbols")); FullScanFilePhaseForTesting?.Invoke(record.Path, "symbols"); FileIssue? symbolRegexTimeoutIssue; if (string.Equals(record.Lang, "csharp", StringComparison.Ordinal) @@ -168,7 +175,7 @@ private static Task[] StartFullScanExtractionWorkers( filePath, projectRoot, record.Path, - Volatile.Read(ref activeExtractionPhases[workerIndex])!.Format(), + Volatile.Read(ref activeExtractionPhases[capturedWorkerIndex])!.Format(), true, hasOversizeLine, loaded.ConflictMarkerLine, @@ -212,7 +219,7 @@ private static Task[] StartFullScanExtractionWorkers( } else { - Volatile.Write(ref activeExtractionPhases[workerIndex], new(record.Path, "references")); + Volatile.Write(ref activeExtractionPhases[capturedWorkerIndex], new(record.Path, "references")); FullScanFilePhaseForTesting?.Invoke(record.Path, "references"); using var regexTimeouts = BoundedRegex.CaptureTimeouts(record.Lang, "reference_extraction"); referenceExtraction = ReferenceExtractor.ExtractDetailedNormalized( @@ -232,7 +239,7 @@ private static Task[] StartFullScanExtractionWorkers( references = referenceExtraction.References; referenceRegexTimeoutIssue = BuildRegexTimeoutIssue(record.Path, regexTimeouts); } - Volatile.Write(ref activeExtractionPhases[workerIndex], new(record.Path, "validating")); + Volatile.Write(ref activeExtractionPhases[capturedWorkerIndex], new(record.Path, "validating")); issues = FileIndexer.ValidateContent(record.Path, rawBytes, content, record.Lang, loaded.Inspection, loaded.Facts); if (symbolRegexTimeoutIssue != null) issues = AppendIssue(issues, symbolRegexTimeoutIssue); @@ -249,7 +256,7 @@ private static Task[] StartFullScanExtractionWorkers( } else { - Volatile.Write(ref activeExtractionPhases[workerIndex], new(record.Path, "validating")); + Volatile.Write(ref activeExtractionPhases[capturedWorkerIndex], new(record.Path, "validating")); issues = FileIndexer.ValidateContent(record.Path, rawBytes, content, record.Lang, loaded.Inspection, loaded.Facts); } extractionResults.Add( @@ -380,16 +387,24 @@ private static Task[] StartFullScanExtractionWorkers( } catch (Exception ex) { - var failedPhase = Volatile.Read(ref activeExtractionPhases[workerIndex])?.Phase ?? "unknown"; + var failedPhase = Volatile.Read(ref activeExtractionPhases[capturedWorkerIndex])?.Phase ?? "unknown"; extractionResults.Add(FullScanFileWorkItem.Failure(fileIndex, filePath, displayRelativePath, failedPhase, ex), extractionCancellationToken); } finally { - Volatile.Write(ref activeExtractionPhases[workerIndex], null); + Volatile.Write(ref activeExtractionPhases[capturedWorkerIndex], null); } } - }, cancellationToken, TaskCreationOptions.LongRunning, TaskScheduler.Default)) - .ToArray(); + }, cancellationToken, TaskCreationOptions.LongRunning, TaskScheduler.Default); + startedWorkerCount++; + } + } + catch + { + attachWorkers(workers[..startedWorkerCount]); + throw; + } + attachWorkers(workers); return workers; } } diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.FilePersistence.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.FilePersistence.cs index 6279bfc38e..7f81169e81 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.FilePersistence.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.FilePersistence.cs @@ -24,7 +24,7 @@ private sealed class FullScanFilePersistenceContext internal required PostExtractionHookRunner PostExtractionHooks { get; init; } internal required SymbolExtractionWorkerClient SymbolExtractionWorker { get; init; } internal required CancellationToken CancellationToken { get; init; } - internal required Action> InsertIssuesForIndexedFile { get; init; } + internal required FullScanExtractionSession ExtractionSession { get; init; } internal required Action WriteProjectRootOnce { get; init; } internal required Action SetPhase { get; init; } } @@ -95,7 +95,7 @@ private static FullScanFilePersistenceResult PersistFullScanFile( var generatedIssues = AppendIssueIfMissing( RequireWorkItemIssues(item), context.GeneratedSuppressionIssue); - context.InsertIssuesForIndexedFile(fileId, generatedIssues); + context.ExtractionSession.InsertIssuesForIndexedFile(fileId, generatedIssues); context.SetPhase( FormatIndexPhasePath(record.Path, "committing"), "committing"); @@ -170,7 +170,7 @@ private static FullScanFilePersistenceResult PersistFullScanFile( : AppendIssue([symbolRegexTimeoutIssue], issue); writer.InsertSymbols([], cancellationToken); writer.InsertReferencesInAtomicFileScope([], cancellationToken); - context.InsertIssuesForIndexedFile(fileId, capIssues); + context.ExtractionSession.InsertIssuesForIndexedFile(fileId, capIssues); txn.Commit(); return new FullScanFilePersistenceResult( chunks.Count, @@ -330,7 +330,7 @@ private static FullScanFilePersistenceResult PersistFullScanFile( mutualRecursionRefreshNeeded = true; context.SetPhase(FormatIndexPhasePath(record.Path, "validating"), "validating"); var issues = RequireWorkItemIssues(item); - context.InsertIssuesForIndexedFile(fileId, issues); + context.ExtractionSession.InsertIssuesForIndexedFile(fileId, issues); context.SetPhase(FormatIndexPhasePath(record.Path, "committing"), "committing"); context.WriteProjectRootOnce(); txn.Commit(); diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.Interruption.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.Interruption.cs index 791794e81a..6c0c3c9062 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.Interruption.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.Interruption.cs @@ -82,7 +82,7 @@ private static void ThrowIfFullScanExtractionStalled( long lastProgressTimestamp, string? currentFile, ActiveExtractionPhase?[] activeExtractionPhases, - Action cancelStalledWork) + CancellationTokenSource stalledWorkCancellation) { if (!TryGetFullScanExtractionStallPath( filesProcessed, @@ -96,7 +96,7 @@ private static void ThrowIfFullScanExtractionStalled( return; } - cancelStalledWork(); + stalledWorkCancellation.Cancel(); throw new IndexExtractionStalledException(filesProcessed, filesTotal, timeout, activePath); } diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.PreWriteSession.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.PreWriteSession.cs new file mode 100644 index 0000000000..31a7234d38 --- /dev/null +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.PreWriteSession.cs @@ -0,0 +1,121 @@ +using CodeIndex.Database; +using CodeIndex.Indexer; + +namespace CodeIndex.Cli; + +public static partial class IndexCommandRunner +{ + private sealed partial class FullScanPreWriteSession( + FullScanPreWriteRequest request, + FullScanPreWriteState state) + { + internal FullScanPreWriteRequest Request { get; } = request; + internal FullScanPreWriteState State { get; } = state; + } + + private readonly record struct FullScanPreWriteRequest( + FullScanPreWriteCore Core, + FullScanPreWriteBaseline Baseline, + FullScanPreWriteContracts Contracts, + FullScanPreWriteRuntime Runtime, + FullScanPreWriteReusePolicy Reuse); + + private readonly record struct FullScanPreWriteCore( + DbWriter Writer, + FileIndexer Indexer, + IndexCommandOptions Options, + string ProjectRoot); + + private readonly record struct FullScanPreWriteBaseline( + bool PriorIndexComplete, + int PriorReadiness, + bool PriorSymbolsOnlyGraphOmitted, + bool? PriorCSharpStaticInterfaceSourceEvidence, + bool StartedWithNoIndexedFiles, + bool ScanHadErrors, + bool ProjectRootWritten); + + private readonly record struct FullScanPreWriteContracts( + bool SymbolKindFilterMatchesPrior, + bool CSharpSymbolNameContractMatchesCurrent, + bool CSharpIndexedProjectRootCompatible, + bool CSharpHotspotTrustMatchesCurrent, + bool RequiresConservativeCSharpSourceRefresh, + bool ForceExtractorRefresh); + + private readonly record struct FullScanPreWriteRuntime( + FullScanFileTarget[] FileTargets, + IReadOnlyList CSharpPrepassTargets, + IReadOnlyDictionary FileLanguages, + int CSharpPrepassCapacity, + int ExtractionParallelism, + int FilesCount, + string ActualMode, + CancellationToken CancellationToken); + + private readonly record struct FullScanPreWriteReusePolicy( + bool CanSkipTargetsBeforeContentLoad, + bool SqlGraphContractMatchesCurrent, + bool HdlGraphContractMatchesCurrent, + IReadOnlyDictionary HotspotFamilyTrustMatchesCurrent, + bool JavaScriptTypeScriptRefreshRequired); + + private sealed class FullScanPreWriteState + { + internal required FullScanPreWriteMutableScanState Scan { get; init; } + internal required FullScanPreWriteCSharpState CSharp { get; init; } + internal required FullScanPreWriteSelectionState Selection { get; init; } + internal required FullScanPreWriteDiagnosticsState Diagnostics { get; init; } + } + + private sealed class FullScanPreWriteMutableScanState + { + internal required FilePurgePlan StaleFilePurgePlan { get; set; } + internal bool DeferCSharpMutationsForIncompleteScan { get; set; } + internal int Purged { get; set; } + internal bool FtsMutated { get; set; } + internal bool HadCSharpStaticInterfaceContractsBeforePurge { get; set; } + } + + private sealed class FullScanPreWriteCSharpState + { + internal ReusableIndexedFileStatsSnapshot? ReusableIndexedFileStats { get; set; } + internal Dictionary? CSharpPrepassStatReuse { get; set; } + internal Dictionary? + WorkspaceFileSnapshots + { get; set; } + internal CSharpStaticInterfaceWorkspaceSymbols Workspace { get; set; } = + null!; + internal CSharpPrepassSymbolArtifactCache? PrepassSymbolArtifacts { get; set; } + internal bool ForceFullRefreshFromInvalidatedNoOp { get; set; } + internal bool PreservePriorPositiveSourceNoOp { get; set; } + internal FullScanPreWriteCSharpEvidence Evidence { get; } = new(); + } + + private sealed class FullScanPreWriteCSharpEvidence + { + internal bool ForStamp { get; set; } + internal bool Complete { get; set; } + } + + private sealed class FullScanPreWriteSelectionState + { + internal List? ExtractionFileIndexes { get; set; } + internal int ExtractionWorkItemCount { get; set; } + internal int Processed { get; set; } + internal int Skipped { get; set; } + internal required ReadableFileByteTracker ReadableFileBytes { get; init; } + internal HashSet? ReusedHotspotFamilyLanguages { get; set; } + internal HashSet? SkippedSymbolExtractorLanguages { get; set; } + internal bool UseFtsBulkLoad { get; set; } + } + + private sealed class FullScanPreWriteDiagnosticsState + { + internal required List ErrorList { get; init; } + internal required List FileErrorList { get; init; } + internal required List WarningList { get; init; } + internal required HashSet ReportedCSharpWorkspaceFailures { get; init; } + internal int Errors { get; set; } + } +} diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.PreWriteValidation.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.PreWriteValidation.cs new file mode 100644 index 0000000000..5964483dd8 --- /dev/null +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.PreWriteValidation.cs @@ -0,0 +1,130 @@ +using CodeIndex.Indexer; + +namespace CodeIndex.Cli; + +public static partial class IndexCommandRunner +{ + private readonly record struct FullScanPreWriteInputValidationResult( + bool IsValid, + string ChangedPath); + + private sealed partial class FullScanPreWriteSession + { + internal FullScanPreWriteInputValidationResult PrepareWriteBoundary( + FileIndexer.ScanInputSnapshot? inputSnapshot) + { + PrepareExtractionTargets(); + DecideFtsBulkLoad(); + RevalidateFinalCSharpNoOp(); + var inputValidation = + ValidateBeforeWriteScanInput(inputSnapshot); + if (inputValidation.IsValid) + ValidateBeforeWriteCSharpFileSnapshots(); + return inputValidation; + } + + internal FullScanPreWriteInputValidationResult + ValidateBeforeWriteScanInput( + FileIndexer.ScanInputSnapshot? inputSnapshot) + { + if (inputSnapshot == null) + return new(true, string.Empty); + + FullScanInputSnapshotBarrierForTesting?.Invoke("before_write"); + var isValid = Request.Core.Indexer.TryValidateScanInputSnapshot( + inputSnapshot, + out var changedPath, + Request.Runtime.CancellationToken); + return new(isValid, changedPath); + } + + internal void ValidateBeforeWriteCSharpFileSnapshots() + { + var options = Request.Core.Options; + var scan = State.Scan; + if (options.SymbolsOnly + || scan.DeferCSharpMutationsForIncompleteScan) + { + return; + } + + var csharp = State.CSharp; + var changedFilePath = string.Empty; + var stableFiles = csharp.WorkspaceFileSnapshots == null + || CSharpStaticInterfacePrepass.TryValidateFileStatSnapshots( + Request.Runtime.CSharpPrepassTargets, + csharp.WorkspaceFileSnapshots, + out changedFilePath, + Request.Runtime.CancellationToken); + if (stableFiles) + return; + + csharp.PrepassSymbolArtifacts?.Clear(); + csharp.PrepassSymbolArtifacts = null; + var driftPath = FormatCSharpWorkspaceSnapshotPath( + Request.Core.ProjectRoot, + changedFilePath); + var incompleteWorkspace = + new CSharpStaticInterfaceWorkspaceSymbols( + [], + HasStaticInterfaceContracts: true, + SourceContractEvidenceComplete: false, + IncompleteSourcePaths: [driftPath]); + DeferCSharpMutationsForIncompleteWorkspace(incompleteWorkspace); + csharp.PreservePriorPositiveSourceNoOp = false; + csharp.Evidence.ForStamp = false; + csharp.Evidence.Complete = false; + csharp.WorkspaceFileSnapshots = null; + csharp.Workspace = incompleteWorkspace; + State.Selection.UseFtsBulkLoad = false; + + DeferCSharpExtractionTargets(); + } + + private void DeferCSharpExtractionTargets() + { + var fileTargets = Request.Runtime.FileTargets; + var selection = State.Selection; + var deferredCSharpIndexes = + new List(Request.Runtime.CSharpPrepassTargets.Count); + if (selection.ExtractionFileIndexes == null) + { + selection.ExtractionFileIndexes = + new List(fileTargets.Length); + for (var fileIndex = 0; + fileIndex < fileTargets.Length; + fileIndex++) + { + if (fileTargets[fileIndex].Language == "csharp") + deferredCSharpIndexes.Add(fileIndex); + else + selection.ExtractionFileIndexes.Add(fileIndex); + } + } + else + { + for (var extractionIndex = + selection.ExtractionFileIndexes.Count - 1; + extractionIndex >= 0; + extractionIndex--) + { + var fileIndex = + selection.ExtractionFileIndexes[extractionIndex]; + if (fileTargets[fileIndex].Language != "csharp") + continue; + + deferredCSharpIndexes.Add(fileIndex); + selection.ExtractionFileIndexes.RemoveAt( + extractionIndex); + } + } + + foreach (var fileIndex in deferredCSharpIndexes) + RecordDeferredFullScanCSharpTarget(fileIndex); + + selection.ExtractionFileIndexes.Sort(); + selection.ExtractionWorkItemCount = + selection.ExtractionFileIndexes.Count; + } + } +} diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.Progress.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.Progress.cs index b5f06fc525..61abaa85b1 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.Progress.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.Progress.cs @@ -9,36 +9,30 @@ private sealed class FullScanProgressSession : IDisposable private readonly IndexCommandOptions options; private readonly int filesCount; private readonly IndexProgressReporter indexProgress; - private readonly Func getProcessed; - private readonly Func isProgressVisible; - private readonly Func getCurrentJsonIndexFile; - private readonly Func getActiveExtractionPhases; + private readonly FullScanPreWriteSelectionState selection; private bool redirectedIndexingMessagePrinted; private long lastJsonProgressAt = Stopwatch.GetTimestamp(); private CancellationTokenSource? jsonHeartbeatCts; private Task? jsonHeartbeatTask; + internal bool IndexProgressVisible { get; set; } + internal string? CurrentJsonIndexFile { get; set; } + internal ActiveExtractionPhase?[] ActiveExtractionPhases { get; set; } = []; internal FullScanProgressSession( IndexCommandOptions options, int filesCount, IndexProgressReporter indexProgress, - Func getProcessed, - Func isProgressVisible, - Func getCurrentJsonIndexFile, - Func getActiveExtractionPhases) + FullScanPreWriteSelectionState selection) { this.options = options; this.filesCount = filesCount; this.indexProgress = indexProgress; - this.getProcessed = getProcessed; - this.isProgressVisible = isProgressVisible; - this.getCurrentJsonIndexFile = getCurrentJsonIndexFile; - this.getActiveExtractionPhases = getActiveExtractionPhases; + this.selection = selection; } internal void EnsureIndexingActivityVisible() { - if (options.Json || options.Quiet || isProgressVisible()) + if (options.Json || options.Quiet || IndexProgressVisible) return; if (indexProgress.Interactive) @@ -59,7 +53,7 @@ internal void ReportJsonIndexProgressIfNeeded() if (!options.Json || options.Quiet || filesCount == 0) return; - var processed = getProcessed(); + var processed = selection.Processed; var now = Stopwatch.GetTimestamp(); if (processed == 0 || processed == filesCount @@ -104,14 +98,14 @@ await Task.Delay(TimeSpan.FromSeconds(5), token) break; var file = GetJsonIndexHeartbeatPath( - getCurrentJsonIndexFile(), + CurrentJsonIndexFile, FormatActiveExtractionPhases( - getActiveExtractionPhases())); + ActiveExtractionPhases)); var fileSuffix = string.IsNullOrEmpty(file) ? string.Empty : $": {file}"; ConsoleUi.TryWriteErrorLine( - $"cdidx: still indexing {getProcessed():N0}/{filesCount:N0} file(s){fileSuffix}..."); + $"cdidx: still indexing {selection.Processed:N0}/{filesCount:N0} file(s){fileSuffix}..."); } }, token); diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ResultConsumer.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ResultConsumer.cs index 373bbc458f..16d2c8337a 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ResultConsumer.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.ResultConsumer.cs @@ -10,414 +10,346 @@ namespace CodeIndex.Cli; public static partial class IndexCommandRunner { - private sealed class FullScanExtractionConsumerContext - { - internal required DbWriter Writer { get; init; } - internal required FileIndexer Indexer { get; init; } - internal required IndexCommandOptions Options { get; init; } - internal required string ProjectRoot { get; init; } - internal required FullScanFileTarget[] FileTargets { get; init; } - internal required int FilesCount { get; init; } - internal required int ProcessedBeforeExtraction { get; init; } - internal required bool ForceExtractorRefresh { get; init; } - internal required bool StartedWithNoIndexedFiles { get; init; } - internal required bool PriorSymbolsOnlyGraphOmitted { get; init; } - internal required bool SymbolKindFilterMatchesPrior { get; init; } - internal required bool CSharpIndexedProjectRootCompatible { get; init; } - internal required bool CSharpSymbolNameContractMatchesCurrent { get; init; } - internal required bool SqlGraphContractMatchesCurrent { get; init; } - internal required bool HdlGraphContractMatchesCurrent { get; init; } - internal required ReadableFileByteTracker ReadableFileBytes { get; init; } - internal required PostExtractionHookRunner PostExtractionHooks { get; init; } - internal required SymbolExtractionWorkerClient SymbolExtractionWorker { get; init; } - internal required IndexProgressReporter IndexProgress { get; init; } - internal required BlockingCollection ExtractionResults { get; init; } - internal required Task[] Workers { get; init; } - internal required TimeSpan ExtractionStallTimeout { get; init; } - internal required ActiveExtractionPhase?[] ActiveExtractionPhases { get; init; } - internal required CancellationToken CancellationToken { get; init; } - internal required Action CancelExtraction { get; init; } - internal required Action EnsureIndexingActivityVisible { get; init; } - internal required Action ReportJsonIndexProgressIfNeeded { get; init; } - internal required Action ThrowIfFullScanCancelled { get; init; } - internal required Action PublishProcessedCount { get; init; } - internal required Action SetCurrentJsonIndexFile { get; init; } - internal required Func GetCurrentJsonIndexFile { get; init; } - internal required Func GetDeferCSharpMutationsForIncompleteScan { get; init; } - internal required Func GetFtsMutated { get; init; } - internal required Func GetCSharpWorkspace { get; init; } - internal required Func GetCSharpPrepassSymbolArtifacts { get; init; } - internal required Func?> GetCSharpWorkspaceFileSnapshots { get; init; } - internal required Action DeferCSharpMutationsForLoadedSnapshotDrift { get; init; } - internal required Func TargetRequiresJavaScriptTypeScriptRefresh { get; init; } - internal required Func AllowReuseWithCurrentHotspotFamilyTrust { get; init; } - internal required Action RequireTypeScriptAugmentationRefresh { get; init; } - internal required Action WriteProjectRootOnce { get; init; } - internal required Action> InsertIssuesForIndexedFile { get; init; } - internal required Action CountFreshInsertedRows { get; init; } - internal required FullScanExtractionConsumerState State { get; init; } - } - - private sealed class FullScanExtractionConsumerState - { - internal int Processed { get; set; } - internal int Skipped { get; set; } - internal int Warnings { get; set; } - internal int ErrorsAdded { get; set; } - internal bool FtsMutated { get; set; } - internal bool MutualRecursionRefreshNeeded { get; set; } - internal bool CSharpMetadataTargetsNeedRefresh { get; set; } - internal int SymbolsDroppedByKindFilter { get; set; } - internal long ExtractedFiles { get; set; } - internal long ExtractedChunks { get; set; } - internal long ExtractedSymbols { get; set; } - internal long ExtractedReferences { get; set; } - internal HashSet? ReusedHotspotFamilyLanguages { get; set; } - internal HashSet? SkippedSymbolExtractorLanguages { get; set; } - internal required HashSet IndexedSymbolExtractorLanguages { get; init; } - internal required List ErrorList { get; init; } - internal required List FileErrorList { get; init; } - internal required List WarningList { get; init; } - } + private readonly record struct FullScanExtractionConsumerResources( + PostExtractionHookRunner PostExtractionHooks, + SymbolExtractionWorkerClient SymbolExtractionWorker, + BlockingCollection ExtractionResults, + Task[] Workers, + TimeSpan ExtractionStallTimeout, + ActiveExtractionPhase?[] ActiveExtractionPhases, + CancellationTokenSource ExtractionCancellation, + int ProcessedBeforeExtraction); - private static FullScanExtractionConsumerState ConsumeFullScanExtractionResults( - FullScanExtractionConsumerContext context) + private sealed partial class FullScanExtractionSession { - var lastExtractionProgressAt = Stopwatch.GetTimestamp(); - while (!context.ExtractionResults.IsCompleted) + internal void ConsumeFullScanExtractionResults( + in FullScanExtractionConsumerResources resources) { - var processed = context.ProcessedBeforeExtraction + context.State.Processed; - context.ThrowIfFullScanCancelled(processed, context.FilesCount); - if (!context.ExtractionResults.TryTake(out var item, millisecondsTimeout: 100)) + var lastExtractionProgressAt = Stopwatch.GetTimestamp(); + while (!resources.ExtractionResults.IsCompleted) { - ThrowIfFullScanExtractionStalled( - processed, - context.FilesCount, - context.ExtractionStallTimeout, - lastExtractionProgressAt, - context.GetCurrentJsonIndexFile(), - context.ActiveExtractionPhases, - context.CancelExtraction); - continue; + var processed = resources.ProcessedBeforeExtraction + Processed; + ThrowIfFullScanCancelled(processed, FilesCount); + if (!resources.ExtractionResults.TryTake(out var item, millisecondsTimeout: 100)) + { + ThrowIfFullScanExtractionStalled( + processed, + FilesCount, + resources.ExtractionStallTimeout, + lastExtractionProgressAt, + CurrentJsonIndexFile, + resources.ActiveExtractionPhases, + resources.ExtractionCancellation); + continue; + } + + lastExtractionProgressAt = Stopwatch.GetTimestamp(); + CurrentJsonIndexFile = item.RelativePath; + ProcessFullScanExtractionItem(in resources, item); + CompleteFullScanExtractionItem(resources.ProcessedBeforeExtraction); } - lastExtractionProgressAt = Stopwatch.GetTimestamp(); - context.SetCurrentJsonIndexFile(item.RelativePath); - ProcessFullScanExtractionItem(context, item); - CompleteFullScanExtractionItem(context); + Task.WaitAll(resources.Workers, CancellationToken); } - Task.WaitAll(context.Workers, context.CancellationToken); - return context.State; - } - - private static void ProcessFullScanExtractionItem( - FullScanExtractionConsumerContext context, - FullScanFileWorkItem item) - { - var state = context.State; - var options = context.Options; - var writer = context.Writer; - var indexFilePhase = item.FailurePhase ?? "preparing"; - var itemFileExtracted = item.Record == null ? 0L : 1L; - var itemChunksExtracted = item.Chunks?.Count ?? 0L; - var itemSymbolsExtracted = item.Symbols?.Count ?? 0L; - var itemReferencesExtracted = item.References?.Count ?? 0L; - context.EnsureIndexingActivityVisible(); - if (item.Exception is IndexExtractionStalledException stalledException) - RethrowPreservingStackTrace(stalledException); - - try + private void ProcessFullScanExtractionItem( + in FullScanExtractionConsumerResources resources, + FullScanFileWorkItem item) { - if (ShouldDeferFullScanCSharpItem(context, item)) + var options = Options; + var writer = Writer; + var indexFilePhase = item.FailurePhase ?? "preparing"; + var itemFileExtracted = item.Record == null ? 0L : 1L; + var itemChunksExtracted = item.Chunks?.Count ?? 0L; + var itemSymbolsExtracted = item.Symbols?.Count ?? 0L; + var itemReferencesExtracted = item.References?.Count ?? 0L; + FullScanProgress.EnsureIndexingActivityVisible(); + if (item.Exception is IndexExtractionStalledException stalledException) + RethrowPreservingStackTrace(stalledException); + + try { - state.Skipped++; - return; - } + if (ShouldDeferFullScanCSharpItem(item)) + { + Skipped++; + return; + } - if (item.Exception != null) - RethrowPreservingStackTrace(item.Exception); + if (item.Exception != null) + RethrowPreservingStackTrace(item.Exception); - if (item.Record == null) - { - RecordSkippedFullScanExtractionItem(context, item); - return; - } + if (item.Record == null) + { + RecordSkippedFullScanExtractionItem(item); + return; + } - var record = item.Record; - context.ReadableFileBytes.Remember(item.FileIndex, record.Size); - if (item.Warning != null && !options.Json && !options.Quiet) - { - context.IndexProgress.Pause(); - ConsoleUi.PrintWarning(item.Warning); - context.IndexProgress.Resume(); - } + var record = item.Record; + ReadableFileBytes.Remember(item.FileIndex, record.Size); + if (item.Warning != null && !options.Json && !options.Quiet) + { + IndexProgress.Pause(); + ConsoleUi.PrintWarning(item.Warning); + IndexProgress.Resume(); + } - var generatedSuppressionIssue = item.GeneratedSuppressionChecked - ? item.GeneratedSuppressionIssue - : context.Indexer.BuildGeneratedCodeExtractionSkippedIssue(record.Path); - var existingId = GetReusableFullScanFileId(context, record, generatedSuppressionIssue); - if (existingId != null) - { - RecordReusedFullScanExtractionItem(context, record); - return; - } + var generatedSuppressionIssue = item.GeneratedSuppressionChecked + ? item.GeneratedSuppressionIssue + : Indexer.BuildGeneratedCodeExtractionSkippedIssue(record.Path); + var existingId = GetReusableFullScanFileId(record, generatedSuppressionIssue); + if (existingId != null) + { + RecordReusedFullScanExtractionItem(record); + return; + } - if (record.Lang == "csharp") - state.CSharpMetadataTargetsNeedRefresh = true; - if (record.Lang == "typescript") - context.RequireTypeScriptAugmentationRefresh(); + if (record.Lang == "csharp") + CSharpMetadataTargetsNeedRefresh = true; + if (record.Lang == "typescript") + RequireTypeScriptAugmentationRefresh(); - var persistence = PersistFullScanFile(new FullScanFilePersistenceContext - { - Writer = writer, - Indexer = context.Indexer, - Options = options, - ProjectRoot = context.ProjectRoot, - Item = item, - Record = record, - GeneratedSuppressionIssue = generatedSuppressionIssue, - StartedWithNoIndexedFiles = context.StartedWithNoIndexedFiles, - DeferCSharpMutationsForIncompleteScan = - context.GetDeferCSharpMutationsForIncompleteScan(), - CSharpWorkspace = context.GetCSharpWorkspace(), - CSharpPrepassSymbolArtifacts = - context.GetCSharpPrepassSymbolArtifacts(), - PostExtractionHooks = context.PostExtractionHooks, - SymbolExtractionWorker = context.SymbolExtractionWorker, - CancellationToken = context.CancellationToken, - InsertIssuesForIndexedFile = context.InsertIssuesForIndexedFile, - WriteProjectRootOnce = context.WriteProjectRootOnce, - SetPhase = (path, phase) => + var persistence = PersistFullScanFile(new FullScanFilePersistenceContext + { + Writer = writer, + Indexer = Indexer, + Options = options, + ProjectRoot = ProjectRoot, + Item = item, + Record = record, + GeneratedSuppressionIssue = generatedSuppressionIssue, + StartedWithNoIndexedFiles = StartedWithNoIndexedFiles, + DeferCSharpMutationsForIncompleteScan = + DeferCSharpMutationsForIncompleteScan, + CSharpWorkspace = CSharpWorkspace, + CSharpPrepassSymbolArtifacts = + CSharpPrepassSymbolArtifacts, + PostExtractionHooks = resources.PostExtractionHooks, + SymbolExtractionWorker = resources.SymbolExtractionWorker, + CancellationToken = CancellationToken, + ExtractionSession = this, + WriteProjectRootOnce = WriteProjectRootOnce, + SetPhase = (path, phase) => + { + CurrentJsonIndexFile = path; + indexFilePhase = phase; + }, + }); + itemChunksExtracted = persistence.ExtractedChunks; + itemSymbolsExtracted = persistence.ExtractedSymbols; + itemReferencesExtracted = persistence.ExtractedReferences; + SymbolsDroppedByKindFilter += persistence.SymbolsDroppedByKindFilter; + MutualRecursionRefreshNeeded |= persistence.MutualRecursionRefreshNeeded; + CSharpMetadataTargetsNeedRefresh |= persistence.CSharpMetadataTargetsNeedRefresh; + FtsMutated = true; + if (persistence.StampSymbolExtractorLanguage + && !string.IsNullOrWhiteSpace(record.Lang)) { - context.SetCurrentJsonIndexFile(path); - indexFilePhase = phase; - }, - }); - itemChunksExtracted = persistence.ExtractedChunks; - itemSymbolsExtracted = persistence.ExtractedSymbols; - itemReferencesExtracted = persistence.ExtractedReferences; - state.SymbolsDroppedByKindFilter += persistence.SymbolsDroppedByKindFilter; - state.MutualRecursionRefreshNeeded |= persistence.MutualRecursionRefreshNeeded; - state.CSharpMetadataTargetsNeedRefresh |= persistence.CSharpMetadataTargetsNeedRefresh; - state.FtsMutated = true; - if (persistence.StampSymbolExtractorLanguage - && !string.IsNullOrWhiteSpace(record.Lang)) + IndexedSymbolExtractorLanguages.Add(record.Lang); + } + CountFreshInsertedRows( + persistence.PersistedChunks, + persistence.PersistedSymbols, + persistence.PersistedReferences); + IndexProgress.WriteVerbose(persistence.VerboseMessage); + } + catch (IndexExtractionStalledException) { - state.IndexedSymbolExtractorLanguages.Add(record.Lang); + throw; } - context.CountFreshInsertedRows( - persistence.PersistedChunks, - persistence.PersistedSymbols, - persistence.PersistedReferences); - context.IndexProgress.WriteVerbose(persistence.VerboseMessage); - } - catch (IndexExtractionStalledException) - { - throw; - } - catch (Exception ex) - { - LogIndexFileFailure("index_file_failed", item.FilePath, indexFilePhase, ex); - state.ErrorsAdded++; - var errorMessage = FormatIndexFileException(ex); - state.ErrorList.Add(new CliJsonMessage(item.FilePath, errorMessage)); - if (state.FileErrorList.Count < PartialIndexFileErrorLimit) - state.FileErrorList.Add(BuildIndexFileError(item.RelativePath, indexFilePhase, ex)); - if (!options.Json) + catch (Exception ex) { - context.IndexProgress.Pause(); - ConsoleUi.ClearProgressLine(); - ConsoleUi.TryWriteErrorLine( - FormatPerFileErrorLine("ERR ", item.FilePath, ex, errorMessage)); - context.IndexProgress.Resume(); + LogIndexFileFailure("index_file_failed", item.FilePath, indexFilePhase, ex); + ErrorsAdded++; + var errorMessage = FormatIndexFileException(ex); + ErrorList.Add(new CliJsonMessage(item.FilePath, errorMessage)); + if (FileErrorList.Count < PartialIndexFileErrorLimit) + FileErrorList.Add(BuildIndexFileError(item.RelativePath, indexFilePhase, ex)); + if (!options.Json) + { + IndexProgress.Pause(); + ConsoleUi.ClearProgressLine(); + ConsoleUi.TryWriteErrorLine( + FormatPerFileErrorLine("ERR ", item.FilePath, ex, errorMessage)); + IndexProgress.Resume(); + } + } + finally + { + ExtractedFiles += itemFileExtracted; + ExtractedChunks += itemChunksExtracted; + ExtractedSymbols += itemSymbolsExtracted; + ExtractedReferences += itemReferencesExtracted; } - } - finally - { - state.ExtractedFiles += itemFileExtracted; - state.ExtractedChunks += itemChunksExtracted; - state.ExtractedSymbols += itemSymbolsExtracted; - state.ExtractedReferences += itemReferencesExtracted; - } - } - - private static bool ShouldDeferFullScanCSharpItem( - FullScanExtractionConsumerContext context, - FullScanFileWorkItem item) - { - if (item.FileIndex < 0 - || context.FileTargets[item.FileIndex].Language != "csharp") - { - return false; } - var deferCurrentItem = context.GetDeferCSharpMutationsForIncompleteScan(); - if (!deferCurrentItem - && item.Exception is CSharpWorkspaceSnapshotDriftException driftException) + private bool ShouldDeferFullScanCSharpItem( + FullScanFileWorkItem item) { - context.DeferCSharpMutationsForLoadedSnapshotDrift(driftException.Path); - context.State.FtsMutated = context.GetFtsMutated(); - return true; - } + if (item.FileIndex < 0 + || FileTargets[item.FileIndex].Language != "csharp") + { + return false; + } - var workspaceFileSnapshots = context.GetCSharpWorkspaceFileSnapshots(); - if (!deferCurrentItem - && item.Record != null - && workspaceFileSnapshots != null - && !CSharpStaticInterfacePrepass.TryValidateLoadedFileStatSnapshot( - item.FilePath, - context.FileTargets[item.FileIndex].IndexPath, - context.FileTargets[item.FileIndex].DisplayRelativePath, - item.Record.Size, - item.Record.Modified, - workspaceFileSnapshots, - out var changedPath, - context.CancellationToken)) - { - context.DeferCSharpMutationsForLoadedSnapshotDrift( - changedPath ?? context.FileTargets[item.FileIndex].DisplayRelativePath); - context.State.FtsMutated = context.GetFtsMutated(); - return true; - } + var deferCurrentItem = DeferCSharpMutationsForIncompleteScan; + if (!deferCurrentItem + && item.Exception is CSharpWorkspaceSnapshotDriftException driftException) + { + DeferCSharpMutationsForLoadedSnapshotDrift(driftException.Path); + return true; + } - return deferCurrentItem; - } + var workspaceFileSnapshots = CSharpWorkspaceFileSnapshots; + if (!deferCurrentItem + && item.Record != null + && workspaceFileSnapshots != null + && !CSharpStaticInterfacePrepass.TryValidateLoadedFileStatSnapshot( + item.FilePath, + FileTargets[item.FileIndex].IndexPath, + FileTargets[item.FileIndex].DisplayRelativePath, + item.Record.Size, + item.Record.Modified, + workspaceFileSnapshots, + out var changedPath, + CancellationToken)) + { + DeferCSharpMutationsForLoadedSnapshotDrift( + changedPath ?? FileTargets[item.FileIndex].DisplayRelativePath); + return true; + } - private static void RecordSkippedFullScanExtractionItem( - FullScanExtractionConsumerContext context, - FullScanFileWorkItem item) - { - var state = context.State; - var path = item.RelativePath; - state.Warnings++; - state.WarningList.Add(new CliJsonMessage(path, item.Warning ?? "File skipped")); - if (!context.Options.Json - && !context.Options.Quiet - && item.Warning != null) - { - context.IndexProgress.Pause(); - ConsoleUi.PrintWarning(item.Warning); - context.IndexProgress.Resume(); + return deferCurrentItem; } - if (!context.Writer.HasFileAtPath(path)) + private void RecordSkippedFullScanExtractionItem( + FullScanFileWorkItem item) { - state.Skipped++; - return; - } + var path = item.RelativePath; + Warnings++; + WarningList.Add(new CliJsonMessage(path, item.Warning ?? "File skipped")); + if (!Options.Json + && !Options.Quiet + && item.Warning != null) + { + IndexProgress.Pause(); + ConsoleUi.PrintWarning(item.Warning); + IndexProgress.Resume(); + } - using var deleteTxn = context.Writer.BeginTransaction( - context.CancellationToken, - "full scan delete skipped file"); - if (!context.Writer.DeleteFileByPath(path)) - return; + if (!Writer.HasFileAtPath(path)) + { + Skipped++; + return; + } - state.CSharpMetadataTargetsNeedRefresh = true; - context.RequireTypeScriptAugmentationRefresh(); - context.WriteProjectRootOnce(); - deleteTxn.Commit(); - state.FtsMutated = true; - } + using var deleteTxn = Writer.BeginTransaction( + CancellationToken, + "full scan delete skipped file"); + if (!Writer.DeleteFileByPath(path)) + return; - private static long? GetReusableFullScanFileId( - FullScanExtractionConsumerContext context, - FileRecord record, - FileIssue? generatedSuppressionIssue) - { - var options = context.Options; - if (context.ForceExtractorRefresh - || options.Rebuild - || context.StartedWithNoIndexedFiles - || options.SymbolsOnly) - { - return null; + CSharpMetadataTargetsNeedRefresh = true; + RequireTypeScriptAugmentationRefresh(); + WriteProjectRootOnce(); + deleteTxn.Commit(); + FtsMutated = true; } - var targetRequiresRefresh = - context.TargetRequiresJavaScriptTypeScriptRefresh(record.Lang, record.Path); - return context.Writer.GetReusableUnchangedFileId( - record.Path, - record.Modified, - record.Checksum, - size: record.Size, - lines: record.Lines, - language: record.Lang, - generated: record.Generated, - maxSymbolsPerFile: options.MaxSymbolsPerFile, - maxReferencesPerFile: options.MaxReferencesPerFile, - generatedExtractionSuppressed: generatedSuppressionIssue != null, - allowReuse: context.SymbolKindFilterMatchesPrior - && !targetRequiresRefresh - && !context.PriorSymbolsOnlyGraphOmitted - && (record.Lang != "csharp" || context.CSharpIndexedProjectRootCompatible) - && (record.Lang != "csharp" || context.CSharpSymbolNameContractMatchesCurrent) - && (record.Lang != "csharp" || !context.GetCSharpWorkspace().HasStaticInterfaceContracts) - && (record.Lang != "sql" || context.SqlGraphContractMatchesCurrent) - && (record.Lang is not ("verilog" or "systemverilog" or "vhdl") - || context.HdlGraphContractMatchesCurrent) - && context.AllowReuseWithCurrentHotspotFamilyTrust(record.Lang)); - } + private long? GetReusableFullScanFileId( + FileRecord record, + FileIssue? generatedSuppressionIssue) + { + var options = Options; + if (ForceExtractorRefresh + || options.Rebuild + || StartedWithNoIndexedFiles + || options.SymbolsOnly) + { + return null; + } - private static void RecordReusedFullScanExtractionItem( - FullScanExtractionConsumerContext context, - FileRecord record) - { - var state = context.State; - var stalePurged = context.GetDeferCSharpMutationsForIncompleteScan() - ? 0 - : context.Writer.PurgeStaleFilesSharingChecksum( - context.ProjectRoot, + var targetRequiresRefresh = + TargetRequiresJavaScriptTypeScriptRefresh(record.Lang, record.Path); + return Writer.GetReusableUnchangedFileId( record.Path, - record.Checksum); - if (stalePurged > 0) - { - state.FtsMutated = true; - state.CSharpMetadataTargetsNeedRefresh = true; - context.RequireTypeScriptAugmentationRefresh(); - if (!context.Options.SymbolsOnly) - state.MutualRecursionRefreshNeeded = true; + record.Modified, + record.Checksum, + size: record.Size, + lines: record.Lines, + language: record.Lang, + generated: record.Generated, + maxSymbolsPerFile: options.MaxSymbolsPerFile, + maxReferencesPerFile: options.MaxReferencesPerFile, + generatedExtractionSuppressed: generatedSuppressionIssue != null, + allowReuse: SymbolKindFilterMatchesPrior + && !targetRequiresRefresh + && !PriorSymbolsOnlyGraphOmitted + && (record.Lang != "csharp" || CSharpIndexedProjectRootCompatible) + && (record.Lang != "csharp" || CSharpSymbolNameContractMatchesCurrent) + && (record.Lang != "csharp" || !CSharpWorkspace.HasStaticInterfaceContracts) + && (record.Lang != "sql" || SqlGraphContractMatchesCurrent) + && (record.Lang is not ("verilog" or "systemverilog" or "vhdl") + || HdlGraphContractMatchesCurrent) + && AllowReuseWithCurrentHotspotFamilyTrust(record.Lang)); } - state.Skipped++; - if (!string.IsNullOrWhiteSpace(record.Lang)) - { - state.SkippedSymbolExtractorLanguages ??= - new HashSet(StringComparer.Ordinal); - state.SkippedSymbolExtractorLanguages.Add(record.Lang); - } - if (FileIndexer.SupportsHotspotFamilyMarkerLanguage(record.Lang) - && record.Lang != null) + private void RecordReusedFullScanExtractionItem( + FileRecord record) { - state.ReusedHotspotFamilyLanguages ??= - new HashSet(StringComparer.Ordinal); - state.ReusedHotspotFamilyLanguages.Add(record.Lang); - } - if (context.Options.Verbose - && !context.Options.Json - && !context.Options.Quiet) - { - context.IndexProgress.Pause(); - ConsoleUi.ClearProgressLine(); - CommandOutputWriter.WriteLine($" [SKIP] {record.Path}"); - context.IndexProgress.Resume(); + var stalePurged = DeferCSharpMutationsForIncompleteScan + ? 0 + : Writer.PurgeStaleFilesSharingChecksum( + ProjectRoot, + record.Path, + record.Checksum); + if (stalePurged > 0) + { + FtsMutated = true; + CSharpMetadataTargetsNeedRefresh = true; + RequireTypeScriptAugmentationRefresh(); + if (!Options.SymbolsOnly) + MutualRecursionRefreshNeeded = true; + } + + Skipped++; + if (!string.IsNullOrWhiteSpace(record.Lang)) + { + SkippedSymbolExtractorLanguages ??= + new HashSet(StringComparer.Ordinal); + SkippedSymbolExtractorLanguages.Add(record.Lang); + } + if (FileIndexer.SupportsHotspotFamilyMarkerLanguage(record.Lang) + && record.Lang != null) + { + ReusedHotspotFamilyLanguages ??= + new HashSet(StringComparer.Ordinal); + ReusedHotspotFamilyLanguages.Add(record.Lang); + } + if (Options.Verbose + && !Options.Json + && !Options.Quiet) + { + IndexProgress.Pause(); + ConsoleUi.ClearProgressLine(); + CommandOutputWriter.WriteLine($" [SKIP] {record.Path}"); + IndexProgress.Resume(); + } } - } - private static void CompleteFullScanExtractionItem( - FullScanExtractionConsumerContext context) - { - context.State.Processed++; - var processed = context.ProcessedBeforeExtraction + context.State.Processed; - context.PublishProcessedCount(processed); - context.SetCurrentJsonIndexFile(null); - context.ThrowIfFullScanCancelled(processed, context.FilesCount); - context.ReportJsonIndexProgressIfNeeded(); - if (context.Options.Json || context.Options.Quiet) - return; + private void CompleteFullScanExtractionItem(int processedBeforeExtraction) + { + Processed++; + var processed = processedBeforeExtraction + Processed; + ProcessedCount = processed; + CurrentJsonIndexFile = null; + ThrowIfFullScanCancelled(processed, FilesCount); + FullScanProgress.ReportJsonIndexProgressIfNeeded(); + if (Options.Json || Options.Quiet) + return; - context.IndexProgress.Pause(); - ConsoleUi.PrintProgress(processed, context.FilesCount); - context.IndexProgress.Resume(); + IndexProgress.Pause(); + ConsoleUi.PrintProgress(processed, FilesCount); + IndexProgress.Resume(); + } } } diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.TargetSelection.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.TargetSelection.cs index 8d8f8f105c..8124f34ffd 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.TargetSelection.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.TargetSelection.cs @@ -5,399 +5,375 @@ namespace CodeIndex.Cli; public static partial class IndexCommandRunner { - private sealed class FullScanTargetSelectionContext + private sealed partial class FullScanPreWriteSession { - internal required DbWriter Writer { get; init; } - internal required FileIndexer Indexer { get; init; } - internal required IndexCommandOptions Options { get; init; } - internal required string ProjectRoot { get; init; } - internal required FullScanFileTarget[] FileTargets { get; init; } - internal required IReadOnlyList - CSharpPrepassTargets - { get; init; } - internal required FilePurgePlan StaleFilePurgePlan { get; init; } - internal required bool CanSkipTargetsBeforeContentLoad { get; init; } - internal required bool StartedWithNoIndexedFiles { get; init; } - internal required bool CSharpIndexedProjectRootCompatible + internal void PrepareExtractionTargets() { - get; - init; - } - - internal required int ExtractionParallelism { get; init; } - internal required bool? PriorCSharpStaticInterfaceSourceEvidence - { - get; - init; - } - - internal required CSharpStaticInterfaceWorkspaceSymbols CSharpWorkspace - { - get; - init; - } - - internal Dictionary? - CSharpWorkspaceFileSnapshots - { get; init; } - internal required bool ForceFullCSharpRefreshFromInvalidatedNoOp - { - get; - init; + ThrowIfFullScanCancelled(); + var selection = State.Selection; + if (Request.Reuse.CanSkipTargetsBeforeContentLoad) + { + SelectReusableFullScanTargets(); + } + else if (State.Scan.DeferCSharpMutationsForIncompleteScan) + { + SelectFullScanTargetsWithDeferredCSharp(); + } + else + { + selection.ExtractionWorkItemCount = + Request.Runtime.FileTargets.Length; + } } - internal required bool PreservePriorPositiveCSharpSourceNoOp + private void SelectReusableFullScanTargets() { - get; - init; - } + var fileTargets = Request.Runtime.FileTargets; + var statPreflightMatched = new bool[fileTargets.Length]; + var csharpNoOpHasInterveningWork = + State.Scan.StaleFilePurgePlan.Count > 0; + for (var fileIndex = 0; + fileIndex < fileTargets.Length; + fileIndex++) + { + ThrowIfFullScanCancelled(); + if (State.Scan.DeferCSharpMutationsForIncompleteScan + && fileTargets[fileIndex].Language == "csharp") + { + continue; + } + + statPreflightMatched[fileIndex] = + GetFullScanTargetStatMatch(fileIndex, true) != null; + if (!statPreflightMatched[fileIndex]) + csharpNoOpHasInterveningWork = true; + } - internal required bool CSharpSourceEvidenceForStamp { get; init; } - internal required bool CSharpSourceEvidenceComplete { get; init; } - internal required CancellationToken CancellationToken { get; init; } - internal required Action ThrowIfFullScanCancelled { get; init; } - internal required Func - GetDeferCSharpMutationsForIncompleteScan - { get; init; } - internal required Func - GetFullScanTargetStatMatch - { get; init; } - internal required Action - RecordFullScanTargetStatSkip - { get; init; } - internal required Action - DeferCSharpMutationsForIncompleteWorkspace - { get; init; } - internal required Func - IsExistingCSharpSymbolPathNowNonCSharp - { get; init; } - } + var revalidatedMatches = + new IndexedFileStatReuseResult?[fileTargets.Length]; + RevalidateNonCSharpFullScanTargets( + statPreflightMatched, + revalidatedMatches, + ref csharpNoOpHasInterveningWork); + var preservedCSharpNoOpInvalidated = + RevalidateCSharpFullScanTargets( + statPreflightMatched, + revalidatedMatches, + csharpNoOpHasInterveningWork); + if (preservedCSharpNoOpInvalidated) + RebuildInvalidatedFullScanCSharpNoOp(revalidatedMatches); + + var selection = State.Selection; + selection.ExtractionFileIndexes = + new List(fileTargets.Length); + for (var fileIndex = 0; + fileIndex < fileTargets.Length; + fileIndex++) + { + ThrowIfFullScanCancelled(); + if (State.Scan.DeferCSharpMutationsForIncompleteScan + && fileTargets[fileIndex].Language == "csharp") + { + RecordDeferredFullScanCSharpTarget(fileIndex); + continue; + } + + var revalidated = revalidatedMatches[fileIndex]; + if (revalidated != null) + { + RecordFullScanTargetStatSkip( + fileIndex, + revalidated.Value); + } + else + { + selection.ExtractionFileIndexes.Add(fileIndex); + } + } - private sealed class FullScanTargetSelectionState - { - internal List? ExtractionFileIndexes { get; set; } - internal int ExtractionWorkItemCount { get; set; } - internal required CSharpStaticInterfaceWorkspaceSymbols CSharpWorkspace - { - get; - set; + selection.ExtractionWorkItemCount = + selection.ExtractionFileIndexes.Count; } - internal Dictionary? - CSharpWorkspaceFileSnapshots - { get; set; } - internal bool ForceFullCSharpRefreshFromInvalidatedNoOp { get; set; } - internal bool PreservePriorPositiveCSharpSourceNoOp { get; set; } - internal bool CSharpSourceEvidenceForStamp { get; set; } - internal bool CSharpSourceEvidenceComplete { get; set; } - } - - private sealed record FullScanTargetSelectionResult( - List? ExtractionFileIndexes, - int ExtractionWorkItemCount, - CSharpStaticInterfaceWorkspaceSymbols CSharpWorkspace, - Dictionary? - CSharpWorkspaceFileSnapshots, - bool ForceFullCSharpRefreshFromInvalidatedNoOp, - bool PreservePriorPositiveCSharpSourceNoOp, - bool CSharpSourceEvidenceForStamp, - bool CSharpSourceEvidenceComplete); - - private static FullScanTargetSelectionResult - PrepareFullScanExtractionTargets( - FullScanTargetSelectionContext context) - { - context.ThrowIfFullScanCancelled(); - var state = new FullScanTargetSelectionState - { - CSharpWorkspace = context.CSharpWorkspace, - CSharpWorkspaceFileSnapshots = - context.CSharpWorkspaceFileSnapshots, - ForceFullCSharpRefreshFromInvalidatedNoOp = - context.ForceFullCSharpRefreshFromInvalidatedNoOp, - PreservePriorPositiveCSharpSourceNoOp = - context.PreservePriorPositiveCSharpSourceNoOp, - CSharpSourceEvidenceForStamp = - context.CSharpSourceEvidenceForStamp, - CSharpSourceEvidenceComplete = - context.CSharpSourceEvidenceComplete, - }; - - if (context.CanSkipTargetsBeforeContentLoad) - { - SelectReusableFullScanTargets(context, state); - } - else if (context.GetDeferCSharpMutationsForIncompleteScan()) - { - SelectFullScanTargetsWithDeferredCSharp(context, state); - } - else + private void RevalidateNonCSharpFullScanTargets( + IReadOnlyList statPreflightMatched, + IList revalidatedMatches, + ref bool csharpNoOpHasInterveningWork) { - state.ExtractionWorkItemCount = context.FileTargets.Length; + var fileTargets = Request.Runtime.FileTargets; + for (var fileIndex = 0; + fileIndex < fileTargets.Length; + fileIndex++) + { + ThrowIfFullScanCancelled(); + if (fileTargets[fileIndex].Language == "csharp") + continue; + + var revalidated = statPreflightMatched[fileIndex] + ? GetFullScanTargetStatMatch(fileIndex, false) + : null; + revalidatedMatches[fileIndex] = revalidated; + if (revalidated == null) + csharpNoOpHasInterveningWork = true; + } } - return new FullScanTargetSelectionResult( - state.ExtractionFileIndexes, - state.ExtractionWorkItemCount, - state.CSharpWorkspace, - state.CSharpWorkspaceFileSnapshots, - state.ForceFullCSharpRefreshFromInvalidatedNoOp, - state.PreservePriorPositiveCSharpSourceNoOp, - state.CSharpSourceEvidenceForStamp, - state.CSharpSourceEvidenceComplete); - } - - private static void SelectReusableFullScanTargets( - FullScanTargetSelectionContext context, - FullScanTargetSelectionState state) - { - var fileTargets = context.FileTargets; - var statPreflightMatched = new bool[fileTargets.Length]; - var csharpNoOpHasInterveningWork = - context.StaleFilePurgePlan.Count > 0; - for (var fileIndex = 0; fileIndex < fileTargets.Length; fileIndex++) + private bool RevalidateCSharpFullScanTargets( + IReadOnlyList statPreflightMatched, + IList revalidatedMatches, + bool csharpNoOpHasInterveningWork) { - context.ThrowIfFullScanCancelled(); - if (context.GetDeferCSharpMutationsForIncompleteScan() - && fileTargets[fileIndex].Language == "csharp") + var fileTargets = Request.Runtime.FileTargets; + var csharp = State.CSharp; + var preservedCSharpNoOpInvalidated = false; + for (var fileIndex = 0; + fileIndex < fileTargets.Length; + fileIndex++) { - continue; + ThrowIfFullScanCancelled(); + if (fileTargets[fileIndex].Language != "csharp" + || State.Scan.DeferCSharpMutationsForIncompleteScan) + { + continue; + } + + var revalidated = statPreflightMatched[fileIndex] + ? GetFullScanTargetStatMatch( + fileIndex, + csharp.PreservePriorPositiveSourceNoOp + && !csharpNoOpHasInterveningWork) + : null; + revalidatedMatches[fileIndex] = revalidated; + if (csharp.PreservePriorPositiveSourceNoOp + && revalidated == null) + { + preservedCSharpNoOpInvalidated = true; + } } - statPreflightMatched[fileIndex] = - context.GetFullScanTargetStatMatch( - fileIndex, - true) != null; - if (!statPreflightMatched[fileIndex]) - csharpNoOpHasInterveningWork = true; - } - - var revalidatedMatches = - new IndexedFileStatReuseResult?[fileTargets.Length]; - RevalidateNonCSharpFullScanTargets( - context, - statPreflightMatched, - revalidatedMatches, - ref csharpNoOpHasInterveningWork); - var preservedCSharpNoOpInvalidated = - RevalidateCSharpFullScanTargets( - context, - state, - statPreflightMatched, - revalidatedMatches, - csharpNoOpHasInterveningWork); - if (preservedCSharpNoOpInvalidated) - { - RebuildInvalidatedFullScanCSharpNoOp( - context, - state, - revalidatedMatches); + return preservedCSharpNoOpInvalidated; } - state.ExtractionFileIndexes = - new List(fileTargets.Length); - for (var fileIndex = 0; fileIndex < fileTargets.Length; fileIndex++) + private void RebuildInvalidatedFullScanCSharpNoOp( + IList revalidatedMatches) { - context.ThrowIfFullScanCancelled(); - if (context.GetDeferCSharpMutationsForIncompleteScan() - && fileTargets[fileIndex].Language == "csharp") + var request = Request; + var core = request.Core; + var baseline = request.Baseline; + var contracts = request.Contracts; + var runtime = request.Runtime; + var scan = State.Scan; + var csharp = State.CSharp; + csharp.Workspace = BuildStableFullScanCSharpWorkspace( + core.ProjectRoot, + runtime.CSharpPrepassTargets, + out var workspaceFileSnapshots, + () => CSharpStaticInterfacePrepass.BuildWorkspaceSymbols( + core.Writer, + core.Indexer, + runtime.CSharpPrepassTargets, + includeExistingSymbols: + contracts.CSharpIndexedProjectRootCompatible + && !core.Options.Rebuild + && !baseline.StartedWithNoIndexedFiles, + canReuseExistingSymbolsWithoutRead: null, + parallelism: runtime.ExtractionParallelism, + excludedExistingFileIds: + scan.StaleFilePurgePlan.FileIds, + isExistingSymbolPathExcluded: + IsExistingCSharpSymbolPathNowNonCSharp, + patternConfigsAlreadyLoaded: true, + cancellationToken: runtime.CancellationToken), + runtime.CancellationToken); + csharp.WorkspaceFileSnapshots = workspaceFileSnapshots; + csharp.PreservePriorPositiveSourceNoOp = false; + if (!csharp.Workspace.SourceContractEvidenceComplete) { - RecordDeferredFullScanCSharpTarget(context, fileIndex); - continue; + var incompleteSourcePaths = + csharp.Workspace.IncompleteSourcePaths; + DeferCSharpMutationsForIncompleteWorkspace(csharp.Workspace); + csharp.Evidence.ForStamp = false; + csharp.Evidence.Complete = false; + csharp.Workspace = + new CSharpStaticInterfaceWorkspaceSymbols( + [], + false, + SourceContractEvidenceComplete: false, + IncompleteSourcePaths: incompleteSourcePaths); + return; } - var revalidated = revalidatedMatches[fileIndex]; - if (revalidated != null) + var requiresFullCSharpRefresh = + baseline.PriorCSharpStaticInterfaceSourceEvidence == true + || csharp.Workspace.HasStaticInterfaceContracts + || csharp.Workspace.RequiresMemberReadReferenceRefresh; + csharp.ForceFullRefreshFromInvalidatedNoOp = + requiresFullCSharpRefresh; + csharp.Evidence.ForStamp = + csharp.Workspace.HasSourceStaticInterfaceContracts; + csharp.Evidence.Complete = true; + if (!requiresFullCSharpRefresh) + return; + + csharp.Workspace = csharp.Workspace with { - context.RecordFullScanTargetStatSkip( - fileIndex, - revalidated.Value); - } - else + HasStaticInterfaceContracts = true, + }; + for (var fileIndex = 0; + fileIndex < runtime.FileTargets.Length; + fileIndex++) { - state.ExtractionFileIndexes.Add(fileIndex); + if (runtime.FileTargets[fileIndex].Language == "csharp") + revalidatedMatches[fileIndex] = null; } } - state.ExtractionWorkItemCount = - state.ExtractionFileIndexes.Count; - } - - private static void RevalidateNonCSharpFullScanTargets( - FullScanTargetSelectionContext context, - IReadOnlyList statPreflightMatched, - IList revalidatedMatches, - ref bool csharpNoOpHasInterveningWork) - { - for (var fileIndex = 0; - fileIndex < context.FileTargets.Length; - fileIndex++) + private void SelectFullScanTargetsWithDeferredCSharp() { - context.ThrowIfFullScanCancelled(); - if (context.FileTargets[fileIndex].Language == "csharp") - continue; + var fileTargets = Request.Runtime.FileTargets; + var selection = State.Selection; + selection.ExtractionFileIndexes = + new List(fileTargets.Length); + for (var fileIndex = 0; + fileIndex < fileTargets.Length; + fileIndex++) + { + if (fileTargets[fileIndex].Language != "csharp") + { + selection.ExtractionFileIndexes.Add(fileIndex); + continue; + } - var revalidated = statPreflightMatched[fileIndex] - ? context.GetFullScanTargetStatMatch(fileIndex, false) - : null; - revalidatedMatches[fileIndex] = revalidated; - if (revalidated == null) - csharpNoOpHasInterveningWork = true; + RecordDeferredFullScanCSharpTarget(fileIndex); + } + + selection.ExtractionWorkItemCount = + selection.ExtractionFileIndexes.Count; } - } - private static bool RevalidateCSharpFullScanTargets( - FullScanTargetSelectionContext context, - FullScanTargetSelectionState state, - IReadOnlyList statPreflightMatched, - IList revalidatedMatches, - bool csharpNoOpHasInterveningWork) - { - var preservedCSharpNoOpInvalidated = false; - for (var fileIndex = 0; - fileIndex < context.FileTargets.Length; - fileIndex++) + private void RecordDeferredFullScanCSharpTarget(int fileIndex) { - context.ThrowIfFullScanCancelled(); - if (context.FileTargets[fileIndex].Language != "csharp" - || context.GetDeferCSharpMutationsForIncompleteScan()) + long currentSize = 0; + try { - continue; + var info = new FileInfo( + Request.Runtime.FileTargets[fileIndex].FilePath); + if (info.Exists && info.Length >= 0) + currentSize = info.Length; } - - var revalidated = statPreflightMatched[fileIndex] - ? context.GetFullScanTargetStatMatch( - fileIndex, - state.PreservePriorPositiveCSharpSourceNoOp - && !csharpNoOpHasInterveningWork) - : null; - revalidatedMatches[fileIndex] = revalidated; - if (state.PreservePriorPositiveCSharpSourceNoOp - && revalidated == null) + catch (Exception ex) when ( + ex is IOException + or UnauthorizedAccessException + or NotSupportedException + or ArgumentException) { - preservedCSharpNoOpInvalidated = true; } - } - - return preservedCSharpNoOpInvalidated; - } - private static void RebuildInvalidatedFullScanCSharpNoOp( - FullScanTargetSelectionContext context, - FullScanTargetSelectionState state, - IList revalidatedMatches) - { - state.CSharpWorkspace = BuildStableFullScanCSharpWorkspace( - context.ProjectRoot, - context.CSharpPrepassTargets, - out var workspaceFileSnapshots, - () => CSharpStaticInterfacePrepass.BuildWorkspaceSymbols( - context.Writer, - context.Indexer, - context.CSharpPrepassTargets, - includeExistingSymbols: - context.CSharpIndexedProjectRootCompatible - && !context.Options.Rebuild - && !context.StartedWithNoIndexedFiles, - canReuseExistingSymbolsWithoutRead: null, - parallelism: context.ExtractionParallelism, - excludedExistingFileIds: - context.StaleFilePurgePlan.FileIds, - isExistingSymbolPathExcluded: - context.IsExistingCSharpSymbolPathNowNonCSharp, - patternConfigsAlreadyLoaded: true, - cancellationToken: context.CancellationToken), - context.CancellationToken); - state.CSharpWorkspaceFileSnapshots = workspaceFileSnapshots; - state.PreservePriorPositiveCSharpSourceNoOp = false; - if (!state.CSharpWorkspace.SourceContractEvidenceComplete) - { - var incompleteSourcePaths = - state.CSharpWorkspace.IncompleteSourcePaths; - context.DeferCSharpMutationsForIncompleteWorkspace( - state.CSharpWorkspace); - state.CSharpSourceEvidenceForStamp = false; - state.CSharpSourceEvidenceComplete = false; - state.CSharpWorkspace = - new CSharpStaticInterfaceWorkspaceSymbols( - [], - false, - SourceContractEvidenceComplete: false, - IncompleteSourcePaths: incompleteSourcePaths); - return; + RecordFullScanTargetStatSkip( + fileIndex, + new IndexedFileStatReuseResult(0, currentSize)); } - var requiresFullCSharpRefresh = - context.PriorCSharpStaticInterfaceSourceEvidence == true - || state.CSharpWorkspace.HasStaticInterfaceContracts - || state.CSharpWorkspace - .RequiresMemberReadReferenceRefresh; - state.ForceFullCSharpRefreshFromInvalidatedNoOp = - requiresFullCSharpRefresh; - state.CSharpSourceEvidenceForStamp = - state.CSharpWorkspace.HasSourceStaticInterfaceContracts; - state.CSharpSourceEvidenceComplete = true; - if (!requiresFullCSharpRefresh) - return; - - state.CSharpWorkspace = state.CSharpWorkspace with + private IndexedFileStatReuseResult? GetFullScanTargetStatMatch( + int fileIndex, + bool allowCSharpPrepassCache) { - HasStaticInterfaceContracts = true, - }; - for (var fileIndex = 0; - fileIndex < context.FileTargets.Length; - fileIndex++) - { - if (context.FileTargets[fileIndex].Language == "csharp") - revalidatedMatches[fileIndex] = null; + var request = Request; + var core = request.Core; + var baseline = request.Baseline; + var contracts = request.Contracts; + var reuse = request.Reuse; + if (!reuse.CanSkipTargetsBeforeContentLoad) + return null; + + var csharp = State.CSharp; + var target = request.Runtime.FileTargets[fileIndex]; + var language = target.Language; + var targetRequiresRefresh = + reuse.JavaScriptTypeScriptRefreshRequired + && (IsJavaScriptTypeScriptLanguage(language) + || IsJavaScriptTypeScriptConfigPath(target.IndexPath)); + var allowReuse = contracts.SymbolKindFilterMatchesPrior + && !targetRequiresRefresh + && !baseline.PriorSymbolsOnlyGraphOmitted + && (language != "csharp" + || contracts.CSharpIndexedProjectRootCompatible) + && (language != "csharp" + || contracts.CSharpSymbolNameContractMatchesCurrent) + && (language != "csharp" + || !csharp.Workspace.HasStaticInterfaceContracts) + && (language != "sql" + || reuse.SqlGraphContractMatchesCurrent) + && (language is not ("verilog" or "systemverilog" or "vhdl") + || reuse.HdlGraphContractMatchesCurrent) + && AllowReuseWithCurrentHotspotFamilyTrust( + language, + reuse.HotspotFamilyTrustMatchesCurrent); + if (!allowReuse) + return null; + + if (allowCSharpPrepassCache + && language == "csharp" + && csharp.CSharpPrepassStatReuse != null + && csharp.CSharpPrepassStatReuse.TryGetValue( + target.IndexPath, + out var cachedCSharpPrepassReuse)) + { + return cachedCSharpPrepassReuse; + } + + return IndexedFileStatReuse.TryGetReusableUnchangedFile( + csharp.ReusableIndexedFileStats!, + target.FilePath, + target.IndexPath, + language, + target.GeneratedExtractionSuppressed); } - } - private static void SelectFullScanTargetsWithDeferredCSharp( - FullScanTargetSelectionContext context, - FullScanTargetSelectionState state) - { - state.ExtractionFileIndexes = - new List(context.FileTargets.Length); - for (var fileIndex = 0; - fileIndex < context.FileTargets.Length; - fileIndex++) + private void RecordFullScanTargetStatSkip( + int fileIndex, + IndexedFileStatReuseResult existingFile) { - if (context.FileTargets[fileIndex].Language != "csharp") + var runtime = Request.Runtime; + var options = Request.Core.Options; + var selection = State.Selection; + var target = runtime.FileTargets[fileIndex]; + var language = target.Language; + selection.Skipped++; + selection.Processed++; + selection.ReadableFileBytes.Remember(fileIndex, existingFile.Size); + if (!string.IsNullOrWhiteSpace(language)) { - state.ExtractionFileIndexes.Add(fileIndex); - continue; + selection.SkippedSymbolExtractorLanguages ??= + new HashSet(StringComparer.Ordinal); + selection.SkippedSymbolExtractorLanguages.Add(language); } - RecordDeferredFullScanCSharpTarget(context, fileIndex); - } - - state.ExtractionWorkItemCount = - state.ExtractionFileIndexes.Count; - } + if (FileIndexer.SupportsHotspotFamilyMarkerLanguage(language) + && language != null) + { + selection.ReusedHotspotFamilyLanguages ??= + new HashSet(StringComparer.Ordinal); + selection.ReusedHotspotFamilyLanguages.Add(language); + } - private static void RecordDeferredFullScanCSharpTarget( - FullScanTargetSelectionContext context, - int fileIndex) - { - long currentSize = 0; - try - { - var info = new FileInfo( - context.FileTargets[fileIndex].FilePath); - if (info.Exists && info.Length >= 0) - currentSize = info.Length; + if (options.Verbose && !options.Json && !options.Quiet) + { + CommandOutputWriter.WriteLine( + $" [SKIP] {target.IndexPath} (unchanged)"); + } } - catch (Exception ex) when ( - ex is IOException - or UnauthorizedAccessException - or NotSupportedException - or ArgumentException) + + private void ThrowIfFullScanCancelled() { + var runtime = Request.Runtime; + if (!runtime.CancellationToken.IsCancellationRequested) + return; + + throw new IndexInterruptedException( + State.Selection.Processed, + runtime.FilesCount, + runtime.ActualMode); } - - context.RecordFullScanTargetStatSkip( - fileIndex, - new IndexedFileStatReuseResult(0, currentSize)); } } diff --git a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs index b10f8b52bb..bdbd638408 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.FullScan.cs @@ -116,14 +116,6 @@ void WriteProjectRootOnce() } } - void ThrowIfFullScanCancelled(int filesProcessed, int? filesTotal) - { - if (!cancellationToken.IsCancellationRequested) - return; - - throw new IndexInterruptedException(filesProcessed, filesTotal, actualMode); - } - int? initialScanFileCapacity = options.Rebuild ? null : writer.GetIndexedFileCount(); var discovery = DiscoverFullScanFiles( indexer, @@ -151,11 +143,24 @@ void ThrowIfFullScanCancelled(int filesProcessed, int? filesTotal) csharpPrepassCapacity); var fileTargets = targetPreparation.FileTargets; var csharpPrepassTargets = targetPreparation.CSharpPrepassTargets; - var readableFileBytes = new ReadableFileByteTracker( - files.Count, - fileIndex => files[fileIndex], - projectRoot, - indexRunDiagnostics); + var preWriteSelection = new FullScanPreWriteSelectionState + { + ReadableFileBytes = new ReadableFileByteTracker( + files.Count, + fileIndex => files[fileIndex], + projectRoot, + indexRunDiagnostics), + }; + void ThrowIfFullScanCancelled(int filesProcessed, int? filesTotal) + { + if (!cancellationToken.IsCancellationRequested) + return; + + throw new IndexInterruptedException( + filesProcessed, + filesTotal, + actualMode); + } var errorList = discovery.ErrorList; var fileErrorList = errorList .Take(PartialIndexFileErrorLimit) @@ -244,33 +249,27 @@ void ThrowIfFullScanCancelled(int filesProcessed, int? filesTotal) : null; var purgedRefs = 0; - int processed = 0, skipped = 0, warnings = warningList.Count, errors = errorList.Count; + int warnings = warningList.Count, errors = errorList.Count; var ftsMutated = purged > 0; var symbolsDroppedByKindFilter = 0; var mutualRecursionRefreshNeeded = !options.SymbolsOnly && (!writer.ReferenceIdentityContractMatchesCurrent() || purged > 0); - var indexProgressVisible = false; + FullScanProgressSession? fullScanProgressForResume = null; var indexProgress = new IndexProgressReporter( options, "Indexing...", spinnerFrames, ConsoleUi.TryWriteErrorLine, - canResume: () => processed < files.Count && !indexProgressVisible, + canResume: () => preWriteSelection.Processed < files.Count + && !fullScanProgressForResume!.IndexProgressVisible, clearProgressLineBeforeWrite: true); - HashSet? reusedHotspotFamilyLanguages = null; - HashSet? skippedSymbolExtractorLanguages = null; var indexedSymbolExtractorLanguages = new HashSet(languageCounts.Count, StringComparer.Ordinal); - string? currentJsonIndexFile = null; - ActiveExtractionPhase?[] activeExtractionPhases = []; - using var fullScanProgress = new FullScanProgressSession( + using var fullScanProgress = fullScanProgressForResume = new FullScanProgressSession( options, files.Count, indexProgress, - () => processed, - () => indexProgressVisible, - () => currentJsonIndexFile, - () => activeExtractionPhases); + preWriteSelection); var extractionParallelism = Math.Max(1, options.Parallelism); var typeScriptAugmentationNeedsRefresh = !options.SymbolsOnly && (options.Rebuild @@ -299,37 +298,6 @@ void RecordCSharpWorkspaceFailure(string path, string phase, Exception exception fileErrorList.Add(BuildIndexFileError(path, phase, exception)); } - void DeferCSharpMutationsForIncompleteWorkspace( - CSharpStaticInterfaceWorkspaceSymbols workspace) - { - if (workspace.SourceContractEvidenceComplete) - return; - - deferCSharpMutationsForIncompleteScan = true; - staleFilePurgePlan = FilePurgePlan.Empty; - purged = 0; - ftsMutated = false; - hadCSharpStaticInterfaceContractsBeforePurge = false; - - var incompletePaths = workspace.IncompleteSourcePaths; - if (incompletePaths == null || incompletePaths.Count == 0) - { - RecordCSharpWorkspaceFailure( - "", - "csharp_prepass", - new IOException("C# static-interface workspace preflight could not read a source file.")); - return; - } - - foreach (var path in incompletePaths.Take(PartialIndexFileErrorLimit)) - { - RecordCSharpWorkspaceFailure( - path, - "csharp_prepass", - new IOException("C# static-interface workspace preflight could not read this source file.")); - } - } - void RequireTypeScriptAugmentationRefresh() { if (!typeScriptAugmentationReadyCleared) @@ -379,112 +347,78 @@ bool FullScanJavaScriptTypeScriptConfigChanged() return false; } - bool TargetRequiresJavaScriptTypeScriptRefresh(string? language, string indexPath) - => javaScriptTypeScriptRefreshRequired - && (IsJavaScriptTypeScriptLanguage(language) || IsJavaScriptTypeScriptConfigPath(indexPath)); - - void InsertIssuesForIndexedFile(long fileId, IReadOnlyList issues) - { - if (startedWithNoIndexedFiles) - writer.InsertIssuesForNewFile(fileId, issues); - else - writer.InsertIssues(fileId, issues); - } - - bool IsExistingCSharpSymbolPathNowNonCSharp(string indexPath) + var preWriteState = new FullScanPreWriteState { - var currentPath = Path.Combine( - projectRoot, - FileIndexer.NormalizeRelativePathForCurrentPlatform(indexPath)); - return scanResult.FileLanguages.TryGetValue(currentPath, out var currentLanguage) - && currentLanguage != "csharp"; - } - - var csharpPreflight = PrepareFullScanCSharpWorkspace( - new FullScanCSharpPreflightContext + Scan = new FullScanPreWriteMutableScanState { - Writer = writer, - Indexer = indexer, - Options = options, - ProjectRoot = projectRoot, - FileTargets = fileTargets, - CSharpPrepassTargets = csharpPrepassTargets, - CSharpPrepassCapacity = csharpPrepassCapacity, + HadCSharpStaticInterfaceContractsBeforePurge = + hadCSharpStaticInterfaceContractsBeforePurge, StaleFilePurgePlan = staleFilePurgePlan, - StartedWithNoIndexedFiles = startedWithNoIndexedFiles, - PriorIndexComplete = priorIndexComplete, - PriorReadiness = priorReadiness, - ScanHadErrors = scanHadErrors, - ForceExtractorRefresh = forceExtractorRefresh, - PriorSymbolsOnlyGraphOmitted = priorSymbolsOnlyGraphOmitted, - SymbolKindFilterMatchesPrior = symbolKindFilterMatchesPrior, - CSharpSymbolNameContractMatchesCurrent = + DeferCSharpMutationsForIncompleteScan = + deferCSharpMutationsForIncompleteScan, + Purged = purged, + FtsMutated = ftsMutated, + }, + CSharp = new FullScanPreWriteCSharpState(), + Selection = preWriteSelection, + Diagnostics = new FullScanPreWriteDiagnosticsState + { + ErrorList = errorList, + FileErrorList = fileErrorList, + WarningList = warningList, + ReportedCSharpWorkspaceFailures = + reportedCSharpWorkspaceFailures, + Errors = errors, + }, + }; + var preWriteSession = new FullScanPreWriteSession( + new FullScanPreWriteRequest( + new FullScanPreWriteCore( + writer, + indexer, + options, + projectRoot), + new FullScanPreWriteBaseline( + priorIndexComplete, + priorReadiness, + priorSymbolsOnlyGraphOmitted, + priorCSharpStaticInterfaceSourceEvidence, + startedWithNoIndexedFiles, + scanHadErrors, + projectRootWritten), + new FullScanPreWriteContracts( + symbolKindFilterMatchesPrior, csharpSymbolNameContractMatchesCurrent, - CSharpIndexedProjectRootCompatible = csharpIndexedProjectRootCompatible, - CSharpHotspotTrustMatchesCurrent = AllowReuseWithCurrentHotspotFamilyTrust( "csharp", hotspotFamilyTrustMatchesCurrent), - RequiresConservativeCSharpSourceRefresh = requiresConservativeCSharpSourceRefresh, - HadCSharpStaticInterfaceContractsBeforePurge = - hadCSharpStaticInterfaceContractsBeforePurge, - PriorCSharpStaticInterfaceSourceEvidence = - priorCSharpStaticInterfaceSourceEvidence, - ProjectRootWritten = projectRootWritten, - ExtractionParallelism = extractionParallelism, - FilesCount = files.Count, - ActualMode = actualMode, - CancellationToken = cancellationToken, - IsExistingCSharpSymbolPathNowNonCSharp = - IsExistingCSharpSymbolPathNowNonCSharp, - GetDeferCSharpMutationsForIncompleteScan = - () => deferCSharpMutationsForIncompleteScan, - GetPurged = () => purged, - DeferCSharpMutationsForIncompleteWorkspace = - DeferCSharpMutationsForIncompleteWorkspace, - }); + forceExtractorRefresh), + new FullScanPreWriteRuntime( + fileTargets, + csharpPrepassTargets, + scanResult.FileLanguages, + csharpPrepassCapacity, + extractionParallelism, + files.Count, + actualMode, + cancellationToken), + new FullScanPreWriteReusePolicy( + CanSkipTargetsBeforeContentLoad: + !forceExtractorRefresh + && !options.Rebuild + && !startedWithNoIndexedFiles + && !options.SymbolsOnly, + sqlGraphContractMatchesCurrent, + hdlGraphContractMatchesCurrent, + hotspotFamilyTrustMatchesCurrent, + javaScriptTypeScriptRefreshRequired)), + preWriteState); + preWriteSession.PrepareCSharpWorkspace(); + var csharpPreflight = preWriteState.CSharp; var reusableIndexedFileStats = csharpPreflight.ReusableIndexedFileStats; - var csharpPrepassStatReuse = - csharpPreflight.CSharpPrepassStatReuse; - var csharpWorkspaceFileSnapshots = - csharpPreflight.CSharpWorkspaceFileSnapshots; - var csharpWorkspace = csharpPreflight.CSharpWorkspace; - var csharpPrepassSymbolArtifacts = - csharpPreflight.CSharpPrepassSymbolArtifacts; - var forceFullCSharpRefreshFromInvalidatedNoOp = - csharpPreflight.ForceFullCSharpRefreshFromInvalidatedNoOp; - var preservePriorPositiveCSharpSourceNoOp = - csharpPreflight.PreservePriorPositiveCSharpSourceNoOp; - var csharpSourceEvidenceForStamp = - csharpPreflight.CSharpSourceEvidenceForStamp; - var csharpSourceEvidenceComplete = - csharpPreflight.CSharpSourceEvidenceComplete; - - void DeferCSharpMutationsForLoadedSnapshotDrift(string path) - { - csharpPrepassSymbolArtifacts?.Clear(); - csharpPrepassSymbolArtifacts = null; - path = FormatCSharpWorkspaceSnapshotPath(projectRoot, path); - deferCSharpMutationsForIncompleteScan = true; - preservePriorPositiveCSharpSourceNoOp = false; - csharpSourceEvidenceForStamp = false; - csharpSourceEvidenceComplete = false; - csharpWorkspaceFileSnapshots = null; - csharpWorkspace = new CSharpStaticInterfaceWorkspaceSymbols( - [], - HasStaticInterfaceContracts: true, - SourceContractEvidenceComplete: false, - IncompleteSourcePaths: [path]); - writer.SetCSharpStaticInterfaceSourceEvidence(null); - RecordCSharpWorkspaceFailure( - path, - "csharp_workspace_validation", - new IOException( - "A C# source changed after workspace preflight; rerun indexing to refresh the complete C# graph.")); - } if (options.MemoryTrace) memorySamples.Add(CaptureMemorySample("csharp_prepass", stopwatch)); @@ -501,336 +435,58 @@ void DeferCSharpMutationsForLoadedSnapshotDrift(string path) var persistedChunks = 0L; var persistedSymbols = 0L; var persistedReferences = 0L; - void CountFreshInsertedRows( - int chunkCount = 0, - int symbolCount = 0, - int referenceCount = 0) + var inputValidation = preWriteSession.PrepareWriteBoundary( + discovery.InputSnapshot); + if (!inputValidation.IsValid) { - persistedFiles++; - persistedChunks += chunkCount; - persistedSymbols += symbolCount; - persistedReferences += referenceCount; - if (!startedWithNoIndexedFiles) - return; - - freshCountFiles++; - freshCountChunks += chunkCount; - freshCountSymbols += symbolCount; - freshCountReferences += referenceCount; - } - - var canSkipFullScanTargetsBeforeContentLoad = !forceExtractorRefresh - && !options.Rebuild - && !startedWithNoIndexedFiles - && !options.SymbolsOnly; - - IndexedFileStatReuseResult? GetFullScanTargetStatMatch( - int fileIndex, - bool allowCSharpPrepassCache) - { - if (!canSkipFullScanTargetsBeforeContentLoad) - return null; - - var target = fileTargets[fileIndex]; - var language = target.Language; - var targetRequiresRefresh = TargetRequiresJavaScriptTypeScriptRefresh(language, target.IndexPath); - var allowReuse = symbolKindFilterMatchesPrior - && !targetRequiresRefresh - && !priorSymbolsOnlyGraphOmitted - && (language != "csharp" || csharpIndexedProjectRootCompatible) - && (language != "csharp" || csharpSymbolNameContractMatchesCurrent) - && (language != "csharp" || !csharpWorkspace.HasStaticInterfaceContracts) - && (language != "sql" || sqlGraphContractMatchesCurrent) - && (language is not ("verilog" or "systemverilog" or "vhdl") || hdlGraphContractMatchesCurrent) - && AllowReuseWithCurrentHotspotFamilyTrust(language, hotspotFamilyTrustMatchesCurrent); - var existingFile = !allowReuse - ? null - : allowCSharpPrepassCache - && language == "csharp" - && csharpPrepassStatReuse != null - && csharpPrepassStatReuse.TryGetValue(target.IndexPath, out var cachedCSharpPrepassReuse) - ? cachedCSharpPrepassReuse - : IndexedFileStatReuse.TryGetReusableUnchangedFile( - reusableIndexedFileStats!, - target.FilePath, - target.IndexPath, - language, - target.GeneratedExtractionSuppressed); - return existingFile; - } - - void RecordFullScanTargetStatSkip(int fileIndex, IndexedFileStatReuseResult existingFile) - { - var target = fileTargets[fileIndex]; - var language = target.Language; - skipped++; - processed++; - readableFileBytes.Remember(fileIndex, existingFile.Size); - if (!string.IsNullOrWhiteSpace(language)) - { - skippedSymbolExtractorLanguages ??= new HashSet(StringComparer.Ordinal); - skippedSymbolExtractorLanguages.Add(language); - } - if (FileIndexer.SupportsHotspotFamilyMarkerLanguage(language) && language != null) - { - reusedHotspotFamilyLanguages ??= new HashSet(StringComparer.Ordinal); - reusedHotspotFamilyLanguages.Add(language); - } - if (options.Verbose && !options.Json && !options.Quiet) - CommandOutputWriter.WriteLine($" [SKIP] {target.IndexPath} (unchanged)"); - } - - var targetSelection = PrepareFullScanExtractionTargets( - new FullScanTargetSelectionContext - { - Writer = writer, - Indexer = indexer, - Options = options, - ProjectRoot = projectRoot, - FileTargets = fileTargets, - CSharpPrepassTargets = csharpPrepassTargets, - StaleFilePurgePlan = staleFilePurgePlan, - CanSkipTargetsBeforeContentLoad = - canSkipFullScanTargetsBeforeContentLoad, - StartedWithNoIndexedFiles = startedWithNoIndexedFiles, - CSharpIndexedProjectRootCompatible = - csharpIndexedProjectRootCompatible, - ExtractionParallelism = extractionParallelism, - PriorCSharpStaticInterfaceSourceEvidence = - priorCSharpStaticInterfaceSourceEvidence, - CSharpWorkspace = csharpWorkspace, - CSharpWorkspaceFileSnapshots = - csharpWorkspaceFileSnapshots, - ForceFullCSharpRefreshFromInvalidatedNoOp = - forceFullCSharpRefreshFromInvalidatedNoOp, - PreservePriorPositiveCSharpSourceNoOp = - preservePriorPositiveCSharpSourceNoOp, - CSharpSourceEvidenceForStamp = - csharpSourceEvidenceForStamp, - CSharpSourceEvidenceComplete = - csharpSourceEvidenceComplete, - CancellationToken = cancellationToken, - ThrowIfFullScanCancelled = - () => ThrowIfFullScanCancelled( - processed, - files.Count), - GetDeferCSharpMutationsForIncompleteScan = - () => deferCSharpMutationsForIncompleteScan, - GetFullScanTargetStatMatch = - GetFullScanTargetStatMatch, - RecordFullScanTargetStatSkip = - RecordFullScanTargetStatSkip, - DeferCSharpMutationsForIncompleteWorkspace = - DeferCSharpMutationsForIncompleteWorkspace, - IsExistingCSharpSymbolPathNowNonCSharp = - IsExistingCSharpSymbolPathNowNonCSharp, - }); - var extractionFileIndexes = - targetSelection.ExtractionFileIndexes; - var extractionWorkItemCount = - targetSelection.ExtractionWorkItemCount; - csharpWorkspace = targetSelection.CSharpWorkspace; - csharpWorkspaceFileSnapshots = - targetSelection.CSharpWorkspaceFileSnapshots; - forceFullCSharpRefreshFromInvalidatedNoOp = - targetSelection - .ForceFullCSharpRefreshFromInvalidatedNoOp; - preservePriorPositiveCSharpSourceNoOp = - targetSelection.PreservePriorPositiveCSharpSourceNoOp; - csharpSourceEvidenceForStamp = - targetSelection.CSharpSourceEvidenceForStamp; - csharpSourceEvidenceComplete = - targetSelection.CSharpSourceEvidenceComplete; - - var useFtsBulkLoad = ShouldUseFullScanFtsBulkLoad( - options.Rebuild, - startedWithNoIndexedFiles, - extractionWorkItemCount, - staleFilePurgePlan, - scanHadErrors, - readableFileBytes, - reusableIndexedFileStats!, - fileTargets, - extractionFileIndexes, - () => ThrowIfFullScanCancelled(processed, files.Count)); - - var finalCSharpRevalidation = - RevalidateFinalFullScanCSharpNoOp( - new FullScanCSharpFinalRevalidationContext + return WriteFullScanSnapshotFailure( + inputValidation.ChangedPath, + new FullScanSnapshotFailureContext { Writer = writer, - Indexer = indexer, Options = options, + Stopwatch = stopwatch, + JsonContext = jsonContext, ProjectRoot = projectRoot, - FileTargets = fileTargets, - CSharpPrepassTargets = csharpPrepassTargets, - StaleFilePurgePlan = staleFilePurgePlan, - StartedWithNoIndexedFiles = - startedWithNoIndexedFiles, - CSharpIndexedProjectRootCompatible = - csharpIndexedProjectRootCompatible, - ExtractionParallelism = extractionParallelism, - PriorCSharpStaticInterfaceSourceEvidence = - priorCSharpStaticInterfaceSourceEvidence, - ReusableIndexedFileStats = - reusableIndexedFileStats!, - ExtractionFileIndexes = - extractionFileIndexes, - ExtractionWorkItemCount = - extractionWorkItemCount, - UseFtsBulkLoad = useFtsBulkLoad, - CSharpWorkspace = csharpWorkspace, - CSharpWorkspaceFileSnapshots = - csharpWorkspaceFileSnapshots, - ForceFullCSharpRefreshFromInvalidatedNoOp = - forceFullCSharpRefreshFromInvalidatedNoOp, - PreservePriorPositiveCSharpSourceNoOp = - preservePriorPositiveCSharpSourceNoOp, - CSharpSourceEvidenceForStamp = - csharpSourceEvidenceForStamp, - CSharpSourceEvidenceComplete = - csharpSourceEvidenceComplete, - CancellationToken = cancellationToken, - DeferCSharpMutationsForIncompleteWorkspace = - DeferCSharpMutationsForIncompleteWorkspace, - IsExistingCSharpSymbolPathNowNonCSharp = - IsExistingCSharpSymbolPathNowNonCSharp, + PriorReadiness = priorReadiness, + CSharpSymbolNameContractMatchesCurrent = csharpSymbolNameContractMatchesCurrent, + PriorMetadataTargetCsharpMatchesCurrent = priorMetadataTargetCsharpMatchesCurrent, + PriorFoldVersion = priorFoldVersion, + PriorFoldFingerprint = priorFoldFingerprint, + MemorySamples = memorySamples, + LanguageCounts = languageCounts, + FilesCount = files.Count, + Skipped = preWriteSelection.Skipped, + DanglingSymlinkCount = scanResult.DanglingSymlinks.Count, + Warnings = warnings, + Errors = preWriteState.Diagnostics.Errors, + SymbolsDroppedByKindFilter = symbolsDroppedByKindFilter, + ErrorList = errorList, + FileErrorList = fileErrorList, + WarningList = warningList, }); - extractionFileIndexes = - finalCSharpRevalidation.ExtractionFileIndexes; - extractionWorkItemCount = - finalCSharpRevalidation.ExtractionWorkItemCount; - useFtsBulkLoad = finalCSharpRevalidation.UseFtsBulkLoad; - csharpWorkspace = finalCSharpRevalidation.CSharpWorkspace; - csharpWorkspaceFileSnapshots = - finalCSharpRevalidation.CSharpWorkspaceFileSnapshots; - forceFullCSharpRefreshFromInvalidatedNoOp = - finalCSharpRevalidation - .ForceFullCSharpRefreshFromInvalidatedNoOp; - preservePriorPositiveCSharpSourceNoOp = - finalCSharpRevalidation - .PreservePriorPositiveCSharpSourceNoOp; - csharpSourceEvidenceForStamp = - finalCSharpRevalidation.CSharpSourceEvidenceForStamp; - csharpSourceEvidenceComplete = - finalCSharpRevalidation.CSharpSourceEvidenceComplete; - skipped -= finalCSharpRevalidation.PromotedCSharpTargetCount; - processed -= finalCSharpRevalidation.PromotedCSharpTargetCount; - if (finalCSharpRevalidation.PromotedAllCSharpTargets) - { - skippedSymbolExtractorLanguages?.Remove("csharp"); - reusedHotspotFamilyLanguages?.Remove("csharp"); } - if (discovery.InputSnapshot != null) - { - FullScanInputSnapshotBarrierForTesting?.Invoke("before_write"); - if (!indexer.TryValidateScanInputSnapshot( - discovery.InputSnapshot, - out var changedScanInputPath, - cancellationToken)) - { - return WriteFullScanSnapshotFailure( - changedScanInputPath, - new FullScanSnapshotFailureContext - { - Writer = writer, - Options = options, - Stopwatch = stopwatch, - JsonContext = jsonContext, - ProjectRoot = projectRoot, - PriorReadiness = priorReadiness, - CSharpSymbolNameContractMatchesCurrent = csharpSymbolNameContractMatchesCurrent, - PriorMetadataTargetCsharpMatchesCurrent = priorMetadataTargetCsharpMatchesCurrent, - PriorFoldVersion = priorFoldVersion, - PriorFoldFingerprint = priorFoldFingerprint, - MemorySamples = memorySamples, - LanguageCounts = languageCounts, - FilesCount = files.Count, - Skipped = skipped, - DanglingSymlinkCount = scanResult.DanglingSymlinks.Count, - Warnings = warnings, - Errors = errors, - SymbolsDroppedByKindFilter = symbolsDroppedByKindFilter, - ErrorList = errorList, - FileErrorList = fileErrorList, - WarningList = warningList, - }); - } - } - - if (!options.SymbolsOnly && !deferCSharpMutationsForIncompleteScan) - { - var changedFilePath = string.Empty; - var stableFiles = csharpWorkspaceFileSnapshots == null - || CSharpStaticInterfacePrepass.TryValidateFileStatSnapshots( - csharpPrepassTargets, - csharpWorkspaceFileSnapshots, - out changedFilePath, - cancellationToken); - if (!stableFiles) - { - csharpPrepassSymbolArtifacts?.Clear(); - csharpPrepassSymbolArtifacts = null; - var driftPath = FormatCSharpWorkspaceSnapshotPath(projectRoot, changedFilePath); - var incompleteWorkspace = new CSharpStaticInterfaceWorkspaceSymbols( - [], - HasStaticInterfaceContracts: true, - SourceContractEvidenceComplete: false, - IncompleteSourcePaths: [driftPath]); - DeferCSharpMutationsForIncompleteWorkspace(incompleteWorkspace); - preservePriorPositiveCSharpSourceNoOp = false; - csharpSourceEvidenceForStamp = false; - csharpSourceEvidenceComplete = false; - csharpWorkspaceFileSnapshots = null; - csharpWorkspace = incompleteWorkspace; - useFtsBulkLoad = false; - - var deferredCSharpIndexes = new List(csharpPrepassTargets.Count); - if (extractionFileIndexes == null) - { - extractionFileIndexes = new List(fileTargets.Length); - for (var fileIndex = 0; fileIndex < fileTargets.Length; fileIndex++) - { - if (fileTargets[fileIndex].Language == "csharp") - deferredCSharpIndexes.Add(fileIndex); - else - extractionFileIndexes.Add(fileIndex); - } - } - else - { - for (var extractionIndex = extractionFileIndexes.Count - 1; extractionIndex >= 0; extractionIndex--) - { - var fileIndex = extractionFileIndexes[extractionIndex]; - if (fileTargets[fileIndex].Language != "csharp") - continue; - deferredCSharpIndexes.Add(fileIndex); - extractionFileIndexes.RemoveAt(extractionIndex); - } - } - - foreach (var fileIndex in deferredCSharpIndexes) - { - long currentSize = 0; - try - { - var info = new FileInfo(fileTargets[fileIndex].FilePath); - if (info.Exists && info.Length >= 0) - currentSize = info.Length; - } - catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or NotSupportedException or ArgumentException) - { - } - RecordFullScanTargetStatSkip( - fileIndex, - new IndexedFileStatReuseResult(0, currentSize)); - } - - extractionFileIndexes.Sort(); - extractionWorkItemCount = extractionFileIndexes.Count; - } - } + staleFilePurgePlan = preWriteState.Scan.StaleFilePurgePlan; + deferCSharpMutationsForIncompleteScan = + preWriteState.Scan.DeferCSharpMutationsForIncompleteScan; + purged = preWriteState.Scan.Purged; + ftsMutated = preWriteState.Scan.FtsMutated; + errors = preWriteState.Diagnostics.Errors; + var extractionFileIndexes = + preWriteSelection.ExtractionFileIndexes; + var extractionWorkItemCount = + preWriteSelection.ExtractionWorkItemCount; + var useFtsBulkLoad = preWriteSelection.UseFtsBulkLoad; + var csharpWorkspace = csharpPreflight.Workspace; + var csharpWorkspaceFileSnapshots = + csharpPreflight.WorkspaceFileSnapshots; + var preservePriorPositiveCSharpSourceNoOp = + csharpPreflight.PreservePriorPositiveSourceNoOp; + var csharpSourceEvidenceForStamp = + csharpPreflight.Evidence.ForStamp; + var csharpSourceEvidenceComplete = + csharpPreflight.Evidence.Complete; // The captured scan has now crossed its only pre-write authority barrier. Start the // outer write scopes immediately afterwards so no durable readiness, evidence, purge, @@ -945,122 +601,100 @@ void RecordFullScanTargetStatSkip(int fileIndex, IndexedFileStatReuseResult exis fullScanProgress.ReportJsonIndexProgressIfNeeded(); - var extractionPipeline = RunFullScanExtractionPipeline( - new FullScanExtractionPipelineContext + var extractionSession = new FullScanExtractionSession + { + Request = new FullScanExtractionRequest( + new FullScanExtractionCore( + writer, + indexer, + options, + projectRoot, + fileTargets, + preWriteSelection.ReadableFileBytes, + indexProgress, + fullScanProgress), + new FullScanExtractionWork( + extractionFileIndexes, + extractionWorkItemCount, + extractionParallelism, + files.Count, + forceExtractorRefresh, + authoritativeFreshFoldRowsClaim, + cancellationToken, + actualMode), + new FullScanExtractionContracts( + priorSymbolsOnlyGraphOmitted, + symbolKindFilterMatchesPrior, + csharpIndexedProjectRootCompatible, + csharpSymbolNameContractMatchesCurrent, + sqlGraphContractMatchesCurrent, + hdlGraphContractMatchesCurrent, + startedWithNoIndexedFiles), + new FullScanExtractionReuse( + javaScriptTypeScriptRefreshRequired, + hotspotFamilyTrustMatchesCurrent)), + State = new FullScanExtractionState { - Writer = writer, - Indexer = indexer, - Options = options, - ProjectRoot = projectRoot, - FileTargets = fileTargets, - ExtractionFileIndexes = extractionFileIndexes, - ExtractionWorkItemCount = extractionWorkItemCount, - ExtractionParallelism = extractionParallelism, - FilesCount = files.Count, - ForceExtractorRefresh = forceExtractorRefresh, - StartedWithNoIndexedFiles = startedWithNoIndexedFiles, - AuthoritativeFreshFoldRowsClaim = authoritativeFreshFoldRowsClaim, - PriorSymbolsOnlyGraphOmitted = - priorSymbolsOnlyGraphOmitted, - SymbolKindFilterMatchesPrior = - symbolKindFilterMatchesPrior, - CSharpIndexedProjectRootCompatible = - csharpIndexedProjectRootCompatible, - CSharpSymbolNameContractMatchesCurrent = - csharpSymbolNameContractMatchesCurrent, - SqlGraphContractMatchesCurrent = - sqlGraphContractMatchesCurrent, - HdlGraphContractMatchesCurrent = - hdlGraphContractMatchesCurrent, - ReadableFileBytes = readableFileBytes, - IndexProgress = indexProgress, - FullScanProgress = fullScanProgress, - CancellationToken = cancellationToken, - GetProcessedCount = () => processed, - PublishProcessedCount = value => processed = value, - ThrowIfFullScanCancelled = - ThrowIfFullScanCancelled, - SetIndexProgressVisible = - value => indexProgressVisible = value, - SetActiveExtractionPhases = - phases => activeExtractionPhases = phases, - SetCurrentJsonIndexFile = - path => currentJsonIndexFile = path, - GetCurrentJsonIndexFile = - () => currentJsonIndexFile, - GetDeferCSharpMutationsForIncompleteScan = - () => deferCSharpMutationsForIncompleteScan, - GetFtsMutated = () => ftsMutated, - GetCSharpWorkspace = () => csharpWorkspace, - GetCSharpPrepassSymbolArtifacts = - () => csharpPrepassSymbolArtifacts, - GetCSharpWorkspaceFileSnapshots = - () => csharpWorkspaceFileSnapshots, - DeferCSharpMutationsForLoadedSnapshotDrift = - DeferCSharpMutationsForLoadedSnapshotDrift, - TargetRequiresJavaScriptTypeScriptRefresh = - TargetRequiresJavaScriptTypeScriptRefresh, - AllowReuseWithCurrentHotspotFamilyTrust = language => - AllowReuseWithCurrentHotspotFamilyTrust( - language, - hotspotFamilyTrustMatchesCurrent), - RequireTypeScriptAugmentationRefresh = - RequireTypeScriptAugmentationRefresh, - WriteProjectRootOnce = WriteProjectRootOnce, - InsertIssuesForIndexedFile = - InsertIssuesForIndexedFile, - CountFreshInsertedRows = CountFreshInsertedRows, - ConsumerState = new FullScanExtractionConsumerState + PreWrite = preWriteState, + Refresh = new FullScanExtractionRefreshState { FtsMutated = ftsMutated, - MutualRecursionRefreshNeeded = - mutualRecursionRefreshNeeded, - CSharpMetadataTargetsNeedRefresh = - csharpMetadataTargetsNeedRefresh, - SymbolsDroppedByKindFilter = - symbolsDroppedByKindFilter, - ReusedHotspotFamilyLanguages = - reusedHotspotFamilyLanguages, - SkippedSymbolExtractorLanguages = - skippedSymbolExtractorLanguages, - IndexedSymbolExtractorLanguages = - indexedSymbolExtractorLanguages, - ErrorList = errorList, - FileErrorList = fileErrorList, - WarningList = warningList, + MutualRecursionRefreshNeeded = mutualRecursionRefreshNeeded, + CSharpMetadataTargetsNeedRefresh = csharpMetadataTargetsNeedRefresh, + SymbolsDroppedByKindFilter = symbolsDroppedByKindFilter, + ReusedHotspotFamilyLanguages = preWriteSelection.ReusedHotspotFamilyLanguages, + SkippedSymbolExtractorLanguages = preWriteSelection.SkippedSymbolExtractorLanguages, + IndexedSymbolExtractorLanguages = indexedSymbolExtractorLanguages, }, - }); - var postExtractionHooks = - extractionPipeline.PostExtractionHooks; - var extractionState = extractionPipeline.ConsumerState; - if (extractionState != null) - { - skipped += extractionState.Skipped; - warnings += extractionState.Warnings; - errors += extractionState.ErrorsAdded; - ftsMutated = extractionState.FtsMutated; - mutualRecursionRefreshNeeded = - extractionState.MutualRecursionRefreshNeeded; - csharpMetadataTargetsNeedRefresh = - extractionState.CSharpMetadataTargetsNeedRefresh; - symbolsDroppedByKindFilter = - extractionState.SymbolsDroppedByKindFilter; - extractedFiles += extractionState.ExtractedFiles; - extractedChunks += extractionState.ExtractedChunks; - extractedSymbols += extractionState.ExtractedSymbols; - extractedReferences += extractionState.ExtractedReferences; - reusedHotspotFamilyLanguages = - extractionState.ReusedHotspotFamilyLanguages; - skippedSymbolExtractorLanguages = - extractionState.SkippedSymbolExtractorLanguages; - } + }, + External = new FullScanExtractionExternalOperations( + RequireTypeScriptAugmentationRefresh, + WriteProjectRootOnce), + }; + var postExtractionHooks = RunFullScanExtractionPipeline(extractionSession); + preWriteState.CSharp.PrepassSymbolArtifacts = null; + deferCSharpMutationsForIncompleteScan = + preWriteState.Scan.DeferCSharpMutationsForIncompleteScan; + csharpWorkspaceFileSnapshots = + preWriteState.CSharp.WorkspaceFileSnapshots; + csharpWorkspace = preWriteState.CSharp.Workspace; + preservePriorPositiveCSharpSourceNoOp = + preWriteState.CSharp.PreservePriorPositiveSourceNoOp; + csharpSourceEvidenceForStamp = preWriteState.CSharp.Evidence.ForStamp; + csharpSourceEvidenceComplete = preWriteState.CSharp.Evidence.Complete; + preWriteSelection.Skipped += extractionSession.State.Counts.Skipped; + persistedFiles += extractionSession.State.PersistenceCounts.PersistedFiles; + persistedChunks += extractionSession.State.PersistenceCounts.PersistedChunks; + persistedSymbols += extractionSession.State.PersistenceCounts.PersistedSymbols; + persistedReferences += extractionSession.State.PersistenceCounts.PersistedReferences; + freshCountFiles += extractionSession.State.PersistenceCounts.FreshFiles; + freshCountChunks += extractionSession.State.PersistenceCounts.FreshChunks; + freshCountSymbols += extractionSession.State.PersistenceCounts.FreshSymbols; + freshCountReferences += extractionSession.State.PersistenceCounts.FreshReferences; + warnings += extractionSession.State.Counts.Warnings; + errors = preWriteState.Diagnostics.Errors + extractionSession.State.Counts.Errors; + ftsMutated = extractionSession.State.Refresh.FtsMutated; + mutualRecursionRefreshNeeded = + extractionSession.State.Refresh.MutualRecursionRefreshNeeded; + csharpMetadataTargetsNeedRefresh = + extractionSession.State.Refresh.CSharpMetadataTargetsNeedRefresh; + symbolsDroppedByKindFilter = + extractionSession.State.Refresh.SymbolsDroppedByKindFilter; + extractedFiles += extractionSession.State.Counts.ExtractedFiles; + extractedChunks += extractionSession.State.Counts.ExtractedChunks; + extractedSymbols += extractionSession.State.Counts.ExtractedSymbols; + extractedReferences += extractionSession.State.Counts.ExtractedReferences; + preWriteSelection.ReusedHotspotFamilyLanguages = + extractionSession.State.Refresh.ReusedHotspotFamilyLanguages; + preWriteSelection.SkippedSymbolExtractorLanguages = + extractionSession.State.Refresh.SkippedSymbolExtractorLanguages; indexProgress.Pause(); if (options.MemoryTrace) memorySamples.Add(CaptureMemorySample("extraction", stopwatch)); - ThrowIfFullScanCancelled(processed, files.Count); + ThrowIfFullScanCancelled(preWriteSelection.Processed, files.Count); var referenceIdentityReadyForMutualRecursionRefresh = !deferCSharpMutationsForIncompleteScan && mutualRecursionRefreshNeeded ? writer.CSharpFamilyTrustAllowsReferenceIdentityReady( @@ -1127,7 +761,7 @@ void RecordFullScanTargetStatSkip(int fileIndex, IndexedFileStatReuseResult exis } if (options.MemoryTrace && !willRebuildTypeScriptAugmentation) memorySamples.Add(CaptureMemorySample("reference_graph", stopwatch)); - ThrowIfFullScanCancelled(processed, files.Count); + ThrowIfFullScanCancelled(preWriteSelection.Processed, files.Count); if (ftsBulkLoad != null) { var phase = ftsMutated ? "rebuilding text index" : "restoring text index triggers"; @@ -1162,7 +796,7 @@ void RecordFullScanTargetStatSkip(int fileIndex, IndexedFileStatReuseResult exis } if (options.MemoryTrace) memorySamples.Add(CaptureMemorySample("text_index", stopwatch)); - ThrowIfFullScanCancelled(processed, files.Count); + ThrowIfFullScanCancelled(preWriteSelection.Processed, files.Count); var readinessStableFiles = true; string? readinessChangedFilePath = null; if (discovery.InputSnapshot != null) @@ -1210,10 +844,21 @@ void RecordFullScanTargetStatSkip(int fileIndex, IndexedFileStatReuseResult exis cancellationToken); if (!readinessStableFiles || !readinessStableScanInputs) { - DeferCSharpMutationsForLoadedSnapshotDrift( + extractionSession.DeferCSharpMutationsForLoadedSnapshotDrift( readinessChangedFilePath ?? readinessChangedScanInputPath ?? ""); + deferCSharpMutationsForIncompleteScan = + preWriteState.Scan.DeferCSharpMutationsForIncompleteScan; + preservePriorPositiveCSharpSourceNoOp = + preWriteState.CSharp.PreservePriorPositiveSourceNoOp; + csharpSourceEvidenceForStamp = preWriteState.CSharp.Evidence.ForStamp; + csharpSourceEvidenceComplete = preWriteState.CSharp.Evidence.Complete; + csharpWorkspaceFileSnapshots = + preWriteState.CSharp.WorkspaceFileSnapshots; + csharpWorkspace = preWriteState.CSharp.Workspace; + errors = preWriteState.Diagnostics.Errors + + extractionSession.State.Counts.Errors; } } @@ -1325,9 +970,9 @@ void RecordFullScanTargetStatSkip(int fileIndex, IndexedFileStatReuseResult exis CancellationToken = cancellationToken, Errors = errors, FileErrorList = fileErrorList, - Processed = processed, + Processed = preWriteSelection.Processed, FileCount = files.Count, - Skipped = skipped, + Skipped = preWriteSelection.Skipped, Purged = purged, ScanHadErrors = scanHadErrors, StartedWithNoIndexedFiles = startedWithNoIndexedFiles, @@ -1342,16 +987,16 @@ void RecordFullScanTargetStatSkip(int fileIndex, IndexedFileStatReuseResult exis TypeScriptAugmentationDirtyNames = typeScriptAugmentationDirtyNames, UseScopedTypeScriptAugmentationRefresh = useScopedTypeScriptAugmentationRefresh, LanguageCounts = languageCounts, - ReusedHotspotFamilyLanguages = reusedHotspotFamilyLanguages, + ReusedHotspotFamilyLanguages = preWriteSelection.ReusedHotspotFamilyLanguages, PriorHotspotFamilyVersions = priorHotspotFamilyVersions, PriorHotspotFamilyMarkerFingerprints = priorHotspotFamilyMarkerFingerprints, CurrentHotspotFamilyMarkerFingerprints = currentHotspotFamilyMarkerFingerprints, IndexedSymbolExtractorLanguages = indexedSymbolExtractorLanguages, - SkippedSymbolExtractorLanguages = skippedSymbolExtractorLanguages, + SkippedSymbolExtractorLanguages = preWriteSelection.SkippedSymbolExtractorLanguages, PriorFoldVersion = priorFoldVersion, PriorFoldFingerprint = priorFoldFingerprint, ScanResult = scanResult, - ReadableFileBytes = readableFileBytes, + ReadableFileBytes = preWriteSelection.ReadableFileBytes, MemorySamples = memorySamples, TypeScriptAugmentationOwnsDeferredReferenceGraphRefresh = deferMutualRecursionRefreshToTypeScriptAugmentation @@ -1446,7 +1091,7 @@ void RecordFullScanTargetStatSkip(int fileIndex, IndexedFileStatReuseResult exis ExtractedReferences = extractedReferences, PersistedReferences = persistedReferences, FilesCount = files.Count, - Skipped = skipped, + Skipped = preWriteSelection.Skipped, Purged = purged, ScanResult = scanResult, LanguageCounts = languageCounts, diff --git a/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.ParallelConsumer.cs b/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.ParallelConsumer.cs new file mode 100644 index 0000000000..b255949b6b --- /dev/null +++ b/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.ParallelConsumer.cs @@ -0,0 +1,353 @@ +using CodeIndex.Indexer; + +namespace CodeIndex.Cli; + +public static partial class IndexCommandRunner +{ + private sealed partial class UpdateFileLoopSession + { + private void ConsumeParallelUpdateResult(UpdateParallelExtractionResult item) + { + var target = item.Target; + if (cancellationToken.IsCancellationRequested && item.Record != null) + { + DemoteReadinessOnce(); + csharpMetadataTargetsNeedRefresh = true; + } + ThrowIfUpdateCancelled(); + updateProgress.Start(); + var relPath = target.RelativePath; + currentUpdatePath = relPath; + currentUpdatePhase = item.FailurePhase ?? "preparing"; + var absPath = target.FilePath; + var dbPath = target.IndexPath; + var fileBatchMarked = false; + var csharpWorkspaceSnapshot = csharpWorkspaceSnapshots![dbPath]; + try + { + if (item.Record != null) + { + readableFileBytes.Remember(item.TargetIndex, item.Record.Size); + if (item.Warning != null && !options.Json && !options.Quiet) + { + updateProgress.Pause(); + ConsoleUi.PrintWarning(item.Warning); + updateProgress.Resume(); + } + DemoteReadinessOnce(); + csharpMetadataTargetsNeedRefresh = true; + } + var sourceContractSeenBeforeObservation = + postExtractionHooks.Value.SawCSharpStaticInterfaceSourceContract; + postExtractionHooks.Value.ObserveCSharpStaticInterfaceSourceContractEvidence( + item.HasCSharpStaticInterfaceSourceContract); + if (!csharpWorkspace.HasSourceStaticInterfaceContracts + && !sourceContractSeenBeforeObservation + && postExtractionHooks.Value.SawCSharpStaticInterfaceSourceContract) + { + parallelSourceWorkspaceDriftDetected = true; + RecordCSharpWorkspaceDrift( + relPath, + "A C# static-interface contract appeared after workspace preflight."); + skipped++; + return; + } + if (item.Exception is IndexExtractionStalledException stalledException) + { + if (!string.Equals( + item.FailurePhase, + "reading", + StringComparison.Ordinal)) + { + DemoteReadinessOnce(); + csharpMetadataTargetsNeedRefresh = true; + writer.MarkBatchInProgress(); + fileBatchMarked = true; + } + RethrowPreservingStackTrace( + new IndexExtractionStalledException( + updated + removed, + targetPaths.Count, + stalledException.Timeout, + stalledException.ActivePath, + stalledException.WorkerError)); + } + if (item.Exception is CSharpWorkspaceChangedException + or CSharpWorkspaceSnapshotDriftException) + { + RecordCSharpWorkspaceDrift( + relPath, + item.Exception.Message, + "reading"); + skipped++; + return; + } + if (item.Exception is FileIndexer.BinaryFileSkippedException + or FileIndexer.FileTooLargeSkippedException) + { + var skippedFile = HandleSkippedUpdateFile( + new SkippedUpdateFileHandlingContext + { + Writer = writer, + Indexer = indexer, + Options = options, + AbsolutePath = absPath, + RelativePath = relPath, + IndexPath = dbPath, + KnownLanguage = item.KnownLanguage, + ProjectRootWritten = persistenceOperations.IsProjectRootWritten(), + TargetIndex = item.TargetIndex, + ReadableFileBytes = readableFileBytes, + HasCSharpWorkspaceSnapshot = true, + CSharpWorkspaceSnapshot = csharpWorkspaceSnapshot, + CSharpWorkspaceSnapshots = csharpWorkspaceSnapshots, + WarningList = warningList, + UpdateProgress = updateProgress, + CancellationToken = cancellationToken, + DemoteReadinessOnce = DemoteReadinessOnce, + SetCurrentUpdatePhase = + phase => currentUpdatePhase = phase, + RecordCSharpWorkspaceDrift = + RecordCSharpWorkspaceDrift, + RecordUpdateFileFailure = + RecordUpdateFileFailure, + PurgeStaleUpdateCleanupPaths = + PurgeStaleUpdateCleanupPaths, + RequireTypeScriptAugmentationRefresh = + RequireTypeScriptAugmentationRefresh, + WriteProjectRootOnce = WriteProjectRootOnce, + RecordDynamicGraphFileRefresh = + RecordDynamicGraphFileRefresh, + }, + item.Exception); + updated += skippedFile.Updated; + skipped += skippedFile.Skipped; + warnings += skippedFile.Warnings; + mutualRecursionRefreshNeeded |= + skippedFile.MutualRecursionRefreshNeeded; + if (skippedFile.Updated > 0) + { + ftsMutated = true; + parallelExtractionEventForTesting?.Invoke( + new UpdateParallelExtractionTestEvent( + UpdateParallelExtractionEventKind.PersistenceCompleted, + item.TargetIndex, + target.DisplayRelativePath, + WorkerIndex: -1)); + } + return; + } + if (item.Exception is FileNotFoundException or DirectoryNotFoundException) + { + RecordCSharpWorkspaceDrift( + relPath, + "The C# file disappeared during its authoritative update pass."); + skipped++; + return; + } + if (item.Exception != null) + { + if (item.Exception is OperationCanceledException) + ThrowIfUpdateCancelled(); + if (!string.Equals( + item.FailurePhase, + "reading", + StringComparison.Ordinal)) + { + csharpMetadataTargetsNeedRefresh = true; + } + RecordUpdateFileFailure( + relPath, + item.FailurePhase ?? "reading", + item.Exception); + return; + } + + var record = item.Record!; + currentUpdatePhase = "validating"; + if (record.Lang != "csharp" + || !CSharpStaticInterfacePrepass.TryValidateLoadedFileStatSnapshot( + absPath, + dbPath, + target.DisplayRelativePath, + record.Size, + record.Modified, + csharpWorkspaceSnapshots, + out _, + cancellationToken)) + { + RecordCSharpWorkspaceDrift( + relPath, + "The C# file changed after extraction and before its authoritative update was persisted.", + "reading"); + skipped++; + return; + } + + currentUpdatePhase = "reading"; + parallelExtractionEventForTesting?.Invoke( + new UpdateParallelExtractionTestEvent( + UpdateParallelExtractionEventKind.PersistenceStarted, + item.TargetIndex, + target.DisplayRelativePath, + WorkerIndex: -1)); + var persistence = PersistPrecomputedUpdateFile( + item, + persistenceOperations.IsProjectRootWritten(), + ref fileBatchMarked); + symbolsDroppedByKindFilter += + persistence.SymbolsDroppedByKindFilter; + mutualRecursionRefreshNeeded |= + persistence.MutualRecursionRefreshNeeded; + updated++; + ftsMutated = true; + UpdateFileCommittedForTesting?.Invoke( + updated + removed, + targetPaths.Count); + parallelExtractionEventForTesting?.Invoke( + new UpdateParallelExtractionTestEvent( + UpdateParallelExtractionEventKind.PersistenceCompleted, + item.TargetIndex, + target.DisplayRelativePath, + WorkerIndex: -1)); + ThrowIfUpdateCancelled(); + updateProgress.WriteVerbose(persistence.VerboseMessage); + } + catch (IndexExtractionStalledException) + { + throw; + } + catch (Exception ex) + { + if (fileBatchMarked) + writer.ClearBatchInProgress(); + if (ex is CSharpWorkspaceChangedException) + { + RecordCSharpWorkspaceDrift(relPath, ex.Message); + skipped++; + return; + } + if (ex is OperationCanceledException) + ThrowIfUpdateCancelled(); + RecordUpdateFileFailure(relPath, currentUpdatePhase, ex); + } + } + + private void SetUpdatePhase(string path, string phase) + { + currentUpdatePath = path; + currentUpdatePhase = phase; + } + + private UpdateFilePersistenceResult PersistPrecomputedUpdateFile( + UpdateParallelExtractionResult item, + bool projectRootWritten, + ref bool fileBatchMarked) + { + var record = item.Record!; + var mutualRecursionRefreshNeeded = false; + + writer.MarkBatchInProgress(); + fileBatchMarked = true; + using var txn = writer.BeginTransaction( + cancellationToken, + "update precomputed file"); + var stalePurged = PurgeStaleUpdateCleanupPaths( + record.Path, + record.Checksum, + projectRootWritten); + if (stalePurged > 0 && !options.SymbolsOnly) + mutualRecursionRefreshNeeded = true; + if (stalePurged > 0) + RequireTypeScriptAugmentationRefresh(); + WriteProjectRootOnce(); + var fileId = writer.UpsertFile(record, out var referenceIdentityChanged); + if (!options.SymbolsOnly && referenceIdentityChanged) + mutualRecursionRefreshNeeded = true; + + SetUpdatePhase( + FormatIndexPhasePath(item.Target.DisplayRelativePath, "chunking"), + "chunking"); + var chunks = ReassignChunkFileIds(item.Chunks!, fileId); + if (item.GeneratedSuppressionIssue != null) + { + writer.InsertChunks(chunks, cancellationToken); + writer.InsertSymbols([], cancellationToken); + writer.InsertReferencesInAtomicFileScope( + [], + refreshMutualRecursionFlags: false, + cancellationToken); + SetUpdatePhase( + FormatIndexPhasePath(item.Target.DisplayRelativePath, "validating"), + "validating"); + writer.InsertIssues(fileId, item.Issues!); + SetUpdatePhase( + FormatIndexPhasePath(item.Target.DisplayRelativePath, "committing"), + "committing"); + writer.ClearBatchInProgress(); + txn.Commit(); + fileBatchMarked = false; + RecordDynamicGraphFileRefresh(record.Lang); + return new UpdateFilePersistenceResult( + 0, + mutualRecursionRefreshNeeded, + $" [OK ] {item.Target.RelativePath} ({chunks.Count} chunks, generated-code extraction skipped)"); + } + + SetUpdatePhase( + FormatIndexPhasePath(item.Target.DisplayRelativePath, "symbols"), + "symbols"); + var symbols = ReassignSymbolFileIds(item.Symbols!, fileId); + if (item.SymbolCapExceeded) + { + writer.InsertSymbols([], cancellationToken); + writer.InsertReferencesInAtomicFileScope( + [], + refreshMutualRecursionFlags: false, + cancellationToken); + writer.InsertIssues(fileId, item.Issues!); + writer.ClearBatchInProgress(); + txn.Commit(); + fileBatchMarked = false; + RecordDynamicGraphFileRefresh(record.Lang); + return new UpdateFilePersistenceResult( + 0, + mutualRecursionRefreshNeeded, + $" [SKIP] {item.Target.RelativePath} ({item.Issues![^1].Message})"); + } + + writer.InsertChunks(chunks, cancellationToken); + FileIndexer.ValidateSymbolLineRanges(record, symbols); + writer.InsertSymbols(symbols, cancellationToken); + SetUpdatePhase( + FormatIndexPhasePath(item.Target.DisplayRelativePath, "references"), + "references"); + var references = ReassignReferenceFileIds(item.References!, fileId); + writer.InsertReferencesInAtomicFileScope( + references, + refreshMutualRecursionFlags: false, + cancellationToken); + SetUpdatePhase( + FormatIndexPhasePath(item.Target.DisplayRelativePath, "validating"), + "validating"); + writer.InsertIssues(fileId, item.Issues!); + SetUpdatePhase( + FormatIndexPhasePath(item.Target.DisplayRelativePath, "committing"), + "committing"); + writer.ClearBatchInProgress(); + txn.Commit(); + fileBatchMarked = false; + RecordDynamicGraphFileRefresh(record.Lang); + if (!options.SymbolsOnly + && (symbols.Count > 0 || references.Count > 0)) + { + mutualRecursionRefreshNeeded = true; + } + + return new UpdateFilePersistenceResult( + 0, + mutualRecursionRefreshNeeded, + $" [OK ] {item.Target.RelativePath} ({chunks.Count} chunks, {symbols.Count} symbols, {references.Count} refs)"); + } + } +} diff --git a/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.SerialConsumer.cs b/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.SerialConsumer.cs new file mode 100644 index 0000000000..15dc6bdd52 --- /dev/null +++ b/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.SerialConsumer.cs @@ -0,0 +1,568 @@ +using CodeIndex.Database; +using CodeIndex.Indexer; +using CodeIndex.Models; + +namespace CodeIndex.Cli; + +public static partial class IndexCommandRunner +{ + private sealed partial class UpdateFileLoopSession + { + private void ConsumeSerialUpdateTarget( + UpdateFileTarget target, + int targetIndex, + LazyDisposable symbolExtractionWorker) + { + ThrowIfUpdateCancelled(); + updateProgress.Start(); + var relPath = target.RelativePath; + currentUpdatePath = relPath; + currentUpdatePhase = "preparing"; + var absPath = target.FilePath; + var dbPath = target.IndexPath; + var fileBatchMarked = false; + string? knownLanguage = null; + CSharpStaticInterfacePrepass.FileStatSnapshot csharpWorkspaceSnapshot = default; + var hasCSharpWorkspaceSnapshot = csharpWorkspaceSnapshots != null + && csharpWorkspaceSnapshots.TryGetValue(dbPath, out csharpWorkspaceSnapshot); + try + { + if (hasCSharpWorkspaceSnapshot + && !CSharpStaticInterfacePrepass.TryValidateLoadedFileStatSnapshot( + absPath, + dbPath, + relPath, + csharpWorkspaceSnapshot.Size, + csharpWorkspaceSnapshot.ModifiedUtc, + csharpWorkspaceSnapshots!, + out _, + cancellationToken)) + { + RecordCSharpWorkspaceDrift( + relPath, + "The C# file changed before its authoritative update pass."); + skipped++; + return; + } + + if (!File.Exists(LongPath.EnsureWindowsPrefix(absPath))) + { + if (hasCSharpWorkspaceSnapshot) + { + RecordCSharpWorkspaceDrift( + relPath, + "The C# file disappeared after contract preflight."); + skipped++; + return; + } + + using var deleteTxn = writer.BeginTransaction(cancellationToken, "update delete missing target"); + if (writer.DeleteFileByPath(dbPath)) + { + DemoteReadinessOnce(); + WriteProjectRootOnce(); + RequireTypeScriptAugmentationRefresh(); + deleteTxn.Commit(); + removed++; + ftsMutated = true; + mutualRecursionRefreshNeeded = true; + updateProgress.WriteVerbose($" [DEL ] {relPath}"); + } + else + { + skipped++; + updateProgress.WriteVerbose($" [SKIP] {relPath} (not in DB)"); + } + return; + } + + var pathFilter = indexer.EvaluatePathFilter(absPath); + RecordScanErrors(pathFilter.Errors); + if (pathFilter.ShouldSkip) + { + if (!pathFilter.ShouldDeleteExisting) + { + skipped++; + if (options.Verbose && !options.Json && !options.Quiet) + { + updateProgress.Pause(); + CommandOutputWriter.WriteLine($" [SKIP] {relPath} ({DescribePathFilter(pathFilter.FilterKind)})"); + updateProgress.Resume(); + } + return; + } + + using var deleteTxn = writer.BeginTransaction(cancellationToken, "update delete skipped path"); + if (writer.DeleteFileByPath(dbPath)) + { + DemoteReadinessOnce(); + WriteProjectRootOnce(); + RequireTypeScriptAugmentationRefresh(); + deleteTxn.Commit(); + removed++; + ftsMutated = true; + mutualRecursionRefreshNeeded = true; + if (options.Verbose && !options.Json && !options.Quiet) + { + updateProgress.Pause(); + CommandOutputWriter.WriteLine($" [DEL ] {relPath} ({DescribePathFilter(pathFilter.FilterKind)})"); + updateProgress.Resume(); + } + } + else + { + skipped++; + if (options.Verbose && !options.Json) + { + updateProgress.Pause(); + CommandOutputWriter.WriteLine($" [SKIP] {relPath} ({DescribePathFilter(pathFilter.FilterKind)})"); + updateProgress.Resume(); + } + } + return; + } + + var indexability = indexer.GetFileIndexabilityForIndexing(absPath); + var detection = indexer.TryDetectLanguageForIndexing(absPath, knownIndexability: indexability); + if (hasCSharpWorkspaceSnapshot + && (indexability != FileIndexer.FileProbeStatus.Supported + || detection.Status != FileIndexer.FileProbeStatus.Supported + || detection.Language != "csharp")) + { + RecordCSharpWorkspaceDrift( + relPath, + "The C# file changed language or indexability after contract preflight."); + skipped++; + return; + } + if (!hasCSharpWorkspaceSnapshot + && csharpWorkspaceSnapshots != null + && indexability == FileIndexer.FileProbeStatus.Supported + && detection.Status == FileIndexer.FileProbeStatus.Supported + && detection.Language == "csharp") + { + RecordCSharpWorkspaceDrift( + relPath, + "A C# target appeared after the authoritative workspace target set was captured."); + skipped++; + return; + } + if (indexability == FileIndexer.FileProbeStatus.Missing || detection.Status == FileIndexer.FileProbeStatus.Missing) + { + var message = $"{relPath}: skipped because it was deleted during indexing."; + warnings++; + warningList.Add(new CliJsonMessage(relPath, message)); + if (!options.Json && !options.Quiet) + { + updateProgress.Pause(); + ConsoleUi.PrintWarning(message); + updateProgress.Resume(); + } + + using var deleteTxn = writer.BeginTransaction(cancellationToken, "update delete missing during probe"); + if (writer.DeleteFileByPath(dbPath)) + { + DemoteReadinessOnce(); + WriteProjectRootOnce(); + RequireTypeScriptAugmentationRefresh(); + deleteTxn.Commit(); + removed++; + ftsMutated = true; + mutualRecursionRefreshNeeded = true; + } + else + { + skipped++; + } + return; + } + + if (indexability == FileIndexer.FileProbeStatus.ProbeFailed || detection.Status == FileIndexer.FileProbeStatus.ProbeFailed) + { + DemoteReadinessOnce(); + + errors++; + errorList.Add(new CliJsonMessage(relPath, "Could not probe file for indexability/language.")); + if (fileErrorList.Count < PartialIndexFileErrorLimit) + { + fileErrorList.Add(new StatusIndexFileError + { + File = FileIndexer.NormalizePathSeparators(relPath), + Category = "file_read_error", + Phase = "reading", + Detail = "Could not probe file for indexability/language.", + }); + } + if (!options.Json) + { + updateProgress.Pause(); + if (options.Verbose) + CommandErrorWriter.WriteStderr($" [ERR ] {relPath}: Could not probe file for indexability/language."); + else + CommandErrorWriter.WriteStderr($" [ERR ] {relPath}: Could not probe file for indexability/language."); + updateProgress.Resume(); + } + return; + } + + if (indexability != FileIndexer.FileProbeStatus.Supported || detection.Status != FileIndexer.FileProbeStatus.Supported) + { + if (!writer.HasFileAtPath(dbPath)) + { + using var purgeTxn = writer.BeginTransaction(cancellationToken, "update purge unsupported renamed target"); + var purged = PurgeStaleUpdateCleanupPaths( + dbPath, + checksum: null, + includeDirectoryAndStem: persistenceOperations.IsProjectRootWritten()); + if (purged > 0) + { + DemoteReadinessOnce(); + WriteProjectRootOnce(); + RequireTypeScriptAugmentationRefresh(); + purgeTxn.Commit(); + removed += purged; + ftsMutated = true; + mutualRecursionRefreshNeeded = true; + if (options.Verbose && !options.Json && !options.Quiet) + { + updateProgress.Pause(); + CommandOutputWriter.WriteLine($" [DEL ] {relPath} (unsupported renamed target)"); + updateProgress.Resume(); + } + } + else + { + skipped++; + if (options.Verbose && !options.Json && !options.Quiet) + { + updateProgress.Pause(); + CommandOutputWriter.WriteLine($" [SKIP] {relPath} (unsupported type)"); + updateProgress.Resume(); + } + } + return; + } + + DemoteReadinessOnce(); + using var deleteTxn = writer.BeginTransaction(cancellationToken, "update delete unsupported target"); + if (writer.DeleteFileByPath(dbPath)) + { + WriteProjectRootOnce(); + RequireTypeScriptAugmentationRefresh(); + deleteTxn.Commit(); + removed++; + ftsMutated = true; + mutualRecursionRefreshNeeded = true; + if (options.Verbose && !options.Json && !options.Quiet) + { + updateProgress.Pause(); + CommandOutputWriter.WriteLine($" [DEL ] {relPath} (no longer indexable)"); + updateProgress.Resume(); + } + } + else + { + skipped++; + if (options.Verbose && !options.Json) + { + updateProgress.Pause(); + CommandOutputWriter.WriteLine($" [SKIP] {relPath} (unsupported type)"); + updateProgress.Resume(); + } + } + return; + } + + if (FileIndexer.TryGetFileIdentity(absPath, out var identity, out var linkCount) + && linkCount > 1 + && !visitedFileIdentities.Add(identity)) + { + var message = "Skipped hardlinked file because the same file content was already indexed from another path."; + warnings++; + warningList.Add(new CliJsonMessage(relPath, message)); + if (!options.Json && !options.Quiet) + { + updateProgress.Pause(); + ConsoleUi.PrintWarning($"{relPath}: {message}"); + updateProgress.Resume(); + } + + using var deleteTxn = writer.BeginTransaction(); + if (writer.DeleteFileByPath(dbPath)) + { + DemoteReadinessOnce(); + WriteProjectRootOnce(); + RequireTypeScriptAugmentationRefresh(); + deleteTxn.Commit(); + removed++; + ftsMutated = true; + mutualRecursionRefreshNeeded = true; + } + else + { + skipped++; + } + return; + } + + var statReusableLanguage = GetStatReusableLanguage(absPath, detection); + var generatedExtractionSuppressed = indexer.IsGeneratedCodeExtractionSuppressed(dbPath); + var statMatchedFile = IndexedFileStatReuse.TryGetReusableUnchangedFile( + writer, + absPath, + dbPath, + statReusableLanguage, + options.MaxFileSizeBytes ?? FileIndexer.DefaultMaxFileSizeBytes, + options.MaxSymbolsPerFile, + options.MaxReferencesPerFile, + generatedExtractionSuppressed, + allowReuse: symbolKindFilterMatchesPrior + && (statReusableLanguage != "csharp" || csharpSymbolNameContractMatchesCurrent) + && (statReusableLanguage != "csharp" || !csharpWorkspace.HasStaticInterfaceContracts) + && (statReusableLanguage != "sql" || sqlGraphContractMatchesCurrent) + && (statReusableLanguage is not ("verilog" or "systemverilog" or "vhdl") || hdlGraphContractMatchesCurrent)); + if (statMatchedFile != null) + { + skipped++; + readableFileBytes.Remember(targetIndex, statMatchedFile.Value.Size); + if (options.Verbose && !options.Json && !options.Quiet) + { + updateProgress.Pause(); + CommandOutputWriter.WriteLine($" [SKIP] {relPath} (unchanged)"); + updateProgress.Resume(); + } + return; + } + + knownLanguage = scannedUpdateLanguages == null + ? statReusableLanguage + : FileIndexer.GetReusableDetectedLanguage(absPath, scannedUpdateLanguages); + + currentUpdatePhase = "reading"; + UpdateFileContentLoadForTesting?.Invoke(relPath); + var loaded = indexer.BuildLoadedRecordWithRawBytes( + absPath, + relPath, + knownLanguage, + cancellationToken); + var record = loaded.Record; + if (hasCSharpWorkspaceSnapshot + && (record.Lang != "csharp" + || !CSharpStaticInterfacePrepass.TryValidateLoadedFileStatSnapshot( + absPath, + dbPath, + relPath, + record.Size, + record.Modified, + csharpWorkspaceSnapshots!, + out _, + cancellationToken))) + { + RecordCSharpWorkspaceDrift( + relPath, + "The C# file changed while the authoritative update pass was reading it."); + skipped++; + return; + } + readableFileBytes.Remember(targetIndex, record.Size); + var warning = loaded.Warning; + var generatedSuppressionIssue = generatedExtractionSuppressed + ? indexer.BuildGeneratedCodeExtractionSkippedIssue(record.Path) + : null; + + if (warning != null && !options.Json && !options.Quiet) + { + updateProgress.Pause(); + ConsoleUi.PrintWarning(warning); + updateProgress.Resume(); + } + + var existingId = writer.GetReusableUnchangedFileId( + record.Path, + record.Modified, + record.Checksum, + size: record.Size, + lines: record.Lines, + language: record.Lang, + generated: record.Generated, + maxSymbolsPerFile: options.MaxSymbolsPerFile, + maxReferencesPerFile: options.MaxReferencesPerFile, + generatedExtractionSuppressed: generatedExtractionSuppressed, + allowReuse: symbolKindFilterMatchesPrior + && (record.Lang != "csharp" || csharpSymbolNameContractMatchesCurrent) + && (record.Lang != "csharp" || !csharpWorkspace.HasStaticInterfaceContracts) + && (record.Lang != "sql" || sqlGraphContractMatchesCurrent) + && (record.Lang is not ("verilog" or "systemverilog" or "vhdl") || hdlGraphContractMatchesCurrent)); + if (existingId != null) + { + using var purgeTxn = writer.BeginTransaction(cancellationToken, "update purge unchanged stale paths"); + var purged = PurgeStaleUpdateCleanupPaths( + record.Path, + record.Checksum, + includeDirectoryAndStem: persistenceOperations.IsProjectRootWritten()); + if (purged > 0) + { + DemoteReadinessOnce(); + WriteProjectRootOnce(); + RequireTypeScriptAugmentationRefresh(); + purgeTxn.Commit(); + removed += purged; + ftsMutated = true; + mutualRecursionRefreshNeeded = true; + } + skipped++; + if (options.Verbose && !options.Json && !options.Quiet) + { + updateProgress.Pause(); + CommandOutputWriter.WriteLine(purged > 0 + ? $" [SKIP] {relPath} (unchanged; purged {purged:N0} stale renamed path(s))" + : $" [SKIP] {relPath} (unchanged)"); + updateProgress.Resume(); + } + return; + } + + DemoteReadinessOnce(); + if (record.Lang == "csharp") + csharpMetadataTargetsNeedRefresh = true; + _ = postExtractionHooks.Value; + var persistence = PersistSerialUpdateFile( + relPath, + absPath, + record, + loaded, + generatedSuppressionIssue, + symbolExtractionWorker.Value, + persistenceOperations.IsProjectRootWritten(), + ref fileBatchMarked); + symbolsDroppedByKindFilter += persistence.SymbolsDroppedByKindFilter; + mutualRecursionRefreshNeeded |= persistence.MutualRecursionRefreshNeeded; + updated++; + ftsMutated = true; + UpdateFileCommittedForTesting?.Invoke(updated + removed, targetPaths.Count); + ThrowIfUpdateCancelled(); + updateProgress.WriteVerbose(persistence.VerboseMessage); + } + catch (IndexExtractionStalledException) + { + throw; + } + catch (Exception ex) + { + if (ex is CSharpWorkspaceChangedException) + { + if (fileBatchMarked) + writer.ClearBatchInProgress(); + RecordCSharpWorkspaceDrift(relPath, ex.Message); + skipped++; + return; + } + + if (ex is FileIndexer.BinaryFileSkippedException + or FileIndexer.FileTooLargeSkippedException) + { + if (fileBatchMarked) + writer.ClearBatchInProgress(); + + var skippedFile = HandleSkippedUpdateFile( + new SkippedUpdateFileHandlingContext + { + Writer = writer, + Indexer = indexer, + Options = options, + AbsolutePath = absPath, + RelativePath = relPath, + IndexPath = dbPath, + KnownLanguage = knownLanguage, + ProjectRootWritten = persistenceOperations.IsProjectRootWritten(), + TargetIndex = targetIndex, + ReadableFileBytes = readableFileBytes, + HasCSharpWorkspaceSnapshot = + hasCSharpWorkspaceSnapshot, + CSharpWorkspaceSnapshot = + csharpWorkspaceSnapshot, + CSharpWorkspaceSnapshots = + csharpWorkspaceSnapshots, + WarningList = warningList, + UpdateProgress = updateProgress, + CancellationToken = cancellationToken, + DemoteReadinessOnce = DemoteReadinessOnce, + SetCurrentUpdatePhase = + phase => currentUpdatePhase = phase, + RecordCSharpWorkspaceDrift = + RecordCSharpWorkspaceDrift, + RecordUpdateFileFailure = + RecordUpdateFileFailure, + PurgeStaleUpdateCleanupPaths = + PurgeStaleUpdateCleanupPaths, + RequireTypeScriptAugmentationRefresh = + RequireTypeScriptAugmentationRefresh, + WriteProjectRootOnce = WriteProjectRootOnce, + RecordDynamicGraphFileRefresh = + RecordDynamicGraphFileRefresh, + }, + ex); + updated += skippedFile.Updated; + skipped += skippedFile.Skipped; + warnings += skippedFile.Warnings; + mutualRecursionRefreshNeeded |= + skippedFile.MutualRecursionRefreshNeeded; + if (skippedFile.Updated > 0) + { + ftsMutated = true; + } + return; + } + + if (ex is FileNotFoundException or DirectoryNotFoundException) + { + if (fileBatchMarked) + writer.ClearBatchInProgress(); + + if (hasCSharpWorkspaceSnapshot) + { + RecordCSharpWorkspaceDrift( + relPath, + "The C# file disappeared during its authoritative update pass."); + skipped++; + return; + } + + var message = $"{relPath}: skipped because it was deleted during indexing."; + warnings++; + warningList.Add(new CliJsonMessage(relPath, message)); + if (!options.Json && !options.Quiet) + { + updateProgress.Pause(); + ConsoleUi.PrintWarning(message); + updateProgress.Resume(); + } + + if (writer.HasFileAtPath(dbPath)) + { + DemoteReadinessOnce(); + using var deleteTxn = writer.BeginTransaction(cancellationToken, "update delete missing during write"); + if (writer.DeleteFileByPath(dbPath)) + { + WriteProjectRootOnce(); + RequireTypeScriptAugmentationRefresh(); + deleteTxn.Commit(); + removed++; + ftsMutated = true; + mutualRecursionRefreshNeeded = true; + } + } + else + { + skipped++; + } + return; + } + + if (fileBatchMarked) + writer.ClearBatchInProgress(); + RecordUpdateFileFailure(relPath, currentUpdatePhase, ex); + } + } + } +} diff --git a/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.SerialPersistence.cs b/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.SerialPersistence.cs new file mode 100644 index 0000000000..84d9d5fedc --- /dev/null +++ b/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.SerialPersistence.cs @@ -0,0 +1,261 @@ +using CodeIndex.Indexer; +using CodeIndex.Indexer.Extensibility; +using CodeIndex.Indexer.Hooks; +using CodeIndex.Models; + +namespace CodeIndex.Cli; + +public static partial class IndexCommandRunner +{ + private sealed partial class UpdateFileLoopSession + { + private UpdateFilePersistenceResult PersistSerialUpdateFile( + string relativePath, + string absolutePath, + FileRecord record, + LoadedFileRecord loaded, + FileIssue? generatedSuppressionIssue, + SymbolExtractionWorkerClient symbolExtractionWorker, + bool projectRootWritten, + ref bool fileBatchMarked) + { + var mutualRecursionRefreshNeeded = false; + var symbolsDroppedByKindFilter = 0; + + writer.MarkBatchInProgress(); + fileBatchMarked = true; + var recordRequiresTypeScriptAugmentationRefresh = record.Lang == "typescript"; + using var txn = writer.BeginTransaction(cancellationToken, "update file"); + if (recordRequiresTypeScriptAugmentationRefresh) + RequireTypeScriptAugmentationRefresh(); + var stalePurged = PurgeStaleUpdateCleanupPaths( + record.Path, + record.Checksum, + projectRootWritten); + if (stalePurged > 0) + { + RequireTypeScriptAugmentationRefresh(); + if (!options.SymbolsOnly) + mutualRecursionRefreshNeeded = true; + } + WriteProjectRootOnce(); + var fileId = writer.UpsertFile(record, out var referenceIdentityChanged); + if (!options.SymbolsOnly && referenceIdentityChanged) + mutualRecursionRefreshNeeded = true; + SetUpdatePhase(FormatIndexPhasePath(relativePath, "chunking"), "chunking"); + var chunks = ChunkSplitter.SplitNormalized( + fileId, + loaded.Content, + loaded.Facts); + if (generatedSuppressionIssue != null) + { + writer.InsertChunks(chunks, cancellationToken); + writer.InsertSymbols([], cancellationToken); + writer.InsertReferencesInAtomicFileScope( + [], + refreshMutualRecursionFlags: false, + cancellationToken); + SetUpdatePhase( + FormatIndexPhasePath(relativePath, "validating"), + "validating"); + var generatedIssues = AppendIssueIfMissing( + FileIndexer.ValidateContent( + record.Path, + loaded.RawBytes, + loaded.Content, + record.Lang, + loaded.Inspection, + loaded.Facts), + generatedSuppressionIssue); + writer.InsertIssues(fileId, generatedIssues); + SetUpdatePhase( + FormatIndexPhasePath(relativePath, "committing"), + "committing"); + writer.ClearBatchInProgress(); + txn.Commit(); + fileBatchMarked = false; + RecordDynamicGraphFileRefresh(record.Lang); + return new UpdateFilePersistenceResult( + 0, + mutualRecursionRefreshNeeded, + $" [OK ] {relativePath} ({chunks.Count} chunks, generated-code extraction skipped)"); + } + + SetUpdatePhase(FormatIndexPhasePath(relativePath, "symbols"), "symbols"); + var symbolExtraction = ExtractSymbolsWithStallTimeout( + fileId, + record.Lang, + loaded.Content, + absolutePath, + projectRoot, + record.Path, + FormatIndexPhasePath(relativePath, "symbols"), + true, + loaded.HasOversizeLine, + loaded.ConflictMarkerLine, + symbolExtractionWorker, + cancellationToken); + var symbols = symbolExtraction.Symbols; + var symbolRegexTimeoutIssue = symbolExtraction.RegexTimeoutIssue; + var fileContext = new FileContext( + projectRoot, + record.Path, + absolutePath, + record.Lang); + var sourceContractSeenBeforeObservation = + postExtractionHooks.Value.SawCSharpStaticInterfaceSourceContract; + postExtractionHooks.Value.ObserveCSharpStaticInterfaceSourceSymbols( + fileContext, + symbols); + if (record.Lang == "csharp" + && !csharpWorkspace.HasSourceStaticInterfaceContracts + && !sourceContractSeenBeforeObservation + && postExtractionHooks.Value.SawCSharpStaticInterfaceSourceContract) + { + writer.SetCSharpStaticInterfaceSourceEvidence(null); + throw new CSharpWorkspaceChangedException( + "A C# static-interface contract appeared after workspace preflight."); + } + if (symbols.Count > options.MaxSymbolsPerFile) + { + var issue = BuildSymbolCountExceededIssue( + record.Path, + symbols.Count, + options.MaxSymbolsPerFile); + IReadOnlyList capIssues = symbolRegexTimeoutIssue == null + ? [issue] + : AppendIssue([symbolRegexTimeoutIssue], issue); + writer.InsertSymbols([], cancellationToken); + writer.InsertReferencesInAtomicFileScope( + [], + refreshMutualRecursionFlags: false, + cancellationToken); + writer.InsertIssues(fileId, capIssues); + writer.ClearBatchInProgress(); + txn.Commit(); + fileBatchMarked = false; + RecordDynamicGraphFileRefresh(record.Lang); + return new UpdateFilePersistenceResult( + 0, + mutualRecursionRefreshNeeded, + $" [SKIP] {relativePath} ({issue.Message})"); + } + + var familyScopeKey = indexer.GetFamilyScopeKey(absolutePath, record.Lang); + SymbolExtractor.ApplyFamilyScope(symbols, familyScopeKey, record.Lang); + postExtractionHooks.Value.OnSymbolsExtractedAfterSourceObservation( + fileContext, + symbols, + loaded.Content, + familyScopeKey); + symbolsDroppedByKindFilter = options.SymbolKindFilter.Apply(symbols); + if (symbols.Count > options.MaxSymbolsPerFile) + { + var issue = BuildSymbolCountExceededIssue( + record.Path, + symbols.Count, + options.MaxSymbolsPerFile); + IReadOnlyList capIssues = symbolRegexTimeoutIssue == null + ? [issue] + : AppendIssue([symbolRegexTimeoutIssue], issue); + writer.InsertSymbols([], cancellationToken); + writer.InsertReferencesInAtomicFileScope( + [], + refreshMutualRecursionFlags: false, + cancellationToken); + writer.InsertIssues(fileId, capIssues); + writer.ClearBatchInProgress(); + txn.Commit(); + fileBatchMarked = false; + RecordDynamicGraphFileRefresh(record.Lang); + return new UpdateFilePersistenceResult( + symbolsDroppedByKindFilter, + mutualRecursionRefreshNeeded, + $" [SKIP] {relativePath} ({issue.Message})"); + } + + writer.InsertChunks(chunks, cancellationToken); + FileIndexer.ValidateSymbolLineRanges(record, symbols); + writer.InsertSymbols(symbols, cancellationToken); + SetUpdatePhase( + FormatIndexPhasePath(relativePath, "references"), + "references"); + List references; + FileIssue? referenceRegexTimeoutIssue; + ReferenceExtractionResult referenceExtraction; + using (var regexTimeouts = BoundedRegex.CaptureTimeouts( + record.Lang, + "reference_extraction")) + { + referenceExtraction = ReferenceExtractor.ExtractDetailedNormalized( + fileId, + record.Lang, + loaded.Content, + loaded.HasOversizeLine, + symbols, + record.Path, + record.Lang == "csharp" ? csharpWorkspace.Symbols : null, + cancellationToken, + maxReferenceCount: options.MaxReferencesPerFile + 1, + conflictMarkerLine: loaded.ConflictMarkerLine, + workspaceRoot: projectRoot, + csharpStaticInterfaceMemberLookups: + csharpWorkspace.StaticInterfaceMemberLookups, + csharpQualifiedPatternLookups: + csharpWorkspace.QualifiedPatternLookups); + references = referenceExtraction.References; + referenceRegexTimeoutIssue = + BuildRegexTimeoutIssue(record.Path, regexTimeouts); + } + postExtractionHooks.Value.OnReferencesExtracted(fileContext, references); + FileIssue? referenceCapIssue = null; + if (references.Count > options.MaxReferencesPerFile) + { + referenceCapIssue = BuildReferenceCountExceededIssue( + record.Path, + references.Count, + options.MaxReferencesPerFile); + references = []; + } + writer.InsertReferencesInAtomicFileScope( + references, + refreshMutualRecursionFlags: false, + cancellationToken); + SetUpdatePhase( + FormatIndexPhasePath(relativePath, "validating"), + "validating"); + IReadOnlyList issues = FileIndexer.ValidateContent( + record.Path, + loaded.RawBytes, + loaded.Content, + record.Lang, + loaded.Inspection, + loaded.Facts); + if (symbolRegexTimeoutIssue != null) + issues = AppendIssue(issues, symbolRegexTimeoutIssue); + if (referenceRegexTimeoutIssue != null) + issues = AppendIssue(issues, referenceRegexTimeoutIssue); + issues = AppendReferenceExtractionDiagnosticIssues( + issues, + record.Path, + referenceExtraction.Diagnostics); + if (referenceCapIssue != null) + issues = AppendIssue(issues, referenceCapIssue); + writer.InsertIssues(fileId, issues); + SetUpdatePhase( + FormatIndexPhasePath(relativePath, "committing"), + "committing"); + writer.ClearBatchInProgress(); + txn.Commit(); + fileBatchMarked = false; + RecordDynamicGraphFileRefresh(record.Lang); + if (!options.SymbolsOnly && (symbols.Count > 0 || references.Count > 0)) + mutualRecursionRefreshNeeded = true; + + return new UpdateFilePersistenceResult( + symbolsDroppedByKindFilter, + mutualRecursionRefreshNeeded, + $" [OK ] {relativePath} ({chunks.Count} chunks, {symbols.Count} symbols, {references.Count} refs)"); + } + } +} diff --git a/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.Session.cs b/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.Session.cs new file mode 100644 index 0000000000..f5a1c8111f --- /dev/null +++ b/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.Session.cs @@ -0,0 +1,251 @@ +using System.Diagnostics; +using CodeIndex.Database; +using CodeIndex.Diagnostics; +using CodeIndex.Indexer; +using CodeIndex.Indexer.Extensibility; +using CodeIndex.Indexer.Hooks; +using CodeIndex.Models; + +namespace CodeIndex.Cli; + +public static partial class IndexCommandRunner +{ + private readonly record struct UpdateFileLoopRuntime( + DbWriter Writer, + FileIndexer Indexer, + IndexCommandOptions Options, + string ProjectRoot, + IndexProgressReporter Progress, + CancellationToken CancellationToken, + IReadOnlyCollection TargetPaths, + List MemorySamples); + + private record struct UpdateFileLoopCounters( + int Updated, + int Removed, + int Skipped, + int Warnings, + int Errors, + int SymbolsDroppedByKindFilter); + + private record struct UpdateFileLoopRefreshState( + bool FtsMutated, + bool MutualRecursionRefreshNeeded, + bool CSharpMetadataTargetsNeedRefresh, + string? CurrentPath, + string CurrentPhase, + bool ParallelSourceWorkspaceDriftDetected, + ReadableFileByteTracker? ReadableFileBytes); + + private readonly record struct UpdateFileLoopWorkspace( + CSharpStaticInterfaceWorkspaceSymbols CSharpWorkspace, + Dictionary? CSharpWorkspaceSnapshots, + IReadOnlyDictionary? ScannedUpdateLanguages, + bool SymbolKindFilterMatchesPrior, + bool CSharpSymbolNameContractMatchesCurrent, + bool SqlGraphContractMatchesCurrent, + bool HdlGraphContractMatchesCurrent); + + private readonly record struct UpdateFileLoopOutput( + Stopwatch Stopwatch, + List? IndexRunDiagnostics, + LazyDisposable PostExtractionHooks, + HashSet VisitedFileIdentities, + List ErrorList, + List FileErrorList, + List WarningList); + + private readonly record struct UpdateFileLoopReadinessOperations( + Action, string> RecordScanErrors, + Action RecordCSharpWorkspaceDrift, + Action DemoteReadinessOnce, + Action RequireTypeScriptAugmentationRefresh, + Action RecordDynamicGraphFileRefresh); + + private readonly record struct UpdateFileLoopPersistenceOperations( + Action WriteProjectRootOnce, + Func PurgeStaleUpdateCleanupPaths, + Func IsProjectRootWritten); + + private readonly record struct UpdateFileLoopParallelTesting( + Action? ExtractionEvent, + Func? ExtractionFailure, + Func? ExtractionStallTimeout, + Action? ExtractionWorkersStopped); + + private readonly record struct UpdateFileLoopRefreshResult( + bool FtsMutated, + bool MutualRecursionRefreshNeeded, + bool CSharpMetadataTargetsNeedRefresh); + + private readonly record struct UpdateFileLoopOutcome( + UpdateFileLoopCounters Counters, + UpdateFileLoopRefreshResult Refresh, + ReadableFileByteTracker ReadableFileBytes); + + private sealed partial class UpdateFileLoopSession + { + private readonly UpdateFileLoopRuntime runtime; + private UpdateFileLoopCounters counters; + private UpdateFileLoopRefreshState refresh; + private readonly UpdateFileLoopWorkspace workspace; + private readonly UpdateFileLoopOutput output; + private readonly UpdateFileLoopReadinessOperations readiness; + private readonly UpdateFileLoopPersistenceOperations persistenceOperations; + private readonly UpdateFileLoopParallelTesting testing; + + internal UpdateFileLoopSession( + UpdateFileLoopRuntime runtime, + UpdateFileLoopCounters counters, + UpdateFileLoopRefreshState refresh, + UpdateFileLoopWorkspace workspace, + UpdateFileLoopOutput output, + UpdateFileLoopReadinessOperations readiness, + UpdateFileLoopPersistenceOperations persistence, + UpdateFileLoopParallelTesting testing) + { + this.runtime = runtime; + this.counters = counters; + this.refresh = refresh; + this.workspace = workspace; + this.output = output; + this.readiness = readiness; + persistenceOperations = persistence; + this.testing = testing; + } + + private DbWriter writer => runtime.Writer; + private FileIndexer indexer => runtime.Indexer; + private IndexCommandOptions options => runtime.Options; + private string projectRoot => runtime.ProjectRoot; + private IndexProgressReporter updateProgress => runtime.Progress; + private CancellationToken cancellationToken => runtime.CancellationToken; + private IReadOnlyCollection targetPaths => runtime.TargetPaths; + private List memorySamples => runtime.MemorySamples; + private int updated { get => counters.Updated; set => counters.Updated = value; } + private int removed { get => counters.Removed; set => counters.Removed = value; } + private int skipped { get => counters.Skipped; set => counters.Skipped = value; } + private int warnings { get => counters.Warnings; set => counters.Warnings = value; } + private int errors { get => counters.Errors; set => counters.Errors = value; } + private int symbolsDroppedByKindFilter + { + get => counters.SymbolsDroppedByKindFilter; + set => counters.SymbolsDroppedByKindFilter = value; + } + private bool ftsMutated { get => refresh.FtsMutated; set => refresh.FtsMutated = value; } + private bool mutualRecursionRefreshNeeded + { + get => refresh.MutualRecursionRefreshNeeded; + set => refresh.MutualRecursionRefreshNeeded = value; + } + private bool csharpMetadataTargetsNeedRefresh + { + get => refresh.CSharpMetadataTargetsNeedRefresh; + set => refresh.CSharpMetadataTargetsNeedRefresh = value; + } + private string? currentUpdatePath + { + get => refresh.CurrentPath; + set => refresh.CurrentPath = value; + } + private string currentUpdatePhase + { + get => refresh.CurrentPhase; + set => refresh.CurrentPhase = value; + } + private bool parallelSourceWorkspaceDriftDetected + { + get => refresh.ParallelSourceWorkspaceDriftDetected; + set => refresh.ParallelSourceWorkspaceDriftDetected = value; + } + private ReadableFileByteTracker readableFileBytes + { + get => refresh.ReadableFileBytes!; + set => refresh.ReadableFileBytes = value; + } + private CSharpStaticInterfaceWorkspaceSymbols csharpWorkspace => workspace.CSharpWorkspace; + private Dictionary? + csharpWorkspaceSnapshots => workspace.CSharpWorkspaceSnapshots; + private IReadOnlyDictionary? scannedUpdateLanguages + => workspace.ScannedUpdateLanguages; + private bool symbolKindFilterMatchesPrior => workspace.SymbolKindFilterMatchesPrior; + private bool csharpSymbolNameContractMatchesCurrent + => workspace.CSharpSymbolNameContractMatchesCurrent; + private bool sqlGraphContractMatchesCurrent => workspace.SqlGraphContractMatchesCurrent; + private bool hdlGraphContractMatchesCurrent => workspace.HdlGraphContractMatchesCurrent; + private Stopwatch stopwatch => output.Stopwatch; + private List? indexRunDiagnostics => output.IndexRunDiagnostics; + private LazyDisposable postExtractionHooks + => output.PostExtractionHooks; + private HashSet visitedFileIdentities + => output.VisitedFileIdentities; + private List errorList => output.ErrorList; + private List fileErrorList => output.FileErrorList; + private List warningList => output.WarningList; + private Action? parallelExtractionEventForTesting + => testing.ExtractionEvent; + private Func? parallelExtractionFailureForTesting + => testing.ExtractionFailure; + private Func? extractionStallTimeoutForTesting => testing.ExtractionStallTimeout; + private Action? parallelExtractionWorkersStoppedForTesting => testing.ExtractionWorkersStopped; + + private void RecordScanErrors( + IEnumerable scanErrors, + string fatalPhase = "discovery") + => readiness.RecordScanErrors(scanErrors, fatalPhase); + + private void RecordCSharpWorkspaceDrift( + string relativePath, + string detail, + string fatalPhase = "reading") + => readiness.RecordCSharpWorkspaceDrift(relativePath, detail, fatalPhase); + + private void DemoteReadinessOnce() => readiness.DemoteReadinessOnce(); + private void WriteProjectRootOnce() => persistenceOperations.WriteProjectRootOnce(); + private void RequireTypeScriptAugmentationRefresh() + => readiness.RequireTypeScriptAugmentationRefresh(); + + private int PurgeStaleUpdateCleanupPaths( + string retainedRelativePath, + string? checksum, + bool includeDirectoryAndStem) + => persistenceOperations.PurgeStaleUpdateCleanupPaths( + retainedRelativePath, + checksum, + includeDirectoryAndStem); + + private void RecordDynamicGraphFileRefresh(string? language) + => readiness.RecordDynamicGraphFileRefresh(language); + + private void RecordUpdateFileFailure( + string relativePath, + string phase, + Exception exception) + { + DemoteReadinessOnce(); + LogIndexFileFailure("index_update_file_failed", relativePath, phase, exception); + + errors++; + var errorMessage = FormatIndexFileException(exception); + errorList.Add(new CliJsonMessage(relativePath, errorMessage)); + if (fileErrorList.Count < PartialIndexFileErrorLimit) + fileErrorList.Add(BuildIndexFileError(relativePath, phase, exception)); + if (!options.Json) + { + updateProgress.Pause(); + CommandErrorWriter.WriteStderr( + FormatPerFileErrorLine("ERR ", relativePath, exception, errorMessage)); + updateProgress.Resume(); + } + } + + private void ThrowIfUpdateCancelled() + { + if (!cancellationToken.IsCancellationRequested) + return; + + updateProgress.Pause(); + throw new IndexInterruptedException(updated + removed, targetPaths.Count); + } + } +} diff --git a/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.cs b/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.cs index 2a42efaaef..44df2fd7eb 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.Update.FileLoop.cs @@ -10,1207 +10,257 @@ namespace CodeIndex.Cli; public static partial class IndexCommandRunner { - private sealed class UpdateFileLoopContext + private sealed partial class UpdateFileLoopSession { - internal required DbWriter Writer { get; init; } - internal required FileIndexer Indexer { get; init; } - internal required IndexCommandOptions Options { get; init; } - internal required Stopwatch Stopwatch { get; init; } - internal required string ProjectRoot { get; init; } - internal List? IndexRunDiagnostics { get; init; } - internal required IReadOnlyCollection TargetPaths { get; init; } - internal required IndexProgressReporter UpdateProgress { get; init; } - internal required List MemorySamples { get; init; } - internal required int Updated { get; init; } - internal required int Removed { get; init; } - internal required int Skipped { get; init; } - internal required bool FtsMutated { get; init; } - internal required bool MutualRecursionRefreshNeeded { get; init; } - internal required bool CSharpMetadataTargetsNeedRefresh { get; init; } - internal required int SymbolsDroppedByKindFilter { get; init; } - internal required CSharpStaticInterfaceWorkspaceSymbols CSharpWorkspace { get; init; } - internal Dictionary? CSharpWorkspaceSnapshots { get; init; } - internal IReadOnlyDictionary? ScannedUpdateLanguages { get; init; } - internal required bool SymbolKindFilterMatchesPrior { get; init; } - internal required bool CSharpSymbolNameContractMatchesCurrent { get; init; } - internal required bool SqlGraphContractMatchesCurrent { get; init; } - internal required bool HdlGraphContractMatchesCurrent { get; init; } - internal required LazyDisposable PostExtractionHooks { get; init; } - internal required HashSet VisitedFileIdentities { get; init; } - internal required List ErrorList { get; init; } - internal required List FileErrorList { get; init; } - internal required List WarningList { get; init; } - internal required CancellationToken CancellationToken { get; init; } - internal required Action, string> RecordScanErrors { get; init; } - internal required Action RecordCSharpWorkspaceDrift { get; init; } - internal required Action DemoteReadinessOnce { get; init; } - internal required Action WriteProjectRootOnce { get; init; } - internal required Action RequireTypeScriptAugmentationRefresh { get; init; } - internal required Func PurgeStaleUpdateCleanupPaths { get; init; } - internal required Action RecordDynamicGraphFileRefresh { get; init; } - internal required Action RecordUpdateFileFailure { get; init; } - internal required Func IsProjectRootWritten { get; init; } - } - - private sealed record UpdateFileLoopResult( - int Updated, - int Removed, - int Skipped, - int Warnings, - int Errors, - bool FtsMutated, - bool MutualRecursionRefreshNeeded, - bool CSharpMetadataTargetsNeedRefresh, - int SymbolsDroppedByKindFilter, - ReadableFileByteTracker ReadableFileBytes); - - private static UpdateFileLoopResult RunUpdateFileLoop(UpdateFileLoopContext context) - { - var writer = context.Writer; - var indexer = context.Indexer; - var options = context.Options; - var stopwatch = context.Stopwatch; - var projectRoot = context.ProjectRoot; - var indexRunDiagnostics = context.IndexRunDiagnostics; - var targetPaths = context.TargetPaths; - var updateProgress = context.UpdateProgress; - var memorySamples = context.MemorySamples; - var updated = context.Updated; - var removed = context.Removed; - var skipped = context.Skipped; - var warnings = 0; - var errors = 0; - var ftsMutated = context.FtsMutated; - var mutualRecursionRefreshNeeded = context.MutualRecursionRefreshNeeded; - var csharpMetadataTargetsNeedRefresh = context.CSharpMetadataTargetsNeedRefresh; - var symbolsDroppedByKindFilter = context.SymbolsDroppedByKindFilter; - var csharpWorkspace = context.CSharpWorkspace; - var csharpWorkspaceSnapshots = context.CSharpWorkspaceSnapshots; - var scannedUpdateLanguages = context.ScannedUpdateLanguages; - var symbolKindFilterMatchesPrior = context.SymbolKindFilterMatchesPrior; - var csharpSymbolNameContractMatchesCurrent = - context.CSharpSymbolNameContractMatchesCurrent; - var sqlGraphContractMatchesCurrent = context.SqlGraphContractMatchesCurrent; - var hdlGraphContractMatchesCurrent = context.HdlGraphContractMatchesCurrent; - var postExtractionHooks = context.PostExtractionHooks; - var visitedFileIdentities = context.VisitedFileIdentities; - var errorList = context.ErrorList; - var fileErrorList = context.FileErrorList; - var warningList = context.WarningList; - var cancellationToken = context.CancellationToken; - var parallelExtractionEventForTesting = - UpdateParallelExtractionEventForTesting; - var parallelExtractionFailureForTesting = - UpdateParallelExtractionFailureForTesting; - var extractionStallTimeoutForTesting = - IndexExtractionStallTimeoutForTesting; - var parallelExtractionWorkersStoppedForTesting = - UpdateParallelExtractionWorkersStoppedForTesting; - - void RecordScanErrors( - IEnumerable scanErrors, - string fatalPhase = "discovery") - => context.RecordScanErrors(scanErrors, fatalPhase); - - void RecordCSharpWorkspaceDrift( - string relativePath, - string detail, - string fatalPhase = "reading") - => context.RecordCSharpWorkspaceDrift(relativePath, detail, fatalPhase); - - void DemoteReadinessOnce() => context.DemoteReadinessOnce(); - void WriteProjectRootOnce() => context.WriteProjectRootOnce(); - void RequireTypeScriptAugmentationRefresh() - => context.RequireTypeScriptAugmentationRefresh(); - - int PurgeStaleUpdateCleanupPaths( - string retainedRelativePath, - string? checksum, - bool includeDirectoryAndStem) - => context.PurgeStaleUpdateCleanupPaths( - retainedRelativePath, - checksum, - includeDirectoryAndStem); - - void RecordDynamicGraphFileRefresh(string? language) - => context.RecordDynamicGraphFileRefresh(language); - - void RecordUpdateFileFailure( - string relativePath, - string phase, - Exception exception) - => context.RecordUpdateFileFailure(relativePath, phase, exception); - - void ThrowIfUpdateCancelled() + internal UpdateFileLoopOutcome Run() { - if (!cancellationToken.IsCancellationRequested) - return; - - updateProgress.Pause(); - throw new IndexInterruptedException(updated + removed, targetPaths.Count); - } - - updateProgress.Start(); - - var updateTargets = new UpdateFileTarget[targetPaths.Count]; - var updateTargetIndex = 0; - foreach (var targetPath in targetPaths) - updateTargets[updateTargetIndex++] = UpdateFileTarget.Create(projectRoot, targetPath); - var readableFileBytes = new ReadableFileByteTracker( - updateTargets.Length, - targetIndex => updateTargets[targetIndex].FilePath, - projectRoot, - indexRunDiagnostics); + updateProgress.Start(); - WriteIndexJsonLiveness(options, $"updating {ConsoleUi.Counted(targetPaths.Count, "file")}..."); - string? currentUpdatePath = null; - var currentUpdatePhase = "preparing"; - var updateHeartbeat = StartIndexJsonPhaseHeartbeat( - options, - "updating index", - () => currentUpdatePath == null - ? $"{updated + removed + skipped:N0}/{targetPaths.Count:N0} files processed" - : $"{updated + removed + skipped:N0}/{targetPaths.Count:N0} files processed, current {currentUpdatePath}"); - var updateExtractionWorkStarted = 0; - void NotifyUpdateExtractionWorkStarted() - { - if (Interlocked.Exchange(ref updateExtractionWorkStarted, 1) == 0) - UpdateExtractionWorkStartedForTesting?.Invoke(); - } + var updateTargets = new UpdateFileTarget[targetPaths.Count]; + var updateTargetIndex = 0; + foreach (var targetPath in targetPaths) + updateTargets[updateTargetIndex++] = UpdateFileTarget.Create(projectRoot, targetPath); + readableFileBytes = new ReadableFileByteTracker( + updateTargets.Length, + targetIndex => updateTargets[targetIndex].FilePath, + projectRoot, + indexRunDiagnostics); + + WriteIndexJsonLiveness(options, $"updating {ConsoleUi.Counted(targetPaths.Count, "file")}..."); + var updateHeartbeat = StartIndexJsonPhaseHeartbeat( + options, + "updating index", + () => currentUpdatePath == null + ? $"{updated + removed + skipped:N0}/{targetPaths.Count:N0} files processed" + : $"{updated + removed + skipped:N0}/{targetPaths.Count:N0} files processed, current {currentUpdatePath}"); + var updateExtractionWorkStarted = 0; + void NotifyUpdateExtractionWorkStarted() + { + if (Interlocked.Exchange(ref updateExtractionWorkStarted, 1) == 0) + UpdateExtractionWorkStartedForTesting?.Invoke(); + } - using var symbolExtractionWorker = new LazyDisposable(() => - { - NotifyUpdateExtractionWorkStarted(); - return new SymbolExtractionWorkerClient(options.MaxFileSizeBytes); - }); - var parallelExtractionFallbackReason = options.Parallelism <= 1 - ? "parallelism_one" - : !csharpWorkspace.HasStaticInterfaceContracts - ? "non_authoritative_csharp_workspace" - : csharpWorkspaceSnapshots == null - ? "missing_csharp_workspace_snapshots" - : csharpWorkspaceSnapshots.Count < 2 - ? "insufficient_authoritative_csharp_targets" - : options.SymbolKindFilter.IsActive - ? "active_symbol_kind_filter" - : UpdateFileContentLoadForTesting != null - ? "content_load_test_hook" - : postExtractionHooks.Value.HasHooks - ? "post_extraction_hooks" - : null; - var parallelizeAuthoritativeCSharpUpdates = - parallelExtractionFallbackReason == null; - var parallelExtractionWorkerCount = parallelizeAuthoritativeCSharpUpdates - ? Math.Min(options.Parallelism, csharpWorkspaceSnapshots!.Count) - : 0; - var parallelExtractionWindowCapacity = parallelizeAuthoritativeCSharpUpdates - ? checked(parallelExtractionWorkerCount * 2) - : 0; - UpdateParallelExtractionSchedulingForTesting?.Invoke( - parallelizeAuthoritativeCSharpUpdates, - parallelExtractionFallbackReason, - parallelExtractionWorkerCount, - parallelExtractionWindowCapacity); - using var parallelExtractionPipeline = - new LazyDisposable(() => + using var symbolExtractionWorker = new LazyDisposable(() => { NotifyUpdateExtractionWorkStarted(); - return new UpdateParallelExtractionPipeline( - indexer, - options, - projectRoot, - csharpWorkspace, - csharpWorkspaceSnapshots!, - parallelExtractionWorkerCount, - parallelExtractionEventForTesting, - parallelExtractionFailureForTesting, - extractionStallTimeoutForTesting, - parallelExtractionWorkersStoppedForTesting); + return new SymbolExtractionWorkerClient(options.MaxFileSizeBytes); }); - - var parallelSourceWorkspaceDriftDetected = false; - void ConsumeParallelUpdateResult(UpdateParallelExtractionResult item) - { - var target = item.Target; - if (cancellationToken.IsCancellationRequested && item.Record != null) - { - DemoteReadinessOnce(); - csharpMetadataTargetsNeedRefresh = true; - } - ThrowIfUpdateCancelled(); - updateProgress.Start(); - var relPath = target.RelativePath; - currentUpdatePath = relPath; - currentUpdatePhase = item.FailurePhase ?? "preparing"; - var absPath = target.FilePath; - var dbPath = target.IndexPath; - var fileBatchMarked = false; - var csharpWorkspaceSnapshot = csharpWorkspaceSnapshots![dbPath]; - try - { - if (item.Record != null) - { - readableFileBytes.Remember(item.TargetIndex, item.Record.Size); - if (item.Warning != null && !options.Json && !options.Quiet) - { - updateProgress.Pause(); - ConsoleUi.PrintWarning(item.Warning); - updateProgress.Resume(); - } - DemoteReadinessOnce(); - csharpMetadataTargetsNeedRefresh = true; - } - var sourceContractSeenBeforeObservation = - postExtractionHooks.Value.SawCSharpStaticInterfaceSourceContract; - postExtractionHooks.Value.ObserveCSharpStaticInterfaceSourceContractEvidence( - item.HasCSharpStaticInterfaceSourceContract); - if (!csharpWorkspace.HasSourceStaticInterfaceContracts - && !sourceContractSeenBeforeObservation - && postExtractionHooks.Value.SawCSharpStaticInterfaceSourceContract) - { - parallelSourceWorkspaceDriftDetected = true; - RecordCSharpWorkspaceDrift( - relPath, - "A C# static-interface contract appeared after workspace preflight."); - skipped++; - return; - } - if (item.Exception is IndexExtractionStalledException stalledException) + var parallelExtractionFallbackReason = options.Parallelism <= 1 + ? "parallelism_one" + : !csharpWorkspace.HasStaticInterfaceContracts + ? "non_authoritative_csharp_workspace" + : csharpWorkspaceSnapshots == null + ? "missing_csharp_workspace_snapshots" + : csharpWorkspaceSnapshots.Count < 2 + ? "insufficient_authoritative_csharp_targets" + : options.SymbolKindFilter.IsActive + ? "active_symbol_kind_filter" + : UpdateFileContentLoadForTesting != null + ? "content_load_test_hook" + : postExtractionHooks.Value.HasHooks + ? "post_extraction_hooks" + : null; + var parallelizeAuthoritativeCSharpUpdates = + parallelExtractionFallbackReason == null; + var parallelExtractionWorkerCount = parallelizeAuthoritativeCSharpUpdates + ? Math.Min(options.Parallelism, csharpWorkspaceSnapshots!.Count) + : 0; + var parallelExtractionWindowCapacity = parallelizeAuthoritativeCSharpUpdates + ? checked(parallelExtractionWorkerCount * 2) + : 0; + UpdateParallelExtractionSchedulingForTesting?.Invoke( + parallelizeAuthoritativeCSharpUpdates, + parallelExtractionFallbackReason, + parallelExtractionWorkerCount, + parallelExtractionWindowCapacity); + using var parallelExtractionPipeline = + new LazyDisposable(() => { - if (!string.Equals( - item.FailurePhase, - "reading", - StringComparison.Ordinal)) - { - DemoteReadinessOnce(); - csharpMetadataTargetsNeedRefresh = true; - writer.MarkBatchInProgress(); - fileBatchMarked = true; - } - RethrowPreservingStackTrace( - new IndexExtractionStalledException( - updated + removed, - targetPaths.Count, - stalledException.Timeout, - stalledException.ActivePath, - stalledException.WorkerError)); - } - if (item.Exception is CSharpWorkspaceChangedException - or CSharpWorkspaceSnapshotDriftException) - { - RecordCSharpWorkspaceDrift( - relPath, - item.Exception.Message, - "reading"); - skipped++; - return; - } - if (item.Exception is FileIndexer.BinaryFileSkippedException - or FileIndexer.FileTooLargeSkippedException) - { - var skippedFile = HandleSkippedUpdateFile( - new SkippedUpdateFileHandlingContext - { - Writer = writer, - Indexer = indexer, - Options = options, - AbsolutePath = absPath, - RelativePath = relPath, - IndexPath = dbPath, - KnownLanguage = item.KnownLanguage, - ProjectRootWritten = context.IsProjectRootWritten(), - TargetIndex = item.TargetIndex, - ReadableFileBytes = readableFileBytes, - HasCSharpWorkspaceSnapshot = true, - CSharpWorkspaceSnapshot = csharpWorkspaceSnapshot, - CSharpWorkspaceSnapshots = csharpWorkspaceSnapshots, - WarningList = warningList, - UpdateProgress = updateProgress, - CancellationToken = cancellationToken, - DemoteReadinessOnce = DemoteReadinessOnce, - SetCurrentUpdatePhase = - phase => currentUpdatePhase = phase, - RecordCSharpWorkspaceDrift = - RecordCSharpWorkspaceDrift, - RecordUpdateFileFailure = - RecordUpdateFileFailure, - PurgeStaleUpdateCleanupPaths = - PurgeStaleUpdateCleanupPaths, - RequireTypeScriptAugmentationRefresh = - RequireTypeScriptAugmentationRefresh, - WriteProjectRootOnce = WriteProjectRootOnce, - RecordDynamicGraphFileRefresh = - RecordDynamicGraphFileRefresh, - }, - item.Exception); - updated += skippedFile.Updated; - skipped += skippedFile.Skipped; - warnings += skippedFile.Warnings; - mutualRecursionRefreshNeeded |= - skippedFile.MutualRecursionRefreshNeeded; - if (skippedFile.Updated > 0) - { - ftsMutated = true; - parallelExtractionEventForTesting?.Invoke( - new UpdateParallelExtractionTestEvent( - UpdateParallelExtractionEventKind.PersistenceCompleted, - item.TargetIndex, - target.DisplayRelativePath, - WorkerIndex: -1)); - } - return; - } - if (item.Exception is FileNotFoundException or DirectoryNotFoundException) - { - RecordCSharpWorkspaceDrift( - relPath, - "The C# file disappeared during its authoritative update pass."); - skipped++; - return; - } - if (item.Exception != null) - { - if (item.Exception is OperationCanceledException) - ThrowIfUpdateCancelled(); - if (!string.Equals( - item.FailurePhase, - "reading", - StringComparison.Ordinal)) - { - csharpMetadataTargetsNeedRefresh = true; - } - RecordUpdateFileFailure( - relPath, - item.FailurePhase ?? "reading", - item.Exception); - return; - } - - var record = item.Record!; - currentUpdatePhase = "validating"; - if (record.Lang != "csharp" - || !CSharpStaticInterfacePrepass.TryValidateLoadedFileStatSnapshot( - absPath, - dbPath, - target.DisplayRelativePath, - record.Size, - record.Modified, - csharpWorkspaceSnapshots, - out _, - cancellationToken)) - { - RecordCSharpWorkspaceDrift( - relPath, - "The C# file changed after extraction and before its authoritative update was persisted.", - "reading"); - skipped++; - return; - } + NotifyUpdateExtractionWorkStarted(); + return new UpdateParallelExtractionPipeline( + indexer, + options, + projectRoot, + csharpWorkspace, + csharpWorkspaceSnapshots!, + parallelExtractionWorkerCount, + parallelExtractionEventForTesting, + parallelExtractionFailureForTesting, + extractionStallTimeoutForTesting, + parallelExtractionWorkersStoppedForTesting); + }); - currentUpdatePhase = "reading"; - parallelExtractionEventForTesting?.Invoke( - new UpdateParallelExtractionTestEvent( - UpdateParallelExtractionEventKind.PersistenceStarted, - item.TargetIndex, - target.DisplayRelativePath, - WorkerIndex: -1)); - var persistence = PersistPrecomputedUpdateFile( - new UpdatePrecomputedFilePersistenceContext - { - Writer = writer, - Options = options, - Item = item, - ProjectRootWritten = context.IsProjectRootWritten(), - CancellationToken = cancellationToken, - RequireTypeScriptAugmentationRefresh = - RequireTypeScriptAugmentationRefresh, - PurgeStaleUpdateCleanupPaths = - PurgeStaleUpdateCleanupPaths, - WriteProjectRootOnce = WriteProjectRootOnce, - RecordDynamicGraphFileRefresh = - RecordDynamicGraphFileRefresh, - SetBatchMarkerOwned = owned => fileBatchMarked = owned, - SetPhase = (path, phase) => - { - currentUpdatePath = path; - currentUpdatePhase = phase; - }, - }); - symbolsDroppedByKindFilter += - persistence.SymbolsDroppedByKindFilter; - mutualRecursionRefreshNeeded |= - persistence.MutualRecursionRefreshNeeded; - updated++; - ftsMutated = true; - UpdateFileCommittedForTesting?.Invoke( - updated + removed, - targetPaths.Count); - parallelExtractionEventForTesting?.Invoke( - new UpdateParallelExtractionTestEvent( - UpdateParallelExtractionEventKind.PersistenceCompleted, - item.TargetIndex, - target.DisplayRelativePath, - WorkerIndex: -1)); - ThrowIfUpdateCancelled(); - updateProgress.WriteVerbose(persistence.VerboseMessage); - } - catch (IndexExtractionStalledException) - { - throw; - } - catch (Exception ex) + try { - if (fileBatchMarked) - writer.ClearBatchInProgress(); - if (ex is CSharpWorkspaceChangedException) + for (var targetIndex = 0; targetIndex < updateTargets.Length; targetIndex++) { - RecordCSharpWorkspaceDrift(relPath, ex.Message); - skipped++; - return; - } - if (ex is OperationCanceledException) ThrowIfUpdateCancelled(); - RecordUpdateFileFailure(relPath, currentUpdatePhase, ex); - } - } - try - { - for (var targetIndex = 0; targetIndex < updateTargets.Length; targetIndex++) - { - ThrowIfUpdateCancelled(); - if (parallelizeAuthoritativeCSharpUpdates - && csharpWorkspaceSnapshots!.ContainsKey( - updateTargets[targetIndex].IndexPath)) - { - IReadOnlyList parallelWindow; - try + if (parallelizeAuthoritativeCSharpUpdates + && csharpWorkspaceSnapshots!.ContainsKey( + updateTargets[targetIndex].IndexPath)) { - parallelWindow = TryBuildUpdateParallelWindow( - indexer, - updateTargets, - targetIndex, - parallelExtractionWindowCapacity, - csharpWorkspaceSnapshots, - scannedUpdateLanguages, - cancellationToken); - } - catch (OperationCanceledException) - when (cancellationToken.IsCancellationRequested) - { - ThrowIfUpdateCancelled(); - throw; - } - catch (Exception) - { - // Probing belongs to the per-file serial error boundary. If a - // speculative window probe fails, let the ordinary consumer - // retry the natural first target and contain any repeat there. - parallelizeAuthoritativeCSharpUpdates = false; - parallelWindow = []; - } - if (parallelWindow.Count > 0) - { - UpdateParallelExtractionWindowResult windowResult; + IReadOnlyList parallelWindow; try { - windowResult = parallelExtractionPipeline.Value.ExtractWindow( - parallelWindow, - cancellationToken, - (path, phase) => - { - currentUpdatePath = FormatIndexPhasePath(path, phase); - currentUpdatePhase = phase; - }); + parallelWindow = TryBuildUpdateParallelWindow( + indexer, + updateTargets, + targetIndex, + parallelExtractionWindowCapacity, + csharpWorkspaceSnapshots, + scannedUpdateLanguages, + cancellationToken); } catch (OperationCanceledException) + when (cancellationToken.IsCancellationRequested) { - var sawValidatedLoad = false; - foreach (var request in parallelWindow) - { - var loaded = request.Progress.GetLoadedRecord(); - if (loaded.Record == null) - continue; - - sawValidatedLoad = true; - break; - } - if (sawValidatedLoad) - { - DemoteReadinessOnce(); - csharpMetadataTargetsNeedRefresh = true; - } ThrowIfUpdateCancelled(); throw; } - var results = windowResult.Results; - parallelSourceWorkspaceDriftDetected = false; - var recoverSerialSuffix = false; - var consumedWindowCount = 0; - var consumableResultCount = windowResult.FatalWasNormalized - ? results.Count - 1 - : results.Count; - for (var resultIndex = 0; - resultIndex < consumableResultCount; - resultIndex++) + catch (Exception) { - var item = results[resultIndex]; - ConsumeParallelUpdateResult(item); - consumedWindowCount++; - if (item.Exception is IndexExtractionStalledException - || parallelSourceWorkspaceDriftDetected) - { - recoverSerialSuffix = true; - break; - } + // Probing belongs to the per-file serial error boundary. If a + // speculative window probe fails, let the ordinary consumer + // retry the natural first target and contain any repeat there. + parallelizeAuthoritativeCSharpUpdates = false; + parallelWindow = []; } - if (windowResult.FatalWasNormalized - && !recoverSerialSuffix) + if (parallelWindow.Count > 0) { - var sourceContractPrecedesFatal = - windowResult - .UnconsumedSourceContractCandidateBeforeFatal - && !csharpWorkspace - .HasSourceStaticInterfaceContracts - && !postExtractionHooks.Value - .SawCSharpStaticInterfaceSourceContract; - if (sourceContractPrecedesFatal) + UpdateParallelExtractionWindowResult windowResult; + try { - // The natural first unconsumed target must be - // retried before a later extraction fatal can - // become terminal. Keep the fatal's readiness - // and batch effects out of this recovery path. - recoverSerialSuffix = true; + windowResult = parallelExtractionPipeline.Value.ExtractWindow( + parallelWindow, + cancellationToken, + (path, phase) => + { + currentUpdatePath = FormatIndexPhasePath(path, phase); + currentUpdatePhase = phase; + }); } - else + catch (OperationCanceledException) { - var actualFatal = - windowResult.ActualFatalResult!; - if (actualFatal.Record != null - || !string.Equals( - actualFatal.FailurePhase, - "reading", - StringComparison.Ordinal)) + var sawValidatedLoad = false; + foreach (var request in parallelWindow) { - DemoteReadinessOnce(); - csharpMetadataTargetsNeedRefresh = true; + var loaded = request.Progress.GetLoadedRecord(); + if (loaded.Record == null) + continue; + + sawValidatedLoad = true; + break; } - if (!string.Equals( - actualFatal.FailurePhase, - "reading", - StringComparison.Ordinal)) + if (sawValidatedLoad) { - writer.MarkBatchInProgress(); + DemoteReadinessOnce(); + csharpMetadataTargetsNeedRefresh = true; } - - var normalizedFatal = results[^1]; - ConsumeParallelUpdateResult(normalizedFatal); - consumedWindowCount++; - if (parallelSourceWorkspaceDriftDetected) - recoverSerialSuffix = true; - } - } - if (recoverSerialSuffix) - { - parallelizeAuthoritativeCSharpUpdates = false; - targetIndex += consumedWindowCount - 1; - continue; - } - if (results.Count != parallelWindow.Count) - { - throw new InvalidOperationException( - "A shortened parallel update window returned without a terminal extraction error."); - } - targetIndex += parallelWindow.Count - 1; - continue; - } - } - - var target = updateTargets[targetIndex]; - ThrowIfUpdateCancelled(); - updateProgress.Start(); - var relPath = target.RelativePath; - currentUpdatePath = relPath; - currentUpdatePhase = "preparing"; - var absPath = target.FilePath; - var dbPath = target.IndexPath; - var fileBatchMarked = false; - string? knownLanguage = null; - CSharpStaticInterfacePrepass.FileStatSnapshot csharpWorkspaceSnapshot = default; - var hasCSharpWorkspaceSnapshot = csharpWorkspaceSnapshots != null - && csharpWorkspaceSnapshots.TryGetValue(dbPath, out csharpWorkspaceSnapshot); - try - { - if (hasCSharpWorkspaceSnapshot - && !CSharpStaticInterfacePrepass.TryValidateLoadedFileStatSnapshot( - absPath, - dbPath, - relPath, - csharpWorkspaceSnapshot.Size, - csharpWorkspaceSnapshot.ModifiedUtc, - csharpWorkspaceSnapshots!, - out _, - cancellationToken)) - { - RecordCSharpWorkspaceDrift( - relPath, - "The C# file changed before its authoritative update pass."); - skipped++; - continue; - } - - if (!File.Exists(LongPath.EnsureWindowsPrefix(absPath))) - { - if (hasCSharpWorkspaceSnapshot) - { - RecordCSharpWorkspaceDrift( - relPath, - "The C# file disappeared after contract preflight."); - skipped++; - continue; - } - - using var deleteTxn = writer.BeginTransaction(cancellationToken, "update delete missing target"); - if (writer.DeleteFileByPath(dbPath)) - { - DemoteReadinessOnce(); - WriteProjectRootOnce(); - RequireTypeScriptAugmentationRefresh(); - deleteTxn.Commit(); - removed++; - ftsMutated = true; - mutualRecursionRefreshNeeded = true; - updateProgress.WriteVerbose($" [DEL ] {relPath}"); - } - else - { - skipped++; - updateProgress.WriteVerbose($" [SKIP] {relPath} (not in DB)"); - } - continue; - } - - var pathFilter = indexer.EvaluatePathFilter(absPath); - RecordScanErrors(pathFilter.Errors); - if (pathFilter.ShouldSkip) - { - if (!pathFilter.ShouldDeleteExisting) - { - skipped++; - if (options.Verbose && !options.Json && !options.Quiet) - { - updateProgress.Pause(); - CommandOutputWriter.WriteLine($" [SKIP] {relPath} ({DescribePathFilter(pathFilter.FilterKind)})"); - updateProgress.Resume(); - } - continue; - } - - using var deleteTxn = writer.BeginTransaction(cancellationToken, "update delete skipped path"); - if (writer.DeleteFileByPath(dbPath)) - { - DemoteReadinessOnce(); - WriteProjectRootOnce(); - RequireTypeScriptAugmentationRefresh(); - deleteTxn.Commit(); - removed++; - ftsMutated = true; - mutualRecursionRefreshNeeded = true; - if (options.Verbose && !options.Json && !options.Quiet) - { - updateProgress.Pause(); - CommandOutputWriter.WriteLine($" [DEL ] {relPath} ({DescribePathFilter(pathFilter.FilterKind)})"); - updateProgress.Resume(); + ThrowIfUpdateCancelled(); + throw; } - } - else - { - skipped++; - if (options.Verbose && !options.Json) - { - updateProgress.Pause(); - CommandOutputWriter.WriteLine($" [SKIP] {relPath} ({DescribePathFilter(pathFilter.FilterKind)})"); - updateProgress.Resume(); - } - } - continue; - } - - var indexability = indexer.GetFileIndexabilityForIndexing(absPath); - var detection = indexer.TryDetectLanguageForIndexing(absPath, knownIndexability: indexability); - if (hasCSharpWorkspaceSnapshot - && (indexability != FileIndexer.FileProbeStatus.Supported - || detection.Status != FileIndexer.FileProbeStatus.Supported - || detection.Language != "csharp")) - { - RecordCSharpWorkspaceDrift( - relPath, - "The C# file changed language or indexability after contract preflight."); - skipped++; - continue; - } - if (!hasCSharpWorkspaceSnapshot - && csharpWorkspaceSnapshots != null - && indexability == FileIndexer.FileProbeStatus.Supported - && detection.Status == FileIndexer.FileProbeStatus.Supported - && detection.Language == "csharp") - { - RecordCSharpWorkspaceDrift( - relPath, - "A C# target appeared after the authoritative workspace target set was captured."); - skipped++; - continue; - } - if (indexability == FileIndexer.FileProbeStatus.Missing || detection.Status == FileIndexer.FileProbeStatus.Missing) - { - var message = $"{relPath}: skipped because it was deleted during indexing."; - warnings++; - warningList.Add(new CliJsonMessage(relPath, message)); - if (!options.Json && !options.Quiet) - { - updateProgress.Pause(); - ConsoleUi.PrintWarning(message); - updateProgress.Resume(); - } - - using var deleteTxn = writer.BeginTransaction(cancellationToken, "update delete missing during probe"); - if (writer.DeleteFileByPath(dbPath)) - { - DemoteReadinessOnce(); - WriteProjectRootOnce(); - RequireTypeScriptAugmentationRefresh(); - deleteTxn.Commit(); - removed++; - ftsMutated = true; - mutualRecursionRefreshNeeded = true; - } - else - { - skipped++; - } - continue; - } - - if (indexability == FileIndexer.FileProbeStatus.ProbeFailed || detection.Status == FileIndexer.FileProbeStatus.ProbeFailed) - { - DemoteReadinessOnce(); - - errors++; - errorList.Add(new CliJsonMessage(relPath, "Could not probe file for indexability/language.")); - if (fileErrorList.Count < PartialIndexFileErrorLimit) - { - fileErrorList.Add(new StatusIndexFileError + var results = windowResult.Results; + parallelSourceWorkspaceDriftDetected = false; + var recoverSerialSuffix = false; + var consumedWindowCount = 0; + var consumableResultCount = windowResult.FatalWasNormalized + ? results.Count - 1 + : results.Count; + for (var resultIndex = 0; + resultIndex < consumableResultCount; + resultIndex++) { - File = FileIndexer.NormalizePathSeparators(relPath), - Category = "file_read_error", - Phase = "reading", - Detail = "Could not probe file for indexability/language.", - }); - } - if (!options.Json) - { - updateProgress.Pause(); - if (options.Verbose) - CommandErrorWriter.WriteStderr($" [ERR ] {relPath}: Could not probe file for indexability/language."); - else - CommandErrorWriter.WriteStderr($" [ERR ] {relPath}: Could not probe file for indexability/language."); - updateProgress.Resume(); - } - continue; - } - - if (indexability != FileIndexer.FileProbeStatus.Supported || detection.Status != FileIndexer.FileProbeStatus.Supported) - { - if (!writer.HasFileAtPath(dbPath)) - { - using var purgeTxn = writer.BeginTransaction(cancellationToken, "update purge unsupported renamed target"); - var purged = PurgeStaleUpdateCleanupPaths( - dbPath, - checksum: null, - includeDirectoryAndStem: context.IsProjectRootWritten()); - if (purged > 0) - { - DemoteReadinessOnce(); - WriteProjectRootOnce(); - RequireTypeScriptAugmentationRefresh(); - purgeTxn.Commit(); - removed += purged; - ftsMutated = true; - mutualRecursionRefreshNeeded = true; - if (options.Verbose && !options.Json && !options.Quiet) + var item = results[resultIndex]; + ConsumeParallelUpdateResult(item); + consumedWindowCount++; + if (item.Exception is IndexExtractionStalledException + || parallelSourceWorkspaceDriftDetected) { - updateProgress.Pause(); - CommandOutputWriter.WriteLine($" [DEL ] {relPath} (unsupported renamed target)"); - updateProgress.Resume(); + recoverSerialSuffix = true; + break; } } - else + if (windowResult.FatalWasNormalized + && !recoverSerialSuffix) { - skipped++; - if (options.Verbose && !options.Json && !options.Quiet) + var sourceContractPrecedesFatal = + windowResult + .UnconsumedSourceContractCandidateBeforeFatal + && !csharpWorkspace + .HasSourceStaticInterfaceContracts + && !postExtractionHooks.Value + .SawCSharpStaticInterfaceSourceContract; + if (sourceContractPrecedesFatal) + { + // The natural first unconsumed target must be + // retried before a later extraction fatal can + // become terminal. Keep the fatal's readiness + // and batch effects out of this recovery path. + recoverSerialSuffix = true; + } + else { - updateProgress.Pause(); - CommandOutputWriter.WriteLine($" [SKIP] {relPath} (unsupported type)"); - updateProgress.Resume(); + var actualFatal = + windowResult.ActualFatalResult!; + if (actualFatal.Record != null + || !string.Equals( + actualFatal.FailurePhase, + "reading", + StringComparison.Ordinal)) + { + DemoteReadinessOnce(); + csharpMetadataTargetsNeedRefresh = true; + } + if (!string.Equals( + actualFatal.FailurePhase, + "reading", + StringComparison.Ordinal)) + { + writer.MarkBatchInProgress(); + } + + var normalizedFatal = results[^1]; + ConsumeParallelUpdateResult(normalizedFatal); + consumedWindowCount++; + if (parallelSourceWorkspaceDriftDetected) + recoverSerialSuffix = true; } } - continue; - } - - DemoteReadinessOnce(); - using var deleteTxn = writer.BeginTransaction(cancellationToken, "update delete unsupported target"); - if (writer.DeleteFileByPath(dbPath)) - { - WriteProjectRootOnce(); - RequireTypeScriptAugmentationRefresh(); - deleteTxn.Commit(); - removed++; - ftsMutated = true; - mutualRecursionRefreshNeeded = true; - if (options.Verbose && !options.Json && !options.Quiet) + if (recoverSerialSuffix) { - updateProgress.Pause(); - CommandOutputWriter.WriteLine($" [DEL ] {relPath} (no longer indexable)"); - updateProgress.Resume(); + parallelizeAuthoritativeCSharpUpdates = false; + targetIndex += consumedWindowCount - 1; + continue; } - } - else - { - skipped++; - if (options.Verbose && !options.Json) + if (results.Count != parallelWindow.Count) { - updateProgress.Pause(); - CommandOutputWriter.WriteLine($" [SKIP] {relPath} (unsupported type)"); - updateProgress.Resume(); + throw new InvalidOperationException( + "A shortened parallel update window returned without a terminal extraction error."); } - } - continue; - } - - if (FileIndexer.TryGetFileIdentity(absPath, out var identity, out var linkCount) - && linkCount > 1 - && !visitedFileIdentities.Add(identity)) - { - var message = "Skipped hardlinked file because the same file content was already indexed from another path."; - warnings++; - warningList.Add(new CliJsonMessage(relPath, message)); - if (!options.Json && !options.Quiet) - { - updateProgress.Pause(); - ConsoleUi.PrintWarning($"{relPath}: {message}"); - updateProgress.Resume(); - } - - using var deleteTxn = writer.BeginTransaction(); - if (writer.DeleteFileByPath(dbPath)) - { - DemoteReadinessOnce(); - WriteProjectRootOnce(); - RequireTypeScriptAugmentationRefresh(); - deleteTxn.Commit(); - removed++; - ftsMutated = true; - mutualRecursionRefreshNeeded = true; - } - else - { - skipped++; - } - continue; - } - - var statReusableLanguage = GetStatReusableLanguage(absPath, detection); - var generatedExtractionSuppressed = indexer.IsGeneratedCodeExtractionSuppressed(dbPath); - var statMatchedFile = IndexedFileStatReuse.TryGetReusableUnchangedFile( - writer, - absPath, - dbPath, - statReusableLanguage, - options.MaxFileSizeBytes ?? FileIndexer.DefaultMaxFileSizeBytes, - options.MaxSymbolsPerFile, - options.MaxReferencesPerFile, - generatedExtractionSuppressed, - allowReuse: symbolKindFilterMatchesPrior - && (statReusableLanguage != "csharp" || csharpSymbolNameContractMatchesCurrent) - && (statReusableLanguage != "csharp" || !csharpWorkspace.HasStaticInterfaceContracts) - && (statReusableLanguage != "sql" || sqlGraphContractMatchesCurrent) - && (statReusableLanguage is not ("verilog" or "systemverilog" or "vhdl") || hdlGraphContractMatchesCurrent)); - if (statMatchedFile != null) - { - skipped++; - readableFileBytes.Remember(targetIndex, statMatchedFile.Value.Size); - if (options.Verbose && !options.Json && !options.Quiet) - { - updateProgress.Pause(); - CommandOutputWriter.WriteLine($" [SKIP] {relPath} (unchanged)"); - updateProgress.Resume(); - } - continue; - } - - knownLanguage = scannedUpdateLanguages == null - ? statReusableLanguage - : FileIndexer.GetReusableDetectedLanguage(absPath, scannedUpdateLanguages); - - currentUpdatePhase = "reading"; - UpdateFileContentLoadForTesting?.Invoke(relPath); - var loaded = indexer.BuildLoadedRecordWithRawBytes( - absPath, - relPath, - knownLanguage, - cancellationToken); - var record = loaded.Record; - if (hasCSharpWorkspaceSnapshot - && (record.Lang != "csharp" - || !CSharpStaticInterfacePrepass.TryValidateLoadedFileStatSnapshot( - absPath, - dbPath, - relPath, - record.Size, - record.Modified, - csharpWorkspaceSnapshots!, - out _, - cancellationToken))) - { - RecordCSharpWorkspaceDrift( - relPath, - "The C# file changed while the authoritative update pass was reading it."); - skipped++; - continue; - } - readableFileBytes.Remember(targetIndex, record.Size); - var warning = loaded.Warning; - var generatedSuppressionIssue = generatedExtractionSuppressed - ? indexer.BuildGeneratedCodeExtractionSkippedIssue(record.Path) - : null; - - if (warning != null && !options.Json && !options.Quiet) - { - updateProgress.Pause(); - ConsoleUi.PrintWarning(warning); - updateProgress.Resume(); - } - - var existingId = writer.GetReusableUnchangedFileId( - record.Path, - record.Modified, - record.Checksum, - size: record.Size, - lines: record.Lines, - language: record.Lang, - generated: record.Generated, - maxSymbolsPerFile: options.MaxSymbolsPerFile, - maxReferencesPerFile: options.MaxReferencesPerFile, - generatedExtractionSuppressed: generatedExtractionSuppressed, - allowReuse: symbolKindFilterMatchesPrior - && (record.Lang != "csharp" || csharpSymbolNameContractMatchesCurrent) - && (record.Lang != "csharp" || !csharpWorkspace.HasStaticInterfaceContracts) - && (record.Lang != "sql" || sqlGraphContractMatchesCurrent) - && (record.Lang is not ("verilog" or "systemverilog" or "vhdl") || hdlGraphContractMatchesCurrent)); - if (existingId != null) - { - using var purgeTxn = writer.BeginTransaction(cancellationToken, "update purge unchanged stale paths"); - var purged = PurgeStaleUpdateCleanupPaths( - record.Path, - record.Checksum, - includeDirectoryAndStem: context.IsProjectRootWritten()); - if (purged > 0) - { - DemoteReadinessOnce(); - WriteProjectRootOnce(); - RequireTypeScriptAugmentationRefresh(); - purgeTxn.Commit(); - removed += purged; - ftsMutated = true; - mutualRecursionRefreshNeeded = true; - } - skipped++; - if (options.Verbose && !options.Json && !options.Quiet) - { - updateProgress.Pause(); - CommandOutputWriter.WriteLine(purged > 0 - ? $" [SKIP] {relPath} (unchanged; purged {purged:N0} stale renamed path(s))" - : $" [SKIP] {relPath} (unchanged)"); - updateProgress.Resume(); - } - continue; - } - - DemoteReadinessOnce(); - if (record.Lang == "csharp") - csharpMetadataTargetsNeedRefresh = true; - var persistence = PersistUpdateFile(new UpdateFilePersistenceContext - { - Writer = writer, - Indexer = indexer, - Options = options, - ProjectRoot = projectRoot, - RelativePath = relPath, - AbsolutePath = absPath, - Record = record, - Loaded = loaded, - GeneratedSuppressionIssue = generatedSuppressionIssue, - CSharpWorkspace = csharpWorkspace, - PostExtractionHooks = postExtractionHooks.Value, - SymbolExtractionWorker = symbolExtractionWorker.Value, - ProjectRootWritten = context.IsProjectRootWritten(), - CancellationToken = cancellationToken, - RequireTypeScriptAugmentationRefresh = RequireTypeScriptAugmentationRefresh, - PurgeStaleUpdateCleanupPaths = PurgeStaleUpdateCleanupPaths, - WriteProjectRootOnce = WriteProjectRootOnce, - RecordDynamicGraphFileRefresh = RecordDynamicGraphFileRefresh, - SetBatchMarkerOwned = owned => fileBatchMarked = owned, - SetPhase = (path, phase) => - { - currentUpdatePath = path; - currentUpdatePhase = phase; - }, - }); - symbolsDroppedByKindFilter += persistence.SymbolsDroppedByKindFilter; - mutualRecursionRefreshNeeded |= persistence.MutualRecursionRefreshNeeded; - updated++; - ftsMutated = true; - UpdateFileCommittedForTesting?.Invoke(updated + removed, targetPaths.Count); - ThrowIfUpdateCancelled(); - updateProgress.WriteVerbose(persistence.VerboseMessage); - } - catch (IndexExtractionStalledException) - { - throw; - } - catch (Exception ex) - { - if (ex is CSharpWorkspaceChangedException) - { - if (fileBatchMarked) - writer.ClearBatchInProgress(); - RecordCSharpWorkspaceDrift(relPath, ex.Message); - skipped++; - continue; - } - - if (ex is FileIndexer.BinaryFileSkippedException - or FileIndexer.FileTooLargeSkippedException) - { - if (fileBatchMarked) - writer.ClearBatchInProgress(); - - var skippedFile = HandleSkippedUpdateFile( - new SkippedUpdateFileHandlingContext - { - Writer = writer, - Indexer = indexer, - Options = options, - AbsolutePath = absPath, - RelativePath = relPath, - IndexPath = dbPath, - KnownLanguage = knownLanguage, - ProjectRootWritten = context.IsProjectRootWritten(), - TargetIndex = targetIndex, - ReadableFileBytes = readableFileBytes, - HasCSharpWorkspaceSnapshot = - hasCSharpWorkspaceSnapshot, - CSharpWorkspaceSnapshot = - csharpWorkspaceSnapshot, - CSharpWorkspaceSnapshots = - csharpWorkspaceSnapshots, - WarningList = warningList, - UpdateProgress = updateProgress, - CancellationToken = cancellationToken, - DemoteReadinessOnce = DemoteReadinessOnce, - SetCurrentUpdatePhase = - phase => currentUpdatePhase = phase, - RecordCSharpWorkspaceDrift = - RecordCSharpWorkspaceDrift, - RecordUpdateFileFailure = - RecordUpdateFileFailure, - PurgeStaleUpdateCleanupPaths = - PurgeStaleUpdateCleanupPaths, - RequireTypeScriptAugmentationRefresh = - RequireTypeScriptAugmentationRefresh, - WriteProjectRootOnce = WriteProjectRootOnce, - RecordDynamicGraphFileRefresh = - RecordDynamicGraphFileRefresh, - }, - ex); - updated += skippedFile.Updated; - skipped += skippedFile.Skipped; - warnings += skippedFile.Warnings; - mutualRecursionRefreshNeeded |= - skippedFile.MutualRecursionRefreshNeeded; - if (skippedFile.Updated > 0) - { - ftsMutated = true; - } - continue; - } - - if (ex is FileNotFoundException or DirectoryNotFoundException) - { - if (fileBatchMarked) - writer.ClearBatchInProgress(); - - if (hasCSharpWorkspaceSnapshot) - { - RecordCSharpWorkspaceDrift( - relPath, - "The C# file disappeared during its authoritative update pass."); - skipped++; + targetIndex += parallelWindow.Count - 1; continue; } - - var message = $"{relPath}: skipped because it was deleted during indexing."; - warnings++; - warningList.Add(new CliJsonMessage(relPath, message)); - if (!options.Json && !options.Quiet) - { - updateProgress.Pause(); - ConsoleUi.PrintWarning(message); - updateProgress.Resume(); - } - - if (writer.HasFileAtPath(dbPath)) - { - DemoteReadinessOnce(); - using var deleteTxn = writer.BeginTransaction(cancellationToken, "update delete missing during write"); - if (writer.DeleteFileByPath(dbPath)) - { - WriteProjectRootOnce(); - RequireTypeScriptAugmentationRefresh(); - deleteTxn.Commit(); - removed++; - ftsMutated = true; - mutualRecursionRefreshNeeded = true; - } - } - else - { - skipped++; - } - continue; } - if (fileBatchMarked) - writer.ClearBatchInProgress(); - RecordUpdateFileFailure(relPath, currentUpdatePhase, ex); + ConsumeSerialUpdateTarget( + updateTargets[targetIndex], + targetIndex, + symbolExtractionWorker); } } + finally + { + StopIndexJsonPhaseHeartbeat(updateHeartbeat); + } + if (options.MemoryTrace) + memorySamples.Add(CaptureMemorySample("extraction", stopwatch)); + return new UpdateFileLoopOutcome( + counters, + new UpdateFileLoopRefreshResult( + ftsMutated, + mutualRecursionRefreshNeeded, + csharpMetadataTargetsNeedRefresh), + readableFileBytes); } - finally - { - StopIndexJsonPhaseHeartbeat(updateHeartbeat); - } - if (options.MemoryTrace) - memorySamples.Add(CaptureMemorySample("extraction", stopwatch)); - return new UpdateFileLoopResult( - updated, - removed, - skipped, - warnings, - errors, - ftsMutated, - mutualRecursionRefreshNeeded, - csharpMetadataTargetsNeedRefresh, - symbolsDroppedByKindFilter, - readableFileBytes); } } diff --git a/src/CodeIndex/Cli/IndexCommandRunner.Update.FilePersistence.cs b/src/CodeIndex/Cli/IndexCommandRunner.Update.FilePersistence.cs index 143e3d91ac..4af45d87bd 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.Update.FilePersistence.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.Update.FilePersistence.cs @@ -8,29 +8,6 @@ namespace CodeIndex.Cli; public static partial class IndexCommandRunner { - private sealed class UpdateFilePersistenceContext - { - internal required DbWriter Writer { get; init; } - internal required FileIndexer Indexer { get; init; } - internal required IndexCommandOptions Options { get; init; } - internal required string ProjectRoot { get; init; } - internal required string RelativePath { get; init; } - internal required string AbsolutePath { get; init; } - internal required FileRecord Record { get; init; } - internal required LoadedFileRecord Loaded { get; init; } - internal FileIssue? GeneratedSuppressionIssue { get; init; } - internal required CSharpStaticInterfaceWorkspaceSymbols CSharpWorkspace { get; init; } - internal required PostExtractionHookRunner PostExtractionHooks { get; init; } - internal required SymbolExtractionWorkerClient SymbolExtractionWorker { get; init; } - internal required bool ProjectRootWritten { get; init; } - internal required CancellationToken CancellationToken { get; init; } - internal required Action RequireTypeScriptAugmentationRefresh { get; init; } - internal required Func PurgeStaleUpdateCleanupPaths { get; init; } - internal required Action WriteProjectRootOnce { get; init; } - internal required Action RecordDynamicGraphFileRefresh { get; init; } - internal required Action SetBatchMarkerOwned { get; init; } - internal required Action SetPhase { get; init; } - } private sealed record UpdateFilePersistenceResult( int SymbolsDroppedByKindFilter, @@ -62,252 +39,6 @@ private sealed class SkippedUpdateFilePersistenceContext private sealed record SkippedUpdateFilePersistenceResult( bool MutualRecursionRefreshNeeded); - private static UpdateFilePersistenceResult PersistUpdateFile( - UpdateFilePersistenceContext context) - { - var writer = context.Writer; - var options = context.Options; - var record = context.Record; - var loaded = context.Loaded; - var cancellationToken = context.CancellationToken; - var mutualRecursionRefreshNeeded = false; - var symbolsDroppedByKindFilter = 0; - - writer.MarkBatchInProgress(); - context.SetBatchMarkerOwned(true); - var recordRequiresTypeScriptAugmentationRefresh = record.Lang == "typescript"; - using var txn = writer.BeginTransaction(cancellationToken, "update file"); - if (recordRequiresTypeScriptAugmentationRefresh) - context.RequireTypeScriptAugmentationRefresh(); - var stalePurged = context.PurgeStaleUpdateCleanupPaths( - record.Path, - record.Checksum, - context.ProjectRootWritten); - if (stalePurged > 0) - { - context.RequireTypeScriptAugmentationRefresh(); - if (!options.SymbolsOnly) - mutualRecursionRefreshNeeded = true; - } - context.WriteProjectRootOnce(); - var fileId = writer.UpsertFile(record, out var referenceIdentityChanged); - if (!options.SymbolsOnly && referenceIdentityChanged) - mutualRecursionRefreshNeeded = true; - context.SetPhase(FormatIndexPhasePath(context.RelativePath, "chunking"), "chunking"); - var chunks = ChunkSplitter.SplitNormalized( - fileId, - loaded.Content, - loaded.Facts); - if (context.GeneratedSuppressionIssue != null) - { - writer.InsertChunks(chunks, cancellationToken); - writer.InsertSymbols([], cancellationToken); - writer.InsertReferencesInAtomicFileScope( - [], - refreshMutualRecursionFlags: false, - cancellationToken); - context.SetPhase( - FormatIndexPhasePath(context.RelativePath, "validating"), - "validating"); - var generatedIssues = AppendIssueIfMissing( - FileIndexer.ValidateContent( - record.Path, - loaded.RawBytes, - loaded.Content, - record.Lang, - loaded.Inspection, - loaded.Facts), - context.GeneratedSuppressionIssue); - writer.InsertIssues(fileId, generatedIssues); - context.SetPhase( - FormatIndexPhasePath(context.RelativePath, "committing"), - "committing"); - writer.ClearBatchInProgress(); - txn.Commit(); - context.SetBatchMarkerOwned(false); - context.RecordDynamicGraphFileRefresh(record.Lang); - return new UpdateFilePersistenceResult( - 0, - mutualRecursionRefreshNeeded, - $" [OK ] {context.RelativePath} ({chunks.Count} chunks, generated-code extraction skipped)"); - } - - context.SetPhase(FormatIndexPhasePath(context.RelativePath, "symbols"), "symbols"); - var symbolExtraction = ExtractSymbolsWithStallTimeout( - fileId, - record.Lang, - loaded.Content, - context.AbsolutePath, - context.ProjectRoot, - record.Path, - FormatIndexPhasePath(context.RelativePath, "symbols"), - true, - loaded.HasOversizeLine, - loaded.ConflictMarkerLine, - context.SymbolExtractionWorker, - cancellationToken); - var symbols = symbolExtraction.Symbols; - var symbolRegexTimeoutIssue = symbolExtraction.RegexTimeoutIssue; - var fileContext = new FileContext( - context.ProjectRoot, - record.Path, - context.AbsolutePath, - record.Lang); - var sourceContractSeenBeforeObservation = - context.PostExtractionHooks.SawCSharpStaticInterfaceSourceContract; - context.PostExtractionHooks.ObserveCSharpStaticInterfaceSourceSymbols( - fileContext, - symbols); - if (record.Lang == "csharp" - && !context.CSharpWorkspace.HasSourceStaticInterfaceContracts - && !sourceContractSeenBeforeObservation - && context.PostExtractionHooks.SawCSharpStaticInterfaceSourceContract) - { - writer.SetCSharpStaticInterfaceSourceEvidence(null); - throw new CSharpWorkspaceChangedException( - "A C# static-interface contract appeared after workspace preflight."); - } - if (symbols.Count > options.MaxSymbolsPerFile) - { - var issue = BuildSymbolCountExceededIssue( - record.Path, - symbols.Count, - options.MaxSymbolsPerFile); - IReadOnlyList capIssues = symbolRegexTimeoutIssue == null - ? [issue] - : AppendIssue([symbolRegexTimeoutIssue], issue); - writer.InsertSymbols([], cancellationToken); - writer.InsertReferencesInAtomicFileScope( - [], - refreshMutualRecursionFlags: false, - cancellationToken); - writer.InsertIssues(fileId, capIssues); - writer.ClearBatchInProgress(); - txn.Commit(); - context.SetBatchMarkerOwned(false); - context.RecordDynamicGraphFileRefresh(record.Lang); - return new UpdateFilePersistenceResult( - 0, - mutualRecursionRefreshNeeded, - $" [SKIP] {context.RelativePath} ({issue.Message})"); - } - - var familyScopeKey = context.Indexer.GetFamilyScopeKey(context.AbsolutePath, record.Lang); - SymbolExtractor.ApplyFamilyScope(symbols, familyScopeKey, record.Lang); - context.PostExtractionHooks.OnSymbolsExtractedAfterSourceObservation( - fileContext, - symbols, - loaded.Content, - familyScopeKey); - symbolsDroppedByKindFilter = options.SymbolKindFilter.Apply(symbols); - if (symbols.Count > options.MaxSymbolsPerFile) - { - var issue = BuildSymbolCountExceededIssue( - record.Path, - symbols.Count, - options.MaxSymbolsPerFile); - IReadOnlyList capIssues = symbolRegexTimeoutIssue == null - ? [issue] - : AppendIssue([symbolRegexTimeoutIssue], issue); - writer.InsertSymbols([], cancellationToken); - writer.InsertReferencesInAtomicFileScope( - [], - refreshMutualRecursionFlags: false, - cancellationToken); - writer.InsertIssues(fileId, capIssues); - writer.ClearBatchInProgress(); - txn.Commit(); - context.SetBatchMarkerOwned(false); - context.RecordDynamicGraphFileRefresh(record.Lang); - return new UpdateFilePersistenceResult( - symbolsDroppedByKindFilter, - mutualRecursionRefreshNeeded, - $" [SKIP] {context.RelativePath} ({issue.Message})"); - } - - writer.InsertChunks(chunks, cancellationToken); - FileIndexer.ValidateSymbolLineRanges(record, symbols); - writer.InsertSymbols(symbols, cancellationToken); - context.SetPhase( - FormatIndexPhasePath(context.RelativePath, "references"), - "references"); - List references; - FileIssue? referenceRegexTimeoutIssue; - ReferenceExtractionResult referenceExtraction; - using (var regexTimeouts = BoundedRegex.CaptureTimeouts( - record.Lang, - "reference_extraction")) - { - referenceExtraction = ReferenceExtractor.ExtractDetailedNormalized( - fileId, - record.Lang, - loaded.Content, - loaded.HasOversizeLine, - symbols, - record.Path, - record.Lang == "csharp" ? context.CSharpWorkspace.Symbols : null, - cancellationToken, - maxReferenceCount: options.MaxReferencesPerFile + 1, - conflictMarkerLine: loaded.ConflictMarkerLine, - workspaceRoot: context.ProjectRoot, - csharpStaticInterfaceMemberLookups: - context.CSharpWorkspace.StaticInterfaceMemberLookups, - csharpQualifiedPatternLookups: - context.CSharpWorkspace.QualifiedPatternLookups); - references = referenceExtraction.References; - referenceRegexTimeoutIssue = - BuildRegexTimeoutIssue(record.Path, regexTimeouts); - } - context.PostExtractionHooks.OnReferencesExtracted(fileContext, references); - FileIssue? referenceCapIssue = null; - if (references.Count > options.MaxReferencesPerFile) - { - referenceCapIssue = BuildReferenceCountExceededIssue( - record.Path, - references.Count, - options.MaxReferencesPerFile); - references = []; - } - writer.InsertReferencesInAtomicFileScope( - references, - refreshMutualRecursionFlags: false, - cancellationToken); - context.SetPhase( - FormatIndexPhasePath(context.RelativePath, "validating"), - "validating"); - IReadOnlyList issues = FileIndexer.ValidateContent( - record.Path, - loaded.RawBytes, - loaded.Content, - record.Lang, - loaded.Inspection, - loaded.Facts); - if (symbolRegexTimeoutIssue != null) - issues = AppendIssue(issues, symbolRegexTimeoutIssue); - if (referenceRegexTimeoutIssue != null) - issues = AppendIssue(issues, referenceRegexTimeoutIssue); - issues = AppendReferenceExtractionDiagnosticIssues( - issues, - record.Path, - referenceExtraction.Diagnostics); - if (referenceCapIssue != null) - issues = AppendIssue(issues, referenceCapIssue); - writer.InsertIssues(fileId, issues); - context.SetPhase( - FormatIndexPhasePath(context.RelativePath, "committing"), - "committing"); - writer.ClearBatchInProgress(); - txn.Commit(); - context.SetBatchMarkerOwned(false); - context.RecordDynamicGraphFileRefresh(record.Lang); - if (!options.SymbolsOnly && (symbols.Count > 0 || references.Count > 0)) - mutualRecursionRefreshNeeded = true; - - return new UpdateFilePersistenceResult( - symbolsDroppedByKindFilter, - mutualRecursionRefreshNeeded, - $" [OK ] {context.RelativePath} ({chunks.Count} chunks, {symbols.Count} symbols, {references.Count} refs)"); - } private static SkippedUpdateFilePersistenceResult PersistSkippedUpdateFile( SkippedUpdateFilePersistenceContext context) diff --git a/src/CodeIndex/Cli/IndexCommandRunner.Update.ParallelExtraction.cs b/src/CodeIndex/Cli/IndexCommandRunner.Update.ParallelExtraction.cs index 9b18090e46..4c6780617b 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.Update.ParallelExtraction.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.Update.ParallelExtraction.cs @@ -736,21 +736,6 @@ private void DisposeResources() } } - private sealed class UpdatePrecomputedFilePersistenceContext - { - internal required DbWriter Writer { get; init; } - internal required IndexCommandOptions Options { get; init; } - internal required UpdateParallelExtractionResult Item { get; init; } - internal required bool ProjectRootWritten { get; init; } - internal required CancellationToken CancellationToken { get; init; } - internal required Action RequireTypeScriptAugmentationRefresh { get; init; } - internal required Func PurgeStaleUpdateCleanupPaths { get; init; } - internal required Action WriteProjectRootOnce { get; init; } - internal required Action RecordDynamicGraphFileRefresh { get; init; } - internal required Action SetBatchMarkerOwned { get; init; } - internal required Action SetPhase { get; init; } - } - private static IReadOnlyList TryBuildUpdateParallelWindow( FileIndexer indexer, UpdateFileTarget[] targets, @@ -1101,116 +1086,4 @@ void ThrowInjectedFailure() } } - private static UpdateFilePersistenceResult PersistPrecomputedUpdateFile( - UpdatePrecomputedFilePersistenceContext context) - { - var writer = context.Writer; - var options = context.Options; - var item = context.Item; - var record = item.Record!; - var cancellationToken = context.CancellationToken; - var mutualRecursionRefreshNeeded = false; - - writer.MarkBatchInProgress(); - context.SetBatchMarkerOwned(true); - using var txn = writer.BeginTransaction( - cancellationToken, - "update precomputed file"); - var stalePurged = context.PurgeStaleUpdateCleanupPaths( - record.Path, - record.Checksum, - context.ProjectRootWritten); - if (stalePurged > 0 && !options.SymbolsOnly) - mutualRecursionRefreshNeeded = true; - if (stalePurged > 0) - context.RequireTypeScriptAugmentationRefresh(); - context.WriteProjectRootOnce(); - var fileId = writer.UpsertFile(record, out var referenceIdentityChanged); - if (!options.SymbolsOnly && referenceIdentityChanged) - mutualRecursionRefreshNeeded = true; - - context.SetPhase( - FormatIndexPhasePath(item.Target.DisplayRelativePath, "chunking"), - "chunking"); - var chunks = ReassignChunkFileIds(item.Chunks!, fileId); - if (item.GeneratedSuppressionIssue != null) - { - writer.InsertChunks(chunks, cancellationToken); - writer.InsertSymbols([], cancellationToken); - writer.InsertReferencesInAtomicFileScope( - [], - refreshMutualRecursionFlags: false, - cancellationToken); - context.SetPhase( - FormatIndexPhasePath(item.Target.DisplayRelativePath, "validating"), - "validating"); - writer.InsertIssues(fileId, item.Issues!); - context.SetPhase( - FormatIndexPhasePath(item.Target.DisplayRelativePath, "committing"), - "committing"); - writer.ClearBatchInProgress(); - txn.Commit(); - context.SetBatchMarkerOwned(false); - context.RecordDynamicGraphFileRefresh(record.Lang); - return new UpdateFilePersistenceResult( - 0, - mutualRecursionRefreshNeeded, - $" [OK ] {item.Target.RelativePath} ({chunks.Count} chunks, generated-code extraction skipped)"); - } - - context.SetPhase( - FormatIndexPhasePath(item.Target.DisplayRelativePath, "symbols"), - "symbols"); - var symbols = ReassignSymbolFileIds(item.Symbols!, fileId); - if (item.SymbolCapExceeded) - { - writer.InsertSymbols([], cancellationToken); - writer.InsertReferencesInAtomicFileScope( - [], - refreshMutualRecursionFlags: false, - cancellationToken); - writer.InsertIssues(fileId, item.Issues!); - writer.ClearBatchInProgress(); - txn.Commit(); - context.SetBatchMarkerOwned(false); - context.RecordDynamicGraphFileRefresh(record.Lang); - return new UpdateFilePersistenceResult( - 0, - mutualRecursionRefreshNeeded, - $" [SKIP] {item.Target.RelativePath} ({item.Issues![^1].Message})"); - } - - writer.InsertChunks(chunks, cancellationToken); - FileIndexer.ValidateSymbolLineRanges(record, symbols); - writer.InsertSymbols(symbols, cancellationToken); - context.SetPhase( - FormatIndexPhasePath(item.Target.DisplayRelativePath, "references"), - "references"); - var references = ReassignReferenceFileIds(item.References!, fileId); - writer.InsertReferencesInAtomicFileScope( - references, - refreshMutualRecursionFlags: false, - cancellationToken); - context.SetPhase( - FormatIndexPhasePath(item.Target.DisplayRelativePath, "validating"), - "validating"); - writer.InsertIssues(fileId, item.Issues!); - context.SetPhase( - FormatIndexPhasePath(item.Target.DisplayRelativePath, "committing"), - "committing"); - writer.ClearBatchInProgress(); - txn.Commit(); - context.SetBatchMarkerOwned(false); - context.RecordDynamicGraphFileRefresh(record.Lang); - if (!options.SymbolsOnly - && (symbols.Count > 0 || references.Count > 0)) - { - mutualRecursionRefreshNeeded = true; - } - - return new UpdateFilePersistenceResult( - 0, - mutualRecursionRefreshNeeded, - $" [OK ] {item.Target.RelativePath} ({chunks.Count} chunks, {symbols.Count} symbols, {references.Count} refs)"); - } } diff --git a/src/CodeIndex/Cli/IndexCommandRunner.Update.cs b/src/CodeIndex/Cli/IndexCommandRunner.Update.cs index b1ede7a389..513f4a5171 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.Update.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.Update.cs @@ -342,28 +342,6 @@ void RecordScanErrors( } } - void RecordUpdateFileFailure( - string relativePath, - string phase, - Exception exception) - { - DemoteReadinessOnce(); - LogIndexFileFailure("index_update_file_failed", relativePath, phase, exception); - - errors++; - var errorMessage = FormatIndexFileException(exception); - errorList.Add(new CliJsonMessage(relativePath, errorMessage)); - if (fileErrorList.Count < PartialIndexFileErrorLimit) - fileErrorList.Add(BuildIndexFileError(relativePath, phase, exception)); - if (!options.Json) - { - updateProgress.Pause(); - CommandErrorWriter.WriteStderr( - FormatPerFileErrorLine("ERR ", relativePath, exception, errorMessage)); - updateProgress.Resume(); - } - } - void ThrowIfUpdateCancelled() { if (!cancellationToken.IsCancellationRequested) @@ -686,56 +664,72 @@ bool TryValidateCSharpWorkspaceInputSnapshot( useUpdateSecondaryIndexStaging, cancellationToken); - var updateLoop = RunUpdateFileLoop(new UpdateFileLoopContext - { - Writer = writer, - Indexer = indexer, - Options = options, - Stopwatch = stopwatch, - ProjectRoot = projectRoot, - IndexRunDiagnostics = indexRunDiagnostics, - TargetPaths = targetPaths, - UpdateProgress = updateProgress, - MemorySamples = memorySamples, - Updated = updated, - Removed = removed, - Skipped = skipped, - FtsMutated = ftsMutated, - MutualRecursionRefreshNeeded = mutualRecursionRefreshNeeded, - CSharpMetadataTargetsNeedRefresh = csharpMetadataTargetsNeedRefresh, - SymbolsDroppedByKindFilter = symbolsDroppedByKindFilter, - CSharpWorkspace = csharpWorkspace, - CSharpWorkspaceSnapshots = csharpWorkspaceSnapshots, - ScannedUpdateLanguages = scannedUpdateLanguages, - SymbolKindFilterMatchesPrior = symbolKindFilterMatchesPrior, - CSharpSymbolNameContractMatchesCurrent = csharpSymbolNameContractMatchesCurrent, - SqlGraphContractMatchesCurrent = sqlGraphContractMatchesCurrent, - HdlGraphContractMatchesCurrent = hdlGraphContractMatchesCurrent, - PostExtractionHooks = postExtractionHooks, - VisitedFileIdentities = visitedFileIdentities, - ErrorList = errorList, - FileErrorList = fileErrorList, - WarningList = warningList, - CancellationToken = cancellationToken, - RecordScanErrors = RecordScanErrors, - RecordCSharpWorkspaceDrift = RecordCSharpWorkspaceDrift, - DemoteReadinessOnce = DemoteReadinessOnce, - WriteProjectRootOnce = WriteProjectRootOnce, - RequireTypeScriptAugmentationRefresh = RequireTypeScriptAugmentationRefresh, - PurgeStaleUpdateCleanupPaths = PurgeStaleUpdateCleanupPaths, - RecordDynamicGraphFileRefresh = RecordDynamicGraphFileRefresh, - RecordUpdateFileFailure = RecordUpdateFileFailure, - IsProjectRootWritten = () => projectRootWritten, - }); - updated = updateLoop.Updated; - removed = updateLoop.Removed; - skipped = updateLoop.Skipped; - warnings += updateLoop.Warnings; - errors += updateLoop.Errors; - ftsMutated = updateLoop.FtsMutated; - mutualRecursionRefreshNeeded = updateLoop.MutualRecursionRefreshNeeded; - csharpMetadataTargetsNeedRefresh = updateLoop.CSharpMetadataTargetsNeedRefresh; - symbolsDroppedByKindFilter = updateLoop.SymbolsDroppedByKindFilter; + var updateLoop = new UpdateFileLoopSession( + new UpdateFileLoopRuntime( + writer, + indexer, + options, + projectRoot, + updateProgress, + cancellationToken, + targetPaths, + memorySamples), + new UpdateFileLoopCounters( + updated, + removed, + skipped, + Warnings: 0, + Errors: 0, + symbolsDroppedByKindFilter), + new UpdateFileLoopRefreshState( + ftsMutated, + mutualRecursionRefreshNeeded, + csharpMetadataTargetsNeedRefresh, + CurrentPath: null, + CurrentPhase: "preparing", + ParallelSourceWorkspaceDriftDetected: false, + ReadableFileBytes: null), + new UpdateFileLoopWorkspace( + csharpWorkspace, + csharpWorkspaceSnapshots, + scannedUpdateLanguages, + symbolKindFilterMatchesPrior, + csharpSymbolNameContractMatchesCurrent, + sqlGraphContractMatchesCurrent, + hdlGraphContractMatchesCurrent), + new UpdateFileLoopOutput( + stopwatch, + indexRunDiagnostics, + postExtractionHooks, + visitedFileIdentities, + errorList, + fileErrorList, + warningList), + new UpdateFileLoopReadinessOperations( + RecordScanErrors, + RecordCSharpWorkspaceDrift, + DemoteReadinessOnce, + RequireTypeScriptAugmentationRefresh, + RecordDynamicGraphFileRefresh), + new UpdateFileLoopPersistenceOperations( + WriteProjectRootOnce, + PurgeStaleUpdateCleanupPaths, + () => projectRootWritten), + new UpdateFileLoopParallelTesting( + UpdateParallelExtractionEventForTesting, + UpdateParallelExtractionFailureForTesting, + IndexExtractionStallTimeoutForTesting, + UpdateParallelExtractionWorkersStoppedForTesting)) + .Run(); + updated = updateLoop.Counters.Updated; + removed = updateLoop.Counters.Removed; + skipped = updateLoop.Counters.Skipped; + warnings += updateLoop.Counters.Warnings; + errors += updateLoop.Counters.Errors; + ftsMutated = updateLoop.Refresh.FtsMutated; + mutualRecursionRefreshNeeded = updateLoop.Refresh.MutualRecursionRefreshNeeded; + csharpMetadataTargetsNeedRefresh = updateLoop.Refresh.CSharpMetadataTargetsNeedRefresh; + symbolsDroppedByKindFilter = updateLoop.Counters.SymbolsDroppedByKindFilter; var readableFileBytes = updateLoop.ReadableFileBytes; if (options.ChangedBetweenSpecified diff --git a/src/CodeIndex/Cli/IndexCommandRunner.cs b/src/CodeIndex/Cli/IndexCommandRunner.cs index aa3f1d2d55..6dd20e76f1 100644 --- a/src/CodeIndex/Cli/IndexCommandRunner.cs +++ b/src/CodeIndex/Cli/IndexCommandRunner.cs @@ -82,6 +82,7 @@ public void Dispose() internal static Action? FullScanWritePhaseStartedForTesting { get; set; } internal static Action? FullScanExtractionSchedulingForTesting { get; set; } internal static Action? FullScanExtractionWorkStartedForTesting { get; set; } + internal static Action? FullScanExtractionWorkersStoppedForTesting { get; set; } internal static Action? FullScanFileContentLoadForTesting { get; set; } internal static Action? FullScanFilePhaseForTesting { get; set; } internal static Action? FullScanExtractionQueueCapacityForTesting { get; set; } diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Batch.cs b/src/CodeIndex/Cli/QueryCommandRunner.Batch.cs index 642fe21039..69c10d20b9 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Batch.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Batch.cs @@ -1,4 +1,3 @@ -using System.Globalization; using System.Runtime.CompilerServices; using System.Text; using System.Text.Json; @@ -12,7 +11,7 @@ namespace CodeIndex.Cli; public static partial class QueryCommandRunner { private const int BatchMaxCapturedOutputChars = JsonEnvelopeWrapper.MaxCapturedOutputChars; - private static readonly ConditionalWeakTable s_batchInputPumps = new(); + private static readonly ConditionalWeakTable s_batchInputPumps = []; internal static Action? BatchParallelCommandStartedForTesting { get; set; } internal static Action? BatchParallelCommandCompletedForTesting { get; set; } internal static Action? BatchInputLineReadForTesting { get; set; } @@ -21,943 +20,6 @@ public static partial class QueryCommandRunner internal static Func? BatchParallelReaderFactoryForTesting { get; set; } internal static Action? BatchParallelDatabaseValidatingForTesting { get; set; } - public static int RunBatch( - string[] cmdArgs, - JsonSerializerOptions jsonOptions, - string appVersion = "", - CancellationToken cancellationToken = default) - { - var dbPath = Path.Combine(".cdidx", "codeindex.db"); - var dbPathExplicit = false; - var jsonSummary = false; - var maxInputLines = BatchDefaultInputLines; - var maxOutputChars = BatchDefaultTotalOutputChars; - var maxOutputCharsSpecified = false; - var parallelism = 1; - var parallelismSpecified = false; - var includeRawStreams = false; - for (var i = 0; i < cmdArgs.Length; i++) - { - var arg = cmdArgs[i]; - if (arg == "--json-summary") - { - jsonSummary = true; - continue; - } - - if (arg == "--include-raw-streams") - { - includeRawStreams = true; - continue; - } - - if (arg == "--db") - { - if (i + 1 >= cmdArgs.Length || string.IsNullOrWhiteSpace(cmdArgs[i + 1])) - { - CommandErrorWriter.WriteStderr(BuildMissingOptionValueError("--db")); - return CommandExitCodes.UsageError; - } - dbPath = cmdArgs[++i]; - dbPathExplicit = true; - continue; - } - - if (arg.StartsWith("--db=", StringComparison.Ordinal)) - { - dbPath = arg["--db=".Length..]; - if (string.IsNullOrWhiteSpace(dbPath)) - { - CommandErrorWriter.WriteStderr(BuildMissingOptionValueError("--db")); - return CommandExitCodes.UsageError; - } - dbPathExplicit = true; - continue; - } - - if (arg == "--max-input-lines" || arg.StartsWith("--max-input-lines=", StringComparison.Ordinal)) - { - if (!TryReadBatchBoundedOption( - cmdArgs, - ref i, - arg, - "--max-input-lines", - 1, - BatchMaxInputLines, - out maxInputLines)) - { - return CommandExitCodes.UsageError; - } - continue; - } - - if (arg == "--max-output-chars" || arg.StartsWith("--max-output-chars=", StringComparison.Ordinal)) - { - if (!TryReadBatchBoundedOption( - cmdArgs, - ref i, - arg, - "--max-output-chars", - BatchMinTotalOutputChars, - BatchMaxTotalOutputChars, - out maxOutputChars)) - { - return CommandExitCodes.UsageError; - } - maxOutputCharsSpecified = true; - continue; - } - - if (arg == "--parallel" || arg.StartsWith("--parallel=", StringComparison.Ordinal)) - { - if (!TryReadBatchBoundedOption( - cmdArgs, - ref i, - arg, - "--parallel", - 1, - BatchMaxParallelism, - out parallelism)) - { - return CommandExitCodes.UsageError; - } - parallelismSpecified = true; - continue; - } - - CommandErrorWriter.WriteStderr($"Error: {ConsoleUi.FormatBoundedValue(arg)} is not supported for batch."); - CommandErrorWriter.WriteStderr($"Usage: {ConsoleUi.GetUsageLine("batch")}"); - return CommandExitCodes.UsageError; - } - - if (parallelismSpecified && !jsonSummary) - { - CommandErrorWriter.WriteStderr("Error: --parallel requires --json-summary so concurrent child output can be isolated and emitted in input order."); - CommandErrorWriter.WriteStderr($"Usage: {ConsoleUi.GetUsageLine("batch")}"); - return CommandExitCodes.UsageError; - } - if (maxOutputCharsSpecified && !jsonSummary) - { - CommandErrorWriter.WriteStderr("Error: --max-output-chars requires --json-summary because ordinary batch output streams directly."); - CommandErrorWriter.WriteStderr($"Usage: {ConsoleUi.GetUsageLine("batch")}"); - return CommandExitCodes.UsageError; - } - if (includeRawStreams && !jsonSummary) - { - CommandErrorWriter.WriteStderr("Error: --include-raw-streams requires --json-summary."); - CommandErrorWriter.WriteStderr($"Usage: {ConsoleUi.GetUsageLine("batch")}"); - return CommandExitCodes.UsageError; - } - - var isUri = dbPath.StartsWith("file:", StringComparison.OrdinalIgnoreCase); - if (!isUri && !File.Exists(dbPath)) - { - CommandErrorWriter.WriteStderr($"Error [{CommandErrorCodes.DbNotFound}]: database not found at {FormatDbDiagnosticValue(Path.GetFullPath(dbPath))}"); - CommandErrorWriter.WriteStderr("Hint: create or refresh the index with `cdidx index ` (or `cdidx .`) and then rerun this command."); - return CommandExitCodes.DatabaseError; - } - - if (parallelism > 1) - { - BatchParallelSession? firstSession = null; - DbContext? validationDb = null; - try - { - validationDb = new DbContext(DbOpenIntent.QueryOnly, dbPath, cancellationToken); - BatchParallelDatabaseValidatingForTesting?.Invoke(); - if (!validationDb.TryValidateIsCodeIndexDb(out var validationReason)) - return WriteInvalidCodeIndexDbError(dbPath, validationReason, json: false, jsonOptions); - var transferredDb = validationDb; - validationDb = null; - firstSession = BatchParallelSession.FromValidated( - dbPath, - transferredDb, - cancellationToken); - } - catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) - { - firstSession?.Dispose(); - return WriteBatchSetupCancellationSummary( - maxInputLines, - maxOutputChars, - parallelism, - jsonOptions); - } - catch - { - firstSession?.Dispose(); - throw; - } - finally - { - validationDb?.Dispose(); - } - - try - { - return RunBatchParallel( - dbPath, - dbPathExplicit, - maxInputLines, - maxOutputChars, - parallelism, - includeRawStreams, - jsonOptions, - appVersion, - firstSession!, - cancellationToken); - } - catch - { - firstSession?.Dispose(); - throw; - } - } - - try - { - using var db = new DbContext(DbOpenIntent.QueryOnly, dbPath, cancellationToken); - if (!db.TryValidateIsCodeIndexDb(out var validationReason)) - return WriteInvalidCodeIndexDbError(dbPath, validationReason, json: false, jsonOptions); - - s_batchReader = new DbReader(db); - s_batchDbPath = dbPath; - s_batchDbPathExplicit = dbPathExplicit; - var jsonOutput = jsonSummary - ? new BatchJsonOutputWriter( - Console.Out, - maxOutputChars, - BatchTerminalOutputReserveChars, - jsonOptions) - : null; - var firstFailure = CommandExitCodes.Success; - var lineNumber = 0; - var commandsProcessed = 0; - var lineErrors = 0; - var commandFailures = 0; - var outputLimitReached = false; - var inputLimitReached = false; - var batchInput = GetBatchInputPump(Console.In); - while (true) - { - BatchPumpedLine? pumpedLine; - try - { - pumpedLine = batchInput.ReadAsync(cancellationToken) - .AsTask() - .GetAwaiter() - .GetResult(); - } - catch (OperationCanceledException) when ( - jsonSummary - && cancellationToken.IsCancellationRequested) - { - firstFailure = CommandExitCodes.CancelledBySignal; - break; - } - if (pumpedLine is null) - break; - - var line = pumpedLine.Value.Line; - var lineExceededLimit = pumpedLine.Value.ExceededLimit; - lineNumber++; - BatchInputLineReadForTesting?.Invoke(lineNumber); - if (cancellationToken.IsCancellationRequested) - { - if (!jsonSummary) - cancellationToken.ThrowIfCancellationRequested(); - - firstFailure = CommandExitCodes.CancelledBySignal; - if (lineExceededLimit || !string.IsNullOrWhiteSpace(line)) - { - var lineError = BuildBatchCancellationLineError(lineNumber); - if (!WriteBatchLineErrorJson(lineNumber, lineError, jsonOutput!)) - { - WriteBatchOutputLimitErrorJson( - lineNumber, - commandName: null, - CommandExitCodes.CancelledBySignal, - maxOutputChars, - jsonOutput!); - outputLimitReached = true; - } - lineErrors++; - } - break; - } - - if (lineNumber > maxInputLines) - { - var lineError = new BatchLineError( - $"batch input exceeds the {maxInputLines} line limit.", - CommandExitCodes.UsageError, - Hint: "Split the request into smaller batch invocations.", - ErrorCode: CommandErrorCodes.UsageError, - Category: "batch_input_line_limit"); - if (jsonSummary) - jsonOutput!.WriteTerminal(BuildBatchLineErrorJson(lineNumber, lineError)); - else - WriteBatchLineErrorDiagnostic(lineError, jsonOptions); - lineErrors++; - inputLimitReached = true; - if (firstFailure == CommandExitCodes.Success) - firstFailure = CommandExitCodes.UsageError; - break; - } - - if (lineExceededLimit) - { - var lineError = new BatchLineError( - $"batch line {lineNumber} exceeds the {BatchMaxLineChars} character limit.", - CommandExitCodes.UsageError, - Hint: "Split the command across smaller arguments or reduce the input record.", - ErrorCode: CommandErrorCodes.UsageError, - Category: "batch_input_line_length_limit"); - if (jsonSummary) - { - if (!WriteBatchLineErrorJson(lineNumber, lineError, jsonOutput!)) - { - WriteBatchOutputLimitErrorJson( - lineNumber, - commandName: null, - CommandExitCodes.UsageError, - maxOutputChars, - jsonOutput!); - outputLimitReached = true; - } - } - else - WriteBatchLineErrorDiagnostic(lineError, jsonOptions); - lineErrors++; - if (firstFailure == CommandExitCodes.Success) - firstFailure = CommandExitCodes.UsageError; - if (outputLimitReached) - break; - continue; - } - - if (string.IsNullOrWhiteSpace(line)) - continue; - - if (!TryParseBatchLine(line, lineNumber, jsonOptions, !jsonSummary, out var commandName, out var subArgs, out var parseExitCode, out var parseError)) - { - if (jsonSummary) - { - if (!WriteBatchLineErrorJson( - lineNumber, - parseError ?? BuildGenericBatchLineError(lineNumber), - jsonOutput!)) - { - WriteBatchOutputLimitErrorJson( - lineNumber, - commandName: null, - parseExitCode, - maxOutputChars, - jsonOutput!); - outputLimitReached = true; - } - } - lineErrors++; - if (firstFailure == CommandExitCodes.Success) - firstFailure = parseExitCode; - if (outputLimitReached) - break; - continue; - } - - commandsProcessed++; - var batchResult = jsonSummary - ? RunBatchQueryCommandWithJsonRecord( - lineNumber, - commandName, - subArgs, - maxOutputChars, - includeRawStreams, - jsonOutput!, - jsonOptions, - appVersion, - cancellationToken) - : new BatchCommandRunResult( - RunBatchQueryCommand(commandName, subArgs, jsonOptions, appVersion, cancellationToken), - OutputLimitReached: false, - CancellationObserved: false); - var exitCode = batchResult.ExitCode; - if (exitCode != CommandExitCodes.Success) - { - commandFailures++; - if (firstFailure == CommandExitCodes.Success) - firstFailure = exitCode; - } - if (batchResult.OutputLimitReached) - { - outputLimitReached = true; - break; - } - if (batchResult.CancellationObserved) - { - firstFailure = CommandExitCodes.CancelledBySignal; - break; - } - } - - if (jsonSummary) - WriteBatchSummaryJson( - lineNumber, - commandsProcessed, - lineErrors, - commandFailures, - firstFailure, - outputLimitReached, - inputLimitReached, - maxInputLines, - maxOutputChars, - parallelism, - jsonOutput!); - - return firstFailure; - } - catch (OperationCanceledException) when ( - jsonSummary - && cancellationToken.IsCancellationRequested) - { - return WriteBatchSetupCancellationSummary( - maxInputLines, - maxOutputChars, - parallelism, - jsonOptions); - } - finally - { - s_batchReader = null; - s_batchDbPath = null; - s_batchDbPathExplicit = false; - } - } - - private static bool TryReadBatchBoundedOption( - string[] args, - ref int index, - string currentArg, - string optionName, - int minimum, - int maximum, - out int value) - { - value = 0; - string rawValue; - if (currentArg == optionName) - { - if (index + 1 >= args.Length || string.IsNullOrWhiteSpace(args[index + 1])) - { - CommandErrorWriter.WriteStderr(BuildMissingOptionValueError(optionName)); - return false; - } - rawValue = args[++index]; - } - else - { - rawValue = currentArg[(optionName.Length + 1)..]; - if (string.IsNullOrWhiteSpace(rawValue)) - { - CommandErrorWriter.WriteStderr(BuildMissingOptionValueError(optionName)); - return false; - } - } - - if (!int.TryParse(rawValue, NumberStyles.None, CultureInfo.InvariantCulture, out value) - || value < minimum - || value > maximum) - { - CommandErrorWriter.WriteStderr($"Error: {optionName} must be an integer from {minimum} to {maximum}."); - CommandErrorWriter.WriteStderr($"Usage: {ConsoleUi.GetUsageLine("batch")}"); - return false; - } - - return true; - } - - private static int RunBatchParallel( - string dbPath, - bool dbPathExplicit, - int maxInputLines, - int maxOutputChars, - int parallelism, - bool includeRawStreams, - JsonSerializerOptions jsonOptions, - string appVersion, - BatchParallelSession firstSession, - CancellationToken cancellationToken) - { - var sessions = new BatchParallelSession[parallelism]; - sessions[0] = firstSession; - for (var index = 1; index < sessions.Length; index++) - sessions[index] = new BatchParallelSession(dbPath); - var availableSessions = new Queue(sessions); - using var consoleOwnership = ConsoleStreamOwnership.Enter(); - var originalOut = Console.Out; - var originalError = Console.Error; - var stdoutRouter = new BatchConsoleRouter(originalOut); - var stderrRouter = new BatchConsoleRouter(originalError); - var jsonOutput = new BatchJsonOutputWriter( - originalOut, - maxOutputChars, - BatchTerminalOutputReserveChars, - jsonOptions); - var firstFailure = CommandExitCodes.Success; - var lineNumber = 0; - var commandsProcessed = 0; - var lineErrors = 0; - var commandFailures = 0; - var outputLimitReached = false; - var inputLimitReached = false; - var cancellationObserved = false; - using var stopProducing = new CancellationTokenSource(); - using var producerCancellation = CancellationTokenSource.CreateLinkedTokenSource( - cancellationToken, - stopProducing.Token); - var batchInput = GetBatchInputPump(Console.In); - var input = Channel.CreateBounded( - new BoundedChannelOptions(1) - { - FullMode = BoundedChannelFullMode.Wait, - SingleReader = true, - SingleWriter = true, - }); - - Console.SetOut(stdoutRouter); - Console.SetError(stderrRouter); - try - { - var producer = Task.Run(async () => - { - try - { - while (!stopProducing.IsCancellationRequested) - { - if (cancellationToken.IsCancellationRequested) - { - cancellationObserved = true; - break; - } - - var pumpedLine = await batchInput.ReadAsync(producerCancellation.Token) - .ConfigureAwait(false); - if (pumpedLine is null) - break; - - lineNumber++; - BatchInputLineReadForTesting?.Invoke(lineNumber); - if (cancellationToken.IsCancellationRequested) - { - cancellationObserved = true; - if (pumpedLine.Value.ExceededLimit - || !string.IsNullOrWhiteSpace(pumpedLine.Value.Line)) - { - await input.Writer.WriteAsync( - new BatchPendingItem( - lineNumber, - null, - [], - BuildBatchCancellationLineError(lineNumber), - Terminal: true), - stopProducing.Token) - .ConfigureAwait(false); - lineErrors++; - } - break; - } - - if (lineNumber > maxInputLines) - { - var lineError = new BatchLineError( - $"batch input exceeds the {maxInputLines} line limit.", - CommandExitCodes.UsageError, - Hint: "Split the request into smaller batch invocations.", - ErrorCode: CommandErrorCodes.UsageError, - Category: "batch_input_line_limit"); - await input.Writer.WriteAsync( - new BatchPendingItem(lineNumber, null, [], lineError, Terminal: true), - stopProducing.Token) - .ConfigureAwait(false); - lineErrors++; - inputLimitReached = true; - break; - } - - if (pumpedLine.Value.ExceededLimit) - { - var lineError = new BatchLineError( - $"batch line {lineNumber} exceeds the {BatchMaxLineChars} character limit.", - CommandExitCodes.UsageError, - Hint: "Split the command across smaller arguments or reduce the input record.", - ErrorCode: CommandErrorCodes.UsageError, - Category: "batch_input_line_length_limit"); - await input.Writer.WriteAsync( - new BatchPendingItem(lineNumber, null, [], lineError, Terminal: false), - stopProducing.Token) - .ConfigureAwait(false); - lineErrors++; - continue; - } - - var line = pumpedLine.Value.Line; - if (string.IsNullOrWhiteSpace(line)) - continue; - - BatchPendingItem item; - if (!TryParseBatchLine( - line, - lineNumber, - jsonOptions, - writeDiagnostics: false, - out var commandName, - out var subArgs, - out _, - out var parseError)) - { - item = new BatchPendingItem( - lineNumber, - null, - [], - parseError ?? BuildGenericBatchLineError(lineNumber), - Terminal: false); - lineErrors++; - } - else - { - item = new BatchPendingItem( - lineNumber, - commandName, - subArgs, - null, - Terminal: false); - commandsProcessed++; - } - - BatchParallelItemPreparedForTesting?.Invoke(lineNumber); - await input.Writer.WriteAsync(item, stopProducing.Token) - .ConfigureAwait(false); - } - - input.Writer.TryComplete(); - } - catch (OperationCanceledException) when (producerCancellation.IsCancellationRequested) - { - if (cancellationToken.IsCancellationRequested) - cancellationObserved = true; - input.Writer.TryComplete(); - } - catch (Exception ex) - { - input.Writer.TryComplete(ex); - throw; - } - }); - var active = new Queue<( - BatchPendingItem Item, - Task Result, - BatchParallelSession? Session)>(); - - try - { - while (!outputLimitReached) - { - while (active.Count < parallelism && input.Reader.TryRead(out var item)) - { - BatchParallelSession? session = null; - Task result; - if (item.Error is not null) - { - result = Task.FromResult(null); - } - else - { - session = availableSessions.Dequeue(); - var assignedSession = session; - result = Task.Run( - () => RunBatchParallelCommand( - item.LineNumber, - item.CommandName!, - item.Arguments, - dbPath, - dbPathExplicit, - assignedSession, - stdoutRouter, - stderrRouter, - jsonOptions, - appVersion, - cancellationToken)); - } - active.Enqueue((item, result, session)); - } - - if (active.Count > 0 - && (active.Peek().Result.IsCompleted - || active.Count == parallelism - || input.Reader.Completion.IsCompleted)) - { - var (item, resultTask, session) = active.Dequeue(); - var result = resultTask.GetAwaiter().GetResult(); - if (session is not null) - availableSessions.Enqueue(session); - if (item.Error is not null) - { - if (item.Error.ExitCode is CommandExitCodes.CancelledBySignal - or CommandExitCodes.LegacyInterrupted) - { - cancellationObserved = true; - } - if (firstFailure == CommandExitCodes.Success) - firstFailure = item.Error.ExitCode; - - if (item.Terminal) - { - jsonOutput.WriteTerminal(BuildBatchLineErrorJson(item.LineNumber, item.Error)); - continue; - } - - if (!WriteBatchLineErrorJson(item.LineNumber, item.Error, jsonOutput)) - { - WriteBatchOutputLimitErrorJson( - item.LineNumber, - commandName: null, - item.Error.ExitCode, - maxOutputChars, - jsonOutput); - outputLimitReached = true; - break; - } - continue; - } - - if (result is null) - { - throw new InvalidOperationException( - "A parallel batch command completed without a result."); - } - if (WriteBatchCommandRecordJson( - item.LineNumber, - item.CommandName!, - item.Arguments, - result.ExitCode, - result.Stdout, - result.Stderr, - result.Error, - ClassifyBatchOutput(item.CommandName!, item.Arguments), - includeRawStreams, - jsonOutput)) - { - if (result.ExitCode != CommandExitCodes.Success) - { - if (result.ExitCode is CommandExitCodes.CancelledBySignal - or CommandExitCodes.LegacyInterrupted) - { - cancellationObserved = true; - } - commandFailures++; - if (firstFailure == CommandExitCodes.Success) - firstFailure = result.ExitCode; - } - continue; - } - - WriteBatchOutputLimitErrorJson( - item.LineNumber, - item.CommandName, - result.ExitCode, - maxOutputChars, - jsonOutput); - outputLimitReached = true; - commandFailures++; - if (firstFailure == CommandExitCodes.Success) - firstFailure = CommandExitCodes.InvalidArgument; - break; - } - - if (outputLimitReached) - break; - if (active.Count == 0 && input.Reader.Completion.IsCompleted) - break; - if (active.Count > 0 && active.Peek().Result.IsCompleted) - continue; - - using var waitCancellation = new CancellationTokenSource(); - var waitForInput = input.Reader.WaitToReadAsync(waitCancellation.Token).AsTask(); - if (active.Count == 0) - { - if (!waitForInput.GetAwaiter().GetResult()) - break; - continue; - } - - var completed = Task.WhenAny(active.Peek().Result, waitForInput) - .GetAwaiter() - .GetResult(); - if (ReferenceEquals(completed, active.Peek().Result)) - { - waitCancellation.Cancel(); - try - { - waitForInput.GetAwaiter().GetResult(); - } - catch (OperationCanceledException) when (waitCancellation.IsCancellationRequested) - { - } - } - } - } - catch - { - stopProducing.Cancel(); - while (active.Count > 0) - { - try - { - active.Dequeue().Result.GetAwaiter().GetResult(); - } - catch - { - // Preserve the first failure while ensuring sibling workers have exited. - } - } - - try - { - producer.GetAwaiter().GetResult(); - } - catch - { - // Preserve the first failure from the consumer or ordered worker. - } - throw; - } - - if (outputLimitReached) - { - stopProducing.Cancel(); - while (active.Count > 0) - active.Dequeue().Result.GetAwaiter().GetResult(); - } - - try - { - producer.GetAwaiter().GetResult(); - } - catch (OperationCanceledException) when ( - outputLimitReached - && !cancellationToken.IsCancellationRequested) - { - } - - if (cancellationObserved) - firstFailure = CommandExitCodes.CancelledBySignal; - - WriteBatchSummaryJson( - lineNumber, - commandsProcessed, - lineErrors, - commandFailures, - firstFailure, - outputLimitReached, - inputLimitReached, - maxInputLines, - maxOutputChars, - parallelism, - jsonOutput); - return firstFailure; - } - finally - { - try - { - foreach (var session in sessions) - session.Dispose(); - } - finally - { - ConsoleStreamOwnership.Restore(originalOut, originalError); - } - } - } - - private static BatchParallelCommandResult RunBatchParallelCommand( - int lineNumber, - string commandName, - string[] subArgs, - string dbPath, - bool dbPathExplicit, - BatchParallelSession session, - BatchConsoleRouter stdoutRouter, - BatchConsoleRouter stderrRouter, - JsonSerializerOptions jsonOptions, - string appVersion, - CancellationToken cancellationToken) - { - using var stdout = new BatchBoundedStringWriter(BatchMaxCapturedOutputChars, "stdout"); - using var stderr = new BatchBoundedStringWriter(BatchMaxCapturedOutputChars, "stderr"); - using var stdoutRouterRegistration = ScopedConsoleOutput.Register(stdoutRouter); - using var stderrRouterRegistration = ScopedConsoleError.Register(stderrRouter); - using var stdoutScope = stdoutRouter.Push(stdout); - using var stderrScope = stderrRouter.Push(stderr); - var exitCode = CommandExitCodes.DatabaseError; - JsonObject? error = null; - try - { - cancellationToken.ThrowIfCancellationRequested(); - if (!session.TryGetReader(cancellationToken, out var reader, out var validationReason)) - { - exitCode = WriteInvalidCodeIndexDbError(dbPath, validationReason, json: false, jsonOptions); - } - else - { - s_batchReader = reader; - s_batchDbPath = dbPath; - s_batchDbPathExplicit = dbPathExplicit; - BatchParallelCommandStartedForTesting?.Invoke(lineNumber); - cancellationToken.ThrowIfCancellationRequested(); - exitCode = RunBatchQueryCommand(commandName, subArgs, jsonOptions, appVersion, cancellationToken); - BatchParallelCommandCompletedForTesting?.Invoke(lineNumber); - } - } - catch (BatchOutputCaptureLimitExceededException ex) - { - exitCode = CommandExitCodes.InvalidArgument; - error = BuildBatchCaptureLimitError(ex); - } - catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) - { - exitCode = CommandExitCodes.CancelledBySignal; - error = BuildBatchCancellationError(); - } - catch (TimeoutException) - { - exitCode = CommandExitCodes.RuntimeError; - error = BuildBatchTimeoutError(); - } - catch (Exception ex) - { - exitCode = CommandExitCodes.RuntimeError; - error = BuildBatchTypedError( - "batch command failed without affecting other batch items.", - exitCode, - "Retry the item directly if command-specific diagnostics are required.", - CommandErrorCodes.CommandFailed, - SafeDiagnosticFormatter.FormatCategoryType( - "batch_command_failure", - ex.GetType().Name), - "command"); - } - finally - { - s_batchReader = null; - s_batchDbPath = null; - s_batchDbPathExplicit = false; - s_activeQueryProjectRoot = null; - } - - return new BatchParallelCommandResult(exitCode, stdout.ToString(), stderr.ToString(), error); - } - private static void WriteBatchSummaryJson( int inputLinesRead, int commandsProcessed, @@ -1643,7 +705,7 @@ private static bool TryParseBatchLine( } commandName = values[0]; - subArgs = values.Skip(1).ToArray(); + subArgs = [.. values.Skip(1)]; return true; } catch (Exception ex) when (ex is JsonException or InvalidDataException) @@ -1762,7 +824,7 @@ private static bool TryParseBatchCommandObject( values.Add(value); } - subArgs = values.ToArray(); + subArgs = [.. values]; return true; } @@ -1868,12 +930,12 @@ private sealed record BatchLineError( string? Category = null, bool WriteAsJson = false); - private sealed record BatchCommandRunResult( + private readonly record struct BatchCommandRunResult( int ExitCode, bool OutputLimitReached, bool CancellationObserved); - private sealed record BatchPendingItem( + private readonly record struct BatchPendingItem( int LineNumber, string? CommandName, string[] Arguments, @@ -1885,7 +947,11 @@ private static BatchInputPump GetBatchInputPump(TextReader reader) private sealed class BatchInputPump { + // Output-limit recovery can own one channel slot plus one producer-local line. + private const int ReplayLineCapacity = 2; private readonly Channel _lines; + private readonly object _replayGate = new(); + private readonly LinkedList _replayLines = []; public BatchInputPump(TextReader reader) { @@ -1904,7 +970,38 @@ public BatchInputPump(TextReader reader) _ = Task.Run(() => Pump(reader)); } - public async ValueTask ReadAsync(CancellationToken cancellationToken) + public ValueTask ReadAsync(CancellationToken cancellationToken) + { + lock (_replayGate) + { + if (_replayLines.First is not { } first) + return ReadPumpedLineAsync(cancellationToken); + + _replayLines.RemoveFirst(); + return ValueTask.FromResult(first.Value); + } + } + + public void ReplayBeforeBufferedInput(IReadOnlyList lines) + { + if (lines.Count == 0) + return; + + lock (_replayGate) + { + if (_replayLines.Count + lines.Count > ReplayLineCapacity) + { + throw new InvalidOperationException( + "Parallel batch replay exceeded the bounded input ownership window."); + } + + for (var index = lines.Count - 1; index >= 0; index--) + _replayLines.AddFirst(lines[index]); + } + } + + private async ValueTask ReadPumpedLineAsync( + CancellationToken cancellationToken) { try { @@ -1918,11 +1015,13 @@ public BatchInputPump(TextReader reader) private void Pump(TextReader reader) { + long sequence = 0; try { while (TryReadBatchLine(reader, out var line, out var exceededLimit)) { - _lines.Writer.WriteAsync(new BatchPumpedLine(line, exceededLimit)) + _lines.Writer.WriteAsync( + new BatchPumpedLine(++sequence, line, exceededLimit)) .AsTask() .GetAwaiter() .GetResult(); @@ -1936,9 +1035,12 @@ private void Pump(TextReader reader) } } - private readonly record struct BatchPumpedLine(string? Line, bool ExceededLimit); + private readonly record struct BatchPumpedLine( + long Sequence, + string? Line, + bool ExceededLimit); - private sealed record BatchParallelCommandResult( + private readonly record struct BatchParallelCommandResult( int ExitCode, string Stdout, string Stderr, diff --git a/src/CodeIndex/Cli/QueryCommandRunner.BatchExecution.cs b/src/CodeIndex/Cli/QueryCommandRunner.BatchExecution.cs new file mode 100644 index 0000000000..ac1f497f13 --- /dev/null +++ b/src/CodeIndex/Cli/QueryCommandRunner.BatchExecution.cs @@ -0,0 +1,435 @@ +using System.Text.Json; +using CodeIndex.Database; + +namespace CodeIndex.Cli; + +public static partial class QueryCommandRunner +{ + private static int ExecuteBatch( + in BatchExecutionPlan plan, + JsonSerializerOptions jsonOptions, + string appVersion, + CancellationToken cancellationToken) + { + if (plan.Parallelism > 1) + { + BatchParallelSession? firstSession = null; + DbContext? validationDb = null; + try + { + validationDb = new DbContext(DbOpenIntent.QueryOnly, plan.DbPath, cancellationToken); + BatchParallelDatabaseValidatingForTesting?.Invoke(); + if (!validationDb.TryValidateIsCodeIndexDb(out var validationReason)) + return WriteInvalidCodeIndexDbError(plan.DbPath, validationReason, json: false, jsonOptions); + var transferredDb = validationDb; + validationDb = null; + firstSession = BatchParallelSession.FromValidated( + plan.DbPath, + transferredDb, + cancellationToken); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + firstSession?.Dispose(); + return WriteBatchSetupCancellationSummary( + plan.MaxInputLines, + plan.MaxOutputChars, + plan.Parallelism, + jsonOptions); + } + catch + { + firstSession?.Dispose(); + throw; + } + finally + { + validationDb?.Dispose(); + } + + try + { + return RunBatchParallel( + plan, + jsonOptions, + appVersion, + firstSession!, + cancellationToken); + } + catch + { + firstSession?.Dispose(); + throw; + } + } + + return RunBatchSerial(in plan, jsonOptions, appVersion, cancellationToken); + } + + private static int RunBatchSerial( + in BatchExecutionPlan plan, + JsonSerializerOptions jsonOptions, + string appVersion, + CancellationToken cancellationToken) + { + try + { + using var db = new DbContext(DbOpenIntent.QueryOnly, plan.DbPath, cancellationToken); + if (!db.TryValidateIsCodeIndexDb(out var validationReason)) + return WriteInvalidCodeIndexDbError(plan.DbPath, validationReason, json: false, jsonOptions); + + s_batchReader = new DbReader(db); + s_batchDbPath = plan.DbPath; + s_batchDbPathExplicit = plan.DbPathExplicit; + var jsonOutput = plan.JsonSummary + ? new BatchJsonOutputWriter( + Console.Out, + plan.MaxOutputChars, + BatchTerminalOutputReserveChars, + jsonOptions) + : null; + var state = new BatchExecutionState(); + var batchInput = GetBatchInputPump(Console.In); + while (true) + { + BatchPumpedLine? pumpedLine; + try + { + pumpedLine = batchInput.ReadAsync(cancellationToken) + .AsTask() + .GetAwaiter() + .GetResult(); + } + catch (OperationCanceledException) when ( + plan.JsonSummary + && cancellationToken.IsCancellationRequested) + { + state.FirstFailure = CommandExitCodes.CancelledBySignal; + state.CancellationObserved = true; + break; + } + if (pumpedLine is null) + break; + + var currentLine = pumpedLine.Value; + var preparation = PrepareBatchLine( + in currentLine, + in plan, + state, + jsonOptions, + writeDiagnostics: !plan.JsonSummary, + cancellationToken, + out var item); + switch (preparation) + { + case BatchLinePreparationKind.Blank: + continue; + + case BatchLinePreparationKind.CancellationWithoutRecord: + state.FirstFailure = CommandExitCodes.CancelledBySignal; + break; + + case BatchLinePreparationKind.CancellationRecord: + state.FirstFailure = CommandExitCodes.CancelledBySignal; + if (!WriteBatchLineErrorJson(item.LineNumber, item.Error!, jsonOutput!)) + { + WriteBatchOutputLimitErrorJson( + item.LineNumber, + commandName: null, + CommandExitCodes.CancelledBySignal, + plan.MaxOutputChars, + jsonOutput!); + state.OutputLimitReached = true; + } + RecordPreparedBatchLine(state, preparation); + break; + + case BatchLinePreparationKind.InputLimit: + if (plan.JsonSummary) + jsonOutput!.WriteTerminal(BuildBatchLineErrorJson(item.LineNumber, item.Error!)); + else + WriteBatchLineErrorDiagnostic(item.Error!, jsonOptions); + RecordPreparedBatchLine(state, preparation); + RecordBatchFirstFailure(state, CommandExitCodes.UsageError); + break; + + case BatchLinePreparationKind.LineLengthError: + if (plan.JsonSummary) + { + if (!WriteBatchLineErrorJson(item.LineNumber, item.Error!, jsonOutput!)) + { + WriteBatchOutputLimitErrorJson( + item.LineNumber, + commandName: null, + CommandExitCodes.UsageError, + plan.MaxOutputChars, + jsonOutput!); + state.OutputLimitReached = true; + } + } + else + { + WriteBatchLineErrorDiagnostic(item.Error!, jsonOptions); + } + RecordPreparedBatchLine(state, preparation); + RecordBatchFirstFailure(state, CommandExitCodes.UsageError); + if (state.OutputLimitReached) + break; + continue; + + case BatchLinePreparationKind.ParseError: + if (plan.JsonSummary + && !WriteBatchLineErrorJson(item.LineNumber, item.Error!, jsonOutput!)) + { + WriteBatchOutputLimitErrorJson( + item.LineNumber, + commandName: null, + item.Error!.ExitCode, + plan.MaxOutputChars, + jsonOutput!); + state.OutputLimitReached = true; + } + RecordPreparedBatchLine(state, preparation); + RecordBatchFirstFailure(state, item.Error!.ExitCode); + if (state.OutputLimitReached) + break; + continue; + + case BatchLinePreparationKind.Command: + RecordPreparedBatchLine(state, preparation); + break; + + default: + throw new InvalidOperationException( + $"Unknown batch line preparation kind '{preparation}'."); + } + + if (preparation != BatchLinePreparationKind.Command) + break; + + var batchResult = plan.JsonSummary + ? RunBatchQueryCommandWithJsonRecord( + item.LineNumber, + item.CommandName!, + item.Arguments, + plan.MaxOutputChars, + plan.IncludeRawStreams, + jsonOutput!, + jsonOptions, + appVersion, + cancellationToken) + : new BatchCommandRunResult( + RunBatchQueryCommand( + item.CommandName!, + item.Arguments, + jsonOptions, + appVersion, + cancellationToken), + OutputLimitReached: false, + CancellationObserved: false); + if (batchResult.ExitCode != CommandExitCodes.Success) + { + state.CommandFailures++; + RecordBatchFirstFailure(state, batchResult.ExitCode); + } + if (batchResult.OutputLimitReached) + { + state.OutputLimitReached = true; + break; + } + if (batchResult.CancellationObserved) + { + state.FirstFailure = CommandExitCodes.CancelledBySignal; + break; + } + } + + if (plan.JsonSummary) + { + WriteBatchSummaryJson( + state.InputLinesRead, + state.CommandsProcessed, + state.LineErrors, + state.CommandFailures, + state.FirstFailure, + state.OutputLimitReached, + state.InputLimitReached, + plan.MaxInputLines, + plan.MaxOutputChars, + plan.Parallelism, + jsonOutput!); + } + + return state.FirstFailure; + } + catch (OperationCanceledException) when ( + plan.JsonSummary + && cancellationToken.IsCancellationRequested) + { + return WriteBatchSetupCancellationSummary( + plan.MaxInputLines, + plan.MaxOutputChars, + plan.Parallelism, + jsonOptions); + } + finally + { + s_batchReader = null; + s_batchDbPath = null; + s_batchDbPathExplicit = false; + } + } + + private static BatchLinePreparationKind PrepareBatchLine( + in BatchPumpedLine pumpedLine, + in BatchExecutionPlan plan, + BatchExecutionState state, + JsonSerializerOptions jsonOptions, + bool writeDiagnostics, + CancellationToken cancellationToken, + out BatchPendingItem item) + { + state.InputLinesRead++; + BatchInputLineReadForTesting?.Invoke(state.InputLinesRead); + if (cancellationToken.IsCancellationRequested) + { + if (!plan.JsonSummary) + cancellationToken.ThrowIfCancellationRequested(); + + state.CancellationObserved = true; + if (pumpedLine.ExceededLimit || !string.IsNullOrWhiteSpace(pumpedLine.Line)) + { + item = new BatchPendingItem( + state.InputLinesRead, + null, + [], + BuildBatchCancellationLineError(state.InputLinesRead), + Terminal: true); + return BatchLinePreparationKind.CancellationRecord; + } + + item = default; + return BatchLinePreparationKind.CancellationWithoutRecord; + } + + if (state.InputLinesRead > plan.MaxInputLines) + { + var lineError = new BatchLineError( + $"batch input exceeds the {plan.MaxInputLines} line limit.", + CommandExitCodes.UsageError, + Hint: "Split the request into smaller batch invocations.", + ErrorCode: CommandErrorCodes.UsageError, + Category: "batch_input_line_limit"); + item = new BatchPendingItem( + state.InputLinesRead, + null, + [], + lineError, + Terminal: true); + return BatchLinePreparationKind.InputLimit; + } + + if (pumpedLine.ExceededLimit) + { + var lineError = new BatchLineError( + $"batch line {state.InputLinesRead} exceeds the {BatchMaxLineChars} character limit.", + CommandExitCodes.UsageError, + Hint: "Split the command across smaller arguments or reduce the input record.", + ErrorCode: CommandErrorCodes.UsageError, + Category: "batch_input_line_length_limit"); + item = new BatchPendingItem( + state.InputLinesRead, + null, + [], + lineError, + Terminal: false); + return BatchLinePreparationKind.LineLengthError; + } + + if (string.IsNullOrWhiteSpace(pumpedLine.Line)) + { + item = default; + return BatchLinePreparationKind.Blank; + } + + if (!TryParseBatchLine( + pumpedLine.Line, + state.InputLinesRead, + jsonOptions, + writeDiagnostics, + out var commandName, + out var subArgs, + out _, + out var parseError)) + { + item = new BatchPendingItem( + state.InputLinesRead, + null, + [], + parseError ?? BuildGenericBatchLineError(state.InputLinesRead), + Terminal: false); + return BatchLinePreparationKind.ParseError; + } + + item = new BatchPendingItem( + state.InputLinesRead, + commandName, + subArgs, + null, + Terminal: false); + return BatchLinePreparationKind.Command; + } + + private static void RecordPreparedBatchLine( + BatchExecutionState state, + BatchLinePreparationKind preparation) + { + if (preparation == BatchLinePreparationKind.Command) + { + state.CommandsProcessed++; + return; + } + + state.LineErrors++; + if (preparation == BatchLinePreparationKind.InputLimit) + state.InputLimitReached = true; + } + + private static void RecordBatchFirstFailure(BatchExecutionState state, int exitCode) + { + if (state.FirstFailure == CommandExitCodes.Success) + state.FirstFailure = exitCode; + } + + private readonly record struct BatchExecutionPlan( + string DbPath, + bool DbPathExplicit, + bool JsonSummary, + int MaxInputLines, + int MaxOutputChars, + int Parallelism, + bool IncludeRawStreams); + + private sealed class BatchExecutionState + { + public int InputLinesRead { get; set; } + public int CommandsProcessed { get; set; } + public int LineErrors { get; set; } + public int CommandFailures { get; set; } + public int FirstFailure { get; set; } = CommandExitCodes.Success; + public bool OutputLimitReached { get; set; } + public bool InputLimitReached { get; set; } + public bool CancellationObserved { get; set; } + } + + private enum BatchLinePreparationKind + { + Blank, + Command, + ParseError, + LineLengthError, + InputLimit, + CancellationWithoutRecord, + CancellationRecord, + } +} diff --git a/src/CodeIndex/Cli/QueryCommandRunner.BatchFacade.cs b/src/CodeIndex/Cli/QueryCommandRunner.BatchFacade.cs new file mode 100644 index 0000000000..b7e2fd1812 --- /dev/null +++ b/src/CodeIndex/Cli/QueryCommandRunner.BatchFacade.cs @@ -0,0 +1,197 @@ +using System.Globalization; +using System.Text.Json; +using CodeIndex.Diagnostics; + +namespace CodeIndex.Cli; + +public static partial class QueryCommandRunner +{ + public static int RunBatch( + string[] cmdArgs, + JsonSerializerOptions jsonOptions, + string appVersion = "", + CancellationToken cancellationToken = default) + { + var dbPath = Path.Combine(".cdidx", "codeindex.db"); + var dbPathExplicit = false; + var jsonSummary = false; + var maxInputLines = BatchDefaultInputLines; + var maxOutputChars = BatchDefaultTotalOutputChars; + var maxOutputCharsSpecified = false; + var parallelism = 1; + var parallelismSpecified = false; + var includeRawStreams = false; + for (var i = 0; i < cmdArgs.Length; i++) + { + var arg = cmdArgs[i]; + if (arg == "--json-summary") + { + jsonSummary = true; + continue; + } + + if (arg == "--include-raw-streams") + { + includeRawStreams = true; + continue; + } + + if (arg == "--db") + { + if (i + 1 >= cmdArgs.Length || string.IsNullOrWhiteSpace(cmdArgs[i + 1])) + { + CommandErrorWriter.WriteStderr(BuildMissingOptionValueError("--db")); + return CommandExitCodes.UsageError; + } + dbPath = cmdArgs[++i]; + dbPathExplicit = true; + continue; + } + + if (arg.StartsWith("--db=", StringComparison.Ordinal)) + { + dbPath = arg["--db=".Length..]; + if (string.IsNullOrWhiteSpace(dbPath)) + { + CommandErrorWriter.WriteStderr(BuildMissingOptionValueError("--db")); + return CommandExitCodes.UsageError; + } + dbPathExplicit = true; + continue; + } + + if (arg == "--max-input-lines" || arg.StartsWith("--max-input-lines=", StringComparison.Ordinal)) + { + if (!TryReadBatchBoundedOption( + cmdArgs, + ref i, + arg, + "--max-input-lines", + 1, + BatchMaxInputLines, + out maxInputLines)) + { + return CommandExitCodes.UsageError; + } + continue; + } + + if (arg == "--max-output-chars" || arg.StartsWith("--max-output-chars=", StringComparison.Ordinal)) + { + if (!TryReadBatchBoundedOption( + cmdArgs, + ref i, + arg, + "--max-output-chars", + BatchMinTotalOutputChars, + BatchMaxTotalOutputChars, + out maxOutputChars)) + { + return CommandExitCodes.UsageError; + } + maxOutputCharsSpecified = true; + continue; + } + + if (arg == "--parallel" || arg.StartsWith("--parallel=", StringComparison.Ordinal)) + { + if (!TryReadBatchBoundedOption( + cmdArgs, + ref i, + arg, + "--parallel", + 1, + BatchMaxParallelism, + out parallelism)) + { + return CommandExitCodes.UsageError; + } + parallelismSpecified = true; + continue; + } + + CommandErrorWriter.WriteStderr($"Error: {ConsoleUi.FormatBoundedValue(arg)} is not supported for batch."); + CommandErrorWriter.WriteStderr($"Usage: {ConsoleUi.GetUsageLine("batch")}"); + return CommandExitCodes.UsageError; + } + + if (parallelismSpecified && !jsonSummary) + { + CommandErrorWriter.WriteStderr("Error: --parallel requires --json-summary so concurrent child output can be isolated and emitted in input order."); + CommandErrorWriter.WriteStderr($"Usage: {ConsoleUi.GetUsageLine("batch")}"); + return CommandExitCodes.UsageError; + } + if (maxOutputCharsSpecified && !jsonSummary) + { + CommandErrorWriter.WriteStderr("Error: --max-output-chars requires --json-summary because ordinary batch output streams directly."); + CommandErrorWriter.WriteStderr($"Usage: {ConsoleUi.GetUsageLine("batch")}"); + return CommandExitCodes.UsageError; + } + if (includeRawStreams && !jsonSummary) + { + CommandErrorWriter.WriteStderr("Error: --include-raw-streams requires --json-summary."); + CommandErrorWriter.WriteStderr($"Usage: {ConsoleUi.GetUsageLine("batch")}"); + return CommandExitCodes.UsageError; + } + + var isUri = dbPath.StartsWith("file:", StringComparison.OrdinalIgnoreCase); + if (!isUri && !File.Exists(dbPath)) + { + CommandErrorWriter.WriteStderr($"Error [{CommandErrorCodes.DbNotFound}]: database not found at {FormatDbDiagnosticValue(Path.GetFullPath(dbPath))}"); + CommandErrorWriter.WriteStderr("Hint: create or refresh the index with `cdidx index ` (or `cdidx .`) and then rerun this command."); + return CommandExitCodes.DatabaseError; + } + + var plan = new BatchExecutionPlan( + dbPath, + dbPathExplicit, + jsonSummary, + maxInputLines, + maxOutputChars, + parallelism, + includeRawStreams); + return ExecuteBatch(in plan, jsonOptions, appVersion, cancellationToken); + } + + private static bool TryReadBatchBoundedOption( + string[] args, + ref int index, + string currentArg, + string optionName, + int minimum, + int maximum, + out int value) + { + value = 0; + string rawValue; + if (currentArg == optionName) + { + if (index + 1 >= args.Length || string.IsNullOrWhiteSpace(args[index + 1])) + { + CommandErrorWriter.WriteStderr(BuildMissingOptionValueError(optionName)); + return false; + } + rawValue = args[++index]; + } + else + { + rawValue = currentArg[(optionName.Length + 1)..]; + if (string.IsNullOrWhiteSpace(rawValue)) + { + CommandErrorWriter.WriteStderr(BuildMissingOptionValueError(optionName)); + return false; + } + } + + if (!int.TryParse(rawValue, NumberStyles.None, CultureInfo.InvariantCulture, out value) + || value < minimum + || value > maximum) + { + CommandErrorWriter.WriteStderr($"Error: {optionName} must be an integer from {minimum} to {maximum}."); + CommandErrorWriter.WriteStderr($"Usage: {ConsoleUi.GetUsageLine("batch")}"); + return false; + } + + return true; + } +} diff --git a/src/CodeIndex/Cli/QueryCommandRunner.BatchParallelExecution.cs b/src/CodeIndex/Cli/QueryCommandRunner.BatchParallelExecution.cs new file mode 100644 index 0000000000..a932a7ef85 --- /dev/null +++ b/src/CodeIndex/Cli/QueryCommandRunner.BatchParallelExecution.cs @@ -0,0 +1,676 @@ +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Threading.Channels; +using CodeIndex.Diagnostics; + +namespace CodeIndex.Cli; + +public static partial class QueryCommandRunner +{ + private static int RunBatchParallel( + BatchExecutionPlan plan, + JsonSerializerOptions jsonOptions, + string appVersion, + BatchParallelSession firstSession, + CancellationToken cancellationToken) + { + var sessions = new BatchParallelSession[plan.Parallelism]; + sessions[0] = firstSession; + for (var index = 1; index < sessions.Length; index++) + sessions[index] = new BatchParallelSession(plan.DbPath); + var availableSessions = new Queue(sessions); + using var consoleOwnership = ConsoleStreamOwnership.Enter(); + var originalOut = Console.Out; + var originalError = Console.Error; + var stdoutRouter = new BatchConsoleRouter(originalOut); + var stderrRouter = new BatchConsoleRouter(originalError); + var jsonOutput = new BatchJsonOutputWriter( + originalOut, + plan.MaxOutputChars, + BatchTerminalOutputReserveChars, + jsonOptions); + var state = new BatchExecutionState(); + var replayTracker = new BatchParallelReplayTracker(); + using var stopProducing = new CancellationTokenSource(); + using var producerCancellation = CancellationTokenSource.CreateLinkedTokenSource( + cancellationToken, + stopProducing.Token); + var batchInput = GetBatchInputPump(Console.In); + var input = Channel.CreateBounded( + new BoundedChannelOptions(1) + { + FullMode = BoundedChannelFullMode.Wait, + SingleReader = true, + SingleWriter = true, + }); + + Console.SetOut(stdoutRouter); + Console.SetError(stderrRouter); + try + { + var producerPlan = new BatchParallelProducerPlan( + plan, + jsonOptions, + state, + replayTracker, + batchInput, + input.Writer, + new BatchParallelProducerCancellation( + stopProducing.Token, + producerCancellation.Token, + cancellationToken)); + var producer = Task.Run( + () => ProduceBatchItemsAsync(producerPlan), + CancellationToken.None); + var active = new Queue(); + var consumerPlan = new BatchParallelConsumerPlan( + plan, + jsonOptions, + appVersion, + availableSessions, + new BatchParallelOutputServices(stdoutRouter, stderrRouter, jsonOutput), + state, + replayTracker, + cancellationToken); + + try + { + ConsumeBatchItemsInOrder(in consumerPlan, input.Reader, active); + } + catch + { + stopProducing.Cancel(); + DrainBatchWorkers(active, preserveEarlierFailure: true); + + try + { + producer.GetAwaiter().GetResult(); + } + catch + { + // Preserve the first failure from the consumer or ordered worker. + } + throw; + } + + if (state.OutputLimitReached) + { + stopProducing.Cancel(); + DrainBatchWorkers(active, preserveEarlierFailure: false); + } + + try + { + producer.GetAwaiter().GetResult(); + } + catch (OperationCanceledException) when ( + state.OutputLimitReached + && !cancellationToken.IsCancellationRequested) + { + } + + if (state.OutputLimitReached && !cancellationToken.IsCancellationRequested) + { + batchInput.ReplayBeforeBufferedInput( + replayTracker.TakeForReplay(state)); + } + + if (state.CancellationObserved) + state.FirstFailure = CommandExitCodes.CancelledBySignal; + + WriteBatchSummaryJson( + state.InputLinesRead, + state.CommandsProcessed, + state.LineErrors, + state.CommandFailures, + state.FirstFailure, + state.OutputLimitReached, + state.InputLimitReached, + plan.MaxInputLines, + plan.MaxOutputChars, + plan.Parallelism, + jsonOutput); + return state.FirstFailure; + } + finally + { + try + { + foreach (var session in sessions) + session.Dispose(); + } + finally + { + ConsoleStreamOwnership.Restore(originalOut, originalError); + } + } + } + + private static async Task ProduceBatchItemsAsync(BatchParallelProducerPlan producer) + { + var plan = producer.ExecutionPlan; + try + { + while (!producer.Cancellation.StopToken.IsCancellationRequested) + { + if (producer.Cancellation.CallerToken.IsCancellationRequested) + { + producer.State.CancellationObserved = true; + break; + } + + var pumpedLine = await producer.Input.ReadAsync(producer.Cancellation.ReadToken) + .ConfigureAwait(false); + if (pumpedLine is null) + break; + + var currentLine = pumpedLine.Value; + var preparation = PrepareBatchLine( + in currentLine, + in plan, + producer.State, + producer.JsonOptions, + writeDiagnostics: false, + producer.Cancellation.CallerToken, + out var item); + if (preparation == BatchLinePreparationKind.Blank) + continue; + if (preparation == BatchLinePreparationKind.CancellationWithoutRecord) + break; + + producer.ReplayTracker.Register(in currentLine, preparation); + var pendingItem = new BatchParallelPendingItem( + item, + currentLine.Sequence); + switch (preparation) + { + case BatchLinePreparationKind.CancellationRecord: + case BatchLinePreparationKind.InputLimit: + await producer.Output.WriteAsync( + pendingItem, + producer.Cancellation.StopToken) + .ConfigureAwait(false); + RecordPreparedBatchLine(producer.State, preparation); + producer.ReplayTracker.MarkCountersRecorded(currentLine.Sequence); + break; + + case BatchLinePreparationKind.LineLengthError: + await producer.Output.WriteAsync( + pendingItem, + producer.Cancellation.StopToken) + .ConfigureAwait(false); + RecordPreparedBatchLine(producer.State, preparation); + producer.ReplayTracker.MarkCountersRecorded(currentLine.Sequence); + continue; + + case BatchLinePreparationKind.ParseError: + case BatchLinePreparationKind.Command: + RecordPreparedBatchLine(producer.State, preparation); + producer.ReplayTracker.MarkCountersRecorded(currentLine.Sequence); + BatchParallelItemPreparedForTesting?.Invoke(item.LineNumber); + await producer.Output.WriteAsync( + pendingItem, + producer.Cancellation.StopToken) + .ConfigureAwait(false); + continue; + + default: + throw new InvalidOperationException( + $"Unknown batch line preparation kind '{preparation}'."); + } + + break; + } + + producer.Output.TryComplete(); + } + catch (OperationCanceledException) when (producer.Cancellation.ReadToken.IsCancellationRequested) + { + if (producer.Cancellation.CallerToken.IsCancellationRequested) + producer.State.CancellationObserved = true; + producer.Output.TryComplete(); + } + catch (Exception ex) + { + producer.Output.TryComplete(ex); + throw; + } + } + + private static void ConsumeBatchItemsInOrder( + in BatchParallelConsumerPlan consumer, + ChannelReader input, + Queue active) + { + while (!consumer.State.OutputLimitReached) + { + FillBatchWorkerQueue(in consumer, input, active); + if (ShouldConsumeOldestBatchItem(in consumer, input, active)) + { + ConsumeOldestBatchItem(in consumer, active); + continue; + } + + if (active.Count == 0 && input.Completion.IsCompleted) + break; + if (active.Count > 0 && active.Peek().Result.IsCompleted) + continue; + if (!WaitForBatchInputOrOldestWorker(input, active)) + break; + } + } + + private static void FillBatchWorkerQueue( + in BatchParallelConsumerPlan consumer, + ChannelReader input, + Queue active) + { + while (active.Count < consumer.ExecutionPlan.Parallelism + && input.TryRead(out var pendingItem)) + { + consumer.ReplayTracker.Commit(pendingItem.InputSequence); + var item = pendingItem.Item; + BatchParallelSession? session = null; + Task result; + if (item.Error is not null) + { + result = Task.FromResult(null); + } + else + { + session = consumer.AvailableSessions.Dequeue(); + var assignedSession = session; + var executionPlan = consumer.ExecutionPlan; + var stdoutRouter = consumer.Output.StdoutRouter; + var stderrRouter = consumer.Output.StderrRouter; + var jsonOptions = consumer.JsonOptions; + var appVersion = consumer.AppVersion; + var cancellationToken = consumer.CancellationToken; + result = Task.Run( + () => RunBatchParallelCommand( + item.LineNumber, + item.CommandName!, + item.Arguments, + executionPlan.DbPath, + executionPlan.DbPathExplicit, + assignedSession, + stdoutRouter, + stderrRouter, + jsonOptions, + appVersion, + cancellationToken)); + } + active.Enqueue(new BatchActiveItem(item, result, session)); + } + } + + private static bool ShouldConsumeOldestBatchItem( + in BatchParallelConsumerPlan consumer, + ChannelReader input, + Queue active) + => active.Count > 0 + && (active.Peek().Result.IsCompleted + || active.Count == consumer.ExecutionPlan.Parallelism + || input.Completion.IsCompleted); + + private static void ConsumeOldestBatchItem( + in BatchParallelConsumerPlan consumer, + Queue active) + { + var activeItem = active.Dequeue(); + var item = activeItem.Item; + var result = activeItem.Result.GetAwaiter().GetResult(); + if (activeItem.Session is not null) + consumer.AvailableSessions.Enqueue(activeItem.Session); + if (item.Error is not null) + { + if (item.Error.ExitCode is CommandExitCodes.CancelledBySignal + or CommandExitCodes.LegacyInterrupted) + { + consumer.State.CancellationObserved = true; + } + RecordBatchFirstFailure(consumer.State, item.Error.ExitCode); + + if (item.Terminal) + { + consumer.Output.JsonOutput.WriteTerminal( + BuildBatchLineErrorJson(item.LineNumber, item.Error)); + return; + } + + if (!WriteBatchLineErrorJson(item.LineNumber, item.Error, consumer.Output.JsonOutput)) + { + WriteBatchOutputLimitErrorJson( + item.LineNumber, + commandName: null, + item.Error.ExitCode, + consumer.ExecutionPlan.MaxOutputChars, + consumer.Output.JsonOutput); + consumer.State.OutputLimitReached = true; + } + return; + } + + if (result is null) + { + throw new InvalidOperationException( + "A parallel batch command completed without a result."); + } + var completedResult = result.Value; + if (WriteBatchCommandRecordJson( + item.LineNumber, + item.CommandName!, + item.Arguments, + completedResult.ExitCode, + completedResult.Stdout, + completedResult.Stderr, + completedResult.Error, + ClassifyBatchOutput(item.CommandName!, item.Arguments), + consumer.ExecutionPlan.IncludeRawStreams, + consumer.Output.JsonOutput)) + { + if (completedResult.ExitCode != CommandExitCodes.Success) + { + if (completedResult.ExitCode is CommandExitCodes.CancelledBySignal + or CommandExitCodes.LegacyInterrupted) + { + consumer.State.CancellationObserved = true; + } + consumer.State.CommandFailures++; + RecordBatchFirstFailure(consumer.State, completedResult.ExitCode); + } + return; + } + + WriteBatchOutputLimitErrorJson( + item.LineNumber, + item.CommandName, + completedResult.ExitCode, + consumer.ExecutionPlan.MaxOutputChars, + consumer.Output.JsonOutput); + consumer.State.OutputLimitReached = true; + consumer.State.CommandFailures++; + RecordBatchFirstFailure(consumer.State, CommandExitCodes.InvalidArgument); + } + + private static bool WaitForBatchInputOrOldestWorker( + ChannelReader input, + Queue active) + { + using var waitCancellation = new CancellationTokenSource(); + var waitForInput = input.WaitToReadAsync(waitCancellation.Token).AsTask(); + if (active.Count == 0) + return waitForInput.GetAwaiter().GetResult(); + + var completed = Task.WhenAny(active.Peek().Result, waitForInput) + .GetAwaiter() + .GetResult(); + if (ReferenceEquals(completed, active.Peek().Result)) + { + waitCancellation.Cancel(); + try + { + waitForInput.GetAwaiter().GetResult(); + } + catch (OperationCanceledException) when (waitCancellation.IsCancellationRequested) + { + } + } + + return true; + } + + private static void DrainBatchWorkers( + Queue active, + bool preserveEarlierFailure) + { + while (active.Count > 0) + { + if (!preserveEarlierFailure) + { + active.Dequeue().Result.GetAwaiter().GetResult(); + continue; + } + + try + { + active.Dequeue().Result.GetAwaiter().GetResult(); + } + catch + { + // Preserve the first failure while ensuring sibling workers have exited. + } + } + } + + private static BatchParallelCommandResult RunBatchParallelCommand( + int lineNumber, + string commandName, + string[] subArgs, + string dbPath, + bool dbPathExplicit, + BatchParallelSession session, + BatchConsoleRouter stdoutRouter, + BatchConsoleRouter stderrRouter, + JsonSerializerOptions jsonOptions, + string appVersion, + CancellationToken cancellationToken) + { + using var stdout = new BatchBoundedStringWriter(BatchMaxCapturedOutputChars, "stdout"); + using var stderr = new BatchBoundedStringWriter(BatchMaxCapturedOutputChars, "stderr"); + using var stdoutRouterRegistration = ScopedConsoleOutput.Register(stdoutRouter); + using var stderrRouterRegistration = ScopedConsoleError.Register(stderrRouter); + using var stdoutScope = stdoutRouter.Push(stdout); + using var stderrScope = stderrRouter.Push(stderr); + var exitCode = CommandExitCodes.DatabaseError; + JsonObject? error = null; + try + { + cancellationToken.ThrowIfCancellationRequested(); + if (!session.TryGetReader(cancellationToken, out var reader, out var validationReason)) + { + exitCode = WriteInvalidCodeIndexDbError(dbPath, validationReason, json: false, jsonOptions); + } + else + { + s_batchReader = reader; + s_batchDbPath = dbPath; + s_batchDbPathExplicit = dbPathExplicit; + BatchParallelCommandStartedForTesting?.Invoke(lineNumber); + cancellationToken.ThrowIfCancellationRequested(); + exitCode = RunBatchQueryCommand(commandName, subArgs, jsonOptions, appVersion, cancellationToken); + BatchParallelCommandCompletedForTesting?.Invoke(lineNumber); + } + } + catch (BatchOutputCaptureLimitExceededException ex) + { + exitCode = CommandExitCodes.InvalidArgument; + error = BuildBatchCaptureLimitError(ex); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + exitCode = CommandExitCodes.CancelledBySignal; + error = BuildBatchCancellationError(); + } + catch (TimeoutException) + { + exitCode = CommandExitCodes.RuntimeError; + error = BuildBatchTimeoutError(); + } + catch (Exception ex) + { + exitCode = CommandExitCodes.RuntimeError; + error = BuildBatchTypedError( + "batch command failed without affecting other batch items.", + exitCode, + "Retry the item directly if command-specific diagnostics are required.", + CommandErrorCodes.CommandFailed, + SafeDiagnosticFormatter.FormatCategoryType( + "batch_command_failure", + ex.GetType().Name), + "command"); + } + finally + { + s_batchReader = null; + s_batchDbPath = null; + s_batchDbPathExplicit = false; + s_activeQueryProjectRoot = null; + } + + return new BatchParallelCommandResult(exitCode, stdout.ToString(), stderr.ToString(), error); + } + + private readonly record struct BatchParallelProducerPlan( + BatchExecutionPlan ExecutionPlan, + JsonSerializerOptions JsonOptions, + BatchExecutionState State, + BatchParallelReplayTracker ReplayTracker, + BatchInputPump Input, + ChannelWriter Output, + BatchParallelProducerCancellation Cancellation); + + private readonly record struct BatchParallelProducerCancellation( + CancellationToken StopToken, + CancellationToken ReadToken, + CancellationToken CallerToken); + + private readonly record struct BatchParallelConsumerPlan( + BatchExecutionPlan ExecutionPlan, + JsonSerializerOptions JsonOptions, + string AppVersion, + Queue AvailableSessions, + BatchParallelOutputServices Output, + BatchExecutionState State, + BatchParallelReplayTracker ReplayTracker, + CancellationToken CancellationToken); + + private readonly record struct BatchParallelOutputServices( + BatchConsoleRouter StdoutRouter, + BatchConsoleRouter StderrRouter, + BatchJsonOutputWriter JsonOutput); + + private readonly record struct BatchActiveItem( + BatchPendingItem Item, + Task Result, + BatchParallelSession? Session); + + private readonly record struct BatchParallelPendingItem( + BatchPendingItem Item, + long InputSequence); + + private readonly record struct BatchParallelReplayEntry( + BatchPumpedLine Input, + BatchLinePreparationKind Preparation, + bool CountersRecorded); + + private sealed class BatchParallelReplayTracker + { + // A bounded channel slot, the producer's local item, and the consumer's + // just-read item can overlap until the consumer records its commit. + private const int Capacity = 3; + private readonly object _gate = new(); + private readonly List _uncommitted = new(Capacity); + + public void Register( + in BatchPumpedLine input, + BatchLinePreparationKind preparation) + { + lock (_gate) + { + if (_uncommitted.Count == Capacity) + { + throw new InvalidOperationException( + "Parallel batch input ownership exceeded its bounded channel window."); + } + + _uncommitted.Add(new BatchParallelReplayEntry( + input, + preparation, + CountersRecorded: false)); + } + } + + public void MarkCountersRecorded(long sequence) + { + lock (_gate) + { + for (var index = 0; index < _uncommitted.Count; index++) + { + var entry = _uncommitted[index]; + if (entry.Input.Sequence != sequence) + continue; + + _uncommitted[index] = entry with { CountersRecorded = true }; + return; + } + + // The consumer can commit the line immediately after the + // channel write completes, before the producer reaches here. + // A committed line no longer needs rollback metadata. + } + } + + public void Commit(long sequence) + { + lock (_gate) + { + for (var index = 0; index < _uncommitted.Count; index++) + { + if (_uncommitted[index].Input.Sequence != sequence) + continue; + + _uncommitted.RemoveAt(index); + return; + } + } + + throw new InvalidOperationException( + "Parallel batch input was dispatched without a matching ownership lease."); + } + + public BatchPumpedLine[] TakeForReplay(BatchExecutionState state) + { + BatchParallelReplayEntry[] entries; + lock (_gate) + { + entries = [.. _uncommitted]; + _uncommitted.Clear(); + } + + Array.Sort( + entries, + static (left, right) => left.Input.Sequence.CompareTo(right.Input.Sequence)); + var replay = new BatchPumpedLine[entries.Length]; + for (var index = 0; index < entries.Length; index++) + { + var entry = entries[index]; + replay[index] = entry.Input; + state.InputLinesRead--; + if (!entry.CountersRecorded) + continue; + + if (entry.Preparation == BatchLinePreparationKind.Command) + { + state.CommandsProcessed--; + } + else + { + state.LineErrors--; + if (entry.Preparation == BatchLinePreparationKind.InputLimit) + state.InputLimitReached = false; + } + } + + if (state.InputLinesRead < 0 + || state.CommandsProcessed < 0 + || state.LineErrors < 0) + { + throw new InvalidOperationException( + "Parallel batch replay produced invalid committed counters."); + } + + return replay; + } + } +} diff --git a/src/CodeIndex/Cli/QueryCommandRunner.Search.cs b/src/CodeIndex/Cli/QueryCommandRunner.Search.cs index 9c346725ae..dd1c9d7d4a 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.Search.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.Search.cs @@ -1,6 +1,5 @@ using System.Text.Json; using CodeIndex.Database; -using CodeIndex.Models; namespace CodeIndex.Cli; @@ -109,957 +108,12 @@ private static int RunSearchCore( command: invocationContext.CommandName, omitNullUsage: true); } - if (options.OpenIssuesPath != null && options.OutputFormat != OutputFormatIssueDrafts) - { - WriteUsageError( - "--open-issues can only be used with `cdidx search --format issue-drafts`.", - options, - "Use an open-issues JSON file from `gh issue list --state open --json number,title,labels,url`."); - return CommandExitCodes.UsageError; - } - if (options.OpenIssuesRepository != null && !IssueDuplicatePreflight.IsGitHubOpenIssuesSource(options.OpenIssuesPath)) - { - WriteUsageError( - "--repo can only be used with `--open-issues github`.", - options, - "Use `--open-issues github --repo owner/name` to fetch open issues directly from GitHub."); - return CommandExitCodes.UsageError; - } - if (options.IssueState != IssueDuplicatePreflight.DefaultIssueState && !IssueDuplicatePreflight.IsGitHubOpenIssuesSource(options.OpenIssuesPath)) - { - WriteUsageError("--issue-state can only be used with `--open-issues github`.", options, "Use `--open-issues github --repo owner/name --issue-state all`."); - return CommandExitCodes.UsageError; - } - if (options.DuplicatePreflightTuningExplicit && options.OutputFormat != OutputFormatIssueDrafts) - { - WriteUsageError( - "--duplicate-confidence and --duplicate-threshold can only be used with `cdidx search --format issue-drafts`.", - options, - "Use these controls when exporting issue draft JSON with duplicate-preflight metadata."); - return CommandExitCodes.UsageError; - } - if ((options.IncludeRecipeQueries.Count > 0 || options.ExcludeRecipeQueries.Count > 0) && options.RecipeName == null) - { - WriteUsageError( - "--include-query and --exclude-query can only be used with --recipe.", - options, - "Use `--recipe risky-code --include-query raw-diagnostic-echo` to run a child query subset."); - return CommandExitCodes.UsageError; - } - if (options.SearchCursor.HasValue && options.RecipeName == null) - { - WriteUsageError( - "--cursor can only be used with --recipe.", - options, - "Use `--recipe risky-code/raw-diagnostic-echo --format compact --cursor ` to fetch the next page for one child query."); - return CommandExitCodes.UsageError; - } - if (options.UnusedCursorOffset.HasValue) - { - WriteUsageError( - "--cursor for search must be a search pagination cursor returned by recipe search.", - options, - "Use `--cursor ` only with `--recipe`; `unused:` cursors are for `cdidx unused`."); - return CommandExitCodes.UsageError; - } - if (options.OutlineCursorOffset.HasValue) - { - WriteUsageError( - "--cursor for search must be a search pagination cursor returned by recipe search.", - options, - "`outline:` cursors are for `cdidx outline `."); - return CommandExitCodes.UsageError; - } - if (options.DependencyCycleCursor.HasValue) - { - WriteUsageError( - "--cursor for search must be a search pagination cursor returned by recipe search.", - options, - "Dependency-cycle cursors are for `cdidx deps --cycles`."); - return CommandExitCodes.UsageError; - } - if (options.AuditScopeExplicit && options.RecipeName == null && options.ListRecipes) - { - WriteUsageError( - "--audit-scope cannot be combined with `cdidx search --list-recipes`.", - options, - "Use `--query ` with --list-recipes to filter recipe discovery, or run an ad hoc search with `--source-only`."); - return CommandExitCodes.UsageError; - } - if (options.ShowExcluded && options.RecipeName == null) - { - WriteUsageError( - "--show-excluded is only supported with `cdidx search --recipe `.", - options, - "Use it with a recipe run to include the effective scope and exclusion diagnostics in JSON output."); - return CommandExitCodes.UsageError; - } - if ((options.IssueTitle != null || options.IssueLabels.Count > 0) && options.OutputFormat != OutputFormatIssueDrafts) - { - WriteUsageError( - "--issue-title and --issue-label can only be used with `cdidx search --format issue-drafts`.", - options, - "Use these hints when exporting issue draft JSON for a plain search."); - return CommandExitCodes.UsageError; - } - if (options.SnippetLines == 0 && options.OutputFormat != OutputFormatIssueDrafts) - { - WriteUsageError( - "--snippet-lines 0 is only supported with --format issue-drafts.", - options, - "Use `--format issue-drafts --snippet-lines 0` for path/line-only draft evidence, or pass a positive snippet line count for search output."); - return CommandExitCodes.UsageError; - } - if (options.IssueTitle != null && options.RecipeName != null) - { - WriteUsageError( - "--issue-title is only supported for ad hoc search issue drafts.", - options, - "Recipe issue-drafts produce one draft per recipe query, so their titles are derived from the recipe metadata."); - return CommandExitCodes.UsageError; - } - if (options.OutputFormat == OutputFormatIssueDrafts && options.CountOnly) - { - WriteUsageError( - "--count cannot be combined with --format issue-drafts.", - options, - "Issue-draft export needs result evidence; remove --count."); - return CommandExitCodes.UsageError; - } - if (options.NamesOnly && !options.ListRecipes) - { - WriteUsageError( - "--names is only supported with `cdidx recipes` or `cdidx search --list-recipes`.", - options, - "Use `cdidx recipes --names --json` for a small deterministic recipe-name list."); - return CommandExitCodes.UsageError; - } - if (options.NamesOnly && options.SummaryOnly) - { - WriteUsageError( - "--names cannot be combined with --summary-only.", - options, - "Use one recipe-list shape at a time."); - return CommandExitCodes.UsageError; - } - if (options.SummaryOnly - && !options.ListRecipes - && options.NamedSearchQueries.Count == 0 - && !(options.RecipeName != null - && (options.CountOnly - || options.Compact - || options.OutputFormat == OutputFormatCompact - || options.OutputFormat == OutputFormatIssueDrafts))) - { - WriteUsageError( - "--summary-only is only supported with `cdidx recipes` / `cdidx search --list-recipes`, named-query count output, recipe count output, or recipe issue-drafts output.", - options, - "Use `cdidx recipes --summary-only --json`, `cdidx search --named-query = --summary-only --json`, `cdidx search --recipe --format compact --summary-only --json`, `cdidx search --recipe --format count --summary-only`, or `cdidx search --recipe --format issue-drafts --summary-only`."); - return CommandExitCodes.UsageError; - } - if (options.OutputFormat == OutputFormatIssueDrafts && options.JsonOutputFormat == JsonOutputFormatArray) - { - WriteUsageError( - "--json=array is not supported with --format issue-drafts because draft export is a JSON object.", - options, - "Use plain `--json` or omit --json when exporting issue drafts."); - return CommandExitCodes.UsageError; - } - if (options.SearchCursor.HasValue && options.OutputFormat == OutputFormatIssueDrafts) - { - WriteUsageError( - "--cursor cannot be combined with --format issue-drafts.", - options, - "Use --cursor with recipe JSON or compact output, then export issue drafts after choosing the desired query page."); - return CommandExitCodes.UsageError; - } - var exactSearch = exact || options.TokenBoundary; - if (exactSearch && options.Prefix) - { - WriteSearchValidationError( - "--prefix cannot be combined with --exact / --exact-substring / --token-boundary (exact uses instr(), not FTS5 prefix phrases).", - options, - "Drop --prefix to keep the exact substring path, or drop the exact-mode flag to opt into FTS5 prefix matching."); + if (!TryValidateSearchOptions(options, exact, invocationContext)) return CommandExitCodes.UsageError; - } - if (options.GroupBy != null && (options.ListRecipes || options.NamedSearchQueries.Count > 0)) - { - var mode = options.ListRecipes - ? "--list-recipes" - : "--named-query"; - WriteUsageError( - $"--group-by is not supported with {mode}.", - options, - "Use `cdidx search --group-by file --count` or remove --group-by for recipe-list and named-batch output."); - return CommandExitCodes.UsageError; - } - if (options.OutputFormat == OutputFormatGrouped && (options.ListRecipes || options.NamedSearchQueries.Count > 0 || options.RecipeName != null)) - { - var mode = options.ListRecipes - ? "--list-recipes" - : options.NamedSearchQueries.Count > 0 - ? "--named-query" - : "--recipe"; - WriteUsageError( - "--format grouped is only supported for plain search output.", - options, - invocationContext.RecipeNameIsPositional && mode == "--recipe" - ? "Run a plain `cdidx search --format grouped`; audit recipe execution does not support grouped output." - : $"Remove {mode}, or run a plain `cdidx search --format grouped`."); - return CommandExitCodes.UsageError; - } - if (TryWriteCappedJsonDiagnosticsUsageError(invocationContext.CommandName, options)) + if (!TryCreateSearchRoutePlan(cmdArgs, options, exact, cancellationToken, out var route)) return CommandExitCodes.UsageError; - if (options.ListRecipes) - { - if (options.FirstPerFile || options.SampleSize.HasValue) - { - WriteUsageError( - "row-selection controls are not supported with --list-recipes because recipe discovery does not emit search rows.", - options, - "Remove --first-per-file / --sample, or execute a recipe or plain search that returns rows."); - return CommandExitCodes.UsageError; - } - if (options.RecipeName != null || options.NamedSearchQueries.Count > 0 || options.ExtraNames.Count > 0) - { - WriteUsageError( - "--list-recipes cannot be combined with --recipe, --named-query, or extra positional arguments.", - options, - "Run `cdidx search --list-recipes --query ` to filter built-in audit recipes by recipe, query, label, severity, path, or search text."); - return CommandExitCodes.UsageError; - } - if (options.OutputFormat is not OutputFormatText and not OutputFormatJson and not OutputFormatCompact) - { - WriteUsageError( - "--format count/csv/tsv/lsp/qf/sarif/issue-drafts is not supported with --list-recipes.", - options, - "Use plain text output, `--json` / `--format json` for the full recipe list, or `--format compact` for a compact summary."); - return CommandExitCodes.UsageError; - } - if (options.JsonOutputFormat == JsonOutputFormatArray) - { - WriteUsageError( - "--json=array is not supported with --list-recipes because recipe-list output is a JSON object.", - options, - "Use plain `--json` for the recipe-list object."); - return CommandExitCodes.UsageError; - } - return WriteSearchRecipeList(options, jsonOptions, invocationContext.CommandName); - } - if (options.NamedSearchQueries.Count > 0) - { - if (options.FirstPerFile || options.SampleSize.HasValue) - { - WriteUsageError( - "row-selection controls are not supported with --named-query because named batches do not expose selector accounting.", - options, - "Remove --first-per-file / --sample, or run each query as a plain search or recipe row output."); - return CommandExitCodes.UsageError; - } - if (options.Query != null || options.RecipeName != null || options.ExtraNames.Count > 0) - { - WriteUsageError( - "--named-query cannot be combined with a positional query, --query, --recipe, or extra positional arguments.", - options, - "Pass one or more `--named-query =` values, or run a plain `cdidx search `."); - return CommandExitCodes.UsageError; - } - if (options.OpenIssuesPath != null) - { - WriteUsageError( - "--open-issues can only be used with `cdidx search --recipe --format issue-drafts`.", - options, - "Remove --open-issues for ad hoc named batches."); - return CommandExitCodes.UsageError; - } - if (options.OutputFormat is not OutputFormatText and not OutputFormatJson and not OutputFormatCount and not OutputFormatCompact) - { - WriteUsageError( - "--format csv/tsv/lsp/qf/sarif/issue-drafts is not supported with --named-query.", - options, - "Use plain text output, `--json`, `--format count`, or `--format compact` for grouped ad hoc results."); - return CommandExitCodes.UsageError; - } - if (options.JsonOutputFormat == JsonOutputFormatArray) - { - WriteUsageError( - "--json=array is not supported with --named-query because named batch output is grouped by query.", - options, - "Use plain `--json` for the grouped named-query object."); - return CommandExitCodes.UsageError; - } - if (options.MaxJsonBytes.HasValue && !options.Json) - { - WriteUsageError( - "--max-json-bytes is only supported with JSON search output.", - options, - "Use `--json` or `--format compact` with --named-query when bounding named batch output."); - return CommandExitCodes.UsageError; - } - - if (options.CountOnly || options.SummaryOnly) - return RunSearchNamedBatchCount(options, jsonOptions, exactSearch); - - return RunSearchNamedBatch(options, jsonOptions, exactSearch); - } - if (options.RecipeName != null) - { - if (options.TokenBoundary) - { - WriteUsageError( - "--token-boundary is only supported for ad hoc search and --named-query batches, not recipe execution.", - options, - "Run an individual query without --recipe if token-boundary filtering is required."); - return CommandExitCodes.UsageError; - } - if (options.Query != null || options.ExtraNames.Count > 0) - { - WriteUsageError( - "--recipe expands into its own curated query set and cannot be combined with a search query.", - options, - "Remove the positional query, or run a plain `cdidx search ` without --recipe."); - return CommandExitCodes.UsageError; - } - if (options.Prefix) - { - WriteUsageError( - "--prefix is not supported with --recipe because each recipe query defines its own match mode.", - options, - "Remove --prefix, or run the individual query from the recipe list yourself."); - return CommandExitCodes.UsageError; - } - if (options.OutputFormat is not OutputFormatText and not OutputFormatJson and not OutputFormatCount and not OutputFormatCompact and not OutputFormatSarif and not OutputFormatIssueDrafts) - { - WriteUsageError( - "--format csv/tsv/lsp/qf is not supported with --recipe.", - options, - "Use `--count` / `--format count` for count-only recipe output, `--json` for grouped recipe results, `--format compact` for summary-first compact JSON, `--format sarif` for audit findings, or `--format issue-drafts` for draft exports."); - return CommandExitCodes.UsageError; - } - if (options.JsonOutputFormat == JsonOutputFormatArray) - { - WriteUsageError( - "--json=array is not supported with --recipe because recipe output is grouped by query.", - options, - "Use plain `--json` for the grouped recipe object."); - return CommandExitCodes.UsageError; - } - if (options.OutputFormat == OutputFormatSarif - && (options.CountOnly - || options.SummaryOnly - || options.GroupBy != null - || options.CountBy != null - || options.UniqueBy != null - || options.ResultsOnly - || options.SearchFields != null - || options.FirstPerFile - || options.SampleSize.HasValue - || options.GroupedPerFileLimitExplicit - || (options.JsonOutputFormatExplicit && options.JsonOutputFormat == JsonOutputFormatNdjson))) - { - WriteUsageError( - "--format sarif cannot be combined with recipe count, summary, aggregation, projection, row-selection, or NDJSON controls.", - options, - "Use `--recipe --format sarif` with result filters and `--limit` / `--total-limit`, or choose the JSON/count output shape instead."); - return CommandExitCodes.UsageError; - } - if (options.GroupedPerFileLimitExplicit) - { - WriteUsageError( - "--per-file-limit is not supported with --recipe because recipe execution does not produce grouped search output.", - options, - "Use --first-per-file for one selected recipe row per file, or remove --recipe and use grouped ad hoc search output."); - return CommandExitCodes.UsageError; - } - if ((options.FirstPerFile || options.SampleSize.HasValue) - && options.SearchCursor.HasValue) - { - WriteUsageError( - "recipe row-selection controls cannot be combined with --cursor because raw recipe cursors cannot preserve selector state.", - options, - "Remove --cursor and rerun selection from the beginning, or remove --first-per-file / --sample to resume from the cursor."); - return CommandExitCodes.UsageError; - } - if ((options.FirstPerFile || options.SampleSize.HasValue) - && (options.CountOnly - || options.GroupBy != null - || options.CountBy != null - || options.UniqueBy != null - || options.ResultsOnly - || (options.SummaryOnly && (options.Compact || options.OutputFormat == OutputFormatCompact)))) - { - WriteUsageError( - "recipe row-selection controls cannot be combined with count, aggregation, results-only, or summary-only compact output.", - options, - "Remove --first-per-file / --sample to keep the non-row output, or choose text, JSON, compact, NDJSON, or issue-drafts row output."); - return CommandExitCodes.UsageError; - } - if (options.MaxJsonBytes.HasValue && !SupportsSearchJsonByteLimit(options)) - { - WriteUsageError( - "--max-json-bytes is only supported with JSON search output.", - options, - "Use `--json=ndjson`, `--format count`, `--format compact`, grouped/count-by JSON, or `--format issue-drafts` with --max-json-bytes."); - return CommandExitCodes.UsageError; - } - if (options.ResultsOnly && options.JsonOutputFormat != JsonOutputFormatNdjson) - { - WriteUsageError( - "--results-only is only supported with NDJSON recipe output.", - options, - "Use `--recipe --results-only --search-fields path,line,query_name`, or remove --results-only."); - return CommandExitCodes.UsageError; - } - if (options.GroupBy != null) - { - if (!IsSupportedSearchGroupByValue(options.GroupBy)) - { - WriteUsageError( - "--group-by for recipe search must be one of file, symbol, origin, return-type, or subsystem.", - options, - $"Use `{invocationContext.RecipeCommandPrefix} --group-by file --count`, `--group-by symbol --count`, `--group-by return-type --count`, `--group-by subsystem --count`, or `--count-by origin`."); - return CommandExitCodes.UsageError; - } - if (!options.CountOnly) - { - WriteUsageError( - $"{invocationContext.RecipeExecutionName} --group-by requires --count.", - options, - "Add --count to request grouped recipe result counts, or remove --group-by to print matching snippets."); - return CommandExitCodes.UsageError; - } - } - if (options.CountBy != null && !IsSupportedSearchAggregationValue(options.CountBy)) - { - WriteUsageError( - "--count-by for recipe search must be one of path, file, symbol, origin, return-type, or subsystem.", - options, - "Use `--count-by path`, `--count-by symbol`, `--count-by return-type`, `--count-by subsystem`, or `--count-by origin`."); - return CommandExitCodes.UsageError; - } - if (options.UniqueBy != null && !IsSupportedSearchAggregationValue(options.UniqueBy)) - { - WriteUsageError( - "--unique for recipe search must be one of path, file, symbol, origin, return-type, or subsystem.", - options, - "Use `--unique path`, `--unique symbol`, `--unique return-type`, `--unique subsystem`, or `--unique origin`."); - return CommandExitCodes.UsageError; - } - if (options.CountBy != null && options.UniqueBy != null) - { - WriteUsageError( - "--count-by cannot be combined with --unique.", - options, - "Run one recipe aggregation mode at a time."); - return CommandExitCodes.UsageError; - } - if (options.GroupBy != null && (options.CountBy != null || options.UniqueBy != null)) - { - WriteUsageError( - "--group-by cannot be combined with --count-by or --unique.", - options, - "Use either `--group-by --count`, `--count-by `, or `--unique `."); - return CommandExitCodes.UsageError; - } - if ((options.GroupBy != null || options.CountBy != null || options.UniqueBy != null) && (options.ResultsOnly || options.SearchFields != null)) - { - WriteUsageError( - "recipe aggregation cannot be combined with --results-only or --search-fields.", - options, - "Run the aggregation separately, or remove --count-by/--group-by to stream projected recipe rows."); - return CommandExitCodes.UsageError; - } - - if (options.CountOnly || (options.SummaryOnly && (options.Compact || options.OutputFormat == OutputFormatCompact))) - { - if (options.GroupBy != null || options.CountBy != null || options.UniqueBy != null) - return RunSearchRecipeAggregation(options, jsonOptions, exact); - - return RunSearchRecipeCount(options, jsonOptions, exact); - } - - if (options.CountBy != null || options.UniqueBy != null) - return RunSearchRecipeAggregation(options, jsonOptions, exact); - - if (options.OutputFormat == OutputFormatIssueDrafts) - return RunSearchRecipeIssueDrafts(options, jsonOptions, exact, cancellationToken); - - return RunSearchRecipe(options, jsonOptions, exact); - } - if (TryWriteBlankQueryError(options, "search")) - return CommandExitCodes.UsageError; - if (options.Query == null) - { - return CommandErrorWriter.WriteJsonOrHuman( - options.Json, - jsonOptions, - "search requires a query argument", - CommandExitCodes.UsageError, - BuildMissingSearchQueryHint(cmdArgs), - GetUsageLineOrThrow("search"), - CommandErrorCodes.UsageError, - category: "usage"); - } - if (options.Query.Length > QueryLimits.MaxQueryLength) - { - WriteUsageError( - QueryLimits.FormatQueryTooLongError(), - options, - "Shorten the search text or split generated input into smaller queries before running `cdidx search`."); - return CommandExitCodes.UsageError; - } - if (TryWriteUnexpectedExtraPositionals("search", options)) - return CommandExitCodes.UsageError; - if ((options.FirstPerFile || options.SampleSize.HasValue) - && (options.CountOnly - || options.GroupBy != null - || options.CountBy != null - || options.UniqueBy != null - || options.OutputFormat == OutputFormatGrouped)) - { - WriteUsageError( - "search row-selection controls cannot be combined with count or aggregation output.", - options, - "Remove --first-per-file / --sample to count the full filtered population, or choose a row output that reports selector accounting."); - return CommandExitCodes.UsageError; - } - if ((options.FirstPerFile || options.SampleSize.HasValue) && options.ResultsOnly) - { - WriteUsageError( - "search row-selection controls cannot be combined with --results-only because that stream omits selector accounting.", - options, - "Remove --results-only to retain the NDJSON terminal record, or remove --first-per-file / --sample."); - return CommandExitCodes.UsageError; - } - if ((options.FirstPerFile || options.SampleSize.HasValue) - && options.JsonOutputFormat == JsonOutputFormatArray) - { - WriteUsageError( - "search row-selection controls cannot be combined with metadata-free --json=array output.", - options, - "Add --json-envelope to retain selector accounting, use --json=ndjson / --format compact, or remove --first-per-file / --sample."); - return CommandExitCodes.UsageError; - } - if ((options.FirstPerFile || options.SampleSize.HasValue) - && options.OutputFormat is not OutputFormatText - and not OutputFormatJson - and not OutputFormatCompact - and not OutputFormatIssueDrafts) - { - WriteUsageError( - "search row-selection controls are only supported by text, JSON, compact, and issue-drafts row output.", - options, - "Choose an output shape that reports selector accounting, or remove --first-per-file / --sample."); - return CommandExitCodes.UsageError; - } - if (options.GroupBy != null) - { - if (!IsSupportedSearchGroupByValue(options.GroupBy)) - { - WriteUsageError( - "--group-by for search must be one of file, symbol, origin, return-type, or subsystem.", - options, - "Use `cdidx search --group-by file --count`, `--group-by symbol --count`, `--group-by return-type --count`, `--group-by subsystem --count`, or `--count-by origin`."); - return CommandExitCodes.UsageError; - } - if (!options.CountOnly) - { - WriteUsageError( - "search --group-by requires --count.", - options, - "Add --count to request grouped result counts, or remove --group-by to print matching snippets."); - return CommandExitCodes.UsageError; - } - if (options.OutputFormat is not OutputFormatText and not OutputFormatJson and not OutputFormatCount) - { - WriteUsageError( - "--group-by for search only supports plain count output or JSON.", - options, - "Use `--count`, optionally with `--json`, instead of compact/location formats."); - return CommandExitCodes.UsageError; - } - if (options.JsonOutputFormat == JsonOutputFormatArray) - { - WriteUsageError( - "--json=array is not supported with search --group-by because grouped count output is a JSON object.", - options, - "Use plain `--json` for the grouped-count object."); - return CommandExitCodes.UsageError; - } - } - if (options.CountBy != null && options.UniqueBy != null) - { - WriteUsageError( - "--count-by cannot be combined with --unique.", - options, - "Run one aggregation mode at a time."); - return CommandExitCodes.UsageError; - } - if (options.GroupBy != null && (options.CountBy != null || options.UniqueBy != null)) - { - WriteUsageError( - "--group-by cannot be combined with --count-by or --unique.", - options, - "Use either `--group-by --count`, `--count-by `, or `--unique `."); - return CommandExitCodes.UsageError; - } - if ((options.CountBy != null || options.UniqueBy != null) && options.JsonOutputFormat == JsonOutputFormatArray) - { - WriteUsageError( - "--json=array is not supported with search aggregation because aggregation output is a JSON object.", - options, - "Use plain `--json` for `--count-by` or `--unique` aggregation output."); - return CommandExitCodes.UsageError; - } - if (options.OutputFormat == OutputFormatGrouped && options.JsonOutputFormat == JsonOutputFormatArray) - { - WriteUsageError( - "--json=array is not supported with search --format grouped because grouped output is a JSON object.", - options, - "Use plain `--json` or omit --json when using `--format grouped`."); - return CommandExitCodes.UsageError; - } - if (options.ResultsOnly && options.JsonOutputFormat != JsonOutputFormatNdjson) - { - WriteUsageError( - "--results-only is only supported with NDJSON search output.", - options, - "Use `--results-only --json=ndjson`, or remove --results-only when using --json=array."); - return CommandExitCodes.UsageError; - } - if (options.MaxJsonBytes.HasValue && !SupportsSearchJsonByteLimit(options)) - { - WriteUsageError( - "--max-json-bytes is only supported with JSON search output.", - options, - "Use `--json=ndjson`, `--json=array`, `--format count`, `--format compact`, grouped/count-by JSON, or `--format issue-drafts` with --max-json-bytes."); - return CommandExitCodes.UsageError; - } - if (options.CountBy != null && !IsSupportedSearchAggregationValue(options.CountBy)) - { - WriteUsageError( - "--count-by for search must be one of path, file, symbol, origin, return-type, or subsystem.", - options, - "Use `--count-by path`, `--count-by symbol`, `--count-by return-type`, `--count-by subsystem`, or `--count-by origin`."); - return CommandExitCodes.UsageError; - } - if (options.UniqueBy != null && !IsSupportedSearchAggregationValue(options.UniqueBy)) - { - WriteUsageError( - "--unique for search must be one of path, file, symbol, origin, return-type, or subsystem.", - options, - "Use `--unique path`, `--unique symbol`, `--unique return-type`, `--unique subsystem`, or `--unique origin`."); - return CommandExitCodes.UsageError; - } - if (options.OutputFormat == OutputFormatIssueDrafts) - return RunSearchIssueDrafts(options, jsonOptions, exactSearch, cancellationToken); - - var exactSubstringHint = SearchQueryAdvisor.BuildExactSubstringHint(options.Query, options.RawFts, exactSearch, options.Prefix); - var ndjsonOptions = options.JsonOutputFormat == JsonOutputFormatNdjson ? GetCompactJsonOptions(jsonOptions) : jsonOptions; - string? jsonDoneTerminalLine = null; - return WithDb(options, jsonOptions, reader => - { - if (options.GroupBy != null) - { - return RunGroupedSearchCount(reader, options, jsonOptions, exactSearch, exactSubstringHint); - } - if (options.CountBy != null || options.UniqueBy != null) - { - return RunSearchAggregation(reader, options, jsonOptions, exactSearch, exactSubstringHint); - } - - if (options.CountOnly) - { - var counts = CountSearchMatches(reader, options, exactSearch); - var queryDiagnostics = DbReader.AnalyzeFtsQuery(options.Query, options.RawFts, options.Prefix, options.Lang); - if (counts.Count == 0) - { - if (options.Json) - { - return WriteJsonObjectWithOptionalByteLimit( - BuildCountJsonPayload( - reader, - jsonOptions, - count: 0, - files: 0, - query: options.Query, - queryOptions: options, - ftsQueryDiagnostics: queryDiagnostics, - exactSubstringHint: exactSubstringHint).ToJsonString(jsonOptions), - options, - "search count", - "Narrow the query or increase --max-json-bytes.", - jsonOptions); - } - else - { - Console.WriteLine("0"); - WriteExactSubstringHintIfNeeded(exactSubstringHint); - } - return CommandExitCodes.Success; - } - - if (options.Json) - { - return WriteJsonObjectWithOptionalByteLimit( - BuildCountJsonPayload( - reader, - jsonOptions, - counts.Count, - counts.FileCount, - query: options.Query, - queryOptions: options, - ftsQueryDiagnostics: queryDiagnostics, - exactSubstringHint: exactSubstringHint).ToJsonString(jsonOptions), - options, - "search count", - "Narrow the query or increase --max-json-bytes.", - jsonOptions); - } - else - { - Console.WriteLine($"{counts.Count}"); - WriteExactSubstringHintIfNeeded(exactSubstringHint); - } - return CommandExitCodes.Success; - } - - var ftsQueryDiagnostics = DbReader.AnalyzeFtsQuery(options.Query, options.RawFts, options.Prefix, options.Lang); - var groupedCounts = options.OutputFormat == OutputFormatGrouped - ? CountSearchMatches(reader, options, exactSearch) - : default; - var displayRows = ReadSearchDisplayRows(reader, options, exactSearch, out var boundedSelection); - var sarifSourceRows = displayRows; - var selection = boundedSelection ?? ApplySearchOutputSelection(displayRows, options); - displayRows = selection.Rows; - if (displayRows.Count == 0) - { - if (options.Json && (options.OutputFormat == OutputFormatCsv || options.OutputFormat == OutputFormatTsv)) - { - WriteDelimitedSearchResults([], options); - return ZeroResultExitCode(options); - } - if (options.Json && options.OutputFormat == OutputFormatGrouped) - { - var groupedExitCode = WriteGroupedSearchResults([], groupedCounts, options, jsonOptions); - return groupedExitCode == CommandExitCodes.Success ? ZeroResultExitCode(options) : groupedExitCode; - } - if (options.Json - && options.OutputFormat == OutputFormatCompact - && selection.Selectors.Count > 0) - { - var compactExitCode = WriteCompactSearchResults([], options, jsonOptions, selection); - return compactExitCode == CommandExitCodes.Success ? ZeroResultExitCode(options) : compactExitCode; - } - if (options.Json && TryWriteEmptySearchJsonWithOptionalByteLimit(options, jsonOptions, out var emptyJsonExitCode)) - return emptyJsonExitCode == CommandExitCodes.Success ? ZeroResultExitCode(options) : emptyJsonExitCode; - var emptySarifRunProperties = options.OutputFormat == OutputFormatSarif - ? BuildAdHocSearchSarifRunProperties( - options, - selection, - CountAdHocSearchSarifSourceResults(reader, options, exactSearch, sarifSourceRows), - returnedResultCount: 0) - : null; - if (options.Json && TryWriteEmptyFormattedResult(options, jsonOptions, emptySarifRunProperties)) - return ZeroResultExitCode(options); - if (options.Json) - { - if (options.JsonOutputFormat == JsonOutputFormatArray) - { - return WriteJsonObjectWithOptionalByteLimit( - JsonSerializer.Serialize( - Array.Empty(), - CliJsonSerializerContextFactory.Create(jsonOptions).CompactSearchResultArray), - options, - "search result array", - "Increase --max-json-bytes or remove the byte cap.", - jsonOptions); - } - else - { - var pathHint = BuildSearchPathGlobHint(reader, options); - if (!options.ResultsOnly) - { - var payload = BuildJsonZeroResultPayload( - reader, - ndjsonOptions, - resultsKey: "results", - query: options.Query, - ftsQueryDiagnostics: ftsQueryDiagnostics, - queryOptions: options, - exactSubstringHint: exactSubstringHint, - extraFields: payload => - { - AddSearchPathHint(payload, pathHint); - AddBareTokenSearchHint(payload, options); - }).ToJsonString(ndjsonOptions); - var stream = WriteNdjsonStream( - [new NdjsonOutputRecord(payload, CountsAsResult: false)], - totalCount: 0, - options, - ndjsonOptions, - reader, - "search", - limitTruncated: false, - "Increase --limit or narrow the query to retrieve the remaining search results.", - totalCountAuthoritative: false, - sourceTotal: selection.Selectors.Count > 0 ? selection.SourceTotal : null, - sourceTotalAuthoritative: selection.Selectors.Count > 0 ? selection.SourceTotalAuthoritative : null, - selectedTotal: selection.Selectors.Count > 0 ? selection.SelectedTotal : null, - selectorOmittedCount: selection.Selectors.Count > 0 ? selection.SelectorOmittedCount : null, - limitOmittedCount: selection.Selectors.Count > 0 ? selection.LimitOmittedCount : null, - selectors: selection.Selectors.Count > 0 ? selection.Selectors : null); - jsonDoneTerminalLine = stream.TerminalLine; - return stream.ExitCode == CommandExitCodes.Success ? ZeroResultExitCode(options) : stream.ExitCode; - } - } - } - else if (!options.Json) - { - CommandErrorWriter.WriteStderr(BuildZeroResultLine("No results found", options)); - WriteLangHint(options.Lang, reader); - WriteExactSubstringHintIfNeeded(exactSubstringHint); - var pathHint = BuildSearchPathGlobHint(reader, options); - WriteZeroResultHints(options, reader, filterHint: pathHint?.SuggestedAction); - } - return ZeroResultExitCode(options); - } - - if (options.Json) - { - var compactResults = displayRows.Select(row => row.Compact).ToArray(); - AttachExactSubstringHint(compactResults, exactSubstringHint); - AttachSearchNextSteps(compactResults, options); - if (options.SearchFields != null) - { - var projectedExitCode = WriteProjectedSearchResults( - compactResults, - selection.OriginalCount, - selection.LimitTruncated, - selection.LimitTruncated ? "limit" : null, - selection.TruncationReason is "sample" or "first_per_file" ? selection.TruncationReason : null, - selection.TruncationReason is "sample" or "first_per_file" - ? selection.SelectionOmittedCount - : null, - selection.SourceTotal, - selection.SourceTotalAuthoritative, - selection.SelectedTotal, - selection.SelectorOmittedCount, - selection.LimitOmittedCount, - selection.Selectors, - options, - jsonOptions, - ndjsonOptions, - reader, - out jsonDoneTerminalLine); - return projectedExitCode; - } - if (options.OutputFormat == OutputFormatCompact) - { - return WriteCompactSearchResults(compactResults, options, jsonOptions, selection); - } - if (options.OutputFormat == OutputFormatGrouped) - { - return WriteGroupedSearchResults(displayRows, groupedCounts, options, jsonOptions); - } - if (options.OutputFormat == OutputFormatCsv || options.OutputFormat == OutputFormatTsv) - { - WriteDelimitedSearchResults(displayRows, options); - return CommandExitCodes.Success; - } - if (TryWriteFormattedLocations( - options, - displayRows.SelectMany(row => ToSearchFormattedLocations(row, options.Query, exactSearch)).Take(options.Limit), - jsonOptions)) - return CommandExitCodes.Success; - if (options.OutputFormat == OutputFormatLsp) - { - WriteLspLocations(displayRows.SelectMany(row => ToSearchLspLocations(row, exactSearch)).Take(options.Limit), jsonOptions); - return CommandExitCodes.Success; - } - if (options.OutputFormat == OutputFormatQf) - { - WriteQuickfix(displayRows.SelectMany(row => ToSearchQuickfixItems(row, options.Query, exactSearch)).Take(options.Limit)); - return CommandExitCodes.Success; - } - if (options.OutputFormat == OutputFormatSarif) - { - var sarifItems = displayRows - .SelectMany(row => ToSearchSarifItems(row, options.Query, exactSearch)) - .Take(options.Limit) - .ToList(); - var runProperties = BuildAdHocSearchSarifRunProperties( - options, - selection, - CountAdHocSearchSarifSourceResults(reader, options, exactSearch, sarifSourceRows), - sarifItems.Count); - WriteSarif(sarifItems, jsonOptions, runProperties: runProperties); - return CommandExitCodes.Success; - } - if (options.JsonOutputFormat == JsonOutputFormatArray) - { - return WriteJsonObjectWithOptionalByteLimit( - JsonSerializer.Serialize( - compactResults, - CliJsonSerializerContextFactory.Create(jsonOptions).CompactSearchResultArray), - options, - "search result array", - "Reduce --limit, --snippet-lines, or use `--json=ndjson --max-json-bytes` for streaming output.", - jsonOptions); - } - else - { - return WriteSearchNdjsonResults( - compactResults, - selection.OriginalCount, - selection.LimitTruncated, - selection.LimitTruncated ? "limit" : null, - selection.TruncationReason is "sample" or "first_per_file" ? selection.TruncationReason : null, - selection.TruncationReason is "sample" or "first_per_file" - ? selection.SelectionOmittedCount - : null, - selection.SourceTotal, - selection.SourceTotalAuthoritative, - selection.SelectedTotal, - selection.SelectorOmittedCount, - selection.LimitOmittedCount, - selection.Selectors, - options, - ndjsonOptions, - reader, - out jsonDoneTerminalLine); - } - } - else - { - if (options.OutputFormat == OutputFormatGrouped) - { - WriteGroupedSearchResultsHuman(displayRows, options); - } - else - { - foreach (var row in displayRows) - { - var r = row.Result; - Console.WriteLine($"{r.Path}:{r.StartLine}-{r.EndLine}{FormatSearchVisibilitySuffix(r.Visibility)}"); - var snippetLines = row.Compact.Snippet.Split('\n', StringSplitOptions.None); - foreach (var line in snippetLines) - Console.WriteLine($" {line}"); - Console.WriteLine(); - } - } - var fileCount = displayRows.Select(row => row.Result.Path).Distinct().Count(); - CommandErrorWriter.WriteStderr($"({displayRows.Count} results in {fileCount} files)"); - WriteExactSubstringHintIfNeeded(exactSubstringHint); - WriteSearchNextSteps(displayRows, options); - } - return CommandExitCodes.Success; - }, exitCode => - { - if (options.Json && options.JsonOutputFormat == JsonOutputFormatNdjson && jsonDoneTerminalLine != null && !options.ResultsOnly) - Console.WriteLine(jsonDoneTerminalLine); - }); + return ExecuteSearchRoute(route); } private static List FindUnexpectedRecipePositionals(string[] args) diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchExecution.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchExecution.cs new file mode 100644 index 0000000000..f94492762c --- /dev/null +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchExecution.cs @@ -0,0 +1,168 @@ +using System.Text.Json; +using CodeIndex.Database; +using CodeIndex.Models; + +namespace CodeIndex.Cli; + +public static partial class QueryCommandRunner +{ + private sealed record SearchExecutionPlan( + QueryCommandOptions Options, + JsonSerializerOptions JsonOptions, + JsonSerializerOptions NdjsonOptions, + bool ExactSearch, + string Query, + SearchQueryHint? ExactSubstringHint); + + private sealed record SearchRowExecution( + FtsQueryDiagnostics FtsQueryDiagnostics, + QueryCountResult GroupedCounts, + List DisplayRows, + IReadOnlyList SarifSourceRows, + SearchOutputSelection Selection); + + private sealed class SearchExecutionOutcome + { + public string? JsonDoneTerminalLine { get; set; } + } + + private static SearchExecutionPlan CreateSearchExecutionPlan( + QueryCommandOptions options, + JsonSerializerOptions jsonOptions, + bool exactSearch, + string query) + { + var exactSubstringHint = SearchQueryAdvisor.BuildExactSubstringHint( + query, + options.RawFts, + exactSearch, + options.Prefix); + var ndjsonOptions = options.JsonOutputFormat == JsonOutputFormatNdjson + ? GetCompactJsonOptions(jsonOptions) + : jsonOptions; + return new SearchExecutionPlan( + options, + jsonOptions, + ndjsonOptions, + exactSearch, + query, + exactSubstringHint); + } + + private static int ExecutePlainSearch(SearchExecutionPlan plan) + { + var outcome = new SearchExecutionOutcome(); + return WithDb( + plan.Options, + plan.JsonOptions, + reader => ExecutePlainSearch(reader, plan, outcome), + _ => WritePlainSearchTerminal(plan, outcome)); + } + + private static int ExecutePlainSearch( + DbReader reader, + SearchExecutionPlan plan, + SearchExecutionOutcome outcome) + { + var options = plan.Options; + if (options.GroupBy != null) + { + return RunGroupedSearchCount( + reader, + options, + plan.JsonOptions, + plan.ExactSearch, + plan.ExactSubstringHint); + } + if (options.CountBy != null || options.UniqueBy != null) + { + return RunSearchAggregation( + reader, + options, + plan.JsonOptions, + plan.ExactSearch, + plan.ExactSubstringHint); + } + if (options.CountOnly) + return WritePlainSearchCount(reader, plan); + + var rows = PreparePlainSearchRows(reader, plan); + return rows.DisplayRows.Count == 0 + ? WriteEmptyPlainSearchResults(reader, plan, rows, outcome) + : WritePlainSearchResults(reader, plan, rows, outcome); + } + + private static int WritePlainSearchCount(DbReader reader, SearchExecutionPlan plan) + { + var options = plan.Options; + var counts = CountSearchMatches(reader, options, plan.ExactSearch); + var queryDiagnostics = DbReader.AnalyzeFtsQuery( + plan.Query, + options.RawFts, + options.Prefix, + options.Lang); + if (options.Json) + { + return WriteJsonObjectWithOptionalByteLimit( + BuildCountJsonPayload( + reader, + plan.JsonOptions, + counts.Count, + counts.FileCount, + query: plan.Query, + queryOptions: options, + ftsQueryDiagnostics: queryDiagnostics, + exactSubstringHint: plan.ExactSubstringHint).ToJsonString(plan.JsonOptions), + options, + "search count", + "Narrow the query or increase --max-json-bytes.", + plan.JsonOptions); + } + + Console.WriteLine($"{counts.Count}"); + WriteExactSubstringHintIfNeeded(plan.ExactSubstringHint); + return CommandExitCodes.Success; + } + + private static SearchRowExecution PreparePlainSearchRows( + DbReader reader, + SearchExecutionPlan plan) + { + var options = plan.Options; + var ftsQueryDiagnostics = DbReader.AnalyzeFtsQuery( + plan.Query, + options.RawFts, + options.Prefix, + options.Lang); + var groupedCounts = options.OutputFormat == OutputFormatGrouped + ? CountSearchMatches(reader, options, plan.ExactSearch) + : default; + var displayRows = ReadSearchDisplayRows( + reader, + options, + plan.ExactSearch, + out var boundedSelection); + var sarifSourceRows = displayRows; + var selection = boundedSelection ?? ApplySearchOutputSelection(displayRows, options); + return new SearchRowExecution( + ftsQueryDiagnostics, + groupedCounts, + selection.Rows, + sarifSourceRows, + selection); + } + + private static void WritePlainSearchTerminal( + SearchExecutionPlan plan, + SearchExecutionOutcome outcome) + { + var options = plan.Options; + if (options.Json + && options.JsonOutputFormat == JsonOutputFormatNdjson + && outcome.JsonDoneTerminalLine != null + && !options.ResultsOnly) + { + Console.WriteLine(outcome.JsonDoneTerminalLine); + } + } +} diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchOutput.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchOutput.cs new file mode 100644 index 0000000000..a4bca6b2fb --- /dev/null +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchOutput.cs @@ -0,0 +1,266 @@ +using System.Text.Json; +using CodeIndex.Database; +using CodeIndex.Models; + +namespace CodeIndex.Cli; + +public static partial class QueryCommandRunner +{ + private static int WriteEmptyPlainSearchResults( + DbReader reader, + SearchExecutionPlan plan, + SearchRowExecution rows, + SearchExecutionOutcome outcome) + { + var options = plan.Options; + var jsonOptions = plan.JsonOptions; + var ndjsonOptions = plan.NdjsonOptions; + var exactSearch = plan.ExactSearch; + var query = plan.Query; + var exactSubstringHint = plan.ExactSubstringHint; + var ftsQueryDiagnostics = rows.FtsQueryDiagnostics; + var groupedCounts = rows.GroupedCounts; + var displayRows = rows.DisplayRows; + var sarifSourceRows = rows.SarifSourceRows; + var selection = rows.Selection; + if (options.Json && (options.OutputFormat == OutputFormatCsv || options.OutputFormat == OutputFormatTsv)) + { + WriteDelimitedSearchResults([], options); + return ZeroResultExitCode(options); + } + if (options.Json && options.OutputFormat == OutputFormatGrouped) + { + var groupedExitCode = WriteGroupedSearchResults([], groupedCounts, options, jsonOptions); + return groupedExitCode == CommandExitCodes.Success ? ZeroResultExitCode(options) : groupedExitCode; + } + if (options.Json + && options.OutputFormat == OutputFormatCompact + && selection.Selectors.Count > 0) + { + var compactExitCode = WriteCompactSearchResults([], options, jsonOptions, selection); + return compactExitCode == CommandExitCodes.Success ? ZeroResultExitCode(options) : compactExitCode; + } + if (options.Json && TryWriteEmptySearchJsonWithOptionalByteLimit(options, jsonOptions, out var emptyJsonExitCode)) + return emptyJsonExitCode == CommandExitCodes.Success ? ZeroResultExitCode(options) : emptyJsonExitCode; + var emptySarifRunProperties = options.OutputFormat == OutputFormatSarif + ? BuildAdHocSearchSarifRunProperties( + options, + selection, + CountAdHocSearchSarifSourceResults(reader, options, exactSearch, sarifSourceRows), + returnedResultCount: 0) + : null; + if (options.Json && TryWriteEmptyFormattedResult(options, jsonOptions, emptySarifRunProperties)) + return ZeroResultExitCode(options); + if (options.Json) + { + if (options.JsonOutputFormat == JsonOutputFormatArray) + { + return WriteJsonObjectWithOptionalByteLimit( + JsonSerializer.Serialize( + Array.Empty(), + CliJsonSerializerContextFactory.Create(jsonOptions).CompactSearchResultArray), + options, + "search result array", + "Increase --max-json-bytes or remove the byte cap.", + jsonOptions); + } + else + { + var pathHint = BuildSearchPathGlobHint(reader, options); + if (!options.ResultsOnly) + { + var payload = BuildJsonZeroResultPayload( + reader, + ndjsonOptions, + resultsKey: "results", + query: query, + ftsQueryDiagnostics: ftsQueryDiagnostics, + queryOptions: options, + exactSubstringHint: exactSubstringHint, + extraFields: payload => + { + AddSearchPathHint(payload, pathHint); + AddBareTokenSearchHint(payload, options); + }).ToJsonString(ndjsonOptions); + var stream = WriteNdjsonStream( + [new NdjsonOutputRecord(payload, CountsAsResult: false)], + totalCount: 0, + options, + ndjsonOptions, + reader, + "search", + limitTruncated: false, + "Increase --limit or narrow the query to retrieve the remaining search results.", + totalCountAuthoritative: false, + sourceTotal: selection.Selectors.Count > 0 ? selection.SourceTotal : null, + sourceTotalAuthoritative: selection.Selectors.Count > 0 ? selection.SourceTotalAuthoritative : null, + selectedTotal: selection.Selectors.Count > 0 ? selection.SelectedTotal : null, + selectorOmittedCount: selection.Selectors.Count > 0 ? selection.SelectorOmittedCount : null, + limitOmittedCount: selection.Selectors.Count > 0 ? selection.LimitOmittedCount : null, + selectors: selection.Selectors.Count > 0 ? selection.Selectors : null); + outcome.JsonDoneTerminalLine = stream.TerminalLine; + return stream.ExitCode == CommandExitCodes.Success ? ZeroResultExitCode(options) : stream.ExitCode; + } + } + } + else if (!options.Json) + { + CommandErrorWriter.WriteStderr(BuildZeroResultLine("No results found", options)); + WriteLangHint(options.Lang, reader); + WriteExactSubstringHintIfNeeded(exactSubstringHint); + var pathHint = BuildSearchPathGlobHint(reader, options); + WriteZeroResultHints(options, reader, filterHint: pathHint?.SuggestedAction); + } + return ZeroResultExitCode(options); + } + + private static int WritePlainSearchResults( + DbReader reader, + SearchExecutionPlan plan, + SearchRowExecution rows, + SearchExecutionOutcome outcome) + { + var options = plan.Options; + var jsonOptions = plan.JsonOptions; + var ndjsonOptions = plan.NdjsonOptions; + var exactSearch = plan.ExactSearch; + var query = plan.Query; + var exactSubstringHint = plan.ExactSubstringHint; + var groupedCounts = rows.GroupedCounts; + var displayRows = rows.DisplayRows; + var sarifSourceRows = rows.SarifSourceRows; + var selection = rows.Selection; + if (options.Json) + { + var compactResults = displayRows.Select(row => row.Compact).ToArray(); + AttachExactSubstringHint(compactResults, exactSubstringHint); + AttachSearchNextSteps(compactResults, options); + if (options.SearchFields != null) + { + var projectedExitCode = WriteProjectedSearchResults( + compactResults, + selection.OriginalCount, + selection.LimitTruncated, + selection.LimitTruncated ? "limit" : null, + selection.TruncationReason is "sample" or "first_per_file" ? selection.TruncationReason : null, + selection.TruncationReason is "sample" or "first_per_file" + ? selection.SelectionOmittedCount + : null, + selection.SourceTotal, + selection.SourceTotalAuthoritative, + selection.SelectedTotal, + selection.SelectorOmittedCount, + selection.LimitOmittedCount, + selection.Selectors, + options, + jsonOptions, + ndjsonOptions, + reader, + out var projectedTerminalLine); + outcome.JsonDoneTerminalLine = projectedTerminalLine; + return projectedExitCode; + } + if (options.OutputFormat == OutputFormatCompact) + { + return WriteCompactSearchResults(compactResults, options, jsonOptions, selection); + } + if (options.OutputFormat == OutputFormatGrouped) + { + return WriteGroupedSearchResults(displayRows, groupedCounts, options, jsonOptions); + } + if (options.OutputFormat == OutputFormatCsv || options.OutputFormat == OutputFormatTsv) + { + WriteDelimitedSearchResults(displayRows, options); + return CommandExitCodes.Success; + } + if (TryWriteFormattedLocations( + options, + displayRows.SelectMany(row => ToSearchFormattedLocations(row, query, exactSearch)).Take(options.Limit), + jsonOptions)) + return CommandExitCodes.Success; + if (options.OutputFormat == OutputFormatLsp) + { + WriteLspLocations(displayRows.SelectMany(row => ToSearchLspLocations(row, exactSearch)).Take(options.Limit), jsonOptions); + return CommandExitCodes.Success; + } + if (options.OutputFormat == OutputFormatQf) + { + WriteQuickfix(displayRows.SelectMany(row => ToSearchQuickfixItems(row, query, exactSearch)).Take(options.Limit)); + return CommandExitCodes.Success; + } + if (options.OutputFormat == OutputFormatSarif) + { + var sarifItems = displayRows + .SelectMany(row => ToSearchSarifItems(row, query, exactSearch)) + .Take(options.Limit) + .ToList(); + var runProperties = BuildAdHocSearchSarifRunProperties( + options, + selection, + CountAdHocSearchSarifSourceResults(reader, options, exactSearch, sarifSourceRows), + sarifItems.Count); + WriteSarif(sarifItems, jsonOptions, runProperties: runProperties); + return CommandExitCodes.Success; + } + if (options.JsonOutputFormat == JsonOutputFormatArray) + { + return WriteJsonObjectWithOptionalByteLimit( + JsonSerializer.Serialize( + compactResults, + CliJsonSerializerContextFactory.Create(jsonOptions).CompactSearchResultArray), + options, + "search result array", + "Reduce --limit, --snippet-lines, or use `--json=ndjson --max-json-bytes` for streaming output.", + jsonOptions); + } + else + { + var ndjsonExitCode = WriteSearchNdjsonResults( + compactResults, + selection.OriginalCount, + selection.LimitTruncated, + selection.LimitTruncated ? "limit" : null, + selection.TruncationReason is "sample" or "first_per_file" ? selection.TruncationReason : null, + selection.TruncationReason is "sample" or "first_per_file" + ? selection.SelectionOmittedCount + : null, + selection.SourceTotal, + selection.SourceTotalAuthoritative, + selection.SelectedTotal, + selection.SelectorOmittedCount, + selection.LimitOmittedCount, + selection.Selectors, + options, + ndjsonOptions, + reader, + out var ndjsonTerminalLine); + outcome.JsonDoneTerminalLine = ndjsonTerminalLine; + return ndjsonExitCode; + } + } + else + { + if (options.OutputFormat == OutputFormatGrouped) + { + WriteGroupedSearchResultsHuman(displayRows, options); + } + else + { + foreach (var row in displayRows) + { + var r = row.Result; + Console.WriteLine($"{r.Path}:{r.StartLine}-{r.EndLine}{FormatSearchVisibilitySuffix(r.Visibility)}"); + var snippetLines = row.Compact.Snippet.Split('\n', StringSplitOptions.None); + foreach (var line in snippetLines) + Console.WriteLine($" {line}"); + Console.WriteLine(); + } + } + var fileCount = displayRows.Select(row => row.Result.Path).Distinct().Count(); + CommandErrorWriter.WriteStderr($"({displayRows.Count} results in {fileCount} files)"); + WriteExactSubstringHintIfNeeded(exactSubstringHint); + WriteSearchNextSteps(displayRows, options); + } + return CommandExitCodes.Success; + } +} diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipeExecution.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipeExecution.cs new file mode 100644 index 0000000000..ff3f98e043 --- /dev/null +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipeExecution.cs @@ -0,0 +1,488 @@ +using CodeIndex.Database; +using CodeIndex.Models; + +namespace CodeIndex.Cli; + +public static partial class QueryCommandRunner +{ + private readonly record struct SearchRecipeQueryMaterializationRequest( + DbReader Reader, + SearchRecipeScopeJsonResult Scope, + QueryCommandOptions Options, + SearchAuditRecipeQuery RecipeQuery, + bool Exact, + int? ResultLimit, + bool? RawFtsOverride); + + private readonly record struct SearchRecipeQueryMaterializationResult( + List Rows, + bool SourceTotalAuthoritative); + + private static SearchRecipeQueryMaterializationResult MaterializeSearchRecipeQuery( + in SearchRecipeQueryMaterializationRequest request) + { + var queryScope = BuildSearchRecipeQueryScope(request.Scope, request.RecipeQuery); + var guardFilters = BuildSearchRecipeGuardFilters(request.Options, request.RecipeQuery); + var fetchLimit = request.ResultLimit.HasValue + ? GetSearchRecipeFetchLimit(request.Options, request.ResultLimit.Value, request.RecipeQuery) + : int.MaxValue; + var results = request.Reader.Search( + request.RecipeQuery.Query, + fetchLimit, + request.Options.Lang, + false, + queryScope.PathPatterns, + queryScope.ExcludePaths, + queryScope.ExcludeTests, + !request.Options.NoDedup, + request.Options.Since, + request.Exact, + false, + !request.Options.NoVisibilityRank, + cursor: request.Options.SearchCursor, + guardFilters: guardFilters, + guardWindow: request.Options.GuardWindow, + guardScope: request.Options.GuardScope, + requiredPathPatterns: GetSearchRecipeRequiredPathPatterns(request.Options, request.RecipeQuery), + resultRanking: request.ResultLimit.HasValue + ? GetSearchRecipeResultRanking(request.RecipeQuery.ResultRanking, request.ResultLimit.Value) + : SearchResultRanking.Default); + var sourceTotalAuthoritative = request.ResultLimit.HasValue + && IsSearchRecipeSourceTotalAuthoritative( + request.Options, + request.RecipeQuery, + guardFilters, + results.Count, + fetchLimit); + results = ApplySearchRecipeFileRejectQueries( + request.Reader, + results, + request.Options, + request.RecipeQuery); + var rows = BuildSearchDisplayRows( + results, + request.Options, + request.Exact, + request.RecipeQuery.Query, + rawFtsOverride: request.RawFtsOverride, + recipeQuery: request.RecipeQuery); + rows = ApplySearchRecipeSemanticFilter( + request.Reader, + request.Options, + request.RecipeQuery, + rows); + return new SearchRecipeQueryMaterializationResult(rows, sourceTotalAuthoritative); + } + + private static List CollectSearchRecipeQueryResults( + DbReader reader, + IReadOnlyList recipeQueries, + SearchRecipeScopeJsonResult scope, + QueryCommandOptions options, + bool userExact, + SearchQueryFreshnessContext? freshnessContext, + bool includeAuditClassifications, + out int total, + out int minimumMatchedTotal, + out List freshnessObservations, + out bool hasFailures) + { + var queryResults = new List(); + freshnessObservations = []; + total = 0; + minimumMatchedTotal = 0; + hasFailures = false; + foreach (var recipeQuery in recipeQueries) + { + try + { + var exact = userExact || recipeQuery.ExactSubstring; + var resultLimit = GetSearchRecipeEffectiveResultLimit(options, total); + var materializationRequest = new SearchRecipeQueryMaterializationRequest( + reader, + scope, + options, + recipeQuery, + exact, + resultLimit, + RawFtsOverride: false); + var materialization = MaterializeSearchRecipeQuery(in materializationRequest); + var rows = materialization.Rows; + var summaryEvidencePaths = BuildSearchRecipeTopFiles(rows); + var summaryEvidencePathCount = rows + .Select(row => row.Result.Path) + .Where(path => !string.IsNullOrWhiteSpace(path)) + .Distinct(StringComparer.Ordinal) + .Count(); + var outputSelection = ApplySearchOutputSelection( + rows, + options, + resultLimit, + materialization.SourceTotalAuthoritative); + rows = outputSelection.Rows; + if (includeAuditClassifications) + ApplySearchRecipeAuditClassifications(reader, recipeQuery, recipeQueries, rows); + var minimumOmitted = Math.Max(0, outputSelection.OriginalCount - rows.Count); + var selectionReason = GetSearchRecipeSelectionReason(outputSelection); + total += rows.Count; + minimumMatchedTotal += outputSelection.OriginalCount; + queryResults.Add(new SearchRecipeQueryResultJsonResult( + recipeQuery.Name, + recipeQuery.Query, + recipeQuery.Description, + recipeQuery.RecommendedLabels, + recipeQuery.FalsePositiveGuidance, + [.. recipeQuery.RiskEvidence], + ToSearchRecipeGuardFilterJsonResults(recipeQuery.GuardFilters), + exact, + recipeQuery.Severity, + [.. recipeQuery.PathPatterns], + [.. recipeQuery.ExcludePaths], + [.. recipeQuery.MatchOrigins], + [.. recipeQuery.ExcludeOrigins], + [.. recipeQuery.ResultKinds], + [.. recipeQuery.Classifiers], + recipeQuery.StringComparisonTaxonomy, + recipeQuery.BroadCatchTaxonomy, + recipeQuery.NullableContractTaxonomy, + BuildSearchRecipeClassifierCounts(rows), + rows.Count, + rows.Count, + outputSelection.OriginalCount, + minimumOmitted, + selectionReason, + selectionReason != null ? outputSelection.SelectionOmittedCount : null, + resultLimit, + minimumOmitted, + BuildSearchRecipeTopFiles(rows), + outputSelection.LimitTruncated, + outputSelection.LimitTruncated + && !options.FirstPerFile + && !options.SampleSize.HasValue + && rows.Count > 0 + ? FormatSearchCursor(rows[^1].Result) + : null, + rows.Select(row => row.Compact).ToList(), + outputSelection.SourceTotal, + outputSelection.SourceTotalAuthoritative, + outputSelection.SourceTotalAuthoritative ? null : outputSelection.SourceTotal, + outputSelection.SelectedTotal, + outputSelection.Returned, + outputSelection.SelectorOmittedCount, + outputSelection.LimitOmittedCount, + outputSelection.Selectors) + { + SummaryEvidencePaths = summaryEvidencePaths, + SummaryEvidencePathCount = summaryEvidencePathCount, + SummaryEvidencePathCountAuthoritative = materialization.SourceTotalAuthoritative, + }); + if (freshnessContext != null) + { + freshnessObservations.Add(SuccessfulSearchQueryObservation( + freshnessContext, + recipeQuery.Name, + outputSelection.OriginalCount)); + } + } + catch (Exception ex) when ( + freshnessContext != null + && TryClassifySearchQueryExecutionFailure(ex, out _)) + { + TryClassifySearchQueryExecutionFailure(ex, out var failureReason); + hasFailures = true; + freshnessObservations.Add(FailedSearchQueryObservation( + freshnessContext, + recipeQuery.Name, + failureReason)); + } + } + + return queryResults; + } + + private static List CollectSearchRecipeCompactQueryResults( + DbReader reader, + IReadOnlyList recipeQueries, + SearchRecipeScopeJsonResult scope, + QueryCommandOptions options, + bool userExact, + SearchQueryFreshnessContext freshnessContext, + out int total, + out List freshnessObservations, + out bool hasFailures) + { + var queryResults = new List(); + freshnessObservations = []; + total = 0; + hasFailures = false; + foreach (var recipeQuery in recipeQueries) + { + try + { + var exact = userExact || recipeQuery.ExactSubstring; + var resultLimit = GetSearchRecipeEffectiveResultLimit(options, total); + var materializationRequest = new SearchRecipeQueryMaterializationRequest( + reader, + scope, + options, + recipeQuery, + exact, + resultLimit, + RawFtsOverride: null); + var materialization = MaterializeSearchRecipeQuery(in materializationRequest); + var rows = materialization.Rows; + var outputSelection = ApplySearchOutputSelection( + rows, + options, + resultLimit, + materialization.SourceTotalAuthoritative); + rows = outputSelection.Rows; + if (!options.SummaryOnly) + ApplySearchRecipeAuditClassifications(reader, recipeQuery, recipeQueries, rows); + var minimumOmitted = Math.Max(0, outputSelection.OriginalCount - rows.Count); + var selectionReason = GetSearchRecipeSelectionReason(outputSelection); + total += rows.Count; + queryResults.Add(new SearchRecipeCompactQueryResultJsonResult( + recipeQuery.Name, + recipeQuery.Query, + recipeQuery.Description, + recipeQuery.Severity, + [.. recipeQuery.RiskEvidence], + ToSearchRecipeGuardFilterJsonResults(recipeQuery.GuardFilters), + [.. recipeQuery.PathPatterns], + [.. recipeQuery.ExcludePaths], + [.. recipeQuery.MatchOrigins], + [.. recipeQuery.ExcludeOrigins], + [.. recipeQuery.ResultKinds], + [.. recipeQuery.Classifiers], + recipeQuery.StringComparisonTaxonomy, + recipeQuery.BroadCatchTaxonomy, + BuildSearchRecipeClassifierCounts(rows), + rows.Count, + rows.Count, + outputSelection.OriginalCount, + minimumOmitted, + selectionReason, + selectionReason != null ? outputSelection.SelectionOmittedCount : null, + resultLimit, + minimumOmitted, + BuildSearchRecipeTopFiles(rows), + outputSelection.LimitTruncated, + outputSelection.LimitTruncated + && !options.FirstPerFile + && !options.SampleSize.HasValue + && rows.Count > 0 + ? FormatSearchCursor(rows[^1].Result) + : null, + rows.Select(row => new SearchRecipeCompactResultJsonResult( + row.Result.Path, + row.Result.Lang, + row.Result.Visibility, + [.. recipeQuery.RiskEvidence], + row.Result.StartLine, + row.Result.EndLine, + row.Compact.MatchLines, + row.Compact.EnclosingSymbolName, + row.Compact.EnclosingSymbolKind)).ToList(), + outputSelection.SourceTotal, + outputSelection.SourceTotalAuthoritative, + outputSelection.SourceTotalAuthoritative ? null : outputSelection.SourceTotal, + outputSelection.SelectedTotal, + outputSelection.Returned, + outputSelection.SelectorOmittedCount, + outputSelection.LimitOmittedCount, + outputSelection.Selectors)); + freshnessObservations.Add(SuccessfulSearchQueryObservation( + freshnessContext, + recipeQuery.Name, + outputSelection.OriginalCount)); + } + catch (Exception ex) when (TryClassifySearchQueryExecutionFailure(ex, out _)) + { + TryClassifySearchQueryExecutionFailure(ex, out var failureReason); + hasFailures = true; + freshnessObservations.Add(FailedSearchQueryObservation( + freshnessContext, + recipeQuery.Name, + failureReason)); + } + } + + return queryResults; + } + + private static string? GetSearchRecipeSelectionReason(SearchOutputSelection selection) + => selection.SelectionOmittedCount > 0 + && selection.TruncationReason is "first_per_file" or "sample" + ? selection.TruncationReason + : null; + + private static bool IsSearchRecipeSourceTotalAuthoritative( + QueryCommandOptions options, + SearchAuditRecipeQuery recipeQuery, + IReadOnlyCollection guardFilters, + int resultCount, + int fetchLimit) + => guardFilters.Count == 0 + && recipeQuery.RejectFileQueries.Count == 0 + && recipeQuery.SemanticFilter == SearchRecipeSemanticFilter.None + && !HasSearchOriginFilters(BuildSearchDisplayFacetFilters(options, recipeQuery)) + && resultCount < fetchLimit; + + private static int GetSearchRecipeFetchLimit( + QueryCommandOptions options, + int resultLimit, + SearchAuditRecipeQuery? recipeQuery = null) + { + if (recipeQuery is { SemanticFilter: not SearchRecipeSemanticFilter.None }) + return int.MaxValue; + + var selectionTarget = resultLimit > 0 && options.SampleSize.HasValue + ? Math.Max(resultLimit, options.SampleSize.Value) + : resultLimit; + return FetchLimitForSearchEnvelope(selectionTarget); + } + + private static List CountSearchRecipeQueryResults( + DbReader reader, + IReadOnlyList recipeQueries, + SearchRecipeScopeJsonResult scope, + QueryCommandOptions options, + bool userExact, + SearchQueryFreshnessContext? freshnessContext, + out int total, + out int fileCount, + out List freshnessObservations, + out bool hasFailures) + { + var queryCounts = new List(); + freshnessObservations = []; + var paths = new HashSet(StringComparer.Ordinal); + total = 0; + hasFailures = false; + foreach (var recipeQuery in recipeQueries) + { + try + { + var exact = userExact || recipeQuery.ExactSubstring; + var materializationRequest = new SearchRecipeQueryMaterializationRequest( + reader, + scope, + options, + recipeQuery, + exact, + ResultLimit: null, + RawFtsOverride: false); + var materialization = MaterializeSearchRecipeQuery(in materializationRequest); + var rows = materialization.Rows; + if (options.Json && !options.SummaryOnly) + ApplySearchRecipeAuditClassifications(reader, recipeQuery, recipeQueries, rows); + var count = rows.Count; + var fileCountForQuery = rows.Select(row => row.Result.Path).Distinct(StringComparer.Ordinal).Count(); + foreach (var path in rows.Select(row => row.Result.Path)) + paths.Add(path); + + total += count; + queryCounts.Add(new SearchRecipeCountQueryJsonResult( + recipeQuery.Name, + recipeQuery.Query, + recipeQuery.Description, + recipeQuery.Severity, + count, + count, + 0, + count, + fileCountForQuery, + false, + BuildSearchRecipeClassifierCounts(rows), + BuildSearchRecipeTopFiles(rows))); + if (freshnessContext != null) + { + freshnessObservations.Add(SuccessfulSearchQueryObservation( + freshnessContext, + recipeQuery.Name, + count)); + } + } + catch (Exception ex) when ( + freshnessContext != null + && TryClassifySearchQueryExecutionFailure(ex, out _)) + { + TryClassifySearchQueryExecutionFailure(ex, out var failureReason); + hasFailures = true; + freshnessObservations.Add(FailedSearchQueryObservation( + freshnessContext, + recipeQuery.Name, + failureReason)); + } + } + + fileCount = paths.Count; + return queryCounts; + } + + private static List CollectSearchRecipeAggregationResults( + DbReader reader, + IReadOnlyList recipeQueries, + SearchRecipeScopeJsonResult scope, + QueryCommandOptions options, + bool userExact, + string groupBy, + out int total, + out int fileCount) + { + var queryResults = new List(); + var paths = new HashSet(StringComparer.Ordinal); + total = 0; + foreach (var recipeQuery in recipeQueries) + { + var exact = userExact || recipeQuery.ExactSubstring; + var materializationRequest = new SearchRecipeQueryMaterializationRequest( + reader, + scope, + options, + recipeQuery, + exact, + ResultLimit: null, + RawFtsOverride: false); + var materialization = MaterializeSearchRecipeQuery(in materializationRequest); + var rows = materialization.Rows; + foreach (var path in rows.Select(row => row.Result.Path)) + paths.Add(path); + + var groups = BuildSearchGroupedCounts(groupBy, rows); + var selection = ApplySearchGroupOutputSelection(groups, options); + total += rows.Count; + queryResults.Add(new SearchRecipeAggregationQueryJsonResult( + recipeQuery.Name, + recipeQuery.Query, + recipeQuery.Description, + recipeQuery.Severity, + rows.Count, + rows.Select(row => row.Result.Path).Distinct(StringComparer.Ordinal).Count(), + selection.Groups.Count, + selection.TotalGroups, + selection.Truncated, + options.Limit, + selection.Groups)); + } + + fileCount = paths.Count; + return queryResults; + } + + private static IReadOnlyList BuildSearchRecipeGuardFilters( + QueryCommandOptions options, + SearchAuditRecipeQuery recipeQuery) + { + if (recipeQuery.GuardFilters.Count == 0) + return options.GuardFilters; + if (options.GuardFilters.Count == 0) + return recipeQuery.GuardFilters; + + var guardFilters = new List(recipeQuery.GuardFilters.Count + options.GuardFilters.Count); + guardFilters.AddRange(recipeQuery.GuardFilters); + guardFilters.AddRange(options.GuardFilters); + return guardFilters; + } +} diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs index 5bdd0252cd..0e20add79f 100644 --- a/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchRecipes.cs @@ -2273,400 +2273,6 @@ private static int GetAdHocIssueDraftResultLimit(QueryCommandOptions options) ? Math.Min(options.Limit, options.TotalLimit.Value) : options.Limit; - private static List CollectSearchRecipeQueryResults( - DbReader reader, - IReadOnlyList recipeQueries, - SearchRecipeScopeJsonResult scope, - QueryCommandOptions options, - bool userExact, - SearchQueryFreshnessContext? freshnessContext, - bool includeAuditClassifications, - out int total, - out int minimumMatchedTotal, - out List freshnessObservations, - out bool hasFailures) - { - var queryResults = new List(); - freshnessObservations = []; - total = 0; - minimumMatchedTotal = 0; - hasFailures = false; - foreach (var recipeQuery in recipeQueries) - { - try - { - var exact = userExact || recipeQuery.ExactSubstring; - var queryScope = BuildSearchRecipeQueryScope(scope, recipeQuery); - var resultLimit = GetSearchRecipeEffectiveResultLimit(options, total); - var guardFilters = BuildSearchRecipeGuardFilters(options, recipeQuery); - var fetchLimit = GetSearchRecipeFetchLimit(options, resultLimit, recipeQuery); - var results = reader.Search( - recipeQuery.Query, - fetchLimit, - options.Lang, - false, - queryScope.PathPatterns, - queryScope.ExcludePaths, - queryScope.ExcludeTests, - !options.NoDedup, - options.Since, - exact, - false, - !options.NoVisibilityRank, - cursor: options.SearchCursor, - guardFilters: guardFilters, - guardWindow: options.GuardWindow, - guardScope: options.GuardScope, - requiredPathPatterns: GetSearchRecipeRequiredPathPatterns(options, recipeQuery), - resultRanking: GetSearchRecipeResultRanking(recipeQuery.ResultRanking, resultLimit)); - var sourceTotalAuthoritative = IsSearchRecipeSourceTotalAuthoritative( - options, - recipeQuery, - guardFilters, - results.Count, - fetchLimit); - results = ApplySearchRecipeFileRejectQueries(reader, results, options, recipeQuery); - var rows = BuildSearchDisplayRows(results, options, exact, recipeQuery.Query, rawFtsOverride: false, recipeQuery: recipeQuery); - rows = ApplySearchRecipeSemanticFilter(reader, options, recipeQuery, rows); - var summaryEvidencePaths = BuildSearchRecipeTopFiles(rows); - var summaryEvidencePathCount = rows - .Select(row => row.Result.Path) - .Where(path => !string.IsNullOrWhiteSpace(path)) - .Distinct(StringComparer.Ordinal) - .Count(); - var outputSelection = ApplySearchOutputSelection(rows, options, resultLimit, sourceTotalAuthoritative); - rows = outputSelection.Rows; - if (includeAuditClassifications) - ApplySearchRecipeAuditClassifications(reader, recipeQuery, recipeQueries, rows); - var minimumOmitted = Math.Max(0, outputSelection.OriginalCount - rows.Count); - var selectionReason = GetSearchRecipeSelectionReason(outputSelection); - total += rows.Count; - minimumMatchedTotal += outputSelection.OriginalCount; - queryResults.Add(new SearchRecipeQueryResultJsonResult( - recipeQuery.Name, - recipeQuery.Query, - recipeQuery.Description, - recipeQuery.RecommendedLabels, - recipeQuery.FalsePositiveGuidance, - [.. recipeQuery.RiskEvidence], - ToSearchRecipeGuardFilterJsonResults(recipeQuery.GuardFilters), - exact, - recipeQuery.Severity, - [.. recipeQuery.PathPatterns], - [.. recipeQuery.ExcludePaths], - [.. recipeQuery.MatchOrigins], - [.. recipeQuery.ExcludeOrigins], - [.. recipeQuery.ResultKinds], - [.. recipeQuery.Classifiers], - recipeQuery.StringComparisonTaxonomy, - recipeQuery.BroadCatchTaxonomy, - recipeQuery.NullableContractTaxonomy, - BuildSearchRecipeClassifierCounts(rows), - rows.Count, - rows.Count, - outputSelection.OriginalCount, - minimumOmitted, - selectionReason, - selectionReason != null ? outputSelection.SelectionOmittedCount : null, - resultLimit, - minimumOmitted, - BuildSearchRecipeTopFiles(rows), - outputSelection.LimitTruncated, - outputSelection.LimitTruncated - && !options.FirstPerFile - && !options.SampleSize.HasValue - && rows.Count > 0 - ? FormatSearchCursor(rows[^1].Result) - : null, - rows.Select(row => row.Compact).ToList(), - outputSelection.SourceTotal, - outputSelection.SourceTotalAuthoritative, - outputSelection.SourceTotalAuthoritative ? null : outputSelection.SourceTotal, - outputSelection.SelectedTotal, - outputSelection.Returned, - outputSelection.SelectorOmittedCount, - outputSelection.LimitOmittedCount, - outputSelection.Selectors) - { - SummaryEvidencePaths = summaryEvidencePaths, - SummaryEvidencePathCount = summaryEvidencePathCount, - SummaryEvidencePathCountAuthoritative = sourceTotalAuthoritative, - }); - if (freshnessContext != null) - { - freshnessObservations.Add(SuccessfulSearchQueryObservation( - freshnessContext, - recipeQuery.Name, - outputSelection.OriginalCount)); - } - } - catch (Exception ex) when ( - freshnessContext != null - && TryClassifySearchQueryExecutionFailure(ex, out _)) - { - TryClassifySearchQueryExecutionFailure(ex, out var failureReason); - hasFailures = true; - freshnessObservations.Add(FailedSearchQueryObservation( - freshnessContext, - recipeQuery.Name, - failureReason)); - } - } - - return queryResults; - } - - private static List CollectSearchRecipeCompactQueryResults( - DbReader reader, - IReadOnlyList recipeQueries, - SearchRecipeScopeJsonResult scope, - QueryCommandOptions options, - bool userExact, - SearchQueryFreshnessContext freshnessContext, - out int total, - out List freshnessObservations, - out bool hasFailures) - { - var queryResults = new List(); - freshnessObservations = []; - total = 0; - hasFailures = false; - foreach (var recipeQuery in recipeQueries) - { - try - { - var exact = userExact || recipeQuery.ExactSubstring; - var queryScope = BuildSearchRecipeQueryScope(scope, recipeQuery); - var resultLimit = GetSearchRecipeEffectiveResultLimit(options, total); - var guardFilters = BuildSearchRecipeGuardFilters(options, recipeQuery); - var fetchLimit = GetSearchRecipeFetchLimit(options, resultLimit, recipeQuery); - var results = reader.Search( - recipeQuery.Query, - fetchLimit, - options.Lang, - false, - queryScope.PathPatterns, - queryScope.ExcludePaths, - queryScope.ExcludeTests, - !options.NoDedup, - options.Since, - exact, - false, - !options.NoVisibilityRank, - cursor: options.SearchCursor, - guardFilters: guardFilters, - guardWindow: options.GuardWindow, - guardScope: options.GuardScope, - requiredPathPatterns: GetSearchRecipeRequiredPathPatterns(options, recipeQuery), - resultRanking: GetSearchRecipeResultRanking(recipeQuery.ResultRanking, resultLimit)); - var sourceTotalAuthoritative = IsSearchRecipeSourceTotalAuthoritative( - options, - recipeQuery, - guardFilters, - results.Count, - fetchLimit); - results = ApplySearchRecipeFileRejectQueries(reader, results, options, recipeQuery); - var rows = BuildSearchDisplayRows(results, options, exact, recipeQuery.Query, recipeQuery: recipeQuery); - rows = ApplySearchRecipeSemanticFilter(reader, options, recipeQuery, rows); - var outputSelection = ApplySearchOutputSelection(rows, options, resultLimit, sourceTotalAuthoritative); - rows = outputSelection.Rows; - if (!options.SummaryOnly) - ApplySearchRecipeAuditClassifications(reader, recipeQuery, recipeQueries, rows); - var minimumOmitted = Math.Max(0, outputSelection.OriginalCount - rows.Count); - var selectionReason = GetSearchRecipeSelectionReason(outputSelection); - total += rows.Count; - queryResults.Add(new SearchRecipeCompactQueryResultJsonResult( - recipeQuery.Name, - recipeQuery.Query, - recipeQuery.Description, - recipeQuery.Severity, - [.. recipeQuery.RiskEvidence], - ToSearchRecipeGuardFilterJsonResults(recipeQuery.GuardFilters), - [.. recipeQuery.PathPatterns], - [.. recipeQuery.ExcludePaths], - [.. recipeQuery.MatchOrigins], - [.. recipeQuery.ExcludeOrigins], - [.. recipeQuery.ResultKinds], - [.. recipeQuery.Classifiers], - recipeQuery.StringComparisonTaxonomy, - recipeQuery.BroadCatchTaxonomy, - BuildSearchRecipeClassifierCounts(rows), - rows.Count, - rows.Count, - outputSelection.OriginalCount, - minimumOmitted, - selectionReason, - selectionReason != null ? outputSelection.SelectionOmittedCount : null, - resultLimit, - minimumOmitted, - BuildSearchRecipeTopFiles(rows), - outputSelection.LimitTruncated, - outputSelection.LimitTruncated - && !options.FirstPerFile - && !options.SampleSize.HasValue - && rows.Count > 0 - ? FormatSearchCursor(rows[^1].Result) - : null, - rows.Select(row => new SearchRecipeCompactResultJsonResult( - row.Result.Path, - row.Result.Lang, - row.Result.Visibility, - [.. recipeQuery.RiskEvidence], - row.Result.StartLine, - row.Result.EndLine, - row.Compact.MatchLines, - row.Compact.EnclosingSymbolName, - row.Compact.EnclosingSymbolKind)).ToList(), - outputSelection.SourceTotal, - outputSelection.SourceTotalAuthoritative, - outputSelection.SourceTotalAuthoritative ? null : outputSelection.SourceTotal, - outputSelection.SelectedTotal, - outputSelection.Returned, - outputSelection.SelectorOmittedCount, - outputSelection.LimitOmittedCount, - outputSelection.Selectors)); - freshnessObservations.Add(SuccessfulSearchQueryObservation( - freshnessContext, - recipeQuery.Name, - outputSelection.OriginalCount)); - } - catch (Exception ex) when (TryClassifySearchQueryExecutionFailure(ex, out _)) - { - TryClassifySearchQueryExecutionFailure(ex, out var failureReason); - hasFailures = true; - freshnessObservations.Add(FailedSearchQueryObservation( - freshnessContext, - recipeQuery.Name, - failureReason)); - } - } - - return queryResults; - } - - private static string? GetSearchRecipeSelectionReason(SearchOutputSelection selection) - => selection.SelectionOmittedCount > 0 - && selection.TruncationReason is "first_per_file" or "sample" - ? selection.TruncationReason - : null; - - private static bool IsSearchRecipeSourceTotalAuthoritative( - QueryCommandOptions options, - SearchAuditRecipeQuery recipeQuery, - IReadOnlyCollection guardFilters, - int resultCount, - int fetchLimit) - => guardFilters.Count == 0 - && recipeQuery.RejectFileQueries.Count == 0 - && recipeQuery.SemanticFilter == SearchRecipeSemanticFilter.None - && !HasSearchOriginFilters(BuildSearchDisplayFacetFilters(options, recipeQuery)) - && resultCount < fetchLimit; - - private static int GetSearchRecipeFetchLimit( - QueryCommandOptions options, - int resultLimit, - SearchAuditRecipeQuery? recipeQuery = null) - { - if (recipeQuery is { SemanticFilter: not SearchRecipeSemanticFilter.None }) - return int.MaxValue; - - var selectionTarget = resultLimit > 0 && options.SampleSize.HasValue - ? Math.Max(resultLimit, options.SampleSize.Value) - : resultLimit; - return FetchLimitForSearchEnvelope(selectionTarget); - } - - private static List CountSearchRecipeQueryResults( - DbReader reader, - IReadOnlyList recipeQueries, - SearchRecipeScopeJsonResult scope, - QueryCommandOptions options, - bool userExact, - SearchQueryFreshnessContext? freshnessContext, - out int total, - out int fileCount, - out List freshnessObservations, - out bool hasFailures) - { - var queryCounts = new List(); - freshnessObservations = []; - var paths = new HashSet(StringComparer.Ordinal); - total = 0; - hasFailures = false; - foreach (var recipeQuery in recipeQueries) - { - try - { - var exact = userExact || recipeQuery.ExactSubstring; - var queryScope = BuildSearchRecipeQueryScope(scope, recipeQuery); - var guardFilters = BuildSearchRecipeGuardFilters(options, recipeQuery); - var results = reader.Search( - recipeQuery.Query, - int.MaxValue, - options.Lang, - false, - queryScope.PathPatterns, - queryScope.ExcludePaths, - queryScope.ExcludeTests, - !options.NoDedup, - options.Since, - exact, - false, - !options.NoVisibilityRank, - cursor: options.SearchCursor, - guardFilters: guardFilters, - guardWindow: options.GuardWindow, - guardScope: options.GuardScope, - requiredPathPatterns: GetSearchRecipeRequiredPathPatterns(options, recipeQuery)); - results = ApplySearchRecipeFileRejectQueries(reader, results, options, recipeQuery); - var rows = BuildSearchDisplayRows(results, options, exact, recipeQuery.Query, rawFtsOverride: false, recipeQuery: recipeQuery); - rows = ApplySearchRecipeSemanticFilter(reader, options, recipeQuery, rows); - if (options.Json && !options.SummaryOnly) - ApplySearchRecipeAuditClassifications(reader, recipeQuery, recipeQueries, rows); - var count = rows.Count; - var fileCountForQuery = rows.Select(row => row.Result.Path).Distinct(StringComparer.Ordinal).Count(); - foreach (var path in rows.Select(row => row.Result.Path)) - paths.Add(path); - - total += count; - queryCounts.Add(new SearchRecipeCountQueryJsonResult( - recipeQuery.Name, - recipeQuery.Query, - recipeQuery.Description, - recipeQuery.Severity, - count, - count, - 0, - count, - fileCountForQuery, - false, - BuildSearchRecipeClassifierCounts(rows), - BuildSearchRecipeTopFiles(rows))); - if (freshnessContext != null) - { - freshnessObservations.Add(SuccessfulSearchQueryObservation( - freshnessContext, - recipeQuery.Name, - count)); - } - } - catch (Exception ex) when ( - freshnessContext != null - && TryClassifySearchQueryExecutionFailure(ex, out _)) - { - TryClassifySearchQueryExecutionFailure(ex, out var failureReason); - hasFailures = true; - freshnessObservations.Add(FailedSearchQueryObservation( - freshnessContext, - recipeQuery.Name, - failureReason)); - } - } - - fileCount = paths.Count; - return queryCounts; - } - private static void ApplySearchRecipeAuditClassifications( DbReader reader, SearchAuditRecipeQuery recipeQuery, @@ -4560,82 +4166,6 @@ private sealed class JsonTrustLexicalContextCache public JsonTrustLexicalContext? Context { get; set; } } - private static List CollectSearchRecipeAggregationResults( - DbReader reader, - IReadOnlyList recipeQueries, - SearchRecipeScopeJsonResult scope, - QueryCommandOptions options, - bool userExact, - string groupBy, - out int total, - out int fileCount) - { - var queryResults = new List(); - var paths = new HashSet(StringComparer.Ordinal); - total = 0; - foreach (var recipeQuery in recipeQueries) - { - var exact = userExact || recipeQuery.ExactSubstring; - var queryScope = BuildSearchRecipeQueryScope(scope, recipeQuery); - var guardFilters = BuildSearchRecipeGuardFilters(options, recipeQuery); - var results = reader.Search( - recipeQuery.Query, - int.MaxValue, - options.Lang, - false, - queryScope.PathPatterns, - queryScope.ExcludePaths, - queryScope.ExcludeTests, - !options.NoDedup, - options.Since, - exact, - false, - !options.NoVisibilityRank, - cursor: options.SearchCursor, - guardFilters: guardFilters, - guardWindow: options.GuardWindow, - guardScope: options.GuardScope, - requiredPathPatterns: GetSearchRecipeRequiredPathPatterns(options, recipeQuery)); - results = ApplySearchRecipeFileRejectQueries(reader, results, options, recipeQuery); - var rows = BuildSearchDisplayRows(results, options, exact, recipeQuery.Query, rawFtsOverride: false, recipeQuery: recipeQuery); - rows = ApplySearchRecipeSemanticFilter(reader, options, recipeQuery, rows); - foreach (var path in rows.Select(row => row.Result.Path)) - paths.Add(path); - - var groups = BuildSearchGroupedCounts(groupBy, rows); - var selection = ApplySearchGroupOutputSelection(groups, options); - total += rows.Count; - queryResults.Add(new SearchRecipeAggregationQueryJsonResult( - recipeQuery.Name, - recipeQuery.Query, - recipeQuery.Description, - recipeQuery.Severity, - rows.Count, - rows.Select(row => row.Result.Path).Distinct(StringComparer.Ordinal).Count(), - selection.Groups.Count, - selection.TotalGroups, - selection.Truncated, - options.Limit, - selection.Groups)); - } - - fileCount = paths.Count; - return queryResults; - } - - private static IReadOnlyList BuildSearchRecipeGuardFilters(QueryCommandOptions options, SearchAuditRecipeQuery recipeQuery) - { - if (recipeQuery.GuardFilters.Count == 0) - return options.GuardFilters; - if (options.GuardFilters.Count == 0) - return recipeQuery.GuardFilters; - - var guardFilters = new List(recipeQuery.GuardFilters.Count + options.GuardFilters.Count); - guardFilters.AddRange(recipeQuery.GuardFilters); - guardFilters.AddRange(options.GuardFilters); - return guardFilters; - } - private static SearchRecipeRunSummaryJsonResult BuildSearchRecipeRunSummary( IReadOnlyList queryResults, int limitPerQuery, diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchRouting.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchRouting.cs new file mode 100644 index 0000000000..f24ed29447 --- /dev/null +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchRouting.cs @@ -0,0 +1,496 @@ +using System.Text.Json; + +namespace CodeIndex.Cli; + +public static partial class QueryCommandRunner +{ + private enum SearchExecutionKind + { + RecipeList, + NamedBatchCount, + NamedBatchRows, + RecipeAggregation, + RecipeCount, + RecipeIssueDrafts, + RecipeRows, + PlainIssueDrafts, + PlainRows, + } + + private readonly record struct SearchRoutePlan( + SearchExecutionKind Execution, + QueryCommandOptions Options, + bool Exact, + CancellationToken CancellationToken) + { + internal bool ExactSearch => Exact || Options.TokenBoundary; + internal JsonSerializerOptions JsonOptions => GetSearchInvocationJsonOptions(Options); + } + + private static bool TryCreateSearchRoutePlan( + string[] cmdArgs, + QueryCommandOptions options, + bool exact, + CancellationToken cancellationToken, + out SearchRoutePlan route) + { + route = default; + SearchExecutionKind execution; + if (options.ListRecipes) + { + if (!TryValidateSearchRecipeListRoute(options)) + return false; + execution = SearchExecutionKind.RecipeList; + } + else if (options.NamedSearchQueries.Count > 0) + { + if (!TryValidateSearchNamedBatchRoute(options)) + return false; + execution = options.CountOnly || options.SummaryOnly + ? SearchExecutionKind.NamedBatchCount + : SearchExecutionKind.NamedBatchRows; + } + else if (options.RecipeName != null) + { + if (!TryValidateSearchRecipeRoute(options)) + return false; + execution = GetSearchRecipeExecutionKind(options); + } + else + { + if (!TryValidatePlainSearchRoute(cmdArgs, options)) + return false; + execution = options.OutputFormat == OutputFormatIssueDrafts + ? SearchExecutionKind.PlainIssueDrafts + : SearchExecutionKind.PlainRows; + } + + route = new SearchRoutePlan(execution, options, exact, cancellationToken); + return true; + } + + private static int ExecuteSearchRoute(SearchRoutePlan route) => + route.Execution switch + { + SearchExecutionKind.RecipeList => WriteSearchRecipeList( + route.Options, + route.JsonOptions, + route.Options.InvocationContext.CommandName), + SearchExecutionKind.NamedBatchCount => RunSearchNamedBatchCount( + route.Options, + route.JsonOptions, + route.ExactSearch), + SearchExecutionKind.NamedBatchRows => RunSearchNamedBatch( + route.Options, + route.JsonOptions, + route.ExactSearch), + SearchExecutionKind.RecipeAggregation => RunSearchRecipeAggregation( + route.Options, + route.JsonOptions, + route.Exact), + SearchExecutionKind.RecipeCount => RunSearchRecipeCount( + route.Options, + route.JsonOptions, + route.Exact), + SearchExecutionKind.RecipeIssueDrafts => RunSearchRecipeIssueDrafts( + route.Options, + route.JsonOptions, + route.Exact, + route.CancellationToken), + SearchExecutionKind.RecipeRows => RunSearchRecipe( + route.Options, + route.JsonOptions, + route.Exact), + SearchExecutionKind.PlainIssueDrafts => RunSearchIssueDrafts( + route.Options, + route.JsonOptions, + route.ExactSearch, + route.CancellationToken), + SearchExecutionKind.PlainRows => ExecutePlainSearch( + CreateSearchExecutionPlan( + route.Options, + route.JsonOptions, + route.ExactSearch, + route.Options.Query!)), + _ => throw new InvalidOperationException("Unknown search execution route."), + }; + + private static JsonSerializerOptions GetSearchInvocationJsonOptions(QueryCommandOptions options) => + options.InvocationJsonOptions + ?? throw new InvalidOperationException("Search invocation JSON options were not initialized."); + + private static SearchExecutionKind GetSearchRecipeExecutionKind(QueryCommandOptions options) + { + if (options.CountOnly || (options.SummaryOnly && (options.Compact || options.OutputFormat == OutputFormatCompact))) + { + return HasSearchAggregation(options) + ? SearchExecutionKind.RecipeAggregation + : SearchExecutionKind.RecipeCount; + } + if (HasSearchCountOrUniqueAggregation(options)) + return SearchExecutionKind.RecipeAggregation; + return options.OutputFormat == OutputFormatIssueDrafts + ? SearchExecutionKind.RecipeIssueDrafts + : SearchExecutionKind.RecipeRows; + } + + private static bool TryValidateSearchRecipeListRoute(QueryCommandOptions options) + { + if (HasSearchRowSelectors(options)) + { + return RejectSearchUsage( + options, + "row-selection controls are not supported with --list-recipes because recipe discovery does not emit search rows.", + "Remove --first-per-file / --sample, or execute a recipe or plain search that returns rows."); + } + if (options.RecipeName != null || options.NamedSearchQueries.Count > 0 || options.ExtraNames.Count > 0) + { + return RejectSearchUsage( + options, + "--list-recipes cannot be combined with --recipe, --named-query, or extra positional arguments.", + "Run `cdidx search --list-recipes --query ` to filter built-in audit recipes by recipe, query, label, severity, path, or search text."); + } + if (options.OutputFormat is not OutputFormatText and not OutputFormatJson and not OutputFormatCompact) + { + return RejectSearchUsage( + options, + "--format count/csv/tsv/lsp/qf/sarif/issue-drafts is not supported with --list-recipes.", + "Use plain text output, `--json` / `--format json` for the full recipe list, or `--format compact` for a compact summary."); + } + if (options.JsonOutputFormat == JsonOutputFormatArray) + { + return RejectSearchUsage( + options, + "--json=array is not supported with --list-recipes because recipe-list output is a JSON object.", + "Use plain `--json` for the recipe-list object."); + } + + return true; + } + + private static bool TryValidateSearchNamedBatchRoute(QueryCommandOptions options) + { + if (HasSearchRowSelectors(options)) + { + return RejectSearchUsage( + options, + "row-selection controls are not supported with --named-query because named batches do not expose selector accounting.", + "Remove --first-per-file / --sample, or run each query as a plain search or recipe row output."); + } + if (options.Query != null || options.RecipeName != null || options.ExtraNames.Count > 0) + { + return RejectSearchUsage( + options, + "--named-query cannot be combined with a positional query, --query, --recipe, or extra positional arguments.", + "Pass one or more `--named-query =` values, or run a plain `cdidx search `."); + } + if (options.OpenIssuesPath != null) + { + return RejectSearchUsage( + options, + "--open-issues can only be used with `cdidx search --recipe --format issue-drafts`.", + "Remove --open-issues for ad hoc named batches."); + } + if (options.OutputFormat is not OutputFormatText and not OutputFormatJson and not OutputFormatCount and not OutputFormatCompact) + { + return RejectSearchUsage( + options, + "--format csv/tsv/lsp/qf/sarif/issue-drafts is not supported with --named-query.", + "Use plain text output, `--json`, `--format count`, or `--format compact` for grouped ad hoc results."); + } + if (options.JsonOutputFormat == JsonOutputFormatArray) + { + return RejectSearchUsage( + options, + "--json=array is not supported with --named-query because named batch output is grouped by query.", + "Use plain `--json` for the grouped named-query object."); + } + if (options.MaxJsonBytes.HasValue && !options.Json) + { + return RejectSearchUsage( + options, + "--max-json-bytes is only supported with JSON search output.", + "Use `--json` or `--format compact` with --named-query when bounding named batch output."); + } + + return true; + } + + private static bool TryValidateSearchRecipeRoute(QueryCommandOptions options) => + TryValidateSearchRecipeIdentityAndOutput(options) + && TryValidateSearchRecipeRowSelection(options) + && TryValidateSearchRecipeAggregation(options); + + private static bool TryValidateSearchRecipeIdentityAndOutput(QueryCommandOptions options) + { + if (options.TokenBoundary) + { + return RejectSearchUsage( + options, + "--token-boundary is only supported for ad hoc search and --named-query batches, not recipe execution.", + "Run an individual query without --recipe if token-boundary filtering is required."); + } + if (options.Query != null || options.ExtraNames.Count > 0) + { + return RejectSearchUsage( + options, + "--recipe expands into its own curated query set and cannot be combined with a search query.", + "Remove the positional query, or run a plain `cdidx search ` without --recipe."); + } + if (options.Prefix) + { + return RejectSearchUsage( + options, + "--prefix is not supported with --recipe because each recipe query defines its own match mode.", + "Remove --prefix, or run the individual query from the recipe list yourself."); + } + if (options.OutputFormat is not OutputFormatText and not OutputFormatJson and not OutputFormatCount and not OutputFormatCompact and not OutputFormatSarif and not OutputFormatIssueDrafts) + { + return RejectSearchUsage( + options, + "--format csv/tsv/lsp/qf is not supported with --recipe.", + "Use `--count` / `--format count` for count-only recipe output, `--json` for grouped recipe results, `--format compact` for summary-first compact JSON, `--format sarif` for audit findings, or `--format issue-drafts` for draft exports."); + } + if (options.JsonOutputFormat == JsonOutputFormatArray) + { + return RejectSearchUsage( + options, + "--json=array is not supported with --recipe because recipe output is grouped by query.", + "Use plain `--json` for the grouped recipe object."); + } + if (options.OutputFormat == OutputFormatSarif + && (options.CountOnly + || options.SummaryOnly + || HasSearchAggregation(options) + || options.ResultsOnly + || options.SearchFields != null + || HasSearchRowSelectors(options) + || options.GroupedPerFileLimitExplicit + || (options.JsonOutputFormatExplicit && options.JsonOutputFormat == JsonOutputFormatNdjson))) + { + return RejectSearchUsage( + options, + "--format sarif cannot be combined with recipe count, summary, aggregation, projection, row-selection, or NDJSON controls.", + "Use `--recipe --format sarif` with result filters and `--limit` / `--total-limit`, or choose the JSON/count output shape instead."); + } + if (options.GroupedPerFileLimitExplicit) + { + return RejectSearchUsage( + options, + "--per-file-limit is not supported with --recipe because recipe execution does not produce grouped search output.", + "Use --first-per-file for one selected recipe row per file, or remove --recipe and use grouped ad hoc search output."); + } + + return true; + } + + private static bool TryValidateSearchRecipeRowSelection(QueryCommandOptions options) + { + if (HasSearchRowSelectors(options) && options.SearchCursor.HasValue) + { + return RejectSearchUsage( + options, + "recipe row-selection controls cannot be combined with --cursor because raw recipe cursors cannot preserve selector state.", + "Remove --cursor and rerun selection from the beginning, or remove --first-per-file / --sample to resume from the cursor."); + } + if (HasSearchRowSelectors(options) + && (options.CountOnly + || HasSearchAggregation(options) + || options.ResultsOnly + || (options.SummaryOnly && (options.Compact || options.OutputFormat == OutputFormatCompact)))) + { + return RejectSearchUsage( + options, + "recipe row-selection controls cannot be combined with count, aggregation, results-only, or summary-only compact output.", + "Remove --first-per-file / --sample to keep the non-row output, or choose text, JSON, compact, NDJSON, or issue-drafts row output."); + } + if (options.MaxJsonBytes.HasValue && !SupportsSearchJsonByteLimit(options)) + { + return RejectSearchUsage( + options, + "--max-json-bytes is only supported with JSON search output.", + "Use `--json=ndjson`, `--format count`, `--format compact`, grouped/count-by JSON, or `--format issue-drafts` with --max-json-bytes."); + } + if (options.ResultsOnly && options.JsonOutputFormat != JsonOutputFormatNdjson) + { + return RejectSearchUsage( + options, + "--results-only is only supported with NDJSON recipe output.", + "Use `--recipe --results-only --search-fields path,line,query_name`, or remove --results-only."); + } + + return true; + } + + private static bool TryValidateSearchRecipeAggregation(QueryCommandOptions options) + { + if (options.GroupBy != null) + { + if (!IsSupportedSearchGroupByValue(options.GroupBy)) + { + return RejectSearchUsage( + options, + "--group-by for recipe search must be one of file, symbol, origin, return-type, or subsystem.", + $"Use `{options.InvocationContext.RecipeCommandPrefix} --group-by file --count`, `--group-by symbol --count`, `--group-by return-type --count`, `--group-by subsystem --count`, or `--count-by origin`."); + } + if (!options.CountOnly) + { + return RejectSearchUsage( + options, + $"{options.InvocationContext.RecipeExecutionName} --group-by requires --count.", + "Add --count to request grouped recipe result counts, or remove --group-by to print matching snippets."); + } + } + if (!TryValidateSearchAggregationFields(options, SearchAggregationTarget.Recipe)) + return false; + if (!TryValidateSearchAggregationConflicts(options, SearchAggregationTarget.Recipe)) + return false; + if (HasSearchAggregation(options) && (options.ResultsOnly || options.SearchFields != null)) + { + return RejectSearchUsage( + options, + "recipe aggregation cannot be combined with --results-only or --search-fields.", + "Run the aggregation separately, or remove --count-by/--group-by to stream projected recipe rows."); + } + + return true; + } + + private static bool TryValidatePlainSearchRoute(string[] cmdArgs, QueryCommandOptions options) => + TryValidatePlainSearchQuery(cmdArgs, options) + && TryValidatePlainSearchRowSelection(options) + && TryValidatePlainSearchAggregationShape(options); + + private static bool TryValidatePlainSearchQuery(string[] cmdArgs, QueryCommandOptions options) + { + if (TryWriteBlankQueryError(options, "search")) + return false; + if (options.Query == null) + { + CommandErrorWriter.WriteJsonOrHuman( + options.Json, + GetSearchInvocationJsonOptions(options), + "search requires a query argument", + CommandExitCodes.UsageError, + BuildMissingSearchQueryHint(cmdArgs), + GetUsageLineOrThrow("search"), + CommandErrorCodes.UsageError, + category: "usage"); + return false; + } + if (options.Query.Length > QueryLimits.MaxQueryLength) + return RejectSearchUsage( + options, + QueryLimits.FormatQueryTooLongError(), + "Shorten the search text or split generated input into smaller queries before running `cdidx search`."); + return !TryWriteUnexpectedExtraPositionals("search", options); + } + + private static bool TryValidatePlainSearchRowSelection(QueryCommandOptions options) + { + if (HasSearchRowSelectors(options) + && (options.CountOnly + || HasSearchAggregation(options) + || options.OutputFormat == OutputFormatGrouped)) + { + return RejectSearchUsage( + options, + "search row-selection controls cannot be combined with count or aggregation output.", + "Remove --first-per-file / --sample to count the full filtered population, or choose a row output that reports selector accounting."); + } + if (HasSearchRowSelectors(options) && options.ResultsOnly) + { + return RejectSearchUsage( + options, + "search row-selection controls cannot be combined with --results-only because that stream omits selector accounting.", + "Remove --results-only to retain the NDJSON terminal record, or remove --first-per-file / --sample."); + } + if (HasSearchRowSelectors(options) && options.JsonOutputFormat == JsonOutputFormatArray) + { + return RejectSearchUsage( + options, + "search row-selection controls cannot be combined with metadata-free --json=array output.", + "Add --json-envelope to retain selector accounting, use --json=ndjson / --format compact, or remove --first-per-file / --sample."); + } + if (HasSearchRowSelectors(options) + && options.OutputFormat is not OutputFormatText + and not OutputFormatJson + and not OutputFormatCompact + and not OutputFormatIssueDrafts) + { + return RejectSearchUsage( + options, + "search row-selection controls are only supported by text, JSON, compact, and issue-drafts row output.", + "Choose an output shape that reports selector accounting, or remove --first-per-file / --sample."); + } + + return true; + } + + private static bool TryValidatePlainSearchAggregationShape(QueryCommandOptions options) + { + if (options.GroupBy != null) + { + if (!IsSupportedSearchGroupByValue(options.GroupBy)) + { + return RejectSearchUsage( + options, + "--group-by for search must be one of file, symbol, origin, return-type, or subsystem.", + "Use `cdidx search --group-by file --count`, `--group-by symbol --count`, `--group-by return-type --count`, `--group-by subsystem --count`, or `--count-by origin`."); + } + if (!options.CountOnly) + { + return RejectSearchUsage( + options, + "search --group-by requires --count.", + "Add --count to request grouped result counts, or remove --group-by to print matching snippets."); + } + if (options.OutputFormat is not OutputFormatText and not OutputFormatJson and not OutputFormatCount) + { + return RejectSearchUsage( + options, + "--group-by for search only supports plain count output or JSON.", + "Use `--count`, optionally with `--json`, instead of compact/location formats."); + } + if (options.JsonOutputFormat == JsonOutputFormatArray) + { + return RejectSearchUsage( + options, + "--json=array is not supported with search --group-by because grouped count output is a JSON object.", + "Use plain `--json` for the grouped-count object."); + } + } + if (!TryValidateSearchAggregationConflicts(options, SearchAggregationTarget.Plain)) + return false; + if (HasSearchCountOrUniqueAggregation(options) && options.JsonOutputFormat == JsonOutputFormatArray) + { + return RejectSearchUsage( + options, + "--json=array is not supported with search aggregation because aggregation output is a JSON object.", + "Use plain `--json` for `--count-by` or `--unique` aggregation output."); + } + if (options.OutputFormat == OutputFormatGrouped && options.JsonOutputFormat == JsonOutputFormatArray) + { + return RejectSearchUsage( + options, + "--json=array is not supported with search --format grouped because grouped output is a JSON object.", + "Use plain `--json` or omit --json when using `--format grouped`."); + } + if (options.ResultsOnly && options.JsonOutputFormat != JsonOutputFormatNdjson) + { + return RejectSearchUsage( + options, + "--results-only is only supported with NDJSON search output.", + "Use `--results-only --json=ndjson`, or remove --results-only when using --json=array."); + } + if (options.MaxJsonBytes.HasValue && !SupportsSearchJsonByteLimit(options)) + { + return RejectSearchUsage( + options, + "--max-json-bytes is only supported with JSON search output.", + "Use `--json=ndjson`, `--json=array`, `--format count`, `--format compact`, grouped/count-by JSON, or `--format issue-drafts` with --max-json-bytes."); + } + + return TryValidateSearchAggregationFields(options, SearchAggregationTarget.Plain); + } +} diff --git a/src/CodeIndex/Cli/QueryCommandRunner.SearchValidation.cs b/src/CodeIndex/Cli/QueryCommandRunner.SearchValidation.cs new file mode 100644 index 0000000000..335830b6c8 --- /dev/null +++ b/src/CodeIndex/Cli/QueryCommandRunner.SearchValidation.cs @@ -0,0 +1,296 @@ +namespace CodeIndex.Cli; + +public static partial class QueryCommandRunner +{ + private enum SearchAggregationTarget + { + Plain, + Recipe, + } + + private static bool TryValidateSearchOptions( + QueryCommandOptions options, + bool exact, + QueryCommandInvocationContext invocationContext) => + TryValidateSearchIssueSourceOptions(options) + && TryValidateSearchRecipeControlOptions(options) + && TryValidateSearchIssueShapeOptions(options) + && TryValidateSearchDiscoveryShapeOptions(options) + && TryValidateSearchCrossRouteOptions(options, exact, invocationContext); + + private static bool TryValidateSearchIssueSourceOptions(QueryCommandOptions options) + { + if (options.OpenIssuesPath != null && options.OutputFormat != OutputFormatIssueDrafts) + { + return RejectSearchUsage( + options, + "--open-issues can only be used with `cdidx search --format issue-drafts`.", + "Use an open-issues JSON file from `gh issue list --state open --json number,title,labels,url`."); + } + if (options.OpenIssuesRepository != null && !IssueDuplicatePreflight.IsGitHubOpenIssuesSource(options.OpenIssuesPath)) + { + return RejectSearchUsage( + options, + "--repo can only be used with `--open-issues github`.", + "Use `--open-issues github --repo owner/name` to fetch open issues directly from GitHub."); + } + if (options.IssueState != IssueDuplicatePreflight.DefaultIssueState && !IssueDuplicatePreflight.IsGitHubOpenIssuesSource(options.OpenIssuesPath)) + return RejectSearchUsage(options, "--issue-state can only be used with `--open-issues github`.", "Use `--open-issues github --repo owner/name --issue-state all`."); + if (options.DuplicatePreflightTuningExplicit && options.OutputFormat != OutputFormatIssueDrafts) + { + return RejectSearchUsage( + options, + "--duplicate-confidence and --duplicate-threshold can only be used with `cdidx search --format issue-drafts`.", + "Use these controls when exporting issue draft JSON with duplicate-preflight metadata."); + } + + return true; + } + + private static bool TryValidateSearchRecipeControlOptions(QueryCommandOptions options) + { + if ((options.IncludeRecipeQueries.Count > 0 || options.ExcludeRecipeQueries.Count > 0) && options.RecipeName == null) + { + return RejectSearchUsage( + options, + "--include-query and --exclude-query can only be used with --recipe.", + "Use `--recipe risky-code --include-query raw-diagnostic-echo` to run a child query subset."); + } + if (options.SearchCursor.HasValue && options.RecipeName == null) + { + return RejectSearchUsage( + options, + "--cursor can only be used with --recipe.", + "Use `--recipe risky-code/raw-diagnostic-echo --format compact --cursor ` to fetch the next page for one child query."); + } + if (options.UnusedCursorOffset.HasValue) + { + return RejectSearchUsage( + options, + "--cursor for search must be a search pagination cursor returned by recipe search.", + "Use `--cursor ` only with `--recipe`; `unused:` cursors are for `cdidx unused`."); + } + if (options.OutlineCursorOffset.HasValue) + { + return RejectSearchUsage( + options, + "--cursor for search must be a search pagination cursor returned by recipe search.", + "`outline:` cursors are for `cdidx outline `."); + } + if (options.DependencyCycleCursor.HasValue) + { + return RejectSearchUsage( + options, + "--cursor for search must be a search pagination cursor returned by recipe search.", + "Dependency-cycle cursors are for `cdidx deps --cycles`."); + } + if (options.AuditScopeExplicit && options.RecipeName == null && options.ListRecipes) + { + return RejectSearchUsage( + options, + "--audit-scope cannot be combined with `cdidx search --list-recipes`.", + "Use `--query ` with --list-recipes to filter recipe discovery, or run an ad hoc search with `--source-only`."); + } + if (options.ShowExcluded && options.RecipeName == null) + { + return RejectSearchUsage( + options, + "--show-excluded is only supported with `cdidx search --recipe `.", + "Use it with a recipe run to include the effective scope and exclusion diagnostics in JSON output."); + } + + return true; + } + + private static bool TryValidateSearchIssueShapeOptions(QueryCommandOptions options) + { + if ((options.IssueTitle != null || options.IssueLabels.Count > 0) && options.OutputFormat != OutputFormatIssueDrafts) + { + return RejectSearchUsage( + options, + "--issue-title and --issue-label can only be used with `cdidx search --format issue-drafts`.", + "Use these hints when exporting issue draft JSON for a plain search."); + } + if (options.SnippetLines == 0 && options.OutputFormat != OutputFormatIssueDrafts) + { + return RejectSearchUsage( + options, + "--snippet-lines 0 is only supported with --format issue-drafts.", + "Use `--format issue-drafts --snippet-lines 0` for path/line-only draft evidence, or pass a positive snippet line count for search output."); + } + if (options.IssueTitle != null && options.RecipeName != null) + { + return RejectSearchUsage( + options, + "--issue-title is only supported for ad hoc search issue drafts.", + "Recipe issue-drafts produce one draft per recipe query, so their titles are derived from the recipe metadata."); + } + if (options.OutputFormat == OutputFormatIssueDrafts && options.CountOnly) + { + return RejectSearchUsage( + options, + "--count cannot be combined with --format issue-drafts.", + "Issue-draft export needs result evidence; remove --count."); + } + + return true; + } + + private static bool TryValidateSearchDiscoveryShapeOptions(QueryCommandOptions options) + { + if (options.NamesOnly && !options.ListRecipes) + { + return RejectSearchUsage( + options, + "--names is only supported with `cdidx recipes` or `cdidx search --list-recipes`.", + "Use `cdidx recipes --names --json` for a small deterministic recipe-name list."); + } + if (options.NamesOnly && options.SummaryOnly) + { + return RejectSearchUsage( + options, + "--names cannot be combined with --summary-only.", + "Use one recipe-list shape at a time."); + } + if (options.SummaryOnly + && !options.ListRecipes + && options.NamedSearchQueries.Count == 0 + && !(options.RecipeName != null + && (options.CountOnly + || options.Compact + || options.OutputFormat == OutputFormatCompact + || options.OutputFormat == OutputFormatIssueDrafts))) + { + return RejectSearchUsage( + options, + "--summary-only is only supported with `cdidx recipes` / `cdidx search --list-recipes`, named-query count output, recipe count output, or recipe issue-drafts output.", + "Use `cdidx recipes --summary-only --json`, `cdidx search --named-query = --summary-only --json`, `cdidx search --recipe --format compact --summary-only --json`, `cdidx search --recipe --format count --summary-only`, or `cdidx search --recipe --format issue-drafts --summary-only`."); + } + if (options.OutputFormat == OutputFormatIssueDrafts && options.JsonOutputFormat == JsonOutputFormatArray) + { + return RejectSearchUsage( + options, + "--json=array is not supported with --format issue-drafts because draft export is a JSON object.", + "Use plain `--json` or omit --json when exporting issue drafts."); + } + if (options.SearchCursor.HasValue && options.OutputFormat == OutputFormatIssueDrafts) + { + return RejectSearchUsage( + options, + "--cursor cannot be combined with --format issue-drafts.", + "Use --cursor with recipe JSON or compact output, then export issue drafts after choosing the desired query page."); + } + + return true; + } + + private static bool TryValidateSearchCrossRouteOptions( + QueryCommandOptions options, + bool exact, + QueryCommandInvocationContext invocationContext) + { + if ((exact || options.TokenBoundary) && options.Prefix) + return RejectSearchValidation( + options, + "--prefix cannot be combined with --exact / --exact-substring / --token-boundary (exact uses instr(), not FTS5 prefix phrases).", + "Drop --prefix to keep the exact substring path, or drop the exact-mode flag to opt into FTS5 prefix matching."); + if (options.GroupBy != null && (options.ListRecipes || options.NamedSearchQueries.Count > 0)) + { + var mode = options.ListRecipes + ? "--list-recipes" + : "--named-query"; + return RejectSearchUsage( + options, + $"--group-by is not supported with {mode}.", + "Use `cdidx search --group-by file --count` or remove --group-by for recipe-list and named-batch output."); + } + if (options.OutputFormat == OutputFormatGrouped && (options.ListRecipes || options.NamedSearchQueries.Count > 0 || options.RecipeName != null)) + { + var mode = options.ListRecipes + ? "--list-recipes" + : options.NamedSearchQueries.Count > 0 + ? "--named-query" + : "--recipe"; + return RejectSearchUsage( + options, + "--format grouped is only supported for plain search output.", + invocationContext.RecipeNameIsPositional && mode == "--recipe" + ? "Run a plain `cdidx search --format grouped`; audit recipe execution does not support grouped output." + : $"Remove {mode}, or run a plain `cdidx search --format grouped`."); + } + + return !TryWriteCappedJsonDiagnosticsUsageError(invocationContext.CommandName, options); + } + + private static bool HasSearchRowSelectors(QueryCommandOptions options) => + options.FirstPerFile || options.SampleSize.HasValue; + + private static bool HasSearchAggregation(QueryCommandOptions options) => + options.GroupBy != null || options.CountBy != null || options.UniqueBy != null; + + private static bool HasSearchCountOrUniqueAggregation(QueryCommandOptions options) => + options.CountBy != null || options.UniqueBy != null; + + private static bool TryValidateSearchAggregationFields( + QueryCommandOptions options, + SearchAggregationTarget target) + { + var recipe = target == SearchAggregationTarget.Recipe; + if (options.CountBy != null && !IsSupportedSearchAggregationValue(options.CountBy)) + { + return RejectSearchUsage( + options, + recipe + ? "--count-by for recipe search must be one of path, file, symbol, origin, return-type, or subsystem." + : "--count-by for search must be one of path, file, symbol, origin, return-type, or subsystem.", + "Use `--count-by path`, `--count-by symbol`, `--count-by return-type`, `--count-by subsystem`, or `--count-by origin`."); + } + if (options.UniqueBy != null && !IsSupportedSearchAggregationValue(options.UniqueBy)) + { + return RejectSearchUsage( + options, + recipe + ? "--unique for recipe search must be one of path, file, symbol, origin, return-type, or subsystem." + : "--unique for search must be one of path, file, symbol, origin, return-type, or subsystem.", + "Use `--unique path`, `--unique symbol`, `--unique return-type`, `--unique subsystem`, or `--unique origin`."); + } + + return true; + } + + private static bool TryValidateSearchAggregationConflicts( + QueryCommandOptions options, + SearchAggregationTarget target) + { + if (options.CountBy != null && options.UniqueBy != null) + { + return RejectSearchUsage( + options, + "--count-by cannot be combined with --unique.", + target == SearchAggregationTarget.Recipe + ? "Run one recipe aggregation mode at a time." + : "Run one aggregation mode at a time."); + } + if (options.GroupBy != null && HasSearchCountOrUniqueAggregation(options)) + { + return RejectSearchUsage( + options, + "--group-by cannot be combined with --count-by or --unique.", + "Use either `--group-by --count`, `--count-by `, or `--unique `."); + } + + return true; + } + + private static bool RejectSearchUsage(QueryCommandOptions options, string message, string hint) + { + WriteUsageError(message, options, hint); + return false; + } + + private static bool RejectSearchValidation(QueryCommandOptions options, string message, string hint) + { + WriteSearchValidationError(message, options, hint); + return false; + } +} diff --git a/src/CodeIndex/Database/BoundedIndexedChunkSelector.cs b/src/CodeIndex/Database/BoundedIndexedChunkSelector.cs new file mode 100644 index 0000000000..b5e73110df --- /dev/null +++ b/src/CodeIndex/Database/BoundedIndexedChunkSelector.cs @@ -0,0 +1,533 @@ +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +internal sealed class BoundedIndexedChunkSelector +{ + private const int LegacyResourceReadProgressOperations = 100; + + private static readonly string LegacyPredecessorSql = $""" + SELECT c.id, c.start_line, c.end_line, c.chunk_index + FROM chunks c INDEXED BY {DbReader.LegacyBoundedResourceReadFileIndexName} + WHERE c.file_id = @fileId + AND c.content IS NOT NULL + AND c.start_line <= @startLine + ORDER BY c.start_line DESC, c.chunk_index DESC + LIMIT @chunkLimit + """; + + private static readonly string LegacyEndSql = $""" + SELECT c.id, c.start_line, c.end_line, c.chunk_index + FROM chunks c INDEXED BY {DbReader.LegacyBoundedResourceReadFileIndexName} + WHERE c.file_id = @fileId + AND c.content IS NOT NULL + AND c.end_line >= @startLine + ORDER BY c.end_line, c.start_line, c.chunk_index + LIMIT @chunkLimit + """; + + private static readonly string LegacyForwardSql = $""" + SELECT c.id, c.start_line, c.end_line, c.chunk_index + FROM chunks c INDEXED BY {DbReader.LegacyBoundedResourceReadFileIndexName} + WHERE c.file_id = @fileId + AND c.content IS NOT NULL + AND c.start_line > @startLine + AND c.start_line <= @endLine + ORDER BY c.start_line, c.chunk_index + LIMIT @chunkLimit + """; + + private readonly SqliteConnection _connection; + private readonly CancellationToken _cancellation; + private readonly bool _hasChunksTable; + private readonly IReadOnlySet _chunkIndexes; + private readonly int? _legacyVmStepLimitOverride; + + internal BoundedIndexedChunkSelector( + SqliteConnection connection, + CancellationToken cancellation, + bool hasChunksTable, + IReadOnlySet chunkIndexes, + int? legacyVmStepLimitOverride) + { + _connection = connection; + _cancellation = cancellation; + _hasChunksTable = hasChunksTable; + _chunkIndexes = chunkIndexes; + _legacyVmStepLimitOverride = legacyVmStepLimitOverride; + } + + internal IReadOnlyList? Select( + BoundedIndexedContentContext context, + out BoundedIndexedContentFailure? failure) + { + failure = ValidateFileMetadata(context); + if (failure != null) + return null; + + var queryMode = ResolveChunkQueryMode(context.File.Id, out failure); + if (failure != null) + return null; + if (context.NextLine > context.EffectiveEndLine) + { + failure = new BoundedIndexedContentFailure(BoundedFileReadStatus.InvalidContinuation); + return null; + } + + return SelectChunks(context, queryMode, out failure); + } + + private BoundedIndexedContentFailure? ValidateFileMetadata(BoundedIndexedContentContext context) + { + var file = context.File; + if (file.Size < 0 || file.Lines < 0 || (file.Size == 0 && file.Lines > 0)) + { + return new BoundedIndexedContentFailure( + BoundedFileReadStatus.InvalidTopology, + "resource_file_metadata_inconsistent"); + } + if (file.Lines != 0) + return null; + + if (_hasChunksTable && HasAnyResourceChunk(file.Id)) + { + return new BoundedIndexedContentFailure( + BoundedFileReadStatus.InvalidTopology, + "resource_file_metadata_inconsistent"); + } + if (DbReader.IsAffirmativelyEmptyIndexedFile(file.Lines, file.Checksum)) + return new BoundedIndexedContentFailure(BoundedFileReadStatus.Empty); + if (file.Size == 0 && file.Checksum != null) + { + return new BoundedIndexedContentFailure( + BoundedFileReadStatus.InvalidTopology, + "resource_file_metadata_inconsistent"); + } + return new BoundedIndexedContentFailure( + BoundedFileReadStatus.ContentUnavailable, + "resource_content_unavailable"); + } + + private ChunkQueryMode ResolveChunkQueryMode( + long fileId, + out BoundedIndexedContentFailure? failure) + { + if (!_hasChunksTable) + { + failure = new BoundedIndexedContentFailure( + BoundedFileReadStatus.ContentUnavailable, + "resource_content_unavailable"); + return default; + } + + var hasBoundedIndexes = _chunkIndexes.Contains(DbReader.BoundedResourceReadChunkIndexName) + && _chunkIndexes.Contains(DbReader.BoundedResourceReadChunkEndIndexName); + var mode = hasBoundedIndexes + ? ChunkQueryMode.Bounded + : _chunkIndexes.Contains(DbReader.LegacyBoundedResourceReadFileIndexName) + ? ChunkQueryMode.Legacy + : ChunkQueryMode.Unavailable; + if (mode == ChunkQueryMode.Unavailable) + { + failure = new BoundedIndexedContentFailure( + BoundedFileReadStatus.ContentUnavailable, + "resource_bounded_read_index_unavailable"); + return mode; + } + if (!TryHasNullResourceChunkBoundary(fileId, mode, out var hasNullBoundary)) + { + failure = new BoundedIndexedContentFailure( + BoundedFileReadStatus.ContentUnavailable, + "resource_bounded_read_index_unavailable"); + return mode; + } + if (hasNullBoundary) + { + failure = new BoundedIndexedContentFailure( + BoundedFileReadStatus.InvalidTopology, + "resource_chunk_topology_invalid"); + return mode; + } + + failure = null; + return mode; + } + + private IReadOnlyList? SelectChunks( + BoundedIndexedContentContext context, + ChunkQueryMode queryMode, + out BoundedIndexedContentFailure? failure) + { + var scanEndLine = (int)Math.Min( + context.EffectiveEndLine, + (long)context.NextLine + context.MaxLines - 1L); + var candidates = new ChunkCandidateSet(); + + failure = ReadCoveringCandidates(context, scanEndLine, queryMode, candidates); + if (failure != null) + return null; + + ReadForwardCandidates(context, scanEndLine, queryMode, candidates); + return FinalizeChunkSelection(context, queryMode, candidates, out failure); + } + + private BoundedIndexedContentFailure? ReadCoveringCandidates( + BoundedIndexedContentContext context, + int scanEndLine, + ChunkQueryMode queryMode, + ChunkCandidateSet candidates) + { + var predecessorCount = ReadChunkCandidates( + context, + scanEndLine, + queryMode, + candidates, + queryMode == ChunkQueryMode.Legacy ? LegacyPredecessorSql : DbReader.BoundedResourceReadPredecessorSql, + includeEndLine: false); + if (candidates.IndexUnavailable) + return IndexUnavailableFailure(); + + if (candidates.TopologyFailure != null + || candidates.Chunks.Count > DbReader.MaxBoundedFileReadChunks + || candidates.Covers(context.NextLine) + || predecessorCount <= DbReader.MaxBoundedFileReadChunks) + { + return null; + } + + var endCount = ReadChunkCandidates( + context, + scanEndLine, + queryMode, + candidates, + queryMode == ChunkQueryMode.Legacy ? LegacyEndSql : DbReader.BoundedResourceReadEndSql, + includeEndLine: false); + if (candidates.IndexUnavailable) + return IndexUnavailableFailure(); + if (candidates.TopologyFailure == null + && !candidates.Covers(context.NextLine) + && endCount > DbReader.MaxBoundedFileReadChunks) + { + return new BoundedIndexedContentFailure( + BoundedFileReadStatus.InvalidTopology, + "chunk_candidate_scan_limit_exceeded"); + } + return null; + } + + private void ReadForwardCandidates( + BoundedIndexedContentContext context, + int scanEndLine, + ChunkQueryMode queryMode, + ChunkCandidateSet candidates) + { + if (candidates.TopologyFailure != null + || candidates.Chunks.Count > DbReader.MaxBoundedFileReadChunks) + { + return; + } + + ReadChunkCandidates( + context, + scanEndLine, + queryMode, + candidates, + queryMode == ChunkQueryMode.Legacy ? LegacyForwardSql : DbReader.BoundedResourceReadForwardSql, + includeEndLine: true); + } + + private IReadOnlyList? FinalizeChunkSelection( + BoundedIndexedContentContext context, + ChunkQueryMode queryMode, + ChunkCandidateSet candidates, + out BoundedIndexedContentFailure? failure) + { + if (candidates.IndexUnavailable) + return Fail(IndexUnavailableFailure(), out failure); + if (candidates.TopologyFailure != null) + { + return Fail( + new BoundedIndexedContentFailure( + BoundedFileReadStatus.InvalidTopology, + candidates.TopologyFailure), + out failure); + } + if (candidates.Chunks.Count > DbReader.MaxBoundedFileReadChunks) + { + return Fail( + new BoundedIndexedContentFailure(BoundedFileReadStatus.InvalidTopology, "chunk_limit_exceeded"), + out failure); + } + + candidates.Chunks.Sort(static (left, right) => + { + var byStartLine = left.StartLine.CompareTo(right.StartLine); + return byStartLine != 0 ? byStartLine : left.ChunkIndex.CompareTo(right.ChunkIndex); + }); + if (candidates.Chunks.Count != 0) + { + failure = null; + return candidates.Chunks; + } + + if (!TryHasAnyStoredResourceChunk(context.File.Id, queryMode, out var hasStoredChunk)) + return Fail(IndexUnavailableFailure(), out failure); + var emptyFailure = hasStoredChunk + ? new BoundedIndexedContentFailure( + BoundedFileReadStatus.IncompleteCoverage, + "resource_chunk_coverage_incomplete") + : new BoundedIndexedContentFailure( + BoundedFileReadStatus.ContentUnavailable, + "resource_content_unavailable"); + return Fail(emptyFailure, out failure); + } + + private static IReadOnlyList? Fail( + BoundedIndexedContentFailure value, + out BoundedIndexedContentFailure? failure) + { + failure = value; + return null; + } + + private static BoundedIndexedContentFailure IndexUnavailableFailure() + => new( + BoundedFileReadStatus.ContentUnavailable, + "resource_bounded_read_index_unavailable"); + + private int ReadChunkCandidates( + BoundedIndexedContentContext context, + int scanEndLine, + ChunkQueryMode queryMode, + ChunkCandidateSet candidates, + string commandText, + bool includeEndLine) + { + int ExecuteQuery() + => ExecuteChunkCandidateQuery( + context, + scanEndLine, + candidates, + commandText, + includeEndLine); + + if (queryMode != ChunkQueryMode.Legacy) + return ExecuteQuery(); + if (TryRunLegacyResourceMetadataQuery(ExecuteQuery, out var candidateCount)) + return candidateCount; + + candidates.IndexUnavailable = true; + return 0; + } + + private int ExecuteChunkCandidateQuery( + BoundedIndexedContentContext context, + int scanEndLine, + ChunkCandidateSet candidates, + string commandText, + bool includeEndLine) + { + using var command = _connection.CreateCommand(); + command.CommandText = commandText; + SqliteCommandPolicy.AddInt64(command, "@fileId", context.File.Id); + SqliteCommandPolicy.AddInt32(command, "@startLine", context.NextLine); + if (includeEndLine) + SqliteCommandPolicy.AddInt32(command, "@endLine", scanEndLine); + SqliteCommandPolicy.AddInt32(command, "@chunkLimit", DbReader.MaxBoundedFileReadChunks + 1); + + var rawCandidateCount = 0; + using var reader = command.ExecuteTrackedReader(); + while (reader.TrackedRead()) + { + rawCandidateCount++; + _cancellation.ThrowIfCancellationRequested(); + if (!TryReadChunkCandidate(reader, candidates, out var chunk)) + break; + if (chunk.EndLine < context.NextLine || chunk.StartLine > scanEndLine) + continue; + + if (candidates.RowIds.Add(chunk.RowId)) + candidates.Chunks.Add(chunk); + if (candidates.Chunks.Count > DbReader.MaxBoundedFileReadChunks) + break; + } + return rawCandidateCount; + } + + private static bool TryReadChunkCandidate( + SqliteDataReader reader, + ChunkCandidateSet candidates, + out BoundedIndexedChunk chunk) + { + if (reader.IsDBNull(1) || reader.IsDBNull(2) || reader.IsDBNull(3)) + { + candidates.TopologyFailure = "resource_chunk_topology_invalid"; + chunk = default; + return false; + } + + chunk = new BoundedIndexedChunk( + reader.GetInt64(0), + reader.GetInt32(1), + reader.GetInt32(2), + reader.GetInt32(3)); + if (chunk.StartLine > 0 && chunk.EndLine >= chunk.StartLine && chunk.ChunkIndex >= 0) + return true; + + candidates.TopologyFailure = "resource_chunk_topology_invalid"; + return false; + } + + private bool HasAnyResourceChunk(long fileId) + { + _cancellation.ThrowIfCancellationRequested(); + using var command = _connection.CreateCommand(); + command.CommandText = "SELECT 1 FROM chunks WHERE file_id = @fileId LIMIT 1"; + SqliteCommandPolicy.AddInt64(command, "@fileId", fileId); + var hasChunk = command.ExecuteScalar() != null; + _cancellation.ThrowIfCancellationRequested(); + return hasChunk; + } + + private bool TryHasAnyStoredResourceChunk( + long fileId, + ChunkQueryMode queryMode, + out bool hasChunk) + { + bool ExecuteQuery() + { + _cancellation.ThrowIfCancellationRequested(); + using var command = _connection.CreateCommand(); + var indexName = queryMode == ChunkQueryMode.Legacy + ? DbReader.LegacyBoundedResourceReadFileIndexName + : DbReader.BoundedResourceReadChunkIndexName; + command.CommandText = $""" + SELECT 1 + FROM chunks INDEXED BY {indexName} + WHERE file_id = @fileId AND content IS NOT NULL + LIMIT 1 + """; + SqliteCommandPolicy.AddInt64(command, "@fileId", fileId); + var result = command.ExecuteScalar() != null; + _cancellation.ThrowIfCancellationRequested(); + return result; + } + + if (queryMode != ChunkQueryMode.Legacy) + { + hasChunk = ExecuteQuery(); + return true; + } + return TryRunLegacyResourceMetadataQuery(ExecuteQuery, out hasChunk); + } + + private bool TryHasNullResourceChunkBoundary( + long fileId, + ChunkQueryMode queryMode, + out bool hasNullBoundary) + { + bool ExecuteQuery() + { + var startIndexName = queryMode == ChunkQueryMode.Legacy + ? DbReader.LegacyBoundedResourceReadFileIndexName + : DbReader.BoundedResourceReadChunkIndexName; + var endIndexName = queryMode == ChunkQueryMode.Legacy + ? DbReader.LegacyBoundedResourceReadFileIndexName + : DbReader.BoundedResourceReadChunkEndIndexName; + _cancellation.ThrowIfCancellationRequested(); + var result = ResourceChunkExists($""" + SELECT 1 + FROM chunks INDEXED BY {startIndexName} + WHERE file_id = @fileId + AND content IS NOT NULL + AND start_line IS NULL + LIMIT 1 + """, fileId) || ResourceChunkExists($""" + SELECT 1 + FROM chunks INDEXED BY {endIndexName} + WHERE file_id = @fileId + AND content IS NOT NULL + AND end_line IS NULL + LIMIT 1 + """, fileId); + _cancellation.ThrowIfCancellationRequested(); + return result; + } + + if (queryMode != ChunkQueryMode.Legacy) + { + hasNullBoundary = ExecuteQuery(); + return true; + } + return TryRunLegacyResourceMetadataQuery(ExecuteQuery, out hasNullBoundary); + } + + private bool ResourceChunkExists(string commandText, long fileId) + { + using var command = _connection.CreateCommand(); + command.CommandText = commandText; + SqliteCommandPolicy.AddInt64(command, "@fileId", fileId); + return command.ExecuteScalar() != null; + } + + private bool TryRunLegacyResourceMetadataQuery(Func action, out T result) + { + var maxVmSteps = _legacyVmStepLimitOverride ?? DbReader.MaxLegacyResourceReadSqliteVmSteps; + maxVmSteps = Math.Clamp(maxVmSteps, 1, DbReader.MaxLegacyResourceReadSqliteVmSteps); + var callbackLimit = Math.Max(1, (maxVmSteps + LegacyResourceReadProgressOperations - 1) + / LegacyResourceReadProgressOperations); + var callbackCount = 0; + var budgetExceeded = false; + SQLitePCL.delegate_progress progress = _ => + { + callbackCount++; + if (callbackCount <= callbackLimit) + return 0; + + budgetExceeded = true; + return 1; + }; + + SQLitePCL.raw.sqlite3_progress_handler( + _connection.Handle, + LegacyResourceReadProgressOperations, + progress, + null!); + try + { + result = action(); + return true; + } + catch (SqliteException exception) when ( + budgetExceeded + && !_cancellation.IsCancellationRequested + && exception.SqliteErrorCode == 9) + { + result = default!; + return false; + } + finally + { + SQLitePCL.raw.sqlite3_progress_handler(_connection.Handle, 0, null!, null!); + GC.KeepAlive(progress); + } + } + + private enum ChunkQueryMode + { + Unavailable, + Bounded, + Legacy, + } + + private sealed class ChunkCandidateSet + { + internal List Chunks { get; } = []; + internal HashSet RowIds { get; } = []; + internal string? TopologyFailure { get; set; } + internal bool IndexUnavailable { get; set; } + + internal bool Covers(int line) + => Chunks.Any(chunk => chunk.StartLine <= line && chunk.EndLine >= line); + } +} diff --git a/src/CodeIndex/Database/BoundedIndexedContentReader.cs b/src/CodeIndex/Database/BoundedIndexedContentReader.cs new file mode 100644 index 0000000000..a95f0940ba --- /dev/null +++ b/src/CodeIndex/Database/BoundedIndexedContentReader.cs @@ -0,0 +1,188 @@ +using CodeIndex.Indexer; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +internal readonly record struct BoundedIndexedContentRequest( + ResourceFileMetadata File, + int StartLine, + int EndLine, + int MaxUtf8Bytes, + int MaxLines, + int? ContinuationLine, + int ContinuationByteOffset); + +internal readonly record struct BoundedIndexedContentFailure( + BoundedFileReadStatus Status, + string? Reason = null); + +internal sealed record BoundedIndexedContentContext( + ResourceFileMetadata File, + int StartLine, + int EndLine, + int MaxUtf8Bytes, + int MaxLines, + int NextLine, + int ContinuationByteOffset) +{ + internal int EffectiveEndLine => Math.Min(EndLine, File.Lines); +} + +internal readonly record struct BoundedIndexedChunk( + long RowId, + int StartLine, + int EndLine, + int ChunkIndex); + +internal sealed class BoundedIndexedContentReader +{ + private readonly BoundedIndexedChunkSelector _chunkSelector; + private readonly BoundedUtf8ContentDecoder _decoder; + + internal BoundedIndexedContentReader( + SqliteConnection connection, + CancellationToken cancellation, + bool hasChunksTable, + IReadOnlySet chunkIndexes, + int? scanByteLimitOverride, + int? legacyVmStepLimitOverride) + { + _chunkSelector = new BoundedIndexedChunkSelector( + connection, + cancellation, + hasChunksTable, + chunkIndexes, + legacyVmStepLimitOverride); + _decoder = new BoundedUtf8ContentDecoder( + connection, + cancellation, + scanByteLimitOverride); + } + + internal BoundedFileReadResult Read(BoundedIndexedContentRequest request) + { + var context = NormalizeRequest(request, out var earlyResult); + if (context == null) + return earlyResult!; + + var chunks = _chunkSelector.Select(context, out var selectionFailure); + if (selectionFailure is { } failure) + return CreateResult(context, failure.Status, failure.Reason); + + var decoded = _decoder.Decode(context, chunks!); + return BuildDecodedResult(context, decoded); + } + + private static BoundedIndexedContentContext? NormalizeRequest( + BoundedIndexedContentRequest request, + out BoundedFileReadResult? earlyResult) + { + var path = request.File.Path; + if (string.IsNullOrWhiteSpace(path)) + { + earlyResult = new BoundedFileReadResult { Status = BoundedFileReadStatus.FileNotFound }; + return null; + } + + ValidateBudgets(request); + if (!request.ContinuationLine.HasValue && request.ContinuationByteOffset != 0) + { + earlyResult = CreateInvalidContinuationResult(path); + return null; + } + + var startLine = Math.Max(1, request.StartLine); + var endLine = Math.Max(startLine, request.EndLine); + var nextLine = request.ContinuationLine ?? startLine; + if (nextLine < startLine || nextLine > endLine) + { + earlyResult = CreateInvalidContinuationResult(path); + return null; + } + + earlyResult = null; + return new BoundedIndexedContentContext( + request.File, + startLine, + endLine, + request.MaxUtf8Bytes, + request.MaxLines, + nextLine, + request.ContinuationByteOffset); + } + + private static void ValidateBudgets(BoundedIndexedContentRequest request) + { + if (request.MaxUtf8Bytes is <= 0 or > DbReader.MaxBoundedFileReadUtf8Bytes) + { + throw new ArgumentOutOfRangeException( + "maxUtf8Bytes", + request.MaxUtf8Bytes, + $"UTF-8 byte budget must be between 1 and {DbReader.MaxBoundedFileReadUtf8Bytes}."); + } + if (request.MaxLines is <= 0 or > DbReader.MaxBoundedFileReadLines) + { + throw new ArgumentOutOfRangeException( + "maxLines", + request.MaxLines, + $"Line budget must be between 1 and {DbReader.MaxBoundedFileReadLines}."); + } + if (request.ContinuationByteOffset < 0) + { + throw new ArgumentOutOfRangeException( + "continuationByteOffset", + request.ContinuationByteOffset, + "Continuation byte offset must be non-negative."); + } + } + + private static BoundedFileReadResult CreateInvalidContinuationResult(string path) + => new() + { + Status = BoundedFileReadStatus.InvalidContinuation, + Path = path, + }; + + private static BoundedFileReadResult BuildDecodedResult( + BoundedIndexedContentContext context, + BoundedUtf8DecodeResult decoded) + { + if (decoded.Failure is { } failure) + return CreateResult(context, failure.Status, failure.Reason); + + return new BoundedFileReadResult + { + Status = BoundedFileReadStatus.Success, + Path = context.File.Path, + Lang = context.File.Lang, + TotalLines = context.File.Lines, + RequestedStartLine = context.StartLine, + RequestedEndLine = context.EndLine, + StartLine = decoded.FirstReturnedLine ?? context.NextLine, + EndLine = decoded.LastReturnedLine ?? context.NextLine, + Content = decoded.Content.ToString(), + Utf8Bytes = decoded.Utf8Bytes, + Truncated = decoded.TruncationReason != null, + TruncationReason = decoded.TruncationReason, + NextLine = decoded.TruncationReason != null ? decoded.NextLine : null, + NextByteOffset = decoded.TruncationReason != null ? decoded.NextByteOffset : null, + }; + } + + private static BoundedFileReadResult CreateResult( + BoundedIndexedContentContext context, + BoundedFileReadStatus status, + string? failureReason = null) + => new() + { + Status = status, + FailureReason = failureReason, + Path = context.File.Path, + Lang = context.File.Lang, + TotalLines = context.File.Lines, + RequestedStartLine = context.StartLine, + RequestedEndLine = context.EndLine, + StartLine = context.NextLine, + EndLine = context.NextLine, + }; +} diff --git a/src/CodeIndex/Database/BoundedUtf8ContentDecoder.cs b/src/CodeIndex/Database/BoundedUtf8ContentDecoder.cs new file mode 100644 index 0000000000..c6eca7c7cc --- /dev/null +++ b/src/CodeIndex/Database/BoundedUtf8ContentDecoder.cs @@ -0,0 +1,496 @@ +using System.Buffers; +using System.Text; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +internal sealed record BoundedUtf8DecodeResult( + StringBuilder Content, + BoundedIndexedContentFailure? Failure, + int NextLine, + int NextByteOffset, + int Utf8Bytes, + int? FirstReturnedLine, + int? LastReturnedLine, + string? TruncationReason); + +internal sealed class BoundedUtf8ContentDecoder +{ + private const int ReadBufferSize = 4 * 1024; + + private readonly SqliteConnection _connection; + private readonly CancellationToken _cancellation; + private readonly int? _scanByteLimitOverride; + + internal BoundedUtf8ContentDecoder( + SqliteConnection connection, + CancellationToken cancellation, + int? scanByteLimitOverride) + { + _connection = connection; + _cancellation = cancellation; + _scanByteLimitOverride = scanByteLimitOverride; + } + + internal BoundedUtf8DecodeResult Decode( + BoundedIndexedContentContext context, + IReadOnlyList chunks) + { + var state = new ReadState(context.NextLine, context.ContinuationByteOffset); + var scanByteLimit = _scanByteLimitOverride ?? DbReader.MaxBoundedFileReadScannedUtf8Bytes; + if (scanByteLimit <= 0 || scanByteLimit > DbReader.MaxBoundedFileReadScannedUtf8Bytes) + scanByteLimit = DbReader.MaxBoundedFileReadScannedUtf8Bytes; + var settings = new ChunkDecodeSettings( + context.EffectiveEndLine, + context.MaxUtf8Bytes, + context.MaxLines, + scanByteLimit, + _cancellation); + var content = new StringBuilder(Math.Min(context.MaxUtf8Bytes, ReadBufferSize)); + var buffer = ArrayPool.Shared.Rent(ReadBufferSize); + try + { + DecodeChunks(chunks, settings, content, buffer, ref state); + } + catch (BoundedFileScanLimitException) + { + state.ScanLimitExceeded = true; + } + catch (InvalidDataException) + { + state.InvalidTopologyReason = "invalid_utf8_content"; + } + finally + { + // Indexed source may contain secrets. Clear the entire pooled array before reuse. + // index済みsourceにsecretが含まれうるため、poolへ戻す前に配列全体を消去する。 + ArrayPool.Shared.Return(buffer, clearArray: true); + } + return CreateResult(content, state); + } + + private void DecodeChunks( + IReadOnlyList chunks, + ChunkDecodeSettings settings, + StringBuilder content, + byte[] buffer, + ref ReadState state) + { + foreach (var chunk in chunks) + { + _cancellation.ThrowIfCancellationRequested(); + if (state.Stopped) + break; + if (chunk.EndLine < state.NextLine) + continue; + if (chunk.StartLine > state.NextLine) + { + state.IncompleteCoverage = true; + break; + } + + using var blob = new SqliteBlob(_connection, "chunks", "content", chunk.RowId, readOnly: true); + ReadBoundedChunk(blob, chunk, settings, content, buffer, ref state); + } + } + + private static BoundedUtf8DecodeResult CreateResult(StringBuilder content, ReadState state) + { + BoundedIndexedContentFailure? failure = null; + if (state.ScanLimitExceeded) + { + failure = new BoundedIndexedContentFailure( + BoundedFileReadStatus.InvalidTopology, + "scan_limit_exceeded"); + } + else if (state.InvalidTopologyReason != null) + { + failure = new BoundedIndexedContentFailure( + BoundedFileReadStatus.InvalidTopology, + state.InvalidTopologyReason); + } + else if (state.InvalidContinuation) + { + failure = new BoundedIndexedContentFailure( + BoundedFileReadStatus.InvalidContinuation, + "invalid_continuation"); + } + else if (state.IncompleteCoverage || (!state.Completed && state.TruncationReason == null)) + { + failure = new BoundedIndexedContentFailure( + BoundedFileReadStatus.IncompleteCoverage, + "resource_chunk_coverage_incomplete"); + } + + return new BoundedUtf8DecodeResult( + content, + failure, + state.NextLine, + state.NextByteOffset, + state.Utf8Bytes, + state.FirstReturnedLine, + state.LastReturnedLine, + state.TruncationReason); + } + + private static void ReadBoundedChunk( + SqliteBlob blob, + BoundedIndexedChunk chunk, + ChunkDecodeSettings settings, + StringBuilder content, + byte[] buffer, + ref ReadState state) + { + var cursor = new ChunkReadCursor(chunk.StartLine); + Span runeBytes = stackalloc byte[4]; + Span runeChars = stackalloc char[2]; + while (TryReadBlobByte( + blob, + buffer, + settings.MaxScannedUtf8Bytes, + settings.CancellationToken, + ref cursor, + ref state, + out var firstByte)) + { + if (!ProcessChunkByte( + blob, + chunk, + firstByte, + settings, + content, + buffer, + runeBytes, + runeChars, + ref cursor, + ref state)) + { + return; + } + } + + CompleteChunkFinalLine(chunk, settings, content, cursor, ref state); + } + + private static bool ProcessChunkByte( + SqliteBlob blob, + BoundedIndexedChunk chunk, + byte firstByte, + ChunkDecodeSettings settings, + StringBuilder content, + byte[] buffer, + Span runeBytes, + Span runeChars, + ref ChunkReadCursor cursor, + ref ReadState state) + { + if (cursor.LocalLine > chunk.EndLine) + return false; + if (cursor.LocalLine < state.NextLine) + { + AdvanceSkippedByte(firstByte, ref cursor); + return true; + } + if (cursor.LocalLine > state.NextLine) + { + state.InvalidContinuation = true; + return false; + } + if (firstByte == (byte)'\n') + return ProcessLineBreak(settings, content, ref cursor, ref state); + return ProcessRune( + blob, + firstByte, + settings, + content, + buffer, + runeBytes, + runeChars, + ref cursor, + ref state); + } + + private static void AdvanceSkippedByte(byte value, ref ChunkReadCursor cursor) + { + if (value == (byte)'\n') + { + cursor.LocalLine++; + cursor.LocalByteOffset = 0; + } + else + { + cursor.LocalByteOffset++; + } + } + + private static bool ProcessLineBreak( + ChunkDecodeSettings settings, + StringBuilder content, + ref ChunkReadCursor cursor, + ref ReadState state) + { + if (state.NextByteOffset != cursor.LocalByteOffset) + { + state.InvalidContinuation = true; + return false; + } + if (!CompleteBoundedLine( + cursor.LocalLine, + cursor.LocalByteOffset, + settings, + content, + ref state)) + { + return false; + } + + cursor.LocalLine++; + cursor.LocalByteOffset = 0; + return true; + } + + private static bool ProcessRune( + SqliteBlob blob, + byte firstByte, + ChunkDecodeSettings settings, + StringBuilder content, + byte[] buffer, + Span runeBytes, + Span runeChars, + ref ChunkReadCursor cursor, + ref ReadState state) + { + var runeByteCount = ReadUtf8Rune( + blob, + firstByte, + buffer, + settings.MaxScannedUtf8Bytes, + settings.CancellationToken, + runeBytes, + ref cursor, + ref state, + out var rune); + var scalarStartOffset = cursor.LocalByteOffset; + cursor.LocalByteOffset += runeByteCount; + if (scalarStartOffset < state.NextByteOffset) + { + if (cursor.LocalByteOffset > state.NextByteOffset) + state.InvalidContinuation = true; + return !state.InvalidContinuation; + } + if (scalarStartOffset != state.NextByteOffset) + { + state.InvalidContinuation = true; + return false; + } + if (state.Utf8Bytes > settings.MaxUtf8Bytes - runeByteCount) + { + state.TruncationReason = "max_bytes"; + return false; + } + if (!TryEnterBoundedLine(cursor.LocalLine, settings.MaxLines, ref state)) + return false; + + var runeCharCount = rune.EncodeToUtf16(runeChars); + content.Append(runeChars[..runeCharCount]); + state.Utf8Bytes += runeByteCount; + state.NextByteOffset = cursor.LocalByteOffset; + return true; + } + + private static void CompleteChunkFinalLine( + BoundedIndexedChunk chunk, + ChunkDecodeSettings settings, + StringBuilder content, + ChunkReadCursor cursor, + ref ReadState state) + { + // Persisted chunks omit the separator after their final line. Treat blob EOF as + // that line boundary and synthesize exactly one LF when the requested range continues. + // 永続化chunkは最終行後のseparatorを持たないため、blob EOFを行境界として扱い、 + // 要求範囲が続く場合だけLFを1つ合成する。 + if (state.Stopped || cursor.LocalLine > chunk.EndLine || cursor.LocalLine != state.NextLine) + return; + if (state.NextByteOffset != cursor.LocalByteOffset) + { + state.InvalidContinuation = true; + return; + } + + _ = CompleteBoundedLine( + cursor.LocalLine, + cursor.LocalByteOffset, + settings, + content, + ref state); + } + + private static bool CompleteBoundedLine( + int line, + int lineByteLength, + ChunkDecodeSettings settings, + StringBuilder content, + ref ReadState state) + { + if (!TryEnterBoundedLine(line, settings.MaxLines, ref state)) + return false; + if (line >= settings.EffectiveEndLine) + { + state.Completed = true; + return false; + } + if (state.Utf8Bytes >= settings.MaxUtf8Bytes) + { + state.NextLine = line; + state.NextByteOffset = lineByteLength; + state.TruncationReason = "max_bytes"; + return false; + } + + content.Append('\n'); + state.Utf8Bytes++; + state.NextLine = line + 1; + state.NextByteOffset = 0; + if (state.ReturnedLineCount >= settings.MaxLines) + { + state.TruncationReason = "max_lines"; + return false; + } + return true; + } + + private static bool TryEnterBoundedLine( + int line, + int maxLines, + ref ReadState state) + { + if (state.LastReturnedLine == line) + return true; + if (state.ReturnedLineCount >= maxLines) + { + state.TruncationReason = "max_lines"; + return false; + } + + state.FirstReturnedLine ??= line; + state.LastReturnedLine = line; + state.ReturnedLineCount++; + return true; + } + + private static int ReadUtf8Rune( + SqliteBlob blob, + byte firstByte, + byte[] buffer, + int maxScannedUtf8Bytes, + CancellationToken cancellationToken, + Span runeBytes, + ref ChunkReadCursor cursor, + ref ReadState state, + out Rune rune) + { + var byteCount = firstByte switch + { + <= 0x7f => 1, + >= 0xc2 and <= 0xdf => 2, + >= 0xe0 and <= 0xef => 3, + >= 0xf0 and <= 0xf4 => 4, + _ => throw new InvalidDataException("Indexed chunk contains invalid UTF-8."), + }; + runeBytes[0] = firstByte; + for (var index = 1; index < byteCount; index++) + { + if (!TryReadBlobByte( + blob, + buffer, + maxScannedUtf8Bytes, + cancellationToken, + ref cursor, + ref state, + out runeBytes[index])) + { + throw new InvalidDataException("Indexed chunk ends inside a UTF-8 scalar value."); + } + } + + var status = Rune.DecodeFromUtf8(runeBytes[..byteCount], out rune, out var bytesConsumed); + if (status != OperationStatus.Done || bytesConsumed != byteCount) + throw new InvalidDataException("Indexed chunk contains invalid UTF-8."); + return byteCount; + } + + private static bool TryReadBlobByte( + SqliteBlob blob, + byte[] buffer, + int maxScannedUtf8Bytes, + CancellationToken cancellationToken, + ref ChunkReadCursor cursor, + ref ReadState state, + out byte value) + { + if (cursor.BufferOffset >= cursor.BufferedBytes) + { + cancellationToken.ThrowIfCancellationRequested(); + if (blob.Position >= blob.Length) + { + value = 0; + return false; + } + var remaining = maxScannedUtf8Bytes - state.ScannedUtf8Bytes; + if (remaining <= 0) + throw new BoundedFileScanLimitException(); + cursor.BufferedBytes = blob.Read(buffer, 0, Math.Min(buffer.Length, remaining)); + cursor.BufferOffset = 0; + if (cursor.BufferedBytes == 0) + { + value = 0; + return false; + } + } + + value = buffer[cursor.BufferOffset++]; + state.ScannedUtf8Bytes++; + return true; + } + + private readonly record struct ChunkDecodeSettings( + int EffectiveEndLine, + int MaxUtf8Bytes, + int MaxLines, + int MaxScannedUtf8Bytes, + CancellationToken CancellationToken); + + private sealed class BoundedFileScanLimitException : Exception; + + private struct ChunkReadCursor(int localLine) + { + internal int BufferOffset; + internal int BufferedBytes; + internal int LocalLine = localLine; + internal int LocalByteOffset; + } + + private struct ReadState(int nextLine, int nextByteOffset) + { + internal int NextLine = nextLine; + internal int NextByteOffset = nextByteOffset; + internal int Utf8Bytes; + internal int ScannedUtf8Bytes; + internal int ReturnedLineCount; + internal int? FirstReturnedLine; + internal int? LastReturnedLine; + internal string? TruncationReason; + internal string? InvalidTopologyReason; + internal bool Completed; + internal bool InvalidContinuation; + internal bool IncompleteCoverage; + internal bool ScanLimitExceeded; + + internal readonly bool Stopped + => Completed + || InvalidContinuation + || IncompleteCoverage + || ScanLimitExceeded + || TruncationReason != null; + } +} diff --git a/src/CodeIndex/Database/CSharpBaseListParser.cs b/src/CodeIndex/Database/CSharpBaseListParser.cs new file mode 100644 index 0000000000..fdf0fe4b1e --- /dev/null +++ b/src/CodeIndex/Database/CSharpBaseListParser.cs @@ -0,0 +1,261 @@ +namespace CodeIndex.Database; + +internal enum CSharpBaseListProjection +{ + // DbReader resolves the complete type expression, including generic arguments. + TypeReference, + + // Metadata propagation needs only the qualified type name before generic/constructor syntax. + HeadIdentifier, +} + +internal static class CSharpBaseListParser +{ + internal static List Parse(string? signature, CSharpBaseListProjection projection) + { + var result = new List(); + if (string.IsNullOrWhiteSpace(signature)) + return result; + + var colonIndex = FindBaseListColon(signature); + if (colonIndex < 0) + return result; + + var state = new ScanState(); + var entryStart = colonIndex + 1; + for (var i = entryStart; i < signature.Length; i++) + { + if (!state.IsCodeCharacter(signature, ref i)) + continue; + + var ch = signature[i]; + if (state.IsTopLevel) + { + if (ch is '{' or ';') + { + AddProjectedEntry(result, signature[entryStart..i], projection); + return result; + } + + if (ch == ',') + { + AddProjectedEntry(result, signature[entryStart..i], projection); + entryStart = i + 1; + continue; + } + + if (IsWhereKeyword(signature, i)) + { + AddProjectedEntry(result, signature[entryStart..i], projection); + return result; + } + } + + state.UpdateDelimiterDepth(ch); + } + + AddProjectedEntry(result, signature[entryStart..], projection); + return result; + } + + private static int FindBaseListColon(string signature) + { + var state = new ScanState(); + for (var i = 0; i < signature.Length; i++) + { + if (!state.IsCodeCharacter(signature, ref i)) + continue; + + var ch = signature[i]; + if (state.IsTopLevel) + { + if (ch is '{' or ';' || IsWhereKeyword(signature, i)) + return -1; + + if (ch == ':' && !IsAliasQualifierColon(signature, i)) + return i; + } + + state.UpdateDelimiterDepth(ch); + } + + return -1; + } + + private static bool IsAliasQualifierColon(string signature, int index) + { + return (index > 0 && signature[index - 1] == ':') + || (index + 1 < signature.Length && signature[index + 1] == ':'); + } + + private static bool IsWhereKeyword(string signature, int index) + { + const string Keyword = "where"; + if (index + Keyword.Length > signature.Length + || !signature.AsSpan(index, Keyword.Length).SequenceEqual(Keyword)) + { + return false; + } + + if (index > 0 && IsIdentifierPart(signature[index - 1])) + return false; + + return index + Keyword.Length >= signature.Length + || !IsIdentifierPart(signature[index + Keyword.Length]); + } + + private static bool IsIdentifierPart(char ch) + { + return char.IsLetterOrDigit(ch) || ch is '_' or '@'; + } + + private static void AddProjectedEntry(List result, string rawEntry, CSharpBaseListProjection projection) + { + var entry = rawEntry.Trim(); + if (entry.Length == 0) + return; + + if (projection == CSharpBaseListProjection.TypeReference) + { + result.Add(entry); + return; + } + + if (projection != CSharpBaseListProjection.HeadIdentifier) + throw new ArgumentOutOfRangeException(nameof(projection), projection, null); + + var cut = entry.Length; + for (var i = 0; i < entry.Length; i++) + { + var ch = entry[i]; + if (ch is '<' or '(' || char.IsWhiteSpace(ch)) + { + cut = i; + break; + } + } + + if (cut > 0) + result.Add(entry[..cut]); + } + + private struct ScanState + { + private int _angleDepth; + private int _parenDepth; + private int _squareDepth; + private char _quote; + private bool _escaped; + private bool _verbatimString; + private bool _lineComment; + private bool _blockComment; + + internal readonly bool IsTopLevel => _angleDepth == 0 && _parenDepth == 0 && _squareDepth == 0; + + internal bool IsCodeCharacter(string text, ref int index) + { + var ch = text[index]; + if (_lineComment) + { + if (ch is '\r' or '\n') + _lineComment = false; + return false; + } + + if (_blockComment) + { + if (ch == '*' && index + 1 < text.Length && text[index + 1] == '/') + { + _blockComment = false; + index++; + } + return false; + } + + if (_quote != '\0') + { + if (_escaped) + { + _escaped = false; + return false; + } + + if (!_verbatimString && ch == '\\') + { + _escaped = true; + return false; + } + + if (_verbatimString && ch == '"' && index + 1 < text.Length && text[index + 1] == '"') + { + index++; + return false; + } + + if (ch == _quote) + { + _quote = '\0'; + _verbatimString = false; + } + return false; + } + + if (ch == '/' && index + 1 < text.Length) + { + if (text[index + 1] == '/') + { + _lineComment = true; + index++; + return false; + } + + if (text[index + 1] == '*') + { + _blockComment = true; + index++; + return false; + } + } + + if (ch is '"' or '\'') + { + _quote = ch; + _verbatimString = ch == '"' + && ((index > 0 && text[index - 1] == '@') + || (index > 1 && text[index - 2] == '@' && text[index - 1] == '$')); + return false; + } + + return true; + } + + internal void UpdateDelimiterDepth(char ch) + { + switch (ch) + { + case '<': + if (_parenDepth == 0 && _squareDepth == 0) + _angleDepth++; + break; + case '>': + if (_parenDepth == 0 && _squareDepth == 0 && _angleDepth > 0) + _angleDepth--; + break; + case '(': + _parenDepth++; + break; + case ')': + if (_parenDepth > 0) + _parenDepth--; + break; + case '[': + _squareDepth++; + break; + case ']': + if (_squareDepth > 0) + _squareDepth--; + break; + } + } + } +} diff --git a/src/CodeIndex/Database/DbContext.ConnectionFunctionRegistration.cs b/src/CodeIndex/Database/DbContext.ConnectionFunctionRegistration.cs new file mode 100644 index 0000000000..6a57736f65 --- /dev/null +++ b/src/CodeIndex/Database/DbContext.ConnectionFunctionRegistration.cs @@ -0,0 +1,323 @@ +using CodeIndex.Cli; +using CodeIndex.Diagnostics; +using CodeIndex.Indexer; +using CodeIndex.Models; +using Microsoft.Data.Sqlite; +using System.Runtime.CompilerServices; +using System.Text.Json; + +namespace CodeIndex.Database; + +public partial class DbContext +{ + private static class ConnectionFunctionRegistrar + { + private static readonly ConditionalWeakTable + CSharpCallableTypeKindLookups = new(); + private static readonly ConditionalWeakTable Registrations = new(); + private static readonly object RegistrationLock = new(); + private static readonly ConditionalWeakTable PartialRegistrations = new(); + private static readonly object PartialRegistrationLock = new(); + + internal static void Register(SqliteConnection connection) + { + lock (RegistrationLock) + { + if (Registrations.TryGetValue(connection, out _)) + return; + + RegisterDependencyAndNameNormalizationFunctions(connection); + RegisterCSharpReferenceShapeFunctions(connection); + RegisterCSharpPartialIdentityFunctions(connection); + RegisterCSharpFileAndBaseFunctions(connection); + RegisterSqlResolutionFunctions(connection); + Registrations.Add(connection, new object()); + } + } + + private static void RegisterDependencyAndNameNormalizationFunctions(SqliteConnection connection) + { + connection.CreateFunction( + "markdown_resolve_path", + (string? sourcePath, string? targetPath) => DbReader.ResolveMarkdownDependencyPath(sourcePath, targetPath)); + connection.CreateFunction( + "markdown_normalize_fragment", + (string? fragment) => fragment == null ? null : MarkdownAnchorIdentity.NormalizeHeadingFragment(fragment)); + connection.CreateFunction( + "python_import_resolves", + (string? sourcePath, string? targetPath, string? referenceName, string? referenceKind, string? context, long? columnNumber, string? signature) => + PythonImportBindingResolver.ResolvesDependency(sourcePath, targetPath, referenceName, referenceKind, context, columnNumber, signature)); + connection.CreateFunction( + "python_import_target_name", + (string? sourcePath, string? referenceName, string? context, long? columnNumber, string? signature) => + PythonImportBindingResolver.ResolveTargetName(sourcePath, referenceName, context, columnNumber, signature)); + connection.CreateFunction( + "sql_leaf_name", + (string? name) => string.IsNullOrWhiteSpace(name) ? null : SqlNameResolver.GetLeafName(name)); + connection.CreateFunction( + "sql_leaf_name_folded", + (string? name) => FoldSqlLeafName(name)); + connection.CreateFunction( + "codeindex_name_fold", + (string? name) => NameFold.Fold(name), + isDeterministic: true); + connection.CreateFunction( + "sql_normalize_name", + (string? name) => string.IsNullOrWhiteSpace(name) ? null : SqlNameResolver.NormalizeQualifiedName(name)); + connection.CreateFunction( + "sql_normalize_name_folded", + (string? name) => FoldSqlQualifiedName(name)); + connection.CreateFunction( + "sql_normalize_csharp_verbatim_name", + (string? text) => string.IsNullOrWhiteSpace(text) ? null : CSharpVerbatimNameNormalizer.Normalize(text)); + } + + private static string? FoldSqlLeafName(string? name) + { + if (string.IsNullOrWhiteSpace(name)) + return null; + var leafName = SqlNameResolver.GetLeafName(name); + return leafName.Length == 0 ? null : NameFold.Fold(leafName) ?? leafName; + } + + private static string? FoldSqlQualifiedName(string? name) + { + if (string.IsNullOrWhiteSpace(name)) + return null; + var normalizedName = SqlNameResolver.NormalizeQualifiedName(name); + return normalizedName.Length == 0 ? null : NameFold.Fold(normalizedName) ?? normalizedName; + } + + private static void RegisterCSharpReferenceShapeFunctions(SqliteConnection connection) + { + connection.CreateFunction( + "csharp_identifier_occurrence_count", + (string? text, string? identifier) => CountCSharpIdentifierOccurrences(text, identifier)); + connection.CreateFunction( + "csharp_identifier_occurrence_count_in_line_range", + (string? text, long? chunkStartLine, long? rangeStartLine, long? rangeEndLine, string? identifier) => + CountCSharpIdentifierOccurrencesInLineRange(text, chunkStartLine, rangeStartLine, rangeEndLine, identifier)); + connection.CreateFunction( + "csharp_reference_type_arity", + (string? context, string? identifier, long? columnNumber) => + CSharpTypeReferenceArity.GetReferenceArity(context, identifier, columnNumber)); + connection.CreateFunction( + "csharp_reference_is_member_receiver", + (string? context, string? identifier, long? columnNumber) => + CSharpTypeReferenceArity.IsMemberReceiver(context, identifier, columnNumber)); + connection.CreateFunction( + "csharp_definition_type_arity", + (string? signature, string? identifier, string? symbolKind) => + CSharpTypeReferenceArity.GetDefinitionArity(signature, identifier, symbolKind)); + connection.CreateFunction( + "csharp_constructor_parameter_count", + (string? signature, string? identifier, string? symbolKind) => + CSharpTypeReferenceArity.GetConstructorParameterCount(signature, identifier, symbolKind)); + } + + private static void RegisterCSharpPartialIdentityFunctions(SqliteConnection connection) + { + var typeKinds = CSharpCallableTypeKindLookups.GetValue( + connection, + static _ => new CSharpCallableTypeKindLookup()); + connection.CreateFunction( + "csharp_partial_callable_identity", + (string? signature, string? identifier, string? returnType) => + LogicalPartialSymbolGrouper.BuildCallableIdentity( + signature, identifier, returnType, containerQualifiedName: null, typeKinds: null), + isDeterministic: true); + connection.CreateFunction( + "csharp_partial_callable_identity", + (string? signature, string? identifier, string? returnType, string? containerQualifiedName) => + LogicalPartialSymbolGrouper.BuildCallableIdentity( + signature, identifier, returnType, containerQualifiedName, typeKinds)); + connection.CreateFunction( + "csharp_partial_callable_identity", + (string? signature, string? identifier, string? returnType, string? containerQualifiedName, long? symbolId) => + LogicalPartialSymbolGrouper.BuildCallableIdentity( + signature, identifier, returnType, containerQualifiedName, typeKinds, symbolId)); + connection.CreateFunction( + "csharp_partial_semantic_score", + (string? signature, string? symbolKind) => + LogicalPartialSymbolGrouper.GetSemanticScore(signature, symbolKind), + isDeterministic: true); + connection.CreateFunction( + "csharp_partial_declaration_identity", + (string? signature) => LogicalPartialSymbolGrouper.BuildCanonicalDeclarationIdentity(signature), + isDeterministic: true); + RegisterCSharpPartialDeclaration(connection); + } + + private static void RegisterCSharpFileAndBaseFunctions(SqliteConnection connection) + { + connection.CreateFunction( + "codeindex_generated_file_name", + (string? path) => FileIndexer.HasGeneratedCodeFileName(path ?? string.Empty), + isDeterministic: true); + connection.CreateFunction( + "csharp_invocation_argument_count", + (string? context, string? identifier, long? columnNumber) => + CSharpTypeReferenceArity.GetInvocationArgumentCount(context, identifier, columnNumber)); + connection.CreateFunction( + "csharp_definition_is_value_type", + (string? signature, string? symbolKind) => + CSharpTypeReferenceArity.IsValueTypeDeclaration(signature, symbolKind)); + connection.CreateFunction( + "csharp_base_identifiers_json", + (string? signature) => JsonSerializer.Serialize( + CSharpBaseListParser.Parse(signature, CSharpBaseListProjection.HeadIdentifier), + CliJsonSerializerContext.Default.ListString)); + connection.CreateFunction( + "csharp_base_name_folded", + (string? baseReference) => + { + var leaf = GetCSharpBaseReferenceLeaf(baseReference); + return leaf == null ? null : NameFold.Fold(leaf) ?? leaf; + }); + connection.CreateFunction( + "csharp_base_name", + (string? baseReference) => GetCSharpBaseReferenceLeaf(baseReference)); + connection.CreateFunction( + "csharp_base_reference_matches", + (string? baseReference, string? candidateName, string? candidateQualifiedName, string? derivingQualifiedName) => + CSharpBaseReferenceMatches(baseReference, candidateName, candidateQualifiedName, derivingQualifiedName) ? 1 : 0); + } + + private static void RegisterSqlResolutionFunctions(SqliteConnection connection) + { + connection.CreateFunction( + "sql_normalize_exact_source_name", + (string? text, string? lang) => string.IsNullOrWhiteSpace(text) ? null : ExactSourceSearchNormalizer.Normalize(text, lang)); + connection.CreateFunction( + "sql_segment_count", + (string? name) => string.IsNullOrWhiteSpace(name) ? (int?)null : SqlNameResolver.GetSegmentCount(name)); + RegisterSqlContextFunctions(connection); + RegisterSqlReferenceFunctions(connection); + } + + private static void RegisterSqlContextFunctions(SqliteConnection connection) + { + connection.CreateFunction( + "sql_context_has_name", + (string? context, string? query) => SqlNameResolver.ContextContainsQualifiedName(context, query) ? 1 : 0); + connection.CreateFunction( + "sql_context_has_name_folded", + (string? context, string? query) => SqlNameResolver.ContextContainsQualifiedNameFolded(context, query) ? 1 : 0); + connection.CreateFunction( + "sql_context_has_name_at", + (string? context, string? query, long? columnNumber) => + SqlNameResolver.ContextContainsQualifiedNameAtColumn(context, query, ToNullableInt(columnNumber)) ? 1 : 0); + connection.CreateFunction( + "sql_context_has_name_folded_at", + (string? context, string? query, long? columnNumber) => + SqlNameResolver.ContextContainsQualifiedNameFoldedAtColumn(context, query, ToNullableInt(columnNumber)) ? 1 : 0); + connection.CreateFunction( + "sql_context_like_name_at", + (string? context, string? query, long? columnNumber) => + SqlNameResolver.ContextContainsQualifiedNameLikeAtColumn(context, query, ToNullableInt(columnNumber)) ? 1 : 0); + connection.CreateFunction( + "sql_context_like_name_folded_at", + (string? context, string? query, long? columnNumber) => + SqlNameResolver.ContextContainsQualifiedNameLikeFoldedAtColumn(context, query, ToNullableInt(columnNumber)) ? 1 : 0); + } + + private static void RegisterSqlReferenceFunctions(SqliteConnection connection) + { + connection.CreateFunction( + "sql_resolve_reference_name", + (string? symbolName, string? context, string? containerName) => EmptyAsNull( + SqlNameResolver.ResolveReferenceName(symbolName, context, containerName))); + connection.CreateFunction( + "sql_resolve_reference_name_folded", + (string? symbolName, string? context, string? containerName) => EmptyAsNull( + SqlNameResolver.ResolveReferenceNameFolded(symbolName, context, containerName))); + connection.CreateFunction( + "sql_resolve_reference_name_at", + (string? symbolName, string? context, string? containerName, long? columnNumber) => EmptyAsNull( + SqlNameResolver.ResolveReferenceNameAtColumn(symbolName, context, containerName, ToNullableInt(columnNumber)))); + connection.CreateFunction( + "sql_resolve_reference_name_folded_at", + (string? symbolName, string? context, string? containerName, long? columnNumber) => EmptyAsNull( + SqlNameResolver.ResolveReferenceNameFoldedAtColumn(symbolName, context, containerName, ToNullableInt(columnNumber)))); + connection.CreateFunction( + "sql_resolve_reference_segment_count_at", + (string? symbolName, string? context, string? containerName, long? columnNumber) => (int?)( + SqlNameResolver.ResolveReferenceSegmentCountAtColumn(symbolName, context, containerName, ToNullableInt(columnNumber)) is var count + && count > 0 ? count : null)); + connection.CreateFunction( + "sql_reference_matches_target_at", + (string? symbolName, string? context, string? containerName, long? columnNumber, string? targetName) => + SqlNameResolver.ReferenceMatchesTargetAtColumn( + symbolName, context, containerName, ToNullableInt(columnNumber), targetName) ? 1 : 0); + connection.CreateFunction( + "sql_allow_leaf_fallback_at", + (string? symbolName, string? context, string? containerName, long? columnNumber) => + SqlNameResolver.AllowLeafFallbackAtColumn( + symbolName, context, containerName, ToNullableInt(columnNumber)) ? 1 : 0); + } + + private static string? EmptyAsNull(string value) => value.Length == 0 ? null : value; + + private static int? ToNullableInt(long? value) + => value is null || value < int.MinValue || value > int.MaxValue ? null : (int)value.Value; + + internal static void RegisterCSharpPartialDeclaration(SqliteConnection connection) + { + lock (PartialRegistrationLock) + { + // Raw DbReader connections need this function, while active statements make + // re-registration unsafe; keep its guard independent from the full family. + if (PartialRegistrations.TryGetValue(connection, out _)) + return; + + connection.CreateFunction( + "csharp_is_partial_declaration", + (string? signature, string? kind, string? name) => + LogicalPartialSymbolGrouper.ContainsPartialModifier(signature, kind, name), + isDeterministic: true); + PartialRegistrations.Add(connection, new object()); + } + } + + internal static void RefreshCSharpCallableTypeKinds( + SqliteConnection connection, + IReadOnlySet fileColumns, + IReadOnlySet symbolColumns, + IReadOnlyList? candidateQueries, + bool exact, + bool useFoldedNames) + { + var lookup = CSharpCallableTypeKindLookups.GetValue( + connection, + static _ => new CSharpCallableTypeKindLookup()); + lookup.RefreshIfChanged(connection, fileColumns, symbolColumns, candidateQueries, exact, useFoldedNames); + } + + internal static void RegisterWithRetry( + SqliteConnection connection, + Action? sleep, + int maxAttempts, + CancellationToken cancellationToken, + Action? registerConnectionFunctions) + { + if (maxAttempts <= 0) + throw new ArgumentOutOfRangeException(nameof(maxAttempts), maxAttempts, "Must be at least 1."); + + cancellationToken.ThrowIfCancellationRequested(); + registerConnectionFunctions ??= RegisterConnectionFunctions; + for (var attempt = 1; attempt <= maxAttempts; attempt++) + { + cancellationToken.ThrowIfCancellationRequested(); + try + { + registerConnectionFunctions(connection); + return; + } + catch (SqliteException ex) when (DbConnectionFactory.IsTransientBusyError(ex) && attempt < maxAttempts) + { + DbConnectionFactory.SleepBeforeRetry(50 * attempt, sleep, cancellationToken); + } + } + } + } +} diff --git a/src/CodeIndex/Database/DbContext.ConnectionFunctions.cs b/src/CodeIndex/Database/DbContext.ConnectionFunctions.cs index b33e80a8c3..f82adcf447 100644 --- a/src/CodeIndex/Database/DbContext.ConnectionFunctions.cs +++ b/src/CodeIndex/Database/DbContext.ConnectionFunctions.cs @@ -1,26 +1,10 @@ -using CodeIndex.Cli; -using CodeIndex.Diagnostics; using CodeIndex.Indexer; -using CodeIndex.Models; using Microsoft.Data.Sqlite; -using System.Globalization; -using System.Runtime.CompilerServices; -using System.Runtime.ExceptionServices; -using System.Text.Json; namespace CodeIndex.Database; public partial class DbContext : IDisposable { - private static readonly ConditionalWeakTable - CSharpCallableTypeKindLookups = new(); - private static readonly ConditionalWeakTable - ConnectionFunctionRegistrations = new(); - private static readonly object ConnectionFunctionRegistrationLock = new(); - private static readonly ConditionalWeakTable - CSharpPartialDeclarationFunctionRegistrations = new(); - private static readonly object CSharpPartialDeclarationFunctionRegistrationLock = new(); - private static SqliteConnection OpenArtifactPreservingQueryOnly(string dbPath) { var connection = CreateArtifactPreservingQueryOnlyConnection( @@ -34,272 +18,10 @@ private static SqliteConnection OpenArtifactPreservingQueryOnly(string dbPath) } internal static void RegisterConnectionFunctions(SqliteConnection connection) - { - lock (ConnectionFunctionRegistrationLock) - { - if (ConnectionFunctionRegistrations.TryGetValue(connection, out _)) - return; - - RegisterConnectionFunctionsCore(connection); - ConnectionFunctionRegistrations.Add(connection, new object()); - } - } - - private static void RegisterConnectionFunctionsCore(SqliteConnection connection) - { - static int? ToNullableInt(long? value) - => value is null || value < int.MinValue || value > int.MaxValue ? null : (int)value.Value; - - connection.CreateFunction( - "markdown_resolve_path", - (string? sourcePath, string? targetPath) => DbReader.ResolveMarkdownDependencyPath(sourcePath, targetPath)); - connection.CreateFunction( - "markdown_normalize_fragment", - (string? fragment) => fragment == null - ? null - : MarkdownAnchorIdentity.NormalizeHeadingFragment(fragment)); - connection.CreateFunction( - "python_import_resolves", - (string? sourcePath, string? targetPath, string? referenceName, string? referenceKind, string? context, long? columnNumber, string? signature) => - PythonImportBindingResolver.ResolvesDependency(sourcePath, targetPath, referenceName, referenceKind, context, columnNumber, signature)); - connection.CreateFunction( - "python_import_target_name", - (string? sourcePath, string? referenceName, string? context, long? columnNumber, string? signature) => - PythonImportBindingResolver.ResolveTargetName(sourcePath, referenceName, context, columnNumber, signature)); - connection.CreateFunction( - "sql_leaf_name", - (string? name) => string.IsNullOrWhiteSpace(name) ? null : SqlNameResolver.GetLeafName(name)); - connection.CreateFunction( - "sql_leaf_name_folded", - (string? name) => - { - if (string.IsNullOrWhiteSpace(name)) - return null; - - var leafName = SqlNameResolver.GetLeafName(name); - return leafName.Length == 0 ? null : NameFold.Fold(leafName) ?? leafName; - }); - connection.CreateFunction( - "codeindex_name_fold", - (string? name) => NameFold.Fold(name), - isDeterministic: true); - connection.CreateFunction( - "sql_normalize_name", - (string? name) => string.IsNullOrWhiteSpace(name) ? null : SqlNameResolver.NormalizeQualifiedName(name)); - connection.CreateFunction( - "sql_normalize_name_folded", - (string? name) => - { - if (string.IsNullOrWhiteSpace(name)) - return null; - - var normalizedName = SqlNameResolver.NormalizeQualifiedName(name); - return normalizedName.Length == 0 ? null : NameFold.Fold(normalizedName) ?? normalizedName; - }); - connection.CreateFunction( - "sql_normalize_csharp_verbatim_name", - (string? text) => string.IsNullOrWhiteSpace(text) ? null : CSharpVerbatimNameNormalizer.Normalize(text)); - connection.CreateFunction( - "csharp_identifier_occurrence_count", - (string? text, string? identifier) => CountCSharpIdentifierOccurrences(text, identifier)); - connection.CreateFunction( - "csharp_identifier_occurrence_count_in_line_range", - (string? text, long? chunkStartLine, long? rangeStartLine, long? rangeEndLine, string? identifier) => - CountCSharpIdentifierOccurrencesInLineRange( - text, - chunkStartLine, - rangeStartLine, - rangeEndLine, - identifier)); - connection.CreateFunction( - "csharp_reference_type_arity", - (string? context, string? identifier, long? columnNumber) => - CSharpTypeReferenceArity.GetReferenceArity(context, identifier, columnNumber)); - connection.CreateFunction( - "csharp_reference_is_member_receiver", - (string? context, string? identifier, long? columnNumber) => - CSharpTypeReferenceArity.IsMemberReceiver(context, identifier, columnNumber)); - connection.CreateFunction( - "csharp_definition_type_arity", - (string? signature, string? identifier, string? symbolKind) => - CSharpTypeReferenceArity.GetDefinitionArity(signature, identifier, symbolKind)); - connection.CreateFunction( - "csharp_constructor_parameter_count", - (string? signature, string? identifier, string? symbolKind) => - CSharpTypeReferenceArity.GetConstructorParameterCount(signature, identifier, symbolKind)); - var csharpCallableTypeKinds = CSharpCallableTypeKindLookups.GetValue( - connection, - static _ => new CSharpCallableTypeKindLookup()); - connection.CreateFunction( - "csharp_partial_callable_identity", - (string? signature, string? identifier, string? returnType) => - LogicalPartialSymbolGrouper.BuildCallableIdentity( - signature, - identifier, - returnType, - containerQualifiedName: null, - typeKinds: null), - isDeterministic: true); - connection.CreateFunction( - "csharp_partial_callable_identity", - (string? signature, string? identifier, string? returnType, string? containerQualifiedName) => - LogicalPartialSymbolGrouper.BuildCallableIdentity( - signature, - identifier, - returnType, - containerQualifiedName, - csharpCallableTypeKinds)); - connection.CreateFunction( - "csharp_partial_callable_identity", - (string? signature, string? identifier, string? returnType, string? containerQualifiedName, long? symbolId) => - LogicalPartialSymbolGrouper.BuildCallableIdentity( - signature, - identifier, - returnType, - containerQualifiedName, - csharpCallableTypeKinds, - symbolId)); - connection.CreateFunction( - "csharp_partial_semantic_score", - (string? signature, string? symbolKind) => - LogicalPartialSymbolGrouper.GetSemanticScore(signature, symbolKind), - isDeterministic: true); - connection.CreateFunction( - "csharp_partial_declaration_identity", - (string? signature) => - LogicalPartialSymbolGrouper.BuildCanonicalDeclarationIdentity(signature), - isDeterministic: true); - RegisterCSharpPartialDeclarationFunction(connection); - connection.CreateFunction( - "codeindex_generated_file_name", - (string? path) => FileIndexer.HasGeneratedCodeFileName(path ?? string.Empty), - isDeterministic: true); - connection.CreateFunction( - "csharp_invocation_argument_count", - (string? context, string? identifier, long? columnNumber) => - CSharpTypeReferenceArity.GetInvocationArgumentCount(context, identifier, columnNumber)); - connection.CreateFunction( - "csharp_definition_is_value_type", - (string? signature, string? symbolKind) => - CSharpTypeReferenceArity.IsValueTypeDeclaration(signature, symbolKind)); - connection.CreateFunction( - "csharp_base_identifiers_json", - (string? signature) => - JsonSerializer.Serialize( - DbWriter.ParseCSharpBaseIdentifiers(signature), - CliJsonSerializerContext.Default.ListString)); - connection.CreateFunction( - "csharp_base_name_folded", - (string? baseReference) => - { - var leaf = GetCSharpBaseReferenceLeaf(baseReference); - return leaf == null ? null : NameFold.Fold(leaf) ?? leaf; - }); - connection.CreateFunction( - "csharp_base_name", - (string? baseReference) => GetCSharpBaseReferenceLeaf(baseReference)); - connection.CreateFunction( - "csharp_base_reference_matches", - (string? baseReference, string? candidateName, string? candidateQualifiedName, string? derivingQualifiedName) => - CSharpBaseReferenceMatches( - baseReference, - candidateName, - candidateQualifiedName, - derivingQualifiedName) ? 1 : 0); - connection.CreateFunction( - "sql_normalize_exact_source_name", - (string? text, string? lang) => string.IsNullOrWhiteSpace(text) ? null : ExactSourceSearchNormalizer.Normalize(text, lang)); - connection.CreateFunction( - "sql_segment_count", - (string? name) => string.IsNullOrWhiteSpace(name) ? (int?)null : SqlNameResolver.GetSegmentCount(name)); - connection.CreateFunction( - "sql_context_has_name", - (string? context, string? query) => SqlNameResolver.ContextContainsQualifiedName(context, query) ? 1 : 0); - connection.CreateFunction( - "sql_context_has_name_folded", - (string? context, string? query) => SqlNameResolver.ContextContainsQualifiedNameFolded(context, query) ? 1 : 0); - connection.CreateFunction( - "sql_context_has_name_at", - (string? context, string? query, long? columnNumber) => - SqlNameResolver.ContextContainsQualifiedNameAtColumn(context, query, ToNullableInt(columnNumber)) ? 1 : 0); - connection.CreateFunction( - "sql_context_has_name_folded_at", - (string? context, string? query, long? columnNumber) => - SqlNameResolver.ContextContainsQualifiedNameFoldedAtColumn(context, query, ToNullableInt(columnNumber)) ? 1 : 0); - connection.CreateFunction( - "sql_context_like_name_at", - (string? context, string? query, long? columnNumber) => - SqlNameResolver.ContextContainsQualifiedNameLikeAtColumn(context, query, ToNullableInt(columnNumber)) ? 1 : 0); - connection.CreateFunction( - "sql_context_like_name_folded_at", - (string? context, string? query, long? columnNumber) => - SqlNameResolver.ContextContainsQualifiedNameLikeFoldedAtColumn(context, query, ToNullableInt(columnNumber)) ? 1 : 0); - connection.CreateFunction( - "sql_resolve_reference_name", - (string? symbolName, string? context, string? containerName) => - { - var resolved = SqlNameResolver.ResolveReferenceName(symbolName, context, containerName); - return resolved.Length == 0 ? null : resolved; - }); - connection.CreateFunction( - "sql_resolve_reference_name_folded", - (string? symbolName, string? context, string? containerName) => - { - var resolved = SqlNameResolver.ResolveReferenceNameFolded(symbolName, context, containerName); - return resolved.Length == 0 ? null : resolved; - }); - connection.CreateFunction( - "sql_resolve_reference_name_at", - (string? symbolName, string? context, string? containerName, long? columnNumber) => - { - var resolved = SqlNameResolver.ResolveReferenceNameAtColumn(symbolName, context, containerName, ToNullableInt(columnNumber)); - return resolved.Length == 0 ? null : resolved; - }); - connection.CreateFunction( - "sql_resolve_reference_name_folded_at", - (string? symbolName, string? context, string? containerName, long? columnNumber) => - { - var resolved = SqlNameResolver.ResolveReferenceNameFoldedAtColumn(symbolName, context, containerName, ToNullableInt(columnNumber)); - return resolved.Length == 0 ? null : resolved; - }); - connection.CreateFunction( - "sql_resolve_reference_segment_count_at", - (string? symbolName, string? context, string? containerName, long? columnNumber) => (int?)( - SqlNameResolver.ResolveReferenceSegmentCountAtColumn(symbolName, context, containerName, ToNullableInt(columnNumber)) is var segmentCount - && segmentCount > 0 - ? segmentCount - : null)); - connection.CreateFunction( - "sql_reference_matches_target_at", - (string? symbolName, string? context, string? containerName, long? columnNumber, string? targetName) => - SqlNameResolver.ReferenceMatchesTargetAtColumn(symbolName, context, containerName, ToNullableInt(columnNumber), targetName) ? 1 : 0); - connection.CreateFunction( - "sql_allow_leaf_fallback_at", - (string? symbolName, string? context, string? containerName, long? columnNumber) => - SqlNameResolver.AllowLeafFallbackAtColumn(symbolName, context, containerName, ToNullableInt(columnNumber)) ? 1 : 0); - } + => ConnectionFunctionRegistrar.Register(connection); internal static void RegisterCSharpPartialDeclarationFunction(SqliteConnection connection) - { - lock (CSharpPartialDeclarationFunctionRegistrationLock) - { - // DbReader also accepts caller-owned raw connections, so it must ensure this - // function exists. DbContext connections already registered it, however, and - // SQLite rejects replacing a function while any statement is active. - // DbReader は caller-owned raw connection も受け付けるため、この function を - // 保証する。一方 DbContext connection では登録済みであり、active statement 中の - // 再登録を SQLite が拒否するため、connection 単位で一度だけ登録する。 - if (CSharpPartialDeclarationFunctionRegistrations.TryGetValue(connection, out _)) - return; - - connection.CreateFunction( - "csharp_is_partial_declaration", - (string? signature, string? kind, string? name) => - LogicalPartialSymbolGrouper.ContainsPartialModifier(signature, kind, name), - isDeterministic: true); - CSharpPartialDeclarationFunctionRegistrations.Add(connection, new object()); - } - } + => ConnectionFunctionRegistrar.RegisterCSharpPartialDeclaration(connection); internal static void RefreshCSharpCallableTypeKinds( SqliteConnection connection, @@ -308,18 +30,13 @@ internal static void RefreshCSharpCallableTypeKinds( IReadOnlyList? candidateQueries = null, bool exact = false, bool useFoldedNames = false) - { - var lookup = CSharpCallableTypeKindLookups.GetValue( - connection, - static _ => new CSharpCallableTypeKindLookup()); - lookup.RefreshIfChanged( + => ConnectionFunctionRegistrar.RefreshCSharpCallableTypeKinds( connection, fileColumns, symbolColumns, candidateQueries, exact, useFoldedNames); - } internal static int CountCSharpIdentifierOccurrences(string? text, string? identifier) { @@ -835,25 +552,10 @@ internal static void RegisterConnectionFunctionsWithRetry( int maxAttempts = 5, CancellationToken cancellationToken = default, Action? registerConnectionFunctions = null) - { - if (maxAttempts <= 0) - throw new ArgumentOutOfRangeException(nameof(maxAttempts), maxAttempts, "Must be at least 1."); - - cancellationToken.ThrowIfCancellationRequested(); - registerConnectionFunctions ??= RegisterConnectionFunctions; - for (var attempt = 1; attempt <= maxAttempts; attempt++) - { - cancellationToken.ThrowIfCancellationRequested(); - try - { - registerConnectionFunctions(connection); - return; - } - catch (SqliteException ex) when (DbConnectionFactory.IsTransientBusyError(ex) && attempt < maxAttempts) - { - DbConnectionFactory.SleepBeforeRetry(50 * attempt, sleep, cancellationToken); - } - } - } - + => ConnectionFunctionRegistrar.RegisterWithRetry( + connection, + sleep, + maxAttempts, + cancellationToken, + registerConnectionFunctions); } diff --git a/src/CodeIndex/Database/DbContext.ExistingDatabaseValidation.cs b/src/CodeIndex/Database/DbContext.ExistingDatabaseValidation.cs new file mode 100644 index 0000000000..45be44bb25 --- /dev/null +++ b/src/CodeIndex/Database/DbContext.ExistingDatabaseValidation.cs @@ -0,0 +1,200 @@ +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +internal enum ExistingCodeIndexDbValidationFailure +{ + None, + Missing, + Inaccessible, + InvalidTarget, + InvalidDatabase, + SchemaTooNew, + Exception, +} + +public partial class DbContext +{ + internal const string DatabaseOpenMissingCategory = "missing_database"; + internal const string DatabaseOpenPermissionCategory = "permission_denied"; + internal const string DatabaseOpenSidecarCategory = "sidecar_failure"; + internal const string DatabaseOpenInvalidUriCategory = "invalid_uri"; + internal const string DatabaseOpenUnknownCategory = "unknown_open_failure"; + private const int SqliteCantOpenDirtyWal = 14 | (5 << 8); + + public static bool TryValidateExistingCodeIndexDb( + string dbPath, + out string message, + out bool isNotFound, + CancellationToken cancellationToken = default) + => TryValidateExistingCodeIndexDb( + dbPath, + requireWritable: true, + requireSupportedUserVersion: false, + out message, + out isNotFound, + out _, + cancellationToken); + + internal static bool TryValidateExistingCodeIndexDb( + string dbPath, + bool requireWritable, + bool requireSupportedUserVersion, + out string message, + out bool isNotFound, + out bool isSchemaTooNew, + CancellationToken cancellationToken = default) + => TryValidateExistingCodeIndexDb( + dbPath, + requireWritable, + requireSupportedUserVersion, + out message, + out isNotFound, + out isSchemaTooNew, + out _, + out _, + cancellationToken); + + internal static bool TryValidateExistingCodeIndexDb( + string dbPath, + bool requireWritable, + bool requireSupportedUserVersion, + out string message, + out bool isNotFound, + out bool isSchemaTooNew, + out ExistingCodeIndexDbValidationFailure validationFailure, + out Exception? validationException, + CancellationToken cancellationToken = default) + => TryValidateExistingCodeIndexDb( + dbPath, + openTarget => + { + var mode = requireWritable + ? SqliteConnectionPolicyMode.ReadWrite + : SqliteConnectionPolicyMode.ReadOnly; + return new SqliteConnection(SqliteConnectionPolicy.BuildConnectionString(openTarget, mode)); + }, + static connection => connection.Open(), + sleep: null, + requireWritable, + requireSupportedUserVersion, + out message, + out isNotFound, + out isSchemaTooNew, + out validationFailure, + out validationException, + cancellationToken); + + internal static bool TryValidateExistingCodeIndexDb( + string dbPath, + Func createConnection, + Action openConnection, + Action? sleep, + out string message, + out bool isNotFound, + CancellationToken cancellationToken = default) + => TryValidateExistingCodeIndexDb( + dbPath, + createConnection, + openConnection, + sleep, + requireWritable: true, + requireSupportedUserVersion: false, + out message, + out isNotFound, + out _, + out _, + out _, + cancellationToken); + + internal static bool TryValidateExistingCodeIndexDb( + string dbPath, + Func createConnection, + Action openConnection, + Action? sleep, + out string message, + out bool isNotFound, + out Exception? validationException, + CancellationToken cancellationToken = default) + => TryValidateExistingCodeIndexDb( + dbPath, + createConnection, + openConnection, + sleep, + requireWritable: true, + requireSupportedUserVersion: false, + out message, + out isNotFound, + out _, + out _, + out validationException, + cancellationToken); + + private static bool TryValidateExistingCodeIndexDb( + string dbPath, + Func createConnection, + Action openConnection, + Action? sleep, + bool requireWritable, + bool requireSupportedUserVersion, + out string message, + out bool isNotFound, + out bool isSchemaTooNew, + out ExistingCodeIndexDbValidationFailure validationFailure, + out Exception? validationException, + CancellationToken cancellationToken = default) + => ProjectValidationResult( + ExistingCodeIndexDbValidator.Validate(new ValidationRequest( + dbPath, + createConnection, + openConnection, + sleep, + requireWritable, + requireSupportedUserVersion, + cancellationToken)), + out message, + out isNotFound, + out isSchemaTooNew, + out validationFailure, + out validationException); + + private static bool ProjectValidationResult( + ValidationResult result, + out string message, + out bool isNotFound, + out bool isSchemaTooNew, + out ExistingCodeIndexDbValidationFailure validationFailure, + out Exception? validationException) + { + message = result.Message; + isNotFound = result.IsNotFound; + isSchemaTooNew = result.IsSchemaTooNew; + validationFailure = result.Failure; + validationException = result.Exception; + return result.IsValid; + } + + internal static string ClassifyCantOpenFailure(string dbPath, int sqliteExtendedErrorCode) + => ExistingCodeIndexDbValidator.ClassifyCantOpenFailure(dbPath, sqliteExtendedErrorCode); + + private readonly record struct ValidationRequest( + string DbPath, + Func CreateConnection, + Action OpenConnection, + Action? Sleep, + bool RequireWritable, + bool RequireSupportedUserVersion, + CancellationToken CancellationToken); + + private readonly record struct ValidationResult( + bool IsValid, + string Message, + bool IsNotFound, + bool IsSchemaTooNew, + ExistingCodeIndexDbValidationFailure Failure, + Exception? Exception) + { + internal static ValidationResult Valid { get; } = new( + true, string.Empty, false, false, ExistingCodeIndexDbValidationFailure.None, null); + } +} diff --git a/src/CodeIndex/Database/DbContext.ExistingDatabaseValidator.cs b/src/CodeIndex/Database/DbContext.ExistingDatabaseValidator.cs new file mode 100644 index 0000000000..e696bc88ac --- /dev/null +++ b/src/CodeIndex/Database/DbContext.ExistingDatabaseValidator.cs @@ -0,0 +1,253 @@ +using CodeIndex.Diagnostics; +using CodeIndex.Indexer; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbContext +{ + private static class ExistingCodeIndexDbValidator + { + internal static ValidationResult Validate(ValidationRequest request) + { + request.CancellationToken.ThrowIfCancellationRequested(); + var targetFailure = ValidateTarget(request, out var openTarget); + if (targetFailure.HasValue) + return targetFailure.Value; + + var preflightFailure = ValidateFileSystemTarget(request.DbPath, openTarget); + if (preflightFailure.HasValue) + return preflightFailure.Value; + + try + { + return OpenAndInspect(request, openTarget); + } + catch (Exception ex) when (ex is SqliteException or CodeIndexException) + { + return ProjectException(request.DbPath, openTarget, ex); + } + } + + private static ValidationResult? ValidateTarget(ValidationRequest request, out string openTarget) + { + openTarget = request.DbPath; + if (!SqliteFileUri.StartsWithFileScheme(request.DbPath)) + return null; + + if (!SqliteFileUri.TryValidateBounds(request.DbPath, out var boundsError)) + { + return Failure( + ExistingCodeIndexDbValidationFailure.InvalidTarget, + FormatDatabaseOpenFailure( + DatabaseOpenInvalidUriCategory, + request.DbPath, + boundsError?.Message ?? "Invalid SQLite file URI.")); + } + + if (request.RequireWritable && SqliteFileUri.RequestsReadOnly(request.DbPath)) + { + return Failure( + ExistingCodeIndexDbValidationFailure.Inaccessible, + $"database must be writable: {request.DbPath}"); + } + + if (!TryGetLocalPath(request.DbPath, out var normalized, out var pathFailureReason) + || normalized == null) + { + return Failure( + ExistingCodeIndexDbValidationFailure.InvalidTarget, + FormatDatabaseOpenFailure( + DatabaseOpenInvalidUriCategory, + request.DbPath, + pathFailureReason)); + } + + openTarget = normalized; + return null; + } + + private static ValidationResult? ValidateFileSystemTarget(string dbPath, string openTarget) + { + var preflight = ProbeDatabasePath(openTarget); + if (preflight is not (DatabasePathProbe.Missing + or DatabasePathProbe.PermissionDenied + or DatabasePathProbe.Directory)) + { + return null; + } + + var category = preflight switch + { + DatabasePathProbe.Missing => DatabaseOpenMissingCategory, + DatabasePathProbe.PermissionDenied => DatabaseOpenPermissionCategory, + _ => DatabaseOpenUnknownCategory, + }; + var failure = preflight switch + { + DatabasePathProbe.Missing => ExistingCodeIndexDbValidationFailure.Missing, + DatabasePathProbe.PermissionDenied => ExistingCodeIndexDbValidationFailure.Inaccessible, + _ => ExistingCodeIndexDbValidationFailure.InvalidTarget, + }; + return Failure( + failure, + FormatDatabaseOpenFailure(category, dbPath), + isNotFound: category == DatabaseOpenMissingCategory); + } + + private static ValidationResult OpenAndInspect(ValidationRequest request, string openTarget) + { + using var connection = request.RequireWritable + ? OpenSqliteConnectionWithRetry( + () => request.CreateConnection(openTarget), + request.OpenConnection, + request.Sleep, + dbPath: request.DbPath, + cancellationToken: request.CancellationToken) + : OpenArtifactPreservingQueryOnly(request.DbPath); + return InspectSchema(connection, request); + } + + private static ValidationResult InspectSchema(SqliteConnection connection, ValidationRequest request) + { + using var command = connection.CreateCommand(); + command.CommandText = SqliteCommandPolicy.PragmaSql("application_id"); + if (SqliteCommandPolicy.ReadInt64Scalar(command, "pragma application_id") != ApplicationId) + return InvalidDatabase(request.DbPath); + + if (request.RequireSupportedUserVersion) + { + command.CommandText = SqliteCommandPolicy.PragmaSql("user_version"); + var userVersion = SqliteCommandPolicy.ReadInt32Scalar(command, "pragma user_version"); + if ((userVersion & ~CurrentSchemaVersion) != 0) + { + return Failure( + ExistingCodeIndexDbValidationFailure.SchemaTooNew, + $"database was written by a newer cdidx schema stamp (user_version {userVersion}); this binary supports up to {CurrentSchemaVersion}: {request.DbPath}", + isSchemaTooNew: true); + } + } + + command.CommandText = "SELECT name FROM sqlite_master WHERE type = 'table'"; + using var reader = command.ExecuteReader(); + var tables = new HashSet(StringComparer.Ordinal); + while (reader.Read()) + tables.Add(reader.GetString(0)); + + return RequiredCodeIndexTables.All(tables.Contains) + ? ValidationResult.Valid + : InvalidDatabase(request.DbPath); + } + + private static ValidationResult ProjectException(string dbPath, string openTarget, Exception exception) + { + if (exception is SqliteException { SqliteErrorCode: 14 } cantOpen) + { + var category = ClassifyCantOpenFailure(openTarget, cantOpen.SqliteExtendedErrorCode); + return Failure( + ExistingCodeIndexDbValidationFailure.Exception, + FormatDatabaseOpenFailure(category, dbPath), + isNotFound: category == DatabaseOpenMissingCategory, + exception: exception); + } + + return exception is CodeIndexException codeIndexException + ? Failure(ExistingCodeIndexDbValidationFailure.Exception, codeIndexException.Message, exception: exception) + : Failure(ExistingCodeIndexDbValidationFailure.Exception, InvalidDatabaseMessage(dbPath), exception: exception); + } + + internal static string ClassifyCantOpenFailure(string dbPath, int sqliteExtendedErrorCode) + { + if (sqliteExtendedErrorCode == SqliteCantOpenDirtyWal) + return DatabaseOpenSidecarCategory; + + return ProbeDatabasePath(dbPath) switch + { + DatabasePathProbe.Missing => DatabaseOpenMissingCategory, + DatabasePathProbe.PermissionDenied => DatabaseOpenPermissionCategory, + _ when HasInaccessibleSqliteSidecar(dbPath) => DatabaseOpenSidecarCategory, + _ => DatabaseOpenUnknownCategory, + }; + } + + private static bool HasInaccessibleSqliteSidecar(string dbPath) + => ProbeDatabasePath(dbPath + "-wal") == DatabasePathProbe.PermissionDenied + || ProbeDatabasePath(dbPath + "-shm") == DatabasePathProbe.PermissionDenied; + + private static DatabasePathProbe ProbeDatabasePath(string path) + { + try + { + var normalizedPath = LongPath.EnsureWindowsPrefix(path); + if ((File.GetAttributes(normalizedPath) & FileAttributes.Directory) != 0) + return DatabasePathProbe.Directory; + + using var stream = new FileStream( + normalizedPath, + FileMode.Open, + FileAccess.Read, + FileShare.ReadWrite | FileShare.Delete, + bufferSize: 1, + FileOptions.RandomAccess); + return DatabasePathProbe.Readable; + } + catch (Exception ex) when (ex is FileNotFoundException or DirectoryNotFoundException) + { + return DatabasePathProbe.Missing; + } + catch (Exception ex) when (ex is UnauthorizedAccessException or System.Security.SecurityException) + { + return DatabasePathProbe.PermissionDenied; + } + catch (IOException) + { + return DatabasePathProbe.Unknown; + } + } + + private static string FormatDatabaseOpenFailure(string category, string dbPath, string? detail = null) + { + var displayPath = dbPath; + if (SqliteFileUri.StartsWithFileScheme(dbPath)) + { + var queryIndex = dbPath.IndexOf('?', StringComparison.Ordinal); + if (queryIndex >= 0) + displayPath = dbPath[..queryIndex]; + } + + var pathLabel = DiagnosticSanitizer.ForPath(displayPath); + var sanitizedDetail = string.IsNullOrWhiteSpace(detail) + ? null + : DiagnosticSanitizer.ForMessage(detail); + var prefix = category == DatabaseOpenMissingCategory + ? $"database not found [{category}]" + : $"database open failed [{category}]"; + return sanitizedDetail == null + ? $"{prefix}: {pathLabel}" + : $"{prefix}: {pathLabel}; {sanitizedDetail}"; + } + + private static ValidationResult InvalidDatabase(string dbPath) + => Failure(ExistingCodeIndexDbValidationFailure.InvalidDatabase, InvalidDatabaseMessage(dbPath)); + + private static string InvalidDatabaseMessage(string dbPath) + => $"database is not an existing CodeIndex DB: {dbPath}"; + + private static ValidationResult Failure( + ExistingCodeIndexDbValidationFailure failure, + string message, + bool isNotFound = false, + bool isSchemaTooNew = false, + Exception? exception = null) + => new(false, message, isNotFound, isSchemaTooNew, failure, exception); + + private enum DatabasePathProbe + { + Readable, + Missing, + PermissionDenied, + Directory, + Unknown, + } + } +} diff --git a/src/CodeIndex/Database/DbContext.cs b/src/CodeIndex/Database/DbContext.cs index edc3051577..65e879e341 100644 --- a/src/CodeIndex/Database/DbContext.cs +++ b/src/CodeIndex/Database/DbContext.cs @@ -1,24 +1,10 @@ using CodeIndex.Cli; -using CodeIndex.Diagnostics; using CodeIndex.Indexer; using CodeIndex.Models; using Microsoft.Data.Sqlite; -using System.Globalization; -using System.Runtime.ExceptionServices; namespace CodeIndex.Database; -internal enum ExistingCodeIndexDbValidationFailure -{ - None, - Missing, - Inaccessible, - InvalidTarget, - InvalidDatabase, - SchemaTooNew, - Exception, -} - /// /// Manages SQLite connection and schema initialization. /// SQLite接続とスキーマ初期化を管理する。 @@ -33,12 +19,6 @@ public partial class DbContext : IDisposable public const string CacheSizeEnvironmentVariable = "CDIDX_SQLITE_CACHE_KB"; public const string MmapSizeEnvironmentVariable = "CDIDX_SQLITE_MMAP_BYTES"; public const string BusyTimeoutEnvironmentVariable = "CDIDX_SQLITE_BUSY_TIMEOUT_MS"; - internal const string DatabaseOpenMissingCategory = "missing_database"; - internal const string DatabaseOpenPermissionCategory = "permission_denied"; - internal const string DatabaseOpenSidecarCategory = "sidecar_failure"; - internal const string DatabaseOpenInvalidUriCategory = "invalid_uri"; - internal const string DatabaseOpenUnknownCategory = "unknown_open_failure"; - private const int SqliteCantOpenDirtyWal = 14 | (5 << 8); public const int DefaultWalAutocheckpointPages = 1000; public const string DefaultSynchronousMode = "NORMAL"; public const string SymbolExtractorVersionMetaPrefix = "symbol_extractor_version_"; @@ -440,336 +420,6 @@ internal PreparedCommandCache PreparedCommands _connection, PreparedCommandCache.ReadCapacityFromEnvironment()); - public static bool TryValidateExistingCodeIndexDb( - string dbPath, - out string message, - out bool isNotFound, - CancellationToken cancellationToken = default) - => TryValidateExistingCodeIndexDb( - dbPath, - requireWritable: true, - requireSupportedUserVersion: false, - out message, - out isNotFound, - out _, - cancellationToken); - - internal static bool TryValidateExistingCodeIndexDb( - string dbPath, - bool requireWritable, - bool requireSupportedUserVersion, - out string message, - out bool isNotFound, - out bool isSchemaTooNew, - CancellationToken cancellationToken = default) - => TryValidateExistingCodeIndexDb( - dbPath, - requireWritable, - requireSupportedUserVersion, - out message, - out isNotFound, - out isSchemaTooNew, - out _, - out _, - cancellationToken); - - internal static bool TryValidateExistingCodeIndexDb( - string dbPath, - bool requireWritable, - bool requireSupportedUserVersion, - out string message, - out bool isNotFound, - out bool isSchemaTooNew, - out ExistingCodeIndexDbValidationFailure validationFailure, - out Exception? validationException, - CancellationToken cancellationToken = default) - => TryValidateExistingCodeIndexDb(dbPath, openTarget => - { - var mode = requireWritable ? SqliteConnectionPolicyMode.ReadWrite : SqliteConnectionPolicyMode.ReadOnly; - return new SqliteConnection(SqliteConnectionPolicy.BuildConnectionString(openTarget, mode)); - }, - static connection => connection.Open(), - null, - requireWritable, - requireSupportedUserVersion, - out message, - out isNotFound, - out isSchemaTooNew, - out validationFailure, - out validationException, - cancellationToken); - - internal static bool TryValidateExistingCodeIndexDb( - string dbPath, - Func createConnection, - Action openConnection, - Action? sleep, - out string message, - out bool isNotFound, - CancellationToken cancellationToken = default) - => TryValidateExistingCodeIndexDb( - dbPath, - createConnection, - openConnection, - sleep, - requireWritable: true, - requireSupportedUserVersion: false, - out message, - out isNotFound, - out _, - out _, - out _, - cancellationToken); - - internal static bool TryValidateExistingCodeIndexDb( - string dbPath, - Func createConnection, - Action openConnection, - Action? sleep, - out string message, - out bool isNotFound, - out Exception? validationException, - CancellationToken cancellationToken = default) - => TryValidateExistingCodeIndexDb( - dbPath, - createConnection, - openConnection, - sleep, - requireWritable: true, - requireSupportedUserVersion: false, - out message, - out isNotFound, - out _, - out _, - out validationException, - cancellationToken); - - private static bool TryValidateExistingCodeIndexDb( - string dbPath, - Func createConnection, - Action openConnection, - Action? sleep, - bool requireWritable, - bool requireSupportedUserVersion, - out string message, - out bool isNotFound, - out bool isSchemaTooNew, - out ExistingCodeIndexDbValidationFailure validationFailure, - out Exception? validationException, - CancellationToken cancellationToken = default) - { - message = string.Empty; - isNotFound = false; - isSchemaTooNew = false; - validationFailure = ExistingCodeIndexDbValidationFailure.None; - validationException = null; - cancellationToken.ThrowIfCancellationRequested(); - - if (SqliteFileUri.StartsWithFileScheme(dbPath) && !SqliteFileUri.TryValidateBounds(dbPath, out var boundsError)) - { - validationFailure = ExistingCodeIndexDbValidationFailure.InvalidTarget; - message = FormatDatabaseOpenFailure( - DatabaseOpenInvalidUriCategory, - dbPath, - boundsError?.Message ?? "Invalid SQLite file URI."); - return false; - } - - if (requireWritable && SqliteFileUri.StartsWithFileScheme(dbPath) && SqliteFileUri.RequestsReadOnly(dbPath)) - { - validationFailure = ExistingCodeIndexDbValidationFailure.Inaccessible; - message = $"database must be writable: {dbPath}"; - return false; - } - - var openTarget = dbPath; - if (SqliteFileUri.StartsWithFileScheme(dbPath)) - { - if (!TryGetLocalPath(dbPath, out var normalized, out var pathFailureReason) - || normalized == null) - { - validationFailure = ExistingCodeIndexDbValidationFailure.InvalidTarget; - message = FormatDatabaseOpenFailure( - DatabaseOpenInvalidUriCategory, - dbPath, - pathFailureReason); - return false; - } - - openTarget = normalized; - } - - var preflight = ProbeDatabasePath(openTarget); - if (preflight is DatabasePathProbe.Missing or DatabasePathProbe.PermissionDenied or DatabasePathProbe.Directory) - { - var category = preflight switch - { - DatabasePathProbe.Missing => DatabaseOpenMissingCategory, - DatabasePathProbe.PermissionDenied => DatabaseOpenPermissionCategory, - _ => DatabaseOpenUnknownCategory, - }; - message = FormatDatabaseOpenFailure(category, dbPath); - isNotFound = category == DatabaseOpenMissingCategory; - validationFailure = preflight switch - { - DatabasePathProbe.Missing => ExistingCodeIndexDbValidationFailure.Missing, - DatabasePathProbe.PermissionDenied => ExistingCodeIndexDbValidationFailure.Inaccessible, - _ => ExistingCodeIndexDbValidationFailure.InvalidTarget, - }; - return false; - } - - try - { - using var connection = requireWritable - ? OpenSqliteConnectionWithRetry( - () => createConnection(openTarget), - openConnection, - sleep, - dbPath: dbPath, - cancellationToken: cancellationToken) - : OpenArtifactPreservingQueryOnly(dbPath); - - using var cmd = connection.CreateCommand(); - cmd.CommandText = SqliteCommandPolicy.PragmaSql("application_id"); - if (SqliteCommandPolicy.ReadInt64Scalar(cmd, "pragma application_id") != ApplicationId) - { - validationFailure = ExistingCodeIndexDbValidationFailure.InvalidDatabase; - message = $"database is not an existing CodeIndex DB: {dbPath}"; - return false; - } - - if (requireSupportedUserVersion) - { - cmd.CommandText = SqliteCommandPolicy.PragmaSql("user_version"); - var userVersion = SqliteCommandPolicy.ReadInt32Scalar(cmd, "pragma user_version"); - var unknownBits = userVersion & ~CurrentSchemaVersion; - if (unknownBits != 0) - { - isSchemaTooNew = true; - validationFailure = ExistingCodeIndexDbValidationFailure.SchemaTooNew; - message = $"database was written by a newer cdidx schema stamp (user_version {userVersion}); this binary supports up to {CurrentSchemaVersion}: {dbPath}"; - return false; - } - } - - cmd.CommandText = "SELECT name FROM sqlite_master WHERE type = 'table'"; - using var reader = cmd.ExecuteReader(); - var tables = new HashSet(StringComparer.Ordinal); - while (reader.Read()) - tables.Add(reader.GetString(0)); - - if (RequiredCodeIndexTables.All(tables.Contains)) - return true; - - validationFailure = ExistingCodeIndexDbValidationFailure.InvalidDatabase; - message = $"database is not an existing CodeIndex DB: {dbPath}"; - return false; - } - catch (SqliteException ex) when (ex.SqliteErrorCode is 14) - { - validationFailure = ExistingCodeIndexDbValidationFailure.Exception; - validationException = ex; - var category = ClassifyCantOpenFailure(openTarget, ex.SqliteExtendedErrorCode); - message = FormatDatabaseOpenFailure(category, dbPath); - isNotFound = category == DatabaseOpenMissingCategory; - return false; - } - catch (SqliteException ex) - { - validationFailure = ExistingCodeIndexDbValidationFailure.Exception; - validationException = ex; - message = $"database is not an existing CodeIndex DB: {dbPath}"; - return false; - } - catch (CodeIndexException ex) - { - validationFailure = ExistingCodeIndexDbValidationFailure.Exception; - validationException = ex; - message = ex.Message; - return false; - } - } - - internal static string ClassifyCantOpenFailure(string dbPath, int sqliteExtendedErrorCode) - { - if (sqliteExtendedErrorCode == SqliteCantOpenDirtyWal) - return DatabaseOpenSidecarCategory; - - return ProbeDatabasePath(dbPath) switch - { - DatabasePathProbe.Missing => DatabaseOpenMissingCategory, - DatabasePathProbe.PermissionDenied => DatabaseOpenPermissionCategory, - _ when HasInaccessibleSqliteSidecar(dbPath) => DatabaseOpenSidecarCategory, - _ => DatabaseOpenUnknownCategory, - }; - } - - private static bool HasInaccessibleSqliteSidecar(string dbPath) - => ProbeDatabasePath(dbPath + "-wal") == DatabasePathProbe.PermissionDenied - || ProbeDatabasePath(dbPath + "-shm") == DatabasePathProbe.PermissionDenied; - - private static DatabasePathProbe ProbeDatabasePath(string path) - { - try - { - var attributes = File.GetAttributes(LongPath.EnsureWindowsPrefix(path)); - if ((attributes & FileAttributes.Directory) != 0) - return DatabasePathProbe.Directory; - - using var stream = new FileStream( - LongPath.EnsureWindowsPrefix(path), - FileMode.Open, - FileAccess.Read, - FileShare.ReadWrite | FileShare.Delete, - bufferSize: 1, - FileOptions.RandomAccess); - return DatabasePathProbe.Readable; - } - catch (Exception ex) when (ex is FileNotFoundException or DirectoryNotFoundException) - { - return DatabasePathProbe.Missing; - } - catch (Exception ex) when (ex is UnauthorizedAccessException or System.Security.SecurityException) - { - return DatabasePathProbe.PermissionDenied; - } - catch (IOException) - { - return DatabasePathProbe.Unknown; - } - } - - private static string FormatDatabaseOpenFailure(string category, string dbPath, string? detail = null) - { - var displayPath = dbPath; - if (SqliteFileUri.StartsWithFileScheme(dbPath)) - { - var queryIndex = dbPath.IndexOf('?', StringComparison.Ordinal); - if (queryIndex >= 0) - displayPath = dbPath[..queryIndex]; - } - var pathLabel = DiagnosticSanitizer.ForPath(displayPath); - var sanitizedDetail = string.IsNullOrWhiteSpace(detail) - ? null - : DiagnosticSanitizer.ForMessage(detail); - var prefix = category == DatabaseOpenMissingCategory - ? $"database not found [{category}]" - : $"database open failed [{category}]"; - return sanitizedDetail == null - ? $"{prefix}: {pathLabel}" - : $"{prefix}: {pathLabel}; {sanitizedDetail}"; - } - - private enum DatabasePathProbe - { - Readable, - Missing, - PermissionDenied, - Directory, - Unknown, - } - public DbContext(DbOpenIntent openIntent, string dbPath, CancellationToken cancellationToken = default) : this( openIntent, diff --git a/src/CodeIndex/Database/DbReader.CSharpResolution.cs b/src/CodeIndex/Database/DbReader.CSharpResolution.cs index de9cb68992..47a4bbb40d 100644 --- a/src/CodeIndex/Database/DbReader.CSharpResolution.cs +++ b/src/CodeIndex/Database/DbReader.CSharpResolution.cs @@ -68,17 +68,9 @@ private HashSet GetActiveCSharpTypeNamespaces(string path, int lineNumbe return cachedActiveNamespaces; } - if (!_csharpNamespaceScopesByPath.TryGetValue(path, out var namespaceScopes)) - { - namespaceScopes = LoadCSharpNamespaceScopes(path); - _csharpNamespaceScopesByPath[path] = namespaceScopes; - } - - if (!_csharpUsingNamespaceScopesByPath.TryGetValue(path, out var usingNamespaceScopes)) - { - usingNamespaceScopes = LoadCSharpUsingNamespaceScopes(path); - _csharpUsingNamespaceScopesByPath[path] = usingNamespaceScopes; - } + var usingCatalog = GetCSharpPathUsingCatalog(path); + var namespaceScopes = usingCatalog.NamespaceDeclarations; + var usingNamespaceScopes = usingCatalog.NamespaceImports; var activeNamespaces = new HashSet(StringComparer.Ordinal); foreach (var scope in namespaceScopes) @@ -267,7 +259,10 @@ private static void AddCSharpBaseListDispatchNamesFromSignature(string containin if (lastDot > 0) namespacePrefix = containingTypeName[..lastDot]; - foreach (var baseTypeReference in ParseCSharpBaseTypeReferences(signature)) + var baseTypeReferences = CSharpBaseListParser.Parse( + signature, + CSharpBaseListProjection.TypeReference); + foreach (var baseTypeReference in baseTypeReferences) { var normalizedBase = NormalizeCSharpBaseTypeReference(baseTypeReference); if (string.IsNullOrWhiteSpace(normalizedBase)) @@ -312,7 +307,9 @@ private List ResolveDirectCSharpInheritedContainingTy if (containingTypeScope.Kind is not ("class" or "struct" or "interface")) return []; - var baseTypeReferences = ParseCSharpBaseTypeReferences(containingTypeScope.Signature); + var baseTypeReferences = CSharpBaseListParser.Parse( + containingTypeScope.Signature, + CSharpBaseListProjection.TypeReference); if (baseTypeReferences.Count == 0) return []; @@ -467,11 +464,7 @@ private bool HasActiveCSharpUsingTypeAlias(string path, int lineNumber, string s if (TryResolveActiveCSharpUsingAliasScope(path, lineNumber, symbolName, requireTypeAlias: true, out _)) return true; - if (!_csharpUsingAliasScopesByPath.TryGetValue(path, out var scopes)) - { - scopes = LoadCSharpUsingAliasScopes(path); - _csharpUsingAliasScopesByPath[path] = scopes; - } + var scopes = GetCSharpPathUsingCatalog(path).Aliases; for (var i = scopes.Count - 1; i >= 0; i--) { @@ -517,11 +510,7 @@ private HashSet GetActiveCSharpUsingStaticTargets(string path, int lineN return cachedActiveTargets; } - if (!_csharpUsingStaticScopesByPath.TryGetValue(path, out var scopes)) - { - scopes = LoadCSharpUsingStaticScopes(path); - _csharpUsingStaticScopesByPath[path] = scopes; - } + var scopes = GetCSharpPathUsingCatalog(path).StaticImports; var activeTargets = new HashSet(StringComparer.Ordinal); foreach (var scope in scopes) @@ -546,128 +535,6 @@ private HashSet GetActiveCSharpUsingStaticTargets(string path, int lineN return activeTargets; } - private List LoadCSharpNamespaceScopes(string path) - { - using var cmd = _conn.CreateCommand(); - cmd.CommandText = @" - SELECT s.line, s.body_start_line, s.body_end_line, s.end_line, s.name, s.signature, f.lines - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE f.path = @path - AND f.lang = 'csharp' - AND s.kind = 'namespace' - ORDER BY s.line"; - SqliteCommandPolicy.Add(cmd, "@path", path); - - var scopes = new List(); - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - var line = reader.GetInt32(0); - var startLine = reader.IsDBNull(1) ? line : reader.GetInt32(1); - var endLine = reader.IsDBNull(2) - ? (reader.IsDBNull(3) ? line : reader.GetInt32(3)) - : reader.GetInt32(2); - var signature = GetNullableString(reader, 5); - if (!string.IsNullOrWhiteSpace(signature) - && signature.TrimEnd().EndsWith(';') - && !reader.IsDBNull(6)) - { - endLine = Math.Max(endLine, reader.GetInt32(6)); - } - - if (startLine <= 0 || endLine < startLine) - continue; - - var qualifiedName = NormalizeDbCSharpQualifiedName(reader.GetString(4)) ?? string.Empty; - scopes.Add(new CSharpNamespaceScope(qualifiedName, startLine, endLine)); - } - - return scopes; - } - - private List LoadCSharpUsingNamespaceScopes(string path) - { - var sql = @" - SELECT s.kind, s.line, s.body_start_line, s.body_end_line, s.end_line, s.signature, f.lines - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE f.path = @path - AND f.lang = 'csharp' - AND (s.kind = 'import' OR s.kind = 'namespace') - ORDER BY s.line"; - var cmd = RentCommand(sql, static c => c.Parameters.Add("@path", SqliteType.Text)); - SetParameter(cmd, "@path", path); - - var namespaceScopes = new List<(int StartLine, int EndLine)>(); - var imports = new List<(int Line, string Signature)>(); - try - { - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - var kind = reader.GetString(0); - var line = reader.GetInt32(1); - if (kind == "namespace") - { - var startLine = reader.IsDBNull(2) ? line : reader.GetInt32(2); - var endLine = reader.IsDBNull(3) - ? (reader.IsDBNull(4) ? line : reader.GetInt32(4)) - : reader.GetInt32(3); - var signature = GetNullableString(reader, 5); - if (!string.IsNullOrWhiteSpace(signature) - && signature.TrimEnd().EndsWith(';') - && !reader.IsDBNull(6)) - { - endLine = Math.Max(endLine, reader.GetInt32(6)); - } - - if (startLine > 0 && endLine >= startLine) - namespaceScopes.Add((startLine, endLine)); - continue; - } - - if (!reader.IsDBNull(5)) - imports.Add((line, reader.GetString(5))); - } - } - finally - { - ReleaseCommand(cmd); - } - - var scopes = new List(); - foreach (var import in imports) - { - if (!TryParseCSharpUsingNamespaceImport(import.Signature, out var target, out var isGlobal) - || isGlobal) - { - continue; - } - - var scopeStartLine = 1; - var scopeEndLine = int.MaxValue; - var scopeWidth = int.MaxValue; - foreach (var (startLine, endLine) in namespaceScopes) - { - if (import.Line < startLine || import.Line > endLine) - continue; - - var width = endLine - startLine; - if (width > scopeWidth) - continue; - - scopeStartLine = startLine; - scopeEndLine = endLine; - scopeWidth = width; - } - - scopes.Add(new CSharpUsingNamespaceScope(target!, import.Line, scopeStartLine, scopeEndLine)); - } - - return scopes; - } - private List LoadCSharpContainingTypeScopes(string path) { using var cmd = _conn.CreateCommand(); @@ -743,7 +610,9 @@ AND s.kind IN ('class', 'struct', 'interface') if (!string.Equals(containingTypeScope.Kind, "class", StringComparison.Ordinal)) return null; - var baseTypeReference = ParseCSharpBaseTypeReference(containingTypeScope.Signature); + var baseTypeReference = CSharpBaseListParser.Parse( + containingTypeScope.Signature, + CSharpBaseListProjection.TypeReference).FirstOrDefault(); if (string.IsNullOrWhiteSpace(baseTypeReference)) return null; @@ -824,125 +693,6 @@ AND s.kind IN ('class', 'struct', 'interface') return normalizedReference; } - private static string? ParseCSharpBaseTypeReference(string? signature) - { - var references = ParseCSharpBaseTypeReferences(signature); - return references.Count == 0 ? null : references[0]; - } - - private static List ParseCSharpBaseTypeReferences(string? signature) - { - if (string.IsNullOrWhiteSpace(signature)) - return []; - - var text = signature.TrimEnd(); - if (text.EndsWith("{", StringComparison.Ordinal)) - text = text[..^1].TrimEnd(); - - var colonIndex = FindCSharpBaseListColonIndex(text); - if (colonIndex < 0) - return []; - - var baseList = text[(colonIndex + 1)..]; - var whereIndex = baseList.IndexOf(" where ", StringComparison.Ordinal); - if (whereIndex >= 0) - baseList = baseList[..whereIndex]; - - var entries = new List(); - foreach (var entry in EnumerateCSharpBaseListEntries(baseList)) - { - var trimmed = entry.Trim(); - if (trimmed.Length > 0) - entries.Add(trimmed); - } - - return entries; - } - - private static int FindCSharpBaseListColonIndex(string signature) - { - var angleDepth = 0; - var parenDepth = 0; - var squareDepth = 0; - for (var i = 0; i < signature.Length; i++) - { - switch (signature[i]) - { - case '<': - angleDepth++; - break; - case '>': - if (angleDepth > 0) - angleDepth--; - break; - case '(': - parenDepth++; - break; - case ')': - if (parenDepth > 0) - parenDepth--; - break; - case '[': - squareDepth++; - break; - case ']': - if (squareDepth > 0) - squareDepth--; - break; - case ':': - if (angleDepth == 0 && parenDepth == 0 && squareDepth == 0) - return i; - break; - } - } - - return -1; - } - - private static IEnumerable EnumerateCSharpBaseListEntries(string baseList) - { - var angleDepth = 0; - var parenDepth = 0; - var squareDepth = 0; - var start = 0; - for (var i = 0; i < baseList.Length; i++) - { - switch (baseList[i]) - { - case '<': - angleDepth++; - break; - case '>': - if (angleDepth > 0) - angleDepth--; - break; - case '(': - parenDepth++; - break; - case ')': - if (parenDepth > 0) - parenDepth--; - break; - case '[': - squareDepth++; - break; - case ']': - if (squareDepth > 0) - squareDepth--; - break; - case ',': - if (angleDepth == 0 && parenDepth == 0 && squareDepth == 0) - { - yield return baseList[start..i]; - start = i + 1; - } - break; - } - } - - yield return baseList[start..]; - } - private static string NormalizeCSharpBaseTypeReference(string typeReference) { if (string.IsNullOrWhiteSpace(typeReference)) @@ -1017,286 +767,6 @@ private static string GetLastQualifiedSegment(string qualifiedName) return split < 0 ? qualifiedName : qualifiedName[(split + (split == lastColon ? 2 : 1))..]; } - private List LoadCSharpUsingStaticScopes(string path) - { - var sql = @" - SELECT s.kind, s.line, s.body_start_line, s.body_end_line, s.end_line, s.signature, f.lines - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE f.path = @path - AND f.lang = 'csharp' - AND (s.kind = 'import' OR s.kind = 'namespace') - ORDER BY s.line"; - var cmd = RentCommand(sql, static c => c.Parameters.Add("@path", SqliteType.Text)); - SetParameter(cmd, "@path", path); - - var namespaceScopes = new List<(int StartLine, int EndLine)>(); - var imports = new List<(int Line, string Signature)>(); - try - { - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - var kind = reader.GetString(0); - var line = reader.GetInt32(1); - if (kind == "namespace") - { - var startLine = reader.IsDBNull(2) ? line : reader.GetInt32(2); - var endLine = reader.IsDBNull(3) - ? (reader.IsDBNull(4) ? line : reader.GetInt32(4)) - : reader.GetInt32(3); - var signature = GetNullableString(reader, 5); - if (!string.IsNullOrWhiteSpace(signature) - && signature.TrimEnd().EndsWith(';') - && !reader.IsDBNull(6)) - { - endLine = Math.Max(endLine, reader.GetInt32(6)); - } - - if (startLine > 0 && endLine >= startLine) - namespaceScopes.Add((startLine, endLine)); - continue; - } - - if (!reader.IsDBNull(5)) - imports.Add((line, reader.GetString(5))); - } - } - finally - { - ReleaseCommand(cmd); - } - - var scopes = new List(); - foreach (var import in imports) - { - if (!TryParseCSharpUsingStaticImport(import.Signature, out var target, out var isGlobal) - || isGlobal) - { - continue; - } - - var scopeStartLine = 1; - var scopeEndLine = int.MaxValue; - var scopeWidth = int.MaxValue; - foreach (var (startLine, endLine) in namespaceScopes) - { - if (import.Line < startLine || import.Line > endLine) - continue; - - var width = endLine - startLine; - if (width > scopeWidth) - continue; - - scopeStartLine = startLine; - scopeEndLine = endLine; - scopeWidth = width; - } - - scopes.Add(new CSharpUsingStaticScope(target!, import.Line, scopeStartLine, scopeEndLine)); - } - - return scopes; - } - - private List LoadCSharpUsingAliasScopes(string path) - { - using var cmd = _conn.CreateCommand(); - cmd.CommandText = @" - SELECT s.kind, s.line, s.body_start_line, s.body_end_line, s.end_line, s.signature, f.lines - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE f.path = @path - AND f.lang = 'csharp' - AND (s.kind = 'import' OR s.kind = 'namespace') - ORDER BY s.line"; - SqliteCommandPolicy.Add(cmd, "@path", path); - - var namespaceScopes = new List<(int StartLine, int EndLine)>(); - var imports = new List<(int Line, string Signature)>(); - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - var kind = reader.GetString(0); - var line = reader.GetInt32(1); - if (kind == "namespace") - { - var startLine = reader.IsDBNull(2) ? line : reader.GetInt32(2); - var endLine = reader.IsDBNull(3) - ? (reader.IsDBNull(4) ? line : reader.GetInt32(4)) - : reader.GetInt32(3); - var signature = GetNullableString(reader, 5); - if (!string.IsNullOrWhiteSpace(signature) - && signature.TrimEnd().EndsWith(';') - && !reader.IsDBNull(6)) - { - endLine = Math.Max(endLine, reader.GetInt32(6)); - } - - if (startLine > 0 && endLine >= startLine) - namespaceScopes.Add((startLine, endLine)); - continue; - } - - if (!reader.IsDBNull(5)) - imports.Add((line, reader.GetString(5))); - } - - var scopes = new List(); - foreach (var import in imports) - { - if (!TryParseCSharpUsingAliasImport(import.Signature, out var aliasName, out var targetQualifiedName, out var isGlobal) - || isGlobal) - { - continue; - } - - var scopeStartLine = 1; - var scopeEndLine = int.MaxValue; - var scopeWidth = int.MaxValue; - foreach (var (startLine, endLine) in namespaceScopes) - { - if (import.Line < startLine || import.Line > endLine) - continue; - - var width = endLine - startLine; - if (width > scopeWidth) - continue; - - scopeStartLine = startLine; - scopeEndLine = endLine; - scopeWidth = width; - } - - scopes.Add(new CSharpUsingAliasScope( - aliasName!, - targetQualifiedName!, - import.Line, - scopeStartLine, - scopeEndLine, - IsKnownCSharpTypeQualifiedName(targetQualifiedName!))); - } - - return scopes; - } - - private HashSet GetGlobalCSharpUsingStaticTargets() - { - if (_csharpGlobalUsingStaticTargets != null) - return _csharpGlobalUsingStaticTargets; - - var sql = @" - SELECT s.signature - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE f.lang = 'csharp' - AND s.kind = 'import'"; - - var targets = new HashSet(StringComparer.Ordinal); - var cmd = RentCommand(sql, static _ => { }); - try - { - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - if (reader.IsDBNull(0)) - continue; - if (TryParseCSharpUsingStaticImport(reader.GetString(0), out var target, out var isGlobal) - && isGlobal) - { - targets.Add(target!); - } - } - } - finally - { - ReleaseCommand(cmd); - } - - _csharpGlobalUsingStaticTargets = targets; - return _csharpGlobalUsingStaticTargets; - } - - private Dictionary GetGlobalCSharpUsingAliasesByName() - { - if (_csharpGlobalUsingAliasesByName != null) - return _csharpGlobalUsingAliasesByName; - - var sql = @" - SELECT s.signature - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE f.lang = 'csharp' - AND s.kind = 'import'"; - - var aliases = new Dictionary(StringComparer.Ordinal); - var cmd = RentCommand(sql, static _ => { }); - try - { - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - if (reader.IsDBNull(0)) - continue; - if (TryParseCSharpUsingAliasImport(reader.GetString(0), out var aliasName, out var targetQualifiedName, out var isGlobal) - && isGlobal) - { - aliases[aliasName!] = new CSharpUsingAliasScope( - aliasName!, - targetQualifiedName!, - 0, - 1, - int.MaxValue, - IsKnownCSharpTypeQualifiedName(targetQualifiedName!)); - } - } - } - finally - { - ReleaseCommand(cmd); - } - - _csharpGlobalUsingAliasesByName = aliases; - return _csharpGlobalUsingAliasesByName; - } - - private HashSet GetGlobalCSharpUsingNamespaces() - { - if (_csharpGlobalUsingNamespaces != null) - return _csharpGlobalUsingNamespaces; - - var sql = @" - SELECT s.signature - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE f.lang = 'csharp' - AND s.kind = 'import'"; - - var namespaces = new HashSet(StringComparer.Ordinal); - var cmd = RentCommand(sql, static _ => { }); - try - { - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - if (reader.IsDBNull(0)) - continue; - if (TryParseCSharpUsingNamespaceImport(reader.GetString(0), out var target, out var isGlobal) - && isGlobal) - { - namespaces.Add(target!); - } - } - } - finally - { - ReleaseCommand(cmd); - } - - _csharpGlobalUsingNamespaces = namespaces; - return _csharpGlobalUsingNamespaces; - } - private static bool TryParseCSharpUsingStaticImport(string signature, out string? target, out bool isGlobal) { target = null; @@ -1354,11 +824,7 @@ private bool TryResolveActiveCSharpUsingAliasScope(string path, int lineNumber, if (string.IsNullOrWhiteSpace(aliasName)) return false; - if (!_csharpUsingAliasScopesByPath.TryGetValue(path, out var scopes)) - { - scopes = LoadCSharpUsingAliasScopes(path); - _csharpUsingAliasScopesByPath[path] = scopes; - } + var scopes = GetCSharpPathUsingCatalog(path).Aliases; for (var i = scopes.Count - 1; i >= 0; i--) { diff --git a/src/CodeIndex/Database/DbReader.CSharpUsingCatalog.cs b/src/CodeIndex/Database/DbReader.CSharpUsingCatalog.cs new file mode 100644 index 0000000000..ac34ddea66 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.CSharpUsingCatalog.cs @@ -0,0 +1,282 @@ +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed record CSharpNamespaceScope( + string QualifiedName, + int ScopeStartLine, + int ScopeEndLine); + + private sealed record CSharpUsingStaticScope( + string TargetQualifiedName, + int Line, + int ScopeStartLine, + int ScopeEndLine); + + private sealed record CSharpUsingNamespaceScope( + string TargetQualifiedName, + int Line, + int ScopeStartLine, + int ScopeEndLine); + + private sealed record CSharpUsingAliasScope( + string AliasName, + string TargetQualifiedName, + int Line, + int ScopeStartLine, + int ScopeEndLine, + bool TargetsType); + + private sealed record CSharpRawUsingImport(int Line, string Signature); + + private sealed record CSharpRawPathUsingCatalog( + List NamespaceDeclarations, + List Imports); + + private sealed record CSharpPathUsingCatalog( + List NamespaceDeclarations, + List NamespaceImports, + List StaticImports, + List Aliases); + + private sealed record CSharpGlobalUsingCatalog( + HashSet Namespaces, + HashSet StaticTargets, + Dictionary AliasesByName); + + private CSharpPathUsingCatalog GetCSharpPathUsingCatalog(string path) + { + if (_csharpUsingCatalogsByPath.TryGetValue(path, out var cached)) + return cached; + + var rawCatalog = LoadRawCSharpPathUsingCatalog(path); + var catalog = ProjectCSharpPathUsingCatalog(rawCatalog); + _csharpUsingCatalogsByPath[path] = catalog; + return catalog; + } + + private CSharpRawPathUsingCatalog LoadRawCSharpPathUsingCatalog(string path) + { + const string sql = @" + SELECT s.kind, s.line, s.body_start_line, s.body_end_line, s.end_line, + s.signature, f.lines, s.name + FROM symbols s + JOIN files f ON s.file_id = f.id + WHERE f.path = @path + AND f.lang = 'csharp' + AND (s.kind = 'import' OR s.kind = 'namespace') + ORDER BY s.line"; + var cmd = RentCommand(sql, static c => c.Parameters.Add("@path", SqliteType.Text)); + SetParameter(cmd, "@path", path); + + var namespaceDeclarations = new List(); + var imports = new List(); + try + { + using var reader = cmd.ExecuteTrackedReader(); + while (reader.TrackedRead()) + { + var line = reader.GetInt32(1); + if (reader.GetString(0) == "namespace") + { + var (startLine, endLine) = ReadCSharpNamespaceRange(reader, line); + if (startLine <= 0 || endLine < startLine) + continue; + + var qualifiedName = NormalizeDbCSharpQualifiedName(reader.GetString(7)) + ?? string.Empty; + namespaceDeclarations.Add(new CSharpNamespaceScope( + qualifiedName, + startLine, + endLine)); + continue; + } + + if (!reader.IsDBNull(5)) + imports.Add(new CSharpRawUsingImport(line, reader.GetString(5))); + } + } + finally + { + ReleaseCommand(cmd); + } + + return new CSharpRawPathUsingCatalog(namespaceDeclarations, imports); + } + + private static (int StartLine, int EndLine) ReadCSharpNamespaceRange( + SqliteDataReader reader, + int declarationLine) + { + var startLine = reader.IsDBNull(2) ? declarationLine : reader.GetInt32(2); + var endLine = reader.IsDBNull(3) + ? (reader.IsDBNull(4) ? declarationLine : reader.GetInt32(4)) + : reader.GetInt32(3); + var signature = GetNullableString(reader, 5); + if (!string.IsNullOrWhiteSpace(signature) + && signature.TrimEnd().EndsWith(';') + && !reader.IsDBNull(6)) + { + endLine = Math.Max(endLine, reader.GetInt32(6)); + } + + return (startLine, endLine); + } + + private CSharpPathUsingCatalog ProjectCSharpPathUsingCatalog( + CSharpRawPathUsingCatalog rawCatalog) + { + var namespaceImports = new List(); + var staticImports = new List(); + var aliases = new List(); + foreach (var import in rawCatalog.Imports) + { + var (scopeStartLine, scopeEndLine) = FindCSharpImportScope( + import.Line, + rawCatalog.NamespaceDeclarations); + if (TryParseCSharpUsingNamespaceImport(import.Signature, out var namespaceTarget, out var namespaceIsGlobal) + && !namespaceIsGlobal) + { + namespaceImports.Add(new CSharpUsingNamespaceScope( + namespaceTarget!, import.Line, scopeStartLine, scopeEndLine)); + } + + if (TryParseCSharpUsingStaticImport(import.Signature, out var staticTarget, out var staticIsGlobal) + && !staticIsGlobal) + { + staticImports.Add(new CSharpUsingStaticScope( + staticTarget!, import.Line, scopeStartLine, scopeEndLine)); + } + + if (TryParseCSharpUsingAliasImport( + import.Signature, + out var aliasName, + out var aliasTarget, + out var aliasIsGlobal) + && !aliasIsGlobal) + { + aliases.Add(new CSharpUsingAliasScope( + aliasName!, + aliasTarget!, + import.Line, + scopeStartLine, + scopeEndLine, + IsKnownCSharpTypeQualifiedName(aliasTarget!))); + } + } + + return new CSharpPathUsingCatalog( + rawCatalog.NamespaceDeclarations, + namespaceImports, + staticImports, + aliases); + } + + private static (int StartLine, int EndLine) FindCSharpImportScope( + int importLine, + IReadOnlyList namespaceDeclarations) + { + var scopeStartLine = 1; + var scopeEndLine = int.MaxValue; + var scopeWidth = int.MaxValue; + foreach (var scope in namespaceDeclarations) + { + if (importLine < scope.ScopeStartLine || importLine > scope.ScopeEndLine) + continue; + + var width = scope.ScopeEndLine - scope.ScopeStartLine; + if (width > scopeWidth) + continue; + + scopeStartLine = scope.ScopeStartLine; + scopeEndLine = scope.ScopeEndLine; + scopeWidth = width; + } + + return (scopeStartLine, scopeEndLine); + } + + private CSharpGlobalUsingCatalog GetGlobalCSharpUsingCatalog() + { + if (_csharpGlobalUsingCatalog != null) + return _csharpGlobalUsingCatalog; + + var rawImports = LoadRawGlobalCSharpUsingImports(); + _csharpGlobalUsingCatalog = ProjectGlobalCSharpUsingCatalog(rawImports); + return _csharpGlobalUsingCatalog; + } + + private List LoadRawGlobalCSharpUsingImports() + { + const string sql = @" + SELECT s.signature + FROM symbols s + JOIN files f ON s.file_id = f.id + WHERE f.lang = 'csharp' + AND s.kind = 'import'"; + var cmd = RentCommand(sql, static _ => { }); + var imports = new List(); + try + { + using var reader = cmd.ExecuteTrackedReader(); + while (reader.TrackedRead()) + { + if (!reader.IsDBNull(0)) + imports.Add(reader.GetString(0)); + } + } + finally + { + ReleaseCommand(cmd); + } + + return imports; + } + + private CSharpGlobalUsingCatalog ProjectGlobalCSharpUsingCatalog( + IReadOnlyList rawImports) + { + var namespaces = new HashSet(StringComparer.Ordinal); + var staticTargets = new HashSet(StringComparer.Ordinal); + var aliases = new Dictionary(StringComparer.Ordinal); + foreach (var signature in rawImports) + { + if (TryParseCSharpUsingNamespaceImport(signature, out var namespaceTarget, out var namespaceIsGlobal) + && namespaceIsGlobal) + { + namespaces.Add(namespaceTarget!); + } + + if (TryParseCSharpUsingStaticImport(signature, out var staticTarget, out var staticIsGlobal) + && staticIsGlobal) + { + staticTargets.Add(staticTarget!); + } + + if (TryParseCSharpUsingAliasImport(signature, out var aliasName, out var aliasTarget, out var aliasIsGlobal) + && aliasIsGlobal) + { + aliases[aliasName!] = new CSharpUsingAliasScope( + aliasName!, + aliasTarget!, + 0, + 1, + int.MaxValue, + IsKnownCSharpTypeQualifiedName(aliasTarget!)); + } + } + + return new CSharpGlobalUsingCatalog(namespaces, staticTargets, aliases); + } + + private HashSet GetGlobalCSharpUsingNamespaces() + => GetGlobalCSharpUsingCatalog().Namespaces; + + private HashSet GetGlobalCSharpUsingStaticTargets() + => GetGlobalCSharpUsingCatalog().StaticTargets; + + private Dictionary GetGlobalCSharpUsingAliasesByName() + => GetGlobalCSharpUsingCatalog().AliasesByName; +} diff --git a/src/CodeIndex/Database/DbReader.Dependencies.cs b/src/CodeIndex/Database/DbReader.Dependencies.cs index f0e502e402..e521429351 100644 --- a/src/CodeIndex/Database/DbReader.Dependencies.cs +++ b/src/CodeIndex/Database/DbReader.Dependencies.cs @@ -238,1153 +238,22 @@ public List GetFileDependencies( lang = NormalizeQueryLanguage(lang); if (!_hasReferencesTable) return new List(); cancellationToken.ThrowIfCancellationRequested(); - using var cmd = _conn.CreateCommand(); - var referenceLineJoin = ReferenceLineJoinSql("r"); - var contextSql = ReferenceContextSql("r"); - var hasResolvedReferenceTargets = _referenceIdentityContractCurrent; - var resolutionStateSql = hasResolvedReferenceTargets ? "r.resolution_state" : "NULL"; - var identityScopeConditionSql = """ - ( - (src.lang = 'csharp' AND r.reference_kind NOT IN ('attribute', 'annotation')) - OR (src.lang = 'dependency_lock' AND r.reference_kind = 'dependency') - ) - """; - var referenceIdSql = $"CASE WHEN {identityScopeConditionSql} THEN r.id ELSE 0 END"; - var scopedResolutionStateSql = $"CASE WHEN {identityScopeConditionSql} THEN {resolutionStateSql} ELSE NULL END"; - var identityScopedSql = $"CASE WHEN {identityScopeConditionSql} THEN 1 ELSE 0 END"; - var identityNameEdgePredicate = hasResolvedReferenceTargets - ? "snc.identity_scoped = 0" - : "snc.source_lang <> 'dependency_lock'"; - var resolvedIdentityLimitSql = lang == "csharp" - ? " LIMIT @sourceCandidateLimit" - : string.Empty; - var resolvedIdentityEdgesSql = hasResolvedReferenceTargets - ? @" - SELECT resolved.source_path, - resolved.target_path, - resolved.symbol_name, - COUNT(*) AS ref_count, - resolved.source_lang, - 'resolved_identity' AS origin, - resolved.raw_reference_kind, - resolved.target_kind - FROM ( - SELECT DISTINCT lrp.source_path, - target_file.path AS target_path, - lrp.symbol_name, - lrp.reference_id, - lrp.source_lang, - lrp.raw_reference_kind, - target.kind AS target_kind - FROM logical_references_primary lrp - JOIN symbol_reference_candidates candidate - ON candidate.reference_id = lrp.reference_id - JOIN symbols target ON target.id = candidate.symbol_id - JOIN files target_file ON target_file.id = target.file_id - JOIN target_files scoped_target - ON scoped_target.target_path = target_file.path - AND scoped_target.target_lang = target_file.lang - WHERE lrp.identity_scoped = 1 - AND lrp.resolution_state IN ('resolved', 'resolved_group') - AND lrp.source_path != target_file.path" - : string.Empty; - if (hasResolvedReferenceTargets) - { - AppendDependencySymbolFilter( - cmd, - ref resolvedIdentityEdgesSql, - "lrp.symbol_name", - dependencySymbols, - dependencySymbolFamilies, - suppressDependencyNoise, - "resolvedDependency"); - resolvedIdentityEdgesSql += @" - ORDER BY lrp.source_path, lrp.symbol_name, lrp.reference_id" + resolvedIdentityLimitSql + @" - ) resolved - GROUP BY resolved.source_path, resolved.target_path, resolved.symbol_name, - resolved.source_lang, resolved.raw_reference_kind, resolved.target_kind - UNION ALL - "; - } - // Aggregate logical reference sites per source-file/name first, then join that bounded - // set to distinct target files. This avoids the per-reference × per-symbol explosion that - // could exhaust SQLite temp-store on large indexes with many same-named symbols. - // まず source-file/name 単位に logical reference site 数を集約し、その後で distinct な - // target file と結合することで、大規模 index で SQLite temp-store を枯渇させる - // per-reference × per-symbol の膨張を防ぐ。 - var sourceFilterAlias = "src"; - var targetFilterAlias = "dst"; - var targetLogicalSymbolNameExpr = BuildLogicalDependencySymbolNameExpr("dst", "s.name"); - var targetLogicalSymbolSegmentCountExpr = BuildLogicalDependencySymbolSegmentCountExpr("dst", "s.name"); - var pythonImportMatchSignatureExpr = GetSymbolColumnSql("signature", "NULL", "py_import_match"); - var pythonImportSignatureExpr = GetSymbolColumnSql("signature", "NULL", "py_import"); - var sqlDependencyTargetMatchExpr = @"( - (tf.target_lang != 'sql' - AND NOT (snc.source_lang IN ('msbuild', 'solution') AND snc.logical_reference_kind IN ('import', 'project_reference')) - AND NOT (snc.source_lang = 'markdown' AND snc.logical_reference_kind = 'import') - AND NOT (snc.source_lang = 'markdown' AND snc.logical_reference_kind = 'reference') - AND tf.symbol_name = snc.symbol_name) - OR (snc.source_lang = 'python' - AND tf.target_lang = 'python' - AND EXISTS ( - SELECT 1 FROM symbols py_import_match - WHERE py_import_match.file_id = snc.source_file_id - AND py_import_match.kind = 'import' - AND python_import_target_name(snc.source_path, snc.symbol_name, snc.context, snc.column_number, " + pythonImportMatchSignatureExpr + @") = tf.symbol_name - )) - OR (tf.target_lang = 'sql' AND ( - (tf.symbol_segment_count = snc.symbol_segment_count AND tf.symbol_name = snc.symbol_name COLLATE NOCASE) - OR (sql_segment_count(snc.raw_symbol_name) = 1 - AND snc.allow_leaf_fallback = 1 - AND tf.symbol_segment_count > 1 - AND sql_leaf_name(tf.symbol_name) = snc.raw_symbol_name COLLATE NOCASE - AND NOT EXISTS ( - SELECT 1 - FROM target_files tf_exact - WHERE tf_exact.target_lang = tf.target_lang - AND tf_exact.symbol_segment_count = 1 - AND tf_exact.symbol_name = snc.symbol_name COLLATE NOCASE - ) - AND NOT EXISTS ( - SELECT 1 - FROM target_files tf_resolved - WHERE tf_resolved.target_lang = tf.target_lang - AND tf_resolved.symbol_segment_count = snc.symbol_segment_count - AND tf_resolved.symbol_name = snc.symbol_name COLLATE NOCASE - )) - )) - )"; - var sql = @" - WITH logical_references_primary AS ( - SELECT src.id AS source_file_id, - src.path AS source_path, - src.lang AS source_lang, - " + referenceIdSql + @" AS reference_id, - " + scopedResolutionStateSql + @" AS resolution_state, - " + identityScopedSql + @" AS identity_scoped, - r.symbol_name, - " + contextSql + @" AS context, - r.container_name, - r.line, - r.column_number, - r.reference_kind AS raw_reference_kind, - " + GetLogicalReferenceKindSql("r.reference_kind") + @" AS logical_reference_kind - FROM symbol_references r - JOIN files src ON r.file_id = src.id" + referenceLineJoin + @" - WHERE 1 = 1"; - // `deps` intentionally includes metadata-only reference kinds - // (`attribute` / `annotation`). Same rationale as - // `GetFileDependencyHintsToResolvedType`: renaming or removing a type that - // is only referenced via `[JsonConverter(typeof(User))]` or - // `@Inject(User.class)` still breaks the annotated file at compile time, so - // file-level dependency analysis must treat those sites as real edges. - // Call-graph-specific commands (`callers` / `callees`) keep rejecting - // metadata kinds at the CLI / MCP boundary — that is a separate contract. - // `deps` は metadata-only 参照 (`attribute` / `annotation`) も意図的に - // 含める。`GetFileDependencyHintsToResolvedType` と同じ理由で、 - // `[JsonConverter(typeof(User))]` や `@Inject(User.class)` 経由でしか参照 - // されない型でも、rename / 削除すれば annotated ファイルは compile-time - // で壊れるため、ファイル単位の依存分析では本物の edge として扱う必要が - // ある。call-graph 専用コマンド (`callers` / `callees`) 側では metadata - // 種別の拒否を CLI / MCP boundary で引き続き行う — そちらは別契約。 - sql += $" AND {BuildGraphSupportedLanguagePredicate(cmd, "src", "depsLang")}"; - AppendDependencyGeneratedFilter(ref sql, sourceFilterAlias); - if (lang != null) - sql += " AND src.lang = @lang"; - if (!reverse && pathPatterns is { Count: > 0 }) - { - var ors = new List(pathPatterns.Count); - for (int i = 0; i < pathPatterns.Count; i++) - ors.Add(BuildPathFilterPredicate(sourceFilterAlias, "pathPattern", i, pathPatterns[i])); - sql += " AND (" + string.Join(" OR ", ors) + ")"; - } - if (!reverse && excludePathPatterns is { Count: > 0 }) - { - for (int i = 0; i < excludePathPatterns.Count; i++) - sql += $" AND NOT {BuildPathFilterPredicate(sourceFilterAlias, "excludePath", i, excludePathPatterns[i])}"; - } - if (excludeTests) - sql += $" AND NOT {DependencyTestPathCondition($"{sourceFilterAlias}.path")}"; - sql += @" - GROUP BY src.id, src.path, src.lang, " + referenceIdSql + @", " + scopedResolutionStateSql + @", " + identityScopedSql + @", r.symbol_name, " + contextSql + @", r.container_name, r.line, r.column_number, r.reference_kind, logical_reference_kind - ), - logical_references AS ( - SELECT source_file_id, source_path, source_lang, reference_id, identity_scoped, - " + BuildLogicalReferenceNameExpr("source_lang", "symbol_name", "context", "container_name", "column_number") + @" AS symbol_name, - " + BuildLogicalReferenceSegmentCountExpr("source_lang", "symbol_name", "context", "container_name", "column_number") + @" AS symbol_segment_count, - " + BuildLogicalReferenceLeafFallbackAllowedExpr("source_lang", "symbol_name", "context", "container_name", "column_number") + @" AS allow_leaf_fallback, - symbol_name AS raw_symbol_name, - context, line, column_number, raw_reference_kind, logical_reference_kind, - 0 AS is_attribute_alias, - CASE WHEN logical_reference_kind IN ('attribute', 'annotation') THEN 1 ELSE 0 END AS is_metadata - FROM logical_references_primary - UNION ALL - -- C# attribute suffix alias: [Foo] in source is stored with symbol_name='Foo', - -- but the defining class is named 'FooAttribute'. Emit the canonical 'Foo' + 'Attribute' - -- form so deps can match the class file as a target. The alias rows are flagged - -- so the edges CTE can restrict them to class-like targets and avoid spurious - -- edges to unrelated functions / properties that happen to be named 'FooAttribute'. - -- C# 属性のサフィックス別名: ソース上の [Foo] は symbol_name='Foo' で保存されるが、 - -- 定義クラスは 'FooAttribute' 命名になるため、正規形 'Foo' + 'Attribute' を補って - -- deps がクラス側のファイルを target として join できるようにする。alias 行には - -- フラグを付け、edges CTE 側で class-like target だけに限定する。これにより、 - -- 偶然 'FooAttribute' という名前を持つ関数やプロパティへの誤ったエッジを防ぐ。 - SELECT source_file_id, source_path, source_lang, reference_id, identity_scoped, - symbol_name || 'Attribute' AS symbol_name, - 1 AS symbol_segment_count, - 0 AS allow_leaf_fallback, - symbol_name || 'Attribute' AS raw_symbol_name, - context, line, column_number, raw_reference_kind, logical_reference_kind, - 1 AS is_attribute_alias, - 1 AS is_metadata - FROM logical_references_primary - WHERE source_lang = 'csharp' - AND logical_reference_kind = 'attribute' - AND symbol_name NOT LIKE '%Attribute' - ), - source_name_counts AS ( - -- Grouping includes is_metadata so metadata-only groups ([Foo] / @Foo) - -- can be restricted to class-like targets independently from non-metadata - -- call-graph groups that share the same symbol_name in the same file - -- (e.g. `Foo()` call + `[Foo]` attribute both present in the same source). - -- is_metadata を GROUP BY に含めることで、同じ source file / symbol_name を - -- 共有する metadata 行と call-graph 行 (例: 同じファイル内の `Foo()` 呼び出し - -- と `[Foo]` 属性) を別グループとして扱い、metadata 側だけに class-like - -- target 制限を掛けられるようにする。 - SELECT source_file_id, - source_path, - source_lang, - identity_scoped, - symbol_name, - symbol_segment_count, - allow_leaf_fallback, - raw_symbol_name, - context, - column_number, - raw_reference_kind, - logical_reference_kind, - is_attribute_alias, - is_metadata, - COUNT(*) AS ref_count - FROM logical_references - WHERE 1 = 1"; - // C# reference names match the public dependency symbol, so filtering them here - // bounds source aggregation. Other languages may resolve an alias or path-like - // reference to a differently named target; retain those rows until target_files - // applies the public symbol filter. - // C# の reference 名は公開 dependency symbol と一致するため、ここで絞って source - // 集約を制限する。他言語は alias や path 形式の reference を異なる target 名へ - // 解決する場合があるため、target_files が公開 symbol filter を適用するまで保持する。 - AppendDependencySymbolFilter( - cmd, - ref sql, - "symbol_name", - dependencySymbols, - dependencySymbolFamilies, - suppressDependencyNoise, - "sourceDependency", - "source_lang = 'csharp'"); - sql += @" - GROUP BY source_file_id, source_path, source_lang, identity_scoped, symbol_name, symbol_segment_count, allow_leaf_fallback, raw_symbol_name, context, column_number, raw_reference_kind, logical_reference_kind, is_attribute_alias, is_metadata - )"; - if (lang == "csharp") - { - // Bound the expensive C# name-to-target join by the requested result limit. - // The final edge ranking already uses an over-fetched candidate window; applying - // the same window to source/name groups prevents a tiny --limit from comparing the - // complete C# reference graph before LIMIT can take effect. - // 高コストな C# の name-to-target join を要求された結果上限に連動して制限する。 - // 最終 edge ranking と同じ over-fetch 候補窓を source/name group にも適用し、 - // 小さな --limit でも LIMIT 前に C# 参照グラフ全体を比較する事態を防ぐ。 - sql += @", - csharp_dependency_targets AS ( - SELECT dst.path AS target_path, - " + targetLogicalSymbolNameExpr + @" AS symbol_name, - MAX(CASE WHEN " + BuildMetadataTargetKindExpr("dst") + @" THEN 1 ELSE 0 END) AS has_metadata_target_kind - FROM symbols s - JOIN files dst ON s.file_id = dst.id - WHERE dst.lang = 'csharp'"; - AppendDependencyGeneratedFilter(ref sql, "dst"); - if (reverse && pathPatterns is { Count: > 0 }) - { - var ors = new List(pathPatterns.Count); - for (int i = 0; i < pathPatterns.Count; i++) - ors.Add(BuildPathFilterPredicate("dst", "pathPattern", i, pathPatterns[i])); - sql += " AND (" + string.Join(" OR ", ors) + ")"; - } - if (reverse && excludePathPatterns is { Count: > 0 }) - { - for (int i = 0; i < excludePathPatterns.Count; i++) - sql += $" AND NOT {BuildPathFilterPredicate("dst", "excludePath", i, excludePathPatterns[i])}"; - } - if (excludeTests) - sql += $" AND NOT {DependencyTestPathCondition("dst.path")}"; - AppendDependencySymbolFilter( - cmd, - ref sql, - targetLogicalSymbolNameExpr, - dependencySymbols, - dependencySymbolFamilies, - suppressDependencyNoise, - "boundedTargetDependency"); - sql += @" - GROUP BY dst.path, " + targetLogicalSymbolNameExpr + @" - ), - bounded_source_name_counts AS ( - SELECT snc.* - FROM source_name_counts snc - WHERE EXISTS ( - SELECT 1 - FROM csharp_dependency_targets tf - WHERE tf.symbol_name = snc.symbol_name - AND tf.target_path != snc.source_path - AND (snc.is_metadata = 0 OR tf.has_metadata_target_kind = 1) - ) - ORDER BY snc.ref_count DESC, snc.source_path, snc.symbol_name, snc.context, snc.column_number, snc.raw_reference_kind - LIMIT @sourceCandidateLimit - ),"; - } - else - { - sql += @", - bounded_source_name_counts AS ( - SELECT * FROM source_name_counts - ),"; - } - sql += @" - target_files AS ( - -- Collapse per-symbol rows to one per (target_path, target_lang, symbol_name) - -- and remember whether any of the same-name symbols is a class-like kind - -- via MAX. Keeping kind in DISTINCT would split identical (path, lang, name) - -- rows when one file defines both a class and a same-name function (e.g. a - -- C# constructor), inflating the deps reference count. - -- (target_path, target_lang, symbol_name) 単位に集約し、同名のシンボルの - -- いずれかが class 系であるかを MAX で覚える。kind を DISTINCT に含めると、 - -- 同じ (path, lang, name) でも class と同名 function (C# のコンストラクタ等) - -- が別行として残り、deps の参照カウントが膨らんでしまう。 - -- has_metadata_target_kind further narrows the class-like set to targets - -- that can legitimately be referenced as [Attribute] metadata. For C# - -- we cannot resolve base types transitively at SQL time, so the best - -- portable approximation is an inheritance-clause check: any class - -- declared with a base list is a potential attribute type (direct or - -- indirect Attribute derivation). A plain class FooAttribute with no - -- base clause is not a valid [Foo] target at compile time. - -- Other languages keep the original class-like breadth. Legacy DBs - -- without a signature column degrade to the broad class-like set. - -- has_metadata_target_kind は [Attribute] metadata target として妥当な - -- class-like のみに絞る。C# は SQL 時点で基底型を遡れないため、継承節を - -- 持つクラスを候補とする近似を採る(直接・間接の Attribute 継承を - -- 取りこぼさない)。他言語は class-like 全体を残す。signature 列が無い - -- legacy DB では filter を無効化し class-like 全体に戻る。 - SELECT dst.path AS target_path, - dst.lang AS target_lang, - " + targetLogicalSymbolNameExpr + @" AS symbol_name, - " + targetLogicalSymbolSegmentCountExpr + @" AS symbol_segment_count, - MAX(CASE WHEN s.kind IN ('class','struct','interface') THEN 1 ELSE 0 END) AS has_class_like_kind, - MAX(CASE WHEN s.kind = 'heading' THEN 1 ELSE 0 END) AS has_heading_kind, - MAX(CASE WHEN " + BuildMetadataTargetKindExpr("dst") + @" - THEN 1 ELSE 0 END) AS has_metadata_target_kind - FROM symbols s - JOIN files dst ON s.file_id = dst.id - WHERE 1 = 1"; - sql += $" AND {BuildGraphSupportedLanguagePredicate(cmd, "dst", "depsTargetLang")}"; - AppendDependencyGeneratedFilter(ref sql, targetFilterAlias); - if (lang != null && !lang.Equals("solution", StringComparison.Ordinal)) - sql += " AND dst.lang = @lang"; - if (reverse && pathPatterns is { Count: > 0 }) - { - var ors = new List(pathPatterns.Count); - for (int i = 0; i < pathPatterns.Count; i++) - ors.Add(BuildPathFilterPredicate(targetFilterAlias, "pathPattern", i, pathPatterns[i])); - sql += " AND (" + string.Join(" OR ", ors) + ")"; - } - if (reverse && excludePathPatterns is { Count: > 0 }) - { - for (int i = 0; i < excludePathPatterns.Count; i++) - sql += $" AND NOT {BuildPathFilterPredicate(targetFilterAlias, "excludePath", i, excludePathPatterns[i])}"; - } - if (excludeTests) - sql += $" AND NOT {DependencyTestPathCondition($"{targetFilterAlias}.path")}"; - AppendDependencySymbolFilter( - cmd, - ref sql, - targetLogicalSymbolNameExpr, - dependencySymbols, - dependencySymbolFamilies, - suppressDependencyNoise, - "targetDependency", - "dst.lang = 'csharp'"); - sql += @" - GROUP BY dst.path, dst.lang, " + targetLogicalSymbolNameExpr + @", " + targetLogicalSymbolSegmentCountExpr + @" - ), - path_target_files AS ( - SELECT DISTINCT target_path, target_lang - FROM target_files - ), - metadata_raw_suppression AS ( - -- When a raw C# attribute reference '[Foo]' (stored as symbol_name='Foo', - -- logical_reference_kind='attribute') also has a synthetic suffix alias - -- row that resolves to a class-like 'FooAttribute' target, drop the raw - -- row to avoid creating a duplicate edge to any unrelated 'Foo' symbol - -- (method, property, local class) that merely shares the bare name. - -- 生の C# 属性参照 '[Foo]' (symbol_name='Foo', kind='attribute') に対して - -- 同じ source_file 内で 'FooAttribute' の synthetic alias 行が - -- class 系 target に解決できる場合、この行自体は落として - -- 同名の関数/プロパティ/ローカルクラス 'Foo' への誤依存を防ぐ。 - SELECT DISTINCT lrp.source_file_id, lrp.symbol_name - FROM logical_references_primary lrp - JOIN target_files tf_alias - ON tf_alias.target_lang = lrp.source_lang - AND tf_alias.symbol_name = lrp.symbol_name || 'Attribute' - AND tf_alias.symbol_segment_count = 1 - AND tf_alias.has_metadata_target_kind = 1 - WHERE lrp.source_lang = 'csharp' - AND lrp.logical_reference_kind = 'attribute' - AND lrp.symbol_name NOT LIKE '%Attribute' - ), - target_ambiguity AS ( - -- Count class-like definitions at symbol-identity level rather than - -- file level. Two same-named class-like definitions in the same file - -- (e.g. `namespace A { class FooAttribute { } } namespace B { class - -- FooAttribute { } }` both inside one .cs file) collapse to a single - -- target_files row because target_files is GROUPed by dst.path, so - -- COUNT(DISTINCT target_path) alone would see count=1 and falsely - -- treat the metadata target as unambiguous. Joining target_files back - -- through files + symbols recovers the per-definition row count while - -- still inheriting target_files' lang / path / graph-supported scope - -- (since the join only keeps rows whose (path, lang, name) already - -- appear in target_files). - -- class-like 定義は path 単位ではなく symbol identity 単位で数える。 - -- 同じ .cs ファイル内に別名前空間で同名 class-like が 2 つあるケースは - -- target_files (dst.path で GROUP BY) 上では 1 行に潰れており、 - -- COUNT(DISTINCT target_path) だけでは count=1 となり metadata target - -- が一意と誤判定される。target_files から files + symbols に JOIN し直す - -- ことで定義単位の件数を復元する。JOIN が target_files 既存行にしか - -- 当たらないため、lang / path / graph-supported スコープはそのまま継承。 - SELECT tf.target_lang, - tf.symbol_name, - tf.symbol_segment_count, - COUNT(*) AS class_like_target_count - FROM target_files tf - JOIN files dst - ON dst.path = tf.target_path - AND dst.lang = tf.target_lang - JOIN symbols s - ON s.file_id = dst.id - AND " + targetLogicalSymbolNameExpr + @" = tf.symbol_name - AND " + targetLogicalSymbolSegmentCountExpr + @" = tf.symbol_segment_count - -- Same language-aware metadata-eligibility filter as - -- target_files: C# restricts to `class` with inheritance - -- clause (interface/struct cannot be attribute targets); - -- JS/TS additionally accepts `function` (decorator - -- factory); others keep the class-like candidate set. - -- target_files と同じ言語別 metadata 適格性フィルタ。 - -- C# は class 限定 + 継承節 (interface/struct は除外)。 - -- JS/TS は decorator factory 用に function も許容。 - -- それ以外は class-like 全体を候補にする。 - AND " + BuildMetadataTargetKindExpr("dst") + @" - WHERE tf.has_metadata_target_kind = 1 - GROUP BY tf.target_lang, tf.symbol_name, tf.symbol_segment_count - ), - edges AS ( - " + resolvedIdentityEdgesSql + @" - SELECT snc.source_path, - tf.target_path, - tf.symbol_name, - snc.ref_count, - snc.source_lang, - CASE - WHEN snc.source_lang = 'markdown' AND tf.has_heading_kind = 1 - THEN 'markdown_heading_name_match' - ELSE 'symbol_name_match' - END AS origin, - snc.raw_reference_kind, - CASE WHEN tf.has_heading_kind = 1 THEN 'heading' ELSE 'symbol' END AS target_kind - FROM bounded_source_name_counts snc - JOIN target_files tf - ON " + sqlDependencyTargetMatchExpr + @" - AND tf.target_lang = snc.source_lang - LEFT JOIN metadata_raw_suppression mrs - ON mrs.source_file_id = snc.source_file_id - AND mrs.symbol_name = snc.symbol_name - LEFT JOIN target_ambiguity ta - ON ta.target_lang = snc.source_lang - AND ta.symbol_name = snc.symbol_name - AND ta.symbol_segment_count = snc.symbol_segment_count - WHERE snc.source_path != tf.target_path - AND " + identityNameEdgePredicate + @" - -- All metadata references ([Foo] / @Foo) and their synthetic C# - -- suffix aliases must only match class-like target kinds; otherwise - -- a metadata reference would spuriously depend on any file that - -- merely defines a function / property / variable sharing the name. - -- Non-metadata call-graph refs keep matching any kind so e.g. a - -- constructor call can still tie back to a class definition. - -- metadata 参照 ([Foo] / @Foo) と C# の合成 alias 行はいずれも - -- class 系の target 種別にのみ一致させる。これを許すと同名の - -- 関数/プロパティ/変数を持つだけのファイルまで誤って依存してしまう。 - -- 非 metadata の call-graph 参照は任意の kind に一致させて構わない - -- (コンストラクタ呼び出しがクラス定義に結び付くケースなど)。 - AND (snc.is_metadata = 0 OR tf.has_metadata_target_kind = 1) - -- Drop raw C# '[Foo]' rows when the suffix alias already resolves - -- to a class-like 'FooAttribute' target in the same source file. - -- 同じ source file で suffix alias が class 系 'FooAttribute' に - -- 解決できている C# の raw '[Foo]' 行は落とす。 - AND NOT ( - snc.is_metadata = 1 - AND snc.is_attribute_alias = 0 - AND snc.source_lang = 'csharp' - AND mrs.source_file_id IS NOT NULL - ) - -- Metadata edges only survive when the target symbol resolves to - -- a single class-like definition within scope; ambiguous cases - -- (multiple same-name attribute / annotation classes) are dropped. - -- metadata エッジは同名 class 系 target が 1 つだけのときのみ残す。 - AND (snc.is_metadata = 0 OR COALESCE(ta.class_like_target_count, 0) <= 1) - -- Python names are file-local bindings. A cross-file edge is valid only - -- when an import in the source file names the referenced symbol/module - -- and that module owns the target path. This prevents ubiquitous names - -- such as Path, main, json, and dataclass from joining every definition. - -- Python の名前はファイルローカルな binding である。cross-file edge は - -- source file の import が参照名/module を束縛し、その module が target - -- path を所有する場合だけ残し、Path/main/json/dataclass の誤結合を防ぐ。 - AND (snc.source_lang != 'python' OR EXISTS ( - SELECT 1 - FROM symbols py_import - WHERE py_import.file_id = snc.source_file_id - AND py_import.kind = 'import' - AND python_import_resolves(snc.source_path, tf.target_path, snc.symbol_name, snc.raw_reference_kind, snc.context, snc.column_number, " + pythonImportSignatureExpr + @") - ))"; - AppendDependencySymbolFilter( - cmd, - ref sql, - "tf.symbol_name", - dependencySymbols, - dependencySymbolFamilies, - suppressDependencyNoise: false, - parameterPrefix: "edgeDependencyNames"); - AppendDependencySymbolFilter( - cmd, - ref sql, - "tf.symbol_name", - dependencySymbols: null, - dependencySymbolFamilies: null, - suppressDependencyNoise: suppressDependencyNoise, - parameterPrefix: "edgeDependencyNoise", - filterScopeSql: "NOT (snc.source_lang = 'markdown' AND tf.has_heading_kind = 1)"); - sql += @" - UNION ALL - -- Resolve explicit Markdown links once per target file. Joining these - -- path references through target_files would multiply one link by every - -- heading or symbol declared in the destination document. - -- 明示的な Markdown link は target file ごとに一度だけ解決する。 - -- target_files 経由で結合すると、1 link が宛先 document 内の全見出し - -- / symbol の件数だけ増幅されるため、file-level path として扱う。 - SELECT snc.source_path, - ptf.target_path, - snc.raw_symbol_name, - snc.ref_count, - snc.source_lang, - 'markdown_explicit_link', - snc.raw_reference_kind, - 'file' - FROM bounded_source_name_counts snc - JOIN path_target_files ptf - ON ptf.target_path = markdown_resolve_path(snc.source_path, snc.symbol_name) - WHERE snc.source_lang = 'markdown' - AND snc.logical_reference_kind = 'import' - AND snc.source_path != ptf.target_path"; - AppendDependencySymbolFilter( - cmd, - ref sql, - "snc.raw_symbol_name", - dependencySymbols, - dependencySymbolFamilies, - suppressDependencyNoise: false, - parameterPrefix: "markdownPathDependency"); - sql += @" - UNION ALL - -- Dockerfile stages are symbols within one file, so their dependency edge is - -- intentionally a self-file edge. Keep this exception stage-specific rather - -- than weakening the cross-file contract for other symbols or languages. - -- Dockerfile stage は同一ファイル内の symbol であるため、この依存 edge は - -- 意図的に self-file edge とする。他の symbol / 言語の cross-file 契約を - -- 緩めないよう stage に限定する。 - SELECT snc.source_path, - snc.source_path, - snc.symbol_name, - snc.ref_count, - snc.source_lang, - 'docker_stage_reference', - snc.raw_reference_kind, - 'stage' - FROM source_name_counts snc - JOIN files self_dst ON self_dst.id = snc.source_file_id - WHERE snc.source_lang = 'dockerfile' - AND snc.raw_reference_kind = 'call' - AND EXISTS ( - SELECT 1 - FROM symbols stage - WHERE stage.file_id = snc.source_file_id - AND stage.kind = 'stage' - AND stage.name = snc.symbol_name COLLATE NOCASE - )"; - if (reverse && pathPatterns is { Count: > 0 }) - { - var ors = new List(pathPatterns.Count); - for (int i = 0; i < pathPatterns.Count; i++) - ors.Add(BuildPathFilterPredicate("self_dst", "pathPattern", i, pathPatterns[i])); - sql += " AND (" + string.Join(" OR ", ors) + ")"; - } - if (reverse && excludePathPatterns is { Count: > 0 }) - { - for (int i = 0; i < excludePathPatterns.Count; i++) - sql += $" AND NOT {BuildPathFilterPredicate("self_dst", "excludePath", i, excludePathPatterns[i])}"; - } - if (reverse && excludeTests) - sql += $" AND NOT {DependencyTestPathCondition("self_dst.path")}"; - AppendDependencySymbolFilter( - cmd, - ref sql, - "snc.symbol_name", - dependencySymbols, - dependencySymbolFamilies, - suppressDependencyNoise, - "dockerDependency"); - sql += @" - UNION ALL - SELECT snc.source_path, - ptf.target_path, - snc.symbol_name, - snc.ref_count, - snc.source_lang, - 'explicit_path_reference', - snc.raw_reference_kind, - 'file' - FROM bounded_source_name_counts snc - JOIN path_target_files ptf - ON ptf.target_path = markdown_resolve_path(snc.source_path, snc.symbol_name) - WHERE snc.source_lang IN ('msbuild', 'solution') - AND snc.logical_reference_kind IN ('import', 'project_reference') - AND snc.source_path != ptf.target_path"; - AppendDependencySymbolFilter( - cmd, - ref sql, - "snc.symbol_name", - dependencySymbols, - dependencySymbolFamilies, - suppressDependencyNoise, - "pathDependency"); - var dependencyReferenceOrderSql = suppressDependencyNoise - ? "retained_reference_count DESC, reference_count DESC, source_path, target_path" - : "reference_count DESC, source_path, target_path"; - var finalDependencyReferenceOrderSql = suppressDependencyNoise - ? "limited_edge_totals.retained_reference_count DESC, limited_edge_totals.reference_count DESC, limited_edge_totals.source_path, limited_edge_totals.target_path" - : "limited_edge_totals.reference_count DESC, limited_edge_totals.source_path, limited_edge_totals.target_path"; - var retainedDependencySymbolFilterSql = suppressDependencyNoise - ? " WHERE edges.origin <> 'markdown_heading_name_match'" - : string.Empty; - sql += @" - ), - edge_totals AS ( - SELECT source_path, - target_path, - SUM(ref_count) AS reference_count, - SUM(CASE WHEN origin = 'markdown_heading_name_match' THEN 0 ELSE ref_count END) AS retained_reference_count - FROM edges - GROUP BY source_path, target_path - ), - limited_edge_totals AS ( - SELECT source_path, - target_path, - reference_count, - retained_reference_count - FROM edge_totals - ORDER BY " + dependencyReferenceOrderSql + @" - LIMIT @limit - ), - edge_evidence_rows AS ( - SELECT edges.source_path, - edges.target_path, - edges.source_lang, - edges.origin, - edges.raw_reference_kind, - edges.target_kind, - SUM(edges.ref_count) AS evidence_reference_count - FROM edges - JOIN limited_edge_totals - ON limited_edge_totals.source_path = edges.source_path - AND limited_edge_totals.target_path = edges.target_path - GROUP BY edges.source_path, - edges.target_path, - edges.source_lang, - edges.origin, - edges.raw_reference_kind, - edges.target_kind - ), - ordered_edge_evidence AS ( - SELECT source_path, - target_path, - source_lang || char(31) || - origin || char(31) || - raw_reference_kind || char(31) || - target_kind || char(31) || - evidence_reference_count AS evidence_item - FROM edge_evidence_rows - ORDER BY source_path, target_path, source_lang, origin, raw_reference_kind, target_kind - ), - edge_evidence_payloads AS ( - SELECT source_path, - target_path, - GROUP_CONCAT(evidence_item, char(30)) AS evidence_payload - FROM ordered_edge_evidence - GROUP BY source_path, target_path - ), - distinct_edge_symbols AS ( - SELECT DISTINCT edges.source_path, - edges.target_path, - edges.symbol_name - FROM edges - JOIN limited_edge_totals - ON limited_edge_totals.source_path = edges.source_path - AND limited_edge_totals.target_path = edges.target_path" + retainedDependencySymbolFilterSql + @" - ), - ranked_edge_symbols AS ( - SELECT source_path, - target_path, - symbol_name, - ROW_NUMBER() OVER (PARTITION BY source_path, target_path ORDER BY symbol_name) AS symbol_rank - FROM distinct_edge_symbols - ) - SELECT limited_edge_totals.source_path, - limited_edge_totals.target_path, - limited_edge_totals.reference_count, - COALESCE(GROUP_CONCAT(CASE WHEN ranked_edge_symbols.symbol_rank <= @symbolSampleLimit THEN ranked_edge_symbols.symbol_name END, char(31)), '') AS symbols, - COALESCE(edge_evidence_payloads.evidence_payload, '') AS evidence_payload - FROM limited_edge_totals - LEFT JOIN ranked_edge_symbols - ON ranked_edge_symbols.source_path = limited_edge_totals.source_path - AND ranked_edge_symbols.target_path = limited_edge_totals.target_path - LEFT JOIN edge_evidence_payloads - ON edge_evidence_payloads.source_path = limited_edge_totals.source_path - AND edge_evidence_payloads.target_path = limited_edge_totals.target_path - GROUP BY limited_edge_totals.source_path, - limited_edge_totals.target_path, - limited_edge_totals.reference_count, - edge_evidence_payloads.evidence_payload - ORDER BY " + finalDependencyReferenceOrderSql; - - cmd.CommandText = sql; - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - if (pathPatterns is { Count: > 0 }) - AddPathFilterParameterSet(cmd, "pathPattern", pathPatterns); - if (excludePathPatterns is { Count: > 0 }) - AddPathFilterParameterSet(cmd, "excludePath", excludePathPatterns); - SqliteCommandPolicy.Add(cmd, "@limit", DependencyNoiseProfile.GetRankingCandidateLimit(limit)); - if (lang == "csharp") - SqliteCommandPolicy.Add(cmd, "@sourceCandidateLimit", DependencyNoiseProfile.GetRankingCandidateLimit(limit)); - SqliteCommandPolicy.Add(cmd, "@symbolSampleLimit", DependencySymbolSampleLimit); - - var results = new List(); - cancellationToken.ThrowIfCancellationRequested(); - using var cancellationRegistration = cancellationToken.Register(static state => ((SqliteCommand)state!).Cancel(), cmd); - try - { - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - cancellationToken.ThrowIfCancellationRequested(); - var symbolSamples = ParseDependencySymbols(reader.GetString(3)); - results.Add(new FileDependencyResult - { - SourcePath = reader.GetString(0), - TargetPath = reader.GetString(1), - ReferenceCount = reader.GetInt32(2), - SymbolSamples = symbolSamples, - Symbols = string.Join(",", symbolSamples), - Evidence = ParseDependencyEvidence(reader.GetString(4)), - }); - } - } - catch (SqliteException) when (cancellationToken.IsCancellationRequested) - { - throw new OperationCanceledException(cancellationToken); - } - return RankDependencyResults(results, limit, suppressDependencyNoise); - } - - internal static List ParseDependencyEvidence(string payload) - { - if (string.IsNullOrEmpty(payload)) - return []; - var evidence = new List(); - foreach (var item in payload.Split('\u001e', StringSplitOptions.RemoveEmptyEntries)) - { - var fields = item.Split('\u001f'); - if (fields.Length != 5 || !int.TryParse(fields[4], out var referenceCount)) - continue; - - evidence.Add(new FileDependencyEvidence - { - SourceLanguage = fields[0], - Origin = fields[1], - ReferenceKind = fields[2], - TargetKind = fields[3], - ReferenceCount = referenceCount, - }); - } - - return evidence; - } - - internal static List ParseDependencySymbols(string payload) - => string.IsNullOrEmpty(payload) - ? [] - : payload.Split('\u001f', StringSplitOptions.RemoveEmptyEntries).ToList(); - - private static List RankDependencyResults( - List results, - int limit, - bool suppressDependencyNoise) - { - foreach (var result in results) - { - var rankingReferenceCount = suppressDependencyNoise && result.Evidence is { Count: > 0 } - ? result.Evidence - .Where(static evidence => evidence.Origin != "markdown_heading_name_match") - .Sum(static evidence => evidence.ReferenceCount) - : result.ReferenceCount; - result.RankingScore = result.SymbolSamples is { } symbolSamples - ? DependencyNoiseProfile.ComputeRankingScore(rankingReferenceCount, symbolSamples) - : DependencyNoiseProfile.ComputeRankingScore(rankingReferenceCount, result.Symbols); - } - - return results - .OrderByDescending(result => result.RankingScore) - .ThenByDescending(result => result.ReferenceCount) - .ThenBy(result => result.SourcePath, StringComparer.Ordinal) - .ThenBy(result => result.TargetPath, StringComparer.Ordinal) - .Take(limit) - .ToList(); - } - - public List GetFileDependencyCycleCandidates( - int limit, - out int candidateRowCount, - string? lang = null, - IReadOnlyList? pathPatterns = null, - IReadOnlyList? excludePathPatterns = null, - bool excludeTests = false, - bool reverse = false, - CancellationToken cancellationToken = default, - IReadOnlyList? dependencySymbols = null, - IReadOnlyList? dependencySymbolFamilies = null, - bool suppressDependencyNoise = false) - { - candidateRowCount = 0; - lang = NormalizeQueryLanguage(lang); - if (!_hasReferencesTable || limit <= 0) - return []; - - cancellationToken.ThrowIfCancellationRequested(); - using var cmd = _conn.CreateCommand(); - var constrainedAlias = reverse ? "dst" : "src"; - var cycleMarkdownExplicitLinkSql = _referenceColumns.Contains("target_qualifier") - ? "(src.lang = 'markdown' AND r.reference_kind = 'reference' AND r.target_qualifier IS NOT NULL AND dst.path = markdown_resolve_path(src.path, r.target_qualifier))" - : "0 = 1"; - var cycleNoiseEvidenceScopeSql = - "(" + cycleMarkdownExplicitLinkSql + " OR (src.lang = 'markdown' AND s.kind = 'heading'))"; - var cycleCandidateOrderSql = suppressDependencyNoise - ? "retained_evidence DESC, source_path, target_path" - : "source_path, target_path"; - var retainedCycleSymbolFilterSql = suppressDependencyNoise - ? " WHERE origin <> 'markdown_heading_name_match'" - : string.Empty; - var sql = @" - WITH candidate_edges AS ( - SELECT src.path AS source_path, - dst.path AS target_path, - MAX(CASE - WHEN src.lang = 'markdown' - AND s.kind = 'heading' - AND NOT " + cycleMarkdownExplicitLinkSql + @" THEN 0 - ELSE 1 - END) AS retained_evidence - FROM symbol_references r - JOIN files src ON r.file_id = src.id - JOIN symbols s ON s.name = r.symbol_name - JOIN files dst ON s.file_id = dst.id - WHERE src.path != dst.path - AND src.lang = dst.lang"; - sql += $" AND {BuildGraphSupportedLanguagePredicate(cmd, "src", "depsCycleSourceLang")}"; - sql += $" AND {BuildGraphSupportedLanguagePredicate(cmd, "dst", "depsCycleTargetLang")}"; - AppendDependencyGeneratedFilter(ref sql, "src"); - AppendDependencyGeneratedFilter(ref sql, "dst"); - if (lang != null) - sql += " AND src.lang = @lang AND dst.lang = @lang"; - if (pathPatterns is { Count: > 0 }) - { - var ors = new List(pathPatterns.Count); - for (int i = 0; i < pathPatterns.Count; i++) - ors.Add(BuildPathFilterPredicate(constrainedAlias, "pathPattern", i, pathPatterns[i])); - sql += " AND (" + string.Join(" OR ", ors) + ")"; - } - if (excludePathPatterns is { Count: > 0 }) - { - for (int i = 0; i < excludePathPatterns.Count; i++) - sql += $" AND NOT {BuildPathFilterPredicate(constrainedAlias, "excludePath", i, excludePathPatterns[i])}"; - } - if (excludeTests) - { - sql += $" AND NOT {DependencyTestPathCondition("src.path")}"; - sql += $" AND NOT {DependencyTestPathCondition("dst.path")}"; - } - AppendDependencySymbolFilter( - cmd, - ref sql, - "r.symbol_name", + var request = new DependencyQueryRequest( + limit, + lang, + pathPatterns, + excludePathPatterns, + excludeTests, + reverse, dependencySymbols, dependencySymbolFamilies, - suppressDependencyNoise: false, - parameterPrefix: "cycleDependencyNames"); - AppendDependencySymbolFilter( - cmd, - ref sql, - "r.symbol_name", - dependencySymbols: null, - dependencySymbolFamilies: null, - suppressDependencyNoise: suppressDependencyNoise, - parameterPrefix: "cycleDependencyNoise", - filterScopeSql: "NOT " + cycleNoiseEvidenceScopeSql); - sql += @" - GROUP BY src.path, dst.path - ORDER BY " + cycleCandidateOrderSql + @" - LIMIT @limit - ), - candidate_symbols AS ( - SELECT candidate_edges.source_path, - candidate_edges.target_path, - r.id AS reference_id, - r.symbol_name, - src.lang AS source_lang, - CASE - WHEN " + cycleMarkdownExplicitLinkSql + @" - THEN 'markdown_explicit_link' - WHEN src.lang = 'markdown' AND s.kind = 'heading' - THEN 'markdown_heading_name_match' - ELSE 'symbol_name_match' - END AS origin, - r.reference_kind AS raw_reference_kind, - CASE WHEN s.kind = 'heading' THEN 'heading' ELSE 'symbol' END AS target_kind - FROM candidate_edges - JOIN files src ON src.path = candidate_edges.source_path - JOIN symbol_references r ON r.file_id = src.id - JOIN symbols s ON s.name = r.symbol_name - JOIN files dst ON s.file_id = dst.id - AND dst.path = candidate_edges.target_path - WHERE src.path != dst.path - AND src.lang = dst.lang"; - AppendDependencySymbolFilter( - cmd, - ref sql, - "r.symbol_name", - dependencySymbols, - dependencySymbolFamilies, - suppressDependencyNoise: false, - parameterPrefix: "cycleAggregateNames"); - AppendDependencySymbolFilter( - cmd, - ref sql, - "r.symbol_name", - dependencySymbols: null, - dependencySymbolFamilies: null, - suppressDependencyNoise: suppressDependencyNoise, - parameterPrefix: "cycleAggregateNoise", - filterScopeSql: "NOT " + cycleNoiseEvidenceScopeSql); - sql += @" - ), - edge_reference_totals AS ( - SELECT source_path, - target_path, - COUNT(DISTINCT reference_id) AS reference_count - FROM candidate_symbols - GROUP BY source_path, target_path - ), - edge_evidence_rows AS ( - SELECT source_path, - target_path, - source_lang, - origin, - raw_reference_kind, - target_kind, - COUNT(DISTINCT reference_id) AS evidence_reference_count - FROM candidate_symbols - GROUP BY source_path, - target_path, - source_lang, - origin, - raw_reference_kind, - target_kind - ), - ordered_edge_evidence AS ( - SELECT source_path, - target_path, - source_lang || char(31) || - origin || char(31) || - raw_reference_kind || char(31) || - target_kind || char(31) || - evidence_reference_count AS evidence_item - FROM edge_evidence_rows - ORDER BY source_path, target_path, source_lang, origin, raw_reference_kind, target_kind - ), - edge_evidence_payloads AS ( - SELECT source_path, - target_path, - GROUP_CONCAT(evidence_item, char(30)) AS evidence_payload - FROM ordered_edge_evidence - GROUP BY source_path, target_path - ), - distinct_edge_symbols AS ( - SELECT DISTINCT source_path, - target_path, - symbol_name - FROM candidate_symbols" + retainedCycleSymbolFilterSql + @" - ), - ranked_edge_symbols AS ( - SELECT source_path, - target_path, - symbol_name, - ROW_NUMBER() OVER (PARTITION BY source_path, target_path ORDER BY symbol_name) AS symbol_rank - FROM distinct_edge_symbols - ) - SELECT edge_reference_totals.source_path, - edge_reference_totals.target_path, - edge_reference_totals.reference_count, - COALESCE(GROUP_CONCAT(CASE WHEN symbol_rank <= @symbolSampleLimit THEN symbol_name END, char(31)), '') AS symbols, - COALESCE(edge_evidence_payloads.evidence_payload, '') AS evidence_payload - FROM edge_reference_totals - LEFT JOIN ranked_edge_symbols - ON ranked_edge_symbols.source_path = edge_reference_totals.source_path - AND ranked_edge_symbols.target_path = edge_reference_totals.target_path - LEFT JOIN edge_evidence_payloads - ON edge_evidence_payloads.source_path = edge_reference_totals.source_path - AND edge_evidence_payloads.target_path = edge_reference_totals.target_path - GROUP BY edge_reference_totals.source_path, - edge_reference_totals.target_path, - edge_reference_totals.reference_count, - edge_evidence_payloads.evidence_payload - ORDER BY edge_reference_totals.source_path, edge_reference_totals.target_path"; - - cmd.CommandText = sql; - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - if (pathPatterns is { Count: > 0 }) - AddPathFilterParameterSet(cmd, "pathPattern", pathPatterns); - if (excludePathPatterns is { Count: > 0 }) - AddPathFilterParameterSet(cmd, "excludePath", excludePathPatterns); - SqliteCommandPolicy.Add(cmd, "@limit", limit); - SqliteCommandPolicy.Add(cmd, "@symbolSampleLimit", DependencySymbolSampleLimit); - - var results = new List(); - using var cancellationRegistration = cancellationToken.Register(static state => ((SqliteCommand)state!).Cancel(), cmd); - try - { - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - cancellationToken.ThrowIfCancellationRequested(); - candidateRowCount++; - var symbolSamples = ParseDependencySymbols(reader.GetString(3)); - results.Add(new FileDependencyResult - { - SourcePath = reader.GetString(0), - TargetPath = reader.GetString(1), - ReferenceCount = reader.GetInt32(2), - RankingScore = reader.GetInt32(2), - SymbolSamples = symbolSamples, - Symbols = string.Join(",", symbolSamples), - Evidence = ParseDependencyEvidence(reader.GetString(4)), - }); - } - } - catch (SqliteException) when (cancellationToken.IsCancellationRequested) - { - throw new OperationCanceledException(cancellationToken); - } - return results; - } - - internal static void AppendDependencySymbolFilter( - SqliteCommand cmd, - ref string sql, - string symbolSql, - IReadOnlyList? dependencySymbols, - IReadOnlyList? dependencySymbolFamilies, - bool suppressDependencyNoise, - string parameterPrefix, - string? filterScopeSql = null) - { - if (dependencySymbols is { Count: > 0 } || dependencySymbolFamilies is { Count: > 0 }) - { - var predicates = new List( - (dependencySymbols?.Count ?? 0) + (dependencySymbolFamilies?.Count ?? 0)); - if (dependencySymbols != null) - { - for (var i = 0; i < dependencySymbols.Count; i++) - { - var parameterName = $"@{parameterPrefix}Symbol{i}"; - predicates.Add($"({symbolSql}) = {parameterName}"); - SqliteCommandPolicy.Add(cmd, parameterName, dependencySymbols[i]); - } - } - if (dependencySymbolFamilies != null) - { - for (var i = 0; i < dependencySymbolFamilies.Count; i++) - { - var parameterName = $"@{parameterPrefix}Family{i}"; - predicates.Add($"({symbolSql}) GLOB {parameterName}"); - SqliteCommandPolicy.Add(cmd, parameterName, EscapeSqliteGlobLiteral(dependencySymbolFamilies[i]) + "*"); - } - } - AppendScopedDependencyPredicate(ref sql, "(" + string.Join(" OR ", predicates) + ")", filterScopeSql); - } - - if (!suppressDependencyNoise) - return; - - var noiseParameters = new List(DependencyNoiseProfile.SymbolNames.Length); - for (var i = 0; i < DependencyNoiseProfile.SymbolNames.Length; i++) - { - var parameterName = $"@{parameterPrefix}Noise{i}"; - noiseParameters.Add(parameterName); - SqliteCommandPolicy.Add(cmd, parameterName, DependencyNoiseProfile.SymbolNames[i]); - } - AppendScopedDependencyPredicate( - ref sql, - $"({symbolSql}) COLLATE NOCASE NOT IN ({string.Join(", ", noiseParameters)})", - filterScopeSql); + suppressDependencyNoise); + return ExecuteDependencyQuery(BuildDependencyQueryPlan(request), cancellationToken); } - private static void AppendScopedDependencyPredicate(ref string sql, string predicate, string? filterScopeSql) - { - sql += filterScopeSql == null - ? " AND " + predicate - : $" AND (NOT ({filterScopeSql}) OR {predicate})"; - } - private static string EscapeSqliteGlobLiteral(string value) - => value - .Replace("[", "[[]", StringComparison.Ordinal) - .Replace("*", "[*]", StringComparison.Ordinal) - .Replace("?", "[?]", StringComparison.Ordinal); - - private static string DependencyTestPathCondition(string pathSql) - => "(" + TestPathCondition.Replace("f.path", pathSql) + $" OR lower({pathSql}) LIKE '%.test%/%')"; private void AppendDependencyGeneratedFilter(ref string sql, string fileAlias) - { - if (!IncludeGeneratedScope.Value && _fileColumns.Contains("generated")) - sql += $" AND COALESCE({fileAlias}.generated, 0) = 0"; - } + => sql += BuildDependencyGeneratedFilter(fileAlias); } diff --git a/src/CodeIndex/Database/DbReader.DependencyCycleCandidateSql.cs b/src/CodeIndex/Database/DbReader.DependencyCycleCandidateSql.cs new file mode 100644 index 0000000000..bdd25284c5 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyCycleCandidateSql.cs @@ -0,0 +1,114 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class DependencyCycleCandidateSqlBuilder + { + private readonly DbReader _reader; + private readonly DependencyQueryRequest _request; + private readonly DependencyCycleQueryExpressions _expressions; + private readonly DependencySqlFragmentBuilder _sql = new(); + + internal DependencyCycleCandidateSqlBuilder( + DbReader reader, + DependencyQueryRequest request, + DependencyCycleQueryExpressions expressions) + { + _reader = reader; + _request = request; + _expressions = expressions; + } + + internal DependencySqlFragment Build() + { + AppendCandidateEdgeQuery(); + AppendCandidateScope(); + AppendCandidateFilters(); + return _sql.Build(); + } + + private void AppendCandidateEdgeQuery() + { + _sql.Append(@" + WITH candidate_edges AS ( + SELECT src.path AS source_path, + dst.path AS target_path, + MAX(CASE + WHEN src.lang = 'markdown' + AND s.kind = 'heading' + AND NOT " + _expressions.MarkdownExplicitLink + @" THEN 0 + ELSE 1 + END) AS retained_evidence + FROM symbol_references r + JOIN files src ON r.file_id = src.id + JOIN symbols s ON s.name = r.symbol_name + JOIN files dst ON s.file_id = dst.id + WHERE src.path != dst.path + AND src.lang = dst.lang"); + } + + private void AppendCandidateScope() + { + AppendGraphLanguageScope("src", "depsCycleSourceLang"); + AppendGraphLanguageScope("dst", "depsCycleTargetLang"); + _sql.Append(_reader.BuildDependencyGeneratedFilter("src")); + _sql.Append(_reader.BuildDependencyGeneratedFilter("dst")); + if (_request.Lang != null) + _sql.Append(" AND src.lang = @lang AND dst.lang = @lang"); + if (_request.PathPatterns is { Count: > 0 }) + { + var predicates = new List(_request.PathPatterns.Count); + for (var i = 0; i < _request.PathPatterns.Count; i++) + { + predicates.Add(BuildPathFilterPredicate( + _expressions.ConstrainedAlias, + "pathPattern", + i, + _request.PathPatterns[i])); + } + _sql.Append(" AND (" + string.Join(" OR ", predicates) + ")"); + } + if (_request.ExcludePathPatterns is { Count: > 0 }) + { + for (var i = 0; i < _request.ExcludePathPatterns.Count; i++) + { + _sql.Append($" AND NOT {BuildPathFilterPredicate(_expressions.ConstrainedAlias, "excludePath", i, _request.ExcludePathPatterns[i])}"); + } + } + if (_request.ExcludeTests) + { + _sql.Append($" AND NOT {DependencyTestPathCondition("src.path")}"); + _sql.Append($" AND NOT {DependencyTestPathCondition("dst.path")}"); + } + } + + private void AppendGraphLanguageScope(string fileAlias, string parameterPrefix) + { + var predicate = _reader.BuildDependencyGraphLanguagePredicate(fileAlias, parameterPrefix); + _sql.Append(" AND " + predicate.Sql); + _sql.AddParameters(predicate.Parameters); + } + + private void AppendCandidateFilters() + { + _sql.Append(BuildDependencySymbolFilter( + "r.symbol_name", + _request.DependencySymbols, + _request.DependencySymbolFamilies, + suppressDependencyNoise: false, + parameterPrefix: "cycleDependencyNames")); + _sql.Append(BuildDependencySymbolFilter( + "r.symbol_name", + dependencySymbols: null, + dependencySymbolFamilies: null, + suppressDependencyNoise: _request.SuppressDependencyNoise, + parameterPrefix: "cycleDependencyNoise", + filterScopeSql: "NOT " + _expressions.NoiseEvidenceScope)); + _sql.Append(@" + GROUP BY src.path, dst.path + ORDER BY " + _expressions.CandidateOrder + @" + LIMIT @limit + ),"); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencyCycleEvidenceSql.cs b/src/CodeIndex/Database/DbReader.DependencyCycleEvidenceSql.cs new file mode 100644 index 0000000000..9d7946840d --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyCycleEvidenceSql.cs @@ -0,0 +1,156 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class DependencyCycleEvidenceSqlBuilder + { + private readonly DependencyQueryRequest _request; + private readonly DependencyCycleQueryExpressions _expressions; + private readonly DependencySqlFragmentBuilder _sql = new(); + + internal DependencyCycleEvidenceSqlBuilder( + DependencyQueryRequest request, + DependencyCycleQueryExpressions expressions) + { + _request = request; + _expressions = expressions; + } + + internal DependencySqlFragment Build() + { + AppendCandidateSymbols(); + AppendEdgeAggregates(); + AppendEvidencePayloads(); + AppendFinalProjection(); + return _sql.Build(); + } + + private void AppendCandidateSymbols() + { + _sql.Append(@" + candidate_symbols AS ( + SELECT candidate_edges.source_path, + candidate_edges.target_path, + r.id AS reference_id, + r.symbol_name, + src.lang AS source_lang, + CASE + WHEN " + _expressions.MarkdownExplicitLink + @" + THEN 'markdown_explicit_link' + WHEN src.lang = 'markdown' AND s.kind = 'heading' + THEN 'markdown_heading_name_match' + ELSE 'symbol_name_match' + END AS origin, + r.reference_kind AS raw_reference_kind, + CASE WHEN s.kind = 'heading' THEN 'heading' ELSE 'symbol' END AS target_kind + FROM candidate_edges + JOIN files src ON src.path = candidate_edges.source_path + JOIN symbol_references r ON r.file_id = src.id + JOIN symbols s ON s.name = r.symbol_name + JOIN files dst ON s.file_id = dst.id + AND dst.path = candidate_edges.target_path + WHERE src.path != dst.path + AND src.lang = dst.lang"); + _sql.Append(BuildDependencySymbolFilter( + "r.symbol_name", + _request.DependencySymbols, + _request.DependencySymbolFamilies, + suppressDependencyNoise: false, + parameterPrefix: "cycleAggregateNames")); + _sql.Append(BuildDependencySymbolFilter( + "r.symbol_name", + dependencySymbols: null, + dependencySymbolFamilies: null, + suppressDependencyNoise: _request.SuppressDependencyNoise, + parameterPrefix: "cycleAggregateNoise", + filterScopeSql: "NOT " + _expressions.NoiseEvidenceScope)); + } + + private void AppendEdgeAggregates() + { + _sql.Append(@" + ), + edge_reference_totals AS ( + SELECT source_path, + target_path, + COUNT(DISTINCT reference_id) AS reference_count + FROM candidate_symbols + GROUP BY source_path, target_path + ), + edge_evidence_rows AS ( + SELECT source_path, + target_path, + source_lang, + origin, + raw_reference_kind, + target_kind, + COUNT(DISTINCT reference_id) AS evidence_reference_count + FROM candidate_symbols + GROUP BY source_path, + target_path, + source_lang, + origin, + raw_reference_kind, + target_kind + ),"); + } + + private void AppendEvidencePayloads() + { + _sql.Append(@" + ordered_edge_evidence AS ( + SELECT source_path, + target_path, + source_lang || char(31) || + origin || char(31) || + raw_reference_kind || char(31) || + target_kind || char(31) || + evidence_reference_count AS evidence_item + FROM edge_evidence_rows + ORDER BY source_path, target_path, source_lang, origin, raw_reference_kind, target_kind + ), + edge_evidence_payloads AS ( + SELECT source_path, + target_path, + GROUP_CONCAT(evidence_item, char(30)) AS evidence_payload + FROM ordered_edge_evidence + GROUP BY source_path, target_path + ), + distinct_edge_symbols AS ( + SELECT DISTINCT source_path, + target_path, + symbol_name + FROM candidate_symbols" + _expressions.RetainedSymbolFilter + @" + ), + ranked_edge_symbols AS ( + SELECT source_path, + target_path, + symbol_name, + ROW_NUMBER() OVER (PARTITION BY source_path, target_path ORDER BY symbol_name) AS symbol_rank + FROM distinct_edge_symbols + )"); + } + + private void AppendFinalProjection() + { + _sql.Append(@" + SELECT edge_reference_totals.source_path, + edge_reference_totals.target_path, + edge_reference_totals.reference_count, + COALESCE(GROUP_CONCAT(CASE WHEN symbol_rank <= @symbolSampleLimit THEN symbol_name END, char(31)), '') AS symbols, + COALESCE(edge_evidence_payloads.evidence_payload, '') AS evidence_payload + FROM edge_reference_totals + LEFT JOIN ranked_edge_symbols + ON ranked_edge_symbols.source_path = edge_reference_totals.source_path + AND ranked_edge_symbols.target_path = edge_reference_totals.target_path + LEFT JOIN edge_evidence_payloads + ON edge_evidence_payloads.source_path = edge_reference_totals.source_path + AND edge_evidence_payloads.target_path = edge_reference_totals.target_path + GROUP BY edge_reference_totals.source_path, + edge_reference_totals.target_path, + edge_reference_totals.reference_count, + edge_evidence_payloads.evidence_payload + ORDER BY edge_reference_totals.source_path, edge_reference_totals.target_path"); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencyCycleExecution.cs b/src/CodeIndex/Database/DbReader.DependencyCycleExecution.cs new file mode 100644 index 0000000000..03312daa83 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyCycleExecution.cs @@ -0,0 +1,41 @@ +using CodeIndex.Models; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private List ExecuteDependencyCycleQuery( + DependencyCycleQueryPlan plan, + CancellationToken cancellationToken, + out int candidateRowCount) + { + candidateRowCount = 0; + using var command = _conn.CreateCommand(); + command.CommandText = plan.Sql; + BindDependencyQueryParameters(command, plan.Parameters); + + var results = new List(); + using var cancellationRegistration = cancellationToken.Register( + static state => ((SqliteCommand)state!).Cancel(), + command); + try + { + using var reader = command.ExecuteTrackedReader(); + while (reader.TrackedRead()) + { + cancellationToken.ThrowIfCancellationRequested(); + candidateRowCount++; + var result = ProjectDependencyRow(reader); + result.RankingScore = result.ReferenceCount; + results.Add(result); + } + } + catch (SqliteException) when (cancellationToken.IsCancellationRequested) + { + throw new OperationCanceledException(cancellationToken); + } + + return results; + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencyCyclePlan.cs b/src/CodeIndex/Database/DbReader.DependencyCyclePlan.cs new file mode 100644 index 0000000000..39b977b365 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyCyclePlan.cs @@ -0,0 +1,59 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed record DependencyCycleQueryPlan( + string Sql, + IReadOnlyList Parameters); + + private sealed record DependencyCycleQueryExpressions( + string MarkdownExplicitLink, + string NoiseEvidenceScope, + string CandidateOrder, + string RetainedSymbolFilter, + string ConstrainedAlias); + + private DependencyCycleQueryPlan BuildDependencyCycleQueryPlan(DependencyQueryRequest request) + { + var expressions = BuildDependencyCycleQueryExpressions(request); + var candidates = new DependencyCycleCandidateSqlBuilder(this, request, expressions).Build(); + var evidence = new DependencyCycleEvidenceSqlBuilder(request, expressions).Build(); + var builder = new DependencySqlFragmentBuilder(); + builder.Append(candidates.Sql); + builder.Append(evidence.Sql); + builder.AddParameters(candidates.Parameters); + builder.AddParameters(evidence.Parameters); + AppendDependencyCycleTerminalParameters(builder, request); + var query = builder.Build(); + return new DependencyCycleQueryPlan(query.Sql, query.Parameters); + } + + private DependencyCycleQueryExpressions BuildDependencyCycleQueryExpressions(DependencyQueryRequest request) + { + var markdownExplicitLink = _referenceColumns.Contains("target_qualifier") + ? "(src.lang = 'markdown' AND r.reference_kind = 'reference' AND r.target_qualifier IS NOT NULL AND dst.path = markdown_resolve_path(src.path, r.target_qualifier))" + : "0 = 1"; + return new DependencyCycleQueryExpressions( + markdownExplicitLink, + "(" + markdownExplicitLink + " OR (src.lang = 'markdown' AND s.kind = 'heading'))", + request.SuppressDependencyNoise + ? "retained_evidence DESC, source_path, target_path" + : "source_path, target_path", + request.SuppressDependencyNoise + ? " WHERE origin <> 'markdown_heading_name_match'" + : string.Empty, + request.Reverse ? "dst" : "src"); + } + + private static void AppendDependencyCycleTerminalParameters( + DependencySqlFragmentBuilder builder, + DependencyQueryRequest request) + { + if (request.Lang != null) + builder.AddText("@lang", request.Lang); + AppendDependencyPathParameters(builder, "pathPattern", request.PathPatterns); + AppendDependencyPathParameters(builder, "excludePath", request.ExcludePathPatterns); + builder.AddInt32("@limit", request.Limit); + builder.AddInt32("@symbolSampleLimit", DependencySymbolSampleLimit); + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencyCycleQuery.cs b/src/CodeIndex/Database/DbReader.DependencyCycleQuery.cs new file mode 100644 index 0000000000..4dfeb3a0d3 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyCycleQuery.cs @@ -0,0 +1,41 @@ +using CodeIndex.Models; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + public List GetFileDependencyCycleCandidates( + int limit, + out int candidateRowCount, + string? lang = null, + IReadOnlyList? pathPatterns = null, + IReadOnlyList? excludePathPatterns = null, + bool excludeTests = false, + bool reverse = false, + CancellationToken cancellationToken = default, + IReadOnlyList? dependencySymbols = null, + IReadOnlyList? dependencySymbolFamilies = null, + bool suppressDependencyNoise = false) + { + candidateRowCount = 0; + lang = NormalizeQueryLanguage(lang); + if (!_hasReferencesTable || limit <= 0) + return []; + + cancellationToken.ThrowIfCancellationRequested(); + var request = new DependencyQueryRequest( + limit, + lang, + pathPatterns, + excludePathPatterns, + excludeTests, + reverse, + dependencySymbols, + dependencySymbolFamilies, + suppressDependencyNoise); + return ExecuteDependencyCycleQuery( + BuildDependencyCycleQueryPlan(request), + cancellationToken, + out candidateRowCount); + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencyEdgeSql.cs b/src/CodeIndex/Database/DbReader.DependencyEdgeSql.cs new file mode 100644 index 0000000000..a01c0c135b --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyEdgeSql.cs @@ -0,0 +1,121 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class DependencyEdgeSqlBuilder + { + private readonly DependencyQueryRequest _request; + private readonly DependencyQueryExpressions _expressions; + private readonly DependencySqlFragment _resolvedIdentity; + private readonly DependencySqlFragmentBuilder _sql = new(); + + internal DependencyEdgeSqlBuilder( + DependencyQueryRequest request, + DependencyQueryExpressions expressions, + DependencySqlFragment resolvedIdentity) + { + _request = request; + _expressions = expressions; + _resolvedIdentity = resolvedIdentity; + } + + internal DependencySqlFragment Build() + { + AppendNameMatchedEdges(); + AppendNameFilters(); + return _sql.Build(); + } + + private void AppendNameMatchedEdges() + { + _sql.Append(@" + edges AS ( + "); + _sql.Append(_resolvedIdentity.Sql); + _sql.Append(@" + SELECT snc.source_path, + tf.target_path, + tf.symbol_name, + snc.ref_count, + snc.source_lang, + CASE + WHEN snc.source_lang = 'markdown' AND tf.has_heading_kind = 1 + THEN 'markdown_heading_name_match' + ELSE 'symbol_name_match' + END AS origin, + snc.raw_reference_kind, + CASE WHEN tf.has_heading_kind = 1 THEN 'heading' ELSE 'symbol' END AS target_kind + FROM bounded_source_name_counts snc + JOIN target_files tf + ON " + _expressions.SqlDependencyTargetMatch + @" + AND tf.target_lang = snc.source_lang + LEFT JOIN metadata_raw_suppression mrs + ON mrs.source_file_id = snc.source_file_id + AND mrs.symbol_name = snc.symbol_name + LEFT JOIN target_ambiguity ta + ON ta.target_lang = snc.source_lang + AND ta.symbol_name = snc.symbol_name + AND ta.symbol_segment_count = snc.symbol_segment_count + WHERE snc.source_path != tf.target_path + AND " + _expressions.IdentityNameEdgePredicate + @" + -- All metadata references ([Foo] / @Foo) and their synthetic C# + -- suffix aliases must only match class-like target kinds; otherwise + -- a metadata reference would spuriously depend on any file that + -- merely defines a function / property / variable sharing the name. + -- Non-metadata call-graph refs keep matching any kind so e.g. a + -- constructor call can still tie back to a class definition. + -- metadata 参照 ([Foo] / @Foo) と C# の合成 alias 行はいずれも + -- class 系の target 種別にのみ一致させる。これを許すと同名の + -- 関数/プロパティ/変数を持つだけのファイルまで誤って依存してしまう。 + -- 非 metadata の call-graph 参照は任意の kind に一致させて構わない + -- (コンストラクタ呼び出しがクラス定義に結び付くケースなど)。 + AND (snc.is_metadata = 0 OR tf.has_metadata_target_kind = 1) + -- Drop raw C# '[Foo]' rows when the suffix alias already resolves + -- to a class-like 'FooAttribute' target in the same source file. + -- 同じ source file で suffix alias が class 系 'FooAttribute' に + -- 解決できている C# の raw '[Foo]' 行は落とす。 + AND NOT ( + snc.is_metadata = 1 + AND snc.is_attribute_alias = 0 + AND snc.source_lang = 'csharp' + AND mrs.source_file_id IS NOT NULL + ) + -- Metadata edges only survive when the target symbol resolves to + -- a single class-like definition within scope; ambiguous cases + -- (multiple same-name attribute / annotation classes) are dropped. + -- metadata エッジは同名 class 系 target が 1 つだけのときのみ残す。 + AND (snc.is_metadata = 0 OR COALESCE(ta.class_like_target_count, 0) <= 1) + -- Python names are file-local bindings. A cross-file edge is valid only + -- when an import in the source file names the referenced symbol/module + -- and that module owns the target path. This prevents ubiquitous names + -- such as Path, main, json, and dataclass from joining every definition. + -- Python の名前はファイルローカルな binding である。cross-file edge は + -- source file の import が参照名/module を束縛し、その module が target + -- path を所有する場合だけ残し、Path/main/json/dataclass の誤結合を防ぐ。 + AND (snc.source_lang != 'python' OR EXISTS ( + SELECT 1 + FROM symbols py_import + WHERE py_import.file_id = snc.source_file_id + AND py_import.kind = 'import' + AND python_import_resolves(snc.source_path, tf.target_path, snc.symbol_name, snc.raw_reference_kind, snc.context, snc.column_number, " + _expressions.PythonImportSignature + @") + ))"); + } + + private void AppendNameFilters() + { + _sql.Append(BuildDependencySymbolFilter( + "tf.symbol_name", + _request.DependencySymbols, + _request.DependencySymbolFamilies, + suppressDependencyNoise: false, + parameterPrefix: "edgeDependencyNames")); + _sql.Append(BuildDependencySymbolFilter( + "tf.symbol_name", + dependencySymbols: null, + dependencySymbolFamilies: null, + suppressDependencyNoise: _request.SuppressDependencyNoise, + parameterPrefix: "edgeDependencyNoise", + filterScopeSql: "NOT (snc.source_lang = 'markdown' AND tf.has_heading_kind = 1)")); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencyFinalSql.cs b/src/CodeIndex/Database/DbReader.DependencyFinalSql.cs new file mode 100644 index 0000000000..b023edf92a --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyFinalSql.cs @@ -0,0 +1,129 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private static DependencySqlFragment BuildDependencyFinalSql(DependencyQueryRequest request) + { + var sql = new DependencySqlFragmentBuilder(); + AppendDependencyEdgeTotals(sql, request.SuppressDependencyNoise); + AppendDependencyEvidence(sql); + AppendDependencySymbolsAndSelect(sql, request.SuppressDependencyNoise); + return sql.Build(); + } + + private static void AppendDependencyEdgeTotals( + DependencySqlFragmentBuilder sql, + bool suppressDependencyNoise) + { + var orderSql = suppressDependencyNoise + ? "retained_reference_count DESC, reference_count DESC, source_path, target_path" + : "reference_count DESC, source_path, target_path"; + sql.Append(@" + ), + edge_totals AS ( + SELECT source_path, + target_path, + SUM(ref_count) AS reference_count, + SUM(CASE WHEN origin = 'markdown_heading_name_match' THEN 0 ELSE ref_count END) AS retained_reference_count + FROM edges + GROUP BY source_path, target_path + ), + limited_edge_totals AS ( + SELECT source_path, + target_path, + reference_count, + retained_reference_count + FROM edge_totals + ORDER BY " + orderSql + @" + LIMIT @limit + ),"); + } + + private static void AppendDependencyEvidence(DependencySqlFragmentBuilder sql) + { + sql.Append(@" + edge_evidence_rows AS ( + SELECT edges.source_path, + edges.target_path, + edges.source_lang, + edges.origin, + edges.raw_reference_kind, + edges.target_kind, + SUM(edges.ref_count) AS evidence_reference_count + FROM edges + JOIN limited_edge_totals + ON limited_edge_totals.source_path = edges.source_path + AND limited_edge_totals.target_path = edges.target_path + GROUP BY edges.source_path, + edges.target_path, + edges.source_lang, + edges.origin, + edges.raw_reference_kind, + edges.target_kind + ), + ordered_edge_evidence AS ( + SELECT source_path, + target_path, + source_lang || char(31) || + origin || char(31) || + raw_reference_kind || char(31) || + target_kind || char(31) || + evidence_reference_count AS evidence_item + FROM edge_evidence_rows + ORDER BY source_path, target_path, source_lang, origin, raw_reference_kind, target_kind + ), + edge_evidence_payloads AS ( + SELECT source_path, + target_path, + GROUP_CONCAT(evidence_item, char(30)) AS evidence_payload + FROM ordered_edge_evidence + GROUP BY source_path, target_path + ),"); + } + + private static void AppendDependencySymbolsAndSelect( + DependencySqlFragmentBuilder sql, + bool suppressDependencyNoise) + { + var retainedFilterSql = suppressDependencyNoise + ? " WHERE edges.origin <> 'markdown_heading_name_match'" + : string.Empty; + var finalOrderSql = suppressDependencyNoise + ? "limited_edge_totals.retained_reference_count DESC, limited_edge_totals.reference_count DESC, limited_edge_totals.source_path, limited_edge_totals.target_path" + : "limited_edge_totals.reference_count DESC, limited_edge_totals.source_path, limited_edge_totals.target_path"; + sql.Append(@" + distinct_edge_symbols AS ( + SELECT DISTINCT edges.source_path, + edges.target_path, + edges.symbol_name + FROM edges + JOIN limited_edge_totals + ON limited_edge_totals.source_path = edges.source_path + AND limited_edge_totals.target_path = edges.target_path" + retainedFilterSql + @" + ), + ranked_edge_symbols AS ( + SELECT source_path, + target_path, + symbol_name, + ROW_NUMBER() OVER (PARTITION BY source_path, target_path ORDER BY symbol_name) AS symbol_rank + FROM distinct_edge_symbols + ) + SELECT limited_edge_totals.source_path, + limited_edge_totals.target_path, + limited_edge_totals.reference_count, + COALESCE(GROUP_CONCAT(CASE WHEN ranked_edge_symbols.symbol_rank <= @symbolSampleLimit THEN ranked_edge_symbols.symbol_name END, char(31)), '') AS symbols, + COALESCE(edge_evidence_payloads.evidence_payload, '') AS evidence_payload + FROM limited_edge_totals + LEFT JOIN ranked_edge_symbols + ON ranked_edge_symbols.source_path = limited_edge_totals.source_path + AND ranked_edge_symbols.target_path = limited_edge_totals.target_path + LEFT JOIN edge_evidence_payloads + ON edge_evidence_payloads.source_path = limited_edge_totals.source_path + AND edge_evidence_payloads.target_path = limited_edge_totals.target_path + GROUP BY limited_edge_totals.source_path, + limited_edge_totals.target_path, + limited_edge_totals.reference_count, + edge_evidence_payloads.evidence_payload + ORDER BY " + finalOrderSql); + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencyProjection.cs b/src/CodeIndex/Database/DbReader.DependencyProjection.cs new file mode 100644 index 0000000000..cb733d9b21 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyProjection.cs @@ -0,0 +1,77 @@ +using CodeIndex.Models; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private static FileDependencyResult ProjectDependencyRow(SqliteDataReader reader) + { + var symbolSamples = ParseDependencySymbols(reader.GetString(3)); + return new FileDependencyResult + { + SourcePath = reader.GetString(0), + TargetPath = reader.GetString(1), + ReferenceCount = reader.GetInt32(2), + SymbolSamples = symbolSamples, + Symbols = string.Join(",", symbolSamples), + Evidence = ParseDependencyEvidence(reader.GetString(4)), + }; + } + + internal static List ParseDependencyEvidence(string payload) + { + if (string.IsNullOrEmpty(payload)) + return []; + + var evidence = new List(); + foreach (var item in payload.Split('\u001e', StringSplitOptions.RemoveEmptyEntries)) + { + var fields = item.Split('\u001f'); + if (fields.Length != 5 || !int.TryParse(fields[4], out var referenceCount)) + continue; + + evidence.Add(new FileDependencyEvidence + { + SourceLanguage = fields[0], + Origin = fields[1], + ReferenceKind = fields[2], + TargetKind = fields[3], + ReferenceCount = referenceCount, + }); + } + + return evidence; + } + + internal static List ParseDependencySymbols(string payload) + => string.IsNullOrEmpty(payload) + ? [] + : payload.Split('\u001f', StringSplitOptions.RemoveEmptyEntries).ToList(); + + private static List RankDependencyResults( + List results, + int limit, + bool suppressDependencyNoise) + { + foreach (var result in results) + { + var rankingReferenceCount = suppressDependencyNoise && result.Evidence is { Count: > 0 } + ? result.Evidence + .Where(static evidence => evidence.Origin != "markdown_heading_name_match") + .Sum(static evidence => evidence.ReferenceCount) + : result.ReferenceCount; + result.RankingScore = result.SymbolSamples is { } symbolSamples + ? DependencyNoiseProfile.ComputeRankingScore(rankingReferenceCount, symbolSamples) + : DependencyNoiseProfile.ComputeRankingScore(rankingReferenceCount, result.Symbols); + } + + return results + .OrderByDescending(result => result.RankingScore) + .ThenByDescending(result => result.ReferenceCount) + .ThenBy(result => result.SourcePath, StringComparer.Ordinal) + .ThenBy(result => result.TargetPath, StringComparer.Ordinal) + .Take(limit) + .ToList(); + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencyQueryBinding.cs b/src/CodeIndex/Database/DbReader.DependencyQueryBinding.cs new file mode 100644 index 0000000000..53ceef54a0 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyQueryBinding.cs @@ -0,0 +1,179 @@ +using CodeIndex.Indexer; +using CodeIndex.Models; +using Microsoft.Data.Sqlite; +using System.Text; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private DependencySqlFragment BuildDependencyGraphLanguagePredicate(string fileAlias, string parameterPrefix) + { + var supportedLanguages = GetWorkspaceSupportedReferenceLanguages() + .OrderBy(language => language, StringComparer.Ordinal) + .ToList(); + if (supportedLanguages.Count == 0) + return new DependencySqlFragment("1 = 0", Array.Empty()); + + var parameters = new DependencyQueryParameter[supportedLanguages.Count]; + for (var i = 0; i < supportedLanguages.Count; i++) + { + var parameterName = $"@{parameterPrefix}{i}"; + parameters[i] = DependencyQueryParameter.Text(parameterName, supportedLanguages[i]); + } + + return new DependencySqlFragment( + BuildGraphSupportedLanguagePredicateSql(supportedLanguages, fileAlias, parameterPrefix), + parameters); + } + + private static DependencySqlFragment BuildDependencySymbolFilter( + string symbolSql, + IReadOnlyList? dependencySymbols, + IReadOnlyList? dependencySymbolFamilies, + bool suppressDependencyNoise, + string parameterPrefix, + string? filterScopeSql = null) + { + var sql = new StringBuilder(); + var parameters = new List(); + AppendRequestedDependencySymbols( + sql, + parameters, + symbolSql, + dependencySymbols, + dependencySymbolFamilies, + parameterPrefix, + filterScopeSql); + AppendDependencyNoiseFilter( + sql, + parameters, + symbolSql, + suppressDependencyNoise, + parameterPrefix, + filterScopeSql); + return new DependencySqlFragment(sql.ToString(), parameters.ToArray()); + } + + private static void AppendRequestedDependencySymbols( + StringBuilder sql, + List parameters, + string symbolSql, + IReadOnlyList? dependencySymbols, + IReadOnlyList? dependencySymbolFamilies, + string parameterPrefix, + string? filterScopeSql) + { + if (dependencySymbols is not { Count: > 0 } && dependencySymbolFamilies is not { Count: > 0 }) + return; + + var predicates = new List((dependencySymbols?.Count ?? 0) + (dependencySymbolFamilies?.Count ?? 0)); + if (dependencySymbols != null) + { + for (var i = 0; i < dependencySymbols.Count; i++) + { + var parameterName = $"@{parameterPrefix}Symbol{i}"; + predicates.Add($"({symbolSql}) = {parameterName}"); + parameters.Add(DependencyQueryParameter.Text(parameterName, dependencySymbols[i])); + } + } + if (dependencySymbolFamilies != null) + { + for (var i = 0; i < dependencySymbolFamilies.Count; i++) + { + var parameterName = $"@{parameterPrefix}Family{i}"; + predicates.Add($"({symbolSql}) GLOB {parameterName}"); + parameters.Add(DependencyQueryParameter.Text( + parameterName, + EscapeSqliteGlobLiteral(dependencySymbolFamilies[i]) + "*")); + } + } + + AppendScopedDependencyPredicate(sql, "(" + string.Join(" OR ", predicates) + ")", filterScopeSql); + } + + private static void AppendDependencyNoiseFilter( + StringBuilder sql, + List parameters, + string symbolSql, + bool suppressDependencyNoise, + string parameterPrefix, + string? filterScopeSql) + { + if (!suppressDependencyNoise) + return; + + var parameterNames = new string[DependencyNoiseProfile.SymbolNames.Length]; + for (var i = 0; i < DependencyNoiseProfile.SymbolNames.Length; i++) + { + var parameterName = $"@{parameterPrefix}Noise{i}"; + parameterNames[i] = parameterName; + parameters.Add(DependencyQueryParameter.Text(parameterName, DependencyNoiseProfile.SymbolNames[i])); + } + AppendScopedDependencyPredicate( + sql, + $"({symbolSql}) COLLATE NOCASE NOT IN ({string.Join(", ", parameterNames)})", + filterScopeSql); + } + + internal static void AppendDependencySymbolFilter( + SqliteCommand cmd, + ref string sql, + string symbolSql, + IReadOnlyList? dependencySymbols, + IReadOnlyList? dependencySymbolFamilies, + bool suppressDependencyNoise, + string parameterPrefix, + string? filterScopeSql = null) + { + var fragment = BuildDependencySymbolFilter( + symbolSql, + dependencySymbols, + dependencySymbolFamilies, + suppressDependencyNoise, + parameterPrefix, + filterScopeSql); + sql += fragment.Sql; + BindDependencyQueryParameters(cmd, fragment.Parameters); + } + + private static void AppendScopedDependencyPredicate(StringBuilder sql, string predicate, string? filterScopeSql) + => sql.Append(filterScopeSql == null + ? " AND " + predicate + : $" AND (NOT ({filterScopeSql}) OR {predicate})"); + + private static void BindDependencyQueryParameters( + SqliteCommand command, + IReadOnlyList parameters) + { + for (var i = 0; i < parameters.Count; i++) + { + var parameter = parameters[i]; + switch (parameter.Kind) + { + case DependencyQueryParameterKind.Text: + SqliteCommandPolicy.Add(command, parameter.Name, parameter.TextValue!); + break; + case DependencyQueryParameterKind.Int32: + SqliteCommandPolicy.Add(command, parameter.Name, parameter.Int32Value); + break; + default: + throw new InvalidOperationException($"Unsupported dependency parameter kind: {parameter.Kind}"); + } + } + } + + private static string EscapeSqliteGlobLiteral(string value) + => value + .Replace("[", "[[]", StringComparison.Ordinal) + .Replace("*", "[*]", StringComparison.Ordinal) + .Replace("?", "[?]", StringComparison.Ordinal); + + private static string DependencyTestPathCondition(string pathSql) + => "(" + TestPathCondition.Replace("f.path", pathSql) + $" OR lower({pathSql}) LIKE '%.test%/%')"; + + private string BuildDependencyGeneratedFilter(string fileAlias) + => !IncludeGeneratedScope.Value && _fileColumns.Contains("generated") + ? $" AND COALESCE({fileAlias}.generated, 0) = 0" + : string.Empty; +} diff --git a/src/CodeIndex/Database/DbReader.DependencyQueryExecution.cs b/src/CodeIndex/Database/DbReader.DependencyQueryExecution.cs new file mode 100644 index 0000000000..0b81bdc881 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyQueryExecution.cs @@ -0,0 +1,40 @@ +using CodeIndex.Models; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private List ExecuteDependencyQuery( + DependencyQueryPlan plan, + CancellationToken cancellationToken) + { + using var command = _conn.CreateCommand(); + command.CommandText = plan.Sql; + BindDependencyQueryParameters(command, plan.Parameters); + + var results = new List(); + cancellationToken.ThrowIfCancellationRequested(); + using var cancellationRegistration = cancellationToken.Register( + static state => ((SqliteCommand)state!).Cancel(), + command); + try + { + using var reader = command.ExecuteTrackedReader(); + while (reader.TrackedRead()) + { + cancellationToken.ThrowIfCancellationRequested(); + results.Add(ProjectDependencyRow(reader)); + } + } + catch (SqliteException) when (cancellationToken.IsCancellationRequested) + { + throw new OperationCanceledException(cancellationToken); + } + + return RankDependencyResults( + results, + plan.Request.Limit, + plan.Request.SuppressDependencyNoise); + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencyQueryPlan.cs b/src/CodeIndex/Database/DbReader.DependencyQueryPlan.cs new file mode 100644 index 0000000000..3ed618b560 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyQueryPlan.cs @@ -0,0 +1,215 @@ +using CodeIndex.Indexer; +using CodeIndex.Models; +using System.Text; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed record DependencyQueryRequest( + int Limit, + string? Lang, + IReadOnlyList? PathPatterns, + IReadOnlyList? ExcludePathPatterns, + bool ExcludeTests, + bool Reverse, + IReadOnlyList? DependencySymbols, + IReadOnlyList? DependencySymbolFamilies, + bool SuppressDependencyNoise); + + private sealed record DependencyQueryPlan( + DependencyQueryRequest Request, + string Sql, + IReadOnlyList Parameters); + + private sealed record DependencySqlFragment( + string Sql, + IReadOnlyList Parameters) + { + internal static readonly DependencySqlFragment Empty = new(string.Empty, Array.Empty()); + } + + private enum DependencyQueryParameterKind + { + Text, + Int32, + } + + private readonly record struct DependencyQueryParameter( + string Name, + DependencyQueryParameterKind Kind, + string? TextValue, + int Int32Value) + { + internal static DependencyQueryParameter Text(string name, string value) + => new(name, DependencyQueryParameterKind.Text, value, default); + + internal static DependencyQueryParameter Int32(string name, int value) + => new(name, DependencyQueryParameterKind.Int32, null, value); + } + + private sealed class DependencySqlFragmentBuilder + { + private readonly StringBuilder _sql = new(); + private readonly List _parameters = []; + + internal void Append(string sql) => _sql.Append(sql); + + internal void Append(DependencySqlFragment fragment) + { + _sql.Append(fragment.Sql); + AddParameters(fragment.Parameters); + } + + internal void AddParameters(IReadOnlyList parameters) + { + for (var i = 0; i < parameters.Count; i++) + _parameters.Add(parameters[i]); + } + + internal void AddText(string name, string value) + => _parameters.Add(DependencyQueryParameter.Text(name, value)); + + internal void AddInt32(string name, int value) + => _parameters.Add(DependencyQueryParameter.Int32(name, value)); + + internal DependencySqlFragment Build() + => new(_sql.ToString(), _parameters.ToArray()); + } + + private sealed record DependencyQueryExpressions( + string ReferenceLineJoin, + string ContextSql, + string ReferenceIdSql, + string ScopedResolutionStateSql, + string IdentityScopedSql, + string IdentityNameEdgePredicate, + string TargetLogicalSymbolName, + string TargetLogicalSymbolSegmentCount, + string PythonImportMatchSignature, + string PythonImportSignature, + string SqlDependencyTargetMatch); + + private DependencyQueryExpressions BuildDependencyQueryExpressions() + { + var identityScopeCondition = """ + ( + (src.lang = 'csharp' AND r.reference_kind NOT IN ('attribute', 'annotation')) + OR (src.lang = 'dependency_lock' AND r.reference_kind = 'dependency') + ) + """; + var resolutionState = _referenceIdentityContractCurrent ? "r.resolution_state" : "NULL"; + var pythonImportMatchSignature = GetSymbolColumnSql("signature", "NULL", "py_import_match"); + var sqlDependencyTargetMatch = @"( + (tf.target_lang != 'sql' + AND NOT (snc.source_lang IN ('msbuild', 'solution') AND snc.logical_reference_kind IN ('import', 'project_reference')) + AND NOT (snc.source_lang = 'markdown' AND snc.logical_reference_kind = 'import') + AND NOT (snc.source_lang = 'markdown' AND snc.logical_reference_kind = 'reference') + AND tf.symbol_name = snc.symbol_name) + OR (snc.source_lang = 'python' + AND tf.target_lang = 'python' + AND EXISTS ( + SELECT 1 FROM symbols py_import_match + WHERE py_import_match.file_id = snc.source_file_id + AND py_import_match.kind = 'import' + AND python_import_target_name(snc.source_path, snc.symbol_name, snc.context, snc.column_number, " + pythonImportMatchSignature + @") = tf.symbol_name + )) + OR (tf.target_lang = 'sql' AND ( + (tf.symbol_segment_count = snc.symbol_segment_count AND tf.symbol_name = snc.symbol_name COLLATE NOCASE) + OR (sql_segment_count(snc.raw_symbol_name) = 1 + AND snc.allow_leaf_fallback = 1 + AND tf.symbol_segment_count > 1 + AND sql_leaf_name(tf.symbol_name) = snc.raw_symbol_name COLLATE NOCASE + AND NOT EXISTS ( + SELECT 1 + FROM target_files tf_exact + WHERE tf_exact.target_lang = tf.target_lang + AND tf_exact.symbol_segment_count = 1 + AND tf_exact.symbol_name = snc.symbol_name COLLATE NOCASE + ) + AND NOT EXISTS ( + SELECT 1 + FROM target_files tf_resolved + WHERE tf_resolved.target_lang = tf.target_lang + AND tf_resolved.symbol_segment_count = snc.symbol_segment_count + AND tf_resolved.symbol_name = snc.symbol_name COLLATE NOCASE + )) + )) + )"; + + return new DependencyQueryExpressions( + ReferenceLineJoinSql("r"), + ReferenceContextSql("r"), + $"CASE WHEN {identityScopeCondition} THEN r.id ELSE 0 END", + $"CASE WHEN {identityScopeCondition} THEN {resolutionState} ELSE NULL END", + $"CASE WHEN {identityScopeCondition} THEN 1 ELSE 0 END", + _referenceIdentityContractCurrent ? "snc.identity_scoped = 0" : "snc.source_lang <> 'dependency_lock'", + BuildLogicalDependencySymbolNameExpr("dst", "s.name"), + BuildLogicalDependencySymbolSegmentCountExpr("dst", "s.name"), + pythonImportMatchSignature, + GetSymbolColumnSql("signature", "NULL", "py_import"), + sqlDependencyTargetMatch); + } + + private DependencyQueryPlan BuildDependencyQueryPlan(DependencyQueryRequest request) + { + var expressions = BuildDependencyQueryExpressions(); + var resolvedIdentity = BuildResolvedDependencyIdentitySql(request); + var source = new DependencySourceSqlBuilder(this, request, expressions).Build(); + var sourceCandidate = new DependencySourceCandidateSqlBuilder(this, request, expressions).Build(); + var target = new DependencyTargetSqlBuilder(this, request, expressions).Build(); + var edge = new DependencyEdgeSqlBuilder(request, expressions, resolvedIdentity).Build(); + var specialEdges = new DependencySpecialEdgeSqlBuilder(request).Build(); + var final = BuildDependencyFinalSql(request); + + var builder = new DependencySqlFragmentBuilder(); + builder.Append(source.Sql); + builder.Append(sourceCandidate.Sql); + builder.Append(target.Sql); + builder.Append(edge.Sql); + builder.Append(specialEdges.Sql); + builder.Append(final.Sql); + + // Preserve the historical binding order even though the resolved-identity + // fragment appears after the source/target CTEs in the final SQL text. + builder.AddParameters(resolvedIdentity.Parameters); + builder.AddParameters(source.Parameters); + builder.AddParameters(sourceCandidate.Parameters); + builder.AddParameters(target.Parameters); + builder.AddParameters(edge.Parameters); + builder.AddParameters(specialEdges.Parameters); + AppendDependencyTerminalParameters(builder, request); + var fragment = builder.Build(); + return new DependencyQueryPlan(request, fragment.Sql, fragment.Parameters); + } + + private static void AppendDependencyTerminalParameters( + DependencySqlFragmentBuilder builder, + DependencyQueryRequest request) + { + if (request.Lang != null) + builder.AddText("@lang", request.Lang); + AppendDependencyPathParameters(builder, "pathPattern", request.PathPatterns); + AppendDependencyPathParameters(builder, "excludePath", request.ExcludePathPatterns); + builder.AddInt32("@limit", DependencyNoiseProfile.GetRankingCandidateLimit(request.Limit)); + if (request.Lang == "csharp") + builder.AddInt32("@sourceCandidateLimit", DependencyNoiseProfile.GetRankingCandidateLimit(request.Limit)); + builder.AddInt32("@symbolSampleLimit", DependencySymbolSampleLimit); + } + + private static void AppendDependencyPathParameters( + DependencySqlFragmentBuilder builder, + string parameterPrefix, + IReadOnlyList? pathPatterns) + { + if (pathPatterns == null) + return; + + for (var i = 0; i < pathPatterns.Count; i++) + { + builder.AddText( + SqliteDynamicSql.BuildParameterName(parameterPrefix, i), + BuildPathLikePattern(pathPatterns[i])); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencyResolvedIdentitySql.cs b/src/CodeIndex/Database/DbReader.DependencyResolvedIdentitySql.cs new file mode 100644 index 0000000000..12da16fa56 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyResolvedIdentitySql.cs @@ -0,0 +1,55 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private DependencySqlFragment BuildResolvedDependencyIdentitySql(DependencyQueryRequest request) + { + if (!_referenceIdentityContractCurrent) + return DependencySqlFragment.Empty; + + var builder = new DependencySqlFragmentBuilder(); + builder.Append(@" + SELECT resolved.source_path, + resolved.target_path, + resolved.symbol_name, + COUNT(*) AS ref_count, + resolved.source_lang, + 'resolved_identity' AS origin, + resolved.raw_reference_kind, + resolved.target_kind + FROM ( + SELECT DISTINCT lrp.source_path, + target_file.path AS target_path, + lrp.symbol_name, + lrp.reference_id, + lrp.source_lang, + lrp.raw_reference_kind, + target.kind AS target_kind + FROM logical_references_primary lrp + JOIN symbol_reference_candidates candidate + ON candidate.reference_id = lrp.reference_id + JOIN symbols target ON target.id = candidate.symbol_id + JOIN files target_file ON target_file.id = target.file_id + JOIN target_files scoped_target + ON scoped_target.target_path = target_file.path + AND scoped_target.target_lang = target_file.lang + WHERE lrp.identity_scoped = 1 + AND lrp.resolution_state IN ('resolved', 'resolved_group') + AND lrp.source_path != target_file.path"); + builder.Append(BuildDependencySymbolFilter( + "lrp.symbol_name", + request.DependencySymbols, + request.DependencySymbolFamilies, + request.SuppressDependencyNoise, + "resolvedDependency")); + var limitSql = request.Lang == "csharp" ? " LIMIT @sourceCandidateLimit" : string.Empty; + builder.Append(@" + ORDER BY lrp.source_path, lrp.symbol_name, lrp.reference_id" + limitSql + @" + ) resolved + GROUP BY resolved.source_path, resolved.target_path, resolved.symbol_name, + resolved.source_lang, resolved.raw_reference_kind, resolved.target_kind + UNION ALL + "); + return builder.Build(); + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencySourceCandidateSql.cs b/src/CodeIndex/Database/DbReader.DependencySourceCandidateSql.cs new file mode 100644 index 0000000000..89f5407567 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencySourceCandidateSql.cs @@ -0,0 +1,86 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class DependencySourceCandidateSqlBuilder + { + private readonly DbReader _reader; + private readonly DependencyQueryRequest _request; + private readonly DependencyQueryExpressions _expressions; + + internal DependencySourceCandidateSqlBuilder( + DbReader reader, + DependencyQueryRequest request, + DependencyQueryExpressions expressions) + { + _reader = reader; + _request = request; + _expressions = expressions; + } + + internal DependencySqlFragment Build() + => _request.Lang == "csharp" ? BuildBoundedCSharpSource() : BuildUnboundedSource(); + + private DependencySqlFragment BuildBoundedCSharpSource() + { + var sql = new DependencySqlFragmentBuilder(); + sql.Append(@", + csharp_dependency_targets AS ( + SELECT dst.path AS target_path, + " + _expressions.TargetLogicalSymbolName + @" AS symbol_name, + MAX(CASE WHEN " + _reader.BuildMetadataTargetKindExpr("dst") + @" THEN 1 ELSE 0 END) AS has_metadata_target_kind + FROM symbols s + JOIN files dst ON s.file_id = dst.id + WHERE dst.lang = 'csharp'"); + sql.Append(_reader.BuildDependencyGeneratedFilter("dst")); + AppendReverseTargetScope(sql); + sql.Append(BuildDependencySymbolFilter( + _expressions.TargetLogicalSymbolName, + _request.DependencySymbols, + _request.DependencySymbolFamilies, + _request.SuppressDependencyNoise, + "boundedTargetDependency")); + sql.Append(@" + GROUP BY dst.path, " + _expressions.TargetLogicalSymbolName + @" + ), + bounded_source_name_counts AS ( + SELECT snc.* + FROM source_name_counts snc + WHERE EXISTS ( + SELECT 1 + FROM csharp_dependency_targets tf + WHERE tf.symbol_name = snc.symbol_name + AND tf.target_path != snc.source_path + AND (snc.is_metadata = 0 OR tf.has_metadata_target_kind = 1) + ) + ORDER BY snc.ref_count DESC, snc.source_path, snc.symbol_name, snc.context, snc.column_number, snc.raw_reference_kind + LIMIT @sourceCandidateLimit + ),"); + return sql.Build(); + } + + private void AppendReverseTargetScope(DependencySqlFragmentBuilder sql) + { + if (_request.Reverse && _request.PathPatterns is { Count: > 0 }) + { + var predicates = new List(_request.PathPatterns.Count); + for (var i = 0; i < _request.PathPatterns.Count; i++) + predicates.Add(BuildPathFilterPredicate("dst", "pathPattern", i, _request.PathPatterns[i])); + sql.Append(" AND (" + string.Join(" OR ", predicates) + ")"); + } + if (_request.Reverse && _request.ExcludePathPatterns is { Count: > 0 }) + { + for (var i = 0; i < _request.ExcludePathPatterns.Count; i++) + sql.Append($" AND NOT {BuildPathFilterPredicate("dst", "excludePath", i, _request.ExcludePathPatterns[i])}"); + } + if (_request.ExcludeTests) + sql.Append($" AND NOT {DependencyTestPathCondition("dst.path")}"); + } + + private static DependencySqlFragment BuildUnboundedSource() + => new(@", + bounded_source_name_counts AS ( + SELECT * FROM source_name_counts + ),", Array.Empty()); + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencySourceSql.cs b/src/CodeIndex/Database/DbReader.DependencySourceSql.cs new file mode 100644 index 0000000000..40280bd9a9 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencySourceSql.cs @@ -0,0 +1,161 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class DependencySourceSqlBuilder + { + private const string SourceAlias = "src"; + + private readonly DbReader _reader; + private readonly DependencyQueryRequest _request; + private readonly DependencyQueryExpressions _expressions; + private readonly DependencySqlFragmentBuilder _sql = new(); + + internal DependencySourceSqlBuilder( + DbReader reader, + DependencyQueryRequest request, + DependencyQueryExpressions expressions) + { + _reader = reader; + _request = request; + _expressions = expressions; + } + + internal DependencySqlFragment Build() + { + AppendPrimaryReferences(); + AppendSourceScope(); + AppendLogicalReferences(); + AppendSourceNameCounts(); + return _sql.Build(); + } + + private void AppendPrimaryReferences() + { + _sql.Append(@" + WITH logical_references_primary AS ( + SELECT src.id AS source_file_id, + src.path AS source_path, + src.lang AS source_lang, + " + _expressions.ReferenceIdSql + @" AS reference_id, + " + _expressions.ScopedResolutionStateSql + @" AS resolution_state, + " + _expressions.IdentityScopedSql + @" AS identity_scoped, + r.symbol_name, + " + _expressions.ContextSql + @" AS context, + r.container_name, + r.line, + r.column_number, + r.reference_kind AS raw_reference_kind, + " + GetLogicalReferenceKindSql("r.reference_kind") + @" AS logical_reference_kind + FROM symbol_references r + JOIN files src ON r.file_id = src.id" + _expressions.ReferenceLineJoin + @" + WHERE 1 = 1"); + var languagePredicate = _reader.BuildDependencyGraphLanguagePredicate(SourceAlias, "depsLang"); + _sql.Append(" AND " + languagePredicate.Sql); + _sql.AddParameters(languagePredicate.Parameters); + _sql.Append(_reader.BuildDependencyGeneratedFilter(SourceAlias)); + } + + private void AppendSourceScope() + { + if (_request.Lang != null) + _sql.Append(" AND src.lang = @lang"); + if (!_request.Reverse && _request.PathPatterns is { Count: > 0 }) + { + var predicates = new List(_request.PathPatterns.Count); + for (var i = 0; i < _request.PathPatterns.Count; i++) + predicates.Add(BuildPathFilterPredicate(SourceAlias, "pathPattern", i, _request.PathPatterns[i])); + _sql.Append(" AND (" + string.Join(" OR ", predicates) + ")"); + } + if (!_request.Reverse && _request.ExcludePathPatterns is { Count: > 0 }) + { + for (var i = 0; i < _request.ExcludePathPatterns.Count; i++) + _sql.Append($" AND NOT {BuildPathFilterPredicate(SourceAlias, "excludePath", i, _request.ExcludePathPatterns[i])}"); + } + if (_request.ExcludeTests) + _sql.Append($" AND NOT {DependencyTestPathCondition($"{SourceAlias}.path")}"); + } + + private void AppendLogicalReferences() + { + _sql.Append(@" + GROUP BY src.id, src.path, src.lang, " + _expressions.ReferenceIdSql + @", " + _expressions.ScopedResolutionStateSql + @", " + _expressions.IdentityScopedSql + @", r.symbol_name, " + _expressions.ContextSql + @", r.container_name, r.line, r.column_number, r.reference_kind, logical_reference_kind + ), + logical_references AS ( + SELECT source_file_id, source_path, source_lang, reference_id, identity_scoped, + " + BuildLogicalReferenceNameExpr("source_lang", "symbol_name", "context", "container_name", "column_number") + @" AS symbol_name, + " + BuildLogicalReferenceSegmentCountExpr("source_lang", "symbol_name", "context", "container_name", "column_number") + @" AS symbol_segment_count, + " + BuildLogicalReferenceLeafFallbackAllowedExpr("source_lang", "symbol_name", "context", "container_name", "column_number") + @" AS allow_leaf_fallback, + symbol_name AS raw_symbol_name, + context, line, column_number, raw_reference_kind, logical_reference_kind, + 0 AS is_attribute_alias, + CASE WHEN logical_reference_kind IN ('attribute', 'annotation') THEN 1 ELSE 0 END AS is_metadata + FROM logical_references_primary + UNION ALL + -- C# attribute suffix alias: [Foo] in source is stored with symbol_name='Foo', + -- but the defining class is named 'FooAttribute'. Emit the canonical 'Foo' + 'Attribute' + -- form so deps can match the class file as a target. The alias rows are flagged + -- so the edges CTE can restrict them to class-like targets and avoid spurious + -- edges to unrelated functions / properties that happen to be named 'FooAttribute'. + -- C# 属性のサフィックス別名: ソース上の [Foo] は symbol_name='Foo' で保存されるが、 + -- 定義クラスは 'FooAttribute' 命名になるため、正規形 'Foo' + 'Attribute' を補って + -- deps がクラス側のファイルを target として join できるようにする。alias 行には + -- フラグを付け、edges CTE 側で class-like target だけに限定する。これにより、 + -- 偶然 'FooAttribute' という名前を持つ関数やプロパティへの誤ったエッジを防ぐ。 + SELECT source_file_id, source_path, source_lang, reference_id, identity_scoped, + symbol_name || 'Attribute' AS symbol_name, + 1 AS symbol_segment_count, + 0 AS allow_leaf_fallback, + symbol_name || 'Attribute' AS raw_symbol_name, + context, line, column_number, raw_reference_kind, logical_reference_kind, + 1 AS is_attribute_alias, + 1 AS is_metadata + FROM logical_references_primary + WHERE source_lang = 'csharp' + AND logical_reference_kind = 'attribute' + AND symbol_name NOT LIKE '%Attribute' + ),"); + } + + private void AppendSourceNameCounts() + { + _sql.Append(@" + source_name_counts AS ( + -- Grouping includes is_metadata so metadata-only groups ([Foo] / @Foo) + -- can be restricted to class-like targets independently from non-metadata + -- call-graph groups that share the same symbol_name in the same file + -- (e.g. `Foo()` call + `[Foo]` attribute both present in the same source). + -- is_metadata を GROUP BY に含めることで、同じ source file / symbol_name を + -- 共有する metadata 行と call-graph 行 (例: 同じファイル内の `Foo()` 呼び出し + -- と `[Foo]` 属性) を別グループとして扱い、metadata 側だけに class-like + -- target 制限を掛けられるようにする。 + SELECT source_file_id, + source_path, + source_lang, + identity_scoped, + symbol_name, + symbol_segment_count, + allow_leaf_fallback, + raw_symbol_name, + context, + column_number, + raw_reference_kind, + logical_reference_kind, + is_attribute_alias, + is_metadata, + COUNT(*) AS ref_count + FROM logical_references + WHERE 1 = 1"); + _sql.Append(BuildDependencySymbolFilter( + "symbol_name", + _request.DependencySymbols, + _request.DependencySymbolFamilies, + _request.SuppressDependencyNoise, + "sourceDependency", + "source_lang = 'csharp'")); + _sql.Append(@" + GROUP BY source_file_id, source_path, source_lang, identity_scoped, symbol_name, symbol_segment_count, allow_leaf_fallback, raw_symbol_name, context, column_number, raw_reference_kind, logical_reference_kind, is_attribute_alias, is_metadata + )"); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencySpecialEdgeSql.cs b/src/CodeIndex/Database/DbReader.DependencySpecialEdgeSql.cs new file mode 100644 index 0000000000..f59ec7e684 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencySpecialEdgeSql.cs @@ -0,0 +1,137 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class DependencySpecialEdgeSqlBuilder + { + private readonly DependencyQueryRequest _request; + private readonly DependencySqlFragmentBuilder _sql = new(); + + internal DependencySpecialEdgeSqlBuilder(DependencyQueryRequest request) + { + _request = request; + } + + internal DependencySqlFragment Build() + { + AppendMarkdownEdges(); + AppendDockerEdges(); + AppendProjectPathEdges(); + return _sql.Build(); + } + + private void AppendMarkdownEdges() + { + _sql.Append(@" + UNION ALL + -- Resolve explicit Markdown links once per target file. Joining these + -- path references through target_files would multiply one link by every + -- heading or symbol declared in the destination document. + -- 明示的な Markdown link は target file ごとに一度だけ解決する。 + -- target_files 経由で結合すると、1 link が宛先 document 内の全見出し + -- / symbol の件数だけ増幅されるため、file-level path として扱う。 + SELECT snc.source_path, + ptf.target_path, + snc.raw_symbol_name, + snc.ref_count, + snc.source_lang, + 'markdown_explicit_link', + snc.raw_reference_kind, + 'file' + FROM bounded_source_name_counts snc + JOIN path_target_files ptf + ON ptf.target_path = markdown_resolve_path(snc.source_path, snc.symbol_name) + WHERE snc.source_lang = 'markdown' + AND snc.logical_reference_kind = 'import' + AND snc.source_path != ptf.target_path"); + _sql.Append(BuildDependencySymbolFilter( + "snc.raw_symbol_name", + _request.DependencySymbols, + _request.DependencySymbolFamilies, + suppressDependencyNoise: false, + parameterPrefix: "markdownPathDependency")); + } + + private void AppendDockerEdges() + { + _sql.Append(@" + UNION ALL + -- Dockerfile stages are symbols within one file, so their dependency edge is + -- intentionally a self-file edge. Keep this exception stage-specific rather + -- than weakening the cross-file contract for other symbols or languages. + -- Dockerfile stage は同一ファイル内の symbol であるため、この依存 edge は + -- 意図的に self-file edge とする。他の symbol / 言語の cross-file 契約を + -- 緩めないよう stage に限定する。 + SELECT snc.source_path, + snc.source_path, + snc.symbol_name, + snc.ref_count, + snc.source_lang, + 'docker_stage_reference', + snc.raw_reference_kind, + 'stage' + FROM source_name_counts snc + JOIN files self_dst ON self_dst.id = snc.source_file_id + WHERE snc.source_lang = 'dockerfile' + AND snc.raw_reference_kind = 'call' + AND EXISTS ( + SELECT 1 + FROM symbols stage + WHERE stage.file_id = snc.source_file_id + AND stage.kind = 'stage' + AND stage.name = snc.symbol_name COLLATE NOCASE + )"); + AppendReverseDockerScope(); + _sql.Append(BuildDependencySymbolFilter( + "snc.symbol_name", + _request.DependencySymbols, + _request.DependencySymbolFamilies, + _request.SuppressDependencyNoise, + "dockerDependency")); + } + + private void AppendReverseDockerScope() + { + if (_request.Reverse && _request.PathPatterns is { Count: > 0 }) + { + var predicates = new List(_request.PathPatterns.Count); + for (var i = 0; i < _request.PathPatterns.Count; i++) + predicates.Add(BuildPathFilterPredicate("self_dst", "pathPattern", i, _request.PathPatterns[i])); + _sql.Append(" AND (" + string.Join(" OR ", predicates) + ")"); + } + if (_request.Reverse && _request.ExcludePathPatterns is { Count: > 0 }) + { + for (var i = 0; i < _request.ExcludePathPatterns.Count; i++) + _sql.Append($" AND NOT {BuildPathFilterPredicate("self_dst", "excludePath", i, _request.ExcludePathPatterns[i])}"); + } + if (_request.Reverse && _request.ExcludeTests) + _sql.Append($" AND NOT {DependencyTestPathCondition("self_dst.path")}"); + } + + private void AppendProjectPathEdges() + { + _sql.Append(@" + UNION ALL + SELECT snc.source_path, + ptf.target_path, + snc.symbol_name, + snc.ref_count, + snc.source_lang, + 'explicit_path_reference', + snc.raw_reference_kind, + 'file' + FROM bounded_source_name_counts snc + JOIN path_target_files ptf + ON ptf.target_path = markdown_resolve_path(snc.source_path, snc.symbol_name) + WHERE snc.source_lang IN ('msbuild', 'solution') + AND snc.logical_reference_kind IN ('import', 'project_reference') + AND snc.source_path != ptf.target_path"); + _sql.Append(BuildDependencySymbolFilter( + "snc.symbol_name", + _request.DependencySymbols, + _request.DependencySymbolFamilies, + _request.SuppressDependencyNoise, + "pathDependency")); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.DependencyTargetSql.cs b/src/CodeIndex/Database/DbReader.DependencyTargetSql.cs new file mode 100644 index 0000000000..9498e0f955 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.DependencyTargetSql.cs @@ -0,0 +1,179 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class DependencyTargetSqlBuilder + { + private const string TargetAlias = "dst"; + + private readonly DbReader _reader; + private readonly DependencyQueryRequest _request; + private readonly DependencyQueryExpressions _expressions; + private readonly DependencySqlFragmentBuilder _sql = new(); + + internal DependencyTargetSqlBuilder( + DbReader reader, + DependencyQueryRequest request, + DependencyQueryExpressions expressions) + { + _reader = reader; + _request = request; + _expressions = expressions; + } + + internal DependencySqlFragment Build() + { + AppendTargetFiles(); + AppendTargetScope(); + AppendTargetMetadataQueries(); + return _sql.Build(); + } + + private void AppendTargetFiles() + { + _sql.Append(@" + target_files AS ( + -- Collapse per-symbol rows to one per (target_path, target_lang, symbol_name) + -- and remember whether any of the same-name symbols is a class-like kind + -- via MAX. Keeping kind in DISTINCT would split identical (path, lang, name) + -- rows when one file defines both a class and a same-name function (e.g. a + -- C# constructor), inflating the deps reference count. + -- (target_path, target_lang, symbol_name) 単位に集約し、同名のシンボルの + -- いずれかが class 系であるかを MAX で覚える。kind を DISTINCT に含めると、 + -- 同じ (path, lang, name) でも class と同名 function (C# のコンストラクタ等) + -- が別行として残り、deps の参照カウントが膨らんでしまう。 + -- has_metadata_target_kind further narrows the class-like set to targets + -- that can legitimately be referenced as [Attribute] metadata. For C# + -- we cannot resolve base types transitively at SQL time, so the best + -- portable approximation is an inheritance-clause check: any class + -- declared with a base list is a potential attribute type (direct or + -- indirect Attribute derivation). A plain class FooAttribute with no + -- base clause is not a valid [Foo] target at compile time. + -- Other languages keep the original class-like breadth. Legacy DBs + -- without a signature column degrade to the broad class-like set. + -- has_metadata_target_kind は [Attribute] metadata target として妥当な + -- class-like のみに絞る。C# は SQL 時点で基底型を遡れないため、継承節を + -- 持つクラスを候補とする近似を採る(直接・間接の Attribute 継承を + -- 取りこぼさない)。他言語は class-like 全体を残す。signature 列が無い + -- legacy DB では filter を無効化し class-like 全体に戻る。 + SELECT dst.path AS target_path, + dst.lang AS target_lang, + " + _expressions.TargetLogicalSymbolName + @" AS symbol_name, + " + _expressions.TargetLogicalSymbolSegmentCount + @" AS symbol_segment_count, + MAX(CASE WHEN s.kind IN ('class','struct','interface') THEN 1 ELSE 0 END) AS has_class_like_kind, + MAX(CASE WHEN s.kind = 'heading' THEN 1 ELSE 0 END) AS has_heading_kind, + MAX(CASE WHEN " + _reader.BuildMetadataTargetKindExpr(TargetAlias) + @" + THEN 1 ELSE 0 END) AS has_metadata_target_kind + FROM symbols s + JOIN files dst ON s.file_id = dst.id + WHERE 1 = 1"); + var languagePredicate = _reader.BuildDependencyGraphLanguagePredicate(TargetAlias, "depsTargetLang"); + _sql.Append(" AND " + languagePredicate.Sql); + _sql.AddParameters(languagePredicate.Parameters); + _sql.Append(_reader.BuildDependencyGeneratedFilter(TargetAlias)); + } + + private void AppendTargetScope() + { + if (_request.Lang != null && !_request.Lang.Equals("solution", StringComparison.Ordinal)) + _sql.Append(" AND dst.lang = @lang"); + if (_request.Reverse && _request.PathPatterns is { Count: > 0 }) + { + var predicates = new List(_request.PathPatterns.Count); + for (var i = 0; i < _request.PathPatterns.Count; i++) + predicates.Add(BuildPathFilterPredicate(TargetAlias, "pathPattern", i, _request.PathPatterns[i])); + _sql.Append(" AND (" + string.Join(" OR ", predicates) + ")"); + } + if (_request.Reverse && _request.ExcludePathPatterns is { Count: > 0 }) + { + for (var i = 0; i < _request.ExcludePathPatterns.Count; i++) + _sql.Append($" AND NOT {BuildPathFilterPredicate(TargetAlias, "excludePath", i, _request.ExcludePathPatterns[i])}"); + } + if (_request.ExcludeTests) + _sql.Append($" AND NOT {DependencyTestPathCondition($"{TargetAlias}.path")}"); + _sql.Append(BuildDependencySymbolFilter( + _expressions.TargetLogicalSymbolName, + _request.DependencySymbols, + _request.DependencySymbolFamilies, + _request.SuppressDependencyNoise, + "targetDependency", + "dst.lang = 'csharp'")); + } + + private void AppendTargetMetadataQueries() + { + _sql.Append(@" + GROUP BY dst.path, dst.lang, " + _expressions.TargetLogicalSymbolName + @", " + _expressions.TargetLogicalSymbolSegmentCount + @" + ), + path_target_files AS ( + SELECT DISTINCT target_path, target_lang + FROM target_files + ), + metadata_raw_suppression AS ( + -- When a raw C# attribute reference '[Foo]' (stored as symbol_name='Foo', + -- logical_reference_kind='attribute') also has a synthetic suffix alias + -- row that resolves to a class-like 'FooAttribute' target, drop the raw + -- row to avoid creating a duplicate edge to any unrelated 'Foo' symbol + -- (method, property, local class) that merely shares the bare name. + -- 生の C# 属性参照 '[Foo]' (symbol_name='Foo', kind='attribute') に対して + -- 同じ source_file 内で 'FooAttribute' の synthetic alias 行が + -- class 系 target に解決できる場合、この行自体は落として + -- 同名の関数/プロパティ/ローカルクラス 'Foo' への誤依存を防ぐ。 + SELECT DISTINCT lrp.source_file_id, lrp.symbol_name + FROM logical_references_primary lrp + JOIN target_files tf_alias + ON tf_alias.target_lang = lrp.source_lang + AND tf_alias.symbol_name = lrp.symbol_name || 'Attribute' + AND tf_alias.symbol_segment_count = 1 + AND tf_alias.has_metadata_target_kind = 1 + WHERE lrp.source_lang = 'csharp' + AND lrp.logical_reference_kind = 'attribute' + AND lrp.symbol_name NOT LIKE '%Attribute' + ), + target_ambiguity AS ( + -- Count class-like definitions at symbol-identity level rather than + -- file level. Two same-named class-like definitions in the same file + -- (e.g. `namespace A { class FooAttribute { } } namespace B { class + -- FooAttribute { } }` both inside one .cs file) collapse to a single + -- target_files row because target_files is GROUPed by dst.path, so + -- COUNT(DISTINCT target_path) alone would see count=1 and falsely + -- treat the metadata target as unambiguous. Joining target_files back + -- through files + symbols recovers the per-definition row count while + -- still inheriting target_files' lang / path / graph-supported scope + -- (since the join only keeps rows whose (path, lang, name) already + -- appear in target_files). + -- class-like 定義は path 単位ではなく symbol identity 単位で数える。 + -- 同じ .cs ファイル内に別名前空間で同名 class-like が 2 つあるケースは + -- target_files (dst.path で GROUP BY) 上では 1 行に潰れており、 + -- COUNT(DISTINCT target_path) だけでは count=1 となり metadata target + -- が一意と誤判定される。target_files から files + symbols に JOIN し直す + -- ことで定義単位の件数を復元する。JOIN が target_files 既存行にしか + -- 当たらないため、lang / path / graph-supported スコープはそのまま継承。 + SELECT tf.target_lang, + tf.symbol_name, + tf.symbol_segment_count, + COUNT(*) AS class_like_target_count + FROM target_files tf + JOIN files dst + ON dst.path = tf.target_path + AND dst.lang = tf.target_lang + JOIN symbols s + ON s.file_id = dst.id + AND " + _expressions.TargetLogicalSymbolName + @" = tf.symbol_name + AND " + _expressions.TargetLogicalSymbolSegmentCount + @" = tf.symbol_segment_count + -- Same language-aware metadata-eligibility filter as + -- target_files: C# restricts to `class` with inheritance + -- clause (interface/struct cannot be attribute targets); + -- JS/TS additionally accepts `function` (decorator + -- factory); others keep the class-like candidate set. + -- target_files と同じ言語別 metadata 適格性フィルタ。 + -- C# は class 限定 + 継承節 (interface/struct は除外)。 + -- JS/TS は decorator factory 用に function も許容。 + -- それ以外は class-like 全体を候補にする。 + AND " + _reader.BuildMetadataTargetKindExpr(TargetAlias) + @" + WHERE tf.has_metadata_target_kind = 1 + GROUP BY tf.target_lang, tf.symbol_name, tf.symbol_segment_count + ),"); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.FilesStatus.cs b/src/CodeIndex/Database/DbReader.FilesStatus.cs index 22afb7ab80..978e5cb963 100644 --- a/src/CodeIndex/Database/DbReader.FilesStatus.cs +++ b/src/CodeIndex/Database/DbReader.FilesStatus.cs @@ -34,8 +34,6 @@ public partial class DbReader internal const string BoundedResourceReadChunkIndexName = "idx_chunks_file_start_chunk_nonnull"; internal const string LegacyBoundedResourceReadFileIndexName = "idx_chunks_file"; internal const int MaxLegacyResourceReadSqliteVmSteps = 250_000; - private const int LegacyResourceReadProgressOperations = 100; - private const int BoundedFileReadBufferSize = 4 * 1024; private const string EmptyIndexedContentChecksum = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"; @@ -75,36 +73,6 @@ AND c.start_line > @startLine LIMIT @chunkLimit """; - private static readonly string LegacyBoundedResourceReadPredecessorSql = $""" - SELECT c.id, c.start_line, c.end_line, c.chunk_index - FROM chunks c INDEXED BY {LegacyBoundedResourceReadFileIndexName} - WHERE c.file_id = @fileId - AND c.content IS NOT NULL - AND c.start_line <= @startLine - ORDER BY c.start_line DESC, c.chunk_index DESC - LIMIT @chunkLimit - """; - - private static readonly string LegacyBoundedResourceReadEndSql = $""" - SELECT c.id, c.start_line, c.end_line, c.chunk_index - FROM chunks c INDEXED BY {LegacyBoundedResourceReadFileIndexName} - WHERE c.file_id = @fileId - AND c.content IS NOT NULL - AND c.end_line >= @startLine - ORDER BY c.end_line, c.start_line, c.chunk_index - LIMIT @chunkLimit - """; - - private static readonly string LegacyBoundedResourceReadForwardSql = $""" - SELECT c.id, c.start_line, c.end_line, c.chunk_index - FROM chunks c INDEXED BY {LegacyBoundedResourceReadFileIndexName} - WHERE c.file_id = @fileId - AND c.content IS NOT NULL - AND c.start_line > @startLine - AND c.start_line <= @endLine - ORDER BY c.start_line, c.chunk_index - LIMIT @chunkLimit - """; internal static TimeSpan? FindRegexMatchTimeoutForTesting { @@ -130,302 +98,44 @@ internal static int? LegacyResourceReadSqliteVmStepLimitForTesting set => LegacyResourceReadSqliteVmStepLimitOverride.Value = value; } + private static Regex CreateFindRegexMatcher(string query, bool exact) + => RegexRegistry.CreateFindRegex(query, exact, ResolveFindRegexMatchTimeout()); + + private static TimeSpan ResolveFindRegexMatchTimeout() + => FindRegexMatchTimeoutForTesting is { } timeout && timeout > TimeSpan.Zero + ? timeout + : BoundedRegex.DefaultMatchTimeout; + public FindResults FindInFiles(string query, int limit, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, int before = 0, int after = 0, bool exact = false, int maxLineWidth = LineWidthFormatter.DefaultMaxLineWidth, int? focusLine = null, int? focusColumn = null, bool regex = false, int? maxCandidateFiles = null, int? maxLinesScanned = null, int offset = 0, bool useIndexedLiteralCandidates = false, string? resumePath = null, int? resumeLine = null, int? resumeFileOrdinal = null, int? resumeMatchOrdinal = null, int? resumeByteOffset = null, bool captureContinuation = false, CancellationToken cancellationToken = default) { - cancellationToken.ThrowIfCancellationRequested(); - if (string.IsNullOrWhiteSpace(query) || limit <= 0) - return new FindResults([], new FindScanSummary(0, 0, 0)); - ValidateFindContinuation( - resumePath, - resumeLine, - resumeFileOrdinal, - resumeMatchOrdinal, - resumeByteOffset); - - before = Math.Max(0, before); - after = Math.Max(0, after); - maxLineWidth = LineWidthFormatter.ClampMaxLineWidth(maxLineWidth); - var comparison = exact ? StringComparison.Ordinal : StringComparison.OrdinalIgnoreCase; - var regexMatcher = regex - ? CreateFindRegexMatcher(query, exact) - : null; - - var searchPlan = CreateFindSearchPlan(query, exact, regex, useIndexedLiteralCandidates); - using var fileCmd = CreateFindFileCommand( - searchPlan, + var scanRequest = new IndexedFindScanRequest( + query, lang, pathPatterns, excludePathPatterns, - excludeTests); - - var candidateFiles = CountFindCandidateFiles(lang, pathPatterns, excludePathPatterns, excludeTests); - var filesScanned = 0; - var linesScanned = 0; - var truncated = false; - string? truncationReason = null; - string? nextPath = null; - int? nextLine = null; - int? nextFileOrdinal = null; - int? nextMatchOrdinal = null; - int? nextByteOffset = null; - var resultLimitReached = false; - var matchesSkipped = 0; - offset = Math.Max(0, offset); - var resumePending = resumePath is not null; - var resumeMatchPending = resumeMatchOrdinal.HasValue; - var candidateFileOrdinal = -1; - var results = new List(); - using var fileReader = fileCmd.ExecuteTrackedReader(); - while (fileReader.TrackedRead()) - { - cancellationToken.ThrowIfCancellationRequested(); - if (!captureContinuation && results.Count >= limit) - break; - candidateFileOrdinal++; - var fileId = fileReader.GetInt64(0); - var path = fileReader.GetString(1); - var fileLang = GetNullableString(fileReader, 2); - var totalLines = fileReader.GetInt32(3); - if (resumePending) - { - if (resumeFileOrdinal.HasValue) - { - if (candidateFileOrdinal < resumeFileOrdinal.Value) - continue; - if (candidateFileOrdinal != resumeFileOrdinal.Value - || !string.Equals(path, resumePath, StringComparison.Ordinal)) - { - throw new FindContinuationException( - "cursor_malformed", - "find cursor file position does not match the current candidate order."); - } - } - else if (!string.Equals(path, resumePath, StringComparison.Ordinal)) - { - continue; - } - if (resumeLine > Math.Max(1, totalLines)) - { - throw new FindContinuationException( - "cursor_malformed", - "find cursor line position exceeds the selected file."); - } - resumePending = false; - } - if (maxCandidateFiles.HasValue && filesScanned >= maxCandidateFiles.Value) - { - truncated = true; - truncationReason ??= "candidate_file_limit"; - nextPath = path; - nextLine = 1; - nextFileOrdinal = candidateFileOrdinal; - nextByteOffset = 0; - break; - } - - var firstEligibleLine = string.Equals(path, resumePath, StringComparison.Ordinal) - ? Math.Max(1, resumeLine ?? 1) - : 1; - var firstContextLine = Math.Max(1, firstEligibleLine - before); - filesScanned++; - if (totalLines <= 0) - continue; - - var searchQuery = exact && !regex ? ExactSourceSearchNormalizer.Normalize(query, fileLang) : query; - var pendingMatches = new Queue(); - var snippetWindow = new Queue(); - var snippetLinesByNumber = new Dictionary(); - var acceptedMatches = results.Count; - var stopScanning = false; - - foreach (var indexedLine in EnumerateIndexedFileLines(fileId)) - { - cancellationToken.ThrowIfCancellationRequested(); - if (indexedLine.Number > totalLines) - break; - if (indexedLine.Number < firstContextLine) - continue; - var eligibleForMatch = indexedLine.Number >= firstEligibleLine; - if (maxLinesScanned.HasValue && linesScanned >= maxLinesScanned.Value) - { - truncated = true; - truncationReason ??= "line_scan_limit"; - if (nextPath is null) - { - nextPath = path; - nextLine = indexedLine.Number; - nextFileOrdinal = candidateFileOrdinal; - nextByteOffset = 0; - } - stopScanning = true; - break; - } - if (eligibleForMatch) - { - linesScanned++; - FindLineScannedForTesting?.Invoke(); - cancellationToken.ThrowIfCancellationRequested(); - } - - AddLineToFindWindow(indexedLine, snippetWindow, snippetLinesByNumber); - - if (!resultLimitReached - && (matchesSkipped < offset - || captureContinuation && acceptedMatches <= limit - || !captureContinuation && acceptedMatches < limit) - && eligibleForMatch - && (!focusLine.HasValue || indexedLine.Number == focusLine.Value)) - { - var matchOrdinal = 0; - foreach (var lineMatch in EnumerateFindLineMatches( - indexedLine.Text, - fileLang, - searchQuery, - comparison, - regexMatcher, - exact && !regex, - focusColumn)) - { - if (resumeMatchPending - && indexedLine.Number == firstEligibleLine) - { - if (matchOrdinal < resumeMatchOrdinal!.Value) - { - matchOrdinal++; - continue; - } - var resumeBoundaryByteOffset = Encoding.UTF8.GetByteCount( - indexedLine.Text.AsSpan(0, lineMatch.Column)); - if (matchOrdinal != resumeMatchOrdinal.Value - || resumeBoundaryByteOffset != resumeByteOffset) - { - throw new FindContinuationException( - "cursor_malformed", - "find cursor match position is not a record boundary for the current query."); - } - resumeMatchPending = false; - } - - if (matchesSkipped < offset) - { - matchesSkipped++; - matchOrdinal++; - continue; - } - if (acceptedMatches >= limit) - { - var byteOffset = Encoding.UTF8.GetByteCount( - indexedLine.Text.AsSpan(0, lineMatch.Column)); - nextPath = path; - nextLine = indexedLine.Number; - nextFileOrdinal = candidateFileOrdinal; - nextMatchOrdinal = matchOrdinal; - nextByteOffset = byteOffset; - resultLimitReached = true; - stopScanning = true; - break; - } - - pendingMatches.Enqueue(new PendingFileFindMatch( - indexedLine.Number, - lineMatch.Column, - lineMatch.Length, - Math.Max(1, indexedLine.Number - before), - Math.Min(totalLines, indexedLine.Number + after))); - acceptedMatches++; - matchOrdinal++; - } - } - - FlushReadyFindMatches( - path, - fileLang, - pendingMatches, - snippetLinesByNumber, - results, - maxLineWidth, - indexedLine.Number); - PruneFindWindow(indexedLine.Number, before, pendingMatches, snippetWindow, snippetLinesByNumber); - - if ((stopScanning || !captureContinuation && results.Count >= limit) - && pendingMatches.Count == 0) - break; - } - - FlushReadyFindMatches( - path, - fileLang, - pendingMatches, - snippetLinesByNumber, - results, - maxLineWidth, - int.MaxValue); - if (stopScanning || !captureContinuation && results.Count >= limit) - break; - } - if (resumePending || resumeMatchPending) - { - throw new FindContinuationException( - "cursor_malformed", - "find cursor position does not exist in the current result sequence."); - } - - var capReached = truncated; - return new FindResults( - results, - new FindScanSummary( - candidateFiles, - filesScanned, - linesScanned, - truncated, - capReached, - TimedOut: false, - truncationReason, - maxCandidateFiles, - maxLinesScanned, - searchPlan.Strategy, - searchPlan.FallbackReason, - nextPath, - nextLine, - nextFileOrdinal, - nextMatchOrdinal, - nextByteOffset, - resultLimitReached)); - } - - private static void ValidateFindContinuation( - string? resumePath, - int? resumeLine, - int? resumeFileOrdinal, - int? resumeMatchOrdinal, - int? resumeByteOffset) - { - if (resumePath is null) - { - if (resumeLine.HasValue - || resumeFileOrdinal.HasValue - || resumeMatchOrdinal.HasValue - || resumeByteOffset.HasValue) - { - throw new FindContinuationException( - "cursor_malformed", - "find cursor continuation fields require a resume path."); - } - return; - } - - if (!resumeLine.HasValue - || resumeLine.Value <= 0 - || resumeFileOrdinal is < 0 - || resumeMatchOrdinal is < 0 - || resumeByteOffset is < 0 - || resumeMatchOrdinal.HasValue && !resumeByteOffset.HasValue - || !resumeMatchOrdinal.HasValue && resumeByteOffset is not (null or 0)) - { - throw new FindContinuationException( - "cursor_malformed", - "find cursor continuation position is invalid."); - } + excludeTests, + exact, + focusLine, + focusColumn, + regex, + maxCandidateFiles, + maxLinesScanned, + useIndexedLiteralCandidates, + new FindResumePosition( + resumePath, + resumeLine, + resumeFileOrdinal, + resumeMatchOrdinal, + resumeByteOffset), + cancellationToken); + return new IndexedFindPipeline(this).Find(new IndexedFindListRequest( + scanRequest, + limit, + before, + after, + maxLineWidth, + offset, + captureContinuation)); } public int CountFindCandidateFiles(string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false) @@ -445,452 +155,26 @@ public int CountFindCandidateFiles(string? lang = null, IReadOnlyList? p public FindCountResult CountFindInFiles(string query, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, bool exact = false, int? focusLine = null, int? focusColumn = null, bool regex = false, int? maxCandidateFiles = null, int? maxLinesScanned = null, bool useIndexedLiteralCandidates = false, string? resumePath = null, int? resumeLine = null, int? resumeFileOrdinal = null, int? resumeMatchOrdinal = null, int? resumeByteOffset = null, CancellationToken cancellationToken = default) { - cancellationToken.ThrowIfCancellationRequested(); - if (string.IsNullOrWhiteSpace(query)) - return new FindCountResult(0, 0, new FindScanSummary(0, 0, 0)); - ValidateFindContinuation( - resumePath, - resumeLine, - resumeFileOrdinal, - resumeMatchOrdinal, - resumeByteOffset); - if (resumeMatchOrdinal.HasValue || resumeByteOffset is not (null or 0)) - { - throw new FindContinuationException( - "cursor_malformed", - "find count cursor must resume at a line boundary."); - } - - var comparison = exact ? StringComparison.Ordinal : StringComparison.OrdinalIgnoreCase; - var regexMatcher = regex - ? CreateFindRegexMatcher(query, exact) - : null; - var searchPlan = CreateFindSearchPlan(query, exact, regex, useIndexedLiteralCandidates); - using var fileCmd = CreateFindFileCommand( - searchPlan, + return new IndexedFindPipeline(this).Count(new IndexedFindScanRequest( + query, lang, pathPatterns, excludePathPatterns, - excludeTests); - - var candidateFiles = CountFindCandidateFiles(lang, pathPatterns, excludePathPatterns, excludeTests); - var filesScanned = 0; - var linesScanned = 0; - var truncated = false; - string? truncationReason = null; - string? nextPath = null; - int? nextLine = null; - int? nextFileOrdinal = null; - var count = 0; - var fileCount = 0; - var resumePending = resumePath is not null; - var candidateFileOrdinal = -1; - using var fileReader = fileCmd.ExecuteTrackedReader(); - while (fileReader.TrackedRead()) - { - cancellationToken.ThrowIfCancellationRequested(); - candidateFileOrdinal++; - var path = fileReader.GetString(1); - if (resumePending) - { - if (resumeFileOrdinal.HasValue) - { - if (candidateFileOrdinal < resumeFileOrdinal.Value) - continue; - if (candidateFileOrdinal != resumeFileOrdinal.Value - || !string.Equals(path, resumePath, StringComparison.Ordinal)) - { - throw new FindContinuationException( - "cursor_malformed", - "find count cursor file position does not match the current candidate order."); - } - } - else if (!string.Equals(path, resumePath, StringComparison.Ordinal)) - { - continue; - } - if (resumeLine > Math.Max(1, fileReader.GetInt32(3))) - { - throw new FindContinuationException( - "cursor_malformed", - "find count cursor line position exceeds the selected file."); - } - resumePending = false; - } - if (maxCandidateFiles.HasValue && filesScanned >= maxCandidateFiles.Value) - { - truncated = true; - truncationReason ??= "candidate_file_limit"; - nextPath = path; - nextLine = 1; - nextFileOrdinal = candidateFileOrdinal; - break; - } - - var fileId = fileReader.GetInt64(0); - var fileLang = GetNullableString(fileReader, 2); - var totalLines = fileReader.GetInt32(3); - var firstEligibleLine = string.Equals(path, resumePath, StringComparison.Ordinal) - ? Math.Max(1, resumeLine ?? 1) - : 1; - filesScanned++; - if (totalLines <= 0) - continue; - - var searchQuery = exact && !regex ? ExactSourceSearchNormalizer.Normalize(query, fileLang) : query; - var fileMatches = 0; - var stopScanning = false; - foreach (var indexedLine in EnumerateIndexedFileLines(fileId)) - { - cancellationToken.ThrowIfCancellationRequested(); - if (indexedLine.Number > totalLines) - break; - if (indexedLine.Number < firstEligibleLine) - continue; - if (maxLinesScanned.HasValue && linesScanned >= maxLinesScanned.Value) - { - truncated = true; - truncationReason ??= "line_scan_limit"; - nextPath = path; - nextLine = indexedLine.Number; - nextFileOrdinal = candidateFileOrdinal; - stopScanning = true; - break; - } - linesScanned++; - - if (focusLine.HasValue && indexedLine.Number != focusLine.Value) - continue; - - foreach (var _ in EnumerateFindLineMatches( - indexedLine.Text, - fileLang, - searchQuery, - comparison, - regexMatcher, - exact && !regex, - focusColumn)) - { - fileMatches++; - } - } - - if (fileMatches > 0) - { - count += fileMatches; - fileCount++; - } - if (stopScanning) - break; - } - if (resumePending) - { - throw new FindContinuationException( - "cursor_malformed", - "find count cursor position does not exist in the current candidate sequence."); - } - - var capReached = truncated; - return new FindCountResult( - count, - fileCount, - new FindScanSummary( - candidateFiles, - filesScanned, - linesScanned, - truncated, - capReached, - TimedOut: false, - truncationReason, - maxCandidateFiles, - maxLinesScanned, - searchPlan.Strategy, - searchPlan.FallbackReason, - nextPath, - nextLine, - nextFileOrdinal, - NextByteOffset: nextPath is null ? null : 0)); - } - - private readonly record struct FindSearchPlan( - string Strategy, - string? FallbackReason, - string? TrigramMatchExpression); - - private readonly record struct IndexedLine(int Number, string Text); - - private readonly record struct FindLineMatch(int Column, int Length); - - private readonly record struct PendingFileFindMatch(int LineNumber, int Column, int Length, int SnippetStart, int SnippetEnd); - - private FindSearchPlan CreateFindSearchPlan( - string query, - bool exact, - bool regex, - bool useIndexedLiteralCandidates) - { - if (!useIndexedLiteralCandidates) - return new FindSearchPlan("line_scan", null, null); - if (regex) - return new FindSearchPlan("line_scan", "regex", null); - if (exact) - return new FindSearchPlan("line_scan", "exact_source_normalization", null); - if (query.Length < 3) - return new FindSearchPlan("line_scan", "query_too_short", null); - if (query.Any(character => character < ' ' || character > '~')) - return new FindSearchPlan("line_scan", "unsupported_query_characters", null); - if (!HasTable(DbContext.FtsChunksTrigramTableName)) - return new FindSearchPlan("line_scan", "trigram_index_unavailable", null); - if (DbWriter.IsFtsBulkLoadMarkerSet(GetMetaString(DbWriter.FtsBulkLoadInProgressMetaKey))) - return new FindSearchPlan("line_scan", "trigram_index_rebuilding", null); - if (!HasAllTrigramFtsSyncTriggers()) - return new FindSearchPlan("line_scan", "trigram_index_unsynchronized", null); - - var phrase = "\"" + query.Replace("\"", "\"\"", StringComparison.Ordinal) + "\""; - return new FindSearchPlan("indexed_trigram", null, phrase); - } - - private bool HasAllTrigramFtsSyncTriggers() - { - using var command = _conn.CreateCommand(); - command.CommandText = DbContext.CountFtsChunksTrigramSyncTriggersSql; - return SqliteCommandPolicy.ReadInt32Scalar( - command, - "find trigram FTS synchronization trigger count") == 3; - } - - private SqliteCommand CreateFindFileCommand( - FindSearchPlan searchPlan, - string? lang, - IReadOnlyList? pathPatterns, - IReadOnlyList? excludePathPatterns, - bool excludeTests) - { - var fileCmd = _conn.CreateCommand(); - string sql; - if (searchPlan.TrigramMatchExpression != null) - { - sql = $""" - SELECT f.id, f.path, f.lang, f.lines - FROM ( - SELECT DISTINCT find_chunk.file_id - FROM {DbContext.FtsChunksTrigramTableName} - JOIN chunks find_chunk ON find_chunk.id = {DbContext.FtsChunksTrigramTableName}.rowid - WHERE {DbContext.FtsChunksTrigramTableName} MATCH @trigramQuery - ) find_candidate - JOIN files f ON f.id = find_candidate.file_id - WHERE 1=1 - """; - } - else - { - sql = "SELECT f.id, f.path, f.lang, f.lines FROM files f WHERE 1=1"; - } - if (lang != null) - sql += " AND f.lang = @lang"; - AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); - sql += $" ORDER BY {PathBucketOrder}, f.path"; - fileCmd.CommandText = sql; - if (searchPlan.TrigramMatchExpression != null) - SqliteCommandPolicy.AddText(fileCmd, "@trigramQuery", searchPlan.TrigramMatchExpression); - if (lang != null) - SqliteCommandPolicy.Add(fileCmd, "@lang", lang); - AddPathFilterParameters(fileCmd, pathPatterns, excludePathPatterns); - return fileCmd; - } - - private IEnumerable EnumerateIndexedFileLines(long fileId) - { - using var chunkCmd = _conn.CreateCommand(); - chunkCmd.CommandText = @" - SELECT c.start_line, c.end_line, c.content - FROM chunks c - WHERE c.file_id = @fileId - ORDER BY c.start_line, c.chunk_index"; - SqliteCommandPolicy.Add(chunkCmd, "@fileId", fileId); - - var lastEmittedLine = 0; - using var chunkReader = chunkCmd.ExecuteTrackedReader(); - while (chunkReader.TrackedRead()) - { - var chunkStartLine = chunkReader.GetInt32(0); - var chunkEndLine = chunkReader.GetInt32(1); - var chunkLines = chunkReader.GetString(2).Split('\n'); - var lineCount = chunkEndLine - chunkStartLine + 1; - - for (int i = 0; i < chunkLines.Length && i < lineCount; i++) - { - var absoluteLine = chunkStartLine + i; - if (absoluteLine <= lastEmittedLine) - continue; - - lastEmittedLine = absoluteLine; - yield return new IndexedLine(absoluteLine, chunkLines[i]); - } - } - } - - private static IEnumerable EnumerateFindLineMatches( - string lineText, - string? fileLang, - string searchQuery, - StringComparison comparison, - Regex? regexMatcher, - bool normalizeExactSource, - int? focusColumn) - { - int[]? rawIndexMap = null; - var searchLine = normalizeExactSource - ? ExactSourceSearchNormalizer.Normalize(lineText, fileLang, out rawIndexMap) - : lineText; - - if (regexMatcher != null) - { - foreach (Match match in regexMatcher.Matches(searchLine)) - { - if (!match.Success) - continue; - if (TryCreateFindLineMatch(match.Index, match.Length, rawIndexMap, focusColumn, out var lineMatch)) - yield return lineMatch; - } - - yield break; - } - - for (int searchStart = 0; searchStart < searchLine.Length;) - { - var matchColumn = searchLine.IndexOf(searchQuery, searchStart, comparison); - if (matchColumn < 0) - break; - - if (TryCreateFindLineMatch(matchColumn, searchQuery.Length, rawIndexMap, focusColumn, out var lineMatch)) - yield return lineMatch; - searchStart = matchColumn + 1; - } - } - - private static bool TryCreateFindLineMatch(int matchColumn, int matchLength, int[]? rawIndexMap, int? focusColumn, out FindLineMatch lineMatch) - { - var rawMatchColumn = rawIndexMap == null ? matchColumn : rawIndexMap[matchColumn]; - var rawMatchLength = matchLength; - if (rawIndexMap != null && matchLength > 0) - { - var rawMatchEndIndex = rawIndexMap[matchColumn + matchLength - 1]; - rawMatchLength = rawMatchEndIndex - rawMatchColumn + 1; - } - - lineMatch = new FindLineMatch(rawMatchColumn, rawMatchLength); - var focusEndColumn = rawMatchColumn + Math.Max(1, rawMatchLength); - return !focusColumn.HasValue || (focusColumn.Value >= rawMatchColumn + 1 && focusColumn.Value <= focusEndColumn); - } - - private static Regex CreateFindRegexMatcher(string query, bool exact) - => RegexRegistry.CreateFindRegex(query, exact, ResolveFindRegexMatchTimeout()); - - private static TimeSpan ResolveFindRegexMatchTimeout() - => FindRegexMatchTimeoutForTesting is { } timeout && timeout > TimeSpan.Zero - ? timeout - : BoundedRegex.DefaultMatchTimeout; - - private static void AddLineToFindWindow(IndexedLine indexedLine, Queue snippetWindow, Dictionary snippetLinesByNumber) - { - snippetWindow.Enqueue(indexedLine); - snippetLinesByNumber[indexedLine.Number] = indexedLine.Text; - } - - private static void FlushReadyFindMatches( - string path, - string? fileLang, - Queue pendingMatches, - Dictionary snippetLinesByNumber, - List results, - int maxLineWidth, - int availableThroughLine) - { - while (pendingMatches.Count > 0 && pendingMatches.Peek().SnippetEnd <= availableThroughLine) - { - var pending = pendingMatches.Dequeue(); - var snippetLineNumbers = Enumerable.Range(pending.SnippetStart, pending.SnippetEnd - pending.SnippetStart + 1) - .Where(snippetLinesByNumber.ContainsKey) - .ToList(); - if (snippetLineNumbers.Count == 0) - continue; - - var snippetLines = snippetLineNumbers.Select(line => snippetLinesByNumber[line]).ToList(); - var (snippet, truncationContext) = ClampFindSnippetLines( - snippetLines, - maxLineWidth, - focusLineIndex: snippetLineNumbers.IndexOf(pending.LineNumber), - focusColumn: pending.Column + 1, - focusLength: pending.Length); - var matchLine = snippetLinesByNumber[pending.LineNumber]; - - results.Add(new FileFindResult - { - Path = path, - Lang = fileLang, - Line = pending.LineNumber, - Column = pending.Column + 1, - Length = pending.Length, - OriginalLineLength = matchLine.Length, - StartLine = snippetLineNumbers[0], - EndLine = snippetLineNumbers[^1], - Snippet = snippet, - SnippetTruncated = truncationContext.LineCount > 0, - SnippetTruncationContext = truncationContext, - }); - } - } - - private static (string Text, FileFindSnippetTruncationContext Context) ClampFindSnippetLines( - IReadOnlyList lines, - int maxLineWidth, - int focusLineIndex, - int focusColumn, - int focusLength) - { - if (lines.Count == 0) - return (string.Empty, new FileFindSnippetTruncationContext()); - - var output = new string[lines.Count]; - var truncatedCharCounts = new List(); - for (var i = 0; i < lines.Count; i++) - { - var clamped = i == focusLineIndex - ? LineWidthFormatter.ClampLine(lines[i], maxLineWidth, focusColumn, focusLength) - : LineWidthFormatter.ClampLine(lines[i], maxLineWidth); - output[i] = clamped.Text; - if (clamped.Truncated) - truncatedCharCounts.Add(clamped.TruncatedCharCount); - } - - return ( - string.Join('\n', output), - new FileFindSnippetTruncationContext - { - LineCount = truncatedCharCounts.Count, - CharCounts = truncatedCharCounts, - TotalChars = truncatedCharCounts.Sum(), - Reason = truncatedCharCounts.Count > 0 ? "line_width" : null, - }); - } - - private static void PruneFindWindow( - int currentLine, - int before, - Queue pendingMatches, - Queue snippetWindow, - Dictionary snippetLinesByNumber) - { - var minLineToKeep = currentLine - before; - if (pendingMatches.Count > 0) - minLineToKeep = Math.Min(minLineToKeep, pendingMatches.Peek().SnippetStart); - - while (snippetWindow.Count > 0 && snippetWindow.Peek().Number < minLineToKeep) - { - var removed = snippetWindow.Dequeue(); - snippetLinesByNumber.Remove(removed.Number); - } + excludeTests, + exact, + focusLine, + focusColumn, + regex, + maxCandidateFiles, + maxLinesScanned, + useIndexedLiteralCandidates, + new FindResumePosition( + resumePath, + resumeLine, + resumeFileOrdinal, + resumeMatchOrdinal, + resumeByteOffset), + cancellationToken)); } /// @@ -1210,692 +494,23 @@ internal BoundedFileReadResult GetBoundedFileContent( int continuationByteOffset = 0) { ArgumentNullException.ThrowIfNull(file); - var path = file.Path; - if (string.IsNullOrWhiteSpace(path)) - return new BoundedFileReadResult { Status = BoundedFileReadStatus.FileNotFound }; - if (maxUtf8Bytes is <= 0 or > MaxBoundedFileReadUtf8Bytes) - throw new ArgumentOutOfRangeException(nameof(maxUtf8Bytes), maxUtf8Bytes, - $"UTF-8 byte budget must be between 1 and {MaxBoundedFileReadUtf8Bytes}."); - if (maxLines is <= 0 or > MaxBoundedFileReadLines) - throw new ArgumentOutOfRangeException(nameof(maxLines), maxLines, - $"Line budget must be between 1 and {MaxBoundedFileReadLines}."); - if (continuationByteOffset < 0) - throw new ArgumentOutOfRangeException(nameof(continuationByteOffset), continuationByteOffset, - "Continuation byte offset must be non-negative."); - if (!continuationLine.HasValue && continuationByteOffset != 0) - return new BoundedFileReadResult { Status = BoundedFileReadStatus.InvalidContinuation, Path = path }; - - startLine = Math.Max(1, startLine); - endLine = Math.Max(startLine, endLine); - var nextLine = continuationLine ?? startLine; - if (nextLine < startLine || nextLine > endLine) - return new BoundedFileReadResult { Status = BoundedFileReadStatus.InvalidContinuation, Path = path }; - - var fileId = file.Id; - var lang = file.Lang; - var totalLines = file.Lines; - var fileSize = file.Size; - - BoundedFileReadResult CreateResult(BoundedFileReadStatus status, string? failureReason = null) => new() - { - Status = status, - FailureReason = failureReason, - Path = path, - Lang = lang, - TotalLines = totalLines, - RequestedStartLine = startLine, - RequestedEndLine = endLine, - StartLine = nextLine, - EndLine = nextLine, - }; - - if (fileSize < 0 || totalLines < 0 || (fileSize == 0 && totalLines > 0)) - return CreateResult(BoundedFileReadStatus.InvalidTopology, "resource_file_metadata_inconsistent"); - if (totalLines == 0) - { - if (_hasChunksTable && HasAnyResourceChunk(fileId)) - return CreateResult(BoundedFileReadStatus.InvalidTopology, "resource_file_metadata_inconsistent"); - if (IsAffirmativelyEmptyIndexedFile(totalLines, file.Checksum)) - return CreateResult(BoundedFileReadStatus.Empty); - if (fileSize == 0 && file.Checksum != null) - return CreateResult(BoundedFileReadStatus.InvalidTopology, "resource_file_metadata_inconsistent"); - return CreateResult(BoundedFileReadStatus.ContentUnavailable, "resource_content_unavailable"); - } - if (!_hasChunksTable) - return CreateResult(BoundedFileReadStatus.ContentUnavailable, "resource_content_unavailable"); - var hasBoundedChunkIndexes = _chunkIndexes.Contains(BoundedResourceReadChunkIndexName) - && _chunkIndexes.Contains(BoundedResourceReadChunkEndIndexName); - var useLegacyChunkQueries = !hasBoundedChunkIndexes - && _chunkIndexes.Contains(LegacyBoundedResourceReadFileIndexName); - if (!hasBoundedChunkIndexes && !useLegacyChunkQueries) - return CreateResult(BoundedFileReadStatus.ContentUnavailable, "resource_bounded_read_index_unavailable"); - if (!TryHasNullResourceChunkBoundary(fileId, useLegacyChunkQueries, out var hasNullBoundary)) - return CreateResult(BoundedFileReadStatus.ContentUnavailable, "resource_bounded_read_index_unavailable"); - if (hasNullBoundary) - return CreateResult(BoundedFileReadStatus.InvalidTopology, "resource_chunk_topology_invalid"); - - var effectiveEndLine = Math.Min(endLine, totalLines); - if (nextLine > effectiveEndLine) - return CreateResult(BoundedFileReadStatus.InvalidContinuation); - - // Bound SQLite work as well as returned metadata. Two disjoint index seeks collect - // at most one capped predecessor set and one capped forward set; overlap filtering - // then happens only over those bounded rows in memory. This avoids a full file_id - // scan for late ranges and gaps while retaining legacy chunks that span many lines. - // 返却metadataだけでなくSQLiteの走査量も制限する。互いに素なpredecessor/forwardの - // index seekを各上限件数まで取得し、重複範囲の判定はbounded rowだけmemory上で行う。 - // これにより後方rangeやgapでもfile_id全走査を避け、長いlegacy chunkも扱える。 - var scanEndLine = (int)Math.Min(effectiveEndLine, (long)nextLine + maxLines - 1L); - var chunks = new List(); - var chunkIds = new HashSet(); - string? chunkTopologyFailure = null; - var chunkIndexUnavailable = false; - - int ReadChunkCandidates(string commandText, bool includeEndLine) - { - int ExecuteQuery() - { - using var chunkCmd = _conn.CreateCommand(); - chunkCmd.CommandText = commandText; - SqliteCommandPolicy.AddInt64(chunkCmd, "@fileId", fileId); - SqliteCommandPolicy.AddInt32(chunkCmd, "@startLine", nextLine); - if (includeEndLine) - SqliteCommandPolicy.AddInt32(chunkCmd, "@endLine", scanEndLine); - SqliteCommandPolicy.AddInt32(chunkCmd, "@chunkLimit", MaxBoundedFileReadChunks + 1); - - var rawCandidateCount = 0; - using var chunkReader = chunkCmd.ExecuteTrackedReader(); - while (chunkReader.TrackedRead()) - { - rawCandidateCount++; - _cancellation.ThrowIfCancellationRequested(); - if (chunkReader.IsDBNull(1) || chunkReader.IsDBNull(2) || chunkReader.IsDBNull(3)) - { - chunkTopologyFailure = "resource_chunk_topology_invalid"; - break; - } - - var chunk = new BoundedFileChunk( - chunkReader.GetInt64(0), - chunkReader.GetInt32(1), - chunkReader.GetInt32(2), - chunkReader.GetInt32(3)); - if (chunk.StartLine <= 0 || chunk.EndLine < chunk.StartLine || chunk.ChunkIndex < 0) - { - chunkTopologyFailure = "resource_chunk_topology_invalid"; - break; - } - if (chunk.EndLine < nextLine || chunk.StartLine > scanEndLine) - continue; - - if (chunkIds.Add(chunk.RowId)) - chunks.Add(chunk); - if (chunks.Count > MaxBoundedFileReadChunks) - break; - } - - return rawCandidateCount; - } - - if (!useLegacyChunkQueries) - return ExecuteQuery(); - if (TryRunLegacyResourceMetadataQuery(ExecuteQuery, out var legacyCandidateCount)) - return legacyCandidateCount; - - chunkIndexUnavailable = true; - return 0; - } - - var predecessorCandidateCount = ReadChunkCandidates( - useLegacyChunkQueries - ? LegacyBoundedResourceReadPredecessorSql - : BoundedResourceReadPredecessorSql, - includeEndLine: false); - if (chunkIndexUnavailable) - return CreateResult(BoundedFileReadStatus.ContentUnavailable, "resource_bounded_read_index_unavailable"); - - bool HasChunkCoveringNextLine() - => chunks.Any(chunk => chunk.StartLine <= nextLine && chunk.EndLine >= nextLine); - - if (chunkTopologyFailure == null - && chunks.Count <= MaxBoundedFileReadChunks - && !HasChunkCoveringNextLine() - && predecessorCandidateCount > MaxBoundedFileReadChunks) - { - var endCandidateCount = ReadChunkCandidates( - useLegacyChunkQueries - ? LegacyBoundedResourceReadEndSql - : BoundedResourceReadEndSql, - includeEndLine: false); - if (chunkIndexUnavailable) - return CreateResult(BoundedFileReadStatus.ContentUnavailable, "resource_bounded_read_index_unavailable"); - if (chunkTopologyFailure == null - && !HasChunkCoveringNextLine() - && endCandidateCount > MaxBoundedFileReadChunks) - { - return CreateResult(BoundedFileReadStatus.InvalidTopology, "chunk_candidate_scan_limit_exceeded"); - } - } - - if (chunkTopologyFailure == null && chunks.Count <= MaxBoundedFileReadChunks) - { - ReadChunkCandidates( - useLegacyChunkQueries - ? LegacyBoundedResourceReadForwardSql - : BoundedResourceReadForwardSql, - includeEndLine: true); - } - - if (chunkIndexUnavailable) - return CreateResult(BoundedFileReadStatus.ContentUnavailable, "resource_bounded_read_index_unavailable"); - if (chunkTopologyFailure != null) - return CreateResult(BoundedFileReadStatus.InvalidTopology, chunkTopologyFailure); - if (chunks.Count > MaxBoundedFileReadChunks) - return CreateResult(BoundedFileReadStatus.InvalidTopology, "chunk_limit_exceeded"); - chunks.Sort(static (left, right) => - { - var byStartLine = left.StartLine.CompareTo(right.StartLine); - return byStartLine != 0 ? byStartLine : left.ChunkIndex.CompareTo(right.ChunkIndex); - }); - if (chunks.Count == 0) - { - if (!TryHasAnyStoredResourceChunk(fileId, useLegacyChunkQueries, out var hasStoredChunk)) - return CreateResult(BoundedFileReadStatus.ContentUnavailable, "resource_bounded_read_index_unavailable"); - return hasStoredChunk - ? CreateResult(BoundedFileReadStatus.IncompleteCoverage, "resource_chunk_coverage_incomplete") - : CreateResult(BoundedFileReadStatus.ContentUnavailable, "resource_content_unavailable"); - } - - var state = new BoundedFileReadState(nextLine, continuationByteOffset); - var scanByteLimit = BoundedFileReadScanByteLimitOverride.Value ?? MaxBoundedFileReadScannedUtf8Bytes; - if (scanByteLimit <= 0 || scanByteLimit > MaxBoundedFileReadScannedUtf8Bytes) - scanByteLimit = MaxBoundedFileReadScannedUtf8Bytes; - var content = new StringBuilder(Math.Min(maxUtf8Bytes, BoundedFileReadBufferSize)); - var buffer = ArrayPool.Shared.Rent(BoundedFileReadBufferSize); - try - { - for (var chunkIndex = 0; chunkIndex < chunks.Count; chunkIndex++) - { - var chunk = chunks[chunkIndex]; - _cancellation.ThrowIfCancellationRequested(); - if (state.Stopped) - break; - if (chunk.EndLine < state.NextLine) - continue; - if (chunk.StartLine > state.NextLine) - { - state.IncompleteCoverage = true; - break; - } - - using var blob = new SqliteBlob(_conn, "chunks", "content", chunk.RowId, readOnly: true); - ReadBoundedChunk( - blob, - chunk, - effectiveEndLine, - maxUtf8Bytes, - maxLines, - content, - buffer, - scanByteLimit, - _cancellation, - ref state); - } - } - catch (BoundedFileScanLimitException) - { - state.ScanLimitExceeded = true; - } - catch (InvalidDataException) - { - state.InvalidTopologyReason = "invalid_utf8_content"; - } - finally - { - // Indexed source may contain secrets. Clear the entire pooled array before reuse. - // index済みsourceにsecretが含まれうるため、poolへ戻す前に配列全体を消去する。 - ArrayPool.Shared.Return(buffer, clearArray: true); - } - - if (state.ScanLimitExceeded) - return CreateResult(BoundedFileReadStatus.InvalidTopology, "scan_limit_exceeded"); - if (state.InvalidTopologyReason != null) - return CreateResult(BoundedFileReadStatus.InvalidTopology, state.InvalidTopologyReason); - if (state.InvalidContinuation) - return CreateResult(BoundedFileReadStatus.InvalidContinuation, "invalid_continuation"); - if (state.IncompleteCoverage || (!state.Completed && state.TruncationReason == null)) - return CreateResult(BoundedFileReadStatus.IncompleteCoverage, "resource_chunk_coverage_incomplete"); - - return new BoundedFileReadResult - { - Status = BoundedFileReadStatus.Success, - Path = path, - Lang = lang, - TotalLines = totalLines, - RequestedStartLine = startLine, - RequestedEndLine = endLine, - StartLine = state.FirstReturnedLine ?? nextLine, - EndLine = state.LastReturnedLine ?? nextLine, - Content = content.ToString(), - Utf8Bytes = state.Utf8Bytes, - Truncated = state.TruncationReason != null, - TruncationReason = state.TruncationReason, - NextLine = state.TruncationReason != null ? state.NextLine : null, - NextByteOffset = state.TruncationReason != null ? state.NextByteOffset : null, - }; - } - - private bool HasAnyResourceChunk(long fileId) - { - _cancellation.ThrowIfCancellationRequested(); - using var command = _conn.CreateCommand(); - command.CommandText = "SELECT 1 FROM chunks WHERE file_id = @fileId LIMIT 1"; - SqliteCommandPolicy.AddInt64(command, "@fileId", fileId); - var hasChunk = command.ExecuteScalar() != null; - _cancellation.ThrowIfCancellationRequested(); - return hasChunk; - } - - private bool TryHasAnyStoredResourceChunk(long fileId, bool useLegacyChunkQueries, out bool hasChunk) - { - bool ExecuteQuery() - { - _cancellation.ThrowIfCancellationRequested(); - using var command = _conn.CreateCommand(); - var indexName = useLegacyChunkQueries - ? LegacyBoundedResourceReadFileIndexName - : BoundedResourceReadChunkIndexName; - command.CommandText = $""" - SELECT 1 - FROM chunks INDEXED BY {indexName} - WHERE file_id = @fileId AND content IS NOT NULL - LIMIT 1 - """; - SqliteCommandPolicy.AddInt64(command, "@fileId", fileId); - var result = command.ExecuteScalar() != null; - _cancellation.ThrowIfCancellationRequested(); - return result; - } - - if (!useLegacyChunkQueries) - { - hasChunk = ExecuteQuery(); - return true; - } - - return TryRunLegacyResourceMetadataQuery(ExecuteQuery, out hasChunk); - } - - private bool TryHasNullResourceChunkBoundary(long fileId, bool useLegacyChunkQueries, out bool hasNullBoundary) - { - bool ExecuteQuery() - { - bool Exists(string commandText) - { - using var command = _conn.CreateCommand(); - command.CommandText = commandText; - SqliteCommandPolicy.AddInt64(command, "@fileId", fileId); - return command.ExecuteScalar() != null; - } - - var startIndexName = useLegacyChunkQueries - ? LegacyBoundedResourceReadFileIndexName - : BoundedResourceReadChunkIndexName; - var endIndexName = useLegacyChunkQueries - ? LegacyBoundedResourceReadFileIndexName - : BoundedResourceReadChunkEndIndexName; - _cancellation.ThrowIfCancellationRequested(); - var result = Exists($""" - SELECT 1 - FROM chunks INDEXED BY {startIndexName} - WHERE file_id = @fileId - AND content IS NOT NULL - AND start_line IS NULL - LIMIT 1 - """) || Exists($""" - SELECT 1 - FROM chunks INDEXED BY {endIndexName} - WHERE file_id = @fileId - AND content IS NOT NULL - AND end_line IS NULL - LIMIT 1 - """); - _cancellation.ThrowIfCancellationRequested(); - return result; - } - - if (!useLegacyChunkQueries) - { - hasNullBoundary = ExecuteQuery(); - return true; - } - - return TryRunLegacyResourceMetadataQuery(ExecuteQuery, out hasNullBoundary); - } - - private bool TryRunLegacyResourceMetadataQuery(Func action, out T result) - { - var maxVmSteps = LegacyResourceReadSqliteVmStepLimitOverride.Value - ?? MaxLegacyResourceReadSqliteVmSteps; - maxVmSteps = Math.Clamp(maxVmSteps, 1, MaxLegacyResourceReadSqliteVmSteps); - var callbackLimit = Math.Max(1, (maxVmSteps + LegacyResourceReadProgressOperations - 1) - / LegacyResourceReadProgressOperations); - var callbackCount = 0; - var budgetExceeded = false; - SQLitePCL.delegate_progress progress = _ => - { - callbackCount++; - if (callbackCount <= callbackLimit) - return 0; - - budgetExceeded = true; - return 1; - }; - - SQLitePCL.raw.sqlite3_progress_handler( - _conn.Handle, - LegacyResourceReadProgressOperations, - progress, - null!); - try - { - result = action(); - return true; - } - catch (SqliteException exception) when ( - budgetExceeded - && !_cancellation.IsCancellationRequested - && exception.SqliteErrorCode == 9) - { - result = default!; - return false; - } - finally - { - SQLitePCL.raw.sqlite3_progress_handler(_conn.Handle, 0, null!, null!); - GC.KeepAlive(progress); - } + var reader = new BoundedIndexedContentReader( + _conn, + _cancellation, + _hasChunksTable, + _chunkIndexes, + BoundedFileReadScanByteLimitOverride.Value, + LegacyResourceReadSqliteVmStepLimitOverride.Value); + return reader.Read(new BoundedIndexedContentRequest( + file, + startLine, + endLine, + maxUtf8Bytes, + maxLines, + continuationLine, + continuationByteOffset)); } - private static void ReadBoundedChunk( - SqliteBlob blob, - BoundedFileChunk chunk, - int effectiveEndLine, - int maxUtf8Bytes, - int maxLines, - StringBuilder content, - byte[] buffer, - int maxScannedUtf8Bytes, - CancellationToken cancellationToken, - ref BoundedFileReadState state) - { - var bufferOffset = 0; - var bufferedBytes = 0; - var localLine = chunk.StartLine; - var localByteOffset = 0; - Span runeBytes = stackalloc byte[4]; - Span runeChars = stackalloc char[2]; - - while (TryReadBlobByte(blob, buffer, ref bufferOffset, ref bufferedBytes, maxScannedUtf8Bytes, cancellationToken, ref state, out var firstByte)) - { - if (localLine > chunk.EndLine) - break; - - if (localLine < state.NextLine) - { - if (firstByte == (byte)'\n') - { - localLine++; - localByteOffset = 0; - } - else - { - localByteOffset++; - } - continue; - } - - if (localLine > state.NextLine) - { - state.InvalidContinuation = true; - return; - } - - if (firstByte == (byte)'\n') - { - if (state.NextByteOffset != localByteOffset) - { - state.InvalidContinuation = true; - return; - } - - if (!CompleteBoundedLine( - localLine, - localByteOffset, - effectiveEndLine, - maxUtf8Bytes, - maxLines, - content, - ref state)) - { - return; - } - - localLine++; - localByteOffset = 0; - continue; - } - - var runeByteCount = ReadUtf8Rune( - blob, - firstByte, - buffer, - ref bufferOffset, - ref bufferedBytes, - maxScannedUtf8Bytes, - cancellationToken, - ref state, - runeBytes, - out var rune); - var scalarStartOffset = localByteOffset; - localByteOffset += runeByteCount; - - if (scalarStartOffset < state.NextByteOffset) - { - if (localByteOffset > state.NextByteOffset) - state.InvalidContinuation = true; - if (state.InvalidContinuation) - return; - continue; - } - - if (scalarStartOffset != state.NextByteOffset) - { - state.InvalidContinuation = true; - return; - } - if (state.Utf8Bytes > maxUtf8Bytes - runeByteCount) - { - state.TruncationReason = "max_bytes"; - return; - } - if (!TryEnterBoundedLine(localLine, maxLines, ref state)) - return; - - var runeCharCount = rune.EncodeToUtf16(runeChars); - content.Append(runeChars[..runeCharCount]); - state.Utf8Bytes += runeByteCount; - state.NextByteOffset = localByteOffset; - } - - // Persisted chunks omit the separator after their final line. Treat blob EOF as - // that line boundary and synthesize exactly one LF when the requested range continues. - // 永続化chunkは最終行後のseparatorを持たないため、blob EOFを行境界として扱い、 - // 要求範囲が続く場合だけLFを1つ合成する。 - if (!state.Stopped && localLine <= chunk.EndLine && localLine == state.NextLine) - { - if (state.NextByteOffset != localByteOffset) - { - state.InvalidContinuation = true; - return; - } - - _ = CompleteBoundedLine( - localLine, - localByteOffset, - effectiveEndLine, - maxUtf8Bytes, - maxLines, - content, - ref state); - } - } - - private static bool CompleteBoundedLine( - int line, - int lineByteLength, - int effectiveEndLine, - int maxUtf8Bytes, - int maxLines, - StringBuilder content, - ref BoundedFileReadState state) - { - if (!TryEnterBoundedLine(line, maxLines, ref state)) - return false; - - if (line >= effectiveEndLine) - { - state.Completed = true; - return false; - } - - if (state.Utf8Bytes >= maxUtf8Bytes) - { - state.NextLine = line; - state.NextByteOffset = lineByteLength; - state.TruncationReason = "max_bytes"; - return false; - } - - content.Append('\n'); - state.Utf8Bytes++; - state.NextLine = line + 1; - state.NextByteOffset = 0; - if (state.ReturnedLineCount >= maxLines) - { - state.TruncationReason = "max_lines"; - return false; - } - - return true; - } - - private static bool TryEnterBoundedLine(int line, int maxLines, ref BoundedFileReadState state) - { - if (state.LastReturnedLine == line) - return true; - if (state.ReturnedLineCount >= maxLines) - { - state.TruncationReason = "max_lines"; - return false; - } - - state.FirstReturnedLine ??= line; - state.LastReturnedLine = line; - state.ReturnedLineCount++; - return true; - } - - private static int ReadUtf8Rune( - SqliteBlob blob, - byte firstByte, - byte[] buffer, - ref int bufferOffset, - ref int bufferedBytes, - int maxScannedUtf8Bytes, - CancellationToken cancellationToken, - ref BoundedFileReadState state, - Span runeBytes, - out Rune rune) - { - var byteCount = firstByte switch - { - <= 0x7f => 1, - >= 0xc2 and <= 0xdf => 2, - >= 0xe0 and <= 0xef => 3, - >= 0xf0 and <= 0xf4 => 4, - _ => throw new InvalidDataException("Indexed chunk contains invalid UTF-8."), - }; - - runeBytes[0] = firstByte; - for (var i = 1; i < byteCount; i++) - { - if (!TryReadBlobByte(blob, buffer, ref bufferOffset, ref bufferedBytes, maxScannedUtf8Bytes, cancellationToken, ref state, out runeBytes[i])) - throw new InvalidDataException("Indexed chunk ends inside a UTF-8 scalar value."); - } - - var status = Rune.DecodeFromUtf8(runeBytes[..byteCount], out rune, out var bytesConsumed); - if (status != OperationStatus.Done || bytesConsumed != byteCount) - throw new InvalidDataException("Indexed chunk contains invalid UTF-8."); - return byteCount; - } - - private static bool TryReadBlobByte( - SqliteBlob blob, - byte[] buffer, - ref int bufferOffset, - ref int bufferedBytes, - int maxScannedUtf8Bytes, - CancellationToken cancellationToken, - ref BoundedFileReadState state, - out byte value) - { - if (bufferOffset >= bufferedBytes) - { - cancellationToken.ThrowIfCancellationRequested(); - if (blob.Position >= blob.Length) - { - value = 0; - return false; - } - var remaining = maxScannedUtf8Bytes - state.ScannedUtf8Bytes; - if (remaining <= 0) - throw new BoundedFileScanLimitException(); - bufferedBytes = blob.Read(buffer, 0, Math.Min(buffer.Length, remaining)); - bufferOffset = 0; - if (bufferedBytes == 0) - { - value = 0; - return false; - } - } - - value = buffer[bufferOffset++]; - state.ScannedUtf8Bytes++; - return true; - } - - private readonly record struct BoundedFileChunk(long RowId, int StartLine, int EndLine, int ChunkIndex); - - private sealed class BoundedFileScanLimitException : Exception; - - private struct BoundedFileReadState(int nextLine, int nextByteOffset) - { - public int NextLine = nextLine; - public int NextByteOffset = nextByteOffset; - public int Utf8Bytes; - public int ScannedUtf8Bytes; - public int ReturnedLineCount; - public int? FirstReturnedLine; - public int? LastReturnedLine; - public string? TruncationReason; - public string? InvalidTopologyReason; - public bool Completed; - public bool InvalidContinuation; - public bool IncompleteCoverage; - public bool ScanLimitExceeded; - public readonly bool Stopped => Completed || InvalidContinuation || IncompleteCoverage || ScanLimitExceeded || TruncationReason != null; - } /// /// Return the length of the focused excerpt line when it is part of the reconstructed range. diff --git a/src/CodeIndex/Database/DbReader.GraphQueries.cs b/src/CodeIndex/Database/DbReader.GraphQueries.cs index 0023cda06a..e06fcbe300 100644 --- a/src/CodeIndex/Database/DbReader.GraphQueries.cs +++ b/src/CodeIndex/Database/DbReader.GraphQueries.cs @@ -1,5 +1,4 @@ using CodeIndex.Indexer; -using Microsoft.Data.Sqlite; using System.Text; using System.Text.RegularExpressions; @@ -86,205 +85,30 @@ private List GetCallersCore(string query, int limit, string? lang, return new List(); lang = NormalizeQueryLanguage(lang); query = NormalizeSymbolSearchQuery(query, lang, exact) ?? query ?? string.Empty; - if (!_hasReferencesTable) return new List(); - using var cmd = _conn.CreateCommand(); - var referenceLineJoin = ReferenceLineJoinSql("r"); - var contextSql = ReferenceContextSql("r"); - var selfReferenceSql = _referenceColumns.Contains("is_self_reference") ? "r.is_self_reference" : "0"; - var mutualRecursionSql = _referenceColumns.Contains("is_mutual_recursion") ? "r.is_mutual_recursion" : "0"; - var callerContainerPredicate = BuildCallerContainerPredicate("f", "r"); - var supportedLangPredicate = BuildGraphSupportedLanguagePredicate(cmd, "f", "graphLang"); - - var groupedReferenceKindSql = rawKinds - ? GetGroupedCallerReferenceKindSql("r.reference_kind") - : GetGroupedCallerLogicalReferenceKindSql("r.reference_kind"); - var groupedReferenceKindGroupSql = rawKinds - ? GetRawReferenceKindSql("r.reference_kind") - : GetLogicalReferenceKindSql("r.reference_kind"); - var sql = @" - WITH logical_references AS ( - SELECT f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, - " + groupedReferenceKindSql + @" AS reference_kind, - r.reference_kind AS raw_reference_kind, - " + groupedReferenceKindGroupSql + @" AS count_reference_kind, - COUNT(*) AS reference_count, - " + ReferenceWeightedScoreSql("r.reference_kind") + @" AS weighted_score, - (CAST(r.line AS INTEGER) * 4294967296 + r.column_number) AS location_key, - MAX(" + selfReferenceSql + @") AS is_self_reference, - MAX(" + mutualRecursionSql + @") AS is_mutual_recursion - FROM symbol_references r - JOIN files f ON r.file_id = f.id" + referenceLineJoin + @" - WHERE " + callerContainerPredicate + @" - AND " + GetCallableReferenceKindPredicateSql("r.reference_kind", referenceKind, "f.lang", includeMemberReads) + @" - AND " + supportedLangPredicate; - if (targetSymbolId != null && HasTable("symbol_reference_candidates")) - { - // Candidate membership alone is not an edge: an ambiguous reference can list - // several possible targets. Only authoritative resolution states may contribute - // callers to a candidate-specific inspect bundle. - // candidate membership だけでは edge ではない。ambiguous reference は複数の - // 候補を持ち得るため、candidate 別 inspect bundle の callers には authoritative - // な resolution state だけを採用する。 - sql += _referenceColumns.Contains("resolution_state") - ? " AND r.resolution_state IN ('resolved', 'resolved_group')" - : " AND 1 = 0"; - sql += @" - AND EXISTS ( - SELECT 1 - FROM symbol_reference_candidates AS identity_candidate - WHERE identity_candidate.reference_id = r.id - AND identity_candidate.symbol_id = @targetSymbolId - )"; - } - if (excludeSelfReferences) - sql += $" AND {selfReferenceSql} = 0"; - var allowSqlLeafFallback = AllowSqlLeafFallbackForQuery(query); - var allowCSharpQualifiedContextMatch = SqlNameResolver.HasQualifier(query) - && !HasQualifiedSymbolDefinition(query, lang, pathPatterns, excludePathPatterns, excludeTests); - var allowQualifiedLeafFallback = HasSingleQualifiedSymbolDefinition(query, lang, pathPatterns, excludePathPatterns, excludeTests); - var useSqlQualifiedContextMatch = SqlNameResolver.HasQualifier(query); - var cssScssVariableAlias = ComputeCssScssVariableAlias(query); - var cssScssVariableAliasScope = cssScssVariableAlias != null - ? " AND f.lang = 'css'" - : string.Empty; - if (useSqlQualifiedContextMatch && exact && _foldReady) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: true, like: false); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); - sql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContextMatch && exact) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: false, like: false); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: false); - sql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name = @query COLLATE NOCASE) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContextMatch && _foldReady) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: true, like: true); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); - sql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name LIKE @query ESCAPE '\\') OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContextMatch) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: false, like: true); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: false); - sql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name LIKE @query ESCAPE '\\') OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (exact && _foldReady) - sql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : $" AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}"; - else if (exact) - sql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND (r.symbol_name = @query COLLATE NOCASE OR (r.symbol_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))" - : " AND (r.symbol_name = @query COLLATE NOCASE OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))" - : " AND r.symbol_name = @query COLLATE NOCASE"; - else - sql += cssScssVariableAlias != null - ? $" AND (r.symbol_name LIKE @query ESCAPE '\\' OR (r.symbol_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))" - : " AND (r.symbol_name LIKE @query ESCAPE '\\' OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))"; - if (lang != null) - { - sql += IncludeAmbiguousMSourceForIdentityTarget(lang, targetSymbolId) - ? " AND (f.lang = @lang OR f.lang = 'ambiguous_m')" - : " AND f.lang = @lang"; - } - sql += BuildCSharpBareMemberReferenceFilter( + if (!_hasReferencesTable) + return new List(); + + var request = CreateGraphReferenceQueryRequest( query, + limit, lang, - "f", - "r", - includeQualifiedCommonCalls); - AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); - sql += @" - GROUP BY f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.file_id, r.line, r.column_number, " + groupedReferenceKindGroupSql + @", r.reference_kind - ) - SELECT path, lang, " + BuildCallerKindProjectionSql("r") + @" AS container_kind, " + BuildCallerNameProjectionSql("r") + @" AS container_name, symbol_name, - " + (rawKinds ? GetGroupedCallerReferenceKindSql("r.reference_kind") : GetPreferredLogicalReferenceKindSql("r.reference_kind")) + @" AS reference_kind, - (MIN(location_key) / 4294967296) AS first_line, - (MIN(location_key) % 4294967296) AS first_column, - SUM(r.reference_count) AS reference_count, - GROUP_CONCAT(DISTINCT r.reference_kind) AS reference_kinds, - GROUP_CONCAT(r.count_reference_kind || ':' || r.reference_count) AS reference_kind_counts, - SUM(r.weighted_score) AS weighted_score, - MAX(r.is_self_reference) AS is_self_reference, - MAX(r.is_mutual_recursion) AS is_mutual_recursion - FROM logical_references r - GROUP BY path, lang, container_kind, container_name, symbol_name"; - sql += $" ORDER BY {BuildReferenceRankOrderSql(rankMode, "r.symbol_name")} LIMIT @limit OFFSET @offset"; - - cmd.CommandText = sql; - string callersQueryParam; - if (!exact) - callersQueryParam = $"%{EscapeLikeQuery(query)}%"; - else if (_foldReady) - callersQueryParam = FoldNameForLanguage(query, lang); - else - callersQueryParam = query; - if (exact && _foldReady) - AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); - else - SqliteCommandPolicy.Add(cmd, "@query", callersQueryParam); - SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); - AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback, allowCSharpQualifiedContextMatch); - SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); - if (cssScssVariableAlias != null) - { - var aliasParam = exact && _foldReady - ? NameFold.Fold(cssScssVariableAlias) ?? cssScssVariableAlias - : cssScssVariableAlias; - SqliteCommandPolicy.Add(cmd, "@queryCssScssVariableAlias", aliasParam); - } - SqliteCommandPolicy.Add(cmd, "@rawQuery", query); - SqliteCommandPolicy.Add(cmd, "@rankingQuery", query.Trim()); - if (RequiresReferenceKindParameter(referenceKind)) - SqliteCommandPolicy.Add(cmd, "@referenceKind", referenceKind); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", NormalizeQueryLanguage(lang)); - if (targetSymbolId != null && HasTable("symbol_reference_candidates")) - SqliteCommandPolicy.Add(cmd, "@targetSymbolId", targetSymbolId.Value); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - SqliteCommandPolicy.Add(cmd, "@limit", limit); - SqliteCommandPolicy.Add(cmd, "@offset", Math.Max(0, offset)); - - var results = new List(); - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - var primaryKind = reader.GetString(5); - var kindAggregate = TruncateReferenceKindAggregate(GetNullableString(reader, 9), out var kindsTruncated); - var countAggregate = TruncateReferenceKindAggregate(GetNullableString(reader, 10), out var countsTruncated); - var kinds = ParseDistinctReferenceKinds(kindAggregate, primaryKind); - var counts = ParseReferenceKindCounts(countAggregate, primaryKind, reader.GetInt32(8)); - results.Add(new CallerResult - { - Path = reader.GetString(0), - Lang = GetNullableString(reader, 1), - CallerKind = GetNullableString(reader, 2), - CallerName = GetNullableString(reader, 3), - CalleeName = reader.GetString(4), - ReferenceKind = primaryKind, - ReferenceKinds = kinds, - HasMixedReferenceKinds = kinds.Count > 1, - ReferenceKindCounts = counts, - AggregateTruncated = kindsTruncated || countsTruncated, - ReferenceWeightScore = reader.GetDouble(11), - FirstLine = reader.GetInt32(6), - FirstColumn = reader.GetInt32(7), - ReferenceCount = reader.GetInt32(8), - HasSelfReference = reader.GetInt32(12) != 0, - HasMutualRecursion = reader.GetInt32(13) != 0, - }); - } - return results; + referenceKind, + pathPatterns, + excludePathPatterns, + excludeTests, + exact, + rawKinds, + includeQualifiedCommonCalls, + includeMemberReads, + identitySymbolId: targetSymbolId, + excludeSelfReferences, + offset); + var plan = BuildGraphReferenceQueryPlan( + CallerGraphReferenceDirection, + request, + GraphReferenceQueryShape.List, + rankMode); + return ExecuteGraphReferenceList(plan, ProjectCallerResult); } public int CountCallers(string query, int limit = 20, string? lang = null, string? referenceKind = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, bool exact = false, bool rawKinds = false, bool includeQualifiedCommonCalls = false, bool includeMemberReads = false) @@ -293,115 +117,26 @@ public int CountCallers(string query, int limit = 20, string? lang = null, strin return 0; lang = NormalizeQueryLanguage(lang); query = NormalizeSymbolSearchQuery(query, lang, exact) ?? query ?? string.Empty; - if (!_hasReferencesTable) return 0; - using var cmd = _conn.CreateCommand(); - var referenceLineJoin = ReferenceLineJoinSql("r"); - var contextSql = ReferenceContextSql("r"); - var groupedSql = @" - SELECT path, lang, container_kind, container_name, symbol_name - FROM ( - SELECT f.path AS path, f.lang AS lang, r.container_kind AS container_kind, - r.container_name AS container_name, r.symbol_name AS symbol_name - FROM symbol_references r - JOIN files f ON r.file_id = f.id" + referenceLineJoin + @" - WHERE " + BuildCallerContainerPredicate("f", "r"); - groupedSql += $" AND {BuildGraphSupportedLanguagePredicate(cmd, "f", "graphLang")}"; - - groupedSql += $" AND {GetCallableReferenceKindPredicateSql("r.reference_kind", referenceKind, "f.lang", includeMemberReads)}"; - var allowSqlLeafFallback = AllowSqlLeafFallbackForQuery(query); - var allowCSharpQualifiedContextMatch = SqlNameResolver.HasQualifier(query) - && !HasQualifiedSymbolDefinition(query, lang, pathPatterns, excludePathPatterns, excludeTests); - var allowQualifiedLeafFallback = HasSingleQualifiedSymbolDefinition(query, lang, pathPatterns, excludePathPatterns, excludeTests); - var useSqlQualifiedContextMatch = SqlNameResolver.HasQualifier(query); - var cssScssVariableAlias = ComputeCssScssVariableAlias(query); - var cssScssVariableAliasScope = cssScssVariableAlias != null - ? " AND f.lang = 'css'" - : string.Empty; - if (useSqlQualifiedContextMatch && exact && _foldReady) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: true, like: false); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); - groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContextMatch && exact) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: false, like: false); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: false); - groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name = @query COLLATE NOCASE) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContextMatch && _foldReady) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: true, like: true); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); - groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name LIKE @query ESCAPE '\\') OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContextMatch) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: false, like: true); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: false); - groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name LIKE @query ESCAPE '\\') OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (exact && _foldReady) - groupedSql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : $" AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}"; - else if (exact) - groupedSql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND (r.symbol_name = @query COLLATE NOCASE OR (r.symbol_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))" - : " AND (r.symbol_name = @query COLLATE NOCASE OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))" - : " AND r.symbol_name = @query COLLATE NOCASE"; - else - groupedSql += cssScssVariableAlias != null - ? $" AND (r.symbol_name LIKE @query ESCAPE '\\' OR (r.symbol_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))" - : " AND (r.symbol_name LIKE @query ESCAPE '\\' OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))"; - if (lang != null) - groupedSql += " AND f.lang = @lang"; - groupedSql += BuildCSharpBareMemberReferenceFilter( + if (!_hasReferencesTable) + return 0; + + var request = CreateGraphReferenceQueryRequest( query, + limit, lang, - "f", - "r", - includeQualifiedCommonCalls); - AppendPathFilters(ref groupedSql, pathPatterns, excludePathPatterns, excludeTests); - groupedSql += $" GROUP BY f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.file_id, r.line, r.column_number, {(rawKinds ? GetRawReferenceKindSql("r.reference_kind") : GetLogicalReferenceKindSql("r.reference_kind"))}"; - groupedSql += " ) grouped_call_sites GROUP BY path, lang, container_kind, container_name, symbol_name LIMIT @limit"; - - cmd.CommandText = $"SELECT COUNT(*) FROM ({groupedSql})"; - var value = !exact - ? $"%{EscapeLikeQuery(query)}%" - : _foldReady - ? FoldNameForLanguage(query, lang) - : query; - if (exact && _foldReady) - AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); - else - SqliteCommandPolicy.Add(cmd, "@query", value); - SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); - AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback, allowCSharpQualifiedContextMatch); - SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); - if (cssScssVariableAlias != null) - { - var aliasParam = exact && _foldReady - ? NameFold.Fold(cssScssVariableAlias) ?? cssScssVariableAlias - : cssScssVariableAlias; - SqliteCommandPolicy.Add(cmd, "@queryCssScssVariableAlias", aliasParam); - } - if (RequiresReferenceKindParameter(referenceKind)) - SqliteCommandPolicy.Add(cmd, "@referenceKind", referenceKind); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", NormalizeQueryLanguage(lang)); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - SqliteCommandPolicy.Add(cmd, "@limit", limit); - - var raw = cmd.ExecuteScalar(); - return raw is long l ? (int)l : Convert.ToInt32(raw); + referenceKind, + pathPatterns, + excludePathPatterns, + excludeTests, + exact, + rawKinds, + includeQualifiedCommonCalls, + includeMemberReads); + var plan = BuildGraphReferenceQueryPlan( + CallerGraphReferenceDirection, + request, + GraphReferenceQueryShape.LimitedCount); + return ExecuteGraphReferenceLimitedCount(plan); } public QueryCountResult CountCallersTotal(string query, string? lang = null, string? referenceKind = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, bool exact = false, bool rawKinds = false, bool includeQualifiedCommonCalls = false, bool includeMemberReads = false) @@ -434,131 +169,24 @@ private QueryCountResult CountCallersTotalCore( if (!_hasReferencesTable) return new QueryCountResult(0, 0); - using var cmd = _conn.CreateCommand(); - var referenceLineJoin = ReferenceLineJoinSql("r"); - var contextSql = ReferenceContextSql("r"); - var cssScssVariableAlias = ComputeCssScssVariableAlias(query); - var cssScssVariableAliasScope = cssScssVariableAlias != null - ? " AND f.lang = 'css'" - : string.Empty; - var groupedSql = @" - SELECT path, lang - FROM ( - SELECT f.path AS path, f.lang AS lang, r.container_kind AS container_kind, - r.container_name AS container_name, r.symbol_name AS symbol_name - FROM symbol_references r - JOIN files f ON r.file_id = f.id" + referenceLineJoin + @" - WHERE " + BuildCallerContainerPredicate("f", "r"); - groupedSql += $" AND {BuildGraphSupportedLanguagePredicate(cmd, "f", "graphLang")}"; - - groupedSql += $" AND {GetCallableReferenceKindPredicateSql("r.reference_kind", referenceKind, "f.lang", includeMemberReads)}"; - if (targetSymbolId != null && HasTable("symbol_reference_candidates")) - { - groupedSql += _referenceColumns.Contains("resolution_state") - ? " AND r.resolution_state IN ('resolved', 'resolved_group')" - : " AND 1 = 0"; - groupedSql += @" - AND EXISTS ( - SELECT 1 - FROM symbol_reference_candidates AS identity_candidate - WHERE identity_candidate.reference_id = r.id - AND identity_candidate.symbol_id = @targetSymbolId - )"; - } - var allowSqlLeafFallback = AllowSqlLeafFallbackForQuery(query); - var allowCSharpQualifiedContextMatch = SqlNameResolver.HasQualifier(query) - && !HasQualifiedSymbolDefinition(query, lang, pathPatterns, excludePathPatterns, excludeTests); - var allowQualifiedLeafFallback = HasSingleQualifiedSymbolDefinition(query, lang, pathPatterns, excludePathPatterns, excludeTests); - var useSqlQualifiedContextMatch = SqlNameResolver.HasQualifier(query); - if (useSqlQualifiedContextMatch && exact && _foldReady) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: true, like: false); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); - groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContextMatch && exact) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: false, like: false); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: false); - groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name = @query COLLATE NOCASE) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContextMatch && _foldReady) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: true, like: true); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: true); - groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name LIKE @query ESCAPE '\\') OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContextMatch) - { - var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded: false, like: true); - var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded: false); - groupedSql += $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND r.symbol_name LIKE @query ESCAPE '\\') OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; - } - else if (exact && _foldReady) - groupedSql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (r.symbol_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : $" AND ({BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")} OR (f.lang = 'sql' AND r.symbol_name_folded = @aliasQueryLeafFolded))" - : $" AND {BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query")}"; - else if (exact) - groupedSql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND (r.symbol_name = @query COLLATE NOCASE OR (r.symbol_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))" - : " AND (r.symbol_name = @query COLLATE NOCASE OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))" - : " AND r.symbol_name = @query COLLATE NOCASE"; - else - groupedSql += cssScssVariableAlias != null - ? $" AND (r.symbol_name LIKE @query ESCAPE '\\' OR (r.symbol_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))" - : " AND (r.symbol_name LIKE @query ESCAPE '\\' OR (f.lang = 'sql' AND r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE))"; - if (lang != null) - { - groupedSql += IncludeAmbiguousMSourceForIdentityTarget(lang, targetSymbolId) - ? " AND (f.lang = @lang OR f.lang = 'ambiguous_m')" - : " AND f.lang = @lang"; - } - groupedSql += BuildCSharpBareMemberReferenceFilter( + var request = CreateGraphReferenceQueryRequest( query, + limit: 0, lang, - "f", - "r", - includeQualifiedCommonCalls); - AppendPathFilters(ref groupedSql, pathPatterns, excludePathPatterns, excludeTests); - groupedSql += $" GROUP BY f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.file_id, r.line, r.column_number, {(rawKinds ? GetRawReferenceKindSql("r.reference_kind") : GetLogicalReferenceKindSql("r.reference_kind"))}"; - groupedSql += " ) grouped_call_sites GROUP BY path, lang, container_kind, container_name, symbol_name"; - - cmd.CommandText = $"SELECT COUNT(*), COUNT(DISTINCT path), MAX(CASE WHEN lang = 'sql' THEN 1 ELSE 0 END) FROM ({groupedSql})"; - var value = !exact - ? $"%{EscapeLikeQuery(query)}%" - : _foldReady - ? FoldNameForLanguage(query, lang) - : query; - if (exact && _foldReady) - AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); - else - SqliteCommandPolicy.Add(cmd, "@query", value); - SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); - AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback, allowCSharpQualifiedContextMatch); - SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); - if (cssScssVariableAlias != null) - { - var aliasParam = exact && _foldReady - ? NameFold.Fold(cssScssVariableAlias) ?? cssScssVariableAlias - : cssScssVariableAlias; - SqliteCommandPolicy.Add(cmd, "@queryCssScssVariableAlias", aliasParam); - } - if (RequiresReferenceKindParameter(referenceKind)) - SqliteCommandPolicy.Add(cmd, "@referenceKind", referenceKind); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", NormalizeQueryLanguage(lang)); - if (targetSymbolId != null) - SqliteCommandPolicy.Add(cmd, "@targetSymbolId", targetSymbolId.Value); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - - return ExecuteCountSummary(cmd); + referenceKind, + pathPatterns, + excludePathPatterns, + excludeTests, + exact, + rawKinds, + includeQualifiedCommonCalls, + includeMemberReads, + identitySymbolId: targetSymbolId); + var plan = BuildGraphReferenceQueryPlan( + CallerGraphReferenceDirection, + request, + GraphReferenceQueryShape.TotalCount); + return ExecuteGraphReferenceTotalCount(plan); } /// @@ -596,177 +224,29 @@ private List GetCalleesCore(string query, int limit, string? lang, return new List(); lang = NormalizeQueryLanguage(lang); query = NormalizeSymbolSearchQuery(query, lang, exact) ?? query ?? string.Empty; - if (!_hasReferencesTable) return new List(); - using var cmd = _conn.CreateCommand(); - - var preferredCalleeKindSql = rawKinds - ? GetPreferredReferenceKindSql("r.reference_kind") - : GetPreferredLogicalReferenceKindSql("r.reference_kind"); - var calleeGroupKindSql = rawKinds - ? GetRawReferenceKindSql("r.reference_kind") - : GetLogicalReferenceKindSql("r.reference_kind"); - var referenceSpanLengthSql = _referenceColumns.Contains("span_length") - ? "r.span_length" - : "NULL"; - var sql = $@" - WITH logical_references AS ( - SELECT f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, - {preferredCalleeKindSql} AS reference_kind, - r.reference_kind AS raw_reference_kind, - {calleeGroupKindSql} AS count_reference_kind, - COUNT(*) AS reference_count, - {ReferenceWeightedScoreSql("r.reference_kind")} AS weighted_score, - r.line, - r.column_number, - {referenceSpanLengthSql} AS span_length - FROM symbol_references r - JOIN files f ON r.file_id = f.id - WHERE r.container_name IS NOT NULL - AND {GetCallableReferenceKindPredicateSql("r.reference_kind", referenceKind, "f.lang", includeMemberReads)} - AND {BuildGraphSupportedLanguagePredicate(cmd, "f", "graphLang")}"; - if (sourceSymbolId != null && _referenceColumns.Contains("source_symbol_id")) - sql += " AND r.source_symbol_id = @sourceSymbolId"; - var allowSqlLeafFallback = AllowSqlLeafFallbackForQuery(query); - var allowQualifiedLeafFallback = HasSingleQualifiedSymbolDefinition(query, lang, pathPatterns, excludePathPatterns, excludeTests); - var useSqlQualifiedContainerMatch = SqlNameResolver.HasQualifier(query); - var cssScssVariableAlias = ComputeCssScssVariableAlias(query); - var cssScssVariableAliasScope = cssScssVariableAlias != null - ? " AND f.lang = 'css'" - : string.Empty; - if (exact && useSqlQualifiedContainerMatch && _foldReady) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: true); - sql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name_folded(r.container_name) = @aliasQueryNormalizedFolded) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}) OR {qualifiedLeafFallbackSql})"; - } - else if (exact && useSqlQualifiedContainerMatch) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: false); - sql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name(r.container_name) = @aliasQueryNormalized COLLATE NOCASE) OR ((f.lang != 'sql') AND r.container_name = @query COLLATE NOCASE) OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContainerMatch && _foldReady) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: true); - sql += $" AND (r.container_name LIKE @query ESCAPE '\\' OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContainerMatch) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: false); - sql += $" AND (r.container_name LIKE @query ESCAPE '\\' OR {qualifiedLeafFallbackSql})"; - } - else if (exact && _foldReady) - sql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (r.container_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : $" AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}"; - else if (exact) - sql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND (r.container_name = @query COLLATE NOCASE OR (r.container_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))" - : " AND (r.container_name = @query COLLATE NOCASE OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))" - : " AND r.container_name = @query COLLATE NOCASE"; - else - sql += cssScssVariableAlias != null - ? $" AND (r.container_name LIKE @query ESCAPE '\\' OR (r.container_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))" - : " AND (r.container_name LIKE @query ESCAPE '\\' OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))"; - if (lang != null) - sql += " AND f.lang = @lang"; - if (!includeQualifiedCommonCalls) - sql += BuildCSharpQualifiedCommonCallNoiseFilter("f", "r"); - AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); - sql += $@" - GROUP BY f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.file_id, r.line, r.column_number, {referenceSpanLengthSql}, r.reference_kind - ), - ranked_call_sites AS ( - SELECT logical_references.*, - ROW_NUMBER() OVER ( - PARTITION BY path, lang, container_kind, container_name, symbol_name, reference_kind - ORDER BY CASE WHEN column_number IS NULL THEN 1 ELSE 0 END, - line, - column_number, - COALESCE(span_length, 0) - ) AS location_rank - FROM logical_references - ) - SELECT path, lang, container_kind, container_name, symbol_name, - reference_kind, - MAX(CASE WHEN location_rank = 1 THEN line END) AS first_line, - MAX(CASE WHEN location_rank = 1 THEN column_number END) AS first_column, - MAX(CASE WHEN location_rank = 1 THEN span_length END) AS first_length, - SUM(r.reference_count) AS reference_count, - GROUP_CONCAT(DISTINCT reference_kind) AS reference_kinds, - GROUP_CONCAT(r.count_reference_kind || ':' || r.reference_count) AS reference_kind_counts, - SUM(r.weighted_score) AS weighted_score - FROM ranked_call_sites r - GROUP BY path, lang, container_kind, container_name, symbol_name, reference_kind"; - sql += $" ORDER BY {BuildReferenceRankOrderSql(rankMode, "r.container_name")} LIMIT @limit OFFSET @offset"; - - cmd.CommandText = sql; - string calleesQueryParam; - if (!exact) - calleesQueryParam = $"%{EscapeLikeQuery(query)}%"; - else if (_foldReady) - calleesQueryParam = FoldNameForLanguage(query, lang); - else - calleesQueryParam = query; - if (exact && _foldReady) - AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); - else - SqliteCommandPolicy.Add(cmd, "@query", calleesQueryParam); - SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); - SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); - SqliteCommandPolicy.Add(cmd, "@aliasQueryNormalized", SqlNameResolver.NormalizeQualifiedName(query)); - SqliteCommandPolicy.Add(cmd, "@aliasQueryNormalizedFolded", NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(query)) ?? SqlNameResolver.NormalizeQualifiedName(query)); - SqliteCommandPolicy.Add(cmd, "@aliasQuerySegmentCount", SqlNameResolver.GetSegmentCount(query)); - if (cssScssVariableAlias != null) - { - var aliasParam = exact && _foldReady - ? NameFold.Fold(cssScssVariableAlias) ?? cssScssVariableAlias - : cssScssVariableAlias; - SqliteCommandPolicy.Add(cmd, "@queryCssScssVariableAlias", aliasParam); - } - SqliteCommandPolicy.Add(cmd, "@rawQuery", query); - SqliteCommandPolicy.Add(cmd, "@rankingQuery", query.Trim()); - AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback); - if (RequiresReferenceKindParameter(referenceKind)) - SqliteCommandPolicy.Add(cmd, "@referenceKind", referenceKind); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - if (sourceSymbolId != null && _referenceColumns.Contains("source_symbol_id")) - SqliteCommandPolicy.Add(cmd, "@sourceSymbolId", sourceSymbolId.Value); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - SqliteCommandPolicy.Add(cmd, "@limit", limit); - SqliteCommandPolicy.Add(cmd, "@offset", Math.Max(0, offset)); + if (!_hasReferencesTable) + return new List(); - var results = new List(); - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - var primaryKind = reader.GetString(5); - var kindAggregate = TruncateReferenceKindAggregate(GetNullableString(reader, 10), out var kindsTruncated); - var countAggregate = TruncateReferenceKindAggregate(GetNullableString(reader, 11), out var countsTruncated); - var kinds = ParseDistinctReferenceKinds(kindAggregate, primaryKind); - var counts = ParseReferenceKindCounts(countAggregate, primaryKind, reader.GetInt32(9)); - results.Add(new CalleeResult - { - Path = reader.GetString(0), - Lang = GetNullableString(reader, 1), - CallerKind = GetNullableString(reader, 2), - CallerName = GetNullableString(reader, 3), - CalleeName = reader.GetString(4), - ReferenceKind = primaryKind, - ReferenceKinds = kinds, - HasMixedReferenceKinds = kinds.Count > 1, - ReferenceKindCounts = counts, - AggregateTruncated = kindsTruncated || countsTruncated, - ReferenceWeightScore = reader.GetDouble(12), - FirstLine = reader.GetInt32(6), - FirstColumn = GetNullableInt32(reader, 7), - FirstLength = GetNullableInt32(reader, 8), - ReferenceCount = reader.GetInt32(9), - }); - } - return results; + var request = CreateGraphReferenceQueryRequest( + query, + limit, + lang, + referenceKind, + pathPatterns, + excludePathPatterns, + excludeTests, + exact, + rawKinds, + includeQualifiedCommonCalls, + includeMemberReads, + identitySymbolId: sourceSymbolId, + offset: offset); + var plan = BuildGraphReferenceQueryPlan( + CalleeGraphReferenceDirection, + request, + GraphReferenceQueryShape.List, + rankMode); + return ExecuteGraphReferenceList(plan, ProjectCalleeResult); } public int CountCallees(string query, int limit = 20, string? lang = null, string? referenceKind = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, bool exact = false, bool rawKinds = false, bool includeQualifiedCommonCalls = false, bool includeMemberReads = false) @@ -775,103 +255,26 @@ public int CountCallees(string query, int limit = 20, string? lang = null, strin return 0; lang = NormalizeQueryLanguage(lang); query = NormalizeSymbolSearchQuery(query, lang, exact) ?? query ?? string.Empty; - if (!_hasReferencesTable) return 0; - using var cmd = _conn.CreateCommand(); - var groupedSql = @" - SELECT path, lang, container_kind, container_name, symbol_name, reference_kind - FROM ( - SELECT f.path AS path, f.lang AS lang, r.container_kind AS container_kind, - r.container_name AS container_name, r.symbol_name AS symbol_name, - " + (rawKinds ? GetPreferredReferenceKindSql("r.reference_kind") : GetPreferredLogicalReferenceKindSql("r.reference_kind")) + @" AS reference_kind - FROM symbol_references r - JOIN files f ON r.file_id = f.id - WHERE r.container_name IS NOT NULL"; - groupedSql += $" AND {BuildGraphSupportedLanguagePredicate(cmd, "f", "graphLang")}"; - - groupedSql += $" AND {GetCallableReferenceKindPredicateSql("r.reference_kind", referenceKind, "f.lang", includeMemberReads)}"; - var allowSqlLeafFallback = AllowSqlLeafFallbackForQuery(query); - var allowQualifiedLeafFallback = HasSingleQualifiedSymbolDefinition(query, lang, pathPatterns, excludePathPatterns, excludeTests); - var useSqlQualifiedContainerMatch = SqlNameResolver.HasQualifier(query); - var cssScssVariableAlias = ComputeCssScssVariableAlias(query); - var cssScssVariableAliasScope = cssScssVariableAlias != null - ? " AND f.lang = 'css'" - : string.Empty; - if (exact && useSqlQualifiedContainerMatch && _foldReady) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: true); - groupedSql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name_folded(r.container_name) = @aliasQueryNormalizedFolded) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}) OR {qualifiedLeafFallbackSql})"; - } - else if (exact && useSqlQualifiedContainerMatch) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: false); - groupedSql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name(r.container_name) = @aliasQueryNormalized COLLATE NOCASE) OR ((f.lang != 'sql') AND r.container_name = @query COLLATE NOCASE) OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContainerMatch && _foldReady) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: true); - groupedSql += $" AND (r.container_name LIKE @query ESCAPE '\\' OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContainerMatch) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: false); - groupedSql += $" AND (r.container_name LIKE @query ESCAPE '\\' OR {qualifiedLeafFallbackSql})"; - } - else if (exact && _foldReady) - groupedSql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (r.container_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : $" AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}"; - else if (exact) - groupedSql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND (r.container_name = @query COLLATE NOCASE OR (r.container_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))" - : " AND (r.container_name = @query COLLATE NOCASE OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))" - : " AND r.container_name = @query COLLATE NOCASE"; - else - groupedSql += cssScssVariableAlias != null - ? $" AND (r.container_name LIKE @query ESCAPE '\\' OR (r.container_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))" - : " AND (r.container_name LIKE @query ESCAPE '\\' OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))"; - if (lang != null) - groupedSql += " AND f.lang = @lang"; - if (!includeQualifiedCommonCalls) - groupedSql += BuildCSharpQualifiedCommonCallNoiseFilter("f", "r"); - AppendPathFilters(ref groupedSql, pathPatterns, excludePathPatterns, excludeTests); - groupedSql += $" GROUP BY f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.file_id, r.line, r.column_number, {(rawKinds ? GetRawReferenceKindSql("r.reference_kind") : GetLogicalReferenceKindSql("r.reference_kind"))}"; - groupedSql += " ) grouped_call_sites GROUP BY path, lang, container_kind, container_name, symbol_name, reference_kind LIMIT @limit"; - - cmd.CommandText = $"SELECT COUNT(*) FROM ({groupedSql})"; - var value = !exact - ? $"%{EscapeLikeQuery(query)}%" - : _foldReady - ? FoldNameForLanguage(query, lang) - : query; - if (exact && _foldReady) - AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); - else - SqliteCommandPolicy.Add(cmd, "@query", value); - SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); - SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); - SqliteCommandPolicy.Add(cmd, "@aliasQueryNormalized", SqlNameResolver.NormalizeQualifiedName(query)); - SqliteCommandPolicy.Add(cmd, "@aliasQueryNormalizedFolded", NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(query)) ?? SqlNameResolver.NormalizeQualifiedName(query)); - SqliteCommandPolicy.Add(cmd, "@aliasQuerySegmentCount", SqlNameResolver.GetSegmentCount(query)); - AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback); - if (cssScssVariableAlias != null) - { - var aliasParam = exact && _foldReady - ? NameFold.Fold(cssScssVariableAlias) ?? cssScssVariableAlias - : cssScssVariableAlias; - SqliteCommandPolicy.Add(cmd, "@queryCssScssVariableAlias", aliasParam); - } - if (RequiresReferenceKindParameter(referenceKind)) - SqliteCommandPolicy.Add(cmd, "@referenceKind", referenceKind); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - SqliteCommandPolicy.Add(cmd, "@limit", limit); + if (!_hasReferencesTable) + return 0; - var raw = cmd.ExecuteScalar(); - return raw is long l ? (int)l : Convert.ToInt32(raw); + var request = CreateGraphReferenceQueryRequest( + query, + limit, + lang, + referenceKind, + pathPatterns, + excludePathPatterns, + excludeTests, + exact, + rawKinds, + includeQualifiedCommonCalls, + includeMemberReads); + var plan = BuildGraphReferenceQueryPlan( + CalleeGraphReferenceDirection, + request, + GraphReferenceQueryShape.LimitedCount); + return ExecuteGraphReferenceLimitedCount(plan); } public QueryCountResult CountCalleesTotal(string query, string? lang = null, string? referenceKind = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, bool exact = false, bool rawKinds = false, bool includeQualifiedCommonCalls = false, bool includeMemberReads = false) @@ -905,168 +308,24 @@ private QueryCountResult CountCalleesTotalCore( if (!_hasReferencesTable) return new QueryCountResult(0, 0); - using var cmd = _conn.CreateCommand(); - var groupedSql = @" - SELECT path, lang - FROM ( - SELECT f.path AS path, f.lang AS lang, r.container_kind AS container_kind, - r.container_name AS container_name, r.symbol_name AS symbol_name, - " + (rawKinds ? GetPreferredReferenceKindSql("r.reference_kind") : GetPreferredLogicalReferenceKindSql("r.reference_kind")) + @" AS reference_kind - FROM symbol_references r - JOIN files f ON r.file_id = f.id - WHERE r.container_name IS NOT NULL"; - groupedSql += $" AND {BuildGraphSupportedLanguagePredicate(cmd, "f", "graphLang")}"; - - groupedSql += $" AND {GetCallableReferenceKindPredicateSql("r.reference_kind", referenceKind, "f.lang", includeMemberReads)}"; - if (sourceSymbolId != null && _referenceColumns.Contains("source_symbol_id")) - groupedSql += " AND r.source_symbol_id = @sourceSymbolId"; - var allowSqlLeafFallback = AllowSqlLeafFallbackForQuery(query); - var allowQualifiedLeafFallback = HasSingleQualifiedSymbolDefinition(query, lang, pathPatterns, excludePathPatterns, excludeTests); - var useSqlQualifiedContainerMatch = SqlNameResolver.HasQualifier(query); - var cssScssVariableAlias = ComputeCssScssVariableAlias(query); - var cssScssVariableAliasScope = cssScssVariableAlias != null - ? " AND f.lang = 'css'" - : string.Empty; - if (exact && useSqlQualifiedContainerMatch && _foldReady) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: true); - groupedSql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name_folded(r.container_name) = @aliasQueryNormalizedFolded) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}) OR {qualifiedLeafFallbackSql})"; - } - else if (exact && useSqlQualifiedContainerMatch) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: false); - groupedSql += $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name(r.container_name) = @aliasQueryNormalized COLLATE NOCASE) OR ((f.lang != 'sql') AND r.container_name = @query COLLATE NOCASE) OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContainerMatch && _foldReady) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: true); - groupedSql += $" AND (r.container_name LIKE @query ESCAPE '\\' OR {qualifiedLeafFallbackSql})"; - } - else if (useSqlQualifiedContainerMatch) - { - var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.container_name", "r.container_name_folded", folded: false); - groupedSql += $" AND (r.container_name LIKE @query ESCAPE '\\' OR {qualifiedLeafFallbackSql})"; - } - else if (exact && _foldReady) - groupedSql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (r.container_name_folded = @queryCssScssVariableAlias{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : $" AND ({BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")} OR (f.lang = 'sql' AND sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded))" - : $" AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}"; - else if (exact) - groupedSql += allowSqlLeafFallback - ? cssScssVariableAlias != null - ? $" AND (r.container_name = @query COLLATE NOCASE OR (r.container_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))" - : " AND (r.container_name = @query COLLATE NOCASE OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))" - : " AND r.container_name = @query COLLATE NOCASE"; - else - groupedSql += cssScssVariableAlias != null - ? $" AND (r.container_name LIKE @query ESCAPE '\\' OR (r.container_name = @queryCssScssVariableAlias COLLATE NOCASE{cssScssVariableAliasScope}) OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))" - : " AND (r.container_name LIKE @query ESCAPE '\\' OR (f.lang = 'sql' AND sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE))"; - if (lang != null) - groupedSql += " AND f.lang = @lang"; - if (!includeQualifiedCommonCalls) - groupedSql += BuildCSharpQualifiedCommonCallNoiseFilter("f", "r"); - AppendPathFilters(ref groupedSql, pathPatterns, excludePathPatterns, excludeTests); - groupedSql += $" GROUP BY f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.file_id, r.line, r.column_number, {(rawKinds ? GetRawReferenceKindSql("r.reference_kind") : GetLogicalReferenceKindSql("r.reference_kind"))}"; - groupedSql += " ) grouped_call_sites GROUP BY path, lang, container_kind, container_name, symbol_name, reference_kind"; - - cmd.CommandText = $"SELECT COUNT(*), COUNT(DISTINCT path), MAX(CASE WHEN lang = 'sql' THEN 1 ELSE 0 END) FROM ({groupedSql})"; - var value = !exact - ? $"%{EscapeLikeQuery(query)}%" - : _foldReady - ? FoldNameForLanguage(query, lang) - : query; - if (exact && _foldReady) - AddPersistedFoldedNameQueryParameters(cmd, "@query", query, lang); - else - SqliteCommandPolicy.Add(cmd, "@query", value); - SqliteCommandPolicy.Add(cmd, "@aliasQuery", query); - SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(query)) ?? SqlNameResolver.GetLeafName(query)); - SqliteCommandPolicy.Add(cmd, "@aliasQueryNormalized", SqlNameResolver.NormalizeQualifiedName(query)); - SqliteCommandPolicy.Add(cmd, "@aliasQueryNormalizedFolded", NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(query)) ?? SqlNameResolver.NormalizeQualifiedName(query)); - SqliteCommandPolicy.Add(cmd, "@aliasQuerySegmentCount", SqlNameResolver.GetSegmentCount(query)); - AddQualifiedGraphQueryParameters(cmd, query, allowQualifiedLeafFallback); - if (cssScssVariableAlias != null) - { - var aliasParam = exact && _foldReady - ? NameFold.Fold(cssScssVariableAlias) ?? cssScssVariableAlias - : cssScssVariableAlias; - SqliteCommandPolicy.Add(cmd, "@queryCssScssVariableAlias", aliasParam); - } - if (RequiresReferenceKindParameter(referenceKind)) - SqliteCommandPolicy.Add(cmd, "@referenceKind", referenceKind); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - if (sourceSymbolId != null) - SqliteCommandPolicy.Add(cmd, "@sourceSymbolId", sourceSymbolId.Value); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - - return ExecuteCountSummary(cmd); - } - - private static string ReferenceWeightedScoreSql(string columnSql) => $@" - SUM(CASE {columnSql} - WHEN 'instantiate' THEN 3.0 - WHEN 'generic_type_argument' THEN 0.5 - WHEN 'call' THEN 1.0 - WHEN 'subscribe' THEN 0.1 - WHEN 'unsubscribe' THEN 0.1 - WHEN 'razor_event_binding' THEN 0.1 - ELSE 0.0 - END)"; - - private static string BuildReferenceRankOrderSql( - ReferenceRankMode rankMode, - string queriedNameSql) - => string.Join( - ", ", - ReferenceRankRecipes.Get(rankMode).Select(dimension => dimension switch - { - ReferenceRankDimension.ReferenceWeightScoreDescending => "weighted_score DESC", - ReferenceRankDimension.ReferenceCountDescending => "reference_count DESC", - ReferenceRankDimension.ReferenceKindPriorityAscending => - "CASE reference_kind WHEN 'instantiate' THEN 0 WHEN 'call' THEN 1 WHEN 'generic_type_argument' THEN 2 WHEN 'subscribe' THEN 3 ELSE 4 END", - ReferenceRankDimension.ExactCaseMatchDescending => - $"CASE WHEN {queriedNameSql} = @rawQuery THEN 0 ELSE 1 END", - ReferenceRankDimension.ExactNameMatchDescending => - $"CASE WHEN lower({queriedNameSql}) = lower(@rankingQuery) THEN 0 ELSE 1 END", - ReferenceRankDimension.PathCategoryAscending => GetPathBucketOrderSql("r.path"), - ReferenceRankDimension.PathAscending => "r.path", - ReferenceRankDimension.FirstLineAscending => "first_line", - ReferenceRankDimension.FirstColumnAscending => "first_column", - ReferenceRankDimension.LanguageAscending => "r.lang", - ReferenceRankDimension.ContainerKindAscending => "r.container_kind", - ReferenceRankDimension.ContainerNameAscending => "r.container_name", - ReferenceRankDimension.SymbolNameAscending => "r.symbol_name", - ReferenceRankDimension.ReferenceKindAscending => "reference_kind", - _ => throw new ArgumentOutOfRangeException(nameof(dimension), dimension, null), - })); - - private static IReadOnlyDictionary ParseReferenceKindCounts(string? aggregate, string primaryKind, int fallbackCount) - { - var counts = new Dictionary(StringComparer.Ordinal); - counts["call"] = 0; - counts["instantiate"] = 0; - counts["subscribe"] = 0; - if (!string.IsNullOrWhiteSpace(aggregate)) - { - foreach (var entry in aggregate.Split(',')) - { - var separator = entry.LastIndexOf(':'); - if (separator <= 0 || separator == entry.Length - 1) - continue; - var kind = entry[..separator].Trim(); - if (kind.Length == 0 || !int.TryParse(entry[(separator + 1)..], out var count)) - continue; - counts[kind] = counts.TryGetValue(kind, out var existing) - ? existing + count - : count; - } - } - if (counts.Count == 0 && !string.IsNullOrEmpty(primaryKind)) - counts[primaryKind] = fallbackCount; - return counts; + var request = CreateGraphReferenceQueryRequest( + query, + limit: 0, + lang, + referenceKind, + pathPatterns, + excludePathPatterns, + excludeTests, + exact, + rawKinds, + includeQualifiedCommonCalls, + includeMemberReads, + identitySymbolId: sourceSymbolId); + var plan = BuildGraphReferenceQueryPlan( + CalleeGraphReferenceDirection, + request, + GraphReferenceQueryShape.TotalCount); + return ExecuteGraphReferenceTotalCount(plan); } /// @@ -1140,15 +399,6 @@ ELSE 5 /// SQL 側で要求された LIMIT/OFFSET を適用し、呼び出し側が要求以上の中間ページを /// materialize しないようにする。 /// - private List GetCallersExact(string symbolName, int limit, int offset = 0, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, bool includeAmbiguousMSource = false, bool includeMemberReads = false) - => GetCallersExactCore(symbolName, limit, offset, lang, pathPatterns, excludePathPatterns, excludeTests, targetSymbolIds: null, includeAmbiguousMSource, includeMemberReads); - - private List GetCallersExactForTarget(string symbolName, long targetSymbolId, int limit, int offset, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, bool includeAmbiguousMSource = false, bool includeMemberReads = false) - => GetCallersExactCore(symbolName, limit, offset, lang, pathPatterns, excludePathPatterns, excludeTests, [targetSymbolId], includeAmbiguousMSource, includeMemberReads); - - private List GetCallersExactForTargets(string symbolName, IReadOnlyList targetSymbolIds, int limit, int offset, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, bool includeAmbiguousMSource = false, bool includeMemberReads = false) - => GetCallersExactCore(symbolName, limit, offset, lang, pathPatterns, excludePathPatterns, excludeTests, targetSymbolIds, includeAmbiguousMSource, includeMemberReads); - private List GetCallersExactCore(string symbolName, int limit, int offset, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, IReadOnlyList? targetSymbolIds, bool includeAmbiguousMSource, bool includeMemberReads) { if (!_hasReferencesTable) return new List(); @@ -1222,1189 +472,135 @@ ELSE NULL ? $@" AND ( EXISTS ( - SELECT 1 - FROM symbol_reference_candidates identity_candidate - WHERE identity_candidate.reference_id = r.id - AND identity_candidate.symbol_id IN ({targetSymbolIdsSql}) - AND r.resolution_state IN ('resolved', 'resolved_group') - ) - OR ( - COALESCE(r.resolution_state, 'unresolved') NOT IN ('resolved', 'resolved_group') - " + nameCondition + @" - )" + unscopedPolymorphicNameCondition + @" - )" - : nameCondition; - // impact BFS must share the call-graph contract with `callers`/`callees`/`hotspots`, - // so event subscriptions (`Click += OnClick`) also participate in the transitive - // caller chain. Metadata edges (`attribute`, `annotation`) stay excluded. - // impact の BFS は `callers`/`callees`/`hotspots` と同じ call-graph 契約を共有し、 - // `subscribe` エッジ（`Click += OnClick` 等）も推移 caller に含める。`attribute` / - // `annotation` のような metadata エッジは引き続き除外する。 - var callerContainerPredicate = BuildCallerContainerPredicate("f", "r"); - var sql = $@" - WITH logical_references AS ( - SELECT f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.reference_kind, r.line, - {sourceSymbolIdSql} AS source_symbol_id, - {targetSymbolIdSql} AS target_symbol_id, - MAX({selfReferenceSql}) AS is_self_reference, - MAX({mutualRecursionSql}) AS is_mutual_recursion - FROM symbol_references r - JOIN files f ON r.file_id = f.id{referenceLineJoin} - WHERE {callerContainerPredicate} - AND (r.reference_kind IN {CallGraphReferenceKindsSql}{(includeMemberReads ? " OR r.reference_kind = 'member_read'" : string.Empty)}) - AND {supportedLangFilter} - {targetCondition}"; - if (lang != null) - { - sql += includeAmbiguousMSource - ? " AND (f.lang = @lang OR f.lang = 'ambiguous_m')" - : " AND f.lang = @lang"; - } - sql += BuildCSharpBareMemberReferenceFilter( - symbolName, - lang, - "f", - "r", - includeQualifiedCommonCalls: false); - AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); - sql += @" - GROUP BY f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.reference_kind, r.file_id, r.line, r.column_number, source_symbol_id, target_symbol_id - ) - SELECT path, lang, " + BuildCallerKindProjectionSql("r") + @" AS container_kind, - CASE WHEN lang = 'solution' AND reference_kind = 'project_reference' THEN path - ELSE " + BuildCallerNameProjectionSql("r") + @" END AS container_name, - symbol_name, - reference_kind, MIN(line) AS first_line, COUNT(*) AS reference_count, - MAX(is_self_reference) AS is_self_reference, - MAX(is_mutual_recursion) AS is_mutual_recursion, - source_symbol_id, - CASE - WHEN COUNT(DISTINCT COALESCE(target_symbol_id, -1)) = 1 - THEN MIN(target_symbol_id) - ELSE NULL - END AS target_symbol_id, - GROUP_CONCAT(DISTINCT target_symbol_id) AS target_symbol_ids - FROM logical_references r - GROUP BY path, lang, container_kind, container_name, symbol_name, reference_kind, source_symbol_id"; - sql += $" ORDER BY {GetPathBucketOrderSql("r.path")}, reference_count DESC, r.path, COALESCE(r.container_name, ''), COALESCE(r.container_kind, ''), r.symbol_name, reference_kind, first_line, COALESCE(source_symbol_id, -1) LIMIT @limit OFFSET @offset"; - - cmd.CommandText = sql; - SqliteCommandPolicy.Add(cmd, "@symbolName", symbolName); - SqliteCommandPolicy.Add(cmd, "@aliasQuery", symbolName); - AddQualifiedGraphQueryParameters(cmd, symbolName, allowQualifiedLeafFallback, allowCSharpQualifiedContextMatch); - SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(symbolName)) ?? SqlNameResolver.GetLeafName(symbolName)); - SqliteCommandPolicy.Add(cmd, "@symbolNameLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(symbolName)) ?? SqlNameResolver.GetLeafName(symbolName)); - if (_foldReady) - AddPersistedFoldedNameQueryParameters(cmd, "@symbolNameFolded", symbolName, lang); - for (var i = 0; i < polymorphicCSharpSymbolNames.Count; i++) - { - if (_foldReady) - SqliteCommandPolicy.Add(cmd, $"@polymorphicSymbolNameFolded{i}", NameFold.Fold(polymorphicCSharpSymbolNames[i]) ?? polymorphicCSharpSymbolNames[i]); - else - SqliteCommandPolicy.Add(cmd, $"@polymorphicSymbolName{i}", polymorphicCSharpSymbolNames[i]); - } - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - if (hasIdentityTargetScope) - { - var targetSymbolIdValues = targetSymbolIds! - .Select(static symbolId => symbolId.ToString(System.Globalization.CultureInfo.InvariantCulture)) - .ToList(); - SqliteCommandPolicy.Add(cmd, "@targetSymbolIdsJson", JsonStringListCodec.Serialize(targetSymbolIdValues)); - } - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - SqliteCommandPolicy.Add(cmd, "@limit", limit); - SqliteCommandPolicy.Add(cmd, "@offset", offset); - - var results = new List(); - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - results.Add(new CallerResult - { - Path = reader.GetString(0), - Lang = GetNullableString(reader, 1), - CallerKind = GetNullableString(reader, 2), - CallerName = GetNullableString(reader, 3), - CalleeName = reader.GetString(4), - ReferenceKind = reader.GetString(5), - ReferenceKinds = [reader.GetString(5)], - ReferenceKindCounts = new Dictionary(StringComparer.Ordinal) - { - [reader.GetString(5)] = reader.GetInt32(7), - }, - FirstLine = reader.GetInt32(6), - ReferenceCount = reader.GetInt32(7), - HasSelfReference = reader.GetInt32(8) != 0, - HasMutualRecursion = reader.GetInt32(9) != 0, - CallerSymbolId = reader.IsDBNull(10) ? null : reader.GetInt64(10), - CalleeSymbolId = reader.IsDBNull(11) ? null : reader.GetInt64(11), - CalleeSymbolIds = reader.IsDBNull(12) - ? Array.Empty() - : reader.GetString(12) - .Split(',', StringSplitOptions.RemoveEmptyEntries) - .Select(long.Parse) - .Order() - .ToArray(), - }); - } - return results; - } - - private static string BuildImpactVisitedKey( - CallerResult caller, - string callerName, - bool useCanonicalIdentity, - bool deduplicateLogicalNodes = false) - { - var identity = useCanonicalIdentity && caller.CallerSymbolId is long callerSymbolId - ? $"id:{callerSymbolId}" - : $"{caller.Path}:{callerName}"; - return deduplicateLogicalNodes ? identity : $"{identity}:{caller.ReferenceKind}"; - } - - private static string BuildImpactTraversalNodeKey(long? symbolId, string name) - => symbolId is long canonicalSymbolId ? $"id:{canonicalSymbolId}" : $"name:{name}"; - - // Per-result cap on the number of distinct shortest paths surfaced by impact --with-paths. - // Each call chain row may carry multiple converging paths from the resolved root through - // distinct intermediates; the cap keeps JSON output bounded for diamond-heavy graphs and - // is signaled by ImpactResult.PathsTruncated when exceeded. - // impact --with-paths が 1 caller につき保持する経路数の上限。ダイヤモンド型で多経路が - // 収束する場合に JSON 膨張を抑える役割があり、超過時は PathsTruncated で通知する。 - private const int DefaultImpactPathsPerResult = 10; - internal const int DefaultImpactGraphStateEntryBudget = 10_000; - internal const int DefaultImpactPartialFamilyMemberBudget = 10_000; - internal int ImpactPartialFamilyMemberBudget { get; set; } = DefaultImpactPartialFamilyMemberBudget; - internal const int ImpactBoundaryCallerProbeBudget = 512; - private const int ImpactBoundaryCallerProbePageSize = 64; - - /// - /// Compute transitive callers of a symbol using BFS with exact matching. - /// Returns each unique caller in the call chain with its depth from the root symbol. - /// The bound is inclusive: when is N, - /// callers at depth 1 through N are returned (so a chain A→B→C→D queried against D with - /// maxDepth: 2 yields C at depth 1 and B at depth 2). Truncation is signaled via the - /// Truncated property in results. When Truncated is true, TruncatedReason distinguishes - /// user_limit (raise --limit) from safety_cap (pathological graph). See Issue #1533. - /// When is true, each ImpactResult is populated with the - /// distinct shortest call paths from the resolved root through any intermediates to that - /// caller (issue #1536); converging diamond chains surface every shortest route up to - /// . - /// 完全一致の BFS でシンボルの推移的呼び出し元を算出。各呼び出し元とルートシンボルからの深さを返す。 - /// は inclusive で、N を指定すると depth 1〜N の caller を返す - /// (例: A→B→C→D のチェーンで D を maxDepth: 2 検索すると C(depth=1) と B(depth=2) を返す)。 - /// 結果が切り詰められた場合は Truncated フラグで通知し、TruncatedReason で - /// user_limit (--limit 到達、緩和で増える) と safety_cap (病的グラフ、--limit 緩和では解消しない) を区別する (#1533)。 - /// を true にすると、各 caller に対してルートからの推移経路 - /// （ダイヤモンド収束時は複数）を 件まで付与する（issue #1536）。 - /// - public (List Results, bool Truncated, string? TruncatedReason, string TerminationReason, List Cycles) GetTransitiveCallers(string symbolName, int maxDepth = 5, int limit = 50, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, bool withPaths = false, int maxPathsPerResult = DefaultImpactPathsPerResult, int resultOffset = 0, bool includeMemberReads = false) - { - // Resolve the symbol name through definitions first so case-mismatched queries - // like "run" find the actual "Run" symbol. Falls back to user input if not found. - // 定義を通じてシンボル名を解決し、"run" → "Run" のようなケース違いを補正する。 - // 見つからなければユーザ入力をフォールバック使用。 - var resolvedName = ResolveSymbolName(symbolName, lang); - var hasResolvedIdentityGraph = _referenceIdentityContractCurrent; - var canResolveQualifiedCSharpIdentity = - hasResolvedIdentityGraph - && SqlNameResolver.HasQualifier(symbolName) - && lang is null or "csharp"; - var rootDefinitionLimit = canResolveQualifiedCSharpIdentity - ? DefaultImpactGraphStateEntryBudget - : limit; - var rootDefinitionResolution = ResolveImpactDefinitions(symbolName, rootDefinitionLimit, lang, pathPatterns, excludePathPatterns, excludeTests); - if (rootDefinitionResolution.Definitions.Count == 0 - && !string.Equals(symbolName, resolvedName, StringComparison.Ordinal)) - { - rootDefinitionResolution = ResolveImpactDefinitions(resolvedName, rootDefinitionLimit, lang, pathPatterns, excludePathPatterns, excludeTests); - } - var rootDefinitions = rootDefinitionResolution.Definitions; - var rootDefinitionPaths = rootDefinitions - .Select(definition => definition.Path) - .ToHashSet(StringComparer.OrdinalIgnoreCase); - var isLogicalPartialFamilyRoot = - hasResolvedIdentityGraph - && rootDefinitionResolution.LogicalCount == 1 - && rootDefinitions.Count == 1 - && rootDefinitions[0].Lang == "csharp" - && rootDefinitions[0].PartialFamilyId != null - && rootDefinitionResolution.PhysicalSymbolIds.Count > 0; - var qualifiedCSharpRootSymbolIds = - (canResolveQualifiedCSharpIdentity || isLogicalPartialFamilyRoot) - && rootDefinitions.Count > 0 - && rootDefinitions.All(definition => definition.Lang == "csharp") - && rootDefinitions.All(definition => definition.SymbolId != null) - && (isLogicalPartialFamilyRoot || rootDefinitionResolution.LogicalCount == rootDefinitions.Count) - ? rootDefinitionResolution.PhysicalSymbolIds.ToHashSet() - : []; - if (hasResolvedIdentityGraph - && rootDefinitionPaths.Count > 1 - && qualifiedCSharpRootSymbolIds.Count == 0) - { - return ([], false, null, ImpactTerminationReasons.Completed, []); - } - var ambiguousMRootSymbolId = hasResolvedIdentityGraph - && rootDefinitions.Count == 1 - && lang is "matlab" or "objc" - && string.Equals(rootDefinitions[0].Lang, lang, StringComparison.Ordinal) - ? rootDefinitions[0].SymbolId - : null; - var identityRootSymbolIds = qualifiedCSharpRootSymbolIds.Count > 0 - ? qualifiedCSharpRootSymbolIds - : ambiguousMRootSymbolId is long ambiguousRootSymbolId - ? [ambiguousRootSymbolId] - : []; - var singleIdentityRootSymbolId = identityRootSymbolIds.Count == 1 - ? identityRootSymbolIds.Single() - : (long?)null; - var includeAmbiguousMSource = ambiguousMRootSymbolId != null; - - var results = new List(); - resultOffset = Math.Max(0, resultOffset); - var resultWindowEnd = checked(resultOffset + limit); - var discoveredResultCount = 0; - var visited = new HashSet(StringComparer.OrdinalIgnoreCase); - var rootTraversalNodeKey = identityRootSymbolIds.Count > 1 - ? $"identity:{NameFold.Fold(symbolName) ?? symbolName}" - : BuildImpactTraversalNodeKey(singleIdentityRootSymbolId, resolvedName); - var queue = new Queue<(string Symbol, long? SymbolId, IReadOnlyList? TargetSymbolIds, string NodeKey, int Depth)>(); - if (isLogicalPartialFamilyRoot) - { - queue.Enqueue((resolvedName, null, identityRootSymbolIds.Order().ToArray(), rootTraversalNodeKey, 0)); - } - else if (identityRootSymbolIds.Count > 0) - { - foreach (var identityRootSymbolId in identityRootSymbolIds.Order()) - queue.Enqueue((resolvedName, identityRootSymbolId, null, rootTraversalNodeKey, 0)); - } - else - { - queue.Enqueue((resolvedName, null, null, rootTraversalNodeKey, 0)); - } - visited.Add(resolvedName); - // A partial-family root cap is reported independently on ImpactAnalysisResult. - // It must not masquerade as a traversal/result cap, because raising --limit does - // not expand the family root and the BFS may otherwise have completed normally. - // partial family の root 上限は ImpactAnalysisResult で独立して報告する。 - // --limit 由来の traversal truncation と混同せず、通常完了した BFS を - // safety_cap 扱いしない。 - var truncated = !isLogicalPartialFamilyRoot - && qualifiedCSharpRootSymbolIds.Count > 0 - && rootDefinitionResolution.PhysicalSymbolIdsTruncated; - var maxDepthReached = false; - var cycles = new List(); - var cycleKeys = new HashSet(StringComparer.Ordinal); - var cycleNodesByKey = new Dictionary(StringComparer.Ordinal); - // truncatedReason tracks the *strongest* signal observed: safety_cap wins over - // user_limit because it tells callers that raising --limit alone will not help - // (the input graph is likely pathological). See Issue #1533. - // truncatedReason は強い方の信号を保持する: safety_cap は --limit を緩和しても解消しない - // ことを示すため、user_limit より優先する (#1533)。 - string? truncatedReason = truncated ? ImpactTruncatedReasons.SafetyCap : null; - // Safety cap to prevent infinite loops on pathological graphs / 病的グラフでの無限ループ防止 - const int maxFetchIterations = 1000; - var graphStateEntryBudget = GetImpactGraphStateEntryBudget(resultWindowEnd); - var graphStateBudgetHit = false; - var boundaryProbeBudgetHit = false; - - // Traversal state uses canonical symbol IDs when they are available. Display names are - // applied only while materializing output, so consecutive same-name symbols remain - // distinct path nodes (issue #4847) while legacy graphs retain name-keyed behavior. - // traversal state は利用可能なら正規 symbol ID をキーにする。表示名への変換は出力時 - // だけ行い、同名 symbol が連続する経路も別ノードとして保持する (#4847)。 - Dictionary> parentsByNodeKey = new(StringComparer.OrdinalIgnoreCase); - Dictionary> cycleParentsByKey = new(StringComparer.Ordinal); - Dictionary depthByNodeKey = new(StringComparer.OrdinalIgnoreCase) - { - [rootTraversalNodeKey] = 0, - }; - var resultIndicesByNodeKey = withPaths - ? new Dictionary>(StringComparer.OrdinalIgnoreCase) - : null; - var resultIndexByVisitedKey = isLogicalPartialFamilyRoot - ? new Dictionary(StringComparer.OrdinalIgnoreCase) - : null; - var pathNodesByKey = withPaths - ? new Dictionary(StringComparer.OrdinalIgnoreCase) - : null; - if (withPaths) - { - var rootPathNode = ResolveImpactPathNode( - resolvedName, - singleIdentityRootSymbolId, - kind: null, - lang, - referencePath: null, - referenceLine: null); - if (isLogicalPartialFamilyRoot) - { - var representative = rootDefinitions[0]; - rootPathNode.SymbolId = null; - rootPathNode.Name = resolvedName; - rootPathNode.Kind = representative.Kind; - rootPathNode.Lang = representative.Lang; - rootPathNode.DefinitionPath = representative.Path; - rootPathNode.DefinitionLine = representative.Line; - rootPathNode.Container = representative.ContainerQualifiedName ?? representative.ContainerName; - rootPathNode.PartialFamilyId = representative.PartialFamilyId; - rootPathNode.LogicalTargetKey = $"partial|{representative.PartialFamilyId}"; - } - pathNodesByKey![rootTraversalNodeKey] = rootPathNode; - } - - while (queue.Count > 0 && discoveredResultCount < resultWindowEnd && !graphStateBudgetHit && !boundaryProbeBudgetHit) - { - var (currentSymbol, currentSymbolId, currentTargetSymbolIds, currentNodeKey, depth) = queue.Dequeue(); - - // Fetch callers in pages, filtering out already-visited before counting toward limit. - // This prevents diamond graphs from hiding reachable callers behind visited duplicates. - // ページングで caller を取得し、visited フィルタ後にカウント。 - // ダイヤモンド型グラフで到達可能な caller が visited 重複に隠れるのを防止。 - var needed = resultWindowEnd - discoveredResultCount; - var pageOffset = 0; - var pageSize = Math.Max(1, needed + 1); - var fetchIterations = 0; - - while (discoveredResultCount < resultWindowEnd && fetchIterations < maxFetchIterations && !graphStateBudgetHit && !boundaryProbeBudgetHit) - { - fetchIterations++; - var page = currentTargetSymbolIds is { Count: > 0 } - ? GetCallersExactForTargets(currentSymbol, currentTargetSymbolIds, pageSize, pageOffset, lang, pathPatterns, excludePathPatterns, excludeTests, includeAmbiguousMSource, includeMemberReads) - : currentSymbolId is long targetSymbolId - ? GetCallersExactForTarget(currentSymbol, targetSymbolId, pageSize, pageOffset, lang, pathPatterns, excludePathPatterns, excludeTests, includeAmbiguousMSource, includeMemberReads) - : GetCallersExact(currentSymbol, pageSize, pageOffset, lang, pathPatterns, excludePathPatterns, excludeTests, includeAmbiguousMSource, includeMemberReads); - - if (page.Count == 0) - break; // No more callers for this symbol / このシンボルの caller は尽きた - - foreach (var caller in page) - { - if (discoveredResultCount >= resultWindowEnd) - { - truncated = true; - truncatedReason ??= ImpactTruncatedReasons.UserLimit; - break; - } - - var callerName = caller.CallerName ?? SyntheticTopLevelCallerName; - var callerSymbolId = hasResolvedIdentityGraph ? caller.CallerSymbolId : null; - var calleeSymbolId = hasResolvedIdentityGraph ? caller.CalleeSymbolId : null; - var cycleEdges = BuildImpactCycleEdges( - caller, - callerName, - currentSymbol, - hasResolvedIdentityGraph, - isLogicalPartialFamilyRoot ? identityRootSymbolIds : null, - rootTraversalNodeKey, - resolvedName); - foreach (var cycleEdge in cycleEdges) - { - RegisterImpactCycleNode(cycleNodesByKey, cycleEdge.Caller); - RegisterImpactCycleNode(cycleNodesByKey, cycleEdge.Callee); - if (IsCycleEdge(cycleEdge.Caller.Key, cycleEdge.Callee.Key, cycleParentsByKey)) - AddImpactCycle(cycles, cycleKeys, BuildCycleMembers(cycleEdge.Caller.Key, cycleEdge.Callee.Key, cycleParentsByKey), cycleNodesByKey); - } - if (IsImpactRootCaller(caller, callerName, resolvedName, rootDefinitionPaths, identityRootSymbolIds)) - continue; - var callerNodeKey = BuildImpactTraversalNodeKey(callerSymbolId, callerName); - var key = BuildImpactVisitedKey( - caller, - callerName, - hasResolvedIdentityGraph, - deduplicateLogicalNodes: isLogicalPartialFamilyRoot); - foreach (var cycleEdge in cycleEdges) - { - if (!cycleParentsByKey.TryGetValue(cycleEdge.Caller.Key, out var cycleParentSet)) - { - cycleParentSet = new HashSet(StringComparer.Ordinal); - cycleParentsByKey[cycleEdge.Caller.Key] = cycleParentSet; - } - cycleParentSet.Add(cycleEdge.Callee.Key); - } - if (ImpactGraphStateEntryCount(parentsByNodeKey, cycleParentsByKey, depthByNodeKey, resultIndicesByNodeKey) > graphStateEntryBudget) - { - graphStateBudgetHit = true; - truncated = true; - truncatedReason = ImpactTruncatedReasons.GraphStateBudget; - break; - } - - if (!visited.Add(key)) - { - if (resultIndexByVisitedKey != null - && resultIndexByVisitedKey.TryGetValue(key, out var existingResultIndex)) - { - MergeImpactReferenceEvidence(results[existingResultIndex], caller); - } - // Same-depth convergence: record the additional parent so path - // enumeration can discover this alternate route. Other-depth re-arrivals - // are intentionally dropped — BFS already keeps the shortest route. - // 同 depth で再到達した場合のみ親辺を追加し、別 depth の到達は破棄。 - // BFS により最短経路だけが残る。 - if (withPaths - && depthByNodeKey.TryGetValue(callerNodeKey, out var existingDepth) - && existingDepth == depth + 1) - { - parentsByNodeKey[callerNodeKey].Add(currentNodeKey); - if (ImpactGraphStateEntryCount(parentsByNodeKey, cycleParentsByKey, depthByNodeKey, resultIndicesByNodeKey) > graphStateEntryBudget) - { - graphStateBudgetHit = true; - truncated = true; - truncatedReason = ImpactTruncatedReasons.GraphStateBudget; - break; - } - } - continue; - } - - var includeInPage = discoveredResultCount >= resultOffset; - var resultIndex = -1; - if (includeInPage) - { - results.Add(new ImpactResult - { - Path = caller.Path, - Lang = caller.Lang, - CallerKind = caller.CallerKind, - CallerName = caller.CallerName, - CalleeName = caller.CalleeName, - CallerSymbolId = callerSymbolId, - CalleeSymbolId = calleeSymbolId, - Depth = depth + 1, - FirstLine = caller.FirstLine, - ReferenceCount = caller.ReferenceCount, - ReferenceKind = caller.ReferenceKind, - ReferenceKinds = caller.ReferenceKinds, - ReferenceKindCounts = caller.ReferenceKindCounts, - }); - resultIndex = results.Count - 1; - resultIndexByVisitedKey?.Add(key, resultIndex); - } - discoveredResultCount++; - - if (withPaths) - { - pathNodesByKey!.TryAdd( - callerNodeKey, - ResolveImpactPathNode( - callerName, - callerSymbolId, - caller.CallerKind, - caller.Lang ?? lang, - caller.Path, - caller.FirstLine)); - if (!depthByNodeKey.ContainsKey(callerNodeKey)) - depthByNodeKey[callerNodeKey] = depth + 1; - if (includeInPage) - { - if (!resultIndicesByNodeKey!.TryGetValue(callerNodeKey, out var idxList)) - { - idxList = new List(); - resultIndicesByNodeKey[callerNodeKey] = idxList; - } - idxList.Add(resultIndex); - } - } - else if (!depthByNodeKey.ContainsKey(callerNodeKey)) - { - depthByNodeKey[callerNodeKey] = depth + 1; - } - if (!parentsByNodeKey.TryGetValue(callerNodeKey, out var parentSet)) - { - parentSet = new HashSet(StringComparer.OrdinalIgnoreCase); - parentsByNodeKey[callerNodeKey] = parentSet; - } - parentSet.Add(currentNodeKey); - if (ImpactGraphStateEntryCount(parentsByNodeKey, cycleParentsByKey, depthByNodeKey, resultIndicesByNodeKey) > graphStateEntryBudget) - { - graphStateBudgetHit = true; - truncated = true; - truncatedReason = ImpactTruncatedReasons.GraphStateBudget; - break; - } - - // Only recurse if the just-added caller (at depth + 1) is strictly below - // maxDepth, so that the next BFS step can reach depth + 2 ≤ maxDepth. - // This keeps the maxDepth bound inclusive of depth = maxDepth results. - // 追加した caller (depth + 1) が maxDepth より小さいときだけ再帰し、 - // 次の BFS で depth + 2 ≤ maxDepth まで到達できるようにする。 - // これにより maxDepth は inclusive な上限として機能する。 - if (caller.CallerName != null - && caller.CallerName != SyntheticTopLevelCallerName - && depth + 1 < maxDepth) - { - queue.Enqueue((caller.CallerName, callerSymbolId, null, callerNodeKey, depth + 1)); - } - else if (caller.CallerName != null - && caller.CallerName != SyntheticTopLevelCallerName - && depth + 1 == maxDepth) - { - var boundaryInspection = InspectBoundaryCallersCore( - caller.CallerName, - callerSymbolId, - resolvedName, - rootDefinitionPaths, - identityRootSymbolIds, - visited, - cycleParentsByKey, - cycleNodesByKey, - cycles, - cycleKeys, - hasResolvedIdentityGraph, - lang, - pathPatterns, - excludePathPatterns, - excludeTests, - includeAmbiguousMSource, - includeMemberReads, - isLogicalPartialFamilyRoot ? identityRootSymbolIds : null, - rootTraversalNodeKey, - resolvedName); - maxDepthReached |= boundaryInspection.HasUnvisitedCaller; - if (boundaryInspection.ProbeBudgetHit) - { - boundaryProbeBudgetHit = true; - truncated = true; - truncatedReason = ImpactTruncatedReasons.BoundaryProbeBudget; - break; - } - } - } - - pageOffset += page.Count; - - // If this page was full, there might be more — continue paging - // ページが満杯なら、まだある可能性 — ページングを継続 - if (page.Count < pageSize) - break; - } - - // If fetch iteration cap was hit, mark as truncated / フェッチ反復上限に達した場合も truncated - if (fetchIterations >= maxFetchIterations) - { - truncated = true; - truncatedReason = ImpactTruncatedReasons.SafetyCap; - } - } - - if (queue.Count > 0 && discoveredResultCount >= resultWindowEnd) - { - truncated = true; - truncatedReason ??= ImpactTruncatedReasons.UserLimit; - } - - if (withPaths) - { - var effectiveCap = maxPathsPerResult > 0 ? maxPathsPerResult : DefaultImpactPathsPerResult; - foreach (var (callerNodeKey, indices) in resultIndicesByNodeKey!) - { - var (pathKeys, more) = EnumerateImpactPaths(callerNodeKey, parentsByNodeKey, rootTraversalNodeKey, effectiveCap); - var paths = pathKeys - .Select(path => path.Select(nodeKey => pathNodesByKey![nodeKey].Name).ToList()) - .ToList(); - foreach (var idx in indices) - { - results[idx].Paths = paths; - results[idx].PathDetails = BuildImpactPathDetails(pathKeys, pathNodesByKey!, results[idx]); - results[idx].PathsTruncated = more; - } - } - } - - var terminationReason = truncatedReason switch - { - ImpactTruncatedReasons.GraphStateBudget => ImpactTerminationReasons.GraphStateBudget, - ImpactTruncatedReasons.BoundaryProbeBudget => ImpactTerminationReasons.BoundaryProbeBudget, - ImpactTruncatedReasons.SafetyCap => ImpactTerminationReasons.SafetyCap, - ImpactTruncatedReasons.UserLimit => ImpactTerminationReasons.RowLimitTruncated, - _ when cycles.Count > 0 => ImpactTerminationReasons.CycleDetected, - _ when maxDepthReached => ImpactTerminationReasons.MaxDepthReached, - _ => ImpactTerminationReasons.Completed, - }; - - return (results, truncated, truncatedReason, terminationReason, cycles); - } - - private static int GetImpactGraphStateEntryBudget(int limit) - { - var limitScaled = Math.Max(1, limit) * 200; - return Math.Max(1024, Math.Min(DefaultImpactGraphStateEntryBudget, limitScaled)); - } - - private static void MergeImpactReferenceEvidence(ImpactResult result, CallerResult caller) - { - var counts = result.ReferenceKindCounts.ToDictionary( - pair => pair.Key, - pair => pair.Value, - StringComparer.Ordinal); - foreach (var (kind, count) in caller.ReferenceKindCounts) - { - counts[kind] = counts.TryGetValue(kind, out var existingCount) - ? Math.Max(existingCount, count) - : count; - } - - result.ReferenceKindCounts = counts; - result.ReferenceKinds = counts.Keys.Order(StringComparer.Ordinal).ToArray(); - result.ReferenceCount = counts.Values.Sum(); - result.FirstLine = Math.Min(result.FirstLine, caller.FirstLine); - } - - private static int ImpactGraphStateEntryCount( - Dictionary> parentsByNodeKey, - Dictionary> cycleParentsByNodeKey, - Dictionary depthByNodeKey, - Dictionary>? resultIndicesByNodeKey) - { - var count = depthByNodeKey.Count + parentsByNodeKey.Count + cycleParentsByNodeKey.Count + (resultIndicesByNodeKey?.Count ?? 0); - foreach (var parents in parentsByNodeKey.Values) - count += parents.Count; - foreach (var parents in cycleParentsByNodeKey.Values) - count += parents.Count; - if (resultIndicesByNodeKey != null) - foreach (var indices in resultIndicesByNodeKey.Values) - count += indices.Count; - return count; - } - - private readonly record struct ImpactBoundaryInspection(bool HasUnvisitedCaller, bool ProbeBudgetHit); - private readonly record struct ImpactCycleNode(string Key, long? SymbolId, string Name); - private readonly record struct ImpactCycleEdge(ImpactCycleNode Caller, ImpactCycleNode Callee); - - private static void RegisterImpactCycleNode( - Dictionary nodesByKey, - ImpactCycleNode node) - => nodesByKey.TryAdd(node.Key, new ImpactCycleMemberResult - { - SymbolId = node.SymbolId, - Name = node.Name, - }); - - private static ImpactCycleNode? BuildImpactCycleNode(long? symbolId, string name, bool hasResolvedIdentityGraph) - { - if (!hasResolvedIdentityGraph) - return new ImpactCycleNode($"name:{NameFold.Fold(name) ?? name}", null, name); - if (symbolId is long canonicalSymbolId) - return new ImpactCycleNode($"id:{canonicalSymbolId}", canonicalSymbolId, name); - return null; - } - - private static List BuildImpactCycleEdges( - CallerResult caller, - string callerName, - string calleeName, - bool hasResolvedIdentityGraph, - IReadOnlySet? logicalRootSymbolIds = null, - string? logicalRootKey = null, - string? logicalRootName = null) - { - var callerNode = NormalizeImpactCycleRootNode( - BuildImpactCycleNode(caller.CallerSymbolId, callerName, hasResolvedIdentityGraph), - logicalRootSymbolIds, - logicalRootKey, - logicalRootName); - if (callerNode is not { } canonicalCaller) - return []; - - if (!hasResolvedIdentityGraph) - { - var legacyCallee = BuildImpactCycleNode(symbolId: null, calleeName, hasResolvedIdentityGraph: false)!.Value; - return [new ImpactCycleEdge(canonicalCaller, legacyCallee)]; - } - - var calleeSymbolIds = caller.CalleeSymbolIds.Count > 0 - ? caller.CalleeSymbolIds - : caller.CalleeSymbolId is long calleeSymbolId - ? [calleeSymbolId] - : Array.Empty(); - return calleeSymbolIds - .Distinct() - .Order() - .Select(calleeSymbolId => new ImpactCycleEdge( - canonicalCaller, - NormalizeImpactCycleRootNode( - new ImpactCycleNode($"id:{calleeSymbolId}", calleeSymbolId, calleeName), - logicalRootSymbolIds, - logicalRootKey, - logicalRootName)!.Value)) - .ToList(); - } - - private static ImpactCycleNode? NormalizeImpactCycleRootNode( - ImpactCycleNode? node, - IReadOnlySet? logicalRootSymbolIds, - string? logicalRootKey, - string? logicalRootName) - { - if (node is not { SymbolId: long symbolId } - || logicalRootSymbolIds is not { Count: > 0 } - || !logicalRootSymbolIds.Contains(symbolId)) - { - return node; - } - - return new ImpactCycleNode( - logicalRootKey ?? "logical-partial-root", - SymbolId: null, - logicalRootName ?? node.Value.Name); - } - - private static bool IsImpactRootCaller( - CallerResult caller, - string callerName, - string resolvedName, - HashSet rootDefinitionPaths, - IReadOnlySet? identityRootSymbolIds) - { - if (identityRootSymbolIds is { Count: > 0 } - && caller.CallerSymbolId is long callerSymbolId) - { - return identityRootSymbolIds.Contains(callerSymbolId); - } - return string.Equals(callerName, resolvedName, StringComparison.OrdinalIgnoreCase) - && (rootDefinitionPaths.Count == 0 || rootDefinitionPaths.Contains(caller.Path)); - } - - private ImpactBoundaryInspection InspectBoundaryCallers( - string symbolName, - long? symbolId, - string resolvedName, - HashSet rootDefinitionPaths, - IReadOnlySet? identityRootSymbolIds, - HashSet visited, - Dictionary> cycleParentsByKey, - Dictionary cycleNodesByKey, - List cycles, - HashSet cycleKeys, - bool hasResolvedIdentityGraph, - string? lang, - IReadOnlyList? pathPatterns, - IReadOnlyList? excludePathPatterns, - bool excludeTests, - bool includeAmbiguousMSource, - bool includeMemberReads) - => InspectBoundaryCallersCore( - symbolName, - symbolId, - resolvedName, - rootDefinitionPaths, - identityRootSymbolIds, - visited, - cycleParentsByKey, - cycleNodesByKey, - cycles, - cycleKeys, - hasResolvedIdentityGraph, - lang, - pathPatterns, - excludePathPatterns, - excludeTests, - includeAmbiguousMSource, - includeMemberReads, - logicalRootSymbolIds: null, - logicalRootKey: null, - logicalRootName: null); - - private ImpactBoundaryInspection InspectBoundaryCallersCore( - string symbolName, - long? symbolId, - string resolvedName, - HashSet rootDefinitionPaths, - IReadOnlySet? identityRootSymbolIds, - HashSet visited, - Dictionary> cycleParentsByKey, - Dictionary cycleNodesByKey, - List cycles, - HashSet cycleKeys, - bool hasResolvedIdentityGraph, - string? lang, - IReadOnlyList? pathPatterns, - IReadOnlyList? excludePathPatterns, - bool excludeTests, - bool includeAmbiguousMSource, - bool includeMemberReads, - IReadOnlySet? logicalRootSymbolIds, - string? logicalRootKey, - string? logicalRootName) - { - var offset = 0; - var probes = 0; - while (true) - { - if (probes >= ImpactBoundaryCallerProbeBudget) - return new ImpactBoundaryInspection(HasUnvisitedCaller: true, ProbeBudgetHit: true); - - var pageSize = Math.Min(ImpactBoundaryCallerProbePageSize, ImpactBoundaryCallerProbeBudget - probes); - var page = symbolId is long targetSymbolId - ? GetCallersExactForTarget(symbolName, targetSymbolId, pageSize, offset, lang, pathPatterns, excludePathPatterns, excludeTests, includeAmbiguousMSource, includeMemberReads) - : GetCallersExact(symbolName, pageSize, offset, lang, pathPatterns, excludePathPatterns, excludeTests, includeAmbiguousMSource, includeMemberReads); - if (page.Count == 0) - return new ImpactBoundaryInspection(HasUnvisitedCaller: false, ProbeBudgetHit: false); - probes += page.Count; - - foreach (var caller in page) - { - var callerName = caller.CallerName ?? SyntheticTopLevelCallerName; - var cycleEdges = BuildImpactCycleEdges( - caller, - callerName, - symbolName, - hasResolvedIdentityGraph, - logicalRootSymbolIds, - logicalRootKey, - logicalRootName); - foreach (var cycleEdge in cycleEdges) - { - RegisterImpactCycleNode(cycleNodesByKey, cycleEdge.Caller); - RegisterImpactCycleNode(cycleNodesByKey, cycleEdge.Callee); - if (IsCycleEdge(cycleEdge.Caller.Key, cycleEdge.Callee.Key, cycleParentsByKey)) - AddImpactCycle(cycles, cycleKeys, BuildCycleMembers(cycleEdge.Caller.Key, cycleEdge.Callee.Key, cycleParentsByKey), cycleNodesByKey); - } - var isRoot = IsImpactRootCaller(caller, callerName, resolvedName, rootDefinitionPaths, identityRootSymbolIds); - if (isRoot) - continue; - - foreach (var cycleEdge in cycleEdges) - { - if (!cycleParentsByKey.TryGetValue(cycleEdge.Caller.Key, out var cycleParentSet)) - { - cycleParentSet = new HashSet(StringComparer.Ordinal); - cycleParentsByKey[cycleEdge.Caller.Key] = cycleParentSet; - } - cycleParentSet.Add(cycleEdge.Callee.Key); - } - - var key = BuildImpactVisitedKey( - caller, - callerName, - hasResolvedIdentityGraph, - deduplicateLogicalNodes: logicalRootSymbolIds is not null); - if (!visited.Contains(key)) - return new ImpactBoundaryInspection(HasUnvisitedCaller: true, ProbeBudgetHit: false); - } - - if (page.Count < pageSize) - return new ImpactBoundaryInspection(HasUnvisitedCaller: false, ProbeBudgetHit: false); - offset += page.Count; - } - } - - private static bool IsCycleEdge( - string callerKey, - string currentKey, - Dictionary> parentsByKey) - { - if (string.Equals(callerKey, currentKey, StringComparison.Ordinal)) - return true; - return HasAncestor(currentKey, callerKey, parentsByKey); - } - - private static bool HasAncestor( - string node, - string target, - Dictionary> parentsByKey) - { - var stack = new Stack(); - var seen = new HashSet(StringComparer.Ordinal); - stack.Push(node); - while (stack.Count > 0) - { - var current = stack.Pop(); - if (!seen.Add(current)) - continue; - if (string.Equals(current, target, StringComparison.Ordinal)) - return true; - if (!parentsByKey.TryGetValue(current, out var parents)) - continue; - foreach (var parent in parents) - stack.Push(parent); - } - return false; - } - - private static List BuildCycleMembers( - string callerKey, - string currentKey, - Dictionary> parentsByKey) - { - var members = new HashSet(StringComparer.Ordinal); - if (!TryBuildAncestorPath(currentKey, callerKey, parentsByKey, members)) + SELECT 1 + FROM symbol_reference_candidates identity_candidate + WHERE identity_candidate.reference_id = r.id + AND identity_candidate.symbol_id IN ({targetSymbolIdsSql}) + AND r.resolution_state IN ('resolved', 'resolved_group') + ) + OR ( + COALESCE(r.resolution_state, 'unresolved') NOT IN ('resolved', 'resolved_group') + " + nameCondition + @" + )" + unscopedPolymorphicNameCondition + @" + )" + : nameCondition; + // impact BFS must share the call-graph contract with `callers`/`callees`/`hotspots`, + // so event subscriptions (`Click += OnClick`) also participate in the transitive + // caller chain. Metadata edges (`attribute`, `annotation`) stay excluded. + // impact の BFS は `callers`/`callees`/`hotspots` と同じ call-graph 契約を共有し、 + // `subscribe` エッジ（`Click += OnClick` 等）も推移 caller に含める。`attribute` / + // `annotation` のような metadata エッジは引き続き除外する。 + var callerContainerPredicate = BuildCallerContainerPredicate("f", "r"); + var sql = $@" + WITH logical_references AS ( + SELECT f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.reference_kind, r.line, + {sourceSymbolIdSql} AS source_symbol_id, + {targetSymbolIdSql} AS target_symbol_id, + MAX({selfReferenceSql}) AS is_self_reference, + MAX({mutualRecursionSql}) AS is_mutual_recursion + FROM symbol_references r + JOIN files f ON r.file_id = f.id{referenceLineJoin} + WHERE {callerContainerPredicate} + AND (r.reference_kind IN {CallGraphReferenceKindsSql}{(includeMemberReads ? " OR r.reference_kind = 'member_read'" : string.Empty)}) + AND {supportedLangFilter} + {targetCondition}"; + if (lang != null) { - members.Add(callerKey); - members.Add(currentKey); + sql += includeAmbiguousMSource + ? " AND (f.lang = @lang OR f.lang = 'ambiguous_m')" + : " AND f.lang = @lang"; } - var result = members.ToList(); - result.Sort(StringComparer.Ordinal); - return result; - } - - private static bool TryBuildAncestorPath( - string node, - string target, - Dictionary> parentsByKey, - HashSet members) - { - var seen = new HashSet(StringComparer.Ordinal); - return TryBuildAncestorPathCore(node, target, parentsByKey, members, seen); - } + sql += BuildCSharpBareMemberReferenceFilter( + symbolName, + lang, + "f", + "r", + includeQualifiedCommonCalls: false); + AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); + sql += @" + GROUP BY f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.reference_kind, r.file_id, r.line, r.column_number, source_symbol_id, target_symbol_id + ) + SELECT path, lang, " + BuildCallerKindProjectionSql("r") + @" AS container_kind, + CASE WHEN lang = 'solution' AND reference_kind = 'project_reference' THEN path + ELSE " + BuildCallerNameProjectionSql("r") + @" END AS container_name, + symbol_name, + reference_kind, MIN(line) AS first_line, COUNT(*) AS reference_count, + MAX(is_self_reference) AS is_self_reference, + MAX(is_mutual_recursion) AS is_mutual_recursion, + source_symbol_id, + CASE + WHEN COUNT(DISTINCT COALESCE(target_symbol_id, -1)) = 1 + THEN MIN(target_symbol_id) + ELSE NULL + END AS target_symbol_id, + GROUP_CONCAT(DISTINCT target_symbol_id) AS target_symbol_ids + FROM logical_references r + GROUP BY path, lang, container_kind, container_name, symbol_name, reference_kind, source_symbol_id"; + sql += $" ORDER BY {GetPathBucketOrderSql("r.path")}, reference_count DESC, r.path, COALESCE(r.container_name, ''), COALESCE(r.container_kind, ''), r.symbol_name, reference_kind, first_line, COALESCE(source_symbol_id, -1) LIMIT @limit OFFSET @offset"; - private static bool TryBuildAncestorPathCore( - string node, - string target, - Dictionary> parentsByKey, - HashSet members, - HashSet seen) - { - if (!seen.Add(node)) - return false; - members.Add(node); - if (string.Equals(node, target, StringComparison.Ordinal)) - return true; - if (parentsByKey.TryGetValue(node, out var parents)) + cmd.CommandText = sql; + SqliteCommandPolicy.Add(cmd, "@symbolName", symbolName); + SqliteCommandPolicy.Add(cmd, "@aliasQuery", symbolName); + AddQualifiedGraphQueryParameters(cmd, symbolName, allowQualifiedLeafFallback, allowCSharpQualifiedContextMatch); + SqliteCommandPolicy.Add(cmd, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(symbolName)) ?? SqlNameResolver.GetLeafName(symbolName)); + SqliteCommandPolicy.Add(cmd, "@symbolNameLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(symbolName)) ?? SqlNameResolver.GetLeafName(symbolName)); + if (_foldReady) + AddPersistedFoldedNameQueryParameters(cmd, "@symbolNameFolded", symbolName, lang); + for (var i = 0; i < polymorphicCSharpSymbolNames.Count; i++) { - foreach (var parent in parents) - { - if (TryBuildAncestorPathCore(parent, target, parentsByKey, members, seen)) - return true; - } + if (_foldReady) + SqliteCommandPolicy.Add(cmd, $"@polymorphicSymbolNameFolded{i}", NameFold.Fold(polymorphicCSharpSymbolNames[i]) ?? polymorphicCSharpSymbolNames[i]); + else + SqliteCommandPolicy.Add(cmd, $"@polymorphicSymbolName{i}", polymorphicCSharpSymbolNames[i]); } - - members.Remove(node); - return false; - } - - private static void AddImpactCycle( - List cycles, - HashSet cycleKeys, - List memberKeys, - IReadOnlyDictionary nodesByKey) - { - if (memberKeys.Count == 0) - return; - var key = string.Join("\u001F", memberKeys); - if (!cycleKeys.Add(key)) - return; - var identities = memberKeys - .Select(memberKey => nodesByKey[memberKey]) - .OrderBy(node => node.Name, StringComparer.OrdinalIgnoreCase) - .ThenBy(node => node.SymbolId) - .Select(node => new ImpactCycleMemberResult - { - SymbolId = node.SymbolId, - Name = node.Name, - }) - .ToList(); - cycles.Add(new ImpactCycleResult - { - Members = identities.Select(identity => identity.Name).ToList(), - MemberIdentities = identities.Any(identity => identity.SymbolId != null) ? identities : null, - }); - } - - private static (List> Paths, bool Truncated) EnumerateImpactPaths( - string callerNodeKey, - Dictionary> parentsByNodeKey, - string resolvedRootNodeKey, - int maxPathsPerResult) - { - // DFS upward through canonical node keys. The Stack enumerator yields top-first, - // so the materialized key path is already ordered [resolvedRoot, ..., caller]. - // 正規 node key の親辺を DFS で辿る。Stack は top-first で列挙されるため、 - // key path はそのまま [resolvedRoot, ..., caller] 順になる。 - var paths = new List>(); - var stack = new Stack(); - var onStack = new HashSet(StringComparer.OrdinalIgnoreCase); - var truncatedRef = new bool[1]; - - stack.Push(callerNodeKey); - onStack.Add(callerNodeKey); - Dfs(callerNodeKey); - stack.Pop(); - onStack.Remove(callerNodeKey); - return (paths, truncatedRef[0]); - - void Dfs(string node) + if (lang != null) + SqliteCommandPolicy.Add(cmd, "@lang", lang); + if (hasIdentityTargetScope) { - if (string.Equals(node, resolvedRootNodeKey, StringComparison.OrdinalIgnoreCase)) - { - paths.Add(stack.ToList()); - return; - } - if (!parentsByNodeKey.TryGetValue(node, out var parents)) - return; - foreach (var p in parents) - { - if (onStack.Contains(p)) - continue; - // Only mark truncated when the cap forces us to *skip* a still-unexplored parent; - // hitting cap exactly as the foreach drains naturally is not a truncation. - // 残りの parent を探索できなくなった瞬間にのみ truncated を立てる。foreach が - // 自然に終わるタイミングと一致しただけでは truncation 扱いしない。 - if (paths.Count >= maxPathsPerResult) - { - truncatedRef[0] = true; - return; - } - stack.Push(p); - onStack.Add(p); - Dfs(p); - stack.Pop(); - onStack.Remove(p); - } + var targetSymbolIdValues = targetSymbolIds! + .Select(static symbolId => symbolId.ToString(System.Globalization.CultureInfo.InvariantCulture)) + .ToList(); + SqliteCommandPolicy.Add(cmd, "@targetSymbolIdsJson", JsonStringListCodec.Serialize(targetSymbolIdValues)); } - } - - private ImpactPathNode ResolveImpactPathNode(string name, long? symbolId, string? kind, string? lang, string? referencePath, int? referenceLine) - { - var node = TryResolveImpactPathNodeDefinition(name, symbolId, kind, lang, referencePath) - ?? new ImpactPathNode - { - SymbolId = symbolId, - Name = name, - Kind = kind, - Lang = lang, - }; - node.ReferencePath = referencePath; - node.ReferenceLine = referenceLine; - return node; - } - - private ImpactPathNode? TryResolveImpactPathNodeDefinition(string name, long? symbolId, string? kind, string? lang, string? preferredPath) - { - if (!_symbolColumns.Contains("name") || !_symbolColumns.Contains("kind")) - return null; - - using var cmd = _conn.CreateCommand(); - var containerNameSql = GetSymbolColumnSql("container_name"); - var containerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name"); - var familyKeySql = GetSymbolColumnSql("family_key"); - var namePredicate = _foldReady && _symbolColumns.Contains("name_folded") - ? "s.name_folded = @nameFolded" - : "s.name = @name COLLATE NOCASE"; - - cmd.CommandText = $@" - SELECT s.id, - f.path, - f.lang, - s.kind, - s.name, - s.line, - {containerNameSql} AS container_name, - {containerQualifiedNameSql} AS container_qualified_name, - {familyKeySql} AS family_key, - s.file_id, - COUNT(*) OVER () AS matching_definition_count - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE ((@symbolId IS NOT NULL AND s.id = @symbolId) - OR (@symbolId IS NULL AND {namePredicate})) - AND s.kind NOT IN ('import', 'namespace') - AND (@kind IS NULL OR s.kind = @kind) - AND (@lang IS NULL OR f.lang = @lang) - ORDER BY CASE WHEN @preferredPath IS NOT NULL AND f.path = @preferredPath THEN 0 ELSE 1 END, - f.path, - s.line - LIMIT 1"; - SqliteCommandPolicy.Add(cmd, "@name", name); - SqliteCommandPolicy.AddNullableInt64(cmd, "@symbolId", symbolId); - if (_foldReady && _symbolColumns.Contains("name_folded")) - SqliteCommandPolicy.Add(cmd, "@nameFolded", NameFold.Fold(name) ?? name); - SqliteCommandPolicy.AddNullableText(cmd, "@kind", kind); - SqliteCommandPolicy.AddNullableText(cmd, "@lang", lang); - SqliteCommandPolicy.AddNullableText(cmd, "@preferredPath", preferredPath); + AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); + SqliteCommandPolicy.Add(cmd, "@limit", limit); + SqliteCommandPolicy.Add(cmd, "@offset", offset); + var results = new List(); using var reader = cmd.ExecuteTrackedReader(); - if (!reader.TrackedRead()) - return null; - - var definitionSymbolId = reader.GetInt64(0); - var definitionPath = reader.GetString(1); - var definitionLang = GetNullableString(reader, 2); - var definitionKind = reader.GetString(3); - var definitionName = reader.GetString(4); - var definitionLine = reader.IsDBNull(5) ? (int?)null : reader.GetInt32(5); - var containerName = GetNullableString(reader, 6); - var containerQualifiedName = GetNullableString(reader, 7); - var familyKey = GetNullableString(reader, 8); - var fileId = reader.GetInt64(9); - var matchingDefinitionCount = reader.GetInt64(10); - - return new ImpactPathNode - { - SymbolId = symbolId ?? (matchingDefinitionCount == 1 ? definitionSymbolId : null), - Name = definitionName, - Kind = definitionKind, - Lang = definitionLang, - DefinitionPath = definitionPath, - DefinitionLine = definitionLine, - Container = containerName, - FamilyKey = familyKey, - LogicalTargetKey = BuildImpactPathLogicalTargetKey(definitionLang, definitionKind, familyKey, containerQualifiedName, fileId), - }; - } - - private static string BuildImpactPathLogicalTargetKey(string? lang, string kind, string? familyKey, string? containerQualifiedName, long fileId) - { - if (!string.IsNullOrWhiteSpace(familyKey)) - return $"family|{lang ?? string.Empty}|{kind}|{familyKey}"; - if (!string.IsNullOrWhiteSpace(containerQualifiedName)) - return $"container|{fileId}|{kind}|{containerQualifiedName}"; - return $"file|{fileId}"; - } - - private static List> BuildImpactPathDetails( - List> pathKeys, - IReadOnlyDictionary nodesByKey, - ImpactResult result) - { - var details = new List>(pathKeys.Count); - foreach (var path in pathKeys) + while (reader.TrackedRead()) { - var detailPath = new List(path.Count); - for (var i = 0; i < path.Count; i++) + results.Add(new CallerResult { - var nodeKey = path[i]; - var isResultNode = i == path.Count - 1; - if (!nodesByKey.TryGetValue(nodeKey, out var node)) - node = new ImpactPathNode { Name = nodeKey }; - detailPath.Add(isResultNode - ? CloneImpactPathNodeForResult(node, result) - : CloneImpactPathNode(node)); - } - details.Add(detailPath); + Path = reader.GetString(0), + Lang = GetNullableString(reader, 1), + CallerKind = GetNullableString(reader, 2), + CallerName = GetNullableString(reader, 3), + CalleeName = reader.GetString(4), + ReferenceKind = reader.GetString(5), + ReferenceKinds = [reader.GetString(5)], + ReferenceKindCounts = new Dictionary(StringComparer.Ordinal) + { + [reader.GetString(5)] = reader.GetInt32(7), + }, + FirstLine = reader.GetInt32(6), + ReferenceCount = reader.GetInt32(7), + HasSelfReference = reader.GetInt32(8) != 0, + HasMutualRecursion = reader.GetInt32(9) != 0, + CallerSymbolId = reader.IsDBNull(10) ? null : reader.GetInt64(10), + CalleeSymbolId = reader.IsDBNull(11) ? null : reader.GetInt64(11), + CalleeSymbolIds = reader.IsDBNull(12) + ? Array.Empty() + : reader.GetString(12) + .Split(',', StringSplitOptions.RemoveEmptyEntries) + .Select(long.Parse) + .Order() + .ToArray(), + }); } - return details; - } - - private static ImpactPathNode CloneImpactPathNodeForResult(ImpactPathNode node, ImpactResult result) - { - var clone = CloneImpactPathNode(node); - clone.Kind ??= result.CallerKind; - clone.Lang ??= result.Lang; - clone.ReferencePath = result.Path; - clone.ReferenceLine = result.FirstLine; - return clone; + return results; } - private static ImpactPathNode CloneImpactPathNode(ImpactPathNode node) - => new() - { - SymbolId = node.SymbolId, - Name = node.Name, - Kind = node.Kind, - Lang = node.Lang, - DefinitionPath = node.DefinitionPath, - DefinitionLine = node.DefinitionLine, - Container = node.Container, - FamilyKey = node.FamilyKey, - PartialFamilyId = node.PartialFamilyId, - LogicalTargetKey = node.LogicalTargetKey, - ReferencePath = node.ReferencePath, - ReferenceLine = node.ReferenceLine, - }; - /// /// Analyze impact for a query by combining transitive callers with symbol-resolution /// metadata and a class-like file-dependency fallback when symbol-level callers are absent. @@ -2691,468 +887,6 @@ public ImpactAnalysisResult AnalyzeImpact(string symbolName, int maxDepth = 5, i }; } - private sealed record ImpactDefinitionResolution( - List Definitions, - int PhysicalCount, - int PhysicalFileCount, - int LogicalCount, - int PreciseDefinitionCount, - int PreciseLogicalDefinitionCount, - int PreciseDefinitionFileCount, - int NonCallableDefinitionCount, - SymbolResult? SinglePreciseDefinition, - HashSet PhysicalSymbolIds, - HashSet PhysicalDefinitionPaths, - bool PhysicalSymbolIdsTruncated); - - private ImpactDefinitionResolution ResolveImpactDefinitions( - string resolvedName, - int representativeLimit, - string? lang, - IReadOnlyList? pathPatterns, - IReadOnlyList? excludePathPatterns, - bool excludeTests, - int representativeOffset = 0) - { - var normalizedName = SqlNameResolver.NormalizeQualifiedName(resolvedName); - var leafName = SqlNameResolver.GetLeafName(resolvedName); - var segmentCount = SqlNameResolver.GetSegmentCount(resolvedName); - var allowLeafFallback = !SqlNameResolver.HasQualifier(resolvedName); - EnsureCSharpCallableTypeKinds(lang, [leafName], exact: true); - using var cmd = _conn.CreateCommand(); - var supportedLangFilter = BuildGraphSupportedLanguagePredicate(cmd, "f", "impactDefLang"); - var signatureSql = GetSymbolColumnSql("signature"); - var returnTypeSql = GetSymbolColumnSql("return_type"); - var bodyStartLineSql = GetSymbolColumnSql("body_start_line"); - var bodyEndLineSql = GetSymbolColumnSql("body_end_line"); - var startColumnSql = GetSymbolColumnSql("start_column"); - var identifierStartColumnSql = GetSymbolColumnSql("identifier_start_column"); - var logicalPartialKeySql = LogicalPartialSymbolGrouper.BuildSqlKeyExpression( - "f.lang", - "s.kind", - "s.name", - "s.id", - "f.path", - signatureSql, - GetSymbolColumnSql("container_name"), - GetSymbolColumnSql("container_qualified_name"), - GetSymbolColumnSql("family_key"), - returnTypeSql, - GetSymbolColumnSql("is_partial_declaration"), - _hotspotFamilyReadyLanguages.Contains("csharp")); - var generatedSql = _fileColumns.Contains("generated") - ? "CASE WHEN COALESCE(f.generated, 0) <> 0 OR codeindex_generated_file_name(f.path) THEN 1 ELSE 0 END" - : "CASE WHEN codeindex_generated_file_name(f.path) THEN 1 ELSE 0 END"; - var canonicalPrimaryRankSql = LogicalPartialSymbolGrouper.BuildSqlPrimaryRankExpression( - "s.kind", - bodyStartLineSql, - bodyEndLineSql); - var canonicalSemanticScoreSql = LogicalPartialSymbolGrouper.BuildSqlSemanticScoreExpression( - signatureSql, - "s.kind", - GetSymbolColumnSql("declaration_semantic_score")); - var fallbackCanonicalDeclarationIdentitySql = BuildCanonicalDeclarationIdentitySql(signatureSql); - var canonicalDeclarationIdentitySql = $"CASE WHEN s.kind IN ('function', 'test.method') THEN COALESCE(csharp_partial_callable_identity({signatureSql}, s.name, {returnTypeSql}), {fallbackCanonicalDeclarationIdentitySql}) ELSE {fallbackCanonicalDeclarationIdentitySql} END"; - var nameCondition = _foldReady - ? allowLeafFallback - ? $"({BuildPersistedFoldedNameMatchSql("s.name_folded", "@resolvedNameFolded")} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name_folded(s.name) = @resolvedNameNormalizedFolded) OR sql_leaf_name_folded(s.name) = @resolvedNameLeafFolded)))" - : $"({BuildPersistedFoldedNameMatchSql("s.name_folded", "@resolvedNameFolded")} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name_folded(s.name) = @resolvedNameNormalizedFolded))" - : allowLeafFallback - ? "(s.name = @resolvedName COLLATE NOCASE OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name(s.name) = @resolvedNameNormalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @resolvedNameLeaf COLLATE NOCASE)))" - : "(s.name = @resolvedName COLLATE NOCASE OR (f.lang = 'sql' AND sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name(s.name) = @resolvedNameNormalized COLLATE NOCASE))"; - var csharpExplicitInterfaceClause = allowLeafFallback - ? BuildCSharpExplicitInterfaceShortAliasMatchSql("resolvedName") - : BuildCSharpExplicitInterfaceIdentityMatchSql("resolvedName"); - nameCondition = $"({nameCondition} OR {csharpExplicitInterfaceClause})"; - if (SqlNameResolver.HasQualifier(resolvedName)) - { - var containerNameSql = GetSymbolColumnSql("container_name", "''"); - var containerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name", containerNameSql); - var csharpLeafCondition = _foldReady - ? "s.name_folded = @resolvedNameLeafFolded" - : "s.name = @resolvedNameLeaf COLLATE NOCASE"; - nameCondition = $"({nameCondition} OR (f.lang = 'csharp' AND {csharpLeafCondition} AND ({containerNameSql} = @resolvedNameContainer COLLATE NOCASE OR {containerQualifiedNameSql} = @resolvedNameContainer COLLATE NOCASE OR {containerQualifiedNameSql} COLLATE NOCASE LIKE @resolvedNameContainerSuffix ESCAPE '\\')))"; - } - var matchOrderSql = @"CASE - WHEN s.name = @resolvedName THEN 0 - WHEN f.lang = 'sql' AND sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name(s.name) = @resolvedNameNormalized THEN 1 - WHEN f.lang = 'sql' AND sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name_folded(s.name) = @resolvedNameNormalizedFolded THEN 2 - WHEN @allowLeafFallback = 1 AND f.lang = 'sql' AND sql_leaf_name(s.name) = @resolvedNameLeaf THEN 3 - WHEN @allowLeafFallback = 1 AND f.lang = 'sql' AND sql_leaf_name_folded(s.name) = @resolvedNameLeafFolded THEN 4 - ELSE 5 - END"; - var matchingSql = $@" - SELECT f.path, f.lang, s.kind, s.name, s.line, - {GetSymbolColumnSql("start_line", "s.line")} AS start_line, - {startColumnSql} AS start_column, - {GetSymbolColumnSql("end_line", "s.line")} AS end_line, - {bodyStartLineSql} AS body_start_line, - {bodyEndLineSql} AS body_end_line, - {signatureSql} AS signature, - {GetSymbolColumnSql("container_kind")} AS container_kind, - {GetSymbolColumnSql("container_name")} AS container_name, - {GetSymbolColumnSql("visibility")} AS visibility, - {returnTypeSql} AS return_type, - {GetSymbolColumnSql("container_qualified_name")} AS container_qualified_name, - {logicalPartialKeySql} AS logical_partial_key, - s.id AS symbol_id, - {matchOrderSql} AS match_order, - {PathBucketOrder} AS path_bucket, - {VisibilityOrder} AS visibility_rank, - CASE WHEN s.kind IN ('class', 'struct', 'interface') THEN 1 ELSE 0 END AS is_precise, - CASE WHEN s.kind IN ('namespace', 'import') THEN 1 ELSE 0 END AS is_non_callable, - {canonicalPrimaryRankSql} AS canonical_primary_rank, - {generatedSql} AS canonical_generated_rank, - {canonicalSemanticScoreSql} AS canonical_semantic_score, - {canonicalDeclarationIdentitySql} AS canonical_declaration_identity, - COALESCE({GetSymbolColumnSql("start_column")}, 2147483647) AS stable_start_column, - {identifierStartColumnSql} AS identifier_start_column - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE {nameCondition} - AND {supportedLangFilter}"; - - if (lang != null) - matchingSql += " AND f.lang = @lang"; - AppendPathFilters(ref matchingSql, pathPatterns, excludePathPatterns, excludeTests); - var pathDistinctSql = ReferenceEquals(GetIndexedPathComparer(), StringComparer.Ordinal) - ? "COUNT(DISTINCT path)" - : "COUNT(DISTINCT path COLLATE NOCASE)"; - var precisePathDistinctSql = ReferenceEquals(GetIndexedPathComparer(), StringComparer.Ordinal) - ? "COUNT(DISTINCT CASE WHEN is_precise = 1 THEN path END)" - : "COUNT(DISTINCT CASE WHEN is_precise = 1 THEN path END COLLATE NOCASE)"; - const string canonicalRepresentativeOrder = "canonical_primary_rank, canonical_generated_rank, canonical_semantic_score DESC, canonical_declaration_identity COLLATE BINARY, path COLLATE BINARY, start_line, stable_start_column, symbol_id"; - const string resultOrder = "match_order, path_bucket, visibility_rank, name, path COLLATE BINARY, line, symbol_id"; - var sql = $@" - WITH matching_definitions AS ( - {matchingSql} - ), - ranked_definitions AS ( - SELECT matching_definitions.*, - ROW_NUMBER() OVER ( - PARTITION BY logical_partial_key - ORDER BY {canonicalRepresentativeOrder} - ) AS logical_row_number, - ROW_NUMBER() OVER ( - PARTITION BY logical_partial_key - ORDER BY path COLLATE BINARY, start_line, stable_start_column, symbol_id - ) AS family_member_row_number, - COUNT(*) OVER (PARTITION BY logical_partial_key) AS logical_definition_sites - FROM matching_definitions - ), - family_ranked_definitions AS ( - SELECT ranked_definitions.*, - MAX(CASE WHEN logical_row_number = 1 THEN family_member_row_number END) OVER ( - PARTITION BY logical_partial_key - ) AS representative_member_row_number - FROM ranked_definitions - ), - family_metadata_definitions AS ( - SELECT family_ranked_definitions.*, - MIN(canonical_primary_rank) OVER (PARTITION BY logical_partial_key) AS logical_primary_rank_min, - MAX(canonical_primary_rank) OVER (PARTITION BY logical_partial_key) AS logical_primary_rank_max, - MIN(canonical_generated_rank) OVER (PARTITION BY logical_partial_key) AS logical_generated_rank_min, - MAX(canonical_generated_rank) OVER (PARTITION BY logical_partial_key) AS logical_generated_rank_max, - MIN(canonical_semantic_score) OVER (PARTITION BY logical_partial_key) AS logical_semantic_score_min, - MAX(canonical_semantic_score) OVER (PARTITION BY logical_partial_key) AS logical_semantic_score_max, - MIN(canonical_declaration_identity) OVER (PARTITION BY logical_partial_key) AS logical_declaration_identity_min, - MAX(canonical_declaration_identity) OVER (PARTITION BY logical_partial_key) AS logical_declaration_identity_max, - json_group_array(json_object( - 'symbol_id', symbol_id, - 'path', path, - 'line', line, - 'start_line', start_line, - 'start_column', start_column, - 'end_line', end_line, - 'name', name, - 'signature', signature, - 'identifier_start_column', identifier_start_column, - 'generated', canonical_generated_rank - )) FILTER (WHERE - family_member_row_number <= CASE - WHEN representative_member_row_number <= {LogicalPartialSymbolGrouper.FamilyMemberLimit} - THEN {LogicalPartialSymbolGrouper.FamilyMemberLimit} - ELSE {LogicalPartialSymbolGrouper.FamilyMemberLimit - 1} - END - OR logical_row_number = 1 - ) OVER ( - PARTITION BY logical_partial_key - ORDER BY path COLLATE BINARY, start_line, stable_start_column, symbol_id - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ) AS logical_family_members_json - FROM family_ranked_definitions - ), - logical_definitions AS ( - SELECT * - FROM family_metadata_definitions - WHERE logical_row_number = 1 - ), - requested_definitions AS ( - SELECT logical_partial_key, 1 AS requested_row - FROM logical_definitions - ORDER BY {resultOrder} - LIMIT @definitionLimit OFFSET @definitionOffset - ), - single_precise_definition AS ( - SELECT logical_partial_key, 0 AS requested_row - FROM logical_definitions - WHERE is_precise = 1 - ORDER BY {resultOrder} - LIMIT 1 - ), - selected_definition_keys AS ( - SELECT logical_partial_key, requested_row - FROM requested_definitions - UNION ALL - SELECT precise.logical_partial_key, precise.requested_row - FROM single_precise_definition precise - WHERE NOT EXISTS ( - SELECT 1 - FROM requested_definitions requested - WHERE requested.logical_partial_key = precise.logical_partial_key - ) - ), - definition_stats AS ( - SELECT COUNT(*) AS physical_count, - {pathDistinctSql} AS physical_file_count, - COUNT(DISTINCT logical_partial_key) AS logical_count, - SUM(is_precise) AS precise_count, - COUNT(DISTINCT CASE WHEN is_precise = 1 THEN logical_partial_key END) AS precise_logical_count, - {precisePathDistinctSql} AS precise_file_count, - SUM(is_non_callable) AS non_callable_count - FROM matching_definitions - ) - SELECT logical.path, logical.lang, logical.kind, logical.name, logical.line, - logical.start_line, logical.start_column, logical.end_line, - logical.body_start_line, logical.body_end_line, logical.signature, - logical.container_kind, logical.container_name, - logical.visibility, logical.return_type, - logical.container_qualified_name, logical.logical_partial_key, - logical.symbol_id, logical.logical_definition_sites, selected.requested_row, - stats.physical_count, stats.physical_file_count, stats.logical_count, - stats.precise_count, stats.precise_file_count, stats.non_callable_count, - CASE - WHEN logical.logical_primary_rank_min <> logical.logical_primary_rank_max THEN '{LogicalPartialSymbolGrouper.ImplementationBodyReason}' - WHEN logical.logical_generated_rank_min <> logical.logical_generated_rank_max THEN '{LogicalPartialSymbolGrouper.NonGeneratedSourceReason}' - WHEN logical.logical_semantic_score_min <> logical.logical_semantic_score_max THEN '{LogicalPartialSymbolGrouper.SemanticDeclarationReason}' - WHEN logical.logical_declaration_identity_min <> logical.logical_declaration_identity_max THEN '{LogicalPartialSymbolGrouper.CanonicalDeclarationIdentityReason}' - ELSE '{LogicalPartialSymbolGrouper.StableLocationReason}' - END AS representative_reason, - logical.logical_family_members_json, - CASE WHEN logical.logical_definition_sites > {LogicalPartialSymbolGrouper.FamilyMemberLimit} THEN 1 ELSE 0 END AS family_members_truncated, - logical.identifier_start_column, - stats.precise_logical_count - FROM selected_definition_keys selected - JOIN logical_definitions logical - ON logical.logical_partial_key = selected.logical_partial_key - CROSS JOIN definition_stats stats - ORDER BY {resultOrder}"; - - cmd.CommandText = sql; - SqliteCommandPolicy.Add(cmd, "@resolvedName", resolvedName); - SqliteCommandPolicy.Add(cmd, "@resolvedNameNormalized", normalizedName); - SqliteCommandPolicy.Add(cmd, "@resolvedNameNormalizedFolded", FoldNameForLanguage(normalizedName, lang)); - SqliteCommandPolicy.Add(cmd, "@resolvedNameLeaf", leafName); - SqliteCommandPolicy.Add(cmd, "@resolvedNameLeafFolded", FoldNameForLanguage(leafName, lang)); - SqliteCommandPolicy.Add(cmd, "@resolvedNameSegmentCount", segmentCount); - SqliteCommandPolicy.Add(cmd, "@allowLeafFallback", allowLeafFallback ? 1 : 0); - AddCSharpExplicitInterfaceIdentityQueryParameter(cmd, "resolvedName", resolvedName); - if (SqlNameResolver.HasQualifier(resolvedName)) - { - var container = GetQualifiedQueryContainer(resolvedName); - SqliteCommandPolicy.Add(cmd, "@resolvedNameContainer", container); - SqliteCommandPolicy.Add(cmd, "@resolvedNameContainerSuffix", $"%.{EscapeLikeQuery(container)}"); - } - if (_foldReady) - AddPersistedFoldedNameQueryParameters(cmd, "@resolvedNameFolded", resolvedName, lang); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - SqliteCommandPolicy.Add(cmd, "@definitionLimit", Math.Max(1, representativeLimit)); - SqliteCommandPolicy.Add(cmd, "@definitionOffset", Math.Max(0, representativeOffset)); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - var results = new List(); - SymbolResult? preciseDefinition = null; - var physicalCount = 0; - var physicalFileCount = 0; - var logicalCount = 0; - var preciseCount = 0; - var preciseLogicalCount = 0; - var preciseFileCount = 0; - var nonCallableCount = 0; - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - var definitionSites = reader.GetInt32(18); - var result = new SymbolResult - { - Path = reader.GetString(0), - Lang = reader.GetString(1), - Kind = reader.GetString(2), - Name = reader.GetString(3), - Line = reader.GetInt32(4), - StartLine = !reader.IsDBNull(5) ? reader.GetInt32(5) : reader.GetInt32(4), - StartColumn = !reader.IsDBNull(29) - ? reader.GetInt32(29) - : ResolveSymbolIdentifierStartColumn( - !reader.IsDBNull(6) ? reader.GetInt32(6) : null, - !reader.IsDBNull(10) ? reader.GetString(10) : null, - reader.GetString(3), - reader.GetString(2)), - EndLine = !reader.IsDBNull(7) ? reader.GetInt32(7) : reader.GetInt32(4), - BodyStartLine = !reader.IsDBNull(8) ? reader.GetInt32(8) : null, - BodyEndLine = !reader.IsDBNull(9) ? reader.GetInt32(9) : null, - Signature = !reader.IsDBNull(10) ? reader.GetString(10) : null, - ContainerKind = !reader.IsDBNull(11) ? reader.GetString(11) : null, - ContainerName = !reader.IsDBNull(12) ? reader.GetString(12) : null, - ContainerQualifiedName = !reader.IsDBNull(15) ? reader.GetString(15) : null, - LogicalPartialKey = !reader.IsDBNull(16) ? reader.GetString(16) : null, - Visibility = !reader.IsDBNull(13) ? reader.GetString(13) : null, - ReturnType = !reader.IsDBNull(14) ? reader.GetString(14) : null, - SymbolId = reader.GetInt64(17), - DefinitionSites = definitionSites > 1 ? definitionSites : null, - }; - if (definitionSites > 1) - { - result.PartialFamilyId = LogicalPartialSymbolGrouper.BuildPartialFamilyId(result.LogicalPartialKey!); - result.RepresentativeReason = reader.GetString(26); - result.FamilyMembers = ReadPartialFamilyMembers(reader.GetString(27), result); - result.FamilyMembersTruncated = reader.GetInt64(28) != 0; - } - physicalCount = reader.GetInt32(20); - physicalFileCount = reader.GetInt32(21); - logicalCount = reader.GetInt32(22); - preciseCount = reader.GetInt32(23); - preciseFileCount = reader.GetInt32(24); - nonCallableCount = reader.GetInt32(25); - preciseLogicalCount = reader.GetInt32(30); - if (IsPreciseImpactFallbackKind(result.Kind)) - preciseDefinition ??= result; - if (reader.GetInt32(19) == 1) - results.Add(result); - } - - // Identity-scoped C# graph traversal keeps the representative-only - // definition payload, but retain every selected physical family ID internally so a - // call resolved to a partial declaration reaches the same graph as its implementation. - // identity-scoped C# のグラフ探索では definition の出力は代表1件のまま - // としつつ、partial 宣言側へ解決された call も実装側と同じグラフへ到達できるよう、 - // 選択された family の全 physical ID を内部的に保持する。 - reader.Dispose(); - var physicalSymbolIds = new HashSet(); - var physicalDefinitionPaths = new HashSet(GetIndexedPathComparer()); - var physicalSymbolIdsTruncated = false; - foreach (var definition in results) - { - AddPhysicalDefinition(definition.SymbolId!.Value, definition.Path); - if (physicalSymbolIdsTruncated || definition.DefinitionSites is not > 1) - continue; - - if (!definition.FamilyMembersTruncated) - { - foreach (var member in definition.FamilyMembers ?? []) - { - if (member.SymbolId is long memberSymbolId) - AddPhysicalDefinition(memberSymbolId, member.Path); - if (physicalSymbolIdsTruncated) - break; - } - continue; - } - - var (familyMembers, familyIdsTruncated) = ResolveImpactPhysicalFamilyMembers( - definition, - logicalPartialKeySql, - lang, - pathPatterns, - excludePathPatterns, - excludeTests); - foreach (var familyMember in familyMembers) - { - AddPhysicalDefinition(familyMember.SymbolId, familyMember.Path); - if (physicalSymbolIdsTruncated) - break; - } - physicalSymbolIdsTruncated |= familyIdsTruncated; - } - - return new ImpactDefinitionResolution( - results, - physicalCount, - physicalFileCount, - logicalCount, - preciseCount, - preciseLogicalCount, - preciseFileCount, - nonCallableCount, - preciseLogicalCount == 1 ? preciseDefinition : null, - physicalSymbolIds, - physicalDefinitionPaths, - physicalSymbolIdsTruncated); - - void AddPhysicalDefinition(long symbolId, string path) - { - if (physicalSymbolIds.Contains(symbolId)) - return; - if (physicalSymbolIds.Count >= Math.Max(1, ImpactPartialFamilyMemberBudget)) - { - physicalSymbolIdsTruncated = true; - return; - } - physicalSymbolIds.Add(symbolId); - physicalDefinitionPaths.Add(path); - } - } - - private (List<(long SymbolId, string Path)> Members, bool Truncated) ResolveImpactPhysicalFamilyMembers( - SymbolResult definition, - string logicalPartialKeySql, - string? lang, - IReadOnlyList? pathPatterns, - IReadOnlyList? excludePathPatterns, - bool excludeTests) - { - using var cmd = _conn.CreateCommand(); - var supportedLangFilter = BuildGraphSupportedLanguagePredicate(cmd, "f", "impactFamilyLang"); - var familyKindPredicate = definition.Kind is "function" or "test.method" - ? "s.kind IN ('function', 'test.method')" - : "s.kind = @familyKind"; - var sql = $@" - SELECT s.id, f.path - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE f.lang = @familyLang - AND {familyKindPredicate} - AND s.name = @familyName COLLATE BINARY - AND ({logicalPartialKeySql}) = @logicalPartialKey - AND {supportedLangFilter}"; - if (lang != null) - sql += " AND f.lang = @lang"; - AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); - sql += " ORDER BY s.id LIMIT @familyMemberLimit"; - - cmd.CommandText = sql; - SqliteCommandPolicy.Add(cmd, "@familyLang", definition.Lang!); - if (definition.Kind is not ("function" or "test.method")) - SqliteCommandPolicy.Add(cmd, "@familyKind", definition.Kind); - SqliteCommandPolicy.Add(cmd, "@familyName", definition.Name); - SqliteCommandPolicy.Add(cmd, "@logicalPartialKey", definition.LogicalPartialKey!); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - var familyMemberBudget = Math.Max(1, ImpactPartialFamilyMemberBudget); - SqliteCommandPolicy.Add(cmd, "@familyMemberLimit", familyMemberBudget + 1); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - - var members = new List<(long SymbolId, string Path)>(); - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - if (members.Count >= familyMemberBudget) - return (members, true); - members.Add((reader.GetInt64(0), reader.GetString(1))); - } - return (members, false); - } - // C# convention: a class `FooAttribute` is used in source as `[Foo]`, so the reference // site is stored with `symbol_name = "Foo"`. When a user queries with the class name // (`references FooAttribute`, `inspect FooAttribute`, `analyze_symbol("FooAttribute")`), diff --git a/src/CodeIndex/Database/DbReader.GraphReferenceQueries.cs b/src/CodeIndex/Database/DbReader.GraphReferenceQueries.cs new file mode 100644 index 0000000000..a9de381907 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.GraphReferenceQueries.cs @@ -0,0 +1,659 @@ +using CodeIndex.Indexer; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private enum GraphReferenceQueryShape + { + List, + LimitedCount, + TotalCount, + } + + private sealed record GraphReferenceQueryRequest( + string Query, + int Limit, + string? Lang, + string? ReferenceKind, + IReadOnlyList? PathPatterns, + IReadOnlyList? ExcludePathPatterns, + bool ExcludeTests, + bool Exact, + bool RawKinds, + bool IncludeQualifiedCommonCalls, + bool IncludeMemberReads, + long? IdentitySymbolId, + bool ExcludeSelfReferences, + int Offset); + + private sealed record GraphReferenceMatchPlan( + bool AllowSqlLeafFallback, + bool AllowQualifiedLeafFallback, + bool AllowCSharpQualifiedContextMatch, + bool UsesQualifiedName, + string? CssScssVariableAlias); + + private sealed record GraphReferenceBuildContext( + string SupportedLanguagePredicateSql, + string IdentityFilterSql, + GraphReferenceMatchPlan Match); + + private sealed record GraphReferenceRowLayout( + int FirstLineOrdinal, + int FirstColumnOrdinal, + bool FirstColumnIsNullable, + int? FirstLengthOrdinal, + int ReferenceCountOrdinal, + int ReferenceKindsOrdinal, + int ReferenceKindCountsOrdinal, + int ReferenceWeightOrdinal, + int? SelfReferenceOrdinal, + int? MutualRecursionOrdinal); + + private sealed record GraphReferenceDirectionSpec( + string MatchColumnSql, + string FoldedMatchColumnSql, + string SqlLeafExactMatchSql, + string SqlLeafFoldedMatchSql, + string RankQueriedNameSql, + bool UsesCSharpQualifiedContext, + bool CountGroupsByReferenceKind, + bool NormalizeBoundLanguage, + string IdentityParameterName, + GraphReferenceRowLayout RowLayout, + Func BuildReferenceJoinSql, + Func BuildSourcePredicateSql, + Func BuildIdentityFilterSql, + Func BuildPreNameFilterSql, + Func BuildPostNameFilterSql, + Func BuildQualifiedNameFilterSql, + Func BuildListSql, + Action BindMatchParameters); + + private sealed record GraphReferenceQueryPlan( + string Sql, + GraphReferenceQueryShape Shape, + GraphReferenceDirectionSpec Direction, + GraphReferenceQueryRequest Request, + GraphReferenceMatchPlan Match, + IReadOnlyList SupportedLanguages, + bool BindIdentityParameter); + + private sealed record GraphReferenceRow( + string Path, + string? Lang, + string? CallerKind, + string? CallerName, + string CalleeName, + string ReferenceKind, + IReadOnlyList ReferenceKinds, + IReadOnlyDictionary ReferenceKindCounts, + bool AggregateTruncated, + double ReferenceWeightScore, + int FirstLine, + int? FirstColumn, + int? FirstLength, + int ReferenceCount, + bool HasSelfReference, + bool HasMutualRecursion); + + private static readonly GraphReferenceDirectionSpec CallerGraphReferenceDirection = new( + MatchColumnSql: "r.symbol_name", + FoldedMatchColumnSql: "r.symbol_name_folded", + SqlLeafExactMatchSql: "r.symbol_name = sql_leaf_name(@aliasQuery) COLLATE NOCASE", + SqlLeafFoldedMatchSql: "r.symbol_name_folded = @aliasQueryLeafFolded", + RankQueriedNameSql: "r.symbol_name", + UsesCSharpQualifiedContext: true, + CountGroupsByReferenceKind: false, + NormalizeBoundLanguage: true, + IdentityParameterName: "@targetSymbolId", + RowLayout: new GraphReferenceRowLayout(6, 7, false, null, 8, 9, 10, 11, 12, 13), + BuildReferenceJoinSql: static reader => reader.ReferenceLineJoinSql("r"), + BuildSourcePredicateSql: static _ => BuildCallerContainerPredicate("f", "r"), + BuildIdentityFilterSql: static (reader, request) => reader.BuildCallerIdentityFilterSql(request), + BuildPreNameFilterSql: static (reader, request) => reader.BuildCallerPreNameFilterSql(request), + BuildPostNameFilterSql: static (reader, request) => reader.BuildCallerPostNameFilterSql(request), + BuildQualifiedNameFilterSql: static (reader, request, _) => reader.BuildCallerQualifiedNameFilterSql(request), + BuildListSql: static (reader, request, context) => reader.BuildCallerListSql(request, context), + BindMatchParameters: static (reader, command, plan) => reader.BindCallerMatchParameters(command, plan)); + + private static readonly GraphReferenceDirectionSpec CalleeGraphReferenceDirection = new( + MatchColumnSql: "r.container_name", + FoldedMatchColumnSql: "r.container_name_folded", + SqlLeafExactMatchSql: "sql_leaf_name(r.container_name) = @aliasQuery COLLATE NOCASE", + SqlLeafFoldedMatchSql: "sql_leaf_name_folded(r.container_name) = @aliasQueryLeafFolded", + RankQueriedNameSql: "r.container_name", + UsesCSharpQualifiedContext: false, + CountGroupsByReferenceKind: true, + NormalizeBoundLanguage: false, + IdentityParameterName: "@sourceSymbolId", + RowLayout: new GraphReferenceRowLayout(6, 7, true, 8, 9, 10, 11, 12, null, null), + BuildReferenceJoinSql: static _ => string.Empty, + BuildSourcePredicateSql: static _ => "r.container_name IS NOT NULL", + BuildIdentityFilterSql: static (reader, request) => reader.BuildCalleeIdentityFilterSql(request), + BuildPreNameFilterSql: static (_, _) => string.Empty, + BuildPostNameFilterSql: static (reader, request) => reader.BuildCalleePostNameFilterSql(request), + BuildQualifiedNameFilterSql: static (reader, request, _) => reader.BuildCalleeQualifiedNameFilterSql(request), + BuildListSql: static (reader, request, context) => reader.BuildCalleeListSql(request, context), + BindMatchParameters: static (reader, command, plan) => reader.BindCalleeMatchParameters(command, plan)); + + private static GraphReferenceQueryRequest CreateGraphReferenceQueryRequest( + string query, + int limit, + string? lang, + string? referenceKind, + IReadOnlyList? pathPatterns, + IReadOnlyList? excludePathPatterns, + bool excludeTests, + bool exact, + bool rawKinds, + bool includeQualifiedCommonCalls, + bool includeMemberReads, + long? identitySymbolId = null, + bool excludeSelfReferences = false, + int offset = 0) + => new( + query, + limit, + lang, + referenceKind, + pathPatterns, + excludePathPatterns, + excludeTests, + exact, + rawKinds, + includeQualifiedCommonCalls, + includeMemberReads, + identitySymbolId, + excludeSelfReferences, + offset); + + private GraphReferenceQueryPlan BuildGraphReferenceQueryPlan( + GraphReferenceDirectionSpec direction, + GraphReferenceQueryRequest request, + GraphReferenceQueryShape shape, + ReferenceRankMode rankMode = ReferenceRankMode.Weighted) + { + var supportedLanguages = GetWorkspaceSupportedReferenceLanguages() + .OrderBy(static language => language, StringComparer.Ordinal) + .ToList() + .AsReadOnly(); + var supportedLanguagePredicateSql = BuildGraphSupportedLanguagePredicateSql( + supportedLanguages, + "f", + "graphLang"); + var match = BuildGraphReferenceMatchPlan(direction, request); + var identityFilterSql = direction.BuildIdentityFilterSql(this, request); + var context = new GraphReferenceBuildContext( + supportedLanguagePredicateSql, + identityFilterSql, + match); + var sql = shape switch + { + GraphReferenceQueryShape.List => direction.BuildListSql(this, request, context), + GraphReferenceQueryShape.LimitedCount => BuildGraphReferenceCountSql(direction, request, context, limited: true), + GraphReferenceQueryShape.TotalCount => BuildGraphReferenceCountSql(direction, request, context, limited: false), + _ => throw new ArgumentOutOfRangeException(nameof(shape), shape, null), + }; + if (shape == GraphReferenceQueryShape.List) + sql += $" ORDER BY {BuildReferenceRankOrderSql(rankMode, direction.RankQueriedNameSql)} LIMIT @limit OFFSET @offset"; + + return new GraphReferenceQueryPlan( + sql, + shape, + direction, + request, + match, + supportedLanguages, + BindIdentityParameter: identityFilterSql.Length > 0); + } + + private GraphReferenceMatchPlan BuildGraphReferenceMatchPlan( + GraphReferenceDirectionSpec direction, + GraphReferenceQueryRequest request) + { + var usesQualifiedName = SqlNameResolver.HasQualifier(request.Query); + var allowCSharpQualifiedContextMatch = direction.UsesCSharpQualifiedContext + && usesQualifiedName + && !HasQualifiedSymbolDefinition( + request.Query, + request.Lang, + request.PathPatterns, + request.ExcludePathPatterns, + request.ExcludeTests); + + return new GraphReferenceMatchPlan( + AllowSqlLeafFallbackForQuery(request.Query), + HasSingleQualifiedSymbolDefinition( + request.Query, + request.Lang, + request.PathPatterns, + request.ExcludePathPatterns, + request.ExcludeTests), + allowCSharpQualifiedContextMatch, + usesQualifiedName, + ComputeCssScssVariableAlias(request.Query)); + } + + private static string BuildGraphSupportedLanguagePredicateSql( + IReadOnlyList supportedLanguages, + string fileAlias, + string parameterPrefix) + { + if (supportedLanguages.Count == 0) + return "1 = 0"; + + var parameterNames = Enumerable.Range(0, supportedLanguages.Count) + .Select(index => $"@{parameterPrefix}{index}"); + return $"{fileAlias}.lang IN ({string.Join(", ", parameterNames)})"; + } + + private string BuildGraphReferenceCountSql( + GraphReferenceDirectionSpec direction, + GraphReferenceQueryRequest request, + GraphReferenceBuildContext context, + bool limited) + { + var outerReferenceKindColumn = direction.CountGroupsByReferenceKind + ? ", reference_kind" + : string.Empty; + var innerReferenceKindColumn = direction.CountGroupsByReferenceKind + ? ",\n " + (request.RawKinds + ? GetPreferredReferenceKindSql("r.reference_kind") + : GetPreferredLogicalReferenceKindSql("r.reference_kind")) + " AS reference_kind" + : string.Empty; + var sql = @" + SELECT path, lang, container_kind, container_name, symbol_name" + outerReferenceKindColumn + @" + FROM ( + SELECT f.path AS path, f.lang AS lang, r.container_kind AS container_kind, + r.container_name AS container_name, r.symbol_name AS symbol_name" + innerReferenceKindColumn + @" + FROM symbol_references r + JOIN files f ON r.file_id = f.id" + direction.BuildReferenceJoinSql(this) + @" + WHERE " + direction.BuildSourcePredicateSql(this); + sql += $" AND {context.SupportedLanguagePredicateSql}"; + + sql += $" AND {GetCallableReferenceKindPredicateSql("r.reference_kind", request.ReferenceKind, "f.lang", request.IncludeMemberReads)}"; + AppendGraphReferenceTailFilters(ref sql, direction, request, context); + sql += $" GROUP BY f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.file_id, r.line, r.column_number, {(request.RawKinds ? GetRawReferenceKindSql("r.reference_kind") : GetLogicalReferenceKindSql("r.reference_kind"))}"; + sql += " ) grouped_call_sites GROUP BY path, lang, container_kind, container_name, symbol_name" + outerReferenceKindColumn; + if (limited) + sql += " LIMIT @limit"; + + return limited + ? $"SELECT COUNT(*) FROM ({sql})" + : $"SELECT COUNT(*), COUNT(DISTINCT path), MAX(CASE WHEN lang = 'sql' THEN 1 ELSE 0 END) FROM ({sql})"; + } + + private string BuildCallerListSql( + GraphReferenceQueryRequest request, + GraphReferenceBuildContext context) + { + var groupedReferenceKindSql = request.RawKinds + ? GetGroupedCallerReferenceKindSql("r.reference_kind") + : GetGroupedCallerLogicalReferenceKindSql("r.reference_kind"); + var groupedReferenceKindGroupSql = request.RawKinds + ? GetRawReferenceKindSql("r.reference_kind") + : GetLogicalReferenceKindSql("r.reference_kind"); + var selfReferenceSql = _referenceColumns.Contains("is_self_reference") ? "r.is_self_reference" : "0"; + var mutualRecursionSql = _referenceColumns.Contains("is_mutual_recursion") ? "r.is_mutual_recursion" : "0"; + var sql = @" + WITH logical_references AS ( + SELECT f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, + " + groupedReferenceKindSql + @" AS reference_kind, + r.reference_kind AS raw_reference_kind, + " + groupedReferenceKindGroupSql + @" AS count_reference_kind, + COUNT(*) AS reference_count, + " + ReferenceWeightedScoreSql("r.reference_kind") + @" AS weighted_score, + (CAST(r.line AS INTEGER) * 4294967296 + r.column_number) AS location_key, + MAX(" + selfReferenceSql + @") AS is_self_reference, + MAX(" + mutualRecursionSql + @") AS is_mutual_recursion + FROM symbol_references r + JOIN files f ON r.file_id = f.id" + ReferenceLineJoinSql("r") + @" + WHERE " + BuildCallerContainerPredicate("f", "r") + @" + AND " + GetCallableReferenceKindPredicateSql("r.reference_kind", request.ReferenceKind, "f.lang", request.IncludeMemberReads) + @" + AND " + context.SupportedLanguagePredicateSql; + AppendGraphReferenceTailFilters(ref sql, CallerGraphReferenceDirection, request, context); + sql += @" + GROUP BY f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.file_id, r.line, r.column_number, " + groupedReferenceKindGroupSql + @", r.reference_kind + ) + SELECT path, lang, " + BuildCallerKindProjectionSql("r") + @" AS container_kind, " + BuildCallerNameProjectionSql("r") + @" AS container_name, symbol_name, + " + (request.RawKinds ? GetGroupedCallerReferenceKindSql("r.reference_kind") : GetPreferredLogicalReferenceKindSql("r.reference_kind")) + @" AS reference_kind, + (MIN(location_key) / 4294967296) AS first_line, + (MIN(location_key) % 4294967296) AS first_column, + SUM(r.reference_count) AS reference_count, + GROUP_CONCAT(DISTINCT r.reference_kind) AS reference_kinds, + GROUP_CONCAT(r.count_reference_kind || ':' || r.reference_count) AS reference_kind_counts, + SUM(r.weighted_score) AS weighted_score, + MAX(r.is_self_reference) AS is_self_reference, + MAX(r.is_mutual_recursion) AS is_mutual_recursion + FROM logical_references r + GROUP BY path, lang, container_kind, container_name, symbol_name"; + return sql; + } + + private string BuildCalleeListSql( + GraphReferenceQueryRequest request, + GraphReferenceBuildContext context) + { + var preferredCalleeKindSql = request.RawKinds + ? GetPreferredReferenceKindSql("r.reference_kind") + : GetPreferredLogicalReferenceKindSql("r.reference_kind"); + var calleeGroupKindSql = request.RawKinds + ? GetRawReferenceKindSql("r.reference_kind") + : GetLogicalReferenceKindSql("r.reference_kind"); + var referenceSpanLengthSql = _referenceColumns.Contains("span_length") + ? "r.span_length" + : "NULL"; + var sql = $@" + WITH logical_references AS ( + SELECT f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, + {preferredCalleeKindSql} AS reference_kind, + r.reference_kind AS raw_reference_kind, + {calleeGroupKindSql} AS count_reference_kind, + COUNT(*) AS reference_count, + {ReferenceWeightedScoreSql("r.reference_kind")} AS weighted_score, + r.line, + r.column_number, + {referenceSpanLengthSql} AS span_length + FROM symbol_references r + JOIN files f ON r.file_id = f.id + WHERE r.container_name IS NOT NULL + AND {GetCallableReferenceKindPredicateSql("r.reference_kind", request.ReferenceKind, "f.lang", request.IncludeMemberReads)} + AND {context.SupportedLanguagePredicateSql}"; + AppendGraphReferenceTailFilters(ref sql, CalleeGraphReferenceDirection, request, context); + sql += $@" + GROUP BY f.path, f.lang, r.container_kind, r.container_name, r.symbol_name, r.file_id, r.line, r.column_number, {referenceSpanLengthSql}, r.reference_kind + ), + ranked_call_sites AS ( + SELECT logical_references.*, + ROW_NUMBER() OVER ( + PARTITION BY path, lang, container_kind, container_name, symbol_name, reference_kind + ORDER BY CASE WHEN column_number IS NULL THEN 1 ELSE 0 END, + line, + column_number, + COALESCE(span_length, 0) + ) AS location_rank + FROM logical_references + ) + SELECT path, lang, container_kind, container_name, symbol_name, + reference_kind, + MAX(CASE WHEN location_rank = 1 THEN line END) AS first_line, + MAX(CASE WHEN location_rank = 1 THEN column_number END) AS first_column, + MAX(CASE WHEN location_rank = 1 THEN span_length END) AS first_length, + SUM(r.reference_count) AS reference_count, + GROUP_CONCAT(DISTINCT reference_kind) AS reference_kinds, + GROUP_CONCAT(r.count_reference_kind || ':' || r.reference_count) AS reference_kind_counts, + SUM(r.weighted_score) AS weighted_score + FROM ranked_call_sites r + GROUP BY path, lang, container_kind, container_name, symbol_name, reference_kind"; + return sql; + } + + private void AppendGraphReferenceTailFilters( + ref string sql, + GraphReferenceDirectionSpec direction, + GraphReferenceQueryRequest request, + GraphReferenceBuildContext context) + { + sql += context.IdentityFilterSql; + sql += direction.BuildPreNameFilterSql(this, request); + sql += BuildGraphReferenceNameFilterSql(direction, request, context.Match); + sql += direction.BuildPostNameFilterSql(this, request); + AppendPathFilters(ref sql, request.PathPatterns, request.ExcludePathPatterns, request.ExcludeTests); + } + + private string BuildGraphReferenceNameFilterSql( + GraphReferenceDirectionSpec direction, + GraphReferenceQueryRequest request, + GraphReferenceMatchPlan match) + { + if (match.UsesQualifiedName) + return direction.BuildQualifiedNameFilterSql(this, request, match); + + var cssAliasScope = match.CssScssVariableAlias != null + ? " AND f.lang = 'css'" + : string.Empty; + if (request.Exact && _foldReady) + { + var foldedMatchSql = BuildPersistedFoldedNameMatchSql(direction.FoldedMatchColumnSql, "@query"); + if (!match.AllowSqlLeafFallback) + return $" AND {foldedMatchSql}"; + if (match.CssScssVariableAlias != null) + return $" AND ({foldedMatchSql} OR ({direction.FoldedMatchColumnSql} = @queryCssScssVariableAlias{cssAliasScope}) OR (f.lang = 'sql' AND {direction.SqlLeafFoldedMatchSql}))"; + return $" AND ({foldedMatchSql} OR (f.lang = 'sql' AND {direction.SqlLeafFoldedMatchSql}))"; + } + + if (request.Exact) + { + if (!match.AllowSqlLeafFallback) + return $" AND {direction.MatchColumnSql} = @query COLLATE NOCASE"; + if (match.CssScssVariableAlias != null) + return $" AND ({direction.MatchColumnSql} = @query COLLATE NOCASE OR ({direction.MatchColumnSql} = @queryCssScssVariableAlias COLLATE NOCASE{cssAliasScope}) OR (f.lang = 'sql' AND {direction.SqlLeafExactMatchSql}))"; + return $" AND ({direction.MatchColumnSql} = @query COLLATE NOCASE OR (f.lang = 'sql' AND {direction.SqlLeafExactMatchSql}))"; + } + + if (match.CssScssVariableAlias != null) + return $" AND ({direction.MatchColumnSql} LIKE @query ESCAPE '\\' OR ({direction.MatchColumnSql} = @queryCssScssVariableAlias COLLATE NOCASE{cssAliasScope}) OR (f.lang = 'sql' AND {direction.SqlLeafExactMatchSql}))"; + return $" AND ({direction.MatchColumnSql} LIKE @query ESCAPE '\\' OR (f.lang = 'sql' AND {direction.SqlLeafExactMatchSql}))"; + } + + private string BuildCallerQualifiedNameFilterSql(GraphReferenceQueryRequest request) + { + var folded = _foldReady; + var like = !request.Exact; + var contextSql = ReferenceContextSql("r"); + var qualifiedContextSql = BuildQualifiedContextMatchSql(contextSql, "r.column_number", folded, like); + var csharpQualifiedContextSql = BuildCSharpQualifiedContextFallbackSql(qualifiedContextSql); + var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql("r.symbol_name", "r.symbol_name_folded", folded); + var nonSqlMatchSql = request.Exact + ? folded + ? BuildPersistedFoldedNameMatchSql("r.symbol_name_folded", "@query") + : "r.symbol_name = @query COLLATE NOCASE" + : "r.symbol_name LIKE @query ESCAPE '\\'"; + return $" AND (((f.lang = 'sql') AND {qualifiedContextSql}) OR ((f.lang != 'sql') AND {nonSqlMatchSql}) OR {csharpQualifiedContextSql} OR {qualifiedLeafFallbackSql})"; + } + + private string BuildCalleeQualifiedNameFilterSql(GraphReferenceQueryRequest request) + { + var qualifiedLeafFallbackSql = BuildQualifiedLeafFallbackSql( + "r.container_name", + "r.container_name_folded", + folded: _foldReady); + if (!request.Exact) + return $" AND (r.container_name LIKE @query ESCAPE '\\' OR {qualifiedLeafFallbackSql})"; + if (_foldReady) + return $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name_folded(r.container_name) = @aliasQueryNormalizedFolded) OR ((f.lang != 'sql') AND {BuildPersistedFoldedNameMatchSql("r.container_name_folded", "@query")}) OR {qualifiedLeafFallbackSql})"; + return $" AND (((f.lang = 'sql') AND sql_segment_count(r.container_name) = @aliasQuerySegmentCount AND sql_normalize_name(r.container_name) = @aliasQueryNormalized COLLATE NOCASE) OR ((f.lang != 'sql') AND r.container_name = @query COLLATE NOCASE) OR {qualifiedLeafFallbackSql})"; + } + + private string BuildCallerIdentityFilterSql(GraphReferenceQueryRequest request) + { + if (request.IdentitySymbolId == null || !HasTable("symbol_reference_candidates")) + return string.Empty; + + var resolutionFilterSql = _referenceColumns.Contains("resolution_state") + ? " AND r.resolution_state IN ('resolved', 'resolved_group')" + : " AND 1 = 0"; + return resolutionFilterSql + @" + AND EXISTS ( + SELECT 1 + FROM symbol_reference_candidates AS identity_candidate + WHERE identity_candidate.reference_id = r.id + AND identity_candidate.symbol_id = @targetSymbolId + )"; + } + + private string BuildCalleeIdentityFilterSql(GraphReferenceQueryRequest request) + => request.IdentitySymbolId != null && _referenceColumns.Contains("source_symbol_id") + ? " AND r.source_symbol_id = @sourceSymbolId" + : string.Empty; + + private string BuildCallerPreNameFilterSql(GraphReferenceQueryRequest request) + { + if (!request.ExcludeSelfReferences) + return string.Empty; + var selfReferenceSql = _referenceColumns.Contains("is_self_reference") + ? "r.is_self_reference" + : "0"; + return $" AND {selfReferenceSql} = 0"; + } + + private string BuildCallerPostNameFilterSql(GraphReferenceQueryRequest request) + { + var sql = string.Empty; + if (request.Lang != null) + { + sql += IncludeAmbiguousMSourceForIdentityTarget(request.Lang, request.IdentitySymbolId) + ? " AND (f.lang = @lang OR f.lang = 'ambiguous_m')" + : " AND f.lang = @lang"; + } + sql += BuildCSharpBareMemberReferenceFilter( + request.Query, + request.Lang, + "f", + "r", + request.IncludeQualifiedCommonCalls); + return sql; + } + + private string BuildCalleePostNameFilterSql(GraphReferenceQueryRequest request) + { + var sql = request.Lang != null + ? " AND f.lang = @lang" + : string.Empty; + if (!request.IncludeQualifiedCommonCalls) + sql += BuildCSharpQualifiedCommonCallNoiseFilter("f", "r"); + return sql; + } + + private void BindGraphReferenceQueryPlan(SqliteCommand command, GraphReferenceQueryPlan plan) + { + for (var index = 0; index < plan.SupportedLanguages.Count; index++) + SqliteCommandPolicy.Add(command, $"@graphLang{index}", plan.SupportedLanguages[index]); + + var request = plan.Request; + var value = !request.Exact + ? $"%{EscapeLikeQuery(request.Query)}%" + : _foldReady + ? FoldNameForLanguage(request.Query, request.Lang) + : request.Query; + if (request.Exact && _foldReady) + AddPersistedFoldedNameQueryParameters(command, "@query", request.Query, request.Lang); + else + SqliteCommandPolicy.Add(command, "@query", value); + SqliteCommandPolicy.Add(command, "@aliasQuery", request.Query); + plan.Direction.BindMatchParameters(this, command, plan); + + if (RequiresReferenceKindParameter(request.ReferenceKind)) + SqliteCommandPolicy.Add(command, "@referenceKind", request.ReferenceKind); + if (request.Lang != null) + { + var language = plan.Direction.NormalizeBoundLanguage + ? NormalizeQueryLanguage(request.Lang) + : request.Lang; + SqliteCommandPolicy.Add(command, "@lang", language); + } + if (plan.BindIdentityParameter) + SqliteCommandPolicy.Add(command, plan.Direction.IdentityParameterName, request.IdentitySymbolId!.Value); + AddPathFilterParameters(command, request.PathPatterns, request.ExcludePathPatterns); + if (plan.Shape != GraphReferenceQueryShape.TotalCount) + SqliteCommandPolicy.Add(command, "@limit", request.Limit); + if (plan.Shape == GraphReferenceQueryShape.List) + SqliteCommandPolicy.Add(command, "@offset", Math.Max(0, request.Offset)); + } + + private void BindCallerMatchParameters(SqliteCommand command, GraphReferenceQueryPlan plan) + { + var request = plan.Request; + AddQualifiedGraphQueryParameters( + command, + request.Query, + plan.Match.AllowQualifiedLeafFallback, + plan.Match.AllowCSharpQualifiedContextMatch); + SqliteCommandPolicy.Add(command, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(request.Query)) ?? SqlNameResolver.GetLeafName(request.Query)); + AddCssScssVariableAliasParameter(command, request, plan.Match.CssScssVariableAlias); + if (plan.Shape == GraphReferenceQueryShape.List) + AddGraphReferenceRankingParameters(command, request.Query); + } + + private void BindCalleeMatchParameters(SqliteCommand command, GraphReferenceQueryPlan plan) + { + var request = plan.Request; + var normalizedQuery = SqlNameResolver.NormalizeQualifiedName(request.Query); + SqliteCommandPolicy.Add(command, "@aliasQueryLeafFolded", NameFold.Fold(SqlNameResolver.GetLeafName(request.Query)) ?? SqlNameResolver.GetLeafName(request.Query)); + SqliteCommandPolicy.Add(command, "@aliasQueryNormalized", normalizedQuery); + SqliteCommandPolicy.Add(command, "@aliasQueryNormalizedFolded", NameFold.Fold(normalizedQuery) ?? normalizedQuery); + SqliteCommandPolicy.Add(command, "@aliasQuerySegmentCount", SqlNameResolver.GetSegmentCount(request.Query)); + if (plan.Shape == GraphReferenceQueryShape.List) + { + AddCssScssVariableAliasParameter(command, request, plan.Match.CssScssVariableAlias); + AddGraphReferenceRankingParameters(command, request.Query); + AddQualifiedGraphQueryParameters(command, request.Query, plan.Match.AllowQualifiedLeafFallback); + } + else + { + AddQualifiedGraphQueryParameters(command, request.Query, plan.Match.AllowQualifiedLeafFallback); + AddCssScssVariableAliasParameter(command, request, plan.Match.CssScssVariableAlias); + } + } + + private void AddCssScssVariableAliasParameter( + SqliteCommand command, + GraphReferenceQueryRequest request, + string? cssScssVariableAlias) + { + if (cssScssVariableAlias == null) + return; + var aliasParameter = request.Exact && _foldReady + ? NameFold.Fold(cssScssVariableAlias) ?? cssScssVariableAlias + : cssScssVariableAlias; + SqliteCommandPolicy.Add(command, "@queryCssScssVariableAlias", aliasParameter); + } + + private static void AddGraphReferenceRankingParameters(SqliteCommand command, string query) + { + SqliteCommandPolicy.Add(command, "@rawQuery", query); + SqliteCommandPolicy.Add(command, "@rankingQuery", query.Trim()); + } + + private static string ReferenceWeightedScoreSql(string columnSql) => $@" + SUM(CASE {columnSql} + WHEN 'instantiate' THEN 3.0 + WHEN 'generic_type_argument' THEN 0.5 + WHEN 'call' THEN 1.0 + WHEN 'subscribe' THEN 0.1 + WHEN 'unsubscribe' THEN 0.1 + WHEN 'razor_event_binding' THEN 0.1 + ELSE 0.0 + END)"; + + private static string BuildReferenceRankOrderSql( + ReferenceRankMode rankMode, + string queriedNameSql) + => string.Join( + ", ", + ReferenceRankRecipes.Get(rankMode).Select(dimension => dimension switch + { + ReferenceRankDimension.ReferenceWeightScoreDescending => "weighted_score DESC", + ReferenceRankDimension.ReferenceCountDescending => "reference_count DESC", + ReferenceRankDimension.ReferenceKindPriorityAscending => + "CASE reference_kind WHEN 'instantiate' THEN 0 WHEN 'call' THEN 1 WHEN 'generic_type_argument' THEN 2 WHEN 'subscribe' THEN 3 ELSE 4 END", + ReferenceRankDimension.ExactCaseMatchDescending => + $"CASE WHEN {queriedNameSql} = @rawQuery THEN 0 ELSE 1 END", + ReferenceRankDimension.ExactNameMatchDescending => + $"CASE WHEN lower({queriedNameSql}) = lower(@rankingQuery) THEN 0 ELSE 1 END", + ReferenceRankDimension.PathCategoryAscending => GetPathBucketOrderSql("r.path"), + ReferenceRankDimension.PathAscending => "r.path", + ReferenceRankDimension.FirstLineAscending => "first_line", + ReferenceRankDimension.FirstColumnAscending => "first_column", + ReferenceRankDimension.LanguageAscending => "r.lang", + ReferenceRankDimension.ContainerKindAscending => "r.container_kind", + ReferenceRankDimension.ContainerNameAscending => "r.container_name", + ReferenceRankDimension.SymbolNameAscending => "r.symbol_name", + ReferenceRankDimension.ReferenceKindAscending => "reference_kind", + _ => throw new ArgumentOutOfRangeException(nameof(dimension), dimension, null), + })); + +} diff --git a/src/CodeIndex/Database/DbReader.GraphReferenceQueryExecution.cs b/src/CodeIndex/Database/DbReader.GraphReferenceQueryExecution.cs new file mode 100644 index 0000000000..06f3ff5737 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.GraphReferenceQueryExecution.cs @@ -0,0 +1,144 @@ +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private List ExecuteGraphReferenceList( + GraphReferenceQueryPlan plan, + Func projector) + { + using var command = _conn.CreateCommand(); + command.CommandText = plan.Sql; + BindGraphReferenceQueryPlan(command, plan); + + var results = new List(); + using var reader = command.ExecuteTrackedReader(); + while (reader.TrackedRead()) + results.Add(projector(ReadGraphReferenceRow(reader, plan.Direction.RowLayout))); + return results; + } + + private int ExecuteGraphReferenceLimitedCount(GraphReferenceQueryPlan plan) + { + using var command = _conn.CreateCommand(); + command.CommandText = plan.Sql; + BindGraphReferenceQueryPlan(command, plan); + var raw = command.ExecuteScalar(); + return raw is long count ? (int)count : Convert.ToInt32(raw); + } + + private QueryCountResult ExecuteGraphReferenceTotalCount(GraphReferenceQueryPlan plan) + { + using var command = _conn.CreateCommand(); + command.CommandText = plan.Sql; + BindGraphReferenceQueryPlan(command, plan); + return ExecuteCountSummary(command); + } + + private GraphReferenceRow ReadGraphReferenceRow( + SqliteDataReader reader, + GraphReferenceRowLayout layout) + { + var primaryKind = reader.GetString(5); + var kindAggregate = TruncateReferenceKindAggregate( + GetNullableString(reader, layout.ReferenceKindsOrdinal), + out var kindsTruncated); + var countAggregate = TruncateReferenceKindAggregate( + GetNullableString(reader, layout.ReferenceKindCountsOrdinal), + out var countsTruncated); + var referenceCount = reader.GetInt32(layout.ReferenceCountOrdinal); + var firstColumn = layout.FirstColumnIsNullable + ? GetNullableInt32(reader, layout.FirstColumnOrdinal) + : reader.GetInt32(layout.FirstColumnOrdinal); + return new GraphReferenceRow( + reader.GetString(0), + GetNullableString(reader, 1), + GetNullableString(reader, 2), + GetNullableString(reader, 3), + reader.GetString(4), + primaryKind, + ParseDistinctReferenceKinds(kindAggregate, primaryKind), + ParseReferenceKindCounts(countAggregate, primaryKind, referenceCount), + kindsTruncated || countsTruncated, + reader.GetDouble(layout.ReferenceWeightOrdinal), + reader.GetInt32(layout.FirstLineOrdinal), + firstColumn, + layout.FirstLengthOrdinal is int lengthOrdinal ? GetNullableInt32(reader, lengthOrdinal) : null, + referenceCount, + layout.SelfReferenceOrdinal is int selfOrdinal && reader.GetInt32(selfOrdinal) != 0, + layout.MutualRecursionOrdinal is int mutualOrdinal && reader.GetInt32(mutualOrdinal) != 0); + } + + private static CallerResult ProjectCallerResult(GraphReferenceRow row) + => new() + { + Path = row.Path, + Lang = row.Lang, + CallerKind = row.CallerKind, + CallerName = row.CallerName, + CalleeName = row.CalleeName, + ReferenceKind = row.ReferenceKind, + ReferenceKinds = row.ReferenceKinds, + HasMixedReferenceKinds = row.ReferenceKinds.Count > 1, + ReferenceKindCounts = row.ReferenceKindCounts, + AggregateTruncated = row.AggregateTruncated, + ReferenceWeightScore = row.ReferenceWeightScore, + FirstLine = row.FirstLine, + FirstColumn = row.FirstColumn!.Value, + ReferenceCount = row.ReferenceCount, + HasSelfReference = row.HasSelfReference, + HasMutualRecursion = row.HasMutualRecursion, + }; + + private static CalleeResult ProjectCalleeResult(GraphReferenceRow row) + => new() + { + Path = row.Path, + Lang = row.Lang, + CallerKind = row.CallerKind, + CallerName = row.CallerName, + CalleeName = row.CalleeName, + ReferenceKind = row.ReferenceKind, + ReferenceKinds = row.ReferenceKinds, + HasMixedReferenceKinds = row.ReferenceKinds.Count > 1, + ReferenceKindCounts = row.ReferenceKindCounts, + AggregateTruncated = row.AggregateTruncated, + ReferenceWeightScore = row.ReferenceWeightScore, + FirstLine = row.FirstLine, + FirstColumn = row.FirstColumn, + FirstLength = row.FirstLength, + ReferenceCount = row.ReferenceCount, + }; + + private static IReadOnlyDictionary ParseReferenceKindCounts( + string? aggregate, + string primaryKind, + int fallbackCount) + { + var counts = new Dictionary(StringComparer.Ordinal) + { + ["call"] = 0, + ["instantiate"] = 0, + ["subscribe"] = 0, + }; + if (!string.IsNullOrWhiteSpace(aggregate)) + { + foreach (var entry in aggregate.Split(',')) + { + var separator = entry.LastIndexOf(':'); + if (separator <= 0 || separator == entry.Length - 1) + continue; + var kind = entry[..separator].Trim(); + if (kind.Length == 0 || !int.TryParse(entry[(separator + 1)..], out var count)) + continue; + counts[kind] = counts.TryGetValue(kind, out var existing) + ? existing + count + : count; + } + } + if (counts.Count == 0 && !string.IsNullOrEmpty(primaryKind)) + counts[primaryKind] = fallbackCount; + return counts; + } +} diff --git a/src/CodeIndex/Database/DbReader.ImpactDefinitionFamilies.cs b/src/CodeIndex/Database/DbReader.ImpactDefinitionFamilies.cs new file mode 100644 index 0000000000..aa4a9f7c81 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.ImpactDefinitionFamilies.cs @@ -0,0 +1,178 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private readonly record struct ImpactPhysicalFamilyMember(long SymbolId, string Path); + + private sealed record ImpactDefinitionFamilyResolution( + HashSet PhysicalSymbolIds, + HashSet PhysicalDefinitionPaths, + bool Truncated); + + private static class ImpactDefinitionFamilyResolver + { + private sealed class Accumulator( + int budget, + IEqualityComparer pathComparer) + { + public HashSet SymbolIds { get; } = []; + public HashSet DefinitionPaths { get; } = new(pathComparer); + public bool Truncated { get; private set; } + + public void Add(long symbolId, string path) + { + if (SymbolIds.Contains(symbolId)) + return; + if (SymbolIds.Count >= budget) + { + Truncated = true; + return; + } + + SymbolIds.Add(symbolId); + DefinitionPaths.Add(path); + } + + public void MarkTruncated() => Truncated = true; + } + + public static ImpactDefinitionFamilyResolution Resolve( + DbReader reader, + ImpactDefinitionQueryPlan plan, + IReadOnlyList definitions) + { + var accumulator = new Accumulator( + Math.Max(1, reader.ImpactPartialFamilyMemberBudget), + reader.GetIndexedPathComparer()); + foreach (var definition in definitions) + { + accumulator.Add(definition.SymbolId!.Value, definition.Path); + if (accumulator.Truncated || definition.DefinitionSites is not > 1) + continue; + + if (!definition.FamilyMembersTruncated) + { + AddProjectedMembers(accumulator, definition); + continue; + } + + var (members, familyTruncated) = ResolvePhysicalMembers( + reader, + definition, + plan); + foreach (var member in members) + { + accumulator.Add(member.SymbolId, member.Path); + if (accumulator.Truncated) + break; + } + if (familyTruncated) + accumulator.MarkTruncated(); + } + + return new ImpactDefinitionFamilyResolution( + accumulator.SymbolIds, + accumulator.DefinitionPaths, + accumulator.Truncated); + } + + private static void AddProjectedMembers( + Accumulator accumulator, + SymbolResult definition) + { + foreach (var member in definition.FamilyMembers ?? []) + { + if (member.SymbolId is long memberSymbolId) + accumulator.Add(memberSymbolId, member.Path); + if (accumulator.Truncated) + break; + } + } + + private static (List Members, bool Truncated) + ResolvePhysicalMembers( + DbReader dbReader, + SymbolResult definition, + ImpactDefinitionQueryPlan plan) + { + using var cmd = dbReader._conn.CreateCommand(); + var supportedLangFilter = dbReader.BuildGraphSupportedLanguagePredicate( + cmd, + "f", + "impactFamilyLang"); + var familyKindPredicate = definition.Kind is "function" or "test.method" + ? "s.kind IN ('function', 'test.method')" + : "s.kind = @familyKind"; + var sql = BuildSql( + dbReader, + plan.LogicalPartialKeySql, + familyKindPredicate, + supportedLangFilter, + plan.Request); + cmd.CommandText = sql; + Bind(dbReader, cmd, definition, plan.Request); + + var familyMemberBudget = Math.Max(1, dbReader.ImpactPartialFamilyMemberBudget); + var members = new List(); + using var reader = cmd.ExecuteTrackedReader(); + while (reader.TrackedRead()) + { + if (members.Count >= familyMemberBudget) + return (members, true); + members.Add(new ImpactPhysicalFamilyMember( + reader.GetInt64(0), + reader.GetString(1))); + } + return (members, false); + } + + private static void Bind( + DbReader reader, + Microsoft.Data.Sqlite.SqliteCommand cmd, + SymbolResult definition, + ImpactDefinitionRequest request) + { + SqliteCommandPolicy.Add(cmd, "@familyLang", definition.Lang!); + if (definition.Kind is not ("function" or "test.method")) + SqliteCommandPolicy.Add(cmd, "@familyKind", definition.Kind); + SqliteCommandPolicy.Add(cmd, "@familyName", definition.Name); + SqliteCommandPolicy.Add(cmd, "@logicalPartialKey", definition.LogicalPartialKey!); + if (request.Lang != null) + SqliteCommandPolicy.Add(cmd, "@lang", request.Lang); + SqliteCommandPolicy.Add( + cmd, + "@familyMemberLimit", + Math.Max(1, reader.ImpactPartialFamilyMemberBudget) + 1); + DbReader.AddPathFilterParameters( + cmd, + request.PathPatterns, + request.ExcludePathPatterns); + } + + private static string BuildSql( + DbReader reader, + string logicalPartialKeySql, + string familyKindPredicate, + string supportedLangFilter, + ImpactDefinitionRequest request) + { + var sql = $@" + SELECT s.id, f.path + FROM symbols s + JOIN files f ON s.file_id = f.id + WHERE f.lang = @familyLang + AND {familyKindPredicate} + AND s.name = @familyName COLLATE BINARY + AND ({logicalPartialKeySql}) = @logicalPartialKey + AND {supportedLangFilter}"; + if (request.Lang != null) + sql += " AND f.lang = @lang"; + DbReader.AppendPathFilters( + ref sql, + request.PathPatterns, + request.ExcludePathPatterns, + request.ExcludeTests); + return sql + " ORDER BY s.id LIMIT @familyMemberLimit"; + } + } +} diff --git a/src/CodeIndex/Database/DbReader.ImpactDefinitionProjection.cs b/src/CodeIndex/Database/DbReader.ImpactDefinitionProjection.cs new file mode 100644 index 0000000000..a24279df49 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.ImpactDefinitionProjection.cs @@ -0,0 +1,161 @@ +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private readonly record struct ImpactDefinitionStats( + int PhysicalCount, + int PhysicalFileCount, + int LogicalCount, + int PreciseDefinitionCount, + int PreciseLogicalDefinitionCount, + int PreciseDefinitionFileCount, + int NonCallableDefinitionCount); + + private sealed record ImpactDefinitionProjection( + List Definitions, + ImpactDefinitionStats Stats, + SymbolResult? SinglePreciseDefinition); + + private static class ImpactDefinitionRowProjector + { + private static class Column + { + public const int Path = 0; + public const int Lang = 1; + public const int Kind = 2; + public const int Name = 3; + public const int Line = 4; + public const int StartLine = 5; + public const int StartColumn = 6; + public const int EndLine = 7; + public const int BodyStartLine = 8; + public const int BodyEndLine = 9; + public const int Signature = 10; + public const int ContainerKind = 11; + public const int ContainerName = 12; + public const int Visibility = 13; + public const int ReturnType = 14; + public const int ContainerQualifiedName = 15; + public const int LogicalPartialKey = 16; + public const int SymbolId = 17; + public const int DefinitionSites = 18; + public const int RequestedRow = 19; + public const int PhysicalCount = 20; + public const int PhysicalFileCount = 21; + public const int LogicalCount = 22; + public const int PreciseCount = 23; + public const int PreciseFileCount = 24; + public const int NonCallableCount = 25; + public const int RepresentativeReason = 26; + public const int FamilyMembersJson = 27; + public const int FamilyMembersTruncated = 28; + public const int IdentifierStartColumn = 29; + public const int PreciseLogicalCount = 30; + } + + public static ImpactDefinitionProjection Read(SqliteCommand cmd) + { + var definitions = new List(); + var stats = EmptyStats; + SymbolResult? preciseDefinition = null; + using var reader = cmd.ExecuteTrackedReader(); + while (reader.TrackedRead()) + { + var row = ReadRow(reader); + stats = ReadStats(reader); + if (IsPreciseImpactFallbackKind(row.Result.Kind)) + preciseDefinition ??= row.Result; + if (row.Requested) + definitions.Add(row.Result); + } + + return new ImpactDefinitionProjection( + definitions, + stats, + preciseDefinition); + } + + private static readonly ImpactDefinitionStats EmptyStats = + new(0, 0, 0, 0, 0, 0, 0); + + private static (SymbolResult Result, bool Requested) ReadRow( + SqliteDataReader reader) + { + var definitionSites = reader.GetInt32(Column.DefinitionSites); + var result = new SymbolResult + { + Path = reader.GetString(Column.Path), + Lang = reader.GetString(Column.Lang), + Kind = reader.GetString(Column.Kind), + Name = reader.GetString(Column.Name), + Line = reader.GetInt32(Column.Line), + StartLine = ReadStartLine(reader), + StartColumn = ReadStartColumn(reader), + EndLine = ReadEndLine(reader), + BodyStartLine = GetNullableInt32(reader, Column.BodyStartLine), + BodyEndLine = GetNullableInt32(reader, Column.BodyEndLine), + Signature = GetNullableString(reader, Column.Signature), + ContainerKind = GetNullableString(reader, Column.ContainerKind), + ContainerName = GetNullableString(reader, Column.ContainerName), + ContainerQualifiedName = GetNullableString( + reader, + Column.ContainerQualifiedName), + LogicalPartialKey = GetNullableString(reader, Column.LogicalPartialKey), + Visibility = GetNullableString(reader, Column.Visibility), + ReturnType = GetNullableString(reader, Column.ReturnType), + SymbolId = reader.GetInt64(Column.SymbolId), + DefinitionSites = definitionSites > 1 ? definitionSites : null, + }; + AddFamily(reader, definitionSites, result); + return (result, reader.GetInt32(Column.RequestedRow) == 1); + } + + private static int ReadStartLine(SqliteDataReader reader) + => GetInt32OrFallback(reader, Column.StartLine, Column.Line); + + private static int ReadEndLine(SqliteDataReader reader) + => GetInt32OrFallback(reader, Column.EndLine, Column.Line); + + private static int? ReadStartColumn(SqliteDataReader reader) + { + return GetNullableInt32(reader, Column.IdentifierStartColumn) + ?? ResolveSymbolIdentifierStartColumn( + GetNullableInt32(reader, Column.StartColumn), + GetNullableString(reader, Column.Signature), + reader.GetString(Column.Name), + reader.GetString(Column.Kind)); + } + + private static void AddFamily( + SqliteDataReader reader, + int definitionSites, + SymbolResult result) + { + if (definitionSites <= 1) + return; + + result.PartialFamilyId = + LogicalPartialSymbolGrouper.BuildPartialFamilyId(result.LogicalPartialKey!); + result.RepresentativeReason = reader.GetString(Column.RepresentativeReason); + result.FamilyMembers = ReadPartialFamilyMembers( + reader.GetString(Column.FamilyMembersJson), + result); + result.FamilyMembersTruncated = + reader.GetInt64(Column.FamilyMembersTruncated) != 0; + } + + private static ImpactDefinitionStats ReadStats(SqliteDataReader reader) + { + return new ImpactDefinitionStats( + reader.GetInt32(Column.PhysicalCount), + reader.GetInt32(Column.PhysicalFileCount), + reader.GetInt32(Column.LogicalCount), + reader.GetInt32(Column.PreciseCount), + reader.GetInt32(Column.PreciseLogicalCount), + reader.GetInt32(Column.PreciseFileCount), + reader.GetInt32(Column.NonCallableCount)); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.ImpactDefinitionQuery.cs b/src/CodeIndex/Database/DbReader.ImpactDefinitionQuery.cs new file mode 100644 index 0000000000..d59e650634 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.ImpactDefinitionQuery.cs @@ -0,0 +1,239 @@ +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed record ImpactDefinitionColumnSql( + string StartLine, + string StartColumn, + string EndLine, + string BodyStartLine, + string BodyEndLine, + string Signature, + string ContainerKind, + string ContainerName, + string ContainerQualifiedName, + string FamilyKey, + string Visibility, + string ReturnType, + string IdentifierStartColumn); + + private sealed record ImpactDefinitionQueryPlan( + ImpactDefinitionRequest Request, + string Sql, + string LogicalPartialKeySql); + + private static class ImpactDefinitionQueryBuilder + { + public static ImpactDefinitionQueryPlan Build( + DbReader reader, + SqliteCommand cmd, + ImpactDefinitionRequest request) + { + var supportedLangFilter = reader.BuildGraphSupportedLanguagePredicate( + cmd, + "f", + "impactDefLang"); + var columns = BuildColumnSql(reader); + var canonical = LogicalPartialQuerySql.Build( + reader, + columns.Signature, + columns.ContainerName, + columns.ContainerQualifiedName, + columns.FamilyKey, + columns.ReturnType, + columns.BodyStartLine, + columns.BodyEndLine); + var matchingSql = BuildMatchingSql( + reader, + request, + columns, + canonical, + supportedLangFilter); + var pathCaseSensitive = ReferenceEquals( + reader.GetIndexedPathComparer(), + StringComparer.Ordinal); + var sql = ImpactDefinitionQuerySql.Build(matchingSql, pathCaseSensitive); + return new ImpactDefinitionQueryPlan( + request, + sql, + canonical.LogicalPartialKey); + } + + private static ImpactDefinitionColumnSql BuildColumnSql(DbReader reader) + { + return new ImpactDefinitionColumnSql( + reader.GetSymbolColumnSql("start_line", "s.line"), + reader.GetSymbolColumnSql("start_column"), + reader.GetSymbolColumnSql("end_line", "s.line"), + reader.GetSymbolColumnSql("body_start_line"), + reader.GetSymbolColumnSql("body_end_line"), + reader.GetSymbolColumnSql("signature"), + reader.GetSymbolColumnSql("container_kind"), + reader.GetSymbolColumnSql("container_name"), + reader.GetSymbolColumnSql("container_qualified_name"), + reader.GetSymbolColumnSql("family_key"), + reader.GetSymbolColumnSql("visibility"), + reader.GetSymbolColumnSql("return_type"), + reader.GetSymbolColumnSql("identifier_start_column")); + } + + private static string BuildMatchingSql( + DbReader reader, + ImpactDefinitionRequest request, + ImpactDefinitionColumnSql columns, + LogicalPartialCanonicalSql canonical, + string supportedLangFilter) + { + var nameCondition = BuildNameCondition(reader, request); + var matchOrderSql = BuildMatchOrderSql(); + var sql = $@" + SELECT f.path, f.lang, s.kind, s.name, s.line, + {columns.StartLine} AS start_line, + {columns.StartColumn} AS start_column, + {columns.EndLine} AS end_line, + {columns.BodyStartLine} AS body_start_line, + {columns.BodyEndLine} AS body_end_line, + {columns.Signature} AS signature, + {columns.ContainerKind} AS container_kind, + {columns.ContainerName} AS container_name, + {columns.Visibility} AS visibility, + {columns.ReturnType} AS return_type, + {columns.ContainerQualifiedName} AS container_qualified_name, + {canonical.LogicalPartialKey} AS logical_partial_key, + s.id AS symbol_id, + {matchOrderSql} AS match_order, + {DbReader.PathBucketOrder} AS path_bucket, + {reader.VisibilityOrder} AS visibility_rank, + CASE WHEN s.kind IN ('class', 'struct', 'interface') THEN 1 ELSE 0 END AS is_precise, + CASE WHEN s.kind IN ('namespace', 'import') THEN 1 ELSE 0 END AS is_non_callable, + {canonical.PrimaryRank} AS canonical_primary_rank, + {canonical.Generated} AS canonical_generated_rank, + {canonical.SemanticScore} AS canonical_semantic_score, + {canonical.DeclarationIdentity} AS canonical_declaration_identity, + COALESCE({columns.StartColumn}, 2147483647) AS stable_start_column, + {columns.IdentifierStartColumn} AS identifier_start_column + FROM symbols s + JOIN files f ON s.file_id = f.id + WHERE {nameCondition} + AND {supportedLangFilter}"; + + if (request.Lang != null) + sql += " AND f.lang = @lang"; + DbReader.AppendPathFilters( + ref sql, + request.PathPatterns, + request.ExcludePathPatterns, + request.ExcludeTests); + return sql; + } + + private static string BuildNameCondition( + DbReader reader, + ImpactDefinitionRequest request) + { + var nameCondition = reader._foldReady + ? BuildFoldedNameCondition(request.AllowLeafFallback) + : BuildLegacyNameCondition(request.AllowLeafFallback); + var explicitInterfaceClause = request.AllowLeafFallback + ? reader.BuildCSharpExplicitInterfaceShortAliasMatchSql("resolvedName") + : reader.BuildCSharpExplicitInterfaceIdentityMatchSql("resolvedName"); + nameCondition = $"({nameCondition} OR {explicitInterfaceClause})"; + if (request.AllowLeafFallback) + return nameCondition; + + var csharpLeafCondition = reader._foldReady + ? "s.name_folded = @resolvedNameLeafFolded" + : "s.name = @resolvedNameLeaf COLLATE NOCASE"; + var containerName = reader.GetSymbolColumnSql("container_name", "''"); + var containerQualifiedName = reader.GetSymbolColumnSql( + "container_qualified_name", + containerName); + return $"({nameCondition} OR (f.lang = 'csharp' AND {csharpLeafCondition} AND ({containerName} = @resolvedNameContainer COLLATE NOCASE OR {containerQualifiedName} = @resolvedNameContainer COLLATE NOCASE OR {containerQualifiedName} COLLATE NOCASE LIKE @resolvedNameContainerSuffix ESCAPE '\\')))"; + } + + private static string BuildFoldedNameCondition(bool allowLeafFallback) + { + return allowLeafFallback + ? $"({DbReader.BuildPersistedFoldedNameMatchSql("s.name_folded", "@resolvedNameFolded")} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name_folded(s.name) = @resolvedNameNormalizedFolded) OR sql_leaf_name_folded(s.name) = @resolvedNameLeafFolded)))" + : $"({DbReader.BuildPersistedFoldedNameMatchSql("s.name_folded", "@resolvedNameFolded")} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name_folded(s.name) = @resolvedNameNormalizedFolded))"; + } + + private static string BuildLegacyNameCondition(bool allowLeafFallback) + { + return allowLeafFallback + ? "(s.name = @resolvedName COLLATE NOCASE OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name(s.name) = @resolvedNameNormalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @resolvedNameLeaf COLLATE NOCASE)))" + : "(s.name = @resolvedName COLLATE NOCASE OR (f.lang = 'sql' AND sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name(s.name) = @resolvedNameNormalized COLLATE NOCASE))"; + } + + private static string BuildMatchOrderSql() + { + return @"CASE + WHEN s.name = @resolvedName THEN 0 + WHEN f.lang = 'sql' AND sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name(s.name) = @resolvedNameNormalized THEN 1 + WHEN f.lang = 'sql' AND sql_segment_count(s.name) = @resolvedNameSegmentCount AND sql_normalize_name_folded(s.name) = @resolvedNameNormalizedFolded THEN 2 + WHEN @allowLeafFallback = 1 AND f.lang = 'sql' AND sql_leaf_name(s.name) = @resolvedNameLeaf THEN 3 + WHEN @allowLeafFallback = 1 AND f.lang = 'sql' AND sql_leaf_name_folded(s.name) = @resolvedNameLeafFolded THEN 4 + ELSE 5 + END"; + } + + public static void Bind( + DbReader reader, + SqliteCommand cmd, + ImpactDefinitionQueryPlan plan) + { + var request = plan.Request; + SqliteCommandPolicy.Add(cmd, "@resolvedName", request.ResolvedName); + SqliteCommandPolicy.Add(cmd, "@resolvedNameNormalized", request.NormalizedName); + SqliteCommandPolicy.Add( + cmd, + "@resolvedNameNormalizedFolded", + DbReader.FoldNameForLanguage(request.NormalizedName, request.Lang)); + SqliteCommandPolicy.Add(cmd, "@resolvedNameLeaf", request.LeafName); + SqliteCommandPolicy.Add( + cmd, + "@resolvedNameLeafFolded", + DbReader.FoldNameForLanguage(request.LeafName, request.Lang)); + SqliteCommandPolicy.Add(cmd, "@resolvedNameSegmentCount", request.SegmentCount); + SqliteCommandPolicy.Add(cmd, "@allowLeafFallback", request.AllowLeafFallback ? 1 : 0); + DbReader.AddCSharpExplicitInterfaceIdentityQueryParameter( + cmd, + "resolvedName", + request.ResolvedName); + BindQualifiedContainer(cmd, request); + if (reader._foldReady) + { + DbReader.AddPersistedFoldedNameQueryParameters( + cmd, + "@resolvedNameFolded", + request.ResolvedName, + request.Lang); + } + if (request.Lang != null) + SqliteCommandPolicy.Add(cmd, "@lang", request.Lang); + SqliteCommandPolicy.Add(cmd, "@definitionLimit", request.RepresentativeLimit); + SqliteCommandPolicy.Add(cmd, "@definitionOffset", request.RepresentativeOffset); + DbReader.AddPathFilterParameters( + cmd, + request.PathPatterns, + request.ExcludePathPatterns); + } + + private static void BindQualifiedContainer( + SqliteCommand cmd, + ImpactDefinitionRequest request) + { + if (request.AllowLeafFallback) + return; + + var container = GetQualifiedQueryContainer(request.ResolvedName); + SqliteCommandPolicy.Add(cmd, "@resolvedNameContainer", container); + SqliteCommandPolicy.Add( + cmd, + "@resolvedNameContainerSuffix", + $"%.{EscapeLikeQuery(container)}"); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.ImpactDefinitionQuerySql.cs b/src/CodeIndex/Database/DbReader.ImpactDefinitionQuerySql.cs new file mode 100644 index 0000000000..3788dd2966 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.ImpactDefinitionQuerySql.cs @@ -0,0 +1,173 @@ +using System.Text; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private static class ImpactDefinitionQuerySql + { + private const string CanonicalRepresentativeOrder = + "canonical_primary_rank, canonical_generated_rank, canonical_semantic_score DESC, canonical_declaration_identity COLLATE BINARY, path COLLATE BINARY, start_line, stable_start_column, symbol_id"; + private const string ResultOrder = + "match_order, path_bucket, visibility_rank, name, path COLLATE BINARY, line, symbol_id"; + + public static string Build(string matchingSql, bool pathCaseSensitive) + { + var pathDistinctSql = pathCaseSensitive + ? "COUNT(DISTINCT path)" + : "COUNT(DISTINCT path COLLATE NOCASE)"; + var precisePathDistinctSql = pathCaseSensitive + ? "COUNT(DISTINCT CASE WHEN is_precise = 1 THEN path END)" + : "COUNT(DISTINCT CASE WHEN is_precise = 1 THEN path END COLLATE NOCASE)"; + var sql = new StringBuilder(); + sql.Append($@" + WITH matching_definitions AS ( + {matchingSql} + ), + "); + AppendRankingCtes(sql); + sql.Append(@" + "); + AppendSelectionCtes(sql, pathDistinctSql, precisePathDistinctSql); + sql.Append(@" + "); + AppendFinalSelect(sql); + return sql.ToString(); + } + + private static void AppendRankingCtes(StringBuilder sql) + { + sql.Append($@"ranked_definitions AS ( + SELECT matching_definitions.*, + ROW_NUMBER() OVER ( + PARTITION BY logical_partial_key + ORDER BY {CanonicalRepresentativeOrder} + ) AS logical_row_number, + ROW_NUMBER() OVER ( + PARTITION BY logical_partial_key + ORDER BY path COLLATE BINARY, start_line, stable_start_column, symbol_id + ) AS family_member_row_number, + COUNT(*) OVER (PARTITION BY logical_partial_key) AS logical_definition_sites + FROM matching_definitions + ), + family_ranked_definitions AS ( + SELECT ranked_definitions.*, + MAX(CASE WHEN logical_row_number = 1 THEN family_member_row_number END) OVER ( + PARTITION BY logical_partial_key + ) AS representative_member_row_number + FROM ranked_definitions + ), + family_metadata_definitions AS ( + SELECT family_ranked_definitions.*, + MIN(canonical_primary_rank) OVER (PARTITION BY logical_partial_key) AS logical_primary_rank_min, + MAX(canonical_primary_rank) OVER (PARTITION BY logical_partial_key) AS logical_primary_rank_max, + MIN(canonical_generated_rank) OVER (PARTITION BY logical_partial_key) AS logical_generated_rank_min, + MAX(canonical_generated_rank) OVER (PARTITION BY logical_partial_key) AS logical_generated_rank_max, + MIN(canonical_semantic_score) OVER (PARTITION BY logical_partial_key) AS logical_semantic_score_min, + MAX(canonical_semantic_score) OVER (PARTITION BY logical_partial_key) AS logical_semantic_score_max, + MIN(canonical_declaration_identity) OVER (PARTITION BY logical_partial_key) AS logical_declaration_identity_min, + MAX(canonical_declaration_identity) OVER (PARTITION BY logical_partial_key) AS logical_declaration_identity_max, + json_group_array(json_object( + 'symbol_id', symbol_id, + 'path', path, + 'line', line, + 'start_line', start_line, + 'start_column', start_column, + 'end_line', end_line, + 'name', name, + 'signature', signature, + 'identifier_start_column', identifier_start_column, + 'generated', canonical_generated_rank + )) FILTER (WHERE + family_member_row_number <= CASE + WHEN representative_member_row_number <= {LogicalPartialSymbolGrouper.FamilyMemberLimit} + THEN {LogicalPartialSymbolGrouper.FamilyMemberLimit} + ELSE {LogicalPartialSymbolGrouper.FamilyMemberLimit - 1} + END + OR logical_row_number = 1 + ) OVER ( + PARTITION BY logical_partial_key + ORDER BY path COLLATE BINARY, start_line, stable_start_column, symbol_id + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ) AS logical_family_members_json + FROM family_ranked_definitions + ),"); + } + + private static void AppendSelectionCtes( + StringBuilder sql, + string pathDistinctSql, + string precisePathDistinctSql) + { + sql.Append($@"logical_definitions AS ( + SELECT * + FROM family_metadata_definitions + WHERE logical_row_number = 1 + ), + requested_definitions AS ( + SELECT logical_partial_key, 1 AS requested_row + FROM logical_definitions + ORDER BY {ResultOrder} + LIMIT @definitionLimit OFFSET @definitionOffset + ), + single_precise_definition AS ( + SELECT logical_partial_key, 0 AS requested_row + FROM logical_definitions + WHERE is_precise = 1 + ORDER BY {ResultOrder} + LIMIT 1 + ), + selected_definition_keys AS ( + SELECT logical_partial_key, requested_row + FROM requested_definitions + UNION ALL + SELECT precise.logical_partial_key, precise.requested_row + FROM single_precise_definition precise + WHERE NOT EXISTS ( + SELECT 1 + FROM requested_definitions requested + WHERE requested.logical_partial_key = precise.logical_partial_key + ) + ), + definition_stats AS ( + SELECT COUNT(*) AS physical_count, + {pathDistinctSql} AS physical_file_count, + COUNT(DISTINCT logical_partial_key) AS logical_count, + SUM(is_precise) AS precise_count, + COUNT(DISTINCT CASE WHEN is_precise = 1 THEN logical_partial_key END) AS precise_logical_count, + {precisePathDistinctSql} AS precise_file_count, + SUM(is_non_callable) AS non_callable_count + FROM matching_definitions + )"); + } + + private static void AppendFinalSelect(StringBuilder sql) + { + sql.Append($@"SELECT logical.path, logical.lang, logical.kind, logical.name, logical.line, + logical.start_line, logical.start_column, logical.end_line, + logical.body_start_line, logical.body_end_line, logical.signature, + logical.container_kind, logical.container_name, + logical.visibility, logical.return_type, + logical.container_qualified_name, logical.logical_partial_key, + logical.symbol_id, logical.logical_definition_sites, selected.requested_row, + stats.physical_count, stats.physical_file_count, stats.logical_count, + stats.precise_count, stats.precise_file_count, stats.non_callable_count, + CASE + WHEN logical.logical_primary_rank_min <> logical.logical_primary_rank_max THEN '{LogicalPartialSymbolGrouper.ImplementationBodyReason}' + WHEN logical.logical_generated_rank_min <> logical.logical_generated_rank_max THEN '{LogicalPartialSymbolGrouper.NonGeneratedSourceReason}' + WHEN logical.logical_semantic_score_min <> logical.logical_semantic_score_max THEN '{LogicalPartialSymbolGrouper.SemanticDeclarationReason}' + WHEN logical.logical_declaration_identity_min <> logical.logical_declaration_identity_max THEN '{LogicalPartialSymbolGrouper.CanonicalDeclarationIdentityReason}' + ELSE '{LogicalPartialSymbolGrouper.StableLocationReason}' + END AS representative_reason, + logical.logical_family_members_json, + CASE WHEN logical.logical_definition_sites > {LogicalPartialSymbolGrouper.FamilyMemberLimit} THEN 1 ELSE 0 END AS family_members_truncated, + logical.identifier_start_column, + stats.precise_logical_count + FROM selected_definition_keys selected + JOIN logical_definitions logical + ON logical.logical_partial_key = selected.logical_partial_key + CROSS JOIN definition_stats stats + ORDER BY {ResultOrder}"); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.ImpactDefinitions.cs b/src/CodeIndex/Database/DbReader.ImpactDefinitions.cs new file mode 100644 index 0000000000..baa6999139 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.ImpactDefinitions.cs @@ -0,0 +1,106 @@ +using CodeIndex.Indexer; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed record ImpactDefinitionRequest( + string ResolvedName, + string NormalizedName, + string LeafName, + int SegmentCount, + bool AllowLeafFallback, + int RepresentativeLimit, + int RepresentativeOffset, + string? Lang, + IReadOnlyList? PathPatterns, + IReadOnlyList? ExcludePathPatterns, + bool ExcludeTests) + { + public static ImpactDefinitionRequest Create( + string resolvedName, + int representativeLimit, + int representativeOffset, + string? lang, + IReadOnlyList? pathPatterns, + IReadOnlyList? excludePathPatterns, + bool excludeTests) + { + return new ImpactDefinitionRequest( + resolvedName, + SqlNameResolver.NormalizeQualifiedName(resolvedName), + SqlNameResolver.GetLeafName(resolvedName), + SqlNameResolver.GetSegmentCount(resolvedName), + !SqlNameResolver.HasQualifier(resolvedName), + Math.Max(1, representativeLimit), + Math.Max(0, representativeOffset), + lang, + pathPatterns, + excludePathPatterns, + excludeTests); + } + } + + private sealed record ImpactDefinitionResolution( + List Definitions, + int PhysicalCount, + int PhysicalFileCount, + int LogicalCount, + int PreciseDefinitionCount, + int PreciseLogicalDefinitionCount, + int PreciseDefinitionFileCount, + int NonCallableDefinitionCount, + SymbolResult? SinglePreciseDefinition, + HashSet PhysicalSymbolIds, + HashSet PhysicalDefinitionPaths, + bool PhysicalSymbolIdsTruncated); + + private ImpactDefinitionResolution ResolveImpactDefinitions( + string resolvedName, + int representativeLimit, + string? lang, + IReadOnlyList? pathPatterns, + IReadOnlyList? excludePathPatterns, + bool excludeTests, + int representativeOffset = 0) + { + var request = ImpactDefinitionRequest.Create( + resolvedName, + representativeLimit, + representativeOffset, + lang, + pathPatterns, + excludePathPatterns, + excludeTests); + EnsureCSharpCallableTypeKinds(request.Lang, [request.LeafName], exact: true); + + ImpactDefinitionProjection projection; + ImpactDefinitionQueryPlan plan; + using (var cmd = _conn.CreateCommand()) + { + plan = ImpactDefinitionQueryBuilder.Build(this, cmd, request); + cmd.CommandText = plan.Sql; + ImpactDefinitionQueryBuilder.Bind(this, cmd, plan); + projection = ImpactDefinitionRowProjector.Read(cmd); + } + + var families = ImpactDefinitionFamilyResolver.Resolve( + this, + plan, + projection.Definitions); + var stats = projection.Stats; + return new ImpactDefinitionResolution( + projection.Definitions, + stats.PhysicalCount, + stats.PhysicalFileCount, + stats.LogicalCount, + stats.PreciseDefinitionCount, + stats.PreciseLogicalDefinitionCount, + stats.PreciseDefinitionFileCount, + stats.NonCallableDefinitionCount, + stats.PreciseLogicalDefinitionCount == 1 ? projection.SinglePreciseDefinition : null, + families.PhysicalSymbolIds, + families.PhysicalDefinitionPaths, + families.Truncated); + } +} diff --git a/src/CodeIndex/Database/DbReader.ImpactTraversal.Cycles.cs b/src/CodeIndex/Database/DbReader.ImpactTraversal.Cycles.cs new file mode 100644 index 0000000000..d2a013d5d7 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.ImpactTraversal.Cycles.cs @@ -0,0 +1,221 @@ +using CodeIndex.Models; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private readonly record struct ImpactCycleNode(string Key, long? SymbolId, string Name); + + private readonly record struct ImpactCycleEdge(ImpactCycleNode Caller, ImpactCycleNode Callee); + + private sealed class ImpactCycleTracker + { + private readonly bool _hasResolvedIdentityGraph; + private readonly IReadOnlySet? _logicalRootSymbolIds; + private readonly string _logicalRootKey; + private readonly string _logicalRootName; + private readonly Dictionary> _parentsByKey = new(StringComparer.Ordinal); + private readonly Dictionary _nodesByKey = new(StringComparer.Ordinal); + private readonly HashSet _cycleKeys = new(StringComparer.Ordinal); + + internal ImpactCycleTracker(ImpactTraversalRoot root) + { + _hasResolvedIdentityGraph = root.HasResolvedIdentityGraph; + _logicalRootSymbolIds = root.IsLogicalPartialFamily ? root.IdentitySymbolIds : null; + _logicalRootKey = root.NodeKey; + _logicalRootName = root.ResolvedName; + } + + internal List Results { get; } = []; + + internal int GraphStateEntryCount + { + get + { + var count = _parentsByKey.Count; + foreach (var parents in _parentsByKey.Values) + count += parents.Count; + return count; + } + } + + internal IReadOnlyList Observe( + CallerResult caller, + string callerName, + string calleeName) + { + var edges = BuildEdges(caller, callerName, calleeName); + foreach (var edge in edges) + { + RegisterNode(edge.Caller); + RegisterNode(edge.Callee); + if (IsCycleEdge(edge.Caller.Key, edge.Callee.Key)) + AddCycle(BuildCycleMembers(edge.Caller.Key, edge.Callee.Key)); + } + return edges; + } + + internal void CommitParents(IReadOnlyList edges) + { + foreach (var edge in edges) + { + if (!_parentsByKey.TryGetValue(edge.Caller.Key, out var parents)) + { + parents = new HashSet(StringComparer.Ordinal); + _parentsByKey[edge.Caller.Key] = parents; + } + parents.Add(edge.Callee.Key); + } + } + + private List BuildEdges( + CallerResult caller, + string callerName, + string calleeName) + { + var callerNode = NormalizeLogicalRoot( + ImpactNodeIdentity.CycleNode( + caller.CallerSymbolId, + callerName, + _hasResolvedIdentityGraph)); + if (callerNode is not { } canonicalCaller) + return []; + + if (!_hasResolvedIdentityGraph) + { + var calleeNode = ImpactNodeIdentity.CycleNode(null, calleeName, false)!.Value; + return [new ImpactCycleEdge(canonicalCaller, calleeNode)]; + } + + var calleeSymbolIds = caller.CalleeSymbolIds.Count > 0 + ? caller.CalleeSymbolIds + : caller.CalleeSymbolId is long calleeSymbolId + ? [calleeSymbolId] + : Array.Empty(); + return calleeSymbolIds + .Distinct() + .Order() + .Select(calleeSymbolId => new ImpactCycleEdge( + canonicalCaller, + NormalizeLogicalRoot(new ImpactCycleNode( + $"id:{calleeSymbolId}", + calleeSymbolId, + calleeName))!.Value)) + .ToList(); + } + + private ImpactCycleNode? NormalizeLogicalRoot(ImpactCycleNode? node) + { + if (node is not { SymbolId: long symbolId } + || _logicalRootSymbolIds is not { Count: > 0 } + || !_logicalRootSymbolIds.Contains(symbolId)) + { + return node; + } + return new ImpactCycleNode(_logicalRootKey, null, _logicalRootName); + } + + private void RegisterNode(ImpactCycleNode node) + => _nodesByKey.TryAdd(node.Key, new ImpactCycleMemberResult + { + SymbolId = node.SymbolId, + Name = node.Name, + }); + + private bool IsCycleEdge(string callerKey, string calleeKey) + => string.Equals(callerKey, calleeKey, StringComparison.Ordinal) + || HasAncestor(calleeKey, callerKey); + + private bool HasAncestor(string node, string target) + { + var stack = new Stack(); + var seen = new HashSet(StringComparer.Ordinal); + stack.Push(node); + while (stack.Count > 0) + { + var current = stack.Pop(); + if (!seen.Add(current)) + continue; + if (string.Equals(current, target, StringComparison.Ordinal)) + return true; + if (!_parentsByKey.TryGetValue(current, out var parents)) + continue; + foreach (var parent in parents) + stack.Push(parent); + } + return false; + } + + private List BuildCycleMembers(string callerKey, string calleeKey) + { + var members = new HashSet(StringComparer.Ordinal); + if (!TryBuildAncestorPath(calleeKey, callerKey, members)) + { + members.Add(callerKey); + members.Add(calleeKey); + } + var result = members.ToList(); + result.Sort(StringComparer.Ordinal); + return result; + } + + private bool TryBuildAncestorPath( + string node, + string target, + HashSet members) + { + var seen = new HashSet(StringComparer.Ordinal); + return TryBuildAncestorPathCore(node, target, members, seen); + } + + private bool TryBuildAncestorPathCore( + string node, + string target, + HashSet members, + HashSet seen) + { + if (!seen.Add(node)) + return false; + members.Add(node); + if (string.Equals(node, target, StringComparison.Ordinal)) + return true; + if (_parentsByKey.TryGetValue(node, out var parents)) + { + foreach (var parent in parents) + { + if (TryBuildAncestorPathCore(parent, target, members, seen)) + return true; + } + } + + members.Remove(node); + return false; + } + + private void AddCycle(List memberKeys) + { + if (memberKeys.Count == 0) + return; + var key = string.Join("\u001F", memberKeys); + if (!_cycleKeys.Add(key)) + return; + var identities = memberKeys + .Select(memberKey => _nodesByKey[memberKey]) + .OrderBy(static node => node.Name, StringComparer.OrdinalIgnoreCase) + .ThenBy(static node => node.SymbolId) + .Select(static node => new ImpactCycleMemberResult + { + SymbolId = node.SymbolId, + Name = node.Name, + }) + .ToList(); + Results.Add(new ImpactCycleResult + { + Members = identities.Select(static identity => identity.Name).ToList(), + MemberIdentities = identities.Any(static identity => identity.SymbolId != null) + ? identities + : null, + }); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.ImpactTraversal.Engine.cs b/src/CodeIndex/Database/DbReader.ImpactTraversal.Engine.cs new file mode 100644 index 0000000000..5b1e6baaf3 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.ImpactTraversal.Engine.cs @@ -0,0 +1,266 @@ +using CodeIndex.Models; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class ImpactTraversalEngine + { + private const int MaxFetchIterations = 1000; + + private readonly DbReader _owner; + private readonly ImpactTraversalRequest _request; + private readonly ImpactTraversalRoot _root; + private readonly ImpactTraversalState _state; + + internal ImpactTraversalEngine( + DbReader owner, + ImpactTraversalRequest request, + ImpactTraversalRoot root) + { + _owner = owner; + _request = request; + _root = root; + _state = new ImpactTraversalState(owner, request, root); + } + + internal (List Results, bool Truncated, string? TruncatedReason, string TerminationReason, List Cycles) Run() + { + while (_state.CanTraverse) + { + var node = _state.Queue.Dequeue(); + TraverseNode(in node); + } + + _state.CompleteTraversal(); + _state.Paths.Materialize(_state.Results, _request.MaxPathsPerResult); + return ( + _state.Results, + _state.Truncated, + _state.TruncatedReason, + _state.ResolveTerminationReason(), + _state.Cycles.Results); + } + + private void TraverseNode(in ImpactTraversalFrontierNode node) + { + var needed = _state.ResultWindowEnd - _state.DiscoveredResultCount; + var pageSize = Math.Max(1, needed + 1); + var pageOffset = 0; + var fetchIterations = 0; + + while (_state.CanFetchCurrentNode && fetchIterations < MaxFetchIterations) + { + fetchIterations++; + var page = _owner.ReadImpactCallerPage( + in node, + _request, + pageSize, + pageOffset, + _root.IncludeAmbiguousMSource); + if (page.Count == 0) + break; + + ProcessPage(in node, page); + pageOffset += page.Count; + if (page.Count < pageSize) + break; + } + + if (fetchIterations >= MaxFetchIterations) + _state.MarkSafetyCap(); + } + + private void ProcessPage( + in ImpactTraversalFrontierNode node, + IReadOnlyList page) + { + foreach (var caller in page) + { + if (_state.DiscoveredResultCount >= _state.ResultWindowEnd) + { + _state.MarkUserLimit(); + break; + } + if (!ProcessCaller(in node, caller)) + break; + } + } + + private bool ProcessCaller( + in ImpactTraversalFrontierNode node, + CallerResult caller) + { + var callerName = caller.CallerName ?? SyntheticTopLevelCallerName; + var callerSymbolId = _root.HasResolvedIdentityGraph ? caller.CallerSymbolId : null; + var calleeSymbolId = _root.HasResolvedIdentityGraph ? caller.CalleeSymbolId : null; + var cycleEdges = _state.Cycles.Observe(caller, callerName, node.Symbol); + if (IsRootCaller(caller, callerName)) + return true; + + var callerNodeKey = ImpactNodeIdentity.TraversalKey(callerSymbolId, callerName); + var visitedKey = ImpactNodeIdentity.VisitedKey( + caller, + callerName, + _root.HasResolvedIdentityGraph, + _root.IsLogicalPartialFamily); + _state.Cycles.CommitParents(cycleEdges); + if (_state.CheckGraphStateBudget()) + return false; + + if (!_state.TryVisit(visitedKey)) + { + _state.MergeReferenceEvidence(visitedKey, caller); + if (_state.Paths.RecordSameDepthParent( + callerNodeKey, + node.NodeKey, + node.Depth + 1) + && _state.CheckGraphStateBudget()) + { + return false; + } + return true; + } + + var result = _state.IncludeNextResult + ? BuildResult(caller, callerSymbolId, calleeSymbolId, node.Depth + 1) + : null; + var resultIndex = _state.AddResult(result, visitedKey); + _state.Paths.RecordCaller( + callerNodeKey, + node.NodeKey, + node.Depth + 1, + resultIndex, + caller, + callerName, + callerSymbolId, + _request.Lang); + if (_state.CheckGraphStateBudget()) + return false; + + return ContinueTraversal(in node, caller, callerSymbolId, callerNodeKey); + } + + private bool ContinueTraversal( + in ImpactTraversalFrontierNode node, + CallerResult caller, + long? callerSymbolId, + string callerNodeKey) + { + if (caller.CallerName != null + && caller.CallerName != SyntheticTopLevelCallerName + && node.Depth + 1 < _request.MaxDepth) + { + _state.Queue.Enqueue(new ImpactTraversalFrontierNode( + caller.CallerName, + callerSymbolId, + TargetSymbolIds: null, + callerNodeKey, + node.Depth + 1)); + } + else if (caller.CallerName != null + && caller.CallerName != SyntheticTopLevelCallerName + && node.Depth + 1 == _request.MaxDepth) + { + var inspection = InspectBoundary(caller.CallerName, callerSymbolId); + _state.MaxDepthReached |= inspection.HasUnvisitedCaller; + if (inspection.ProbeBudgetHit) + { + _state.MarkBoundaryProbeBudget(); + return false; + } + } + return true; + } + + private ImpactBoundaryInspection InspectBoundary( + string symbolName, + long? symbolId) + { + var pageOffset = 0; + var probes = 0; + while (true) + { + if (probes >= _state.BoundaryProbeBudget) + return new ImpactBoundaryInspection(true, true); + + var pageSize = Math.Min( + ImpactBoundaryCallerProbePageSize, + _state.BoundaryProbeBudget - probes); + var node = new ImpactTraversalFrontierNode( + symbolName, + symbolId, + TargetSymbolIds: null, + NodeKey: string.Empty, + Depth: 0); + var page = _owner.ReadImpactCallerPage( + in node, + _request, + pageSize, + pageOffset, + _root.IncludeAmbiguousMSource); + if (page.Count == 0) + return new ImpactBoundaryInspection(false, false); + probes += page.Count; + + foreach (var caller in page) + { + var callerName = caller.CallerName ?? SyntheticTopLevelCallerName; + var cycleEdges = _state.Cycles.Observe(caller, callerName, symbolName); + if (IsRootCaller(caller, callerName)) + continue; + + _state.Cycles.CommitParents(cycleEdges); + var visitedKey = ImpactNodeIdentity.VisitedKey( + caller, + callerName, + _root.HasResolvedIdentityGraph, + _root.IsLogicalPartialFamily); + if (!_state.Visited.Contains(visitedKey)) + return new ImpactBoundaryInspection(true, false); + } + + if (page.Count < pageSize) + return new ImpactBoundaryInspection(false, false); + pageOffset += page.Count; + } + } + + private bool IsRootCaller(CallerResult caller, string callerName) + { + if (_root.IdentitySymbolIds.Count > 0 + && caller.CallerSymbolId is long callerSymbolId) + { + return _root.IdentitySymbolIds.Contains(callerSymbolId); + } + return string.Equals( + callerName, + _root.ResolvedName, + StringComparison.OrdinalIgnoreCase) + && (_root.DefinitionPaths.Count == 0 + || _root.DefinitionPaths.Contains(caller.Path)); + } + + private static ImpactResult BuildResult( + CallerResult caller, + long? callerSymbolId, + long? calleeSymbolId, + int depth) + => new() + { + Path = caller.Path, + Lang = caller.Lang, + CallerKind = caller.CallerKind, + CallerName = caller.CallerName, + CalleeName = caller.CalleeName, + CallerSymbolId = callerSymbolId, + CalleeSymbolId = calleeSymbolId, + Depth = depth, + FirstLine = caller.FirstLine, + ReferenceCount = caller.ReferenceCount, + ReferenceKind = caller.ReferenceKind, + ReferenceKinds = caller.ReferenceKinds, + ReferenceKindCounts = caller.ReferenceKindCounts, + }; + } +} diff --git a/src/CodeIndex/Database/DbReader.ImpactTraversal.PathResolution.cs b/src/CodeIndex/Database/DbReader.ImpactTraversal.PathResolution.cs new file mode 100644 index 0000000000..06a2454ed8 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.ImpactTraversal.PathResolution.cs @@ -0,0 +1,125 @@ +using CodeIndex.Models; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private ImpactPathNode ResolveImpactPathNode( + string name, + long? symbolId, + string? kind, + string? lang, + string? referencePath, + int? referenceLine) + { + var node = TryResolveImpactPathNodeDefinition(name, symbolId, kind, lang, referencePath) + ?? new ImpactPathNode + { + SymbolId = symbolId, + Name = name, + Kind = kind, + Lang = lang, + }; + node.ReferencePath = referencePath; + node.ReferenceLine = referenceLine; + return node; + } + + private ImpactPathNode? TryResolveImpactPathNodeDefinition( + string name, + long? symbolId, + string? kind, + string? lang, + string? preferredPath) + { + if (!_symbolColumns.Contains("name") || !_symbolColumns.Contains("kind")) + return null; + + using var cmd = _conn.CreateCommand(); + var containerNameSql = GetSymbolColumnSql("container_name"); + var containerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name"); + var familyKeySql = GetSymbolColumnSql("family_key"); + var namePredicate = _foldReady && _symbolColumns.Contains("name_folded") + ? "s.name_folded = @nameFolded" + : "s.name = @name COLLATE NOCASE"; + + cmd.CommandText = $@" + SELECT s.id, + f.path, + f.lang, + s.kind, + s.name, + s.line, + {containerNameSql} AS container_name, + {containerQualifiedNameSql} AS container_qualified_name, + {familyKeySql} AS family_key, + s.file_id, + COUNT(*) OVER () AS matching_definition_count + FROM symbols s + JOIN files f ON s.file_id = f.id + WHERE ((@symbolId IS NOT NULL AND s.id = @symbolId) + OR (@symbolId IS NULL AND {namePredicate})) + AND s.kind NOT IN ('import', 'namespace') + AND (@kind IS NULL OR s.kind = @kind) + AND (@lang IS NULL OR f.lang = @lang) + ORDER BY CASE WHEN @preferredPath IS NOT NULL AND f.path = @preferredPath THEN 0 ELSE 1 END, + f.path, + s.line + LIMIT 1"; + SqliteCommandPolicy.Add(cmd, "@name", name); + SqliteCommandPolicy.AddNullableInt64(cmd, "@symbolId", symbolId); + if (_foldReady && _symbolColumns.Contains("name_folded")) + SqliteCommandPolicy.Add(cmd, "@nameFolded", NameFold.Fold(name) ?? name); + SqliteCommandPolicy.AddNullableText(cmd, "@kind", kind); + SqliteCommandPolicy.AddNullableText(cmd, "@lang", lang); + SqliteCommandPolicy.AddNullableText(cmd, "@preferredPath", preferredPath); + + using var reader = cmd.ExecuteTrackedReader(); + if (!reader.TrackedRead()) + return null; + + var definitionSymbolId = reader.GetInt64(0); + var definitionPath = reader.GetString(1); + var definitionLang = GetNullableString(reader, 2); + var definitionKind = reader.GetString(3); + var definitionName = reader.GetString(4); + var definitionLine = reader.IsDBNull(5) ? (int?)null : reader.GetInt32(5); + var containerName = GetNullableString(reader, 6); + var containerQualifiedName = GetNullableString(reader, 7); + var familyKey = GetNullableString(reader, 8); + var fileId = reader.GetInt64(9); + var matchingDefinitionCount = reader.GetInt64(10); + + return new ImpactPathNode + { + SymbolId = symbolId ?? (matchingDefinitionCount == 1 ? definitionSymbolId : null), + Name = definitionName, + Kind = definitionKind, + Lang = definitionLang, + DefinitionPath = definitionPath, + DefinitionLine = definitionLine, + Container = containerName, + FamilyKey = familyKey, + LogicalTargetKey = BuildImpactPathLogicalTargetKey( + definitionLang, + definitionKind, + familyKey, + containerQualifiedName, + fileId), + }; + } + + private static string BuildImpactPathLogicalTargetKey( + string? lang, + string kind, + string? familyKey, + string? containerQualifiedName, + long fileId) + { + if (!string.IsNullOrWhiteSpace(familyKey)) + return $"family|{lang ?? string.Empty}|{kind}|{familyKey}"; + if (!string.IsNullOrWhiteSpace(containerQualifiedName)) + return $"container|{fileId}|{kind}|{containerQualifiedName}"; + return $"file|{fileId}"; + } +} diff --git a/src/CodeIndex/Database/DbReader.ImpactTraversal.Paths.cs b/src/CodeIndex/Database/DbReader.ImpactTraversal.Paths.cs new file mode 100644 index 0000000000..93d8b39f2b --- /dev/null +++ b/src/CodeIndex/Database/DbReader.ImpactTraversal.Paths.cs @@ -0,0 +1,228 @@ +using CodeIndex.Models; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class ImpactPathTracker + { + private readonly DbReader _owner; + private readonly bool _withPaths; + private readonly string _rootNodeKey; + private readonly Dictionary> _parentsByNodeKey = new(StringComparer.OrdinalIgnoreCase); + private readonly Dictionary _depthByNodeKey = new(StringComparer.OrdinalIgnoreCase); + private readonly Dictionary>? _resultIndicesByNodeKey; + private readonly Dictionary? _nodesByKey; + + internal ImpactPathTracker(DbReader owner, bool withPaths, ImpactTraversalRoot root) + { + _owner = owner; + _withPaths = withPaths; + _rootNodeKey = root.NodeKey; + _depthByNodeKey[root.NodeKey] = 0; + if (!withPaths) + return; + + _resultIndicesByNodeKey = new Dictionary>(StringComparer.OrdinalIgnoreCase); + _nodesByKey = new Dictionary(StringComparer.OrdinalIgnoreCase) + { + [root.NodeKey] = root.PathNode!, + }; + } + + internal int GraphStateEntryCount + { + get + { + var count = _depthByNodeKey.Count + + _parentsByNodeKey.Count + + (_resultIndicesByNodeKey?.Count ?? 0); + foreach (var parents in _parentsByNodeKey.Values) + count += parents.Count; + if (_resultIndicesByNodeKey != null) + { + foreach (var indices in _resultIndicesByNodeKey.Values) + count += indices.Count; + } + return count; + } + } + + internal bool RecordSameDepthParent( + string callerNodeKey, + string currentNodeKey, + int callerDepth) + { + if (!_withPaths + || !_depthByNodeKey.TryGetValue(callerNodeKey, out var existingDepth) + || existingDepth != callerDepth) + { + return false; + } + + _parentsByNodeKey[callerNodeKey].Add(currentNodeKey); + return true; + } + + internal void RecordCaller( + string callerNodeKey, + string currentNodeKey, + int callerDepth, + int resultIndex, + CallerResult caller, + string callerName, + long? callerSymbolId, + string? requestLang) + { + if (_withPaths) + { + _nodesByKey!.TryAdd( + callerNodeKey, + _owner.ResolveImpactPathNode( + callerName, + callerSymbolId, + caller.CallerKind, + caller.Lang ?? requestLang, + caller.Path, + caller.FirstLine)); + _depthByNodeKey.TryAdd(callerNodeKey, callerDepth); + if (resultIndex >= 0) + { + if (!_resultIndicesByNodeKey!.TryGetValue(callerNodeKey, out var indices)) + { + indices = []; + _resultIndicesByNodeKey[callerNodeKey] = indices; + } + indices.Add(resultIndex); + } + } + else + { + _depthByNodeKey.TryAdd(callerNodeKey, callerDepth); + } + + if (!_parentsByNodeKey.TryGetValue(callerNodeKey, out var parents)) + { + parents = new HashSet(StringComparer.OrdinalIgnoreCase); + _parentsByNodeKey[callerNodeKey] = parents; + } + parents.Add(currentNodeKey); + } + + internal void Materialize(List results, int maxPathsPerResult) + { + if (!_withPaths) + return; + + var effectiveCap = maxPathsPerResult > 0 + ? maxPathsPerResult + : DefaultImpactPathsPerResult; + foreach (var (callerNodeKey, indices) in _resultIndicesByNodeKey!) + { + var (pathKeys, pathsTruncated) = EnumeratePaths(callerNodeKey, effectiveCap); + var paths = pathKeys + .Select(path => path.Select(nodeKey => _nodesByKey![nodeKey].Name).ToList()) + .ToList(); + foreach (var resultIndex in indices) + { + results[resultIndex].Paths = paths; + results[resultIndex].PathDetails = BuildPathDetails(pathKeys, results[resultIndex]); + results[resultIndex].PathsTruncated = pathsTruncated; + } + } + } + + private (List> Paths, bool Truncated) EnumeratePaths( + string callerNodeKey, + int maxPathsPerResult) + { + var paths = new List>(); + var stack = new Stack(); + var onStack = new HashSet(StringComparer.OrdinalIgnoreCase); + var truncated = false; + + stack.Push(callerNodeKey); + onStack.Add(callerNodeKey); + Dfs(callerNodeKey); + return (paths, truncated); + + void Dfs(string node) + { + if (string.Equals(node, _rootNodeKey, StringComparison.OrdinalIgnoreCase)) + { + paths.Add(stack.ToList()); + return; + } + if (!_parentsByNodeKey.TryGetValue(node, out var parents)) + return; + foreach (var parent in parents) + { + if (onStack.Contains(parent)) + continue; + if (paths.Count >= maxPathsPerResult) + { + truncated = true; + return; + } + stack.Push(parent); + onStack.Add(parent); + Dfs(parent); + stack.Pop(); + onStack.Remove(parent); + } + } + } + + private List> BuildPathDetails( + List> pathKeys, + ImpactResult result) + { + var details = new List>(pathKeys.Count); + foreach (var path in pathKeys) + { + var detailPath = new List(path.Count); + for (var i = 0; i < path.Count; i++) + { + var nodeKey = path[i]; + var isResultNode = i == path.Count - 1; + if (!_nodesByKey!.TryGetValue(nodeKey, out var node)) + node = new ImpactPathNode { Name = nodeKey }; + detailPath.Add(isResultNode + ? ClonePathNodeForResult(node, result) + : ClonePathNode(node)); + } + details.Add(detailPath); + } + return details; + } + + private static ImpactPathNode ClonePathNodeForResult( + ImpactPathNode node, + ImpactResult result) + { + var clone = ClonePathNode(node); + clone.Kind ??= result.CallerKind; + clone.Lang ??= result.Lang; + clone.ReferencePath = result.Path; + clone.ReferenceLine = result.FirstLine; + return clone; + } + + private static ImpactPathNode ClonePathNode(ImpactPathNode node) + => new() + { + SymbolId = node.SymbolId, + Name = node.Name, + Kind = node.Kind, + Lang = node.Lang, + DefinitionPath = node.DefinitionPath, + DefinitionLine = node.DefinitionLine, + Container = node.Container, + FamilyKey = node.FamilyKey, + PartialFamilyId = node.PartialFamilyId, + LogicalTargetKey = node.LogicalTargetKey, + ReferencePath = node.ReferencePath, + ReferenceLine = node.ReferenceLine, + }; + } +} diff --git a/src/CodeIndex/Database/DbReader.ImpactTraversal.State.cs b/src/CodeIndex/Database/DbReader.ImpactTraversal.State.cs new file mode 100644 index 0000000000..0a3a4a09f6 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.ImpactTraversal.State.cs @@ -0,0 +1,224 @@ +using CodeIndex.Models; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private static class ImpactNodeIdentity + { + internal static string TraversalKey(long? symbolId, string name) + => symbolId is long canonicalId ? CanonicalIdKey(canonicalId) : $"name:{name}"; + + internal static string VisitedKey( + CallerResult caller, + string callerName, + bool useCanonicalIdentity, + bool deduplicateLogicalNodes) + { + var identity = useCanonicalIdentity && caller.CallerSymbolId is long callerId + ? CanonicalIdKey(callerId) + : $"{caller.Path}:{callerName}"; + return deduplicateLogicalNodes ? identity : $"{identity}:{caller.ReferenceKind}"; + } + + internal static ImpactCycleNode? CycleNode( + long? symbolId, + string name, + bool hasResolvedIdentityGraph) + { + if (!hasResolvedIdentityGraph) + return new ImpactCycleNode($"name:{NameFold.Fold(name) ?? name}", null, name); + return symbolId is long canonicalId + ? new ImpactCycleNode(CanonicalIdKey(canonicalId), canonicalId, name) + : null; + } + + private static string CanonicalIdKey(long symbolId) => $"id:{symbolId}"; + } + + private sealed class ImpactTraversalState + { + private readonly int _resultOffset; + private readonly int _graphStateEntryBudget; + private readonly Dictionary? _resultIndexByVisitedKey; + + internal ImpactTraversalState( + DbReader owner, + ImpactTraversalRequest request, + ImpactTraversalRoot root) + { + _resultOffset = Math.Max(0, request.ResultOffset); + ResultWindowEnd = checked(_resultOffset + request.Limit); + _graphStateEntryBudget = owner.GetImpactGraphStateEntryBudget(ResultWindowEnd); + BoundaryProbeBudget = owner.GetImpactBoundaryCallerProbeBudget(); + Queue = CreateInitialQueue(root); + Visited = new HashSet(StringComparer.OrdinalIgnoreCase) { root.ResolvedName }; + Cycles = new ImpactCycleTracker(root); + Paths = new ImpactPathTracker(owner, request.WithPaths, root); + _resultIndexByVisitedKey = root.IsLogicalPartialFamily + ? new Dictionary(StringComparer.OrdinalIgnoreCase) + : null; + Truncated = root.InitiallyTruncated; + TruncatedReason = root.InitiallyTruncated + ? ImpactTruncatedReasons.SafetyCap + : null; + } + + internal Queue Queue { get; } + internal HashSet Visited { get; } + internal ImpactCycleTracker Cycles { get; } + internal ImpactPathTracker Paths { get; } + internal List Results { get; } = []; + internal int ResultWindowEnd { get; } + internal int BoundaryProbeBudget { get; } + internal int DiscoveredResultCount { get; private set; } + internal bool Truncated { get; private set; } + internal string? TruncatedReason { get; private set; } + internal bool MaxDepthReached { get; set; } + internal bool GraphStateBudgetHit { get; private set; } + internal bool BoundaryProbeBudgetHit { get; private set; } + + internal bool CanTraverse + => Queue.Count > 0 + && DiscoveredResultCount < ResultWindowEnd + && !GraphStateBudgetHit + && !BoundaryProbeBudgetHit; + + internal bool CanFetchCurrentNode + => DiscoveredResultCount < ResultWindowEnd + && !GraphStateBudgetHit + && !BoundaryProbeBudgetHit; + + internal bool IncludeNextResult => DiscoveredResultCount >= _resultOffset; + + private static Queue CreateInitialQueue( + ImpactTraversalRoot root) + { + var capacity = root.IsLogicalPartialFamily + ? 1 + : Math.Max(1, root.IdentitySymbolIds.Count); + var queue = new Queue(capacity); + if (root.IsLogicalPartialFamily) + { + queue.Enqueue(new ImpactTraversalFrontierNode( + root.ResolvedName, + SymbolId: null, + root.IdentitySymbolIds.Order().ToArray(), + root.NodeKey, + Depth: 0)); + } + else if (root.IdentitySymbolIds.Count > 0) + { + foreach (var symbolId in root.IdentitySymbolIds.Order()) + { + queue.Enqueue(new ImpactTraversalFrontierNode( + root.ResolvedName, + symbolId, + TargetSymbolIds: null, + root.NodeKey, + Depth: 0)); + } + } + else + { + queue.Enqueue(new ImpactTraversalFrontierNode( + root.ResolvedName, + SymbolId: null, + TargetSymbolIds: null, + root.NodeKey, + Depth: 0)); + } + return queue; + } + + internal bool TryVisit(string key) => Visited.Add(key); + + internal int AddResult(ImpactResult? result, string visitedKey) + { + var resultIndex = -1; + if (IncludeNextResult) + { + Results.Add(result!); + resultIndex = Results.Count - 1; + _resultIndexByVisitedKey?.Add(visitedKey, resultIndex); + } + DiscoveredResultCount++; + return resultIndex; + } + + internal void MergeReferenceEvidence(string visitedKey, CallerResult caller) + { + if (_resultIndexByVisitedKey != null + && _resultIndexByVisitedKey.TryGetValue(visitedKey, out var resultIndex)) + { + MergeImpactReferenceEvidence(Results[resultIndex], caller); + } + } + + internal bool CheckGraphStateBudget() + { + if (Paths.GraphStateEntryCount + Cycles.GraphStateEntryCount <= _graphStateEntryBudget) + return false; + GraphStateBudgetHit = true; + Truncated = true; + TruncatedReason = ImpactTruncatedReasons.GraphStateBudget; + return true; + } + + internal void MarkBoundaryProbeBudget() + { + BoundaryProbeBudgetHit = true; + Truncated = true; + TruncatedReason = ImpactTruncatedReasons.BoundaryProbeBudget; + } + + internal void MarkUserLimit() + { + Truncated = true; + TruncatedReason ??= ImpactTruncatedReasons.UserLimit; + } + + internal void MarkSafetyCap() + { + Truncated = true; + TruncatedReason = ImpactTruncatedReasons.SafetyCap; + } + + internal void CompleteTraversal() + { + if (Queue.Count > 0 && DiscoveredResultCount >= ResultWindowEnd) + MarkUserLimit(); + } + + internal string ResolveTerminationReason() + => TruncatedReason switch + { + ImpactTruncatedReasons.GraphStateBudget => ImpactTerminationReasons.GraphStateBudget, + ImpactTruncatedReasons.BoundaryProbeBudget => ImpactTerminationReasons.BoundaryProbeBudget, + ImpactTruncatedReasons.SafetyCap => ImpactTerminationReasons.SafetyCap, + ImpactTruncatedReasons.UserLimit => ImpactTerminationReasons.RowLimitTruncated, + _ when Cycles.Results.Count > 0 => ImpactTerminationReasons.CycleDetected, + _ when MaxDepthReached => ImpactTerminationReasons.MaxDepthReached, + _ => ImpactTerminationReasons.Completed, + }; + + private static void MergeImpactReferenceEvidence(ImpactResult result, CallerResult caller) + { + var counts = result.ReferenceKindCounts.ToDictionary( + static pair => pair.Key, + static pair => pair.Value, + StringComparer.Ordinal); + foreach (var (kind, count) in caller.ReferenceKindCounts) + { + counts[kind] = counts.TryGetValue(kind, out var existingCount) + ? Math.Max(existingCount, count) + : count; + } + + result.ReferenceKindCounts = counts; + result.ReferenceKinds = counts.Keys.Order(StringComparer.Ordinal).ToArray(); + result.ReferenceCount = counts.Values.Sum(); + result.FirstLine = Math.Min(result.FirstLine, caller.FirstLine); + } + } +} diff --git a/src/CodeIndex/Database/DbReader.ImpactTraversal.cs b/src/CodeIndex/Database/DbReader.ImpactTraversal.cs new file mode 100644 index 0000000000..1fd501b01a --- /dev/null +++ b/src/CodeIndex/Database/DbReader.ImpactTraversal.cs @@ -0,0 +1,302 @@ +using CodeIndex.Indexer; +using CodeIndex.Models; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + // Per-result cap on the number of distinct shortest paths surfaced by impact --with-paths. + // Each call chain row may carry multiple converging paths from the resolved root through + // distinct intermediates; the cap keeps JSON output bounded for diamond-heavy graphs and + // is signaled by ImpactResult.PathsTruncated when exceeded. + // impact --with-paths が 1 caller につき保持する経路数の上限。ダイヤモンド型で多経路が + // 収束する場合に JSON 膨張を抑える役割があり、超過時は PathsTruncated で通知する。 + private const int DefaultImpactPathsPerResult = 10; + internal const int DefaultImpactGraphStateEntryBudget = 10_000; + internal const int DefaultImpactPartialFamilyMemberBudget = 10_000; + internal int ImpactPartialFamilyMemberBudget { get; set; } = DefaultImpactPartialFamilyMemberBudget; + internal const int ImpactBoundaryCallerProbeBudget = 512; + private const int ImpactBoundaryCallerProbePageSize = 64; + + internal int? ImpactGraphStateEntryBudgetForTesting { get; set; } + internal int? ImpactBoundaryCallerProbeBudgetForTesting { get; set; } + + private sealed record ImpactTraversalRequest( + string SymbolName, + int MaxDepth, + int Limit, + string? Lang, + IReadOnlyList? PathPatterns, + IReadOnlyList? ExcludePathPatterns, + bool ExcludeTests, + bool WithPaths, + int MaxPathsPerResult, + int ResultOffset, + bool IncludeMemberReads); + + private sealed record ImpactTraversalRoot( + string ResolvedName, + HashSet DefinitionPaths, + bool HasResolvedIdentityGraph, + bool IsLogicalPartialFamily, + HashSet IdentitySymbolIds, + bool IncludeAmbiguousMSource, + bool InitiallyTruncated, + string NodeKey, + ImpactPathNode? PathNode); + + private readonly record struct ImpactTraversalFrontierNode( + string Symbol, + long? SymbolId, + IReadOnlyList? TargetSymbolIds, + string NodeKey, + int Depth); + + private readonly record struct ImpactBoundaryInspection( + bool HasUnvisitedCaller, + bool ProbeBudgetHit); + + /// + /// Compute transitive callers of a symbol using BFS with exact matching. + /// Returns each unique caller in the call chain with its depth from the root symbol. + /// The bound is inclusive: when is N, + /// callers at depth 1 through N are returned (so a chain A→B→C→D queried against D with + /// maxDepth: 2 yields C at depth 1 and B at depth 2). Truncation is signaled via the + /// Truncated property in results. When Truncated is true, TruncatedReason distinguishes + /// user_limit (raise --limit) from safety_cap (pathological graph). See Issue #1533. + /// When is true, each ImpactResult is populated with the + /// distinct shortest call paths from the resolved root through any intermediates to that + /// caller (issue #1536); converging diamond chains surface every shortest route up to + /// . + /// 完全一致の BFS でシンボルの推移的呼び出し元を算出。各呼び出し元とルートシンボルからの深さを返す。 + /// は inclusive で、N を指定すると depth 1〜N の caller を返す + /// (例: A→B→C→D のチェーンで D を maxDepth: 2 検索すると C(depth=1) と B(depth=2) を返す)。 + /// 結果が切り詰められた場合は Truncated フラグで通知し、TruncatedReason で + /// user_limit (--limit 到達、緩和で増える) と safety_cap (病的グラフ、--limit 緩和では解消しない) を区別する (#1533)。 + /// を true にすると、各 caller に対してルートからの推移経路 + /// （ダイヤモンド収束時は複数）を 件まで付与する（issue #1536）。 + /// + public (List Results, bool Truncated, string? TruncatedReason, string TerminationReason, List Cycles) GetTransitiveCallers( + string symbolName, + int maxDepth = 5, + int limit = 50, + string? lang = null, + IReadOnlyList? pathPatterns = null, + IReadOnlyList? excludePathPatterns = null, + bool excludeTests = false, + bool withPaths = false, + int maxPathsPerResult = DefaultImpactPathsPerResult, + int resultOffset = 0, + bool includeMemberReads = false) + { + var request = new ImpactTraversalRequest( + symbolName, + maxDepth, + limit, + lang, + pathPatterns, + excludePathPatterns, + excludeTests, + withPaths, + maxPathsPerResult, + resultOffset, + includeMemberReads); + var root = ResolveImpactTraversalRoot(request); + if (root == null) + return ([], false, null, ImpactTerminationReasons.Completed, []); + + return new ImpactTraversalEngine(this, request, root).Run(); + } + + private ImpactTraversalRoot? ResolveImpactTraversalRoot(ImpactTraversalRequest request) + { + var resolvedName = ResolveSymbolName(request.SymbolName, request.Lang); + var hasResolvedIdentityGraph = _referenceIdentityContractCurrent; + var canResolveQualifiedCSharpIdentity = + hasResolvedIdentityGraph + && SqlNameResolver.HasQualifier(request.SymbolName) + && request.Lang is null or "csharp"; + var rootDefinitionLimit = canResolveQualifiedCSharpIdentity + ? DefaultImpactGraphStateEntryBudget + : request.Limit; + var resolution = ResolveImpactRootDefinitions(request, resolvedName, rootDefinitionLimit); + var definitions = resolution.Definitions; + var definitionPaths = definitions + .Select(static definition => definition.Path) + .ToHashSet(StringComparer.OrdinalIgnoreCase); + var isLogicalPartialFamily = IsLogicalPartialFamilyRoot( + hasResolvedIdentityGraph, + resolution, + definitions); + var qualifiedCSharpIds = ResolveQualifiedCSharpRootIds( + canResolveQualifiedCSharpIdentity, + isLogicalPartialFamily, + resolution, + definitions); + + if (hasResolvedIdentityGraph && definitionPaths.Count > 1 && qualifiedCSharpIds.Count == 0) + return null; + + var ambiguousMRootId = ResolveAmbiguousMRootId( + hasResolvedIdentityGraph, + request.Lang, + definitions); + var identityIds = qualifiedCSharpIds.Count > 0 + ? qualifiedCSharpIds + : ambiguousMRootId is long ambiguousId + ? new HashSet { ambiguousId } + : []; + var singleIdentityId = identityIds.Count == 1 ? identityIds.Single() : (long?)null; + var rootNodeKey = identityIds.Count > 1 + ? $"identity:{NameFold.Fold(request.SymbolName) ?? request.SymbolName}" + : ImpactNodeIdentity.TraversalKey(singleIdentityId, resolvedName); + var pathNode = request.WithPaths + ? CreateImpactRootPathNode( + resolvedName, + singleIdentityId, + request.Lang, + isLogicalPartialFamily, + definitions) + : null; + + return new ImpactTraversalRoot( + resolvedName, + definitionPaths, + hasResolvedIdentityGraph, + isLogicalPartialFamily, + identityIds, + ambiguousMRootId != null, + !isLogicalPartialFamily + && qualifiedCSharpIds.Count > 0 + && resolution.PhysicalSymbolIdsTruncated, + rootNodeKey, + pathNode); + } + + private ImpactDefinitionResolution ResolveImpactRootDefinitions( + ImpactTraversalRequest request, + string resolvedName, + int definitionLimit) + { + var resolution = ResolveImpactDefinitions( + request.SymbolName, + definitionLimit, + request.Lang, + request.PathPatterns, + request.ExcludePathPatterns, + request.ExcludeTests); + if (resolution.Definitions.Count == 0 + && !string.Equals(request.SymbolName, resolvedName, StringComparison.Ordinal)) + { + resolution = ResolveImpactDefinitions( + resolvedName, + definitionLimit, + request.Lang, + request.PathPatterns, + request.ExcludePathPatterns, + request.ExcludeTests); + } + return resolution; + } + + private static bool IsLogicalPartialFamilyRoot( + bool hasResolvedIdentityGraph, + ImpactDefinitionResolution resolution, + IReadOnlyList definitions) + => hasResolvedIdentityGraph + && resolution.LogicalCount == 1 + && definitions.Count == 1 + && definitions[0].Lang == "csharp" + && definitions[0].PartialFamilyId != null + && resolution.PhysicalSymbolIds.Count > 0; + + private static HashSet ResolveQualifiedCSharpRootIds( + bool canResolveQualifiedCSharpIdentity, + bool isLogicalPartialFamily, + ImpactDefinitionResolution resolution, + IReadOnlyList definitions) + => (canResolveQualifiedCSharpIdentity || isLogicalPartialFamily) + && definitions.Count > 0 + && definitions.All(static definition => definition.Lang == "csharp") + && definitions.All(static definition => definition.SymbolId != null) + && (isLogicalPartialFamily || resolution.LogicalCount == definitions.Count) + ? resolution.PhysicalSymbolIds.ToHashSet() + : []; + + private static long? ResolveAmbiguousMRootId( + bool hasResolvedIdentityGraph, + string? lang, + IReadOnlyList definitions) + => hasResolvedIdentityGraph + && definitions.Count == 1 + && lang is "matlab" or "objc" + && string.Equals(definitions[0].Lang, lang, StringComparison.Ordinal) + ? definitions[0].SymbolId + : null; + + private ImpactPathNode CreateImpactRootPathNode( + string resolvedName, + long? symbolId, + string? lang, + bool isLogicalPartialFamily, + IReadOnlyList definitions) + { + var node = ResolveImpactPathNode( + resolvedName, + symbolId, + kind: null, + lang, + referencePath: null, + referenceLine: null); + if (!isLogicalPartialFamily) + return node; + + var representative = definitions[0]; + node.SymbolId = null; + node.Name = resolvedName; + node.Kind = representative.Kind; + node.Lang = representative.Lang; + node.DefinitionPath = representative.Path; + node.DefinitionLine = representative.Line; + node.Container = representative.ContainerQualifiedName ?? representative.ContainerName; + node.PartialFamilyId = representative.PartialFamilyId; + node.LogicalTargetKey = $"partial|{representative.PartialFamilyId}"; + return node; + } + + private List ReadImpactCallerPage( + in ImpactTraversalFrontierNode node, + ImpactTraversalRequest request, + int pageSize, + int pageOffset, + bool includeAmbiguousMSource) + { + IReadOnlyList? targetIds = node.TargetSymbolIds is { Count: > 0 } + ? node.TargetSymbolIds + : node.SymbolId is long symbolId + ? [symbolId] + : null; + return GetCallersExactCore( + node.Symbol, + pageSize, + pageOffset, + request.Lang, + request.PathPatterns, + request.ExcludePathPatterns, + request.ExcludeTests, + targetIds, + includeAmbiguousMSource, + request.IncludeMemberReads); + } + + private int GetImpactGraphStateEntryBudget(int limit) + { + if (ImpactGraphStateEntryBudgetForTesting is int testBudget) + return testBudget; + var limitScaled = Math.Max(1, limit) * 200; + return Math.Max(1024, Math.Min(DefaultImpactGraphStateEntryBudget, limitScaled)); + } + + private int GetImpactBoundaryCallerProbeBudget() + => ImpactBoundaryCallerProbeBudgetForTesting ?? ImpactBoundaryCallerProbeBudget; +} diff --git a/src/CodeIndex/Database/DbReader.LogicalPartialQuerySql.cs b/src/CodeIndex/Database/DbReader.LogicalPartialQuerySql.cs new file mode 100644 index 0000000000..51c02b9160 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.LogicalPartialQuerySql.cs @@ -0,0 +1,79 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed record LogicalPartialCanonicalSql( + string LogicalPartialKey, + string Generated, + string PrimaryRank, + string SemanticScore, + string DeclarationIdentity); + + private static class LogicalPartialQuerySql + { + public static LogicalPartialCanonicalSql Build( + DbReader reader, + string signatureSql, + string containerNameSql, + string containerQualifiedNameSql, + string familyKeySql, + string returnTypeSql, + string bodyStartLineSql, + string bodyEndLineSql) + { + var logicalPartialKey = BuildKey( + reader, + signatureSql, + containerNameSql, + containerQualifiedNameSql, + familyKeySql, + returnTypeSql); + var generated = reader._fileColumns.Contains("generated") + ? "CASE WHEN COALESCE(f.generated, 0) <> 0 OR codeindex_generated_file_name(f.path) THEN 1 ELSE 0 END" + : "CASE WHEN codeindex_generated_file_name(f.path) THEN 1 ELSE 0 END"; + var primaryRank = LogicalPartialSymbolGrouper.BuildSqlPrimaryRankExpression( + "s.kind", + bodyStartLineSql, + bodyEndLineSql); + var semanticScore = LogicalPartialSymbolGrouper.BuildSqlSemanticScoreExpression( + signatureSql, + "s.kind", + reader.GetSymbolColumnSql("declaration_semantic_score")); + var fallbackIdentity = BuildDeclarationIdentity(signatureSql); + var declarationIdentity = + $"CASE WHEN s.kind IN ('function', 'test.method') THEN COALESCE(csharp_partial_callable_identity({signatureSql}, s.name, {returnTypeSql}), {fallbackIdentity}) ELSE {fallbackIdentity} END"; + return new LogicalPartialCanonicalSql( + logicalPartialKey, + generated, + primaryRank, + semanticScore, + declarationIdentity); + } + + public static string BuildKey( + DbReader reader, + string signatureSql, + string containerNameSql, + string containerQualifiedNameSql, + string familyKeySql, + string returnTypeSql) + { + return LogicalPartialSymbolGrouper.BuildSqlKeyExpression( + "f.lang", + "s.kind", + "s.name", + "s.id", + "f.path", + signatureSql, + containerNameSql, + containerQualifiedNameSql, + familyKeySql, + returnTypeSql, + reader.GetSymbolColumnSql("is_partial_declaration"), + reader._hotspotFamilyReadyLanguages.Contains("csharp")); + } + + private static string BuildDeclarationIdentity(string signatureSql) + => $"csharp_partial_declaration_identity({signatureSql})"; + } +} diff --git a/src/CodeIndex/Database/DbReader.References.cs b/src/CodeIndex/Database/DbReader.References.cs index 5ff578d918..245870b32c 100644 --- a/src/CodeIndex/Database/DbReader.References.cs +++ b/src/CodeIndex/Database/DbReader.References.cs @@ -653,34 +653,8 @@ private bool ShouldSuppressCSharpQualifiedConstantPatternReference(string path, return false; } - - // Query-side mirror of the C# declaration canonicalizer. Users commonly type source - // spellings such as `@class` or `Outer.@class`; the DB stores the canonical names - // without the verbatim `@`, so query entrypoints normalize to the persisted form first. - // Rust macro names are also accepted with a trailing `!` because the extractor stores - // them without the punctuation, so `my_macro!` and `my_macro` resolve to the same row. - // The normalization is applied when `--lang` is omitted or explicitly `csharp` because - // name-based lookup still needs to treat C# verbatim spellings as canonical symbol names. - // Other languages, including SQL, must preserve leading `@` characters. - // C# 宣言側 canonicalizer の query 側ミラー。`@class` / `Outer.@class` のような source - // spelling を受けても、DB 側の `@` なし canonical 名に合わせてから検索する。 - // Rust macro 名も extractor 側では末尾 `!` を落として保存するため、`my_macro!` と `my_macro` - // を同じ行へ解決できるようにする。 - // `--lang` 未指定または `csharp` 指定では name-based lookup が verbatim spelling を canonical 名へ寄せる。 - // それ以外の言語、特に SQL では先頭 `@` を保持する。 private static string? NormalizeCSharpVerbatimQuery(string? query, string? lang) - { - if (!string.IsNullOrWhiteSpace(lang) && string.Equals(lang, "rust", StringComparison.OrdinalIgnoreCase)) - { - var rustNormalized = NormalizeRustSymbolSearchQuery(query); - return string.IsNullOrWhiteSpace(rustNormalized) ? null : rustNormalized; - } - - if (!string.IsNullOrWhiteSpace(lang) && !string.Equals(lang, "csharp", StringComparison.OrdinalIgnoreCase)) - return query; - var normalized = query == null ? null : NormalizeDbCSharpQualifiedName(query); - return string.IsNullOrWhiteSpace(normalized) ? null : normalized; - } + => SymbolSearchQueryNormalizer.NormalizeCSharpVerbatim(query, lang); private static bool IsBareVerbatimQueryToken(string? value) { diff --git a/src/CodeIndex/Database/DbReader.SymbolSearchBoundedCount.cs b/src/CodeIndex/Database/DbReader.SymbolSearchBoundedCount.cs new file mode 100644 index 0000000000..5dcfecbc51 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.SymbolSearchBoundedCount.cs @@ -0,0 +1,64 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + public int CountSearchSymbols(IReadOnlyList? queries, int limit = 20, string? kind = null, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, DateTime? since = null, bool exact = false, IReadOnlyList? visibilityFilters = null, IReadOnlyList? excludeVisibilityFilters = null) + { + var plan = SymbolSearchQueryPlanBuilder.Build(new SymbolSearchQueryPlan + { + Queries = queries, + Limit = limit, + Kind = kind, + Lang = lang, + PathPatterns = pathPatterns, + ExcludePathPatterns = excludePathPatterns, + ExcludeTests = excludeTests, + Since = since, + Exact = exact, + VisibilityFilters = visibilityFilters, + ExcludeVisibilityFilters = excludeVisibilityFilters, + }); + return ExecuteBoundedSymbolSearch(plan); + } + + private int ExecuteBoundedSymbolSearch(SymbolSearchQueryPlan plan) + { + if (plan.Queries != null + && (plan.Queries.Count > 1 + || string.Equals(plan.Lang, "markdown", StringComparison.Ordinal))) + { + return SearchSymbols( + plan.Queries, + plan.Limit, + plan.Kind, + plan.Lang, + plan.PathPatterns, + plan.ExcludePathPatterns, + plan.ExcludeTests, + plan.Since, + plan.Exact, + plan.VisibilityFilters, + plan.ExcludeVisibilityFilters).Count; + } + + using var cmd = _conn.CreateCommand(); + + var innerSql = @" + SELECT 1 + FROM symbols s + JOIN files f ON s.file_id = f.id + WHERE 1=1"; + + innerSql += SymbolSearchQueryPredicateBuilder.BuildBounded(this, plan); + SymbolSearchQueryPredicateBuilder.AppendFilters(this, ref innerSql, plan, includeLineRange: false); + innerSql += " LIMIT @limit"; + + cmd.CommandText = $"SELECT COUNT(*) FROM ({innerSql})"; + SymbolSearchQueryBinder.BindBoundedQuery(this, cmd, plan); + SymbolSearchQueryBinder.BindFilters(this, cmd, plan, includeLineRange: false); + SqliteCommandPolicy.Add(cmd, "@limit", plan.Limit); + + var raw = cmd.ExecuteScalar(); + return raw is long l ? (int)l : Convert.ToInt32(raw); + } +} diff --git a/src/CodeIndex/Database/DbReader.SymbolSearchList.cs b/src/CodeIndex/Database/DbReader.SymbolSearchList.cs new file mode 100644 index 0000000000..af743d7872 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.SymbolSearchList.cs @@ -0,0 +1,61 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + /// + /// Search symbols by one or more name patterns (OR-joined). Empty/null list returns all symbols matching other filters. + /// When is true, names are matched case-insensitively for equality instead of substring. + /// 複数名前パターン（OR結合）でシンボルを検索。空/null なら他フィルタに一致する全シンボルを返す。 + /// が true の場合、部分一致ではなく大文字小文字を無視した完全一致になる。 + /// + public List SearchSymbols(IReadOnlyList? queries, int limit = 20, string? kind = null, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, DateTime? since = null, bool exact = false, IReadOnlyList? visibilityFilters = null, IReadOnlyList? excludeVisibilityFilters = null, SymbolSortMode sortMode = SymbolSortMode.Name, int? startLine = null, int? endLine = null, bool groupPartials = false, int offset = 0) + { + var plan = SymbolSearchQueryPlanBuilder.Build(new SymbolSearchQueryPlan + { + Queries = queries, + Limit = limit, + Kind = kind, + Lang = lang, + PathPatterns = pathPatterns, + ExcludePathPatterns = excludePathPatterns, + ExcludeTests = excludeTests, + Since = since, + Exact = exact, + VisibilityFilters = visibilityFilters, + ExcludeVisibilityFilters = excludeVisibilityFilters, + SortMode = sortMode, + StartLine = startLine, + EndLine = endLine, + GroupPartials = groupPartials, + Offset = offset, + }); + return ExecuteSymbolSearchList(plan); + } + + private List ExecuteSymbolSearchList(SymbolSearchQueryPlan plan) + { + if (plan.Queries is { Count: > 1 }) + return ExecuteMultiQuerySymbolSearch(plan); + + if (plan.GroupPartials) + { + EnsureCSharpCallableTypeKinds( + plan.Lang, + plan.Queries, + plan.Exact, + plan.Kind); + } + using var cmd = _conn.CreateCommand(); + + var sql = BuildSymbolSearchListSql(plan); + cmd.CommandText = sql; + SymbolSearchQueryBinder.BindFullQueries(this, cmd, plan); + SymbolSearchQueryBinder.BindListOrdering(cmd, plan); + SymbolSearchQueryBinder.BindFilters(this, cmd, plan, includeLineRange: true); + SqliteCommandPolicy.Add(cmd, "@limit", plan.Limit); + SqliteCommandPolicy.Add(cmd, "@offset", Math.Max(0, plan.Offset)); + + using var reader = cmd.ExecuteTrackedReader(); + return SymbolSearchRowProjector.ReadAll(reader, plan); + } +} diff --git a/src/CodeIndex/Database/DbReader.SymbolSearchListQueryBuilder.cs b/src/CodeIndex/Database/DbReader.SymbolSearchListQueryBuilder.cs new file mode 100644 index 0000000000..8cb3d16817 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.SymbolSearchListQueryBuilder.cs @@ -0,0 +1,244 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed record SymbolSearchColumnSql( + string StartLine, + string EndLine, + string BodyStartLine, + string BodyEndLine, + string Signature, + string ContainerKind, + string ContainerName, + string ContainerQualifiedName, + string FamilyKey, + string Visibility, + string ReturnType, + string StartColumn, + string SizeLines); + + private sealed record SymbolSearchRankingSql( + string Join, + string GenericNamePenalty, + string DefinitionSites, + string ReferenceCount, + string HotspotScore, + string RankingReferenceScore, + string RankingHotspotScore, + string StructuralRankPenalty, + string ComplexityScore, + string ExactNameOrder); + + private string BuildSymbolSearchListSql(SymbolSearchQueryPlan plan) + { + var columns = BuildSymbolSearchColumnSql(); + var includeRankSignals = + plan.SortMode != SymbolSortMode.Name && _hasReferencesTable; + var ranking = BuildSymbolSearchRankingSql(columns, includeRankSignals); + var canonical = LogicalPartialQuerySql.Build( + this, + columns.Signature, + columns.ContainerName, + columns.ContainerQualifiedName, + columns.FamilyKey, + columns.ReturnType, + columns.BodyStartLine, + columns.BodyEndLine); + var sql = BuildSymbolSearchSelectSql(columns, ranking, canonical); + sql += SymbolSearchQueryPredicateBuilder.BuildFull(this, plan); + SymbolSearchQueryPredicateBuilder.AppendFilters( + this, + ref sql, + plan, + includeLineRange: true); + if (plan.GroupPartials) + { + sql = BuildLogicalPartialSymbolQuery(sql, plan.SortMode); + } + else + { + sql += BuildSymbolSortOrderBy( + plan.SortMode, + ranking.ExactNameOrder, + ranking.ReferenceCount, + ranking.HotspotScore, + ranking.RankingReferenceScore, + ranking.RankingHotspotScore, + columns.SizeLines, + ranking.ComplexityScore, + columns.StartColumn); + } + + return sql + " LIMIT @limit OFFSET @offset"; + } + + private SymbolSearchColumnSql BuildSymbolSearchColumnSql() + { + var startLine = GetSymbolColumnSql("start_line", "s.line"); + var endLine = GetSymbolColumnSql("end_line", "s.line"); + return new SymbolSearchColumnSql( + startLine, + endLine, + GetSymbolColumnSql("body_start_line"), + GetSymbolColumnSql("body_end_line"), + GetSymbolColumnSql("signature"), + GetSymbolColumnSql("container_kind"), + GetSymbolColumnSql("container_name"), + GetSymbolColumnSql("container_qualified_name"), + GetSymbolColumnSql("family_key"), + GetSymbolColumnSql("visibility"), + GetSymbolColumnSql("return_type"), + GetSymbolColumnSql("start_column", "CAST(2147483647 AS INTEGER)"), + $"CASE WHEN ({endLine}) >= ({startLine}) THEN ({endLine}) - ({startLine}) + 1 ELSE 1 END"); + } + + private SymbolSearchRankingSql BuildSymbolSearchRankingSql( + SymbolSearchColumnSql columns, + bool includeRankSignals) + { + var genericPenalty = includeRankSignals + ? GetGenericSymbolRankNamePenaltySql("s.name") + : "1.0"; + var definitionSites = includeRankSignals + ? "COALESCE(symbol_defs.definition_sites, 1)" + : "CAST(1 AS INTEGER)"; + var conservativeSignal = includeRankSignals + ? $"(f.lang = 'csharp' AND (s.kind = 'property' OR ({definitionSites}) > 1 OR lower(s.name) IN {GenericSymbolRankNamesSql}))" + : "0"; + var referenceCount = includeRankSignals + ? $"CASE WHEN {conservativeSignal} THEN COALESCE(symbol_file_rank.reference_count, 0) ELSE COALESCE(symbol_rank.reference_count, 0) END" + : "CAST(0 AS INTEGER)"; + var hotspotScore = includeRankSignals + ? $"CASE WHEN {conservativeSignal} THEN COALESCE(symbol_file_rank.hotspot_score, 0.0) ELSE COALESCE(symbol_rank.hotspot_score, 0.0) END" + : "CAST(0.0 AS REAL)"; + var dilution = $"CASE WHEN ({definitionSites}) > 1 THEN CAST(({definitionSites}) * ({definitionSites}) AS REAL) ELSE 1.0 END"; + var structuralPenalty = includeRankSignals + ? $"CASE WHEN s.kind IN ('property', 'enum') AND ({columns.SizeLines}) <= 1 THEN 0.1 ELSE 1.0 END" + : "1.0"; + var rankingReference = includeRankSignals + ? $"(({referenceCount}) * ({genericPenalty}) * ({structuralPenalty}) / ({dilution}))" + : referenceCount; + var rankingHotspot = includeRankSignals + ? $"(({hotspotScore}) * ({genericPenalty}) * ({structuralPenalty}) / ({dilution}))" + : hotspotScore; + var cappedReference = $"CASE WHEN ({rankingReference}) > 100.0 THEN 100.0 ELSE ({rankingReference}) END"; + var cappedHotspot = $"CASE WHEN ({rankingHotspot}) > 150.0 THEN 150.0 ELSE ({rankingHotspot}) END"; + var complexity = $@"(({columns.SizeLines} * 16.0) + ({cappedReference} * 0.75) + ({cappedHotspot} * 0.35) + CASE + WHEN {columns.Visibility} IN ('public', 'pub', 'open', 'export') THEN 8.0 + WHEN {columns.Visibility} IN ('protected', 'internal', 'protected internal') THEN 4.0 + ELSE 0.0 + END)"; + return new SymbolSearchRankingSql( + BuildSymbolRankJoin(includeRankSignals), + genericPenalty, + definitionSites, + referenceCount, + hotspotScore, + rankingReference, + rankingHotspot, + structuralPenalty, + complexity, + BuildExactSymbolNameOrderSql()); + } + + private static string BuildSymbolRankJoin(bool includeRankSignals) + { + return includeRankSignals + ? $@" + LEFT JOIN ( + SELECT rf.lang AS lang, + sr.symbol_name AS symbol_name, + COUNT(*) AS reference_count, + SUM({GetHotspotReferenceWeightSql("sr.reference_kind")}) AS hotspot_score + FROM symbol_references sr + JOIN files rf ON rf.id = sr.file_id + WHERE sr.reference_kind IN {CallGraphReferenceKindsSql} + AND sr.symbol_name IS NOT NULL + AND sr.symbol_name <> '' + GROUP BY rf.lang, sr.symbol_name COLLATE NOCASE + ) symbol_rank + ON symbol_rank.lang = f.lang + AND symbol_rank.symbol_name = s.name COLLATE NOCASE + LEFT JOIN ( + SELECT sr.file_id AS file_id, + sr.symbol_name AS symbol_name, + COUNT(*) AS reference_count, + SUM({GetHotspotReferenceWeightSql("sr.reference_kind")}) AS hotspot_score + FROM symbol_references sr + WHERE sr.reference_kind IN {CallGraphReferenceKindsSql} + AND sr.symbol_name IS NOT NULL + AND sr.symbol_name <> '' + GROUP BY sr.file_id, sr.symbol_name COLLATE NOCASE + ) symbol_file_rank + ON symbol_file_rank.file_id = s.file_id + AND symbol_file_rank.symbol_name = s.name COLLATE NOCASE + LEFT JOIN ( + SELECT df.lang AS lang, + ds.name AS symbol_name, + COUNT(*) AS definition_sites + FROM symbols ds + JOIN files df ON df.id = ds.file_id + WHERE ds.name IS NOT NULL + AND ds.name <> '' + GROUP BY df.lang, ds.name COLLATE NOCASE + ) symbol_defs + ON symbol_defs.lang = f.lang + AND symbol_defs.symbol_name = s.name COLLATE NOCASE" + : string.Empty; + } + + private string BuildSymbolSearchSelectSql( + SymbolSearchColumnSql columns, + SymbolSearchRankingSql ranking, + LogicalPartialCanonicalSql canonical) + { + return $@" + SELECT f.path, f.lang, s.kind, {GetSymbolColumnSql("sub_kind")} AS sub_kind, s.name, s.line, + {columns.StartLine} AS start_line, + {GetSymbolColumnSql("start_column")} AS start_column, + {columns.EndLine} AS end_line, + {columns.BodyStartLine} AS body_start_line, + {columns.BodyEndLine} AS body_end_line, + {columns.Signature} AS signature, + {columns.ContainerKind} AS container_kind, + {columns.ContainerName} AS container_name, + {columns.Visibility} AS visibility, + {columns.ReturnType} AS return_type, + {ranking.ReferenceCount} AS reference_count, + {ranking.HotspotScore} AS hotspot_score, + {ranking.RankingReferenceScore} AS ranking_reference_score, + {ranking.RankingHotspotScore} AS ranking_hotspot_score, + {ranking.GenericNamePenalty} AS generic_name_penalty, + {ranking.StructuralRankPenalty} AS structural_rank_penalty, + {ranking.DefinitionSites} AS definition_sites, + {columns.SizeLines} AS size_lines, + {ranking.ComplexityScore} AS complexity_score, + {columns.ContainerQualifiedName} AS container_qualified_name, + {canonical.LogicalPartialKey} AS logical_partial_key, + s.id AS symbol_id, + {ranking.ExactNameOrder} AS exact_name_order, + {PathBucketOrder} AS path_bucket, + {VisibilityOrder} AS visibility_rank, + {columns.StartColumn} AS stable_start_column, + {canonical.PrimaryRank} AS canonical_primary_rank, + {canonical.Generated} AS canonical_generated_rank, + {canonical.SemanticScore} AS canonical_semantic_score, + {canonical.DeclarationIdentity} AS canonical_declaration_identity, + {GetSymbolColumnSql("identifier_start_column")} AS identifier_start_column + FROM symbols s + JOIN files f ON s.file_id = f.id + {ranking.Join} + WHERE 1=1"; + } + + private static string BuildExactSymbolNameOrderSql() + { + return "CASE " + + "WHEN @preferLiteralExactMatch = 1 AND s.name = @rawQuery THEN 0 " + + "WHEN @preferLiteralNormalizedSqlMatch = 1 AND f.lang = 'sql' AND sql_segment_count(s.name) = @rawQuerySegmentCount AND sql_normalize_name(s.name) = @rawQueryNormalized THEN 1 " + + "WHEN @preferCaseInsensitiveExactMatch = 1 AND s.name = @rawQuery COLLATE NOCASE THEN 2 " + + "WHEN @preferCaseInsensitiveNormalizedSqlMatch = 1 AND f.lang = 'sql' AND sql_segment_count(s.name) = @rawQuerySegmentCount AND sql_normalize_name_folded(s.name) = @rawQueryNormalizedFolded THEN 3 " + + "WHEN @preferCaseInsensitiveSqlLeafMatch = 1 AND f.lang = 'sql' AND sql_leaf_name_folded(s.name) = @rawQueryLeafFolded THEN 4 " + + "ELSE 5 END"; + } +} diff --git a/src/CodeIndex/Database/DbReader.SymbolSearchMultiQuery.cs b/src/CodeIndex/Database/DbReader.SymbolSearchMultiQuery.cs new file mode 100644 index 0000000000..ff3f69063d --- /dev/null +++ b/src/CodeIndex/Database/DbReader.SymbolSearchMultiQuery.cs @@ -0,0 +1,59 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + // Run one search per name so a common earlier-sorting name cannot starve another + // query, then round-robin under the original global limit/offset contract. + private List ExecuteMultiQuerySymbolSearch(SymbolSearchQueryPlan plan) + { + var queries = plan.Queries!; + var requestedPrefix = checked(plan.Limit + Math.Max(0, plan.Offset)); + var perName = new List>(queries.Count); + foreach (var query in queries) + { + perName.Add(SearchSymbols( + new[] { query! }, + requestedPrefix, + plan.Kind, + plan.Lang, + plan.PathPatterns, + plan.ExcludePathPatterns, + plan.ExcludeTests, + plan.Since, + plan.Exact, + plan.VisibilityFilters, + plan.ExcludeVisibilityFilters, + plan.SortMode, + plan.StartLine, + plan.EndLine, + plan.GroupPartials)); + } + + var seen = new HashSet(); + var merged = new List(); + var cursors = new int[perName.Count]; + bool advanced; + do + { + advanced = false; + for (var index = 0; index < perName.Count && merged.Count < requestedPrefix; index++) + { + while (cursors[index] < perName[index].Count) + { + var result = perName[index][cursors[index]++]; + if (seen.Add(result.SymbolId)) + { + merged.Add(result); + advanced = true; + break; + } + } + } + } while (advanced && merged.Count < requestedPrefix); + + return merged + .Skip(Math.Max(0, plan.Offset)) + .Take(plan.Limit) + .ToList(); + } +} diff --git a/src/CodeIndex/Database/DbReader.SymbolSearchPredicates.cs b/src/CodeIndex/Database/DbReader.SymbolSearchPredicates.cs new file mode 100644 index 0000000000..abbd692bc7 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.SymbolSearchPredicates.cs @@ -0,0 +1,177 @@ +using CodeIndex.Indexer; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private string BuildQualifiedSymbolMatchSql(string parameterStem, bool useFoldedName, string symbolAlias = "s", string fileAlias = "f") + { + var containerNameSql = GetSymbolColumnSql("container_name", "''", symbolAlias); + var containerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name", containerNameSql, symbolAlias); + var nameMatchSql = useFoldedName + ? $"{symbolAlias}.name_folded = @{parameterStem}CSharpLeafFolded" + : $"{symbolAlias}.name = @{parameterStem}CSharpLeaf COLLATE NOCASE"; + var qualifiedNameMatchSql = useFoldedName + ? $"{symbolAlias}.name_folded = @{parameterStem}CSharpQualifiedFolded" + : $"{symbolAlias}.name = @{parameterStem}CSharpQualified COLLATE NOCASE"; + return $@"({fileAlias}.lang = 'csharp' + AND ({qualifiedNameMatchSql} + OR ({nameMatchSql} + AND ({containerNameSql} = @{parameterStem}Container COLLATE NOCASE + OR {containerQualifiedNameSql} = @{parameterStem}Container COLLATE NOCASE + OR {containerQualifiedNameSql} COLLATE NOCASE LIKE @{parameterStem}ContainerSuffixLike ESCAPE '\'))))"; + } + + private string BuildExactPrimarySymbolNameMatchSql( + string parameterSql, + bool useFoldedName, + string query, + string? lang) + { + var matchSql = useFoldedName + ? BuildPersistedFoldedNameMatchSql("s.name_folded", parameterSql) + : $"s.name = {parameterSql} COLLATE NOCASE"; + if (!string.IsNullOrWhiteSpace(lang) || !SqlNameResolver.HasQualifier(query)) + return matchSql; + + // Preserve direct qualified matching for ordinary and legacy C# rows. Only v3 + // explicit-interface rows have a display alias; those rows must use the C# identity + // clause so `IFoo.this` cannot also match a distinct `IFoo.@this` implementation. + // 通常および legacy C# row の修飾直接一致は維持する。表示 alias を持つ v3 の + // 明示的 interface row だけを C# identity 条件へ限定し、`IFoo.this` が別の + // `IFoo.@this` 実装にも一致しないようにする。 + var displayNameFoldedSql = GetSymbolColumnSql("display_name_folded", "NULL"); + return $"((f.lang <> 'csharp' OR {displayNameFoldedSql} IS NULL) AND {matchSql})"; + } + + private string BuildCSharpExplicitInterfaceIdentityMatchSql( + string parameterStem, + string symbolAlias = "s", + string fileAlias = "f") + { + if (!_csharpSymbolNameContractCurrent + || !_foldMetadataCurrent + || !_symbolColumns.Contains("name_folded")) + return "0"; + + return $"({fileAlias}.lang = 'csharp' AND {symbolAlias}.name_folded = @{parameterStem}CSharpExplicitInterfaceIdentityFolded)"; + } + + private string BuildCSharpExplicitInterfaceShortAliasMatchSql( + string parameterStem, + string symbolAlias = "s", + string fileAlias = "f") + => _foldReady + && _csharpSymbolNameContractCurrent + && _symbolColumns.Contains("display_name_folded") + && HasSymbolIndex("idx_symbols_display_name_folded") + ? $"({fileAlias}.lang = 'csharp' AND {symbolAlias}.display_name_folded = @{parameterStem}LeafFolded)" + : $"({fileAlias}.lang = 'csharp' AND {symbolAlias}.name = @{parameterStem}Leaf COLLATE NOCASE)"; + + private static void AddCSharpExplicitInterfaceIdentityQueryParameter( + SqliteCommand cmd, + string parameterStem, + string query) + { + SqliteCommandPolicy.Add( + cmd, + $"@{parameterStem}CSharpExplicitInterfaceIdentityFolded", + CSharpSymbolNameNormalizer.NormalizeExplicitInterfaceQueryIdentityNameFolded(query)); + } + + private static string GetQualifiedQueryContainer(string query) + { + var normalized = SqlNameResolver.NormalizeQualifiedName(query); + var lastDot = normalized.LastIndexOf('.'); + return lastDot > 0 ? normalized[..lastDot] : string.Empty; + } + + private static string GetQualifiedQuerySuffix(string query) + { + var normalized = SqlNameResolver.NormalizeQualifiedName(query); + var lastDot = normalized.LastIndexOf('.'); + if (lastDot <= 0) + return normalized; + var previousDot = normalized.LastIndexOf('.', lastDot - 1); + return previousDot >= 0 ? normalized[(previousDot + 1)..] : normalized; + } + + private static string GetQualifiedQueryLeaf(string query, string? lang) + { + var leaf = SqlNameResolver.GetLeafName(query); + return NormalizeCSharpVerbatimQuery(leaf, lang) ?? leaf; + } + + private static void AddQualifiedSymbolQueryParameters(SqliteCommand cmd, string parameterStem, string query) + { + var csharpDisplayQuery = + CSharpSymbolNameNormalizer.NormalizeExplicitInterfaceQueryDisplayName(query); + var csharpQualifiedQuery = + NormalizeCSharpVerbatimQuery(csharpDisplayQuery, "csharp") + ?? csharpDisplayQuery; + var container = GetQualifiedQueryContainer(csharpQualifiedQuery); + var csharpLeaf = GetQualifiedQueryLeaf(csharpQualifiedQuery, "csharp"); + SqliteCommandPolicy.Add(cmd, $"@{parameterStem}Container", container); + SqliteCommandPolicy.Add(cmd, $"@{parameterStem}ContainerSuffixLike", $"%.{EscapeLikeQuery(container)}"); + SqliteCommandPolicy.Add(cmd, $"@{parameterStem}CSharpQualified", csharpQualifiedQuery); + SqliteCommandPolicy.Add( + cmd, + $"@{parameterStem}CSharpQualifiedFolded", + NameFold.Fold(csharpQualifiedQuery) ?? csharpQualifiedQuery); + SqliteCommandPolicy.Add(cmd, $"@{parameterStem}CSharpLeaf", csharpLeaf); + SqliteCommandPolicy.Add( + cmd, + $"@{parameterStem}CSharpLeafFolded", + NameFold.Fold(csharpLeaf) ?? csharpLeaf); + } + + private bool HasSingleQualifiedSymbolDefinition(string query, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests) + { + if (!SqlNameResolver.HasQualifier(query)) + return false; + + var matches = SearchSymbols(query, 2, kind: null, lang, pathPatterns: null, excludePathPatterns: null, excludeTests, since: null, exact: false); + if (matches.Count != 1) + return false; + + var leafMatches = SearchSymbols(SqlNameResolver.GetLeafName(query), 2, kind: null, lang, pathPatterns: null, excludePathPatterns: null, excludeTests, since: null, exact: true); + return leafMatches.Count == 1; + } + + private bool HasQualifiedSymbolDefinition(string query, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests) + { + if (!SqlNameResolver.HasQualifier(query)) + return false; + + return SearchSymbols(query, 1, kind: null, lang, pathPatterns: null, excludePathPatterns: null, excludeTests, since: null, exact: false).Count > 0; + } + + private static string BuildQualifiedContextMatchSql(string contextSql, string columnSql, bool folded, bool like) + { + var functionName = (folded, like) switch + { + (true, true) => "sql_context_like_name_folded_at", + (true, false) => "sql_context_has_name_folded_at", + (false, true) => "sql_context_like_name_at", + _ => "sql_context_has_name_at", + }; + return $"({functionName}({contextSql}, @aliasQuery, {columnSql}) = 1 OR {functionName}({contextSql}, @aliasQuerySuffix, {columnSql}) = 1)"; + } + + private static string BuildQualifiedLeafFallbackSql(string nameSql, string foldedNameSql, bool folded) + => folded + ? $"(@allowQualifiedLeafFallback = 1 AND f.lang = 'csharp' AND {foldedNameSql} = @aliasQueryLeafFolded)" + : $"(@allowQualifiedLeafFallback = 1 AND f.lang = 'csharp' AND {nameSql} = @aliasQueryLeaf COLLATE NOCASE)"; + + private static string BuildCSharpQualifiedContextFallbackSql(string qualifiedContextSql) + => $"(@allowCSharpQualifiedContextMatch = 1 AND f.lang = 'csharp' AND {qualifiedContextSql})"; + + private static void AddQualifiedGraphQueryParameters(SqliteCommand cmd, string query, bool allowLeafFallback, bool allowCSharpContextMatch = false) + { + SqliteCommandPolicy.Add(cmd, "@aliasQuerySuffix", GetQualifiedQuerySuffix(query)); + SqliteCommandPolicy.Add(cmd, "@aliasQueryLeaf", SqlNameResolver.GetLeafName(query)); + SqliteCommandPolicy.Add(cmd, "@allowQualifiedLeafFallback", allowLeafFallback ? 1 : 0); + SqliteCommandPolicy.Add(cmd, "@allowCSharpQualifiedContextMatch", allowCSharpContextMatch ? 1 : 0); + } +} diff --git a/src/CodeIndex/Database/DbReader.SymbolSearchProjection.cs b/src/CodeIndex/Database/DbReader.SymbolSearchProjection.cs new file mode 100644 index 0000000000..0328f8dcb9 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.SymbolSearchProjection.cs @@ -0,0 +1,158 @@ +using System.Text.Json; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private static class SymbolSearchRowProjector + { + private const int DefinitionSitesIndex = 22; + private const int UngroupedIdentifierStartColumnIndex = 36; + private const int GroupedIdentifierStartColumnIndex = 31; + + public static List ReadAll( + SqliteDataReader reader, + SymbolSearchQueryPlan plan) + { + var results = new List(); + var includeRankingMetadata = plan.SortMode != SymbolSortMode.Name; + var sortModeName = includeRankingMetadata + ? plan.SortMode.ToString().ToLowerInvariant() + : null; + while (reader.TrackedRead()) + results.Add(Read(reader, plan, includeRankingMetadata, sortModeName)); + return results; + } + + private static SymbolResult Read( + SqliteDataReader reader, + SymbolSearchQueryPlan plan, + bool includeRankingMetadata, + string? sortModeName) + { + var definitionSites = Convert.ToInt32(reader.GetInt64(DefinitionSitesIndex)); + var identifierColumn = plan.GroupPartials + ? GroupedIdentifierStartColumnIndex + : UngroupedIdentifierStartColumnIndex; + var result = new SymbolResult + { + Path = reader.GetString(0), + Lang = GetNullableString(reader, 1), + Kind = reader.GetString(2), + SubKind = GetNullableString(reader, 3), + Name = reader.GetString(4), + Line = reader.GetInt32(5), + StartLine = GetInt32OrFallback(reader, 6, 5), + StartColumn = GetNullableInt32(reader, identifierColumn) + ?? ResolveSymbolIdentifierStartColumn( + GetNullableInt32(reader, 7), + GetNullableString(reader, 11), + reader.GetString(4), + reader.GetString(2)), + EndLine = GetInt32OrFallback(reader, 8, 5), + BodyStartLine = GetNullableInt32(reader, 9), + BodyEndLine = GetNullableInt32(reader, 10), + Signature = GetNullableString(reader, 11), + ContainerKind = GetNullableString(reader, 12), + ContainerName = GetNullableString(reader, 13), + ContainerQualifiedName = GetNullableString(reader, 25), + LogicalPartialKey = GetNullableString(reader, 26), + Visibility = GetNullableString(reader, 14), + ReturnType = GetNullableString(reader, 15), + SortMode = sortModeName, + ReferenceCount = includeRankingMetadata ? Convert.ToInt32(reader.GetInt64(16)) : null, + HotspotScore = includeRankingMetadata ? Math.Round(reader.GetDouble(17), 3) : null, + RankingReferenceScore = includeRankingMetadata ? Math.Round(reader.GetDouble(18), 3) : null, + RankingHotspotScore = includeRankingMetadata ? Math.Round(reader.GetDouble(19), 3) : null, + GenericNamePenalty = includeRankingMetadata ? Math.Round(reader.GetDouble(20), 3) : null, + StructuralRankPenalty = includeRankingMetadata ? Math.Round(reader.GetDouble(21), 3) : null, + DefinitionSites = includeRankingMetadata || (plan.GroupPartials && definitionSites > 1) ? definitionSites : null, + SizeLines = includeRankingMetadata ? Convert.ToInt32(reader.GetInt64(23)) : null, + ComplexityScore = includeRankingMetadata ? Math.Round(reader.GetDouble(24), 3) : null, + SymbolId = reader.GetInt64(27), + }; + AddPartialFamily(reader, plan, definitionSites, result); + return result; + } + + private static void AddPartialFamily( + SqliteDataReader reader, + SymbolSearchQueryPlan plan, + int definitionSites, + SymbolResult result) + { + if (!plan.GroupPartials || definitionSites <= 1) + return; + + result.PartialFamilyId = + LogicalPartialSymbolGrouper.BuildPartialFamilyId(result.LogicalPartialKey!); + result.RepresentativeReason = reader.GetString(28); + result.FamilyMembers = ReadPartialFamilyMembers(reader.GetString(29), result); + result.FamilyMembersTruncated = reader.GetInt64(30) != 0; + } + } + + private static List ReadPartialFamilyMembers(string json, SymbolResult representative) + { + using var document = JsonDocument.Parse(json); + var members = new List(); + foreach (var element in document.RootElement.EnumerateArray()) + { + var symbolId = element.GetProperty("symbol_id").GetInt64(); + var path = element.GetProperty("path").GetString() ?? string.Empty; + var startLine = element.GetProperty("start_line").GetInt32(); + var rawStartColumn = element.GetProperty("start_column").ValueKind == JsonValueKind.Null + ? (int?)null + : element.GetProperty("start_column").GetInt32(); + var memberName = element.GetProperty("name").GetString() ?? representative.Name; + var memberSignature = element.GetProperty("signature").ValueKind == JsonValueKind.Null + ? null + : element.GetProperty("signature").GetString(); + var identifierStartColumn = element.TryGetProperty("identifier_start_column", out var identifierColumnElement) + && identifierColumnElement.ValueKind != JsonValueKind.Null + ? identifierColumnElement.GetInt32() + : (int?)null; + members.Add(new PartialFamilyMember + { + SymbolId = symbolId, + Path = path, + Line = element.GetProperty("line").GetInt32(), + StartLine = startLine, + StartColumn = identifierStartColumn + ?? ResolveSymbolIdentifierStartColumn( + rawStartColumn, + memberSignature, + memberName, + representative.Kind), + EndLine = element.GetProperty("end_line").GetInt32(), + Generated = element.GetProperty("generated").GetInt32() != 0, + Representative = representative.SymbolId == symbolId + || (representative.SymbolId == null + && string.Equals(representative.Path, path, StringComparison.Ordinal) + && representative.StartLine == startLine), + }); + } + return members; + } + + private static int? ResolveSymbolIdentifierStartColumn( + int? declarationStartColumn, + string? signature, + string name, + string kind) + { + if (!declarationStartColumn.HasValue || string.IsNullOrWhiteSpace(signature) || string.IsNullOrEmpty(name)) + return declarationStartColumn; + + var firstLineEnd = signature.IndexOfAny(['\r', '\n']); + var firstLine = firstLineEnd >= 0 ? signature[..firstLineEnd] : signature; + var callable = kind is "function" or "test.method"; + var relativeColumn = callable + ? LogicalPartialSymbolGrouper.FindCallableNameOffset(firstLine, name) + : firstLine.IndexOf(name, StringComparison.Ordinal); + if (relativeColumn < 0 && callable) + relativeColumn = firstLine.IndexOf(name, StringComparison.Ordinal); + return relativeColumn >= 0 ? declarationStartColumn.Value + relativeColumn : declarationStartColumn; + } +} diff --git a/src/CodeIndex/Database/DbReader.SymbolSearchQueryBinding.cs b/src/CodeIndex/Database/DbReader.SymbolSearchQueryBinding.cs new file mode 100644 index 0000000000..1506fc92a5 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.SymbolSearchQueryBinding.cs @@ -0,0 +1,338 @@ +using CodeIndex.Indexer; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private static class SymbolSearchQueryPredicateBuilder + { + public static string BuildFull(DbReader reader, SymbolSearchQueryPlan plan) + { + if (plan.Queries is not { Count: > 0 } queries) + return string.Empty; + + var clauses = plan.Exact + ? queries.Select((query, index) => BuildFullExact(reader, plan, query, index)) + : queries.Select((query, index) => BuildFullLike(reader, query, index)); + return $" AND ({string.Join(" OR ", clauses)})"; + } + + public static string BuildBounded(DbReader reader, SymbolSearchQueryPlan plan) + { + if (plan.Queries is not { Count: 1 } queries) + return string.Empty; + + var query = queries[0]; + var allowLeafFallback = !SqlNameResolver.HasQualifier(query); + var qualifiedSymbolClause = SqlNameResolver.HasQualifier(query) + ? reader.BuildQualifiedSymbolMatchSql("query0", reader._foldReady) + : null; + var csharpExplicitInterfaceClause = allowLeafFallback + ? reader.BuildCSharpExplicitInterfaceShortAliasMatchSql("query0") + : reader.BuildCSharpExplicitInterfaceIdentityMatchSql("query0"); + var rustQualifiedExact = ShouldPreserveRustQualifiedExactQuery(query, plan.Lang, plan.Exact); + var rustQualifiedParts = rustQualifiedExact + ? NormalizeRustQualifiedExactQueryParts(query) + : default; + if (!plan.Exact) + { + return $" AND (s.name LIKE @query0 ESCAPE '\\' OR (f.lang = 'sql' AND sql_normalize_name(s.name) LIKE @query0NormalizedLike ESCAPE '\\'){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause} OR {csharpExplicitInterfaceClause}" : string.Empty)})"; + } + + if (rustQualifiedParts.QualifiedPath != null) + { + return reader._foldReady + ? " AND ((s.container_qualified_name = @query0RustContainer COLLATE NOCASE OR s.container_name = @query0RustContainer COLLATE NOCASE) AND s.name_folded = @query0RustLeafFolded)" + : " AND ((s.container_qualified_name = @query0RustContainer COLLATE NOCASE OR s.container_name = @query0RustContainer COLLATE NOCASE) AND s.name = @query0RustLeaf COLLATE NOCASE)"; + } + + return " AND " + BuildExactCore( + reader, + query, + plan.Lang, + "query0", + allowLeafFallback, + qualifiedSymbolClause, + csharpExplicitInterfaceClause, + markdownAnchorClause: null, + swiftBacktickClause: string.Empty); + } + + public static void AppendFilters( + DbReader reader, + ref string sql, + SymbolSearchQueryPlan plan, + bool includeLineRange) + { + if (plan.Kind != null) + sql += " AND s.kind = @kind"; + if (plan.Lang != null) + sql += SymbolLanguageFileIdFilter; + if (plan.Since != null && reader._fileColumns.Contains("modified")) + sql += " AND f.modified >= @since"; + if (includeLineRange && plan.StartLine != null) + sql += " AND s.line >= @startLine"; + if (includeLineRange && plan.EndLine != null) + sql += " AND s.line <= @endLine"; + AppendPathFilters( + ref sql, + plan.PathPatterns, + plan.ExcludePathPatterns, + plan.ExcludeTests); + reader.AppendVisibilityFilters( + ref sql, + plan.VisibilityFilters, + plan.ExcludeVisibilityFilters); + } + + private static string BuildFullExact( + DbReader reader, + SymbolSearchQueryPlan plan, + string query, + int index) + { + var parameterStem = $"query{index}"; + var rustQualifiedExact = + ShouldPreserveRustQualifiedExactQuery(query, plan.Lang, plan.Exact); + var rustQualifiedParts = rustQualifiedExact + ? NormalizeRustQualifiedExactQueryParts(query) + : default; + if (rustQualifiedParts.QualifiedPath != null) + { + return reader._foldReady + ? $"((s.container_qualified_name = @{parameterStem}RustContainer COLLATE NOCASE OR s.container_name = @{parameterStem}RustContainer COLLATE NOCASE) AND s.name_folded = @{parameterStem}RustLeafFolded)" + : $"((s.container_qualified_name = @{parameterStem}RustContainer COLLATE NOCASE OR s.container_name = @{parameterStem}RustContainer COLLATE NOCASE) AND s.name = @{parameterStem}RustLeaf COLLATE NOCASE)"; + } + + var allowLeafFallback = !SqlNameResolver.HasQualifier(query); + var markdownAnchorClause = reader._symbolColumns.Contains("name_folded") + ? $"(f.lang = 'markdown' AND ((s.kind = 'heading' AND s.name_folded = @{parameterStem}MarkdownHeading) OR (s.kind = 'anchor' AND s.name_folded = @{parameterStem}MarkdownExplicitAnchor COLLATE BINARY)))" + : "0"; + var qualifiedSymbolClause = SqlNameResolver.HasQualifier(query) + ? reader.BuildQualifiedSymbolMatchSql(parameterStem, reader._foldReady) + : null; + var csharpExplicitInterfaceClause = allowLeafFallback + ? reader.BuildCSharpExplicitInterfaceShortAliasMatchSql(parameterStem) + : reader.BuildCSharpExplicitInterfaceIdentityMatchSql(parameterStem); + var swiftBacktickAlias = ComputeSwiftBacktickAlias(query, plan.Lang); + var swiftBacktickClause = swiftBacktickAlias != null + ? reader._foldReady + ? $" OR s.name_folded = @{parameterStem}SwiftBacktickAlias" + : $" OR s.name = @{parameterStem}SwiftBacktickAlias COLLATE NOCASE" + : string.Empty; + return BuildExactCore( + reader, + query, + plan.Lang, + parameterStem, + allowLeafFallback, + qualifiedSymbolClause, + csharpExplicitInterfaceClause, + markdownAnchorClause, + swiftBacktickClause); + } + + private static string BuildExactCore( + DbReader reader, + string query, + string? lang, + string parameterStem, + bool allowLeafFallback, + string? qualifiedSymbolClause, + string csharpExplicitInterfaceClause, + string? markdownAnchorClause, + string swiftBacktickClause) + { + var parameterSql = $"@{parameterStem}"; + var aliasClauses = $"{swiftBacktickClause} OR {csharpExplicitInterfaceClause}"; + if (markdownAnchorClause != null) + aliasClauses += $" OR {markdownAnchorClause}"; + if (reader._foldReady) + { + return allowLeafFallback + ? $"({reader.BuildExactPrimarySymbolNameMatchSql(parameterSql, true, query, lang)}{aliasClauses} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @{parameterStem}SegmentCount AND sql_normalize_name_folded(s.name) = @{parameterStem}NormalizedFolded) OR sql_leaf_name_folded(s.name) = @{parameterStem}LeafFolded)))" + : $"({reader.BuildExactPrimarySymbolNameMatchSql(parameterSql, true, query, lang)}{aliasClauses} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @{parameterStem}SegmentCount AND sql_normalize_name_folded(s.name) = @{parameterStem}NormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})"; + } + + return allowLeafFallback + ? $"({reader.BuildExactPrimarySymbolNameMatchSql(parameterSql, false, query, lang)}{aliasClauses} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @{parameterStem}SegmentCount AND sql_normalize_name(s.name) = @{parameterStem}Normalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @{parameterStem}Leaf COLLATE NOCASE)))" + : $"({reader.BuildExactPrimarySymbolNameMatchSql(parameterSql, false, query, lang)}{aliasClauses} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @{parameterStem}SegmentCount AND sql_normalize_name(s.name) = @{parameterStem}Normalized COLLATE NOCASE){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})"; + } + + private static string BuildFullLike(DbReader reader, string query, int index) + { + var parameterStem = $"query{index}"; + var qualifiedSymbolClause = SqlNameResolver.HasQualifier(query) + ? reader.BuildQualifiedSymbolMatchSql(parameterStem, reader._foldReady) + : null; + var csharpExplicitInterfaceClause = SqlNameResolver.HasQualifier(query) + ? reader.BuildCSharpExplicitInterfaceIdentityMatchSql(parameterStem) + : null; + var markdownAnchorLikeClause = reader._symbolColumns.Contains("name_folded") + ? $" OR (f.lang = 'markdown' AND ((s.kind = 'heading' AND s.name_folded LIKE @{parameterStem}MarkdownHeadingLike ESCAPE '\\') OR (s.kind = 'anchor' AND instr(s.name_folded, @{parameterStem}MarkdownExplicitAnchor) > 0)))" + : string.Empty; + return $"(s.name LIKE @{parameterStem} ESCAPE '\\'{markdownAnchorLikeClause} OR (f.lang = 'sql' AND sql_normalize_name(s.name) LIKE @{parameterStem}NormalizedLike ESCAPE '\\'){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause} OR {csharpExplicitInterfaceClause}" : string.Empty)})"; + } + } + + private static class SymbolSearchQueryBinder + { + public static void BindFullQueries( + DbReader reader, + SqliteCommand command, + SymbolSearchQueryPlan plan) + { + if (plan.Queries == null) + return; + + for (var index = 0; index < plan.Queries.Count; index++) + BindFullQuery(reader, command, plan, plan.Queries[index], index); + } + + public static void BindBoundedQuery( + DbReader reader, + SqliteCommand command, + SymbolSearchQueryPlan plan) + { + if (plan.Queries is not { Count: 1 } queries) + return; + + BindBaseQuery(reader, command, plan, queries[0], 0); + BindQualifiedAndRust(command, plan, queries[0], 0); + } + + public static void BindFilters( + DbReader reader, + SqliteCommand command, + SymbolSearchQueryPlan plan, + bool includeLineRange) + { + if (plan.Kind != null) + SqliteCommandPolicy.Add(command, "@kind", plan.Kind); + if (plan.Lang != null) + SqliteCommandPolicy.Add(command, "@lang", plan.Lang); + if (plan.Since != null && reader._fileColumns.Contains("modified")) + SqliteCommandPolicy.Add(command, "@since", plan.Since.Value); + if (includeLineRange && plan.StartLine != null) + SqliteCommandPolicy.Add(command, "@startLine", plan.StartLine.Value); + if (includeLineRange && plan.EndLine != null) + SqliteCommandPolicy.Add(command, "@endLine", plan.EndLine.Value); + AddPathFilterParameters( + command, + plan.PathPatterns, + plan.ExcludePathPatterns); + AddVisibilityFilterParameters( + command, + plan.VisibilityFilters, + plan.ExcludeVisibilityFilters); + } + + public static void BindListOrdering(SqliteCommand command, SymbolSearchQueryPlan plan) + { + var hasSingleQuery = plan.Queries is { Count: 1 }; + var query = hasSingleQuery ? plan.Queries![0] : string.Empty; + var preferSqlLeafMatch = hasSingleQuery && !SqlNameResolver.HasQualifier(query); + SqliteCommandPolicy.Add(command, "@preferLiteralExactMatch", hasSingleQuery ? 1 : 0); + SqliteCommandPolicy.Add(command, "@preferLiteralNormalizedSqlMatch", hasSingleQuery ? 1 : 0); + SqliteCommandPolicy.Add(command, "@preferCaseInsensitiveExactMatch", hasSingleQuery ? 1 : 0); + SqliteCommandPolicy.Add(command, "@preferCaseInsensitiveNormalizedSqlMatch", hasSingleQuery ? 1 : 0); + SqliteCommandPolicy.Add(command, "@preferCaseInsensitiveSqlLeafMatch", preferSqlLeafMatch ? 1 : 0); + SqliteCommandPolicy.Add(command, "@rawQuery", query); + var normalized = hasSingleQuery ? SqlNameResolver.NormalizeQualifiedName(query) : string.Empty; + SqliteCommandPolicy.Add(command, "@rawQueryNormalized", normalized); + SqliteCommandPolicy.Add(command, "@rawQueryNormalizedFolded", hasSingleQuery ? NameFold.Fold(normalized) ?? normalized : string.Empty); + var leaf = hasSingleQuery ? SqlNameResolver.GetLeafName(query) : string.Empty; + SqliteCommandPolicy.Add(command, "@rawQueryLeaf", leaf); + SqliteCommandPolicy.Add(command, "@rawQueryLeafFolded", hasSingleQuery ? NameFold.Fold(leaf) ?? leaf : string.Empty); + SqliteCommandPolicy.Add(command, "@rawQuerySegmentCount", hasSingleQuery ? SqlNameResolver.GetSegmentCount(query) : 0); + } + + private static void BindFullQuery( + DbReader reader, + SqliteCommand command, + SymbolSearchQueryPlan plan, + string query, + int index) + { + BindBaseQuery(reader, command, plan, query, index); + var parameterStem = $"query{index}"; + if (reader._symbolColumns.Contains("name_folded")) + { + var heading = MarkdownAnchorIdentity.NormalizeHeadingFragment(query); + SqliteCommandPolicy.Add(command, $"@{parameterStem}MarkdownHeading", heading); + SqliteCommandPolicy.Add(command, $"@{parameterStem}MarkdownHeadingLike", $"%{EscapeLikeQuery(heading)}%"); + SqliteCommandPolicy.Add( + command, + $"@{parameterStem}MarkdownExplicitAnchor", + MarkdownAnchorIdentity.NormalizeExplicitAnchorDefinition(query)); + } + + BindQualifiedAndRust(command, plan, query, index); + var swiftBacktickAlias = ComputeSwiftBacktickAlias(query, plan.Lang); + if (swiftBacktickAlias != null) + { + SqliteCommandPolicy.Add( + command, + $"@{parameterStem}SwiftBacktickAlias", + reader._foldReady + ? NameFold.Fold(swiftBacktickAlias) ?? swiftBacktickAlias + : swiftBacktickAlias); + } + } + + private static void BindBaseQuery( + DbReader reader, + SqliteCommand command, + SymbolSearchQueryPlan plan, + string query, + int index) + { + var parameterStem = $"query{index}"; + var parameterName = $"@{parameterStem}"; + var parameterValue = !plan.Exact + ? $"%{EscapeLikeQuery(query)}%" + : reader._foldReady + ? FoldNameForLanguage(query, plan.Lang) + : query; + if (plan.Exact && reader._foldReady) + AddPersistedFoldedNameQueryParameters(command, parameterName, query, plan.Lang); + else + SqliteCommandPolicy.Add(command, parameterName, parameterValue); + + var normalized = SqlNameResolver.NormalizeQualifiedName(query); + SqliteCommandPolicy.Add(command, $"@{parameterStem}Normalized", normalized); + SqliteCommandPolicy.Add(command, $"@{parameterStem}NormalizedFolded", NameFold.Fold(normalized) ?? normalized); + var leaf = GetQualifiedQueryLeaf(query, plan.Lang); + SqliteCommandPolicy.Add(command, $"@{parameterStem}Leaf", leaf); + SqliteCommandPolicy.Add(command, $"@{parameterStem}LeafFolded", NameFold.Fold(leaf) ?? leaf); + SqliteCommandPolicy.Add(command, $"@{parameterStem}SegmentCount", SqlNameResolver.GetSegmentCount(query)); + SqliteCommandPolicy.Add(command, $"@{parameterStem}NormalizedLike", $"%{EscapeLikeQuery(normalized)}%"); + AddCSharpExplicitInterfaceIdentityQueryParameter(command, parameterStem, query); + } + + private static void BindQualifiedAndRust( + SqliteCommand command, + SymbolSearchQueryPlan plan, + string query, + int index) + { + var parameterStem = $"query{index}"; + if (SqlNameResolver.HasQualifier(query)) + AddQualifiedSymbolQueryParameters(command, parameterStem, query); + + var rustQualifiedExact = + ShouldPreserveRustQualifiedExactQuery(query, plan.Lang, plan.Exact); + var rustQualifiedParts = rustQualifiedExact + ? NormalizeRustQualifiedExactQueryParts(query) + : default; + if (rustQualifiedParts.QualifiedPath != null) + { + var rustLeaf = rustQualifiedParts.LeafName ?? string.Empty; + SqliteCommandPolicy.Add(command, $"@{parameterStem}RustContainer", rustQualifiedParts.ContainerPath ?? string.Empty); + SqliteCommandPolicy.Add(command, $"@{parameterStem}RustLeaf", rustLeaf); + SqliteCommandPolicy.Add(command, $"@{parameterStem}RustLeafFolded", NameFold.Fold(rustLeaf) ?? rustLeaf); + } + } + } +} diff --git a/src/CodeIndex/Database/DbReader.SymbolSearchQueryNormalizer.cs b/src/CodeIndex/Database/DbReader.SymbolSearchQueryNormalizer.cs new file mode 100644 index 0000000000..700013d4cf --- /dev/null +++ b/src/CodeIndex/Database/DbReader.SymbolSearchQueryNormalizer.cs @@ -0,0 +1,355 @@ +using System.Text.RegularExpressions; +using CodeIndex.Indexer; +using Regex = CodeIndex.Indexer.BoundedRegex; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private static class SymbolSearchQueryNormalizer + { + private sealed class NormalizedSymbolSearchQueryList : List + { + public NormalizedSymbolSearchQueryList(IEnumerable queries) + : base(queries) + { + } + } + + public static IReadOnlyList MarkNormalized(IEnumerable queries) + => new NormalizedSymbolSearchQueryList(queries); + + public static IReadOnlyList? NormalizeQueries( + IReadOnlyList? queries, + string? lang, + bool exact) + { + if (queries == null) + return null; + if (queries is NormalizedSymbolSearchQueryList) + return queries; + + var seen = new HashSet(StringComparer.Ordinal); + var normalized = new List(); + foreach (var query in queries) + { + var value = NormalizeForSymbolSearch(query, lang, exact) ?? query ?? string.Empty; + if (value.Length == 0 || !seen.Add(value)) + continue; + normalized.Add(value); + } + + return new NormalizedSymbolSearchQueryList(normalized); + } + + public static string? Normalize(string? query, string? lang, bool exact) + { + if (!string.IsNullOrWhiteSpace(lang) + && string.Equals(lang, "rust", StringComparison.OrdinalIgnoreCase)) + { + return RustQueryStrategy.Normalize(query, exact); + } + + if (!string.IsNullOrWhiteSpace(lang) + && string.Equals(lang, "javascript", StringComparison.OrdinalIgnoreCase)) + { + return JavaScriptQueryStrategy.Normalize(query); + } + + var terraformNormalized = TerraformQueryStrategy.Normalize(query, lang); + return terraformNormalized ?? CSharpQueryStrategy.NormalizeVerbatim(query, lang); + } + + public static string? NormalizeForSymbolSearch(string? query, string? lang, bool exact) + { + if (RustQueryStrategy.ShouldPreserveQualifiedExactQuery(query, lang, exact)) + return query?.Trim(); + + if (exact + && !string.IsNullOrWhiteSpace(query) + && SqlNameResolver.HasQualifier(query)) + { + if (string.Equals(NormalizeQueryLanguage(lang), "csharp", StringComparison.Ordinal)) + return CSharpSymbolNameNormalizer.NormalizeExplicitInterfaceQueryDisplayName(query); + + // Without a language filter the query must retain its original spelling for + // non-C# exact matching. C#-specific display and identity aliases are supplied + // through their own SQL parameters. + // 言語フィルターがない場合、C# 以外の完全一致を保つため query の元表記を + // 維持する。C# 専用の表示名・identity alias は個別の SQL parameter で渡す。 + if (string.IsNullOrWhiteSpace(lang)) + { + var terraformNormalized = TerraformQueryStrategy.Normalize(query, lang); + return terraformNormalized ?? query.Trim(); + } + } + + return Normalize(query, lang, exact) ?? query; + } + + public static string? NormalizeCSharpVerbatim(string? query, string? lang) + => CSharpQueryStrategy.NormalizeVerbatim(query, lang); + + public static string? ComputeSwiftBacktickAlias(string? query, string? lang) + => SwiftQueryStrategy.ComputeBacktickAlias(query, lang); + + public static bool ShouldPreserveRustQualifiedExactQuery(string? query, string? lang, bool exact) + => RustQueryStrategy.ShouldPreserveQualifiedExactQuery(query, lang, exact); + + public static (string? QualifiedPath, string? ContainerPath, string? LeafName) + NormalizeRustQualifiedExactQueryParts(string query) + => RustQueryStrategy.NormalizeQualifiedExactQueryParts(query); + + private static class TerraformQueryStrategy + { + private static readonly Regex VarLocalModuleQueryRegex = new( + @"^(?:var|local|module)\.(?[A-Za-z_]\w*)(?:\..*)?$", + RegexOptions.Compiled); + + private static readonly Regex DataQueryRegex = new( + @"^data\.[A-Za-z_]\w*\.(?[A-Za-z_]\w*)(?:\..*)?$", + RegexOptions.Compiled); + + public static string? Normalize(string? query, string? lang) + { + if (!string.IsNullOrWhiteSpace(lang) + && !string.Equals(lang, "terraform", StringComparison.OrdinalIgnoreCase)) + { + return null; + } + + if (string.IsNullOrWhiteSpace(query)) + return null; + + var trimmed = query.Trim(); + if (trimmed.Length == 0) + return null; + + // Terraform dotted prefixes are stored as bare names in references and symbols. + var simpleMatch = VarLocalModuleQueryRegex.Match(trimmed); + if (simpleMatch.Success) + return simpleMatch.Groups["name"].Value; + + var dataMatch = DataQueryRegex.Match(trimmed); + return dataMatch.Success ? dataMatch.Groups["name"].Value : null; + } + } + + private static class JavaScriptQueryStrategy + { + public static string? Normalize(string? query) + { + if (query == null) + return null; + + var trimmed = query.Trim(); + if (trimmed.Length == 0) + return null; + + var commonJsPrefixLength = GetCommonJsPrefixLength(trimmed); + if (commonJsPrefixLength == 0) + return trimmed; + + trimmed = trimmed[commonJsPrefixLength..]; + if (trimmed.Length == 0) + return null; + + trimmed = trimmed.TrimStart(); + if (trimmed.StartsWith(".", StringComparison.Ordinal)) + trimmed = trimmed[1..].TrimStart(); + + return GetCommonJsLeaf(trimmed); + } + + private static int GetCommonJsPrefixLength(string query) + { + if (query.StartsWith("module.exports", StringComparison.Ordinal)) + { + var nextIndex = "module.exports".Length; + return query.Length > nextIndex && query[nextIndex] is '.' or '[' ? nextIndex : 0; + } + + if (query.StartsWith("exports", StringComparison.Ordinal)) + { + var nextIndex = "exports".Length; + return query.Length > nextIndex && query[nextIndex] is '.' or '[' ? nextIndex : 0; + } + + return 0; + } + + private static string? GetCommonJsLeaf(string query) + { + var bracketLeaf = NormalizeBracketLeaf(query); + if (bracketLeaf != null) + return bracketLeaf; + + var leafIndex = query.LastIndexOf('.'); + var leaf = leafIndex >= 0 ? query[(leafIndex + 1)..] : query; + return NormalizeBracketLeaf(leaf) ?? (leaf.Length == 0 ? null : leaf); + } + + private static string? NormalizeBracketLeaf(string query) + { + var trimmed = query.Trim(); + if (trimmed.Length < 3 || trimmed[0] != '[' || trimmed[^1] != ']') + return null; + + var inner = trimmed[1..^1].Trim(); + if (inner.Length < 2) + return null; + + var quote = inner[0]; + if (quote is not '\'' and not '"') + return null; + if (inner[^1] != quote) + return null; + + var leaf = inner[1..^1].Trim(); + return leaf.Length == 0 ? null : leaf; + } + } + + private static class RustQueryStrategy + { + public static string? Normalize(string? query, bool exact = false) + { + if (query == null) + return null; + + var macroQuery = query.Trim(); + if (macroQuery.Length == 0) + return null; + + var isMacroQuery = macroQuery.EndsWith("!", StringComparison.Ordinal); + if (isMacroQuery) + macroQuery = macroQuery[..^1].TrimEnd(); + if (macroQuery.Length == 0) + return null; + + if (exact && isMacroQuery && macroQuery.Contains("::", StringComparison.Ordinal)) + return NormalizeQualifiedMacroQuery(macroQuery); + + var leafIndex = macroQuery.LastIndexOf("::", StringComparison.Ordinal); + if (leafIndex >= 0) + macroQuery = macroQuery[(leafIndex + 2)..].Trim(); + if (macroQuery.StartsWith("r#", StringComparison.Ordinal)) + macroQuery = macroQuery[2..]; + + return macroQuery.Length == 0 ? null : macroQuery; + } + + public static bool ShouldPreserveQualifiedExactQuery(string? query, string? lang, bool exact) + { + return exact + && !string.IsNullOrWhiteSpace(lang) + && string.Equals(lang, "rust", StringComparison.OrdinalIgnoreCase) + && !string.IsNullOrWhiteSpace(query) + && query.Contains("::", StringComparison.Ordinal); + } + + public static (string? QualifiedPath, string? ContainerPath, string? LeafName) + NormalizeQualifiedExactQueryParts(string query) + { + var trimmed = query.Trim(); + if (trimmed.Length == 0) + return (null, null, null); + + if (trimmed.EndsWith("!", StringComparison.Ordinal)) + trimmed = trimmed[..^1].TrimEnd(); + + var normalized = NormalizeQualifiedMacroQuery(trimmed); + if (string.IsNullOrWhiteSpace(normalized)) + return (null, null, null); + + normalized = TrimQualifiedPathPrefixes(normalized.Replace("::", ".")); + var lastDot = normalized.LastIndexOf('.'); + return lastDot < 0 + ? (normalized, string.Empty, normalized) + : (normalized, normalized[..lastDot], normalized[(lastDot + 1)..]); + } + + private static string? NormalizeQualifiedMacroQuery(string query) + { + var segments = query + .Split("::", StringSplitOptions.None) + .Select(segment => segment.Trim()) + .Where(segment => segment.Length > 0) + .Select(segment => segment.StartsWith("r#", StringComparison.Ordinal) ? segment[2..] : segment) + .ToList(); + + return segments.Count == 0 ? null : string.Join("::", segments); + } + + private static string TrimQualifiedPathPrefixes(string query) + { + while (query.StartsWith("crate.", StringComparison.Ordinal) + || query.StartsWith("self.", StringComparison.Ordinal) + || query.StartsWith("super.", StringComparison.Ordinal)) + { + var dotIndex = query.IndexOf('.'); + if (dotIndex < 0 || dotIndex == query.Length - 1) + break; + + query = query[(dotIndex + 1)..]; + } + + return query; + } + } + + private static class SwiftQueryStrategy + { + public static string? ComputeBacktickAlias(string? query, string? lang) + { + if (!string.Equals(lang, "swift", StringComparison.OrdinalIgnoreCase) + || string.IsNullOrWhiteSpace(query)) + { + return null; + } + + var trimmed = query.Trim(); + if (trimmed.Length == 0 + || trimmed.IndexOfAny(['`', ':', '/', '<', '>', '(', ')', '[', ']', ' ']) >= 0) + { + return null; + } + + var lastDot = trimmed.LastIndexOf('.'); + if (lastDot < 0) + return $"`{trimmed}`"; + if (lastDot == 0 || lastDot == trimmed.Length - 1) + return null; + + var prefix = trimmed[..(lastDot + 1)]; + var leaf = trimmed[(lastDot + 1)..]; + return leaf.IndexOf('.') >= 0 ? null : $"{prefix}`{leaf}`"; + } + } + + private static class CSharpQueryStrategy + { + // Query-side mirror of the C# declaration canonicalizer. C# source spellings such as + // `@class` are canonicalized when no language or C# is selected. Other languages, + // especially SQL, retain a leading `@`. Rust also retains its macro query policy. + public static string? NormalizeVerbatim(string? query, string? lang) + { + if (!string.IsNullOrWhiteSpace(lang) + && string.Equals(lang, "rust", StringComparison.OrdinalIgnoreCase)) + { + var rustNormalized = RustQueryStrategy.Normalize(query); + return string.IsNullOrWhiteSpace(rustNormalized) ? null : rustNormalized; + } + + if (!string.IsNullOrWhiteSpace(lang) + && !string.Equals(lang, "csharp", StringComparison.OrdinalIgnoreCase)) + { + return query; + } + + var normalized = query == null ? null : NormalizeDbCSharpQualifiedName(query); + return string.IsNullOrWhiteSpace(normalized) ? null : normalized; + } + } + } +} diff --git a/src/CodeIndex/Database/DbReader.SymbolSearchQueryPlan.cs b/src/CodeIndex/Database/DbReader.SymbolSearchQueryPlan.cs new file mode 100644 index 0000000000..374e9c7ab7 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.SymbolSearchQueryPlan.cs @@ -0,0 +1,37 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed record SymbolSearchQueryPlan + { + public IReadOnlyList? Queries { get; init; } + public int Limit { get; init; } = 20; + public string? Kind { get; init; } + public string? Lang { get; init; } + public IReadOnlyList? PathPatterns { get; init; } + public IReadOnlyList? ExcludePathPatterns { get; init; } + public bool ExcludeTests { get; init; } + public DateTime? Since { get; init; } + public bool Exact { get; init; } + public IReadOnlyList? VisibilityFilters { get; init; } + public IReadOnlyList? ExcludeVisibilityFilters { get; init; } + public SymbolSortMode SortMode { get; init; } = SymbolSortMode.Name; + public int? StartLine { get; init; } + public int? EndLine { get; init; } + public bool GroupPartials { get; init; } + public int Offset { get; init; } + } + + private static class SymbolSearchQueryPlanBuilder + { + public static SymbolSearchQueryPlan Build(SymbolSearchQueryPlan source) + { + var lang = NormalizeQueryLanguage(source.Lang); + return source with + { + Lang = lang, + Queries = NormalizeSymbolSearchQueries(source.Queries, lang, source.Exact), + }; + } + } +} diff --git a/src/CodeIndex/Database/DbReader.SymbolSearchSql.cs b/src/CodeIndex/Database/DbReader.SymbolSearchSql.cs new file mode 100644 index 0000000000..fd441c41f1 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.SymbolSearchSql.cs @@ -0,0 +1,149 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private static string GetGenericSymbolRankNamePenaltySql(string nameSql) + => $"CASE WHEN lower({nameSql}) IN {GenericSymbolRankNamesSql} THEN {GenericSymbolRankNamePenaltySqlLiteral} ELSE 1.0 END"; + + private static string BuildLogicalPartialSymbolQuery(string matchingSymbolsSql, SymbolSortMode sortMode) + { + var orderBy = BuildLogicalPartialSortOrderBy(sortMode); + return $@" + WITH matching_symbols AS ( + {matchingSymbolsSql} + ), + ranked_symbols AS ( + SELECT matching_symbols.*, + ROW_NUMBER() OVER ( + PARTITION BY logical_partial_key + ORDER BY canonical_primary_rank, + canonical_generated_rank, + canonical_semantic_score DESC, + canonical_declaration_identity COLLATE BINARY, + logical_partial_key COLLATE BINARY, + path COLLATE BINARY, + start_line, + stable_start_column, + symbol_id + ) AS logical_row_number, + ROW_NUMBER() OVER ( + PARTITION BY logical_partial_key + ORDER BY path COLLATE BINARY, start_line, stable_start_column, symbol_id + ) AS family_member_row_number, + COUNT(*) OVER (PARTITION BY logical_partial_key) AS logical_definition_sites + FROM matching_symbols + ), + family_ranked_symbols AS ( + SELECT ranked_symbols.*, + MAX(CASE WHEN logical_row_number = 1 THEN family_member_row_number END) OVER ( + PARTITION BY logical_partial_key + ) AS representative_member_row_number + FROM ranked_symbols + ), + logical_symbols AS ( + SELECT family_ranked_symbols.*, + MAX(reference_count) OVER (PARTITION BY logical_partial_key) AS logical_reference_count, + MAX(hotspot_score) OVER (PARTITION BY logical_partial_key) AS logical_hotspot_score, + MAX(ranking_reference_score) OVER (PARTITION BY logical_partial_key) AS logical_ranking_reference_score, + MAX(ranking_hotspot_score) OVER (PARTITION BY logical_partial_key) AS logical_ranking_hotspot_score, + MAX(generic_name_penalty) OVER (PARTITION BY logical_partial_key) AS logical_generic_name_penalty, + MAX(structural_rank_penalty) OVER (PARTITION BY logical_partial_key) AS logical_structural_rank_penalty, + MAX(size_lines) OVER (PARTITION BY logical_partial_key) AS logical_size_lines, + MAX(complexity_score) OVER (PARTITION BY logical_partial_key) AS logical_complexity_score, + MIN(exact_name_order) OVER (PARTITION BY logical_partial_key) AS logical_exact_name_order, + MIN(path_bucket) OVER (PARTITION BY logical_partial_key) AS logical_path_bucket, + MIN(visibility_rank) OVER (PARTITION BY logical_partial_key) AS logical_visibility_rank, + MIN(canonical_primary_rank) OVER (PARTITION BY logical_partial_key) AS logical_primary_rank_min, + MAX(canonical_primary_rank) OVER (PARTITION BY logical_partial_key) AS logical_primary_rank_max, + MIN(canonical_generated_rank) OVER (PARTITION BY logical_partial_key) AS logical_generated_rank_min, + MAX(canonical_generated_rank) OVER (PARTITION BY logical_partial_key) AS logical_generated_rank_max, + MIN(canonical_semantic_score) OVER (PARTITION BY logical_partial_key) AS logical_semantic_score_min, + MAX(canonical_semantic_score) OVER (PARTITION BY logical_partial_key) AS logical_semantic_score_max, + MIN(canonical_declaration_identity) OVER (PARTITION BY logical_partial_key) AS logical_declaration_identity_min, + MAX(canonical_declaration_identity) OVER (PARTITION BY logical_partial_key) AS logical_declaration_identity_max, + json_group_array(json_object( + 'symbol_id', symbol_id, + 'path', path, + 'line', line, + 'start_line', start_line, + 'start_column', start_column, + 'end_line', end_line, + 'name', name, + 'signature', signature, + 'identifier_start_column', identifier_start_column, + 'generated', canonical_generated_rank + )) FILTER (WHERE + family_member_row_number <= CASE + WHEN representative_member_row_number <= {LogicalPartialSymbolGrouper.FamilyMemberLimit} + THEN {LogicalPartialSymbolGrouper.FamilyMemberLimit} + ELSE {LogicalPartialSymbolGrouper.FamilyMemberLimit - 1} + END + OR logical_row_number = 1 + ) OVER ( + PARTITION BY logical_partial_key + ORDER BY path COLLATE BINARY, start_line, stable_start_column, symbol_id + ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING + ) AS logical_family_members_json + FROM family_ranked_symbols + ) + SELECT path, lang, kind, sub_kind, name, line, + start_line, start_column, end_line, + body_start_line, body_end_line, signature, + container_kind, container_name, visibility, return_type, + logical_reference_count, logical_hotspot_score, + logical_ranking_reference_score, logical_ranking_hotspot_score, + logical_generic_name_penalty, logical_structural_rank_penalty, + logical_definition_sites, logical_size_lines, logical_complexity_score, + container_qualified_name, logical_partial_key, symbol_id, + CASE + WHEN logical_primary_rank_min <> logical_primary_rank_max THEN '{LogicalPartialSymbolGrouper.ImplementationBodyReason}' + WHEN logical_generated_rank_min <> logical_generated_rank_max THEN '{LogicalPartialSymbolGrouper.NonGeneratedSourceReason}' + WHEN logical_semantic_score_min <> logical_semantic_score_max THEN '{LogicalPartialSymbolGrouper.SemanticDeclarationReason}' + WHEN logical_declaration_identity_min <> logical_declaration_identity_max THEN '{LogicalPartialSymbolGrouper.CanonicalDeclarationIdentityReason}' + ELSE '{LogicalPartialSymbolGrouper.StableLocationReason}' + END AS representative_reason, + logical_family_members_json, + CASE WHEN logical_definition_sites > {LogicalPartialSymbolGrouper.FamilyMemberLimit} THEN 1 ELSE 0 END AS family_members_truncated, + identifier_start_column + FROM logical_symbols + WHERE logical_row_number = 1 + {orderBy}"; + } + + private static string BuildLogicalPartialSortOrderBy(SymbolSortMode sortMode) + { + const string stableTieBreakers = "logical_path_bucket, logical_visibility_rank, name, path COLLATE BINARY, line, stable_start_column, symbol_id"; + return sortMode switch + { + SymbolSortMode.Hotspot => $"ORDER BY logical_ranking_hotspot_score DESC, logical_ranking_reference_score DESC, logical_hotspot_score DESC, logical_reference_count DESC, logical_size_lines DESC, {stableTieBreakers}", + SymbolSortMode.References => $"ORDER BY logical_ranking_reference_score DESC, logical_ranking_hotspot_score DESC, logical_reference_count DESC, logical_hotspot_score DESC, logical_size_lines DESC, {stableTieBreakers}", + SymbolSortMode.Size => $"ORDER BY logical_size_lines DESC, logical_ranking_reference_score DESC, logical_ranking_hotspot_score DESC, logical_reference_count DESC, {stableTieBreakers}", + SymbolSortMode.Complexity => $"ORDER BY logical_complexity_score DESC, logical_ranking_hotspot_score DESC, logical_ranking_reference_score DESC, logical_reference_count DESC, logical_size_lines DESC, {stableTieBreakers}", + SymbolSortMode.Path => "ORDER BY path COLLATE BINARY, line, stable_start_column, name, symbol_id", + _ => $"ORDER BY logical_exact_name_order, {stableTieBreakers}", + }; + } + + private string BuildSymbolSortOrderBy( + SymbolSortMode sortMode, + string exactNameOrderSql, + string referenceCountSql, + string rawHotspotScoreSql, + string rankingReferenceScoreSql, + string rankingHotspotScoreSql, + string sizeLinesSql, + string complexityScoreSql, + string startColumnSql) + { + var stableTieBreakers = $"{PathBucketOrder}, {VisibilityOrder}, s.name, f.path, s.line, {startColumnSql} ASC, s.id ASC"; + return sortMode switch + { + SymbolSortMode.Hotspot => $" ORDER BY {rankingHotspotScoreSql} DESC, {rankingReferenceScoreSql} DESC, {rawHotspotScoreSql} DESC, {referenceCountSql} DESC, {sizeLinesSql} DESC, {stableTieBreakers}", + SymbolSortMode.References => $" ORDER BY {rankingReferenceScoreSql} DESC, {rankingHotspotScoreSql} DESC, {referenceCountSql} DESC, {rawHotspotScoreSql} DESC, {sizeLinesSql} DESC, {stableTieBreakers}", + SymbolSortMode.Size => $" ORDER BY {sizeLinesSql} DESC, {rankingReferenceScoreSql} DESC, {rankingHotspotScoreSql} DESC, {referenceCountSql} DESC, {stableTieBreakers}", + SymbolSortMode.Complexity => $" ORDER BY {complexityScoreSql} DESC, {rankingHotspotScoreSql} DESC, {rankingReferenceScoreSql} DESC, {referenceCountSql} DESC, {sizeLinesSql} DESC, {stableTieBreakers}", + SymbolSortMode.Path => $" ORDER BY f.path, s.line, {startColumnSql} ASC, s.name, s.id ASC", + _ => $" ORDER BY {exactNameOrderSql}, {stableTieBreakers}", + }; + } +} diff --git a/src/CodeIndex/Database/DbReader.SymbolSearchTotalCount.cs b/src/CodeIndex/Database/DbReader.SymbolSearchTotalCount.cs new file mode 100644 index 0000000000..9f88eb61f8 --- /dev/null +++ b/src/CodeIndex/Database/DbReader.SymbolSearchTotalCount.cs @@ -0,0 +1,69 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + public QueryCountResult CountSearchSymbolsTotal(string? query = null, string? kind = null, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, DateTime? since = null, bool exact = false, IReadOnlyList? visibilityFilters = null, IReadOnlyList? excludeVisibilityFilters = null, bool groupPartials = false) + { + return CountSearchSymbolsTotal(query == null ? null : new[] { NormalizeSymbolSearchQueryForSymbolSearch(query, lang, exact) ?? query ?? string.Empty }, kind, lang, pathPatterns, excludePathPatterns, excludeTests, since, exact, visibilityFilters, excludeVisibilityFilters, groupPartials); + } + + public QueryCountResult CountSearchSymbolsTotal(IReadOnlyList? queries, string? kind = null, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, DateTime? since = null, bool exact = false, IReadOnlyList? visibilityFilters = null, IReadOnlyList? excludeVisibilityFilters = null, bool groupPartials = false) + { + var plan = SymbolSearchQueryPlanBuilder.Build(new SymbolSearchQueryPlan + { + Queries = queries, + Kind = kind, + Lang = lang, + PathPatterns = pathPatterns, + ExcludePathPatterns = excludePathPatterns, + ExcludeTests = excludeTests, + Since = since, + Exact = exact, + VisibilityFilters = visibilityFilters, + ExcludeVisibilityFilters = excludeVisibilityFilters, + GroupPartials = groupPartials, + }); + return ExecuteTotalSymbolSearch(plan); + } + + private QueryCountResult ExecuteTotalSymbolSearch(SymbolSearchQueryPlan plan) + { + if (plan.GroupPartials) + { + EnsureCSharpCallableTypeKinds( + plan.Lang, + plan.Queries, + plan.Exact, + plan.Kind); + } + using var cmd = _conn.CreateCommand(); + + var logicalPartialKeySql = LogicalPartialQuerySql.BuildKey( + this, + GetSymbolColumnSql("signature"), + GetSymbolColumnSql("container_name"), + GetSymbolColumnSql("container_qualified_name"), + GetSymbolColumnSql("family_key"), + GetSymbolColumnSql("return_type")); + var countSql = plan.GroupPartials + ? $"COUNT(DISTINCT ({logicalPartialKeySql}))" + : "COUNT(*)"; + var sql = $@" + SELECT {countSql}, COUNT(DISTINCT f.path) + FROM symbols s + JOIN files f ON s.file_id = f.id + WHERE 1=1"; + + sql += SymbolSearchQueryPredicateBuilder.BuildFull(this, plan); + SymbolSearchQueryPredicateBuilder.AppendFilters(this, ref sql, plan, includeLineRange: false); + + cmd.CommandText = sql; + SymbolSearchQueryBinder.BindFullQueries(this, cmd, plan); + SymbolSearchQueryBinder.BindFilters(this, cmd, plan, includeLineRange: false); + + using var reader = cmd.ExecuteTrackedReader(); + return reader.TrackedRead() + ? new QueryCountResult(reader.GetInt32(0), reader.GetInt32(1)) + : new QueryCountResult(0, 0); + } +} diff --git a/src/CodeIndex/Database/DbReader.cs b/src/CodeIndex/Database/DbReader.cs index 2e67d6f85f..991d984671 100644 --- a/src/CodeIndex/Database/DbReader.cs +++ b/src/CodeIndex/Database/DbReader.cs @@ -80,15 +80,13 @@ public partial class DbReader : IDisposable private readonly IReadOnlySet _fileColumns; private readonly IReadOnlySet _symbolColumns; private readonly IReadOnlySet _referenceColumns; + private readonly IReadOnlySet _chunkColumns; private readonly IReadOnlySet _chunkIndexes; private readonly IReadOnlySet _symbolIndexes; private readonly IReadOnlySet _referenceIndexes; private readonly HashSet _indexedHotspotFamilyLanguages; - private readonly Dictionary> _csharpUsingStaticScopesByPath = new(StringComparer.Ordinal); - private readonly Dictionary> _csharpNamespaceScopesByPath = new(StringComparer.Ordinal); + private readonly Dictionary _csharpUsingCatalogsByPath = new(StringComparer.Ordinal); private readonly Dictionary> _csharpContainingTypeScopesByPath = new(StringComparer.Ordinal); - private readonly Dictionary> _csharpUsingNamespaceScopesByPath = new(StringComparer.Ordinal); - private readonly Dictionary> _csharpUsingAliasScopesByPath = new(StringComparer.Ordinal); private readonly Dictionary>> _activeCSharpTypeNamespacesByPathLine = new(StringComparer.Ordinal); private readonly Dictionary>> _activeCSharpContainingTypeScopesByPathLine = new(StringComparer.Ordinal); private readonly Dictionary>> _activeCSharpUsingStaticTargetsByPathLine = new(StringComparer.Ordinal); @@ -98,9 +96,7 @@ public partial class DbReader : IDisposable private readonly Dictionary> _csharpInheritedContainingTypesByQualifiedName = new(StringComparer.Ordinal); private readonly Dictionary _csharpContainingTypeScopeByQualifiedName = new(StringComparer.Ordinal); private bool _disposed; - private HashSet? _csharpGlobalUsingStaticTargets; - private HashSet? _csharpGlobalUsingNamespaces; - private Dictionary? _csharpGlobalUsingAliasesByName; + private CSharpGlobalUsingCatalog? _csharpGlobalUsingCatalog; private IReadOnlyCollection? _workspaceSupportedReferenceLanguages; internal readonly bool _hasReferencesTable; internal readonly bool _hasHotspotReferenceCountsTable; @@ -280,11 +276,7 @@ ELSE 0 private const string SyntheticTopLevelCallerName = ""; private const string SyntheticTopLevelCallerKind = "function"; - private sealed record CSharpNamespaceScope(string QualifiedName, int ScopeStartLine, int ScopeEndLine); private sealed record CSharpContainingTypeScope(string Path, string Kind, string QualifiedName, string? Visibility, string? Signature, int DeclarationLine, int ScopeStartLine, int ScopeEndLine); - private sealed record CSharpUsingStaticScope(string TargetQualifiedName, int Line, int ScopeStartLine, int ScopeEndLine); - private sealed record CSharpUsingNamespaceScope(string TargetQualifiedName, int Line, int ScopeStartLine, int ScopeEndLine); - private sealed record CSharpUsingAliasScope(string AliasName, string TargetQualifiedName, int Line, int ScopeStartLine, int ScopeEndLine, bool TargetsType); private sealed record CSharpTypeNamespaceCandidate(string QualifiedName, string Path, bool IsFileLocal); private sealed record CSharpContainingTypeCandidate(string QualifiedName, bool AccessibleFromDerivedType); @@ -607,6 +599,7 @@ private DbReader( GeneratedColumnAvailableScope.Value = _fileColumns.Contains("generated"); _symbolColumns = LoadColumns("symbols"); _referenceColumns = LoadColumns("symbol_references"); + _chunkColumns = LoadColumns("chunks"); _chunkIndexes = LoadIndexes("chunks"); _symbolIndexes = LoadIndexes("symbols"); int userVersion; @@ -741,11 +734,8 @@ public void Dispose() return; _disposed = true; - _csharpUsingStaticScopesByPath.Clear(); - _csharpNamespaceScopesByPath.Clear(); + _csharpUsingCatalogsByPath.Clear(); _csharpContainingTypeScopesByPath.Clear(); - _csharpUsingNamespaceScopesByPath.Clear(); - _csharpUsingAliasScopesByPath.Clear(); _activeCSharpTypeNamespacesByPathLine.Clear(); _activeCSharpContainingTypeScopesByPathLine.Clear(); _activeCSharpUsingStaticTargetsByPathLine.Clear(); @@ -754,9 +744,7 @@ public void Dispose() _csharpTypeContainingTypesByName.Clear(); _csharpInheritedContainingTypesByQualifiedName.Clear(); _csharpContainingTypeScopeByQualifiedName.Clear(); - _csharpGlobalUsingStaticTargets = null; - _csharpGlobalUsingNamespaces = null; - _csharpGlobalUsingAliasesByName = null; + _csharpGlobalUsingCatalog = null; } /// diff --git a/src/CodeIndex/Database/DbSymbolReader.HotspotCandidates.cs b/src/CodeIndex/Database/DbSymbolReader.HotspotCandidates.cs new file mode 100644 index 0000000000..6aaa214526 --- /dev/null +++ b/src/CodeIndex/Database/DbSymbolReader.HotspotCandidates.cs @@ -0,0 +1,243 @@ +using System.Text; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private readonly record struct SymbolHotspotCandidatePlan( + string Sql, + List GraphLanguages, + List HotspotFamilyLanguages, + int? CandidateLimit); + + private SymbolHotspotCandidatePlan BuildSymbolHotspotCandidatePlan( + int? resultLimit, + string? kind, + string? lang, + IReadOnlyList? pathPatterns, + IReadOnlyList? excludePathPatterns, + bool excludeTests, + IReadOnlyList? visibilityFilters, + IReadOnlyList? excludeVisibilityFilters) + { + // Ambiguity is computed from the visibility-filtered language/kind candidate set + // before path and test filters. This prevents an out-of-scope duplicate from + // promoting a same-name symbol to codebase-wide counting. Cross-file grouping is + // allowed only for authoritative family keys in fully-ready languages. + // 曖昧性は visibility / language / kind 適用後、path / test 適用前に判定する。 + // scope 外の同名定義による誤った全体集計を防ぎ、cross-file 集約は ready な family に限る。 + var graphLanguages = new List(GetWorkspaceSupportedReferenceLanguages()); + var hotspotFamilyLanguages = new List(_hotspotFamilyReadyLanguages); + hotspotFamilyLanguages.Sort(StringComparer.Ordinal); + var familyLanguageConditionSql = hotspotFamilyLanguages.Count > 0 + ? $"f.lang IN ({BuildIndexedParameterList("hotspotFamilyLang", hotspotFamilyLanguages.Count)})" + : "0"; + var familyKeySql = GetSymbolColumnSql("family_key"); + var familyTargetKeySql = hotspotFamilyLanguages.Count > 0 + ? $@"CASE + WHEN {familyLanguageConditionSql} + AND COALESCE({familyKeySql}, '') <> '' + THEN 'family|' || COALESCE(f.lang, '') || '|' || COALESCE(s.kind, '') || '|' || {familyKeySql} + ELSE NULL + END" + : "NULL"; + var containerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name"); + var containerTargetKeySql = $@"CASE + WHEN COALESCE({containerQualifiedNameSql}, '') <> '' + THEN 'container|' || CAST(s.file_id AS TEXT) || '|' || COALESCE(s.kind, '') || '|' || {containerQualifiedNameSql} + ELSE NULL + END"; + var candidateLimit = GetBoundedHotspotCandidateLimit(resultLimit); + var sql = new StringBuilder(capacity: 5_000); + sql.Append($@" + WITH {BuildBoundedHotspotCandidatePrefix(candidateLimit)}all_candidate_symbols AS MATERIALIZED ( + SELECT s.id, s.file_id, s.name, s.kind, f.path, f.lang, s.line, + {GetSymbolColumnSql("visibility")} AS visibility, + {GetSymbolColumnSql("container_name")} AS container_name, + CASE + WHEN {familyTargetKeySql} IS NOT NULL + THEN {familyTargetKeySql} + WHEN {containerTargetKeySql} IS NOT NULL + THEN {containerTargetKeySql} + ELSE 'file|' || CAST(s.file_id AS TEXT) + END AS logical_target_key, + COALESCE({familyTargetKeySql}, {containerTargetKeySql}) AS count_safe_key + FROM symbols s + JOIN files f ON s.file_id = f.id + WHERE s.kind NOT IN ('import', 'namespace')"); + sql.Append(BuildCSharpHotspotFunctionDefinitionGateSql()); + sql.Append(BuildBoundedHotspotSymbolPredicate(candidateLimit)); + AppendSymbolHotspotCandidateFilters( + sql, + graphLanguages, + kind, + lang, + visibilityFilters, + excludeVisibilityFilters); + sql.Append(@" + ), + name_cardinality AS ( + SELECT lang, + name, + COUNT(*) AS defs, + COUNT(DISTINCT logical_target_key) AS target_groups, + COUNT(DISTINCT count_safe_key) AS count_safe_groups, + COUNT(count_safe_key) AS count_safe_defs + FROM all_candidate_symbols + GROUP BY lang, name + ), + filtered_candidates AS MATERIALIZED ( + SELECT id, + file_id, + name, + kind, + path, + lang, + line, + visibility, + container_name, + logical_target_key + FROM all_candidate_symbols + WHERE 1 = 1"); + AppendSymbolHotspotPathFilters( + sql, + pathPatterns, + excludePathPatterns, + excludeTests); + sql.Append(@" + ),"); + return new SymbolHotspotCandidatePlan( + sql.ToString(), + graphLanguages, + hotspotFamilyLanguages, + candidateLimit); + } + + private void AppendSymbolHotspotCandidateFilters( + StringBuilder sql, + IReadOnlyList graphLanguages, + string? kind, + string? lang, + IReadOnlyList? visibilityFilters, + IReadOnlyList? excludeVisibilityFilters) + { + var filterSql = string.Empty; + if (lang != null) + filterSql += SymbolLanguageFileIdFilter; + else + filterSql += $" AND f.lang IN ({BuildIndexedParameterList("gl", graphLanguages.Count)})"; + if (kind != null) + filterSql += " AND s.kind = @kind"; + AppendVisibilityFilters(ref filterSql, visibilityFilters, excludeVisibilityFilters); + sql.Append(filterSql); + } + + private string BuildCSharpHotspotFunctionDefinitionGateSql() => + _symbolColumns.Contains("body_start_line") + && _symbolColumns.Contains("body_end_line") + && _symbolColumns.Contains("signature") + && _symbolColumns.Contains("container_kind") + ? @" + AND NOT ( + f.lang = 'csharp' + AND s.kind = 'function' + AND s.container_kind = 'function' + AND ( + (s.body_start_line IS NULL AND s.body_end_line IS NULL) + OR (s.container_kind = 'function' AND COALESCE(s.signature, '') LIKE '%.' || s.name || '(%') + ) + )" + : string.Empty; + + private static void AppendSymbolHotspotPathFilters( + StringBuilder sql, + IReadOnlyList? pathPatterns, + IReadOnlyList? excludePathPatterns, + bool excludeTests) + { + if (pathPatterns is { Count: > 0 }) + { + sql.Append(" AND ("); + for (var index = 0; index < pathPatterns.Count; index++) + { + if (index > 0) + sql.Append(" OR "); + sql.Append(BuildPathColumnFilterPredicate("path", "pathPattern", index, pathPatterns[index])); + } + sql.Append(')'); + } + if (excludePathPatterns != null) + { + for (var index = 0; index < excludePathPatterns.Count; index++) + { + sql.Append(" AND NOT "); + sql.Append(BuildPathColumnFilterPredicate( + "path", + "excludePathPattern", + index, + excludePathPatterns[index])); + } + } + if (excludeTests) + { + sql.Append(" AND NOT "); + sql.Append(TestPathCondition.Replace("f.path", "path", StringComparison.Ordinal)); + } + } + + private static string BuildIndexedParameterList(string prefix, int count) + { + var parameters = new StringBuilder(count * (prefix.Length + 4)); + for (var index = 0; index < count; index++) + { + if (index > 0) + parameters.Append(','); + parameters.Append('@'); + parameters.Append(prefix); + parameters.Append(index); + } + return parameters.ToString(); + } + + private static void AddSymbolHotspotParameters( + SqliteCommand command, + SymbolHotspotRowsQuery query, + int? limit, + string? kind, + string? lang, + IReadOnlyList? pathPatterns, + IReadOnlyList? excludePathPatterns, + IReadOnlyList? visibilityFilters, + IReadOnlyList? excludeVisibilityFilters) + { + if (limit.HasValue) + SqliteCommandPolicy.Add(command, "@limit", limit.Value); + var candidatePlan = query.CandidatePlan; + if (candidatePlan.CandidateLimit.HasValue) + { + SqliteCommandPolicy.Add( + command, + "@candidateReferenceLimit", + candidatePlan.CandidateLimit.Value); + } + if (lang != null) + SqliteCommandPolicy.Add(command, "@lang", lang); + else + { + for (var index = 0; index < candidatePlan.GraphLanguages.Count; index++) + SqliteCommandPolicy.Add(command, $"@gl{index}", candidatePlan.GraphLanguages[index]); + } + if (kind != null) + SqliteCommandPolicy.Add(command, "@kind", kind); + AddPathFilterParameters(command, pathPatterns, excludePathPatterns); + AddVisibilityFilterParameters(command, visibilityFilters, excludeVisibilityFilters); + for (var index = 0; index < candidatePlan.HotspotFamilyLanguages.Count; index++) + { + SqliteCommandPolicy.Add( + command, + $"@hotspotFamilyLang{index}", + candidatePlan.HotspotFamilyLanguages[index]); + } + } +} diff --git a/src/CodeIndex/Database/DbSymbolReader.Hotspots.cs b/src/CodeIndex/Database/DbSymbolReader.Hotspots.cs index 83db8b8660..613c985c05 100644 --- a/src/CodeIndex/Database/DbSymbolReader.Hotspots.cs +++ b/src/CodeIndex/Database/DbSymbolReader.Hotspots.cs @@ -11,115 +11,18 @@ public partial class DbReader { private SymbolHotspotRowsQuery BuildGroupedSymbolHotspotRowsQuery(int? resultLimit, string? kind, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, IReadOnlyList? visibilityFilters, IReadOnlyList? excludeVisibilityFilters) { - var containerNameSql = GetSymbolColumnSql("container_name"); - var containerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name"); - var familyKeySql = GetSymbolColumnSql("family_key"); - var hotspotFamilyLangs = _hotspotFamilyReadyLanguages - .OrderBy(value => value, StringComparer.Ordinal) - .ToList(); - var familyLangConditionSql = hotspotFamilyLangs.Count > 0 - ? $"f.lang IN ({string.Join(",", hotspotFamilyLangs.Select((_, i) => $"@hotspotFamilyLang{i}"))})" - : "0"; - var familyTargetKeySql = hotspotFamilyLangs.Count > 0 - ? $@"CASE - WHEN {familyLangConditionSql} - AND COALESCE({familyKeySql}, '') <> '' - THEN 'family|' || COALESCE(f.lang, '') || '|' || COALESCE(s.kind, '') || '|' || {familyKeySql} - ELSE NULL - END" - : "NULL"; - var containerTargetKeySql = $@"CASE - WHEN COALESCE({containerQualifiedNameSql}, '') <> '' - THEN 'container|' || CAST(s.file_id AS TEXT) || '|' || COALESCE(s.kind, '') || '|' || {containerQualifiedNameSql} - ELSE NULL - END"; - var csharpFunctionDefinitionGateSql = _symbolColumns.Contains("body_start_line") - && _symbolColumns.Contains("body_end_line") - && _symbolColumns.Contains("signature") - && _symbolColumns.Contains("container_kind") - ? @" - AND NOT ( - f.lang = 'csharp' - AND s.kind = 'function' - AND s.container_kind = 'function' - AND ( - (s.body_start_line IS NULL AND s.body_end_line IS NULL) - OR (s.container_kind = 'function' AND COALESCE(s.signature, '') LIKE '%.' || s.name || '(%') - ) - )" - : string.Empty; - var graphLangs = GetWorkspaceSupportedReferenceLanguages().ToList(); - var candidateLimit = GetBoundedHotspotCandidateLimit(resultLimit); - var boundedCandidatePrefix = BuildBoundedHotspotCandidatePrefix(candidateLimit); - var boundedSymbolPredicate = BuildBoundedHotspotSymbolPredicate(candidateLimit); - var sql = $@" - WITH {boundedCandidatePrefix}all_candidate_symbols AS MATERIALIZED ( - SELECT s.id, s.file_id, s.name, s.kind, f.path, f.lang, s.line, - {GetSymbolColumnSql("visibility")} AS visibility, - {containerNameSql} AS container_name, - CASE - WHEN {familyTargetKeySql} IS NOT NULL - THEN {familyTargetKeySql} - WHEN {containerTargetKeySql} IS NOT NULL - THEN {containerTargetKeySql} - ELSE 'file|' || CAST(s.file_id AS TEXT) - END AS logical_target_key, - COALESCE({familyTargetKeySql}, {containerTargetKeySql}) AS count_safe_key - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE s.kind NOT IN ('import', 'namespace')" + csharpFunctionDefinitionGateSql + boundedSymbolPredicate; - - if (lang != null) - sql += SymbolLanguageFileIdFilter; - else - sql += $" AND f.lang IN ({string.Join(",", graphLangs.Select((_, i) => $"@gl{i}"))})"; - if (kind != null) - sql += " AND s.kind = @kind"; - AppendVisibilityFilters(ref sql, visibilityFilters, excludeVisibilityFilters); - - sql += @" - ), - name_cardinality AS ( - SELECT lang, - name, - COUNT(*) AS defs, - COUNT(DISTINCT logical_target_key) AS target_groups, - COUNT(DISTINCT count_safe_key) AS count_safe_groups, - COUNT(count_safe_key) AS count_safe_defs - FROM all_candidate_symbols - GROUP BY lang, name - ), - filtered_candidates AS MATERIALIZED ( - SELECT id, - file_id, - name, - kind, - path, - lang, - line, - visibility, - container_name, - logical_target_key - FROM all_candidate_symbols - WHERE 1 = 1"; - if (pathPatterns is { Count: > 0 }) - { - var ors = new List(pathPatterns.Count); - for (int i = 0; i < pathPatterns.Count; i++) - ors.Add(BuildPathColumnFilterPredicate("path", "pathPattern", i, pathPatterns[i])); - sql += " AND (" + string.Join(" OR ", ors) + ")"; - } - if (excludePathPatterns != null) - { - for (int i = 0; i < excludePathPatterns.Count; i++) - sql += $" AND NOT {BuildPathColumnFilterPredicate("path", "excludePathPattern", i, excludePathPatterns[i])}"; - } - if (excludeTests) - sql += $" AND NOT {TestPathCondition.Replace("f.path", "path")}"; - sql += @" - ), + var candidatePlan = BuildSymbolHotspotCandidatePlan( + resultLimit, + kind, + lang, + pathPatterns, + excludePathPatterns, + excludeTests, + visibilityFilters, + excludeVisibilityFilters); + var sql = candidatePlan.Sql + @" logical_references AS MATERIALIZED ( - " + BuildHotspotLogicalReferenceRowsSql(includeLeafMetadata: false, boundedCandidates: candidateLimit.HasValue) + @" + " + BuildHotspotLogicalReferenceRowsSql(includeLeafMetadata: false, boundedCandidates: candidatePlan.CandidateLimit.HasValue) + @" ), file_reference_counts AS MATERIALIZED ( SELECT lang, @@ -296,7 +199,7 @@ JOIN grouped_reference_counts grc GROUP BY hs.name, hs.kind )"; - return new SymbolHotspotRowsQuery(sql, graphLangs, hotspotFamilyLangs, candidateLimit); + return new SymbolHotspotRowsQuery(sql, candidatePlan); } public HotspotCountResult CountGroupedSymbolHotspots(string? kind, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, IReadOnlyList? visibilityFilters = null, IReadOnlyList? excludeVisibilityFilters = null) @@ -883,125 +786,16 @@ WHERE rc.ref_count > 0 private SymbolHotspotRowsQuery BuildSymbolHotspotRowsQuery(int? resultLimit, string? kind, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, IReadOnlyList? visibilityFilters, IReadOnlyList? excludeVisibilityFilters) { - var containerNameSql = GetSymbolColumnSql("container_name"); - var containerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name"); - var familyKeySql = GetSymbolColumnSql("family_key"); - var hotspotFamilyLangs = _hotspotFamilyReadyLanguages - .OrderBy(value => value, StringComparer.Ordinal) - .ToList(); - var familyLangConditionSql = hotspotFamilyLangs.Count > 0 - ? $"f.lang IN ({string.Join(",", hotspotFamilyLangs.Select((_, i) => $"@hotspotFamilyLang{i}"))})" - : "0"; - var familyTargetKeySql = hotspotFamilyLangs.Count > 0 - ? $@"CASE - WHEN {familyLangConditionSql} - AND COALESCE({familyKeySql}, '') <> '' - THEN 'family|' || COALESCE(f.lang, '') || '|' || COALESCE(s.kind, '') || '|' || {familyKeySql} - ELSE NULL - END" - : "NULL"; - var containerTargetKeySql = $@"CASE - WHEN COALESCE({containerQualifiedNameSql}, '') <> '' - THEN 'container|' || CAST(s.file_id AS TEXT) || '|' || COALESCE(s.kind, '') || '|' || {containerQualifiedNameSql} - ELSE NULL - END"; - var csharpFunctionDefinitionGateSql = _symbolColumns.Contains("body_start_line") - && _symbolColumns.Contains("body_end_line") - && _symbolColumns.Contains("signature") - && _symbolColumns.Contains("container_kind") - ? @" - AND NOT ( - f.lang = 'csharp' - AND s.kind = 'function' - AND s.container_kind = 'function' - AND ( - (s.body_start_line IS NULL AND s.body_end_line IS NULL) - OR (s.container_kind = 'function' AND COALESCE(s.signature, '') LIKE '%.' || s.name || '(%') - ) - )" - : string.Empty; - // Ambiguity is computed from the unscoped language/kind candidate set so `--path` - // cannot hide an out-of-scope duplicate and accidentally promote a same-name symbol - // back to codebase-wide counting. Cross-file grouping is allowed only when the - // extractor persisted an authoritative family key on a DB that is stamped as fully - // current for hotspot-family semantics (currently partial-type families). Same-file - // same-container overloads can still share one conservative target key, but only - // unique names or authoritative families may promote to codebase-wide counts. - // 曖昧性は path 非依存の候補集合で判定し、`--path` で隠れた重複定義が一意扱いに - // 戻ってしまうことを防ぐ。cross-file の集約は current な hotspot-family semantics で - // fully-ready と判定された DB 上の正式な family key のみに限定し、same-file の - // same-container overload は保守的な target として扱いつつ、codebase-wide 集計への - // 昇格は一意名か authoritative family のみに限定する。 - var candidateLimit = GetBoundedHotspotCandidateLimit(resultLimit); - var boundedCandidatePrefix = BuildBoundedHotspotCandidatePrefix(candidateLimit); - var boundedSymbolPredicate = BuildBoundedHotspotSymbolPredicate(candidateLimit); - var sql = $@" - WITH {boundedCandidatePrefix}all_candidate_symbols AS MATERIALIZED ( - SELECT s.id, s.file_id, s.name, s.kind, f.path, f.lang, s.line, - {GetSymbolColumnSql("visibility")} AS visibility, - {containerNameSql} AS container_name, - CASE - WHEN {familyTargetKeySql} IS NOT NULL - THEN {familyTargetKeySql} - WHEN {containerTargetKeySql} IS NOT NULL - THEN {containerTargetKeySql} - ELSE 'file|' || CAST(s.file_id AS TEXT) - END AS logical_target_key, - COALESCE({familyTargetKeySql}, {containerTargetKeySql}) AS count_safe_key - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE s.kind NOT IN ('import', 'namespace')" + csharpFunctionDefinitionGateSql + boundedSymbolPredicate; - - var graphLangs = GetWorkspaceSupportedReferenceLanguages().ToList(); - if (lang != null) - sql += SymbolLanguageFileIdFilter; - else - sql += $" AND f.lang IN ({string.Join(",", graphLangs.Select((_, i) => $"@gl{i}"))})"; - if (kind != null) - sql += " AND s.kind = @kind"; - AppendVisibilityFilters(ref sql, visibilityFilters, excludeVisibilityFilters); - - sql += @" - ), - name_cardinality AS ( - SELECT lang, - name, - COUNT(*) AS defs, - COUNT(DISTINCT logical_target_key) AS target_groups, - COUNT(DISTINCT count_safe_key) AS count_safe_groups, - COUNT(count_safe_key) AS count_safe_defs - FROM all_candidate_symbols - GROUP BY lang, name - ), - filtered_candidates AS MATERIALIZED ( - SELECT id, - file_id, - name, - kind, - path, - lang, - line, - visibility, - container_name, - logical_target_key - FROM all_candidate_symbols - WHERE 1 = 1"; - if (pathPatterns != null && pathPatterns.Count > 0) - { - var ors = new List(pathPatterns.Count); - for (int i = 0; i < pathPatterns.Count; i++) - ors.Add(BuildPathColumnFilterPredicate("path", "pathPattern", i, pathPatterns[i])); - sql += " AND (" + string.Join(" OR ", ors) + ")"; - } - if (excludePathPatterns != null) - { - for (int i = 0; i < excludePathPatterns.Count; i++) - sql += $" AND NOT {BuildPathColumnFilterPredicate("path", "excludePathPattern", i, excludePathPatterns[i])}"; - } - if (excludeTests) - sql += $" AND NOT {TestPathCondition.Replace("f.path", "path")}"; - sql += @" - ), + var candidatePlan = BuildSymbolHotspotCandidatePlan( + resultLimit, + kind, + lang, + pathPatterns, + excludePathPatterns, + excludeTests, + visibilityFilters, + excludeVisibilityFilters); + var sql = candidatePlan.Sql + @" grouped_candidates AS ( SELECT MIN(id) AS symbol_id, name, @@ -1043,7 +837,7 @@ JOIN grouped_metadata gm AND gm.kind = gc.kind ), logical_references AS MATERIALIZED ( - " + BuildHotspotLogicalReferenceRowsSql(includeLeafMetadata: true, boundedCandidates: candidateLimit.HasValue) + @" + " + BuildHotspotLogicalReferenceRowsSql(includeLeafMetadata: true, boundedCandidates: candidatePlan.CandidateLimit.HasValue) + @" ), file_reference_counts_exact AS MATERIALIZED ( SELECT lang, @@ -1223,35 +1017,12 @@ file_symbol_counts AS ( FROM filtered_candidates GROUP BY path, COALESCE(lang, '') )"; - return new SymbolHotspotRowsQuery(sql, graphLangs, hotspotFamilyLangs, candidateLimit); - } - - private static void AddSymbolHotspotParameters(SqliteCommand command, SymbolHotspotRowsQuery query, int? limit, string? kind, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, IReadOnlyList? visibilityFilters, IReadOnlyList? excludeVisibilityFilters) - { - if (limit.HasValue) - SqliteCommandPolicy.Add(command, "@limit", limit.Value); - if (query.CandidateLimit.HasValue) - SqliteCommandPolicy.Add(command, "@candidateReferenceLimit", query.CandidateLimit.Value); - if (lang != null) - SqliteCommandPolicy.Add(command, "@lang", lang); - else - { - for (int i = 0; i < query.GraphLanguages.Count; i++) - SqliteCommandPolicy.Add(command, $"@gl{i}", query.GraphLanguages[i]); - } - if (kind != null) - SqliteCommandPolicy.Add(command, "@kind", kind); - AddPathFilterParameters(command, pathPatterns, excludePathPatterns); - AddVisibilityFilterParameters(command, visibilityFilters, excludeVisibilityFilters); - for (int i = 0; i < query.HotspotFamilyLanguages.Count; i++) - SqliteCommandPolicy.Add(command, $"@hotspotFamilyLang{i}", query.HotspotFamilyLanguages[i]); + return new SymbolHotspotRowsQuery(sql, candidatePlan); } private sealed record SymbolHotspotRowsQuery( string Sql, - List GraphLanguages, - List HotspotFamilyLanguages, - int? CandidateLimit); + SymbolHotspotCandidatePlan CandidatePlan); private static HotspotCountResult ExecuteHotspotCountSummary(SqliteCommand command) { diff --git a/src/CodeIndex/Database/DbSymbolReader.Search.cs b/src/CodeIndex/Database/DbSymbolReader.Search.cs index 37d812f66f..4c2123ed46 100644 --- a/src/CodeIndex/Database/DbSymbolReader.Search.cs +++ b/src/CodeIndex/Database/DbSymbolReader.Search.cs @@ -1,10 +1,3 @@ -using System.Globalization; -using System.Text.Json; -using System.Text.RegularExpressions; -using CodeIndex.Indexer; -using Microsoft.Data.Sqlite; -using Regex = CodeIndex.Indexer.BoundedRegex; - namespace CodeIndex.Database; public partial class DbReader @@ -12,33 +5,11 @@ public partial class DbReader private const string GenericSymbolRankNamePenaltySqlLiteral = "0.01"; private const string GenericSymbolRankNamesSql = "('add','all','any','append','appendline','average','build','call','combine','contains','convert','count','create','distinct','equal','equals','execute','exists','file','files','first','firstordefault','get','getboolean','getbyte','getbytes','getchar','getchars','getdatetime','getdecimal','getdouble','getfieldvalue','getfloat','getguid','getint16','getint32','getint64','getordinal','getstring','gettemppath','getvalue','getvalues','groupby','handle','id','invoke','isdbnull','key','kind','last','lastordefault','length','line','list','load','name','orderby','orderbydescending','parse','path','process','read','resolve','run','set','single','singleordefault','skip','start','stop','sum','take','text','thenby','thenbydescending','tolist','tostring','tryparse','type','update','value','values','write')"; - private sealed class NormalizedSymbolSearchQueryList : List - { - public NormalizedSymbolSearchQueryList(IEnumerable queries) - : base(queries) - { - } - } - - private static IReadOnlyList? NormalizeSymbolSearchQueries(IReadOnlyList? queries, string? lang, bool exact) - { - if (queries == null) - return null; - if (queries is NormalizedSymbolSearchQueryList) - return queries; - - var seen = new HashSet(StringComparer.Ordinal); - var normalized = new List(); - foreach (var query in queries) - { - var value = NormalizeSymbolSearchQueryForSymbolSearch(query, lang, exact) ?? query ?? string.Empty; - if (value.Length == 0 || !seen.Add(value)) - continue; - normalized.Add(value); - } - - return new NormalizedSymbolSearchQueryList(normalized); - } + private static IReadOnlyList? NormalizeSymbolSearchQueries( + IReadOnlyList? queries, + string? lang, + bool exact) + => SymbolSearchQueryNormalizer.NormalizeQueries(queries, lang, exact); /// /// Escape LIKE wildcards (%, _) in user input to prevent unintended pattern matching. @@ -84,7 +55,7 @@ public List SearchSymbols(string? query = null, int limit = 20, st return SearchSymbols( normalizedQuery == null ? null - : new NormalizedSymbolSearchQueryList([normalizedQuery]), + : SymbolSearchQueryNormalizer.MarkNormalized([normalizedQuery]), limit, kind, lang, pathPatterns, excludePathPatterns, excludeTests, since, exact, visibilityFilters, excludeVisibilityFilters, sortMode, startLine, endLine, groupPartials, offset); @@ -96,7 +67,7 @@ public int CountSearchSymbols(string? query = null, int limit = 20, string? kind return CountSearchSymbols( normalizedQuery == null ? null - : new NormalizedSymbolSearchQueryList([normalizedQuery]), + : SymbolSearchQueryNormalizer.MarkNormalized([normalizedQuery]), limit, kind, lang, pathPatterns, excludePathPatterns, excludeTests, since, exact, visibilityFilters, excludeVisibilityFilters); } @@ -110,7 +81,7 @@ public bool AnySearchSymbols(IReadOnlyList? queries, string? kind = null foreach (var query in validQueries) { if (CountSearchSymbols( - new NormalizedSymbolSearchQueryList([query]), + SymbolSearchQueryNormalizer.MarkNormalized([query]), 1, kind, lang, @@ -127,1334 +98,18 @@ public bool AnySearchSymbols(IReadOnlyList? queries, string? kind = null return false; } - private string BuildQualifiedSymbolMatchSql(string parameterStem, bool useFoldedName, string symbolAlias = "s", string fileAlias = "f") - { - var containerNameSql = GetSymbolColumnSql("container_name", "''", symbolAlias); - var containerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name", containerNameSql, symbolAlias); - var nameMatchSql = useFoldedName - ? $"{symbolAlias}.name_folded = @{parameterStem}CSharpLeafFolded" - : $"{symbolAlias}.name = @{parameterStem}CSharpLeaf COLLATE NOCASE"; - var qualifiedNameMatchSql = useFoldedName - ? $"{symbolAlias}.name_folded = @{parameterStem}CSharpQualifiedFolded" - : $"{symbolAlias}.name = @{parameterStem}CSharpQualified COLLATE NOCASE"; - return $@"({fileAlias}.lang = 'csharp' - AND ({qualifiedNameMatchSql} - OR ({nameMatchSql} - AND ({containerNameSql} = @{parameterStem}Container COLLATE NOCASE - OR {containerQualifiedNameSql} = @{parameterStem}Container COLLATE NOCASE - OR {containerQualifiedNameSql} COLLATE NOCASE LIKE @{parameterStem}ContainerSuffixLike ESCAPE '\'))))"; - } - - private string BuildExactPrimarySymbolNameMatchSql( - string parameterSql, - bool useFoldedName, - string query, - string? lang) - { - var matchSql = useFoldedName - ? BuildPersistedFoldedNameMatchSql("s.name_folded", parameterSql) - : $"s.name = {parameterSql} COLLATE NOCASE"; - if (!string.IsNullOrWhiteSpace(lang) || !SqlNameResolver.HasQualifier(query)) - return matchSql; - - // Preserve direct qualified matching for ordinary and legacy C# rows. Only v3 - // explicit-interface rows have a display alias; those rows must use the C# identity - // clause so `IFoo.this` cannot also match a distinct `IFoo.@this` implementation. - // 通常および legacy C# row の修飾直接一致は維持する。表示 alias を持つ v3 の - // 明示的 interface row だけを C# identity 条件へ限定し、`IFoo.this` が別の - // `IFoo.@this` 実装にも一致しないようにする。 - var displayNameFoldedSql = GetSymbolColumnSql("display_name_folded", "NULL"); - return $"((f.lang <> 'csharp' OR {displayNameFoldedSql} IS NULL) AND {matchSql})"; - } - - private string BuildCSharpExplicitInterfaceIdentityMatchSql( - string parameterStem, - string symbolAlias = "s", - string fileAlias = "f") - { - if (!_csharpSymbolNameContractCurrent - || !_foldMetadataCurrent - || !_symbolColumns.Contains("name_folded")) - return "0"; - - return $"({fileAlias}.lang = 'csharp' AND {symbolAlias}.name_folded = @{parameterStem}CSharpExplicitInterfaceIdentityFolded)"; - } - - private string BuildCSharpExplicitInterfaceShortAliasMatchSql( - string parameterStem, - string symbolAlias = "s", - string fileAlias = "f") - => _foldReady - && _csharpSymbolNameContractCurrent - && _symbolColumns.Contains("display_name_folded") - && HasSymbolIndex("idx_symbols_display_name_folded") - ? $"({fileAlias}.lang = 'csharp' AND {symbolAlias}.display_name_folded = @{parameterStem}LeafFolded)" - : $"({fileAlias}.lang = 'csharp' AND {symbolAlias}.name = @{parameterStem}Leaf COLLATE NOCASE)"; - - private static void AddCSharpExplicitInterfaceIdentityQueryParameter( - SqliteCommand cmd, - string parameterStem, - string query) - { - SqliteCommandPolicy.Add( - cmd, - $"@{parameterStem}CSharpExplicitInterfaceIdentityFolded", - CSharpSymbolNameNormalizer.NormalizeExplicitInterfaceQueryIdentityNameFolded(query)); - } - - private static string GetQualifiedQueryContainer(string query) - { - var normalized = SqlNameResolver.NormalizeQualifiedName(query); - var lastDot = normalized.LastIndexOf('.'); - return lastDot > 0 ? normalized[..lastDot] : string.Empty; - } - - private static string GetQualifiedQuerySuffix(string query) - { - var normalized = SqlNameResolver.NormalizeQualifiedName(query); - var lastDot = normalized.LastIndexOf('.'); - if (lastDot <= 0) - return normalized; - var previousDot = normalized.LastIndexOf('.', lastDot - 1); - return previousDot >= 0 ? normalized[(previousDot + 1)..] : normalized; - } - - private static string GetQualifiedQueryLeaf(string query, string? lang) - { - var leaf = SqlNameResolver.GetLeafName(query); - return NormalizeCSharpVerbatimQuery(leaf, lang) ?? leaf; - } - - private static void AddQualifiedSymbolQueryParameters(SqliteCommand cmd, string parameterStem, string query) - { - var csharpDisplayQuery = - CSharpSymbolNameNormalizer.NormalizeExplicitInterfaceQueryDisplayName(query); - var csharpQualifiedQuery = - NormalizeCSharpVerbatimQuery(csharpDisplayQuery, "csharp") - ?? csharpDisplayQuery; - var container = GetQualifiedQueryContainer(csharpQualifiedQuery); - var csharpLeaf = GetQualifiedQueryLeaf(csharpQualifiedQuery, "csharp"); - SqliteCommandPolicy.Add(cmd, $"@{parameterStem}Container", container); - SqliteCommandPolicy.Add(cmd, $"@{parameterStem}ContainerSuffixLike", $"%.{EscapeLikeQuery(container)}"); - SqliteCommandPolicy.Add(cmd, $"@{parameterStem}CSharpQualified", csharpQualifiedQuery); - SqliteCommandPolicy.Add( - cmd, - $"@{parameterStem}CSharpQualifiedFolded", - NameFold.Fold(csharpQualifiedQuery) ?? csharpQualifiedQuery); - SqliteCommandPolicy.Add(cmd, $"@{parameterStem}CSharpLeaf", csharpLeaf); - SqliteCommandPolicy.Add( - cmd, - $"@{parameterStem}CSharpLeafFolded", - NameFold.Fold(csharpLeaf) ?? csharpLeaf); - } - - private bool HasSingleQualifiedSymbolDefinition(string query, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests) - { - if (!SqlNameResolver.HasQualifier(query)) - return false; - - var matches = SearchSymbols(query, 2, kind: null, lang, pathPatterns: null, excludePathPatterns: null, excludeTests, since: null, exact: false); - if (matches.Count != 1) - return false; - - var leafMatches = SearchSymbols(SqlNameResolver.GetLeafName(query), 2, kind: null, lang, pathPatterns: null, excludePathPatterns: null, excludeTests, since: null, exact: true); - return leafMatches.Count == 1; - } - - private bool HasQualifiedSymbolDefinition(string query, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests) - { - if (!SqlNameResolver.HasQualifier(query)) - return false; - - return SearchSymbols(query, 1, kind: null, lang, pathPatterns: null, excludePathPatterns: null, excludeTests, since: null, exact: false).Count > 0; - } - - private static string BuildQualifiedContextMatchSql(string contextSql, string columnSql, bool folded, bool like) - { - var functionName = (folded, like) switch - { - (true, true) => "sql_context_like_name_folded_at", - (true, false) => "sql_context_has_name_folded_at", - (false, true) => "sql_context_like_name_at", - _ => "sql_context_has_name_at", - }; - return $"({functionName}({contextSql}, @aliasQuery, {columnSql}) = 1 OR {functionName}({contextSql}, @aliasQuerySuffix, {columnSql}) = 1)"; - } - - private static string BuildQualifiedLeafFallbackSql(string nameSql, string foldedNameSql, bool folded) - => folded - ? $"(@allowQualifiedLeafFallback = 1 AND f.lang = 'csharp' AND {foldedNameSql} = @aliasQueryLeafFolded)" - : $"(@allowQualifiedLeafFallback = 1 AND f.lang = 'csharp' AND {nameSql} = @aliasQueryLeaf COLLATE NOCASE)"; - - private static string BuildCSharpQualifiedContextFallbackSql(string qualifiedContextSql) - => $"(@allowCSharpQualifiedContextMatch = 1 AND f.lang = 'csharp' AND {qualifiedContextSql})"; - - private static void AddQualifiedGraphQueryParameters(SqliteCommand cmd, string query, bool allowLeafFallback, bool allowCSharpContextMatch = false) - { - SqliteCommandPolicy.Add(cmd, "@aliasQuerySuffix", GetQualifiedQuerySuffix(query)); - SqliteCommandPolicy.Add(cmd, "@aliasQueryLeaf", SqlNameResolver.GetLeafName(query)); - SqliteCommandPolicy.Add(cmd, "@allowQualifiedLeafFallback", allowLeafFallback ? 1 : 0); - SqliteCommandPolicy.Add(cmd, "@allowCSharpQualifiedContextMatch", allowCSharpContextMatch ? 1 : 0); - } - - public int CountSearchSymbols(IReadOnlyList? queries, int limit = 20, string? kind = null, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, DateTime? since = null, bool exact = false, IReadOnlyList? visibilityFilters = null, IReadOnlyList? excludeVisibilityFilters = null) - { - lang = DbReader.NormalizeQueryLanguage(lang); - var validQueries = NormalizeSymbolSearchQueries(queries, lang, exact); - if (validQueries != null - && (validQueries.Count > 1 || string.Equals(lang, "markdown", StringComparison.Ordinal))) - return SearchSymbols(validQueries, limit, kind, lang, pathPatterns, excludePathPatterns, excludeTests, since, exact, visibilityFilters, excludeVisibilityFilters).Count; - - using var cmd = _conn.CreateCommand(); - - var innerSql = @" - SELECT 1 - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE 1=1"; - - if (validQueries != null && validQueries.Count == 1) - { - var allowLeafFallback = !SqlNameResolver.HasQualifier(validQueries[0]); - var qualifiedSymbolClause = SqlNameResolver.HasQualifier(validQueries[0]) - ? BuildQualifiedSymbolMatchSql("query0", _foldReady) - : null; - var csharpExplicitInterfaceClause = allowLeafFallback - ? BuildCSharpExplicitInterfaceShortAliasMatchSql("query0") - : BuildCSharpExplicitInterfaceIdentityMatchSql("query0"); - var rustQualifiedExact = ShouldPreserveRustQualifiedExactQuery(validQueries[0], lang, exact); - var rustQualifiedParts = rustQualifiedExact ? NormalizeRustQualifiedExactQueryParts(validQueries[0]) : default; - innerSql += exact - ? rustQualifiedParts.QualifiedPath != null - ? _foldReady - ? " AND ((s.container_qualified_name = @query0RustContainer COLLATE NOCASE OR s.container_name = @query0RustContainer COLLATE NOCASE) AND s.name_folded = @query0RustLeafFolded)" - : " AND ((s.container_qualified_name = @query0RustContainer COLLATE NOCASE OR s.container_name = @query0RustContainer COLLATE NOCASE) AND s.name = @query0RustLeaf COLLATE NOCASE)" - : _foldReady - ? allowLeafFallback - ? $" AND ({BuildExactPrimarySymbolNameMatchSql("@query0", true, validQueries[0], lang)} OR {csharpExplicitInterfaceClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query0SegmentCount AND sql_normalize_name_folded(s.name) = @query0NormalizedFolded) OR sql_leaf_name_folded(s.name) = @query0LeafFolded)))" - : $" AND ({BuildExactPrimarySymbolNameMatchSql("@query0", true, validQueries[0], lang)} OR {csharpExplicitInterfaceClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query0SegmentCount AND sql_normalize_name_folded(s.name) = @query0NormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" - : allowLeafFallback - ? $" AND ({BuildExactPrimarySymbolNameMatchSql("@query0", false, validQueries[0], lang)} OR {csharpExplicitInterfaceClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query0SegmentCount AND sql_normalize_name(s.name) = @query0Normalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @query0Leaf COLLATE NOCASE)))" - : $" AND ({BuildExactPrimarySymbolNameMatchSql("@query0", false, validQueries[0], lang)} OR {csharpExplicitInterfaceClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query0SegmentCount AND sql_normalize_name(s.name) = @query0Normalized COLLATE NOCASE){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" - : $" AND (s.name LIKE @query0 ESCAPE '\\' OR (f.lang = 'sql' AND sql_normalize_name(s.name) LIKE @query0NormalizedLike ESCAPE '\\'){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause} OR {csharpExplicitInterfaceClause}" : string.Empty)})"; - } - if (kind != null) - innerSql += " AND s.kind = @kind"; - if (lang != null) - innerSql += SymbolLanguageFileIdFilter; - if (since != null && _fileColumns.Contains("modified")) - innerSql += " AND f.modified >= @since"; - AppendPathFilters(ref innerSql, pathPatterns, excludePathPatterns, excludeTests); - AppendVisibilityFilters(ref innerSql, visibilityFilters, excludeVisibilityFilters); - innerSql += " LIMIT @limit"; - - cmd.CommandText = $"SELECT COUNT(*) FROM ({innerSql})"; - if (validQueries != null && validQueries.Count == 1) - { - var value = validQueries[0]; - var rustQualifiedExact = ShouldPreserveRustQualifiedExactQuery(value, lang, exact); - var rustQualifiedParts = rustQualifiedExact ? NormalizeRustQualifiedExactQueryParts(value) : default; - var paramValue = !exact - ? $"%{EscapeLikeQuery(value)}%" - : _foldReady - ? FoldNameForLanguage(value, lang) - : value; - if (exact && _foldReady) - AddPersistedFoldedNameQueryParameters(cmd, "@query0", value, lang); - else - SqliteCommandPolicy.Add(cmd, "@query0", paramValue); - SqliteCommandPolicy.Add(cmd, "@query0Normalized", SqlNameResolver.NormalizeQualifiedName(value)); - SqliteCommandPolicy.Add(cmd, "@query0NormalizedFolded", NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(value)) ?? SqlNameResolver.NormalizeQualifiedName(value)); - var queryLeaf = GetQualifiedQueryLeaf(value, lang); - SqliteCommandPolicy.Add(cmd, "@query0Leaf", queryLeaf); - SqliteCommandPolicy.Add(cmd, "@query0LeafFolded", NameFold.Fold(queryLeaf) ?? queryLeaf); - SqliteCommandPolicy.Add(cmd, "@query0SegmentCount", SqlNameResolver.GetSegmentCount(value)); - SqliteCommandPolicy.Add(cmd, "@query0NormalizedLike", $"%{EscapeLikeQuery(SqlNameResolver.NormalizeQualifiedName(value))}%"); - AddCSharpExplicitInterfaceIdentityQueryParameter(cmd, "query0", value); - if (SqlNameResolver.HasQualifier(value)) - AddQualifiedSymbolQueryParameters(cmd, "query0", value); - if (rustQualifiedParts.QualifiedPath != null) - { - SqliteCommandPolicy.Add(cmd, "@query0RustContainer", rustQualifiedParts.ContainerPath ?? string.Empty); - SqliteCommandPolicy.Add(cmd, "@query0RustLeaf", rustQualifiedParts.LeafName ?? string.Empty); - SqliteCommandPolicy.Add(cmd, "@query0RustLeafFolded", NameFold.Fold(rustQualifiedParts.LeafName ?? string.Empty) ?? rustQualifiedParts.LeafName ?? string.Empty); - } - } - if (kind != null) - SqliteCommandPolicy.Add(cmd, "@kind", kind); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - if (since != null && _fileColumns.Contains("modified")) - SqliteCommandPolicy.Add(cmd, "@since", since.Value); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - AddVisibilityFilterParameters(cmd, visibilityFilters, excludeVisibilityFilters); - SqliteCommandPolicy.Add(cmd, "@limit", limit); - - var raw = cmd.ExecuteScalar(); - return raw is long l ? (int)l : Convert.ToInt32(raw); - } - - public QueryCountResult CountSearchSymbolsTotal(string? query = null, string? kind = null, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, DateTime? since = null, bool exact = false, IReadOnlyList? visibilityFilters = null, IReadOnlyList? excludeVisibilityFilters = null, bool groupPartials = false) - { - return CountSearchSymbolsTotal(query == null ? null : new[] { NormalizeSymbolSearchQueryForSymbolSearch(query, lang, exact) ?? query ?? string.Empty }, kind, lang, pathPatterns, excludePathPatterns, excludeTests, since, exact, visibilityFilters, excludeVisibilityFilters, groupPartials); - } - - public QueryCountResult CountSearchSymbolsTotal(IReadOnlyList? queries, string? kind = null, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, DateTime? since = null, bool exact = false, IReadOnlyList? visibilityFilters = null, IReadOnlyList? excludeVisibilityFilters = null, bool groupPartials = false) - { - lang = DbReader.NormalizeQueryLanguage(lang); - var effectiveQueries = NormalizeSymbolSearchQueries(queries, lang, exact); - if (groupPartials) - EnsureCSharpCallableTypeKinds(lang, effectiveQueries, exact, kind); - using var cmd = _conn.CreateCommand(); - - var logicalPartialKeySql = LogicalPartialSymbolGrouper.BuildSqlKeyExpression( - "f.lang", - "s.kind", - "s.name", - "s.id", - "f.path", - GetSymbolColumnSql("signature"), - GetSymbolColumnSql("container_name"), - GetSymbolColumnSql("container_qualified_name"), - GetSymbolColumnSql("family_key"), - GetSymbolColumnSql("return_type"), - GetSymbolColumnSql("is_partial_declaration"), - _hotspotFamilyReadyLanguages.Contains("csharp")); - var countSql = groupPartials - ? $"COUNT(DISTINCT ({logicalPartialKeySql}))" - : "COUNT(*)"; - var sql = $@" - SELECT {countSql}, COUNT(DISTINCT f.path) - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE 1=1"; - - if (effectiveQueries != null && effectiveQueries.Count > 0) - { - var orClauses = exact - ? string.Join(" OR ", effectiveQueries.Select((queryValue, idx) => - { - var rustQualifiedExact = ShouldPreserveRustQualifiedExactQuery(queryValue, lang, exact); - var rustQualifiedParts = rustQualifiedExact ? NormalizeRustQualifiedExactQueryParts(queryValue) : default; - var allowLeafFallback = !SqlNameResolver.HasQualifier(queryValue); - var markdownAnchorClause = _symbolColumns.Contains("name_folded") - ? $"(f.lang = 'markdown' AND ((s.kind = 'heading' AND s.name_folded = @query{idx}MarkdownHeading) OR (s.kind = 'anchor' AND s.name_folded = @query{idx}MarkdownExplicitAnchor COLLATE BINARY)))" - : "0"; - var qualifiedSymbolClause = SqlNameResolver.HasQualifier(queryValue) - ? BuildQualifiedSymbolMatchSql($"query{idx}", _foldReady) - : null; - var csharpExplicitInterfaceClause = allowLeafFallback - ? BuildCSharpExplicitInterfaceShortAliasMatchSql($"query{idx}") - : BuildCSharpExplicitInterfaceIdentityMatchSql($"query{idx}"); - var swiftBacktickAlias = ComputeSwiftBacktickAlias(queryValue, lang); - var swiftBacktickClause = swiftBacktickAlias != null - ? _foldReady - ? $" OR s.name_folded = @query{idx}SwiftBacktickAlias" - : $" OR s.name = @query{idx}SwiftBacktickAlias COLLATE NOCASE" - : string.Empty; - if (rustQualifiedParts.QualifiedPath != null) - return _foldReady - ? $"((s.container_qualified_name = @query{idx}RustContainer COLLATE NOCASE OR s.container_name = @query{idx}RustContainer COLLATE NOCASE) AND s.name_folded = @query{idx}RustLeafFolded)" - : $"((s.container_qualified_name = @query{idx}RustContainer COLLATE NOCASE OR s.container_name = @query{idx}RustContainer COLLATE NOCASE) AND s.name = @query{idx}RustLeaf COLLATE NOCASE)"; - return _foldReady - ? allowLeafFallback - ? $"({BuildExactPrimarySymbolNameMatchSql($"@query{idx}", true, queryValue, lang)}{swiftBacktickClause} OR {csharpExplicitInterfaceClause} OR {markdownAnchorClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded) OR sql_leaf_name_folded(s.name) = @query{idx}LeafFolded)))" - : $"({BuildExactPrimarySymbolNameMatchSql($"@query{idx}", true, queryValue, lang)}{swiftBacktickClause} OR {csharpExplicitInterfaceClause} OR {markdownAnchorClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" - : allowLeafFallback - ? $"({BuildExactPrimarySymbolNameMatchSql($"@query{idx}", false, queryValue, lang)}{swiftBacktickClause} OR {csharpExplicitInterfaceClause} OR {markdownAnchorClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name(s.name) = @query{idx}Normalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @query{idx}Leaf COLLATE NOCASE)))" - : $"({BuildExactPrimarySymbolNameMatchSql($"@query{idx}", false, queryValue, lang)}{swiftBacktickClause} OR {csharpExplicitInterfaceClause} OR {markdownAnchorClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name(s.name) = @query{idx}Normalized COLLATE NOCASE){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})"; - })) - : string.Join(" OR ", effectiveQueries.Select((queryValue, idx) => - { - var qualifiedSymbolClause = SqlNameResolver.HasQualifier(queryValue) - ? BuildQualifiedSymbolMatchSql($"query{idx}", _foldReady) - : null; - var csharpExplicitInterfaceClause = SqlNameResolver.HasQualifier(queryValue) - ? BuildCSharpExplicitInterfaceIdentityMatchSql($"query{idx}") - : null; - var markdownAnchorLikeClause = _symbolColumns.Contains("name_folded") - ? $" OR (f.lang = 'markdown' AND ((s.kind = 'heading' AND s.name_folded LIKE @query{idx}MarkdownHeadingLike ESCAPE '\\') OR (s.kind = 'anchor' AND instr(s.name_folded, @query{idx}MarkdownExplicitAnchor) > 0)))" - : string.Empty; - return $"(s.name LIKE @query{idx} ESCAPE '\\'{markdownAnchorLikeClause} OR (f.lang = 'sql' AND sql_normalize_name(s.name) LIKE @query{idx}NormalizedLike ESCAPE '\\'){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause} OR {csharpExplicitInterfaceClause}" : string.Empty)})"; - })); - sql += $" AND ({orClauses})"; - } - if (kind != null) - sql += " AND s.kind = @kind"; - if (lang != null) - sql += SymbolLanguageFileIdFilter; - if (since != null && _fileColumns.Contains("modified")) - sql += " AND f.modified >= @since"; - AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); - AppendVisibilityFilters(ref sql, visibilityFilters, excludeVisibilityFilters); - - cmd.CommandText = sql; - if (effectiveQueries != null) - { - for (int i = 0; i < effectiveQueries.Count; i++) - { - var value = effectiveQueries[i]; - var paramValue = !exact - ? $"%{EscapeLikeQuery(value)}%" - : _foldReady - ? FoldNameForLanguage(value, lang) - : value; - if (exact && _foldReady) - AddPersistedFoldedNameQueryParameters(cmd, $"@query{i}", value, lang); - else - SqliteCommandPolicy.Add(cmd, $"@query{i}", paramValue); - SqliteCommandPolicy.Add(cmd, $"@query{i}Normalized", SqlNameResolver.NormalizeQualifiedName(value)); - SqliteCommandPolicy.Add(cmd, $"@query{i}NormalizedFolded", NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(value)) ?? SqlNameResolver.NormalizeQualifiedName(value)); - var queryLeaf = GetQualifiedQueryLeaf(value, lang); - SqliteCommandPolicy.Add(cmd, $"@query{i}Leaf", queryLeaf); - SqliteCommandPolicy.Add(cmd, $"@query{i}LeafFolded", NameFold.Fold(queryLeaf) ?? queryLeaf); - SqliteCommandPolicy.Add(cmd, $"@query{i}SegmentCount", SqlNameResolver.GetSegmentCount(value)); - SqliteCommandPolicy.Add(cmd, $"@query{i}NormalizedLike", $"%{EscapeLikeQuery(SqlNameResolver.NormalizeQualifiedName(value))}%"); - AddCSharpExplicitInterfaceIdentityQueryParameter(cmd, $"query{i}", value); - if (_symbolColumns.Contains("name_folded")) - { - var markdownHeadingIdentity = MarkdownAnchorIdentity.NormalizeHeadingFragment(value); - SqliteCommandPolicy.Add(cmd, $"@query{i}MarkdownHeading", markdownHeadingIdentity); - SqliteCommandPolicy.Add(cmd, $"@query{i}MarkdownHeadingLike", $"%{EscapeLikeQuery(markdownHeadingIdentity)}%"); - SqliteCommandPolicy.Add(cmd, $"@query{i}MarkdownExplicitAnchor", MarkdownAnchorIdentity.NormalizeExplicitAnchorDefinition(value)); - } - if (SqlNameResolver.HasQualifier(value)) - AddQualifiedSymbolQueryParameters(cmd, $"query{i}", value); - var rustQualifiedExact = ShouldPreserveRustQualifiedExactQuery(value, lang, exact); - var rustQualifiedParts = rustQualifiedExact - ? NormalizeRustQualifiedExactQueryParts(value) - : default; - if (rustQualifiedParts.QualifiedPath != null) - { - SqliteCommandPolicy.Add(cmd, $"@query{i}RustContainer", rustQualifiedParts.ContainerPath ?? string.Empty); - SqliteCommandPolicy.Add(cmd, $"@query{i}RustLeaf", rustQualifiedParts.LeafName ?? string.Empty); - SqliteCommandPolicy.Add(cmd, $"@query{i}RustLeafFolded", NameFold.Fold(rustQualifiedParts.LeafName ?? string.Empty) ?? rustQualifiedParts.LeafName ?? string.Empty); - } - var swiftBacktickAlias = ComputeSwiftBacktickAlias(value, lang); - if (swiftBacktickAlias != null) - { - SqliteCommandPolicy.Add(cmd, $"@query{i}SwiftBacktickAlias", _foldReady - ? NameFold.Fold(swiftBacktickAlias) ?? swiftBacktickAlias - : swiftBacktickAlias); - } - } - } - if (kind != null) - SqliteCommandPolicy.Add(cmd, "@kind", kind); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - if (since != null && _fileColumns.Contains("modified")) - SqliteCommandPolicy.Add(cmd, "@since", since.Value); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - AddVisibilityFilterParameters(cmd, visibilityFilters, excludeVisibilityFilters); - - using var reader = cmd.ExecuteTrackedReader(); - return reader.TrackedRead() - ? new QueryCountResult(reader.GetInt32(0), reader.GetInt32(1)) - : new QueryCountResult(0, 0); - } - - /// - /// Search symbols by one or more name patterns (OR-joined). Empty/null list returns all symbols matching other filters. - /// When is true, names are matched case-insensitively for equality instead of substring. - /// 複数名前パターン（OR結合）でシンボルを検索。空/null なら他フィルタに一致する全シンボルを返す。 - /// が true の場合、部分一致ではなく大文字小文字を無視した完全一致になる。 - /// - public List SearchSymbols(IReadOnlyList? queries, int limit = 20, string? kind = null, string? lang = null, IReadOnlyList? pathPatterns = null, IReadOnlyList? excludePathPatterns = null, bool excludeTests = false, DateTime? since = null, bool exact = false, IReadOnlyList? visibilityFilters = null, IReadOnlyList? excludeVisibilityFilters = null, SymbolSortMode sortMode = SymbolSortMode.Name, int? startLine = null, int? endLine = null, bool groupPartials = false, int offset = 0) - { - lang = DbReader.NormalizeQueryLanguage(lang); - // Multi-name queries: run one search per name to guarantee per-name candidate coverage - // (a common/earlier-sorting name cannot starve others out of the candidate pool), then - // round-robin interleave the per-name results under a single global `limit` cap so the - // public `limit` contract stays "Max total results", not per-name. - // 複数名指定: 名前ごとに独立検索して候補プールを確保した上で、round-robin で統合し、 - // 最終的に全体で `limit` 件に収める。`limit` は従来どおり「合計の上限」。 - var validQueries = NormalizeSymbolSearchQueries(queries, lang, exact); - if (validQueries != null && validQueries.Count > 1) - { - var requestedPrefix = checked(limit + Math.Max(0, offset)); - var perName = new List>(validQueries.Count); - foreach (var q in validQueries) - perName.Add(SearchSymbols(new[] { q! }, requestedPrefix, kind, lang, pathPatterns, excludePathPatterns, excludeTests, since, exact, visibilityFilters, excludeVisibilityFilters, sortMode, startLine, endLine, groupPartials)); - - // De-duplicate the same physical symbol returned by multiple query branches while - // preserving distinct indexed rows that happen to share a path/line/name/kind tuple. - // 複数 query branch が返した同一 physical symbol は除外する一方、path/line/name/kind が - // 偶然一致する別の indexed row は保持する。 - var seen = new HashSet(); - var merged = new List(); - var cursors = new int[perName.Count]; - bool advanced; - do - { - advanced = false; - for (int i = 0; i < perName.Count && merged.Count < requestedPrefix; i++) - { - while (cursors[i] < perName[i].Count) - { - var r = perName[i][cursors[i]++]; - if (seen.Add(r.SymbolId)) - { - merged.Add(r); - advanced = true; - break; - } - } - } - } while (advanced && merged.Count < requestedPrefix); - return merged.Skip(Math.Max(0, offset)).Take(limit).ToList(); - } - - if (groupPartials) - EnsureCSharpCallableTypeKinds(lang, validQueries, exact, kind); - using var cmd = _conn.CreateCommand(); - - var startLineSql = GetSymbolColumnSql("start_line", "s.line"); - var endLineSql = GetSymbolColumnSql("end_line", "s.line"); - var bodyStartLineSql = GetSymbolColumnSql("body_start_line"); - var bodyEndLineSql = GetSymbolColumnSql("body_end_line"); - var signatureSql = GetSymbolColumnSql("signature"); - var containerKindSql = GetSymbolColumnSql("container_kind"); - var containerNameSql = GetSymbolColumnSql("container_name"); - var containerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name"); - var familyKeySql = GetSymbolColumnSql("family_key"); - var visibilitySql = GetSymbolColumnSql("visibility"); - var returnTypeSql = GetSymbolColumnSql("return_type"); - var startColumnSql = GetSymbolColumnSql("start_column", "CAST(2147483647 AS INTEGER)"); - var sizeLinesSql = $"CASE WHEN ({endLineSql}) >= ({startLineSql}) THEN ({endLineSql}) - ({startLineSql}) + 1 ELSE 1 END"; - var includeRankSignals = sortMode != SymbolSortMode.Name && _hasReferencesTable; - // Keep aggregate grouping aligned with the NOCASE joins below. Binary grouping would emit - // one aggregate row per case variant and multiply the same physical s.id before LIMIT/OFFSET (#4753). - // 集計側と下記 NOCASE JOIN の照合順序を揃え、大小文字 variant による同一 s.id の増殖を防ぐ。 - var symbolRankJoin = includeRankSignals - ? $@" - LEFT JOIN ( - SELECT rf.lang AS lang, - sr.symbol_name AS symbol_name, - COUNT(*) AS reference_count, - SUM({GetHotspotReferenceWeightSql("sr.reference_kind")}) AS hotspot_score - FROM symbol_references sr - JOIN files rf ON rf.id = sr.file_id - WHERE sr.reference_kind IN {CallGraphReferenceKindsSql} - AND sr.symbol_name IS NOT NULL - AND sr.symbol_name <> '' - GROUP BY rf.lang, sr.symbol_name COLLATE NOCASE - ) symbol_rank - ON symbol_rank.lang = f.lang - AND symbol_rank.symbol_name = s.name COLLATE NOCASE - LEFT JOIN ( - SELECT sr.file_id AS file_id, - sr.symbol_name AS symbol_name, - COUNT(*) AS reference_count, - SUM({GetHotspotReferenceWeightSql("sr.reference_kind")}) AS hotspot_score - FROM symbol_references sr - WHERE sr.reference_kind IN {CallGraphReferenceKindsSql} - AND sr.symbol_name IS NOT NULL - AND sr.symbol_name <> '' - GROUP BY sr.file_id, sr.symbol_name COLLATE NOCASE - ) symbol_file_rank - ON symbol_file_rank.file_id = s.file_id - AND symbol_file_rank.symbol_name = s.name COLLATE NOCASE - LEFT JOIN ( - SELECT df.lang AS lang, - ds.name AS symbol_name, - COUNT(*) AS definition_sites - FROM symbols ds - JOIN files df ON df.id = ds.file_id - WHERE ds.name IS NOT NULL - AND ds.name <> '' - GROUP BY df.lang, ds.name COLLATE NOCASE - ) symbol_defs - ON symbol_defs.lang = f.lang - AND symbol_defs.symbol_name = s.name COLLATE NOCASE" - : string.Empty; - var genericNamePenaltySql = includeRankSignals ? GetGenericSymbolRankNamePenaltySql("s.name") : "1.0"; - var definitionSitesSql = includeRankSignals ? "COALESCE(symbol_defs.definition_sites, 1)" : "CAST(1 AS INTEGER)"; - var csharpConservativeRankSignalSql = includeRankSignals - ? $"(f.lang = 'csharp' AND (s.kind = 'property' OR ({definitionSitesSql}) > 1 OR lower(s.name) IN {GenericSymbolRankNamesSql}))" - : "0"; - var referenceCountSql = includeRankSignals - ? $"CASE WHEN {csharpConservativeRankSignalSql} THEN COALESCE(symbol_file_rank.reference_count, 0) ELSE COALESCE(symbol_rank.reference_count, 0) END" - : "CAST(0 AS INTEGER)"; - var hotspotScoreSql = includeRankSignals - ? $"CASE WHEN {csharpConservativeRankSignalSql} THEN COALESCE(symbol_file_rank.hotspot_score, 0.0) ELSE COALESCE(symbol_rank.hotspot_score, 0.0) END" - : "CAST(0.0 AS REAL)"; - var definitionDilutionSql = $"CASE WHEN ({definitionSitesSql}) > 1 THEN CAST(({definitionSitesSql}) * ({definitionSitesSql}) AS REAL) ELSE 1.0 END"; - var structuralRankPenaltySql = includeRankSignals ? $"CASE WHEN s.kind IN ('property', 'enum') AND ({sizeLinesSql}) <= 1 THEN 0.1 ELSE 1.0 END" : "1.0"; - var rankingReferenceScoreSql = includeRankSignals ? $"(({referenceCountSql}) * ({genericNamePenaltySql}) * ({structuralRankPenaltySql}) / ({definitionDilutionSql}))" : referenceCountSql; - var rankingHotspotScoreSql = includeRankSignals ? $"(({hotspotScoreSql}) * ({genericNamePenaltySql}) * ({structuralRankPenaltySql}) / ({definitionDilutionSql}))" : hotspotScoreSql; - var cappedRankingReferenceScoreSql = $"CASE WHEN ({rankingReferenceScoreSql}) > 100.0 THEN 100.0 ELSE ({rankingReferenceScoreSql}) END"; - var cappedRankingHotspotScoreSql = $"CASE WHEN ({rankingHotspotScoreSql}) > 150.0 THEN 150.0 ELSE ({rankingHotspotScoreSql}) END"; - var complexityScoreSql = $@"(({sizeLinesSql} * 16.0) + ({cappedRankingReferenceScoreSql} * 0.75) + ({cappedRankingHotspotScoreSql} * 0.35) + CASE - WHEN {visibilitySql} IN ('public', 'pub', 'open', 'export') THEN 8.0 - WHEN {visibilitySql} IN ('protected', 'internal', 'protected internal') THEN 4.0 - ELSE 0.0 - END)"; - var logicalPartialKeySql = LogicalPartialSymbolGrouper.BuildSqlKeyExpression( - "f.lang", - "s.kind", - "s.name", - "s.id", - "f.path", - signatureSql, - containerNameSql, - containerQualifiedNameSql, - familyKeySql, - returnTypeSql, - GetSymbolColumnSql("is_partial_declaration"), - _hotspotFamilyReadyLanguages.Contains("csharp")); - var generatedSql = _fileColumns.Contains("generated") - ? "CASE WHEN COALESCE(f.generated, 0) <> 0 OR codeindex_generated_file_name(f.path) THEN 1 ELSE 0 END" - : "CASE WHEN codeindex_generated_file_name(f.path) THEN 1 ELSE 0 END"; - var canonicalPrimaryRankSql = LogicalPartialSymbolGrouper.BuildSqlPrimaryRankExpression( - "s.kind", - bodyStartLineSql, - bodyEndLineSql); - var canonicalSemanticScoreSql = LogicalPartialSymbolGrouper.BuildSqlSemanticScoreExpression( - signatureSql, - "s.kind", - GetSymbolColumnSql("declaration_semantic_score")); - var fallbackCanonicalDeclarationIdentitySql = BuildCanonicalDeclarationIdentitySql(signatureSql); - var canonicalDeclarationIdentitySql = $"CASE WHEN s.kind IN ('function', 'test.method') THEN COALESCE(csharp_partial_callable_identity({signatureSql}, s.name, {returnTypeSql}), {fallbackCanonicalDeclarationIdentitySql}) ELSE {fallbackCanonicalDeclarationIdentitySql} END"; - var exactNameOrderSql = "CASE " + - "WHEN @preferLiteralExactMatch = 1 AND s.name = @rawQuery THEN 0 " + - "WHEN @preferLiteralNormalizedSqlMatch = 1 AND f.lang = 'sql' AND sql_segment_count(s.name) = @rawQuerySegmentCount AND sql_normalize_name(s.name) = @rawQueryNormalized THEN 1 " + - "WHEN @preferCaseInsensitiveExactMatch = 1 AND s.name = @rawQuery COLLATE NOCASE THEN 2 " + - "WHEN @preferCaseInsensitiveNormalizedSqlMatch = 1 AND f.lang = 'sql' AND sql_segment_count(s.name) = @rawQuerySegmentCount AND sql_normalize_name_folded(s.name) = @rawQueryNormalizedFolded THEN 3 " + - "WHEN @preferCaseInsensitiveSqlLeafMatch = 1 AND f.lang = 'sql' AND sql_leaf_name_folded(s.name) = @rawQueryLeafFolded THEN 4 " + - "ELSE 5 END"; - - var sql = $@" - SELECT f.path, f.lang, s.kind, {GetSymbolColumnSql("sub_kind")} AS sub_kind, s.name, s.line, - {startLineSql} AS start_line, - {GetSymbolColumnSql("start_column")} AS start_column, - {endLineSql} AS end_line, - {bodyStartLineSql} AS body_start_line, - {bodyEndLineSql} AS body_end_line, - {signatureSql} AS signature, - {containerKindSql} AS container_kind, - {containerNameSql} AS container_name, - {visibilitySql} AS visibility, - {returnTypeSql} AS return_type, - {referenceCountSql} AS reference_count, - {hotspotScoreSql} AS hotspot_score, - {rankingReferenceScoreSql} AS ranking_reference_score, - {rankingHotspotScoreSql} AS ranking_hotspot_score, - {genericNamePenaltySql} AS generic_name_penalty, - {structuralRankPenaltySql} AS structural_rank_penalty, - {definitionSitesSql} AS definition_sites, - {sizeLinesSql} AS size_lines, - {complexityScoreSql} AS complexity_score, - {containerQualifiedNameSql} AS container_qualified_name, - {logicalPartialKeySql} AS logical_partial_key, - s.id AS symbol_id, - {exactNameOrderSql} AS exact_name_order, - {PathBucketOrder} AS path_bucket, - {VisibilityOrder} AS visibility_rank, - {startColumnSql} AS stable_start_column, - {canonicalPrimaryRankSql} AS canonical_primary_rank, - {generatedSql} AS canonical_generated_rank, - {canonicalSemanticScoreSql} AS canonical_semantic_score, - {canonicalDeclarationIdentitySql} AS canonical_declaration_identity, - {GetSymbolColumnSql("identifier_start_column")} AS identifier_start_column - FROM symbols s - JOIN files f ON s.file_id = f.id - {symbolRankJoin} - WHERE 1=1"; - - var effectiveQueries = validQueries; - if (effectiveQueries != null && effectiveQueries.Count > 0) - { - // --exact: Unicode-aware equality when FoldReady (#86), else ASCII COLLATE NOCASE. - // Fold path: `s.name_folded = @qFolded` (indexed by idx_symbols_name_folded), query - // value is pre-folded in .NET with NameFold.Fold so Ä vs ä / 全角 vs 半角 match. - // Fallback: `s.name = @q COLLATE NOCASE` (indexed by idx_symbols_name_nocase). Both - // paths stay SARGable. Using `lower(col)` would force a full scan per name. - // --exact: FoldReady なら Unicode 折り畳み経路、未 ready ならレガシー NOCASE 経路へ fallback。 - var orClauses = exact - ? string.Join(" OR ", effectiveQueries.Select((queryValue, idx) => - { - var rustQualifiedExact = ShouldPreserveRustQualifiedExactQuery(queryValue, lang, exact); - var rustQualifiedParts = rustQualifiedExact ? NormalizeRustQualifiedExactQueryParts(queryValue) : default; - var allowLeafFallback = !SqlNameResolver.HasQualifier(queryValue); - var markdownAnchorClause = _symbolColumns.Contains("name_folded") - ? $"(f.lang = 'markdown' AND ((s.kind = 'heading' AND s.name_folded = @query{idx}MarkdownHeading) OR (s.kind = 'anchor' AND s.name_folded = @query{idx}MarkdownExplicitAnchor COLLATE BINARY)))" - : "0"; - var qualifiedSymbolClause = SqlNameResolver.HasQualifier(queryValue) - ? BuildQualifiedSymbolMatchSql($"query{idx}", _foldReady) - : null; - var csharpExplicitInterfaceClause = allowLeafFallback - ? BuildCSharpExplicitInterfaceShortAliasMatchSql($"query{idx}") - : BuildCSharpExplicitInterfaceIdentityMatchSql($"query{idx}"); - var swiftBacktickAlias = ComputeSwiftBacktickAlias(queryValue, lang); - var swiftBacktickClause = swiftBacktickAlias != null - ? _foldReady - ? $" OR s.name_folded = @query{idx}SwiftBacktickAlias" - : $" OR s.name = @query{idx}SwiftBacktickAlias COLLATE NOCASE" - : string.Empty; - if (rustQualifiedParts.QualifiedPath != null) - return _foldReady - ? $"((s.container_qualified_name = @query{idx}RustContainer COLLATE NOCASE OR s.container_name = @query{idx}RustContainer COLLATE NOCASE) AND s.name_folded = @query{idx}RustLeafFolded)" - : $"((s.container_qualified_name = @query{idx}RustContainer COLLATE NOCASE OR s.container_name = @query{idx}RustContainer COLLATE NOCASE) AND s.name = @query{idx}RustLeaf COLLATE NOCASE)"; - return _foldReady - ? allowLeafFallback - ? $"({BuildExactPrimarySymbolNameMatchSql($"@query{idx}", true, queryValue, lang)}{swiftBacktickClause} OR {csharpExplicitInterfaceClause} OR {markdownAnchorClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded) OR sql_leaf_name_folded(s.name) = @query{idx}LeafFolded)))" - : $"({BuildExactPrimarySymbolNameMatchSql($"@query{idx}", true, queryValue, lang)}{swiftBacktickClause} OR {csharpExplicitInterfaceClause} OR {markdownAnchorClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name_folded(s.name) = @query{idx}NormalizedFolded){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})" - : allowLeafFallback - ? $"({BuildExactPrimarySymbolNameMatchSql($"@query{idx}", false, queryValue, lang)}{swiftBacktickClause} OR {csharpExplicitInterfaceClause} OR {markdownAnchorClause} OR (f.lang = 'sql' AND ((sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name(s.name) = @query{idx}Normalized COLLATE NOCASE) OR sql_leaf_name(s.name) = @query{idx}Leaf COLLATE NOCASE)))" - : $"({BuildExactPrimarySymbolNameMatchSql($"@query{idx}", false, queryValue, lang)}{swiftBacktickClause} OR {csharpExplicitInterfaceClause} OR {markdownAnchorClause} OR (f.lang = 'sql' AND sql_segment_count(s.name) = @query{idx}SegmentCount AND sql_normalize_name(s.name) = @query{idx}Normalized COLLATE NOCASE){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause}" : string.Empty)})"; - })) - : string.Join(" OR ", effectiveQueries.Select((queryValue, idx) => - { - var qualifiedSymbolClause = SqlNameResolver.HasQualifier(queryValue) - ? BuildQualifiedSymbolMatchSql($"query{idx}", _foldReady) - : null; - var csharpExplicitInterfaceClause = SqlNameResolver.HasQualifier(queryValue) - ? BuildCSharpExplicitInterfaceIdentityMatchSql($"query{idx}") - : null; - var markdownAnchorLikeClause = _symbolColumns.Contains("name_folded") - ? $" OR (f.lang = 'markdown' AND ((s.kind = 'heading' AND s.name_folded LIKE @query{idx}MarkdownHeadingLike ESCAPE '\\') OR (s.kind = 'anchor' AND instr(s.name_folded, @query{idx}MarkdownExplicitAnchor) > 0)))" - : string.Empty; - return $"(s.name LIKE @query{idx} ESCAPE '\\'{markdownAnchorLikeClause} OR (f.lang = 'sql' AND sql_normalize_name(s.name) LIKE @query{idx}NormalizedLike ESCAPE '\\'){(qualifiedSymbolClause != null ? $" OR {qualifiedSymbolClause} OR {csharpExplicitInterfaceClause}" : string.Empty)})"; - })); - sql += $" AND ({orClauses})"; - } - if (kind != null) - sql += " AND s.kind = @kind"; - if (lang != null) - sql += SymbolLanguageFileIdFilter; - if (since != null && _fileColumns.Contains("modified")) - sql += " AND f.modified >= @since"; - if (startLine != null) - sql += " AND s.line >= @startLine"; - if (endLine != null) - sql += " AND s.line <= @endLine"; - AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); - AppendVisibilityFilters(ref sql, visibilityFilters, excludeVisibilityFilters); - if (groupPartials) - sql = BuildLogicalPartialSymbolQuery(sql, sortMode); - else - sql += BuildSymbolSortOrderBy(sortMode, exactNameOrderSql, referenceCountSql, hotspotScoreSql, rankingReferenceScoreSql, rankingHotspotScoreSql, sizeLinesSql, complexityScoreSql, startColumnSql); - sql += " LIMIT @limit OFFSET @offset"; - - cmd.CommandText = sql; - if (effectiveQueries != null) - { - for (int idx = 0; idx < effectiveQueries.Count; idx++) - { - string paramValue; - var rustQualifiedExact = ShouldPreserveRustQualifiedExactQuery(effectiveQueries[idx], lang, exact); - var rustQualifiedParts = rustQualifiedExact ? NormalizeRustQualifiedExactQueryParts(effectiveQueries[idx]) : default; - if (!exact) - paramValue = $"%{EscapeLikeQuery(effectiveQueries[idx])}%"; - else if (_foldReady) - paramValue = FoldNameForLanguage(effectiveQueries[idx], lang); - else - paramValue = effectiveQueries[idx]; - if (exact && _foldReady) - { - AddPersistedFoldedNameQueryParameters( - cmd, - $"@query{idx}", - effectiveQueries[idx], - lang); - } - else - { - SqliteCommandPolicy.Add(cmd, $"@query{idx}", paramValue); - } - SqliteCommandPolicy.Add(cmd, $"@query{idx}Normalized", SqlNameResolver.NormalizeQualifiedName(effectiveQueries[idx])); - SqliteCommandPolicy.Add(cmd, $"@query{idx}NormalizedFolded", NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(effectiveQueries[idx])) ?? SqlNameResolver.NormalizeQualifiedName(effectiveQueries[idx])); - var queryLeaf = GetQualifiedQueryLeaf(effectiveQueries[idx], lang); - SqliteCommandPolicy.Add(cmd, $"@query{idx}Leaf", queryLeaf); - SqliteCommandPolicy.Add(cmd, $"@query{idx}LeafFolded", NameFold.Fold(queryLeaf) ?? queryLeaf); - SqliteCommandPolicy.Add(cmd, $"@query{idx}SegmentCount", SqlNameResolver.GetSegmentCount(effectiveQueries[idx])); - SqliteCommandPolicy.Add(cmd, $"@query{idx}NormalizedLike", $"%{EscapeLikeQuery(SqlNameResolver.NormalizeQualifiedName(effectiveQueries[idx]))}%"); - AddCSharpExplicitInterfaceIdentityQueryParameter(cmd, $"query{idx}", effectiveQueries[idx]); - if (_symbolColumns.Contains("name_folded")) - { - var markdownHeadingIdentity = MarkdownAnchorIdentity.NormalizeHeadingFragment(effectiveQueries[idx]); - SqliteCommandPolicy.Add(cmd, $"@query{idx}MarkdownHeading", markdownHeadingIdentity); - SqliteCommandPolicy.Add(cmd, $"@query{idx}MarkdownHeadingLike", $"%{EscapeLikeQuery(markdownHeadingIdentity)}%"); - SqliteCommandPolicy.Add(cmd, $"@query{idx}MarkdownExplicitAnchor", MarkdownAnchorIdentity.NormalizeExplicitAnchorDefinition(effectiveQueries[idx])); - } - if (SqlNameResolver.HasQualifier(effectiveQueries[idx])) - AddQualifiedSymbolQueryParameters(cmd, $"query{idx}", effectiveQueries[idx]); - if (rustQualifiedParts.QualifiedPath != null) - { - SqliteCommandPolicy.Add(cmd, $"@query{idx}RustContainer", rustQualifiedParts.ContainerPath ?? string.Empty); - SqliteCommandPolicy.Add(cmd, $"@query{idx}RustLeaf", rustQualifiedParts.LeafName ?? string.Empty); - SqliteCommandPolicy.Add(cmd, $"@query{idx}RustLeafFolded", NameFold.Fold(rustQualifiedParts.LeafName ?? string.Empty) ?? rustQualifiedParts.LeafName ?? string.Empty); - } - var swiftBacktickAlias = ComputeSwiftBacktickAlias(effectiveQueries[idx], lang); - if (swiftBacktickAlias != null) - { - SqliteCommandPolicy.Add(cmd, $"@query{idx}SwiftBacktickAlias", _foldReady - ? NameFold.Fold(swiftBacktickAlias) ?? swiftBacktickAlias - : swiftBacktickAlias); - } - } - } - var preferLiteralExactMatch = effectiveQueries != null && effectiveQueries.Count == 1; - var preferCaseInsensitiveExactMatch = effectiveQueries != null && effectiveQueries.Count == 1; - var preferSqlLeafMatch = preferCaseInsensitiveExactMatch && !SqlNameResolver.HasQualifier(effectiveQueries![0]); - SqliteCommandPolicy.Add(cmd, "@preferLiteralExactMatch", preferLiteralExactMatch ? 1 : 0); - SqliteCommandPolicy.Add(cmd, "@preferLiteralNormalizedSqlMatch", preferLiteralExactMatch ? 1 : 0); - SqliteCommandPolicy.Add(cmd, "@preferCaseInsensitiveExactMatch", preferCaseInsensitiveExactMatch ? 1 : 0); - SqliteCommandPolicy.Add(cmd, "@preferCaseInsensitiveNormalizedSqlMatch", preferCaseInsensitiveExactMatch ? 1 : 0); - SqliteCommandPolicy.Add(cmd, "@preferCaseInsensitiveSqlLeafMatch", preferSqlLeafMatch ? 1 : 0); - SqliteCommandPolicy.Add(cmd, "@rawQuery", preferLiteralExactMatch ? effectiveQueries![0] : string.Empty); - SqliteCommandPolicy.Add(cmd, "@rawQueryNormalized", preferLiteralExactMatch ? SqlNameResolver.NormalizeQualifiedName(effectiveQueries![0]) : string.Empty); - SqliteCommandPolicy.Add(cmd, "@rawQueryNormalizedFolded", preferLiteralExactMatch ? NameFold.Fold(SqlNameResolver.NormalizeQualifiedName(effectiveQueries![0])) ?? SqlNameResolver.NormalizeQualifiedName(effectiveQueries![0]) : string.Empty); - SqliteCommandPolicy.Add(cmd, "@rawQueryLeaf", preferLiteralExactMatch ? SqlNameResolver.GetLeafName(effectiveQueries![0]) : string.Empty); - SqliteCommandPolicy.Add(cmd, "@rawQueryLeafFolded", preferLiteralExactMatch ? NameFold.Fold(SqlNameResolver.GetLeafName(effectiveQueries![0])) ?? SqlNameResolver.GetLeafName(effectiveQueries![0]) : string.Empty); - SqliteCommandPolicy.Add(cmd, "@rawQuerySegmentCount", preferLiteralExactMatch ? SqlNameResolver.GetSegmentCount(effectiveQueries![0]) : 0); - if (kind != null) - SqliteCommandPolicy.Add(cmd, "@kind", kind); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - if (since != null && _fileColumns.Contains("modified")) - SqliteCommandPolicy.Add(cmd, "@since", since.Value); - if (startLine != null) - SqliteCommandPolicy.Add(cmd, "@startLine", startLine.Value); - if (endLine != null) - SqliteCommandPolicy.Add(cmd, "@endLine", endLine.Value); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - AddVisibilityFilterParameters(cmd, visibilityFilters, excludeVisibilityFilters); - SqliteCommandPolicy.Add(cmd, "@limit", limit); - SqliteCommandPolicy.Add(cmd, "@offset", Math.Max(0, offset)); - - var includeRankingMetadata = sortMode != SymbolSortMode.Name; - var sortModeName = sortMode.ToString().ToLowerInvariant(); - var identifierStartColumnIndex = groupPartials ? 31 : 36; - var results = new List(); - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - var definitionSites = Convert.ToInt32(reader.GetInt64(22)); - var result = new SymbolResult - { - Path = reader.GetString(0), - Lang = GetNullableString(reader, 1), - Kind = reader.GetString(2), - SubKind = GetNullableString(reader, 3), - Name = reader.GetString(4), - Line = reader.GetInt32(5), - StartLine = GetInt32OrFallback(reader, 6, 5), - StartColumn = GetNullableInt32(reader, identifierStartColumnIndex) - ?? ResolveSymbolIdentifierStartColumn( - GetNullableInt32(reader, 7), - GetNullableString(reader, 11), - reader.GetString(4), - reader.GetString(2)), - EndLine = GetInt32OrFallback(reader, 8, 5), - BodyStartLine = GetNullableInt32(reader, 9), - BodyEndLine = GetNullableInt32(reader, 10), - Signature = GetNullableString(reader, 11), - ContainerKind = GetNullableString(reader, 12), - ContainerName = GetNullableString(reader, 13), - ContainerQualifiedName = GetNullableString(reader, 25), - LogicalPartialKey = GetNullableString(reader, 26), - Visibility = GetNullableString(reader, 14), - ReturnType = GetNullableString(reader, 15), - SortMode = includeRankingMetadata ? sortModeName : null, - ReferenceCount = includeRankingMetadata ? Convert.ToInt32(reader.GetInt64(16)) : null, - HotspotScore = includeRankingMetadata ? Math.Round(reader.GetDouble(17), 3) : null, - RankingReferenceScore = includeRankingMetadata ? Math.Round(reader.GetDouble(18), 3) : null, - RankingHotspotScore = includeRankingMetadata ? Math.Round(reader.GetDouble(19), 3) : null, - GenericNamePenalty = includeRankingMetadata ? Math.Round(reader.GetDouble(20), 3) : null, - StructuralRankPenalty = includeRankingMetadata ? Math.Round(reader.GetDouble(21), 3) : null, - DefinitionSites = includeRankingMetadata || (groupPartials && definitionSites > 1) ? definitionSites : null, - SizeLines = includeRankingMetadata ? Convert.ToInt32(reader.GetInt64(23)) : null, - ComplexityScore = includeRankingMetadata ? Math.Round(reader.GetDouble(24), 3) : null, - SymbolId = reader.GetInt64(27), - }; - if (groupPartials && definitionSites > 1) - { - result.PartialFamilyId = LogicalPartialSymbolGrouper.BuildPartialFamilyId(result.LogicalPartialKey!); - result.RepresentativeReason = reader.GetString(28); - result.FamilyMembers = ReadPartialFamilyMembers(reader.GetString(29), result); - result.FamilyMembersTruncated = reader.GetInt64(30) != 0; - } - results.Add(result); - } - return results; - } - - private static List ReadPartialFamilyMembers(string json, SymbolResult representative) - { - using var document = JsonDocument.Parse(json); - var members = new List(); - foreach (var element in document.RootElement.EnumerateArray()) - { - var symbolId = element.GetProperty("symbol_id").GetInt64(); - var path = element.GetProperty("path").GetString() ?? string.Empty; - var startLine = element.GetProperty("start_line").GetInt32(); - var rawStartColumn = element.GetProperty("start_column").ValueKind == JsonValueKind.Null - ? (int?)null - : element.GetProperty("start_column").GetInt32(); - var memberName = element.GetProperty("name").GetString() ?? representative.Name; - var memberSignature = element.GetProperty("signature").ValueKind == JsonValueKind.Null - ? null - : element.GetProperty("signature").GetString(); - var identifierStartColumn = element.TryGetProperty("identifier_start_column", out var identifierColumnElement) - && identifierColumnElement.ValueKind != JsonValueKind.Null - ? identifierColumnElement.GetInt32() - : (int?)null; - members.Add(new PartialFamilyMember - { - SymbolId = symbolId, - Path = path, - Line = element.GetProperty("line").GetInt32(), - StartLine = startLine, - StartColumn = identifierStartColumn - ?? ResolveSymbolIdentifierStartColumn( - rawStartColumn, - memberSignature, - memberName, - representative.Kind), - EndLine = element.GetProperty("end_line").GetInt32(), - Generated = element.GetProperty("generated").GetInt32() != 0, - Representative = representative.SymbolId == symbolId - || (representative.SymbolId == null - && string.Equals(representative.Path, path, StringComparison.Ordinal) - && representative.StartLine == startLine), - }); - } - return members; - } - - private static int? ResolveSymbolIdentifierStartColumn( - int? declarationStartColumn, - string? signature, - string name, - string kind) - { - if (!declarationStartColumn.HasValue || string.IsNullOrWhiteSpace(signature) || string.IsNullOrEmpty(name)) - return declarationStartColumn; - - var firstLineEnd = signature.IndexOfAny(['\r', '\n']); - var firstLine = firstLineEnd >= 0 ? signature[..firstLineEnd] : signature; - var callable = kind is "function" or "test.method"; - var relativeColumn = callable - ? LogicalPartialSymbolGrouper.FindCallableNameOffset(firstLine, name) - : firstLine.IndexOf(name, StringComparison.Ordinal); - if (relativeColumn < 0 && callable) - relativeColumn = firstLine.IndexOf(name, StringComparison.Ordinal); - return relativeColumn >= 0 ? declarationStartColumn.Value + relativeColumn : declarationStartColumn; - } - - private static string GetGenericSymbolRankNamePenaltySql(string nameSql) - => $"CASE WHEN lower({nameSql}) IN {GenericSymbolRankNamesSql} THEN {GenericSymbolRankNamePenaltySqlLiteral} ELSE 1.0 END"; - - private static string BuildCanonicalDeclarationIdentitySql(string signatureSql) - => $"csharp_partial_declaration_identity({signatureSql})"; - - private static string BuildLogicalPartialSymbolQuery(string matchingSymbolsSql, SymbolSortMode sortMode) - { - var orderBy = BuildLogicalPartialSortOrderBy(sortMode); - return $@" - WITH matching_symbols AS ( - {matchingSymbolsSql} - ), - ranked_symbols AS ( - SELECT matching_symbols.*, - ROW_NUMBER() OVER ( - PARTITION BY logical_partial_key - ORDER BY canonical_primary_rank, - canonical_generated_rank, - canonical_semantic_score DESC, - canonical_declaration_identity COLLATE BINARY, - logical_partial_key COLLATE BINARY, - path COLLATE BINARY, - start_line, - stable_start_column, - symbol_id - ) AS logical_row_number, - ROW_NUMBER() OVER ( - PARTITION BY logical_partial_key - ORDER BY path COLLATE BINARY, start_line, stable_start_column, symbol_id - ) AS family_member_row_number, - COUNT(*) OVER (PARTITION BY logical_partial_key) AS logical_definition_sites - FROM matching_symbols - ), - family_ranked_symbols AS ( - SELECT ranked_symbols.*, - MAX(CASE WHEN logical_row_number = 1 THEN family_member_row_number END) OVER ( - PARTITION BY logical_partial_key - ) AS representative_member_row_number - FROM ranked_symbols - ), - logical_symbols AS ( - SELECT family_ranked_symbols.*, - MAX(reference_count) OVER (PARTITION BY logical_partial_key) AS logical_reference_count, - MAX(hotspot_score) OVER (PARTITION BY logical_partial_key) AS logical_hotspot_score, - MAX(ranking_reference_score) OVER (PARTITION BY logical_partial_key) AS logical_ranking_reference_score, - MAX(ranking_hotspot_score) OVER (PARTITION BY logical_partial_key) AS logical_ranking_hotspot_score, - MAX(generic_name_penalty) OVER (PARTITION BY logical_partial_key) AS logical_generic_name_penalty, - MAX(structural_rank_penalty) OVER (PARTITION BY logical_partial_key) AS logical_structural_rank_penalty, - MAX(size_lines) OVER (PARTITION BY logical_partial_key) AS logical_size_lines, - MAX(complexity_score) OVER (PARTITION BY logical_partial_key) AS logical_complexity_score, - MIN(exact_name_order) OVER (PARTITION BY logical_partial_key) AS logical_exact_name_order, - MIN(path_bucket) OVER (PARTITION BY logical_partial_key) AS logical_path_bucket, - MIN(visibility_rank) OVER (PARTITION BY logical_partial_key) AS logical_visibility_rank, - MIN(canonical_primary_rank) OVER (PARTITION BY logical_partial_key) AS logical_primary_rank_min, - MAX(canonical_primary_rank) OVER (PARTITION BY logical_partial_key) AS logical_primary_rank_max, - MIN(canonical_generated_rank) OVER (PARTITION BY logical_partial_key) AS logical_generated_rank_min, - MAX(canonical_generated_rank) OVER (PARTITION BY logical_partial_key) AS logical_generated_rank_max, - MIN(canonical_semantic_score) OVER (PARTITION BY logical_partial_key) AS logical_semantic_score_min, - MAX(canonical_semantic_score) OVER (PARTITION BY logical_partial_key) AS logical_semantic_score_max, - MIN(canonical_declaration_identity) OVER (PARTITION BY logical_partial_key) AS logical_declaration_identity_min, - MAX(canonical_declaration_identity) OVER (PARTITION BY logical_partial_key) AS logical_declaration_identity_max, - json_group_array(json_object( - 'symbol_id', symbol_id, - 'path', path, - 'line', line, - 'start_line', start_line, - 'start_column', start_column, - 'end_line', end_line, - 'name', name, - 'signature', signature, - 'identifier_start_column', identifier_start_column, - 'generated', canonical_generated_rank - )) FILTER (WHERE - family_member_row_number <= CASE - WHEN representative_member_row_number <= {LogicalPartialSymbolGrouper.FamilyMemberLimit} - THEN {LogicalPartialSymbolGrouper.FamilyMemberLimit} - ELSE {LogicalPartialSymbolGrouper.FamilyMemberLimit - 1} - END - OR logical_row_number = 1 - ) OVER ( - PARTITION BY logical_partial_key - ORDER BY path COLLATE BINARY, start_line, stable_start_column, symbol_id - ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING - ) AS logical_family_members_json - FROM family_ranked_symbols - ) - SELECT path, lang, kind, sub_kind, name, line, - start_line, start_column, end_line, - body_start_line, body_end_line, signature, - container_kind, container_name, visibility, return_type, - logical_reference_count, logical_hotspot_score, - logical_ranking_reference_score, logical_ranking_hotspot_score, - logical_generic_name_penalty, logical_structural_rank_penalty, - logical_definition_sites, logical_size_lines, logical_complexity_score, - container_qualified_name, logical_partial_key, symbol_id, - CASE - WHEN logical_primary_rank_min <> logical_primary_rank_max THEN '{LogicalPartialSymbolGrouper.ImplementationBodyReason}' - WHEN logical_generated_rank_min <> logical_generated_rank_max THEN '{LogicalPartialSymbolGrouper.NonGeneratedSourceReason}' - WHEN logical_semantic_score_min <> logical_semantic_score_max THEN '{LogicalPartialSymbolGrouper.SemanticDeclarationReason}' - WHEN logical_declaration_identity_min <> logical_declaration_identity_max THEN '{LogicalPartialSymbolGrouper.CanonicalDeclarationIdentityReason}' - ELSE '{LogicalPartialSymbolGrouper.StableLocationReason}' - END AS representative_reason, - logical_family_members_json, - CASE WHEN logical_definition_sites > {LogicalPartialSymbolGrouper.FamilyMemberLimit} THEN 1 ELSE 0 END AS family_members_truncated, - identifier_start_column - FROM logical_symbols - WHERE logical_row_number = 1 - {orderBy}"; - } - - private static string BuildLogicalPartialSortOrderBy(SymbolSortMode sortMode) - { - const string stableTieBreakers = "logical_path_bucket, logical_visibility_rank, name, path COLLATE BINARY, line, stable_start_column, symbol_id"; - return sortMode switch - { - SymbolSortMode.Hotspot => $"ORDER BY logical_ranking_hotspot_score DESC, logical_ranking_reference_score DESC, logical_hotspot_score DESC, logical_reference_count DESC, logical_size_lines DESC, {stableTieBreakers}", - SymbolSortMode.References => $"ORDER BY logical_ranking_reference_score DESC, logical_ranking_hotspot_score DESC, logical_reference_count DESC, logical_hotspot_score DESC, logical_size_lines DESC, {stableTieBreakers}", - SymbolSortMode.Size => $"ORDER BY logical_size_lines DESC, logical_ranking_reference_score DESC, logical_ranking_hotspot_score DESC, logical_reference_count DESC, {stableTieBreakers}", - SymbolSortMode.Complexity => $"ORDER BY logical_complexity_score DESC, logical_ranking_hotspot_score DESC, logical_ranking_reference_score DESC, logical_reference_count DESC, logical_size_lines DESC, {stableTieBreakers}", - SymbolSortMode.Path => "ORDER BY path COLLATE BINARY, line, stable_start_column, name, symbol_id", - _ => $"ORDER BY logical_exact_name_order, {stableTieBreakers}", - }; - } - - private string BuildSymbolSortOrderBy( - SymbolSortMode sortMode, - string exactNameOrderSql, - string referenceCountSql, - string rawHotspotScoreSql, - string rankingReferenceScoreSql, - string rankingHotspotScoreSql, - string sizeLinesSql, - string complexityScoreSql, - string startColumnSql) - { - var stableTieBreakers = $"{PathBucketOrder}, {VisibilityOrder}, s.name, f.path, s.line, {startColumnSql} ASC, s.id ASC"; - return sortMode switch - { - SymbolSortMode.Hotspot => $" ORDER BY {rankingHotspotScoreSql} DESC, {rankingReferenceScoreSql} DESC, {rawHotspotScoreSql} DESC, {referenceCountSql} DESC, {sizeLinesSql} DESC, {stableTieBreakers}", - SymbolSortMode.References => $" ORDER BY {rankingReferenceScoreSql} DESC, {rankingHotspotScoreSql} DESC, {referenceCountSql} DESC, {rawHotspotScoreSql} DESC, {sizeLinesSql} DESC, {stableTieBreakers}", - SymbolSortMode.Size => $" ORDER BY {sizeLinesSql} DESC, {rankingReferenceScoreSql} DESC, {rankingHotspotScoreSql} DESC, {referenceCountSql} DESC, {stableTieBreakers}", - SymbolSortMode.Complexity => $" ORDER BY {complexityScoreSql} DESC, {rankingHotspotScoreSql} DESC, {rankingReferenceScoreSql} DESC, {referenceCountSql} DESC, {sizeLinesSql} DESC, {stableTieBreakers}", - SymbolSortMode.Path => $" ORDER BY f.path, s.line, {startColumnSql} ASC, s.name, s.id ASC", - _ => $" ORDER BY {exactNameOrderSql}, {stableTieBreakers}", - }; - } - private static string? NormalizeSymbolSearchQuery(string? query, string? lang, bool exact = false) - { - if (!string.IsNullOrWhiteSpace(lang) && string.Equals(lang, "rust", StringComparison.OrdinalIgnoreCase)) - return NormalizeRustSymbolSearchQuery(query, exact); - - if (!string.IsNullOrWhiteSpace(lang) && string.Equals(lang, "javascript", StringComparison.OrdinalIgnoreCase)) - return NormalizeJavaScriptSymbolSearchQuery(query); - - // Terraform dotted prefixes (var.X / local.X / module.X / data.TYPE.X) are stored as bare names in - // the references and symbols tables. Strip the prefix so queries pasted from HCL still resolve. - // Terraform の dotted prefix（var.X / local.X / module.X / data.TYPE.X）は参照/シンボルの bare 名で格納されるため、 - // HCL からそのまま貼り付けたクエリでも解決できるよう prefix を取り除く。 - var terraformNormalized = NormalizeTerraformDottedQuery(query, lang); - if (terraformNormalized != null) - return terraformNormalized; - - return NormalizeCSharpVerbatimQuery(query, lang); - } - - private static readonly Regex TerraformVarLocalModuleQueryRegex = new( - @"^(?:var|local|module)\.(?[A-Za-z_]\w*)(?:\..*)?$", - RegexOptions.Compiled); - - private static readonly Regex TerraformDataQueryRegex = new( - @"^data\.[A-Za-z_]\w*\.(?[A-Za-z_]\w*)(?:\..*)?$", - RegexOptions.Compiled); - - private static string? NormalizeTerraformDottedQuery(string? query, string? lang) - { - if (!string.IsNullOrWhiteSpace(lang) - && !string.Equals(lang, "terraform", StringComparison.OrdinalIgnoreCase)) - { - return null; - } - - if (string.IsNullOrWhiteSpace(query)) - return null; - - var trimmed = query.Trim(); - if (trimmed.Length == 0) - return null; - - var simpleMatch = TerraformVarLocalModuleQueryRegex.Match(trimmed); - if (simpleMatch.Success) - return simpleMatch.Groups["name"].Value; - - var dataMatch = TerraformDataQueryRegex.Match(trimmed); - if (dataMatch.Success) - return dataMatch.Groups["name"].Value; - - return null; - } + => SymbolSearchQueryNormalizer.Normalize(query, lang, exact); private static string? ComputeSwiftBacktickAlias(string? query, string? lang) - { - if (!string.Equals(lang, "swift", StringComparison.OrdinalIgnoreCase) || string.IsNullOrWhiteSpace(query)) - return null; - - var trimmed = query.Trim(); - if (trimmed.Length == 0) - return null; - - if (trimmed.IndexOfAny(['`', ':', '/', '<', '>', '(', ')', '[', ']', ' ']) >= 0) - return null; - - var lastDot = trimmed.LastIndexOf('.'); - if (lastDot < 0) - return $"`{trimmed}`"; - - if (lastDot == 0 || lastDot == trimmed.Length - 1) - return null; - - var prefix = trimmed[..(lastDot + 1)]; - var leaf = trimmed[(lastDot + 1)..]; - if (leaf.IndexOf('.') >= 0) - return null; - - return $"{prefix}`{leaf}`"; - } - - private static string? NormalizeJavaScriptSymbolSearchQuery(string? query) - { - if (query == null) - return null; - - var trimmed = query.Trim(); - if (trimmed.Length == 0) - return null; - - var commonJsPrefixLength = 0; - if (trimmed.StartsWith("module.exports", StringComparison.Ordinal)) - { - var nextIndex = "module.exports".Length; - if (trimmed.Length > nextIndex && trimmed[nextIndex] is '.' or '[') - commonJsPrefixLength = nextIndex; - } - else if (trimmed.StartsWith("exports", StringComparison.Ordinal)) - { - var nextIndex = "exports".Length; - if (trimmed.Length > nextIndex && trimmed[nextIndex] is '.' or '[') - commonJsPrefixLength = nextIndex; - } - - if (commonJsPrefixLength == 0) - return trimmed; - - trimmed = trimmed[commonJsPrefixLength..]; - if (trimmed.Length == 0) - return null; - - trimmed = trimmed.TrimStart(); - if (trimmed.StartsWith(".", StringComparison.Ordinal)) - trimmed = trimmed[1..].TrimStart(); - - var bracketLeaf = NormalizeJavaScriptBracketLeaf(trimmed); - if (bracketLeaf != null) - return bracketLeaf; - - var leafIndex = trimmed.LastIndexOf('.'); - if (leafIndex >= 0) - trimmed = trimmed[(leafIndex + 1)..]; - - bracketLeaf = NormalizeJavaScriptBracketLeaf(trimmed); - if (bracketLeaf != null) - return bracketLeaf; - - return trimmed.Length == 0 ? null : trimmed; - } - - private static string? NormalizeJavaScriptBracketLeaf(string query) - { - var trimmed = query.Trim(); - if (trimmed.Length < 3 || trimmed[0] != '[' || trimmed[^1] != ']') - return null; - - var inner = trimmed[1..^1].Trim(); - if (inner.Length < 2) - return null; - - var quote = inner[0]; - if (quote is not '\'' and not '"') - return null; - - if (inner[^1] != quote) - return null; - - var leaf = inner[1..^1].Trim(); - return leaf.Length == 0 ? null : leaf; - } - - private static string? NormalizeRustSymbolSearchQuery(string? query, bool exact = false) - { - if (query == null) - return null; - - var trimmed = query.Trim(); - if (trimmed.Length == 0) - return null; - - var macroQuery = trimmed; - var isMacroQuery = macroQuery.EndsWith("!", StringComparison.Ordinal); - if (isMacroQuery) - macroQuery = macroQuery[..^1].TrimEnd(); - - if (macroQuery.Length == 0) - return null; - - if (exact && isMacroQuery && macroQuery.Contains("::", StringComparison.Ordinal)) - return NormalizeRustQualifiedMacroQuery(macroQuery); - - var leafIndex = macroQuery.LastIndexOf("::", StringComparison.Ordinal); - if (leafIndex >= 0) - macroQuery = macroQuery[(leafIndex + 2)..].Trim(); - - if (macroQuery.StartsWith("r#", StringComparison.Ordinal)) - macroQuery = macroQuery[2..]; - - return macroQuery.Length == 0 ? null : macroQuery; - } - - private static string? NormalizeRustQualifiedMacroQuery(string query) - { - var segments = query - .Split("::", StringSplitOptions.None) - .Select(segment => segment.Trim()) - .Where(segment => segment.Length > 0) - .Select(segment => segment.StartsWith("r#", StringComparison.Ordinal) ? segment[2..] : segment) - .ToList(); - - return segments.Count == 0 ? null : string.Join("::", segments); - } + => SymbolSearchQueryNormalizer.ComputeSwiftBacktickAlias(query, lang); private static bool ShouldPreserveRustQualifiedExactQuery(string? query, string? lang, bool exact) - { - return exact - && !string.IsNullOrWhiteSpace(lang) - && string.Equals(lang, "rust", StringComparison.OrdinalIgnoreCase) - && !string.IsNullOrWhiteSpace(query) - && query.Contains("::", StringComparison.Ordinal); - } + => SymbolSearchQueryNormalizer.ShouldPreserveRustQualifiedExactQuery(query, lang, exact); private static string? NormalizeSymbolSearchQueryForSymbolSearch(string? query, string? lang, bool exact) - { - if (ShouldPreserveRustQualifiedExactQuery(query, lang, exact)) - return query?.Trim(); - if (exact - && !string.IsNullOrWhiteSpace(query) - && SqlNameResolver.HasQualifier(query)) - { - if (string.Equals( - NormalizeQueryLanguage(lang), - "csharp", - StringComparison.Ordinal)) - { - return CSharpSymbolNameNormalizer.NormalizeExplicitInterfaceQueryDisplayName(query); - } - - // Without a language filter the query must retain its original spelling for - // non-C# exact matching. C#-specific display and identity aliases are supplied - // through their own SQL parameters. - // 言語フィルターがない場合、C# 以外の完全一致を保つため query の元表記を - // 維持する。C# 専用の表示名・identity alias は個別の SQL parameter で渡す。 - if (string.IsNullOrWhiteSpace(lang)) - { - var terraformNormalized = NormalizeTerraformDottedQuery(query, lang); - if (terraformNormalized != null) - return terraformNormalized; - return query.Trim(); - } - } - - return NormalizeSymbolSearchQuery(query, lang, exact) ?? query; - } + => SymbolSearchQueryNormalizer.NormalizeForSymbolSearch(query, lang, exact); private static (string? QualifiedPath, string? ContainerPath, string? LeafName) NormalizeRustQualifiedExactQueryParts(string query) - { - var trimmed = query.Trim(); - if (trimmed.Length == 0) - return (null, null, null); - - if (trimmed.EndsWith("!", StringComparison.Ordinal)) - trimmed = trimmed[..^1].TrimEnd(); - - var normalized = NormalizeRustQualifiedMacroQuery(trimmed); - if (string.IsNullOrWhiteSpace(normalized)) - return (null, null, null); - - normalized = normalized.Replace("::", "."); - while (normalized.StartsWith("crate.", StringComparison.Ordinal) - || normalized.StartsWith("self.", StringComparison.Ordinal) - || normalized.StartsWith("super.", StringComparison.Ordinal)) - { - var dotIndex = normalized.IndexOf('.'); - if (dotIndex < 0 || dotIndex == normalized.Length - 1) - break; - - normalized = normalized[(dotIndex + 1)..]; - } - var lastDot = normalized.LastIndexOf('.'); - if (lastDot < 0) - return (normalized, string.Empty, normalized); - - return (normalized, normalized[..lastDot], normalized[(lastDot + 1)..]); - } - + => SymbolSearchQueryNormalizer.NormalizeRustQualifiedExactQueryParts(query); } diff --git a/src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.Candidates.cs b/src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.Candidates.cs new file mode 100644 index 0000000000..f124495219 --- /dev/null +++ b/src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.Candidates.cs @@ -0,0 +1,455 @@ +using System.Text; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class UnusedCandidateSymbol + { + public long FileId { get; init; } + public string Path { get; init; } = string.Empty; + public string? Lang { get; init; } + public string Kind { get; init; } = string.Empty; + public string Name { get; init; } = string.Empty; + public int Line { get; init; } + public int StartLine { get; init; } + public int EndLine { get; init; } + public string? Signature { get; init; } + public string? Visibility { get; init; } + public string? ReturnType { get; init; } + public string? ContainerKind { get; init; } + public string? ContainerName { get; init; } + public string? ContainerQualifiedName { get; init; } + public bool IsPublicOrExported { get; init; } + public bool IsReflectionOrConfigSuspect { get; init; } + } + + private readonly record struct UnusedCandidateScope( + string? Kind, + string? Lang, + IReadOnlyList? PathPatterns, + IReadOnlyList? ExcludePathPatterns, + bool ExcludeTests, + IReadOnlyList? VisibilityFilters, + IReadOnlyList? ExcludeVisibilityFilters); + + private readonly record struct UnusedCandidateQueryPlan( + string Sql, + IReadOnlyList GraphLanguages); + + private readonly record struct UnusedCandidateBucketSql( + string IsPublicOrExported, + string IsReflectionOrConfigSuspect, + string ProvisionalOrder); + + private const string UnusedCandidateColumns = + "file_id, path, lang, kind, name, line, start_line, end_line, signature, visibility, " + + "return_type, container_kind, container_name, container_qualified_name, " + + "is_public_or_exported, is_reflection_or_config_suspect, provisional_bucket_order"; + + private List FetchUnusedCandidates( + int fetchLimit, + int provisionalBucketOrder, + int offset, + string? kind, + string? lang, + IReadOnlyList? pathPatterns, + IReadOnlyList? excludePathPatterns, + bool excludeTests, + IReadOnlyList? visibilityFilters = null, + IReadOnlyList? excludeVisibilityFilters = null) + { + var scope = new UnusedCandidateScope( + kind, lang, pathPatterns, excludePathPatterns, excludeTests, + visibilityFilters, excludeVisibilityFilters); + var plan = BuildUnusedCandidatePageQuery(scope, resolveSqlReferences: true); + using var cmd = _conn.CreateCommand(); + cmd.CommandText = plan.Sql; + AddUnusedCandidatePageParameters(cmd, plan, scope, provisionalBucketOrder, fetchLimit, offset); + + var results = new List(); + using var reader = cmd.ExecuteTrackedReader(); + while (reader.TrackedRead()) + results.Add(CreateUnusedSymbolResult(ReadUnusedCandidate(reader))); + return results; + } + + private List FetchUnusedCandidateSymbols( + int fetchLimit, + int offset, + int provisionalBucketOrder, + string? kind, + string? lang, + IReadOnlyList? pathPatterns, + IReadOnlyList? excludePathPatterns, + bool excludeTests, + IReadOnlyList? visibilityFilters = null, + IReadOnlyList? excludeVisibilityFilters = null) + { + var scope = new UnusedCandidateScope( + kind, lang, pathPatterns, excludePathPatterns, excludeTests, + visibilityFilters, excludeVisibilityFilters); + var plan = BuildUnusedCandidatePageQuery(scope, resolveSqlReferences: false); + using var cmd = _conn.CreateCommand(); + cmd.CommandText = plan.Sql; + AddUnusedCandidatePageParameters(cmd, plan, scope, provisionalBucketOrder, fetchLimit, offset); + + var candidates = new List(); + using var reader = cmd.ExecuteTrackedReader(); + while (reader.TrackedRead()) + candidates.Add(ReadUnusedCandidate(reader)); + return candidates; + } + + private QueryCountResult CountUnusedCandidates(UnusedCandidateScope scope) + { + var plan = BuildUnusedCandidateCountQuery(scope); + using var cmd = _conn.CreateCommand(); + cmd.CommandText = plan.Sql; + AddUnusedCandidateScopeParameters(cmd, plan, scope); + + using var reader = cmd.ExecuteTrackedReader(); + if (!reader.TrackedRead()) + return new QueryCountResult(0, 0); + return new QueryCountResult( + reader.GetInt32(0), + reader.GetInt32(1), + reader.FieldCount > 2 + && !reader.IsDBNull(2) + && Convert.ToInt32(reader.GetValue(2)) != 0); + } + + private UnusedCandidateQueryPlan BuildUnusedCandidatePageQuery( + UnusedCandidateScope scope, + bool resolveSqlReferences) + { + var graphLanguages = GetUnusedCandidateGraphLanguages(scope.Lang, resolveSqlReferences); + var bucketSql = BuildUnusedCandidateBucketSql(); + var hasUsableUnusedChunks = HasUsableUnusedChunks(); + var useCSharpPartialUseExclusion = CanApplyCSharpPartialUseExclusion( + scope, + hasUsableUnusedChunks); + var sql = new StringBuilder( + capacity: resolveSqlReferences || useCSharpPartialUseExclusion ? 5_000 : 3_000); + if (resolveSqlReferences) + sql.Append("\n WITH unused_candidates AS (\n SELECT "); + else + sql.Append("\n SELECT "); + sql.Append(BuildUnusedCandidateProjectionSql(bucketSql)); + AppendUnusedCandidateSourceSql( + sql, + scope, + graphLanguages, + resolveSqlReferences, + hasUsableUnusedChunks, + useCSharpPartialUseExclusion); + + if (resolveSqlReferences) + { + sql.Append("\n )\n SELECT "); + sql.Append(UnusedCandidateColumns); + sql.Append("\n FROM unused_candidates\n WHERE provisional_bucket_order = @bucketOrder"); + } + else + { + sql.Append(" AND ("); + sql.Append(bucketSql.ProvisionalOrder); + sql.Append(") = @bucketOrder"); + } + sql.Append("\n ORDER BY path, line, name\n LIMIT @limit OFFSET @offset"); + return new UnusedCandidateQueryPlan(sql.ToString(), graphLanguages); + } + + private UnusedCandidateQueryPlan BuildUnusedCandidateCountQuery(UnusedCandidateScope scope) + { + var graphLanguages = GetUnusedCandidateGraphLanguages(scope.Lang, resolveSqlReferences: true); + var hasUsableUnusedChunks = HasUsableUnusedChunks(); + var useCSharpPartialUseExclusion = CanApplyCSharpPartialUseExclusion( + scope, + hasUsableUnusedChunks); + var sql = new StringBuilder(capacity: useCSharpPartialUseExclusion ? 5_000 : 4_000); + sql.Append(@" + SELECT COUNT(*), COUNT(DISTINCT f.path), MAX(CASE WHEN f.lang = 'sql' THEN 1 ELSE 0 END)"); + AppendUnusedCandidateSourceSql( + sql, + scope, + graphLanguages, + resolveSqlReferences: true, + hasUsableUnusedChunks: hasUsableUnusedChunks, + useCSharpPartialUseExclusion: useCSharpPartialUseExclusion); + return new UnusedCandidateQueryPlan(sql.ToString(), graphLanguages); + } + + private IReadOnlyList GetUnusedCandidateGraphLanguages( + string? lang, + bool resolveSqlReferences) + { + var supportedLanguages = GetWorkspaceSupportedReferenceLanguages(); + if (lang != null) + return Array.Empty(); + + var graphLanguages = new List(); + foreach (var language in supportedLanguages) + { + if (resolveSqlReferences || !IsSqlLanguage(language)) + graphLanguages.Add(language); + } + return graphLanguages; + } + + private void AppendUnusedCandidateSourceSql( + StringBuilder sql, + UnusedCandidateScope scope, + IReadOnlyList graphLanguages, + bool resolveSqlReferences, + bool hasUsableUnusedChunks, + bool useCSharpPartialUseExclusion) + { + sql.Append(@" + FROM symbols s + JOIN files f ON s.file_id = f.id + WHERE s.kind NOT IN ('import', 'namespace')"); + sql.Append(BuildUnusedReferenceAbsenceSql(resolveSqlReferences)); + if (useCSharpPartialUseExclusion || (resolveSqlReferences && hasUsableUnusedChunks)) + { + var visibilitySql = $"lower({GetSymbolColumnSql("visibility", "''")})"; + if (resolveSqlReferences && hasUsableUnusedChunks) + { + sql.Append(BuildSameFilePrivateUseExclusionSql( + "s", "f", visibilitySql, + GetSymbolColumnSql("start_line", "s.line"), + GetSymbolColumnSql("end_line", "s.line"))); + } + if (useCSharpPartialUseExclusion) + sql.Append(BuildCSharpPartialContainingTypeUseExclusionSql("s", "f", visibilitySql)); + } + sql.Append("\n AND "); + sql.Append(BuildAmbiguousCSharpEnumMemberExclusionSql( + "s", "f", scope.PathPatterns, scope.ExcludePathPatterns, scope.ExcludeTests)); + AppendUnusedCandidateScopeSql(sql, scope, graphLanguages); + } + + private bool CanApplyCSharpPartialUseExclusion( + UnusedCandidateScope scope, + bool hasUsableUnusedChunks) + { + if (scope.Lang != null + && !string.Equals(scope.Lang, "csharp", StringComparison.Ordinal)) + { + return false; + } + + return hasUsableUnusedChunks + && _symbolColumns.Contains("visibility") + && _symbolColumns.Contains("container_kind") + && _symbolColumns.Contains("container_name") + && _symbolColumns.Contains("signature"); + } + + private bool HasUsableUnusedChunks() + => _hasChunksTable + && _chunkColumns.Contains("file_id") + && _chunkColumns.Contains("start_line") + && _chunkColumns.Contains("end_line") + && _chunkColumns.Contains("content") + && HasTable("chunks"); + + private string BuildUnusedReferenceAbsenceSql(bool resolveSqlReferences) + { + if (!resolveSqlReferences) + { + return @" + AND NOT EXISTS ( + SELECT 1 + FROM symbol_references sr + WHERE sr.symbol_name IS NOT NULL + AND sr.symbol_name <> '' + AND sr.symbol_name = s.name + )"; + } + + var referenceContextSql = ReferenceContextSql("sr"); + return @" + AND NOT EXISTS ( + SELECT 1 + FROM symbol_references sr + JOIN files rf ON rf.id = sr.file_id" + ReferenceLineJoinSql("sr") + @" + WHERE sr.symbol_name = s.name + OR (f.lang = 'sql' AND rf.lang = 'sql' AND ( + (sql_resolve_reference_segment_count_at(sr.symbol_name, " + referenceContextSql + @", sr.container_name, sr.column_number) = sql_segment_count(s.name) + AND sql_reference_matches_target_at(sr.symbol_name, " + referenceContextSql + @", sr.container_name, sr.column_number, s.name) = 1) + OR (sql_segment_count(sr.symbol_name) = 1 + AND sql_allow_leaf_fallback_at(sr.symbol_name, " + referenceContextSql + @", sr.container_name, sr.column_number) = 1 + AND sr.symbol_name = sql_leaf_name(s.name) COLLATE NOCASE + AND NOT EXISTS ( + SELECT 1 + FROM symbols s_exact + JOIN files f_exact ON f_exact.id = s_exact.file_id + WHERE f_exact.lang = 'sql' + AND sql_segment_count(s_exact.name) = sql_resolve_reference_segment_count_at(sr.symbol_name, " + referenceContextSql + @", sr.container_name, sr.column_number) + AND sql_reference_matches_target_at(sr.symbol_name, " + referenceContextSql + @", sr.container_name, sr.column_number, s_exact.name) = 1 + )) + )) + )"; + } + + private void AppendUnusedCandidateScopeSql( + StringBuilder sql, + UnusedCandidateScope scope, + IReadOnlyList graphLanguages) + { + var filters = string.Empty; + if (scope.Lang != null) + filters += SymbolLanguageFileIdFilter; + else + filters += $" AND f.lang IN ({string.Join(",", graphLanguages.Select((_, i) => $"@gl{i}"))})"; + if (scope.Kind != null) + filters += " AND s.kind = @kind"; + AppendPathFilters( + ref filters, + scope.PathPatterns, + scope.ExcludePathPatterns, + scope.ExcludeTests); + AppendVisibilityFilters( + ref filters, + scope.VisibilityFilters, + scope.ExcludeVisibilityFilters); + sql.Append(filters); + } + + private static void AddUnusedCandidatePageParameters( + SqliteCommand command, + UnusedCandidateQueryPlan plan, + UnusedCandidateScope scope, + int provisionalBucketOrder, + int fetchLimit, + int offset) + { + SqliteCommandPolicy.Add(command, "@bucketOrder", provisionalBucketOrder); + SqliteCommandPolicy.Add(command, "@limit", fetchLimit); + SqliteCommandPolicy.Add(command, "@offset", offset); + AddUnusedCandidateScopeParameters(command, plan, scope); + } + + private static void AddUnusedCandidateScopeParameters( + SqliteCommand command, + UnusedCandidateQueryPlan plan, + UnusedCandidateScope scope) + { + if (scope.Lang != null) + SqliteCommandPolicy.Add(command, "@lang", scope.Lang); + else + { + for (var index = 0; index < plan.GraphLanguages.Count; index++) + SqliteCommandPolicy.Add(command, $"@gl{index}", plan.GraphLanguages[index]); + } + if (scope.Kind != null) + SqliteCommandPolicy.Add(command, "@kind", scope.Kind); + AddPathFilterParameters(command, scope.PathPatterns, scope.ExcludePathPatterns); + AddVisibilityFilterParameters( + command, + scope.VisibilityFilters, + scope.ExcludeVisibilityFilters); + } + + private string BuildUnusedCandidateProjectionSql(UnusedCandidateBucketSql bucketSql) => $@"s.file_id, f.path, f.lang, s.kind, s.name, s.line, + {GetSymbolColumnSql("start_line", "s.line")} AS start_line, + {GetSymbolColumnSql("end_line", "s.line")} AS end_line, + {GetSymbolColumnSql("signature")} AS signature, + {GetSymbolColumnSql("visibility")} AS visibility, + {GetSymbolColumnSql("return_type")} AS return_type, + {GetSymbolColumnSql("container_kind")} AS container_kind, + {GetSymbolColumnSql("container_name")} AS container_name, + {GetSymbolColumnSql("container_qualified_name", GetSymbolColumnSql("container_name", "''"))} AS container_qualified_name, + CASE WHEN {bucketSql.IsPublicOrExported} THEN 1 ELSE 0 END AS is_public_or_exported, + CASE WHEN {bucketSql.IsReflectionOrConfigSuspect} THEN 1 ELSE 0 END AS is_reflection_or_config_suspect, + {bucketSql.ProvisionalOrder} AS provisional_bucket_order"; + + private UnusedCandidateBucketSql BuildUnusedCandidateBucketSql() + { + var visibilitySql = $"lower({GetSymbolColumnSql("visibility", "''")})"; + var signatureSql = $"lower({GetSymbolColumnSql("signature", "''")})"; + var isPublicOrExportedSql = $"{visibilitySql} IN ('public', 'open', 'pub', 'export')"; + var reflectionOrConfigSql = $@"( + {isPublicOrExportedSql} + AND s.kind = 'property' + AND ( + lower(f.path) LIKE 'config/%' + OR lower(f.path) LIKE '%/config/%' + OR lower(f.path) LIKE 'settings/%' + OR lower(f.path) LIKE '%/settings/%' + OR lower(f.path) LIKE 'options/%' + OR lower(f.path) LIKE '%/options/%' + OR {signatureSql} LIKE '%iconfiguration%' + OR {signatureSql} LIKE '%configurationsection%' + OR {signatureSql} LIKE '%ioptions%' + OR {signatureSql} LIKE '%options<%' + ) + )"; + var provisionalOrderSql = $@"CASE + WHEN {reflectionOrConfigSql} THEN 3 + WHEN {isPublicOrExportedSql} THEN 2 + WHEN {visibilitySql} IN ('private', 'fileprivate') THEN 0 + ELSE 1 + END"; + return new UnusedCandidateBucketSql( + isPublicOrExportedSql, + reflectionOrConfigSql, + provisionalOrderSql); + } + + private static UnusedCandidateSymbol ReadUnusedCandidate(SqliteDataReader reader) => new() + { + FileId = reader.GetInt64(0), + Path = reader.GetString(1), + Lang = GetNullableString(reader, 2), + Kind = reader.GetString(3), + Name = reader.GetString(4), + Line = reader.GetInt32(5), + StartLine = GetInt32OrFallback(reader, 6, 5), + EndLine = GetInt32OrFallback(reader, 7, 5), + Signature = GetNullableString(reader, 8), + Visibility = GetNullableString(reader, 9), + ReturnType = GetNullableString(reader, 10), + ContainerKind = GetNullableString(reader, 11), + ContainerName = GetNullableString(reader, 12), + ContainerQualifiedName = GetNullableString(reader, 13), + IsPublicOrExported = reader.GetInt32(14) != 0, + IsReflectionOrConfigSuspect = reader.GetInt32(15) != 0, + }; + + private string BuildAmbiguousCSharpEnumMemberExclusionSql( + string symbolAlias, + string fileAlias, + IReadOnlyList? pathPatterns, + IReadOnlyList? excludePathPatterns, + bool excludeTests) + { + var symbolContainerKindSql = GetSymbolColumnSql("container_kind", "''", symbolAlias); + var symbolContainerNameSql = GetSymbolColumnSql("container_name", "''", symbolAlias); + var symbolContainerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name", symbolContainerNameSql, symbolAlias); + var peerContainerKindSql = GetSymbolColumnSql("container_kind", "''", "s_peer"); + var peerContainerNameSql = GetSymbolColumnSql("container_name", "''", "s_peer"); + var peerContainerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name", peerContainerNameSql, "s_peer"); + var peerPathFiltersSql = BuildPathFiltersSql("f_peer", pathPatterns, excludePathPatterns, excludeTests); + + return $@" + NOT ( + {fileAlias}.lang = 'csharp' + AND {symbolAlias}.kind = 'enum' + AND {symbolContainerKindSql} = 'enum' + AND EXISTS ( + SELECT 1 + FROM symbols s_peer + JOIN files f_peer ON f_peer.id = s_peer.file_id + WHERE f_peer.lang = 'csharp' + {peerPathFiltersSql} + AND s_peer.kind = 'enum' + AND {peerContainerKindSql} = 'enum' + AND s_peer.name = {symbolAlias}.name + AND {peerContainerQualifiedNameSql} <> {symbolContainerQualifiedNameSql} + ) + )"; + } +} diff --git a/src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.Classification.cs b/src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.Classification.cs new file mode 100644 index 0000000000..5a03bc408e --- /dev/null +++ b/src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.Classification.cs @@ -0,0 +1,650 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private const string UnusedBucketLikelyPrivate = "likely_unused_private"; + private const string UnusedBucketMaybeNonPublic = "maybe_unused_nonpublic"; + private const string UnusedBucketPublicOrExported = "public_or_exported_no_refs"; + private const string UnusedBucketReflectionOrConfig = "reflection_or_config_suspect"; + private const string UnusedContractDomainPrivate = "private_or_file_local"; + private const string UnusedContractDomainNonPublic = "nonpublic_internal"; + private const string UnusedContractDomainPublicApi = "public_api_surface"; + private const string UnusedContractDomainCli = "cli_contract"; + private const string UnusedContractDomainJson = "json_contract"; + private const string UnusedContractDomainMcp = "mcp_contract"; + private const string UnusedContractDomainLsp = "lsp_contract"; + private const string UnusedContractDomainConfig = "configuration_contract"; + private const string UnusedContractDomainSerialization = "serialization_or_reflection_contract"; + private const string UnusedContractDomainGenerated = "generated_code"; + private const string UnusedContractDomainDocumentation = "documentation_surface"; + private const string UnusedContractDomainTest = "test_contract"; + private const string UnusedContractDomainFrameworkOverride = "framework_override"; + private const string UnusedContractDomainExceptionDiagnostic = "exception_diagnostic"; + + private static readonly string[] UnusedContractPathSegments = ["/contracts/", "/contract/", "/dtos/", "/dto/", "/models/", "/model/", "/schemas/", "/schema/"]; + private static readonly string[] UnusedRecordContractSuffixes = + [ + "Dto", + "DTO", + "Request", + "Response", + "Result", + "Results", + "Model", + "Payload", + "Envelope", + ]; + private static readonly string[] UnusedGeneratedPathMarkers = + [ + ".g.cs", + ".generated.", + "/generated/", + "/obj/", + "/bin/", + ]; + private static readonly string[] UnusedCliPathMarkers = ["/cli/", "/commands/", "/commandline/"]; + private static readonly string[] UnusedMcpPathMarkers = ["/mcp/"]; + private static readonly string[] UnusedLspPathMarkers = ["/lsp/", "/languageserver/"]; + private static readonly string[] UnusedJsonContractTerms = + [ + "Json", + "Dto", + "DTO", + "Request", + "Response", + "Result", + "Results", + "Payload", + "Envelope", + "Contract", + "Schema", + ]; + private static readonly HashSet UnusedFrameworkOverrideMemberNames = new(StringComparer.OrdinalIgnoreCase) + { + "CanRead", + "CanSeek", + "CanWrite", + "Length", + "Position", + "Flush", + "FlushAsync", + "Read", + "ReadAsync", + "ReadByte", + "Seek", + "SetLength", + "Write", + "WriteAsync", + "WriteByte", + "Dispose", + "DisposeAsync", + }; + private static readonly HashSet UnusedExceptionMetadataNames = new(StringComparer.OrdinalIgnoreCase) + { + "CharactersRead", + "Utf8BytesRead", + "SizeLimit", + "Limit", + "ByteCount", + "BytesRead", + "Offset", + "Position", + "Path", + "FileName", + "LineNumber", + "ColumnNumber", + "ActualSize", + "MaxSize", + "Length", + }; + private static readonly string[] UnusedConfigMemberTerms = + [ + "Configuration", + "Config", + "Options", + "Settings", + "Manifest", + "Schema", + "Metadata", + "Limit", + "Max", + "Min", + "Size", + "Bytes", + "Count", + "Capacity", + "Timeout", + "Version", + "Kind", + "Category", + "Severity", + "Source", + "Target", + "Path", + "Name", + "Id", + "Key", + ]; + + private static readonly string[] UnusedJsonContextSuffixes = ["JsonContext"]; + private static readonly string[] UnusedJsonContextSignatureTerms = ["JsonSerializerContext", "JsonSerializable", "JsonSourceGenerationOptions"]; + private static readonly string[] UnusedContractTypeSignatureTerms = ["DataContract", "Serializable", "MessagePackObject", "ProtoContract"]; + private static readonly string[] UnusedContractMemberSignatureTerms = ["JsonProperty", "JsonInclude", "DataMember", "XmlElement", "XmlAttribute"]; + private static readonly string[] UnusedConfigNameSuffixes = ["Options", "Settings", "Configuration", "Config", "Manifest", "Schema"]; + private static readonly string[] UnusedConfigNameTerms = ["Configuration", "IOptions"]; + private static readonly string[] UnusedConfigSignatureTerms = ["IConfiguration", "ConfigurationSection", "IOptions", "Options<"]; + private static readonly string[] UnusedConfigPathSegments = ["/config/", "/configuration/", "/options/", "/settings/", "/manifest/", "/manifests/"]; + private static readonly string[] UnusedMetadataSignatureTerms = ["JsonProperty", "JsonInclude", "DataMember", "Diagnostic", "Metadata", "IConfiguration", "IOptions", "Options<"]; + private static readonly string[] UnusedExceptionSuffixes = ["Exception"]; + private static readonly string[] UnusedTestContainerSuffixes = ["Test", "Tests", "Fixture"]; + private static readonly string[] UnusedFrameworkOverrideSignatureTerms = [" override ", " override\t", " override\r", " override\n"]; + private static readonly string[] UnusedFrameworkContainerSuffixes = ["Stream", "TextReader", "TextWriter"]; + private static readonly string[] UnusedMcpContractTerms = ["Mcp", "JsonRpc"]; + private static readonly string[] UnusedLspContractTerms = ["Lsp", "LanguageServer"]; + private static readonly string[] UnusedCliContractTerms = ["Cli", "CommandLine"]; + private static readonly string[] UnusedCliSignatureTerms = ["CommandLine", "System.CommandLine", "Option<", "Argument<"]; + private static readonly string[] UnusedCliContractSuffixes = ["Command", "Flag", "Flags", "Usage", "ExitCode", "ErrorCode"]; + private static readonly string[] UnusedJsonSignatureTerms = + [ + "JsonProperty", + "JsonInclude", + "JsonSerializerContext", + "DataContract", + "DataMember", + "XmlElement", + "XmlAttribute", + "YamlMember", + "MessagePackObject", + "ProtoContract", + ]; + + internal static readonly string[] OrderedUnusedContractDomains = + [ + UnusedContractDomainPrivate, + UnusedContractDomainNonPublic, + UnusedContractDomainPublicApi, + UnusedContractDomainCli, + UnusedContractDomainJson, + UnusedContractDomainMcp, + UnusedContractDomainLsp, + UnusedContractDomainConfig, + UnusedContractDomainSerialization, + UnusedContractDomainGenerated, + UnusedContractDomainDocumentation, + UnusedContractDomainTest, + UnusedContractDomainFrameworkOverride, + UnusedContractDomainExceptionDiagnostic, + ]; + + private static readonly string[] OrderedUnusedBuckets = + [ + UnusedBucketLikelyPrivate, + UnusedBucketMaybeNonPublic, + UnusedBucketPublicOrExported, + UnusedBucketReflectionOrConfig, + ]; + + [Flags] + private enum UnusedSurfaceFlags : ushort + { + None = 0, + ReflectionOrConfigSuspect = 1 << 0, + DocumentationHeading = 1 << 1, + MarkdownFenceLanguageMarker = 1 << 2, + GeneratedSurface = 1 << 3, + SerializationContract = 1 << 4, + SourceGeneratedJsonContext = 1 << 5, + ContractMember = 1 << 6, + ConfigOrMetadataSurface = 1 << 7, + TestHook = 1 << 8, + ExceptionMetadata = 1 << 9, + ConfigOrMetadataMember = 1 << 10, + } + + private readonly record struct UnusedSurfaceAnalysis(UnusedSurfaceFlags Flags, List Tags); + private readonly record struct UnusedContractDomainClassification(string Domain, List Tags); + private UnusedSymbolResult CreateUnusedSymbolResult(UnusedCandidateSymbol candidate) + { + var kind = NormalizeUnusedSymbolKind(candidate); + var surface = AnalyzeUnusedSurfaces(candidate, kind); + var classification = ClassifyUnusedSymbol( + candidate.IsPublicOrExported, + surface.Tags.Count > 0, + candidate.Visibility); + var reasonTags = BuildUnusedReasonTags(candidate, surface); + var contractDomain = ClassifyUnusedContractDomain( + candidate, + kind, + classification.Bucket, + surface); + return new UnusedSymbolResult + { + Path = candidate.Path, + Lang = candidate.Lang, + Kind = kind, + Name = candidate.Name, + Line = candidate.Line, + StartLine = candidate.StartLine, + EndLine = candidate.EndLine, + Signature = candidate.Signature, + Visibility = candidate.Visibility, + ReturnType = candidate.ReturnType, + ContainerKind = candidate.ContainerKind, + ContainerName = candidate.ContainerName, + UnusedBucket = classification.Bucket, + UnusedConfidence = classification.Confidence, + UnusedReason = classification.Reason, + UnusedReasonTags = reasonTags, + UnusedContractDomain = contractDomain.Domain, + UnusedContractDomainTags = contractDomain.Tags, + }; + } + + private UnusedSurfaceAnalysis AnalyzeUnusedSurfaces( + UnusedCandidateSymbol candidate, + string kind) + { + var flags = UnusedSurfaceFlags.None; + var tags = new List(); + if (candidate.IsReflectionOrConfigSuspect) + AddUnusedSurface(ref flags, UnusedSurfaceFlags.ReflectionOrConfigSuspect, tags, "reflection_or_config_suspect"); + if (IsMarkdownHeadingSymbol(candidate, kind)) + AddUnusedSurface(ref flags, UnusedSurfaceFlags.DocumentationHeading, tags, "documentation_heading"); + if (IsMarkdownFenceSymbol(candidate, kind)) + AddUnusedSurface(ref flags, UnusedSurfaceFlags.MarkdownFenceLanguageMarker, tags, "markdown_fence_language_marker"); + if (IsGeneratedSurface(candidate)) + AddUnusedSurface(ref flags, UnusedSurfaceFlags.GeneratedSurface, tags, "generated_surface"); + AddUnusedSerializationSurfaces(candidate, kind, ref flags, tags); + if (IsConfigOrManifestSurface(candidate, kind)) + AddUnusedSurface(ref flags, UnusedSurfaceFlags.ConfigOrMetadataSurface, tags, "config_or_metadata_surface"); + if (IsTestHookName(candidate.Name)) + AddUnusedSurface(ref flags, UnusedSurfaceFlags.TestHook, tags, "test_hook"); + if (IsExceptionMetadataProperty(candidate, kind)) + AddUnusedSurface(ref flags, UnusedSurfaceFlags.ExceptionMetadata, tags, "exception_metadata"); + if (IsConfigOrMetadataMember(candidate, kind)) + AddUnusedSurface(ref flags, UnusedSurfaceFlags.ConfigOrMetadataMember, tags, "config_or_metadata_member"); + if ((flags & UnusedSurfaceFlags.ReflectionOrConfigSuspect) == 0 + && candidate.IsPublicOrExported + && HasReflectionAttributeContext(kind, candidate.Path, candidate.StartLine)) + { + AddUnusedSurface(ref flags, UnusedSurfaceFlags.ReflectionOrConfigSuspect, tags, "reflection_or_config_suspect"); + } + return new UnusedSurfaceAnalysis(flags, tags); + } + + private static void AddUnusedSerializationSurfaces( + UnusedCandidateSymbol candidate, + string kind, + ref UnusedSurfaceFlags flags, + List tags) + { + if (IsSourceGeneratedJsonContext(candidate)) + { + AddUnusedSurface(ref flags, UnusedSurfaceFlags.SerializationContract, tags, "serialization_contract"); + AddUnusedSurface(ref flags, UnusedSurfaceFlags.SourceGeneratedJsonContext, tags, "source_generated_json_context"); + } + if (IsUnusedContractType(candidate, kind)) + AddUnusedSurface(ref flags, UnusedSurfaceFlags.SerializationContract, tags, "serialization_contract"); + if (IsUnusedContractMember(candidate, kind)) + AddUnusedSurface(ref flags, UnusedSurfaceFlags.ContractMember, tags, "contract_member"); + } + + private static void AddUnusedSurface( + ref UnusedSurfaceFlags flags, + UnusedSurfaceFlags flag, + List tags, + string tag) + { + flags |= flag; + AddUnusedSurfaceTag(tags, tag); + } + + private static void AddUnusedSurfaceTag(List tags, string tag) + { + if (!tags.Contains(tag, StringComparer.Ordinal)) + tags.Add(tag); + } + + private static (string Bucket, string Confidence, string Reason) ClassifyUnusedSymbol( + bool isPublicOrExported, + bool isIntentionalSurfaceSuspect, + string? visibility) + { + if (isIntentionalSurfaceSuspect) + { + return ( + UnusedBucketReflectionOrConfig, + "low", + "symbol with attribute-driven reflection surface, serialization, config, metadata, test-hook, generated, documentation, or compatibility surface and no indexed references"); + } + if (isPublicOrExported) + return (UnusedBucketPublicOrExported, "low", "public/exported symbol with no indexed references"); + if (IsPrivateLikeVisibility(visibility)) + { + return ( + UnusedBucketLikelyPrivate, + "medium", + "private/file-local symbol with no indexed references after same-file text validation"); + } + return (UnusedBucketMaybeNonPublic, "low", "non-public symbol with no indexed references"); + } + + private static List BuildUnusedReasonTags( + UnusedCandidateSymbol candidate, + UnusedSurfaceAnalysis surface) + { + var tags = new List { "no_indexed_references" }; + if (surface.Tags.Count > 0) + { + tags.Add("intentional_surface_suspect"); + tags.Add("reflection_or_config_suspect"); + } + foreach (var surfaceTag in surface.Tags) + AddUnusedSurfaceTag(tags, surfaceTag); + if (candidate.IsPublicOrExported) + tags.Add("public_or_exported"); + else if (IsPrivateLikeVisibility(candidate.Visibility)) + tags.Add("private_or_file_local"); + else + tags.Add("non_public"); + return tags; + } + + private static UnusedContractDomainClassification ClassifyUnusedContractDomain( + UnusedCandidateSymbol candidate, + string kind, + string bucket, + UnusedSurfaceAnalysis surface) + { + var tags = new List(); + if (string.Equals(bucket, UnusedBucketReflectionOrConfig, StringComparison.Ordinal)) + AddUnusedSurfaceTag(tags, "intentional_surface_suspect"); + foreach (var surfaceTag in surface.Tags) + AddUnusedSurfaceTag(tags, surfaceTag); + + if (IsPrivateLikeVisibility(candidate.Visibility)) + return CreateUnusedContractDomain(UnusedContractDomainPrivate, tags, "private_or_file_local"); + if (!candidate.IsPublicOrExported && surface.Tags.Count == 0) + return CreateUnusedContractDomain(UnusedContractDomainNonPublic, tags, "nonpublic_or_protected"); + + AddUnusedSurfaceTag(tags, candidate.IsPublicOrExported ? "public_or_exported" : "nonpublic_or_protected"); + return ClassifyExposedUnusedContractDomain(candidate, kind, surface, tags); + } + + private static UnusedContractDomainClassification ClassifyExposedUnusedContractDomain( + UnusedCandidateSymbol candidate, + string kind, + UnusedSurfaceAnalysis surface, + List tags) + { + if (HasUnusedSurface(surface, UnusedSurfaceFlags.DocumentationHeading)) + return CreateUnusedContractDomain(UnusedContractDomainDocumentation, tags, "documentation_heading"); + if (HasUnusedSurface(surface, UnusedSurfaceFlags.MarkdownFenceLanguageMarker)) + return CreateUnusedContractDomain(UnusedContractDomainDocumentation, tags, "markdown_fence_language_marker"); + if (IsUnusedTestContractSurface(candidate, surface)) + return CreateUnusedContractDomain(UnusedContractDomainTest, tags, "test_surface"); + if (HasUnusedSurface(surface, UnusedSurfaceFlags.GeneratedSurface)) + return CreateUnusedContractDomain(UnusedContractDomainGenerated, tags, "generated_surface"); + if (IsExceptionDiagnosticSurface(candidate, kind, surface)) + return CreateUnusedContractDomain(UnusedContractDomainExceptionDiagnostic, tags, "exception_metadata"); + if (IsFrameworkOverrideSurface(candidate)) + return CreateUnusedContractDomain(UnusedContractDomainFrameworkOverride, tags, "framework_override"); + if (IsMcpContractSurface(candidate)) + return CreateUnusedContractDomain(UnusedContractDomainMcp, tags, "mcp_tool_contract"); + if (IsLspContractSurface(candidate)) + return CreateUnusedContractDomain(UnusedContractDomainLsp, tags, "lsp_protocol_contract"); + if (IsCliContractSurface(candidate)) + return CreateUnusedContractDomain(UnusedContractDomainCli, tags, "cli_option_or_result"); + if (IsConfigurationContractSurface(surface)) + return CreateUnusedContractDomain(UnusedContractDomainConfig, tags, "configuration_or_metadata_contract"); + if (IsJsonContractSurface(candidate, surface)) + return CreateUnusedContractDomain(UnusedContractDomainJson, tags, "json_output_or_input_contract"); + if (HasUnusedSurface(surface, UnusedSurfaceFlags.ReflectionOrConfigSuspect)) + return CreateUnusedContractDomain(UnusedContractDomainSerialization, tags, "reflection_or_serialization_contract"); + return candidate.IsPublicOrExported + ? CreateUnusedContractDomain(UnusedContractDomainPublicApi, tags, "public_api_surface") + : CreateUnusedContractDomain(UnusedContractDomainNonPublic, tags, "nonpublic_or_protected"); + } + + private static UnusedContractDomainClassification CreateUnusedContractDomain( + string domain, + List tags, + string domainTag) + { + AddUnusedSurfaceTag(tags, domain); + AddUnusedSurfaceTag(tags, domainTag); + return new UnusedContractDomainClassification(domain, tags); + } + + private static bool HasUnusedSurface(UnusedSurfaceAnalysis surface, UnusedSurfaceFlags flag) + => (surface.Flags & flag) != 0; + + private static bool IsMarkdownHeadingSymbol(UnusedCandidateSymbol candidate, string kind) + { + return (string.Equals(candidate.Lang, "markdown", StringComparison.OrdinalIgnoreCase) + || string.Equals(candidate.Lang, "md", StringComparison.OrdinalIgnoreCase)) + && (kind.Contains("heading", StringComparison.OrdinalIgnoreCase) + || kind.Contains("header", StringComparison.OrdinalIgnoreCase)); + } + + private static bool IsMarkdownFenceSymbol(UnusedCandidateSymbol candidate, string kind) + { + if ((!string.Equals(candidate.Lang, "markdown", StringComparison.OrdinalIgnoreCase) + && !string.Equals(candidate.Lang, "md", StringComparison.OrdinalIgnoreCase)) + || !string.Equals(kind, "code", StringComparison.OrdinalIgnoreCase)) + { + return false; + } + var signature = candidate.Signature?.TrimStart(); + return signature?.StartsWith("```", StringComparison.Ordinal) == true + || signature?.StartsWith("~~~", StringComparison.Ordinal) == true; + } + + private static bool IsGeneratedSurface(UnusedCandidateSymbol candidate) + => ContainsAny(candidate.Path, UnusedGeneratedPathMarkers); + + private static bool IsSourceGeneratedJsonContext(UnusedCandidateSymbol candidate) + => EndsWithAny(candidate.Name, UnusedJsonContextSuffixes) + || ContainsAny(candidate.Signature, UnusedJsonContextSignatureTerms); + + private static bool IsUnusedContractType(UnusedCandidateSymbol candidate, string kind) + { + if (!IsTypeLikeUnusedKind(kind) || IsPrivateLikeVisibility(candidate.Visibility)) + return false; + return ContainsAny(candidate.Path, UnusedContractPathSegments) + || IsRecordContractType(candidate) + || ContainsAny(candidate.Signature, UnusedContractTypeSignatureTerms); + } + + private static bool IsUnusedContractMember(UnusedCandidateSymbol candidate, string kind) + { + if (!IsDataMemberUnusedKind(kind) || IsPrivateLikeVisibility(candidate.Visibility)) + return false; + return ContainsAny(candidate.Path, UnusedContractPathSegments) + || ContainsAny(candidate.Signature, UnusedContractMemberSignatureTerms); + } + + private static bool IsConfigOrManifestSurface(UnusedCandidateSymbol candidate, string kind) + { + if (IsPrivateLikeVisibility(candidate.Visibility)) + return false; + if (!IsTypeLikeUnusedKind(kind) && !IsDataMemberUnusedKind(kind) && !IsFunctionLikeUnusedKind(kind)) + return false; + if (EndsWithAny(candidate.Name, UnusedConfigNameSuffixes) + || ContainsAny(candidate.Name, UnusedConfigNameTerms) + || ContainsAny(candidate.Signature, UnusedConfigSignatureTerms)) + { + return true; + } + return IsDataMemberUnusedKind(kind) + && (ContainsAny(candidate.Path, UnusedConfigPathSegments) + || ContainsAny(candidate.Signature, UnusedConfigSignatureTerms)); + } + + private static bool IsTestHookName(string? name) + { + return !string.IsNullOrWhiteSpace(name) + && (name.EndsWith("ForTests", StringComparison.OrdinalIgnoreCase) + || name.Contains("ForTest", StringComparison.OrdinalIgnoreCase) + || name.Contains("TestOnly", StringComparison.OrdinalIgnoreCase)); + } + + private static bool IsExceptionMetadataProperty(UnusedCandidateSymbol candidate, string kind) + { + return string.Equals(kind, "property", StringComparison.OrdinalIgnoreCase) + && UnusedExceptionMetadataNames.Contains(candidate.Name) + && (EndsWithAny(candidate.ContainerName, UnusedExceptionSuffixes) + || ContainsAny(candidate.ContainerQualifiedName, UnusedExceptionSuffixes) + || ContainsAny(candidate.Signature, UnusedExceptionSuffixes)); + } + + private static bool IsConfigOrMetadataMember(UnusedCandidateSymbol candidate, string kind) + { + if (!IsDataMemberUnusedKind(kind) || IsPrivateLikeVisibility(candidate.Visibility)) + return false; + if (!ContainsAny(candidate.Name, UnusedConfigMemberTerms)) + return false; + return ContainsAny(candidate.Path, UnusedContractPathSegments) + || ContainsAny(candidate.Path, UnusedConfigPathSegments) + || ContainsAny(candidate.Signature, UnusedMetadataSignatureTerms); + } + + private static bool IsTypeLikeUnusedKind(string kind) + => kind is "class" or "struct" or "record" or "interface" or "enum" or "type"; + + private static bool IsDataMemberUnusedKind(string kind) + => kind is "property" or "field" or "constant" or "enum_member"; + + private static bool IsFunctionLikeUnusedKind(string kind) + => kind is "function" or "method"; + + private static bool IsRecordContractType(UnusedCandidateSymbol candidate) + => ContainsCSharpRecordKeyword(candidate.Signature) + && EndsWithAny(candidate.Name, UnusedRecordContractSuffixes); + + private static bool ContainsCSharpRecordKeyword(string? signature) + { + if (string.IsNullOrWhiteSpace(signature)) + return false; + var trimmed = signature.TrimStart(); + return trimmed.StartsWith("record ", StringComparison.Ordinal) + || trimmed.StartsWith("record(", StringComparison.Ordinal) + || signature.Contains(" record ", StringComparison.Ordinal) + || signature.Contains(" record(", StringComparison.Ordinal); + } + + private static bool EndsWithAny(string? value, IReadOnlyList suffixes) + { + if (string.IsNullOrWhiteSpace(value)) + return false; + foreach (var suffix in suffixes) + { + if (value.EndsWith(suffix, StringComparison.OrdinalIgnoreCase)) + return true; + } + return false; + } + + private static bool ContainsAny(string? value, IReadOnlyList fragments) + { + if (string.IsNullOrWhiteSpace(value)) + return false; + foreach (var fragment in fragments) + { + if (value.Contains(fragment, StringComparison.OrdinalIgnoreCase)) + return true; + } + return false; + } + + private static bool IsUnusedTestContractSurface( + UnusedCandidateSymbol candidate, + UnusedSurfaceAnalysis surface) + { + return IsUnusedTestPath(candidate.Path) + || EndsWithAny(candidate.ContainerName, UnusedTestContainerSuffixes) + || EndsWithAny(candidate.ContainerQualifiedName, UnusedTestContainerSuffixes) + || HasUnusedSurface(surface, UnusedSurfaceFlags.TestHook); + } + + private static bool IsUnusedTestPath(string? path) + { + if (string.IsNullOrWhiteSpace(path)) + return false; + var normalized = path.Replace('\\', '/'); + return normalized.StartsWith("test/", StringComparison.OrdinalIgnoreCase) + || normalized.StartsWith("tests/", StringComparison.OrdinalIgnoreCase) + || normalized.Contains("/test/", StringComparison.OrdinalIgnoreCase) + || normalized.Contains("/tests/", StringComparison.OrdinalIgnoreCase) + || normalized.Contains(".tests/", StringComparison.OrdinalIgnoreCase); + } + + private static bool IsExceptionDiagnosticSurface( + UnusedCandidateSymbol candidate, + string kind, + UnusedSurfaceAnalysis surface) + { + return HasUnusedSurface(surface, UnusedSurfaceFlags.ExceptionMetadata) + || (IsDataMemberUnusedKind(kind) + && (EndsWithAny(candidate.ContainerName, UnusedExceptionSuffixes) + || EndsWithAny(candidate.ContainerQualifiedName, UnusedExceptionSuffixes))) + || (IsTypeLikeUnusedKind(kind) && EndsWithAny(candidate.Name, UnusedExceptionSuffixes)); + } + + private static bool IsFrameworkOverrideSurface(UnusedCandidateSymbol candidate) + { + if (ContainsAny(candidate.Signature, UnusedFrameworkOverrideSignatureTerms)) + return true; + if (!UnusedFrameworkOverrideMemberNames.Contains(candidate.Name)) + return false; + return EndsWithAny(candidate.ContainerName, UnusedFrameworkContainerSuffixes) + || EndsWithAny(candidate.ContainerQualifiedName, UnusedFrameworkContainerSuffixes) + || ContainsAny(candidate.Signature, UnusedFrameworkContainerSuffixes); + } + + private static bool IsMcpContractSurface(UnusedCandidateSymbol candidate) + => ContainsAny(candidate.Path, UnusedMcpPathMarkers) + || ContainsAny(candidate.ContainerName, UnusedMcpContractTerms) + || ContainsAny(candidate.ContainerQualifiedName, UnusedMcpContractTerms) + || ContainsAny(candidate.Signature, UnusedMcpContractTerms); + + private static bool IsLspContractSurface(UnusedCandidateSymbol candidate) + => ContainsAny(candidate.Path, UnusedLspPathMarkers) + || ContainsAny(candidate.ContainerName, UnusedLspContractTerms) + || ContainsAny(candidate.ContainerQualifiedName, UnusedLspContractTerms) + || ContainsAny(candidate.Signature, UnusedLspContractTerms); + + private static bool IsCliContractSurface(UnusedCandidateSymbol candidate) + { + var hasCliContext = ContainsAny(candidate.Path, UnusedCliPathMarkers) + || ContainsAny(candidate.ContainerQualifiedName, UnusedCliContractTerms) + || ContainsAny(candidate.Signature, UnusedCliSignatureTerms); + if (hasCliContext) + return true; + return EndsWithAny(candidate.Name, UnusedCliContractSuffixes) + || EndsWithAny(candidate.ContainerName, UnusedCliContractSuffixes); + } + + private static bool IsConfigurationContractSurface(UnusedSurfaceAnalysis surface) + => HasUnusedSurface(surface, UnusedSurfaceFlags.ConfigOrMetadataSurface) + || HasUnusedSurface(surface, UnusedSurfaceFlags.ConfigOrMetadataMember); + + private static bool IsJsonContractSurface( + UnusedCandidateSymbol candidate, + UnusedSurfaceAnalysis surface) + { + return HasUnusedSurface(surface, UnusedSurfaceFlags.SerializationContract) + || HasUnusedSurface(surface, UnusedSurfaceFlags.ContractMember) + || HasUnusedSurface(surface, UnusedSurfaceFlags.SourceGeneratedJsonContext) + || ContainsAny(candidate.Path, UnusedContractPathSegments) + || EndsWithAny(candidate.Name, UnusedRecordContractSuffixes) + || EndsWithAny(candidate.ContainerName, UnusedRecordContractSuffixes) + || ContainsAny(candidate.Name, UnusedJsonContractTerms) + || ContainsAny(candidate.ContainerName, UnusedJsonContractTerms) + || ContainsAny(candidate.Signature, UnusedJsonSignatureTerms); + } + + private static bool IsPrivateLikeVisibility(string? visibility) + => string.Equals(visibility, "private", StringComparison.OrdinalIgnoreCase) + || string.Equals(visibility, "fileprivate", StringComparison.OrdinalIgnoreCase); +} diff --git a/src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.cs b/src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.cs index 41bd9d84fb..588bbe49a6 100644 --- a/src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.cs +++ b/src/CodeIndex/Database/DbSymbolReader.UnusedSymbols.cs @@ -126,41 +126,6 @@ FROM symbols {ancestorAlias} ), '')"; } - private const string UnusedBucketLikelyPrivate = "likely_unused_private"; - private const string UnusedBucketMaybeNonPublic = "maybe_unused_nonpublic"; - private const string UnusedBucketPublicOrExported = "public_or_exported_no_refs"; - private const string UnusedBucketReflectionOrConfig = "reflection_or_config_suspect"; - private const string UnusedContractDomainPrivate = "private_or_file_local"; - private const string UnusedContractDomainNonPublic = "nonpublic_internal"; - private const string UnusedContractDomainPublicApi = "public_api_surface"; - private const string UnusedContractDomainCli = "cli_contract"; - private const string UnusedContractDomainJson = "json_contract"; - private const string UnusedContractDomainMcp = "mcp_contract"; - private const string UnusedContractDomainLsp = "lsp_contract"; - private const string UnusedContractDomainConfig = "configuration_contract"; - private const string UnusedContractDomainSerialization = "serialization_or_reflection_contract"; - private const string UnusedContractDomainGenerated = "generated_code"; - private const string UnusedContractDomainDocumentation = "documentation_surface"; - private const string UnusedContractDomainTest = "test_contract"; - private const string UnusedContractDomainFrameworkOverride = "framework_override"; - private const string UnusedContractDomainExceptionDiagnostic = "exception_diagnostic"; - internal static readonly string[] OrderedUnusedContractDomains = - [ - UnusedContractDomainPrivate, - UnusedContractDomainNonPublic, - UnusedContractDomainPublicApi, - UnusedContractDomainCli, - UnusedContractDomainJson, - UnusedContractDomainMcp, - UnusedContractDomainLsp, - UnusedContractDomainConfig, - UnusedContractDomainSerialization, - UnusedContractDomainGenerated, - UnusedContractDomainDocumentation, - UnusedContractDomainTest, - UnusedContractDomainFrameworkOverride, - UnusedContractDomainExceptionDiagnostic, - ]; private static readonly HashSet ReflectionPropertyAttributeNames = new(StringComparer.Ordinal) { "jsonpropertyname", @@ -247,120 +212,6 @@ FROM symbols {ancestorAlias} "return", "type", }; - private static readonly string[] UnusedContractPathSegments = - [ - "/contracts/", - "/contract/", - "/dtos/", - "/dto/", - "/models/", - "/model/", - "/schemas/", - "/schema/", - ]; - private static readonly string[] UnusedRecordContractSuffixes = - [ - "Dto", - "DTO", - "Request", - "Response", - "Result", - "Results", - "Model", - "Payload", - "Envelope", - ]; - private static readonly string[] UnusedGeneratedPathMarkers = - [ - ".g.cs", - ".generated.", - "/generated/", - "/obj/", - "/bin/", - ]; - private static readonly string[] UnusedCliPathMarkers = ["/cli/", "/commands/", "/commandline/"]; - private static readonly string[] UnusedMcpPathMarkers = ["/mcp/"]; - private static readonly string[] UnusedLspPathMarkers = ["/lsp/", "/languageserver/"]; - private static readonly string[] UnusedJsonContractTerms = - [ - "Json", - "Dto", - "DTO", - "Request", - "Response", - "Result", - "Results", - "Payload", - "Envelope", - "Contract", - "Schema", - ]; - private static readonly HashSet UnusedFrameworkOverrideMemberNames = new(StringComparer.OrdinalIgnoreCase) - { - "CanRead", - "CanSeek", - "CanWrite", - "Length", - "Position", - "Flush", - "FlushAsync", - "Read", - "ReadAsync", - "ReadByte", - "Seek", - "SetLength", - "Write", - "WriteAsync", - "WriteByte", - "Dispose", - "DisposeAsync", - }; - private static readonly HashSet UnusedExceptionMetadataNames = new(StringComparer.OrdinalIgnoreCase) - { - "CharactersRead", - "Utf8BytesRead", - "SizeLimit", - "Limit", - "ByteCount", - "BytesRead", - "Offset", - "Position", - "Path", - "FileName", - "LineNumber", - "ColumnNumber", - "ActualSize", - "MaxSize", - "Length", - }; - private static readonly string[] UnusedConfigMemberTerms = - [ - "Configuration", - "Config", - "Options", - "Settings", - "Manifest", - "Schema", - "Metadata", - "Limit", - "Max", - "Min", - "Size", - "Bytes", - "Count", - "Capacity", - "Timeout", - "Version", - "Kind", - "Category", - "Severity", - "Source", - "Target", - "Path", - "Name", - "Id", - "Key", - ]; private const int UnusedAttributeContextWindow = 16; private const int UnusedPublicOverfetchMultiplier = 16; private const int UnusedPublicOverfetchMinimum = 64; @@ -377,64 +228,7 @@ FROM symbols {ancestorAlias} @"\{\s*(?:get|set|init)\b|=>", RegexOptions.Compiled | RegexOptions.CultureInvariant); - private sealed class UnusedCandidateSymbol - { - public long FileId { get; init; } - public string Path { get; init; } = string.Empty; - public string? Lang { get; init; } - public string Kind { get; init; } = string.Empty; - public string Name { get; init; } = string.Empty; - public int Line { get; init; } - public int StartLine { get; init; } - public int EndLine { get; init; } - public string? Signature { get; init; } - public string? Visibility { get; init; } - public string? ReturnType { get; init; } - public string? ContainerKind { get; init; } - public string? ContainerName { get; init; } - public string? ContainerQualifiedName { get; init; } - public bool IsPublicOrExported { get; init; } - public bool IsReflectionOrConfigSuspect { get; init; } - public int ProvisionalBucketOrder { get; init; } - } - private readonly record struct UnusedCandidateChunk(int StartLine, int EndLine, string Content); - private readonly record struct UnusedContractDomainClassification(string Domain, List Tags); - - - private string BuildAmbiguousCSharpEnumMemberExclusionSql( - string symbolAlias, - string fileAlias, - IReadOnlyList? pathPatterns, - IReadOnlyList? excludePathPatterns, - bool excludeTests) - { - var symbolContainerKindSql = GetSymbolColumnSql("container_kind", "''", symbolAlias); - var symbolContainerNameSql = GetSymbolColumnSql("container_name", "''", symbolAlias); - var symbolContainerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name", symbolContainerNameSql, symbolAlias); - var peerContainerKindSql = GetSymbolColumnSql("container_kind", "''", "s_peer"); - var peerContainerNameSql = GetSymbolColumnSql("container_name", "''", "s_peer"); - var peerContainerQualifiedNameSql = GetSymbolColumnSql("container_qualified_name", peerContainerNameSql, "s_peer"); - var peerPathFiltersSql = BuildPathFiltersSql("f_peer", pathPatterns, excludePathPatterns, excludeTests); - - return $@" - NOT ( - {fileAlias}.lang = 'csharp' - AND {symbolAlias}.kind = 'enum' - AND {symbolContainerKindSql} = 'enum' - AND EXISTS ( - SELECT 1 - FROM symbols s_peer - JOIN files f_peer ON f_peer.id = s_peer.file_id - WHERE f_peer.lang = 'csharp' - {peerPathFiltersSql} - AND s_peer.kind = 'enum' - AND {peerContainerKindSql} = 'enum' - AND s_peer.name = {symbolAlias}.name - AND {peerContainerQualifiedNameSql} <> {symbolContainerQualifiedNameSql} - ) - )"; - } /// /// Find symbols that have no matching references in the reference table (potential dead code). @@ -568,13 +362,14 @@ private List GetUnusedSymbolsWithoutSqlResolver(int limit, s UnusedPublicOverfetchMaximum); var publicOrExported = new List(targetCount); var fileContentByFileId = new Dictionary(); - var privateLike = CollectUnusedCandidateBucket(targetCount, batchSize, 0, fileContentByFileId, + var canInspectSameFileUse = CanInspectUnusedCandidateSameFileUse(); + var privateLike = CollectUnusedCandidateBucket(targetCount, batchSize, 0, fileContentByFileId, canInspectSameFileUse, kind, lang, pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters); - var maybeNonPublic = CollectUnusedCandidateBucket(targetCount, batchSize, 1, fileContentByFileId, + var maybeNonPublic = CollectUnusedCandidateBucket(targetCount, batchSize, 1, fileContentByFileId, canInspectSameFileUse, kind, lang, pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters); - var reflectionOrConfig = CollectUnusedCandidateBucket(targetCount, batchSize, 3, fileContentByFileId, + var reflectionOrConfig = CollectUnusedCandidateBucket(targetCount, batchSize, 3, fileContentByFileId, canInspectSameFileUse, kind, lang, pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters); - CollectPublicUnusedCandidateBucket(targetCount, batchSize, publicFetchBudget, fileContentByFileId, + CollectPublicUnusedCandidateBucket(targetCount, batchSize, publicFetchBudget, fileContentByFileId, canInspectSameFileUse, publicOrExported, reflectionOrConfig, kind, lang, pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters); var merged = new List(privateLike.Count + maybeNonPublic.Count + publicOrExported.Count + reflectionOrConfig.Count); @@ -596,6 +391,7 @@ private List GetFilteredUnusedSymbolsWithoutSqlResolver(int return []; var fileContentByFileId = new Dictionary(); + var canInspectSameFileUse = CanInspectUnusedCandidateSameFileUse(); var resultsByBucket = CreateUnusedBucketResultLists(); const int batchSize = UnusedPublicOverfetchMaximum; foreach (var provisionalBucket in GetRelevantUnusedProvisionalBuckets(targetBuckets)) @@ -604,14 +400,14 @@ private List GetFilteredUnusedSymbolsWithoutSqlResolver(int while (!AllTargetUnusedBucketsFilled(resultsByBucket, targetBuckets, targetCount)) { var batch = FetchUnusedCandidateSymbols(batchSize, offset, provisionalBucket, kind, lang, - pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters).ToList(); + pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters); if (batch.Count == 0) break; offset += batch.Count; foreach (var candidate in batch) { - if (HasPrivateCSharpUse(candidate, fileContentByFileId)) + if (HasSameFilePrivateUse(candidate, fileContentByFileId, canInspectSameFileUse)) continue; var result = CreateUnusedSymbolResult(candidate); @@ -630,7 +426,7 @@ private List GetFilteredUnusedSymbolsWithoutSqlResolver(int } private List CollectUnusedCandidateBucket(int targetCount, int batchSize, int provisionalBucketOrder, - Dictionary fileContentByFileId, string? kind, string? lang, + Dictionary fileContentByFileId, bool canInspectSameFileUse, string? kind, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, IReadOnlyList? visibilityFilters, IReadOnlyList? excludeVisibilityFilters) { @@ -639,14 +435,14 @@ private List CollectUnusedCandidateBucket(int targetCount, i while (results.Count < targetCount) { var batch = FetchUnusedCandidateSymbols(batchSize, offset, provisionalBucketOrder, kind, lang, - pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters).ToList(); + pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters); if (batch.Count == 0) break; offset += batch.Count; foreach (var candidate in batch) { - if (HasPrivateCSharpUse(candidate, fileContentByFileId)) + if (HasSameFilePrivateUse(candidate, fileContentByFileId, canInspectSameFileUse)) continue; results.Add(CreateUnusedSymbolResult(candidate)); @@ -662,7 +458,7 @@ private List CollectUnusedCandidateBucket(int targetCount, i } private void CollectPublicUnusedCandidateBucket(int targetCount, int batchSize, int candidateBudget, - Dictionary fileContentByFileId, + Dictionary fileContentByFileId, bool canInspectSameFileUse, List publicOrExported, List reflectionOrConfig, string? kind, string? lang, IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, IReadOnlyList? visibilityFilters, IReadOnlyList? excludeVisibilityFilters) @@ -673,14 +469,14 @@ private void CollectPublicUnusedCandidateBucket(int targetCount, int batchSize, && candidatesFetched < candidateBudget) { var batch = FetchUnusedCandidateSymbols(batchSize, offset, 2, kind, lang, - pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters).ToList(); + pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters); if (batch.Count == 0) break; offset += batch.Count; foreach (var candidate in batch) { - if (HasPrivateCSharpUse(candidate, fileContentByFileId)) + if (HasSameFilePrivateUse(candidate, fileContentByFileId, canInspectSameFileUse)) continue; candidatesFetched++; @@ -705,17 +501,15 @@ private void CollectPublicUnusedCandidateBucket(int targetCount, int batchSize, } } - private bool HasPrivateCSharpUse(UnusedCandidateSymbol candidate, Dictionary fileContentByFileId) - => HasSameFilePrivateUse(candidate, fileContentByFileId) - || HasCSharpPartialContainingTypeUse(candidate); - - private bool HasSameFilePrivateUse(UnusedCandidateSymbol candidate, Dictionary fileContentByFileId) + private bool HasSameFilePrivateUse( + UnusedCandidateSymbol candidate, + Dictionary fileContentByFileId, + bool canInspectSameFileUse) { - if (!string.Equals(candidate.Lang, "csharp", StringComparison.Ordinal) + if (!canInspectSameFileUse + || !string.Equals(candidate.Lang, "csharp", StringComparison.Ordinal) || !IsPrivateLikeVisibility(candidate.Visibility) - || candidate.Name.Length == 0 - || !_hasChunksTable - || !HasTable("chunks")) + || candidate.Name.Length == 0) return false; var fileContent = GetUnusedCandidateFileContent(candidate.FileId, fileContentByFileId); @@ -726,76 +520,9 @@ private bool HasSameFilePrivateUse(UnusedCandidateSymbol candidate, Dictionary own_type.file_id - AND peer_type.kind = own_type.kind - AND peer_type.name = own_type.name - JOIN files peer_file ON peer_file.id = peer_type.file_id - JOIN chunks peer_chunk ON peer_chunk.file_id = peer_type.file_id - WHERE own_type.file_id = @fileId - AND own_type.kind = @containerKind - AND own_type.name = @containerName - AND lower({ownSignatureSql}) LIKE '%partial%' - AND lower({peerSignatureSql}) LIKE '%partial%' - AND peer_file.lang = 'csharp' - AND ( - @containerQualifiedName = '' - OR @containerQualifiedName = own_type.name - OR @containerQualifiedName = {ownQualifiedNameSql} - ) - AND ( - @containerQualifiedName = '' - OR @containerQualifiedName = peer_type.name - OR @containerQualifiedName = {peerQualifiedNameSql} - ) - AND {ownTypeShapeSql} = {peerTypeShapeSql} - AND peer_chunk.end_line >= {peerTypeStartLineSql} - AND peer_chunk.start_line <= {peerTypeEndLineSql} - AND csharp_identifier_occurrence_count_in_line_range( - peer_chunk.content, - peer_chunk.start_line, - {peerTypeStartLineSql}, - {peerTypeEndLineSql}, - @symbolName) > 0 - LIMIT 1"; - SqliteCommandPolicy.Add(cmd, "@fileId", candidate.FileId); - SqliteCommandPolicy.Add(cmd, "@containerKind", candidate.ContainerKind); - SqliteCommandPolicy.Add(cmd, "@containerName", candidate.ContainerName); - SqliteCommandPolicy.Add(cmd, "@containerQualifiedName", candidate.ContainerQualifiedName ?? string.Empty); - SqliteCommandPolicy.Add(cmd, "@symbolName", candidate.Name); - - using var reader = cmd.ExecuteTrackedReader(); - return reader.TrackedRead(); - } - - private static bool IsCSharpPartialContainerKind(string? kind) - => kind is "class" or "struct" or "interface"; + private bool CanInspectUnusedCandidateSameFileUse() + => HasUsableUnusedChunks() + && _chunkColumns.Contains("chunk_index"); private string GetUnusedCandidateFileContent(long fileId, Dictionary fileContentByFileId) { @@ -864,163 +591,6 @@ private static void AddUnusedCandidateChunkLines(SortedDictionary l } } - private IEnumerable FetchUnusedCandidateSymbols(int fetchLimit, int offset, int provisionalBucketOrder, string? kind, string? lang, - IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, IReadOnlyList? visibilityFilters = null, IReadOnlyList? excludeVisibilityFilters = null) - { - var graphLangs = GetWorkspaceSupportedReferenceLanguages() - .Where(value => !IsSqlLanguage(value)) - .ToList(); - var visibilitySql = $"lower({GetSymbolColumnSql("visibility", "''")})"; - var signatureSql = $"lower({GetSymbolColumnSql("signature", "''")})"; - const string pathSql = "lower(f.path)"; - var isPublicOrExportedSql = $"{visibilitySql} IN ('public', 'open', 'pub', 'export')"; - var hasConfigContextSql = $@"( - {pathSql} LIKE 'config/%' - OR {pathSql} LIKE '%/config/%' - OR {pathSql} LIKE 'settings/%' - OR {pathSql} LIKE '%/settings/%' - OR {pathSql} LIKE 'options/%' - OR {pathSql} LIKE '%/options/%' - OR {signatureSql} LIKE '%iconfiguration%' - OR {signatureSql} LIKE '%configurationsection%' - OR {signatureSql} LIKE '%ioptions%' - OR {signatureSql} LIKE '%options<%' - )"; - var isReflectionOrConfigSuspectSql = $@"( - {isPublicOrExportedSql} - AND s.kind = 'property' - AND {hasConfigContextSql} - )"; - var provisionalBucketOrderSql = $@" - CASE - WHEN {isReflectionOrConfigSuspectSql} THEN 3 - WHEN {isPublicOrExportedSql} THEN 2 - WHEN {visibilitySql} IN ('private', 'fileprivate') THEN 0 - ELSE 1 - END"; - - var sql = $@" - SELECT s.file_id, f.path, f.lang, s.kind, s.name, s.line, - {GetSymbolColumnSql("start_line", "s.line")} AS start_line, - {GetSymbolColumnSql("end_line", "s.line")} AS end_line, - {GetSymbolColumnSql("signature")} AS signature, - {GetSymbolColumnSql("visibility")} AS visibility, - {GetSymbolColumnSql("return_type")} AS return_type, - {GetSymbolColumnSql("container_kind")} AS container_kind, - {GetSymbolColumnSql("container_name")} AS container_name, - {GetSymbolColumnSql("container_qualified_name", GetSymbolColumnSql("container_name", "''"))} AS container_qualified_name, - CASE WHEN {isPublicOrExportedSql} THEN 1 ELSE 0 END AS is_public_or_exported, - CASE WHEN {isReflectionOrConfigSuspectSql} THEN 1 ELSE 0 END AS is_reflection_or_config_suspect, - {provisionalBucketOrderSql} AS provisional_bucket_order - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE s.kind NOT IN ('import', 'namespace')"; - sql += $"\n AND {BuildAmbiguousCSharpEnumMemberExclusionSql("s", "f", pathPatterns, excludePathPatterns, excludeTests)}"; - sql += """ - AND NOT EXISTS ( - SELECT 1 - FROM symbol_references sr - WHERE sr.symbol_name IS NOT NULL - AND sr.symbol_name <> '' - AND sr.symbol_name = s.name - ) - """; - - if (lang != null) - sql += SymbolLanguageFileIdFilter; - else - sql += $" AND f.lang IN ({string.Join(",", graphLangs.Select((_, i) => $"@gl{i}"))})"; - - if (kind != null) - sql += " AND s.kind = @kind"; - - sql += " AND (" + provisionalBucketOrderSql + ") = @provisionalBucketOrder"; - AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); - AppendVisibilityFilters(ref sql, visibilityFilters, excludeVisibilityFilters); - sql += " ORDER BY f.path, s.line, s.name"; - sql += " LIMIT @limit OFFSET @offset"; - - using var cmd = _conn.CreateCommand(); - cmd.CommandText = sql; - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - else - { - for (int i = 0; i < graphLangs.Count; i++) - SqliteCommandPolicy.Add(cmd, $"@gl{i}", graphLangs[i]); - } - if (kind != null) - SqliteCommandPolicy.Add(cmd, "@kind", kind); - SqliteCommandPolicy.Add(cmd, "@provisionalBucketOrder", provisionalBucketOrder); - SqliteCommandPolicy.Add(cmd, "@limit", fetchLimit); - SqliteCommandPolicy.Add(cmd, "@offset", offset); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - AddVisibilityFilterParameters(cmd, visibilityFilters, excludeVisibilityFilters); - - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - yield return new UnusedCandidateSymbol - { - FileId = reader.GetInt64(0), - Path = reader.GetString(1), - Lang = GetNullableString(reader, 2), - Kind = reader.GetString(3), - Name = reader.GetString(4), - Line = reader.GetInt32(5), - StartLine = GetInt32OrFallback(reader, 6, 5), - EndLine = GetInt32OrFallback(reader, 7, 5), - Signature = GetNullableString(reader, 8), - Visibility = GetNullableString(reader, 9), - ReturnType = GetNullableString(reader, 10), - ContainerKind = GetNullableString(reader, 11), - ContainerName = GetNullableString(reader, 12), - ContainerQualifiedName = GetNullableString(reader, 13), - IsPublicOrExported = reader.GetInt32(14) != 0, - IsReflectionOrConfigSuspect = reader.GetInt32(15) != 0, - ProvisionalBucketOrder = reader.GetInt32(16), - }; - } - } - - private UnusedSymbolResult CreateUnusedSymbolResult(UnusedCandidateSymbol candidate) - { - var kind = NormalizeUnusedSymbolKind(candidate); - var surfaceTags = BuildUnusedIntentionalSurfaceTags(candidate, kind); - if (!surfaceTags.Contains("reflection_or_config_suspect", StringComparer.Ordinal) - && candidate.IsPublicOrExported - && HasReflectionAttributeContext(kind, candidate.Path, candidate.StartLine)) - { - AddUnusedSurfaceTag(surfaceTags, "reflection_or_config_suspect"); - } - - var isIntentionalSurfaceSuspect = surfaceTags.Count > 0; - var classification = ClassifyUnusedSymbol(candidate.IsPublicOrExported, isIntentionalSurfaceSuspect, candidate.Visibility); - var reasonTags = BuildUnusedReasonTags(candidate.IsPublicOrExported, isIntentionalSurfaceSuspect, candidate.Visibility, surfaceTags); - var contractDomain = ClassifyUnusedContractDomain(candidate, kind, classification.Bucket, surfaceTags); - return new UnusedSymbolResult - { - Path = candidate.Path, - Lang = candidate.Lang, - Kind = kind, - Name = candidate.Name, - Line = candidate.Line, - StartLine = candidate.StartLine, - EndLine = candidate.EndLine, - Signature = candidate.Signature, - Visibility = candidate.Visibility, - ReturnType = candidate.ReturnType, - ContainerKind = candidate.ContainerKind, - ContainerName = candidate.ContainerName, - UnusedBucket = classification.Bucket, - UnusedConfidence = classification.Confidence, - UnusedReason = classification.Reason, - UnusedReasonTags = reasonTags, - UnusedContractDomain = contractDomain.Domain, - UnusedContractDomainTags = contractDomain.Tags, - }; - } - private static string NormalizeUnusedSymbolKind(UnusedCandidateSymbol candidate) => NormalizeUnusedSymbolKind(candidate.Lang, candidate.Kind, candidate.Signature); @@ -1247,157 +817,6 @@ private static bool StartsWithCSharpWhereConstraint(string text, int index) return end >= text.Length || char.IsWhiteSpace(text[end]); } - private List FetchUnusedCandidates(int fetchLimit, int provisionalBucketOrder, int offset, string? kind, string? lang, - IReadOnlyList? pathPatterns, IReadOnlyList? excludePathPatterns, bool excludeTests, IReadOnlyList? visibilityFilters = null, IReadOnlyList? excludeVisibilityFilters = null) - { - var graphLangs = GetWorkspaceSupportedReferenceLanguages(); - var visibilitySql = $"lower({GetSymbolColumnSql("visibility", "''")})"; - var signatureSql = $"lower({GetSymbolColumnSql("signature", "''")})"; - const string pathSql = "lower(f.path)"; - var isPublicOrExportedSql = $"{visibilitySql} IN ('public', 'open', 'pub', 'export')"; - var hasConfigContextSql = $@"( - {pathSql} LIKE 'config/%' - OR {pathSql} LIKE '%/config/%' - OR {pathSql} LIKE 'settings/%' - OR {pathSql} LIKE '%/settings/%' - OR {pathSql} LIKE 'options/%' - OR {pathSql} LIKE '%/options/%' - OR {signatureSql} LIKE '%iconfiguration%' - OR {signatureSql} LIKE '%configurationsection%' - OR {signatureSql} LIKE '%ioptions%' - OR {signatureSql} LIKE '%options<%' - )"; - var isReflectionOrConfigSuspectSql = $@"( - {isPublicOrExportedSql} - AND s.kind = 'property' - AND {hasConfigContextSql} - )"; - var provisionalBucketOrderSql = $@" - CASE - WHEN {isReflectionOrConfigSuspectSql} THEN 3 - WHEN {isPublicOrExportedSql} THEN 2 - WHEN {visibilitySql} IN ('private', 'fileprivate') THEN 0 - ELSE 1 - END"; - - var sql = $@" - WITH unused_candidates AS ( - SELECT s.file_id, f.path, f.lang, s.kind, s.name, s.line, - {GetSymbolColumnSql("start_line", "s.line")} AS start_line, - {GetSymbolColumnSql("end_line", "s.line")} AS end_line, - {GetSymbolColumnSql("signature")} AS signature, - {GetSymbolColumnSql("visibility")} AS visibility, - {GetSymbolColumnSql("return_type")} AS return_type, - {GetSymbolColumnSql("container_kind")} AS container_kind, - {GetSymbolColumnSql("container_name")} AS container_name, - {GetSymbolColumnSql("container_qualified_name", GetSymbolColumnSql("container_name", "''"))} AS container_qualified_name, - CASE WHEN {isPublicOrExportedSql} THEN 1 ELSE 0 END AS is_public_or_exported, - CASE WHEN {isReflectionOrConfigSuspectSql} THEN 1 ELSE 0 END AS is_reflection_or_config_suspect, - {provisionalBucketOrderSql} AS provisional_bucket_order - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE s.kind NOT IN ('import', 'namespace') - AND NOT EXISTS ( - SELECT 1 - FROM symbol_references sr - JOIN files rf ON rf.id = sr.file_id" + ReferenceLineJoinSql("sr") + @" - WHERE sr.symbol_name = s.name - OR (f.lang = 'sql' AND rf.lang = 'sql' AND ( - (sql_resolve_reference_segment_count_at(sr.symbol_name, " + ReferenceContextSql("sr") + @", sr.container_name, sr.column_number) = sql_segment_count(s.name) - AND sql_reference_matches_target_at(sr.symbol_name, " + ReferenceContextSql("sr") + @", sr.container_name, sr.column_number, s.name) = 1) - OR (sql_segment_count(sr.symbol_name) = 1 - AND sql_allow_leaf_fallback_at(sr.symbol_name, " + ReferenceContextSql("sr") + @", sr.container_name, sr.column_number) = 1 - AND sr.symbol_name = sql_leaf_name(s.name) COLLATE NOCASE - AND NOT EXISTS ( - SELECT 1 - FROM symbols s_exact - JOIN files f_exact ON f_exact.id = s_exact.file_id - WHERE f_exact.lang = 'sql' - AND sql_segment_count(s_exact.name) = sql_resolve_reference_segment_count_at(sr.symbol_name, " + ReferenceContextSql("sr") + @", sr.container_name, sr.column_number) - AND sql_reference_matches_target_at(sr.symbol_name, " + ReferenceContextSql("sr") + @", sr.container_name, sr.column_number, s_exact.name) = 1 - )) - )) - )"; - if (_hasChunksTable && HasTable("chunks")) - { - sql += BuildSameFilePrivateUseExclusionSql( - "s", - "f", - visibilitySql, - GetSymbolColumnSql("start_line", "s.line"), - GetSymbolColumnSql("end_line", "s.line")); - sql += BuildCSharpPartialContainingTypeUseExclusionSql("s", "f", visibilitySql); - } - sql += $"\n AND {BuildAmbiguousCSharpEnumMemberExclusionSql("s", "f", pathPatterns, excludePathPatterns, excludeTests)}"; - - if (lang != null) - sql += SymbolLanguageFileIdFilter; - else - sql += $" AND f.lang IN ({string.Join(",", graphLangs.Select((_, i) => $"@gl{i}"))})"; - - if (kind != null) - sql += " AND s.kind = @kind"; - - AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); - AppendVisibilityFilters(ref sql, visibilityFilters, excludeVisibilityFilters); - sql += @" - ) - SELECT file_id, path, lang, kind, name, line, start_line, end_line, signature, visibility, - return_type, container_kind, container_name, container_qualified_name, - is_public_or_exported, is_reflection_or_config_suspect, provisional_bucket_order - FROM unused_candidates - WHERE provisional_bucket_order = @bucketOrder - ORDER BY path, line, name - LIMIT @limit OFFSET @offset"; - - using var cmd = _conn.CreateCommand(); - cmd.CommandText = sql; - SqliteCommandPolicy.Add(cmd, "@bucketOrder", provisionalBucketOrder); - SqliteCommandPolicy.Add(cmd, "@limit", fetchLimit); - SqliteCommandPolicy.Add(cmd, "@offset", offset); - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - else - { - var langList = graphLangs.ToList(); - for (int i = 0; i < langList.Count; i++) - SqliteCommandPolicy.Add(cmd, $"@gl{i}", langList[i]); - } - if (kind != null) - SqliteCommandPolicy.Add(cmd, "@kind", kind); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - AddVisibilityFilterParameters(cmd, visibilityFilters, excludeVisibilityFilters); - - var results = new List(); - using var reader = cmd.ExecuteTrackedReader(); - while (reader.TrackedRead()) - { - var candidate = new UnusedCandidateSymbol - { - FileId = reader.GetInt64(0), - Path = reader.GetString(1), - Lang = GetNullableString(reader, 2), - Kind = reader.GetString(3), - Name = reader.GetString(4), - Line = reader.GetInt32(5), - StartLine = GetInt32OrFallback(reader, 6, 5), - EndLine = GetInt32OrFallback(reader, 7, 5), - Signature = GetNullableString(reader, 8), - Visibility = GetNullableString(reader, 9), - ReturnType = GetNullableString(reader, 10), - ContainerKind = GetNullableString(reader, 11), - ContainerName = GetNullableString(reader, 12), - ContainerQualifiedName = GetNullableString(reader, 13), - IsPublicOrExported = reader.GetInt32(14) != 0, - IsReflectionOrConfigSuspect = reader.GetInt32(15) != 0, - ProvisionalBucketOrder = reader.GetInt32(16), - }; - results.Add(CreateUnusedSymbolResult(candidate)); - } - - return results; - } - private static List DiversifyUnusedResults(List results, int limit) { if (results.Count == 0 || limit <= 0) @@ -1617,6 +1036,7 @@ private UnusedCountResult CountUnusedSymbolsDetailedWithoutSqlResolver(string? k var confidenceCounts = new Dictionary(StringComparer.Ordinal); var contractDomainCounts = new Dictionary(StringComparer.Ordinal); var fileContentByFileId = new Dictionary(); + var canInspectSameFileUse = CanInspectUnusedCandidateSameFileUse(); const int batchSize = UnusedPublicOverfetchMaximum; for (var bucket = 0; bucket <= 3; bucket++) { @@ -1624,14 +1044,14 @@ private UnusedCountResult CountUnusedSymbolsDetailedWithoutSqlResolver(string? k while (true) { var batch = FetchUnusedCandidateSymbols(batchSize, offset, bucket, kind, lang, - pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters).ToList(); + pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters); if (batch.Count == 0) break; offset += batch.Count; foreach (var candidate in batch) { - if (HasPrivateCSharpUse(candidate, fileContentByFileId)) + if (HasSameFilePrivateUse(candidate, fileContentByFileId, canInspectSameFileUse)) continue; var result = CreateUnusedSymbolResult(candidate); @@ -1764,82 +1184,14 @@ public QueryCountResult CountUnusedSymbols(string? kind, string? lang, IReadOnly if (!ScopeMayIncludeSqlSymbols(kind, lang, pathPatterns, excludePathPatterns, excludeTests)) return CountUnusedSymbolsWithoutSqlResolver(kind, lang, pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters); - var graphLangs = GetWorkspaceSupportedReferenceLanguages(); - using var cmd = _conn.CreateCommand(); - var referenceLineJoin = ReferenceLineJoinSql("sr"); - var contextSql = ReferenceContextSql("sr"); - var sql = @" - SELECT COUNT(*), COUNT(DISTINCT f.path), MAX(CASE WHEN f.lang = 'sql' THEN 1 ELSE 0 END) - FROM symbols s - JOIN files f ON s.file_id = f.id - WHERE s.kind NOT IN ('import', 'namespace') - AND NOT EXISTS ( - SELECT 1 - FROM symbol_references sr - JOIN files rf ON rf.id = sr.file_id" + referenceLineJoin + @" - WHERE sr.symbol_name = s.name - OR (f.lang = 'sql' AND rf.lang = 'sql' AND ( - (sql_resolve_reference_segment_count_at(sr.symbol_name, " + contextSql + @", sr.container_name, sr.column_number) = sql_segment_count(s.name) - AND sql_reference_matches_target_at(sr.symbol_name, " + contextSql + @", sr.container_name, sr.column_number, s.name) = 1) - OR (sql_segment_count(sr.symbol_name) = 1 - AND sql_allow_leaf_fallback_at(sr.symbol_name, " + contextSql + @", sr.container_name, sr.column_number) = 1 - AND sr.symbol_name = sql_leaf_name(s.name) COLLATE NOCASE - AND NOT EXISTS ( - SELECT 1 - FROM symbols s_exact - JOIN files f_exact ON f_exact.id = s_exact.file_id - WHERE f_exact.lang = 'sql' - AND sql_segment_count(s_exact.name) = sql_resolve_reference_segment_count_at(sr.symbol_name, " + contextSql + @", sr.container_name, sr.column_number) - AND sql_reference_matches_target_at(sr.symbol_name, " + contextSql + @", sr.container_name, sr.column_number, s_exact.name) = 1 - )) - )) - )"; - if (_hasChunksTable && HasTable("chunks")) - { - sql += BuildSameFilePrivateUseExclusionSql( - "s", - "f", - $"lower({GetSymbolColumnSql("visibility", "''")})", - GetSymbolColumnSql("start_line", "s.line"), - GetSymbolColumnSql("end_line", "s.line")); - sql += BuildCSharpPartialContainingTypeUseExclusionSql( - "s", - "f", - $"lower({GetSymbolColumnSql("visibility", "''")})"); - } - sql += $"\n AND {BuildAmbiguousCSharpEnumMemberExclusionSql("s", "f", pathPatterns, excludePathPatterns, excludeTests)}"; - - if (lang != null) - sql += SymbolLanguageFileIdFilter; - else - sql += $" AND f.lang IN ({string.Join(",", graphLangs.Select((_, i) => $"@gl{i}"))})"; - - if (kind != null) - sql += " AND s.kind = @kind"; - - AppendPathFilters(ref sql, pathPatterns, excludePathPatterns, excludeTests); - AppendVisibilityFilters(ref sql, visibilityFilters, excludeVisibilityFilters); - cmd.CommandText = sql; - if (lang != null) - SqliteCommandPolicy.Add(cmd, "@lang", lang); - else - { - var langList = graphLangs.ToList(); - for (int i = 0; i < langList.Count; i++) - SqliteCommandPolicy.Add(cmd, $"@gl{i}", langList[i]); - } - if (kind != null) - SqliteCommandPolicy.Add(cmd, "@kind", kind); - AddPathFilterParameters(cmd, pathPatterns, excludePathPatterns); - AddVisibilityFilterParameters(cmd, visibilityFilters, excludeVisibilityFilters); - - using var reader = cmd.ExecuteTrackedReader(); - if (!reader.TrackedRead()) - return new QueryCountResult(0, 0); - return new QueryCountResult( - reader.GetInt32(0), - reader.GetInt32(1), - reader.FieldCount > 2 && !reader.IsDBNull(2) && Convert.ToInt32(reader.GetValue(2)) != 0); + return CountUnusedCandidates(new UnusedCandidateScope( + kind, + lang, + pathPatterns, + excludePathPatterns, + excludeTests, + visibilityFilters, + excludeVisibilityFilters)); } private QueryCountResult CountFilteredUnusedSymbols(string? kind, string? lang, @@ -1887,6 +1239,7 @@ private QueryCountResult CountUnusedSymbolsWithoutSqlResolver(string? kind, stri var count = 0; var paths = new HashSet(StringComparer.Ordinal); var fileContentByFileId = new Dictionary(); + var canInspectSameFileUse = CanInspectUnusedCandidateSameFileUse(); const int batchSize = UnusedPublicOverfetchMaximum; for (var bucket = 0; bucket <= 3; bucket++) { @@ -1894,14 +1247,14 @@ private QueryCountResult CountUnusedSymbolsWithoutSqlResolver(string? kind, stri while (true) { var batch = FetchUnusedCandidateSymbols(batchSize, offset, bucket, kind, lang, - pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters).ToList(); + pathPatterns, excludePathPatterns, excludeTests, visibilityFilters, excludeVisibilityFilters); if (batch.Count == 0) break; offset += batch.Count; foreach (var candidate in batch) { - if (HasPrivateCSharpUse(candidate, fileContentByFileId)) + if (HasSameFilePrivateUse(candidate, fileContentByFileId, canInspectSameFileUse)) continue; count++; @@ -2782,424 +2135,4 @@ private static bool LooksLikeAttributeBoundaryLine(string line) return line.IndexOf('[') >= 0 || line.IndexOf(']') >= 0; } - private static List BuildUnusedIntentionalSurfaceTags(UnusedCandidateSymbol candidate, string kind) - { - var tags = new List(); - if (candidate.IsReflectionOrConfigSuspect) - AddUnusedSurfaceTag(tags, "reflection_or_config_suspect"); - if (IsMarkdownHeadingSymbol(candidate, kind)) - AddUnusedSurfaceTag(tags, "documentation_heading"); - if (IsMarkdownFenceSymbol(candidate, kind)) - AddUnusedSurfaceTag(tags, "markdown_fence_language_marker"); - if (IsGeneratedSurface(candidate)) - AddUnusedSurfaceTag(tags, "generated_surface"); - if (IsSourceGeneratedJsonContext(candidate)) - { - AddUnusedSurfaceTag(tags, "serialization_contract"); - AddUnusedSurfaceTag(tags, "source_generated_json_context"); - } - if (IsUnusedContractType(candidate, kind)) - AddUnusedSurfaceTag(tags, "serialization_contract"); - if (IsUnusedContractMember(candidate, kind)) - AddUnusedSurfaceTag(tags, "contract_member"); - if (IsConfigOrManifestSurface(candidate, kind)) - AddUnusedSurfaceTag(tags, "config_or_metadata_surface"); - if (IsTestHookName(candidate.Name)) - AddUnusedSurfaceTag(tags, "test_hook"); - if (IsExceptionMetadataProperty(candidate, kind)) - AddUnusedSurfaceTag(tags, "exception_metadata"); - if (IsConfigOrMetadataMember(candidate, kind)) - AddUnusedSurfaceTag(tags, "config_or_metadata_member"); - return tags; - } - - private static void AddUnusedSurfaceTag(List tags, string tag) - { - if (!tags.Contains(tag, StringComparer.Ordinal)) - tags.Add(tag); - } - - private static bool IsMarkdownHeadingSymbol(UnusedCandidateSymbol candidate, string kind) - { - return (string.Equals(candidate.Lang, "markdown", StringComparison.OrdinalIgnoreCase) - || string.Equals(candidate.Lang, "md", StringComparison.OrdinalIgnoreCase)) - && (kind.Contains("heading", StringComparison.OrdinalIgnoreCase) - || kind.Contains("header", StringComparison.OrdinalIgnoreCase)); - } - - private static bool IsMarkdownFenceSymbol(UnusedCandidateSymbol candidate, string kind) - { - if ((!string.Equals(candidate.Lang, "markdown", StringComparison.OrdinalIgnoreCase) - && !string.Equals(candidate.Lang, "md", StringComparison.OrdinalIgnoreCase)) - || !string.Equals(kind, "code", StringComparison.OrdinalIgnoreCase)) - { - return false; - } - - var signature = candidate.Signature?.TrimStart(); - return signature?.StartsWith("```", StringComparison.Ordinal) == true - || signature?.StartsWith("~~~", StringComparison.Ordinal) == true; - } - - private static bool IsGeneratedSurface(UnusedCandidateSymbol candidate) - => ContainsAny(candidate.Path, UnusedGeneratedPathMarkers); - - private static bool IsSourceGeneratedJsonContext(UnusedCandidateSymbol candidate) - { - return EndsWithAny(candidate.Name, ["JsonContext"]) - || ContainsAny(candidate.Signature, ["JsonSerializerContext", "JsonSerializable", "JsonSourceGenerationOptions"]); - } - - private static bool IsUnusedContractType(UnusedCandidateSymbol candidate, string kind) - { - if (!IsTypeLikeUnusedKind(kind) || IsPrivateLikeVisibility(candidate.Visibility)) - return false; - return ContainsAny(candidate.Path, UnusedContractPathSegments) - || IsRecordContractType(candidate) - || ContainsAny(candidate.Signature, ["DataContract", "Serializable", "MessagePackObject", "ProtoContract"]); - } - - private static bool IsUnusedContractMember(UnusedCandidateSymbol candidate, string kind) - { - if (!IsDataMemberUnusedKind(kind) || IsPrivateLikeVisibility(candidate.Visibility)) - return false; - return ContainsAny(candidate.Path, UnusedContractPathSegments) - || ContainsAny(candidate.Signature, ["JsonProperty", "JsonInclude", "DataMember", "XmlElement", "XmlAttribute"]); - } - - private static bool IsConfigOrManifestSurface(UnusedCandidateSymbol candidate, string kind) - { - if (IsPrivateLikeVisibility(candidate.Visibility)) - return false; - if (!IsTypeLikeUnusedKind(kind) && !IsDataMemberUnusedKind(kind) && !IsFunctionLikeUnusedKind(kind)) - return false; - if (EndsWithAny(candidate.Name, ["Options", "Settings", "Configuration", "Config", "Manifest", "Schema"]) - || ContainsAny(candidate.Name, ["Configuration", "IOptions"]) - || ContainsAny(candidate.Signature, ["IConfiguration", "ConfigurationSection", "IOptions", "Options<"])) - { - return true; - } - - return IsDataMemberUnusedKind(kind) - && (ContainsAny(candidate.Path, ["/config/", "/configuration/", "/options/", "/settings/", "/manifest/", "/manifests/"]) - || ContainsAny(candidate.Signature, ["IConfiguration", "ConfigurationSection", "IOptions", "Options<"])); - } - - private static bool IsTestHookName(string? name) - { - return !string.IsNullOrWhiteSpace(name) - && (name.EndsWith("ForTests", StringComparison.OrdinalIgnoreCase) - || name.Contains("ForTest", StringComparison.OrdinalIgnoreCase) - || name.Contains("TestOnly", StringComparison.OrdinalIgnoreCase)); - } - - private static bool IsExceptionMetadataProperty(UnusedCandidateSymbol candidate, string kind) - { - return string.Equals(kind, "property", StringComparison.OrdinalIgnoreCase) - && UnusedExceptionMetadataNames.Contains(candidate.Name) - && (EndsWithAny(candidate.ContainerName, ["Exception"]) - || ContainsAny(candidate.ContainerQualifiedName, ["Exception"]) - || ContainsAny(candidate.Signature, ["Exception"])); - } - - private static bool IsConfigOrMetadataMember(UnusedCandidateSymbol candidate, string kind) - { - if (!IsDataMemberUnusedKind(kind) || IsPrivateLikeVisibility(candidate.Visibility)) - return false; - if (!ContainsAny(candidate.Name, UnusedConfigMemberTerms)) - return false; - - var hasContext = ContainsAny(candidate.Path, UnusedContractPathSegments) - || ContainsAny(candidate.Path, ["/config/", "/configuration/", "/options/", "/settings/", "/manifest/", "/manifests/"]) - || ContainsAny(candidate.Signature, ["JsonProperty", "JsonInclude", "DataMember", "Diagnostic", "Metadata", "IConfiguration", "IOptions", "Options<"]); - return hasContext; - } - - private static bool IsTypeLikeUnusedKind(string kind) - { - return kind is "class" or "struct" or "record" or "interface" or "enum" or "type"; - } - - private static bool IsDataMemberUnusedKind(string kind) - { - return kind is "property" or "field" or "constant" or "enum_member"; - } - - private static bool IsFunctionLikeUnusedKind(string kind) - { - return kind is "function" or "method"; - } - - private static bool IsRecordContractType(UnusedCandidateSymbol candidate) - { - return ContainsCSharpRecordKeyword(candidate.Signature) - && EndsWithAny(candidate.Name, UnusedRecordContractSuffixes); - } - - private static bool ContainsCSharpRecordKeyword(string? signature) - { - if (string.IsNullOrWhiteSpace(signature)) - return false; - - var trimmed = signature.TrimStart(); - return trimmed.StartsWith("record ", StringComparison.Ordinal) - || trimmed.StartsWith("record(", StringComparison.Ordinal) - || signature.Contains(" record ", StringComparison.Ordinal) - || signature.Contains(" record(", StringComparison.Ordinal); - } - - private static bool EndsWithAny(string? value, IReadOnlyList suffixes) - { - if (string.IsNullOrWhiteSpace(value)) - return false; - foreach (var suffix in suffixes) - { - if (value.EndsWith(suffix, StringComparison.OrdinalIgnoreCase)) - return true; - } - - return false; - } - - private static bool ContainsAny(string? value, IReadOnlyList fragments) - { - if (string.IsNullOrWhiteSpace(value)) - return false; - foreach (var fragment in fragments) - { - if (value.Contains(fragment, StringComparison.OrdinalIgnoreCase)) - return true; - } - - return false; - } - - private static (string Bucket, string Confidence, string Reason) ClassifyUnusedSymbol(bool isPublicOrExported, bool isIntentionalSurfaceSuspect, string? visibility) - { - if (isIntentionalSurfaceSuspect) - { - return ( - UnusedBucketReflectionOrConfig, - "low", - "symbol with attribute-driven reflection surface, serialization, config, metadata, test-hook, generated, documentation, or compatibility surface and no indexed references"); - } - - if (isPublicOrExported) - { - return ( - UnusedBucketPublicOrExported, - "low", - "public/exported symbol with no indexed references"); - } - - if (IsPrivateLikeVisibility(visibility)) - { - return ( - UnusedBucketLikelyPrivate, - "medium", - "private/file-local symbol with no indexed references after same-file text validation"); - } - - return ( - UnusedBucketMaybeNonPublic, - "low", - "non-public symbol with no indexed references"); - } - - private static List BuildUnusedReasonTags(bool isPublicOrExported, bool isIntentionalSurfaceSuspect, string? visibility, IReadOnlyList surfaceTags) - { - var tags = new List { "no_indexed_references" }; - if (isIntentionalSurfaceSuspect) - { - tags.Add("intentional_surface_suspect"); - tags.Add("reflection_or_config_suspect"); - } - foreach (var surfaceTag in surfaceTags) - AddUnusedSurfaceTag(tags, surfaceTag); - if (isPublicOrExported) - tags.Add("public_or_exported"); - else if (IsPrivateLikeVisibility(visibility)) - tags.Add("private_or_file_local"); - else - tags.Add("non_public"); - return tags; - } - - private static UnusedContractDomainClassification ClassifyUnusedContractDomain( - UnusedCandidateSymbol candidate, - string kind, - string bucket, - IReadOnlyList surfaceTags) - { - var tags = new List(); - if (string.Equals(bucket, UnusedBucketReflectionOrConfig, StringComparison.Ordinal)) - AddUnusedSurfaceTag(tags, "intentional_surface_suspect"); - foreach (var surfaceTag in surfaceTags) - AddUnusedSurfaceTag(tags, surfaceTag); - - if (IsPrivateLikeVisibility(candidate.Visibility)) - return CreateUnusedContractDomain(UnusedContractDomainPrivate, tags, "private_or_file_local"); - - if (!candidate.IsPublicOrExported && surfaceTags.Count == 0) - return CreateUnusedContractDomain(UnusedContractDomainNonPublic, tags, "nonpublic_or_protected"); - - AddUnusedSurfaceTag(tags, candidate.IsPublicOrExported ? "public_or_exported" : "nonpublic_or_protected"); - - if (HasUnusedSurfaceTag(surfaceTags, "documentation_heading") || IsMarkdownHeadingSymbol(candidate, kind)) - return CreateUnusedContractDomain(UnusedContractDomainDocumentation, tags, "documentation_heading"); - - if (HasUnusedSurfaceTag(surfaceTags, "markdown_fence_language_marker") || IsMarkdownFenceSymbol(candidate, kind)) - return CreateUnusedContractDomain(UnusedContractDomainDocumentation, tags, "markdown_fence_language_marker"); - - if (IsUnusedTestContractSurface(candidate)) - return CreateUnusedContractDomain(UnusedContractDomainTest, tags, "test_surface"); - - if (HasUnusedSurfaceTag(surfaceTags, "generated_surface") || IsGeneratedSurface(candidate)) - return CreateUnusedContractDomain(UnusedContractDomainGenerated, tags, "generated_surface"); - - if (HasUnusedSurfaceTag(surfaceTags, "exception_metadata") || IsExceptionDiagnosticSurface(candidate, kind)) - return CreateUnusedContractDomain(UnusedContractDomainExceptionDiagnostic, tags, "exception_metadata"); - - if (IsFrameworkOverrideSurface(candidate)) - return CreateUnusedContractDomain(UnusedContractDomainFrameworkOverride, tags, "framework_override"); - - if (IsMcpContractSurface(candidate)) - return CreateUnusedContractDomain(UnusedContractDomainMcp, tags, "mcp_tool_contract"); - - if (IsLspContractSurface(candidate)) - return CreateUnusedContractDomain(UnusedContractDomainLsp, tags, "lsp_protocol_contract"); - - if (IsCliContractSurface(candidate)) - return CreateUnusedContractDomain(UnusedContractDomainCli, tags, "cli_option_or_result"); - - if (IsConfigurationContractSurface(candidate, kind, surfaceTags)) - return CreateUnusedContractDomain(UnusedContractDomainConfig, tags, "configuration_or_metadata_contract"); - - if (IsJsonContractSurface(candidate, surfaceTags)) - return CreateUnusedContractDomain(UnusedContractDomainJson, tags, "json_output_or_input_contract"); - - if (HasUnusedSurfaceTag(surfaceTags, "reflection_or_config_suspect")) - return CreateUnusedContractDomain(UnusedContractDomainSerialization, tags, "reflection_or_serialization_contract"); - - return candidate.IsPublicOrExported - ? CreateUnusedContractDomain(UnusedContractDomainPublicApi, tags, "public_api_surface") - : CreateUnusedContractDomain(UnusedContractDomainNonPublic, tags, "nonpublic_or_protected"); - } - - private static UnusedContractDomainClassification CreateUnusedContractDomain(string domain, List tags, params string[] domainTags) - { - AddUnusedSurfaceTag(tags, domain); - foreach (var tag in domainTags) - AddUnusedSurfaceTag(tags, tag); - return new UnusedContractDomainClassification(domain, tags); - } - - private static bool HasUnusedSurfaceTag(IReadOnlyList tags, string tag) - => tags.Contains(tag, StringComparer.Ordinal); - - private static bool IsUnusedTestContractSurface(UnusedCandidateSymbol candidate) - { - return IsUnusedTestPath(candidate.Path) - || EndsWithAny(candidate.ContainerName, ["Test", "Tests", "Fixture"]) - || EndsWithAny(candidate.ContainerQualifiedName, ["Test", "Tests", "Fixture"]) - || IsTestHookName(candidate.Name); - } - - private static bool IsUnusedTestPath(string? path) - { - if (string.IsNullOrWhiteSpace(path)) - return false; - - var normalized = path.Replace('\\', '/'); - return normalized.StartsWith("test/", StringComparison.OrdinalIgnoreCase) - || normalized.StartsWith("tests/", StringComparison.OrdinalIgnoreCase) - || normalized.Contains("/test/", StringComparison.OrdinalIgnoreCase) - || normalized.Contains("/tests/", StringComparison.OrdinalIgnoreCase) - || normalized.Contains(".tests/", StringComparison.OrdinalIgnoreCase); - } - - private static bool IsExceptionDiagnosticSurface(UnusedCandidateSymbol candidate, string kind) - { - return IsExceptionMetadataProperty(candidate, kind) - || (IsDataMemberUnusedKind(kind) - && (EndsWithAny(candidate.ContainerName, ["Exception"]) - || EndsWithAny(candidate.ContainerQualifiedName, ["Exception"]))) - || (IsTypeLikeUnusedKind(kind) && EndsWithAny(candidate.Name, ["Exception"])); - } - - private static bool IsFrameworkOverrideSurface(UnusedCandidateSymbol candidate) - { - if (ContainsAny(candidate.Signature, [" override ", " override\t", " override\r", " override\n"])) - return true; - - if (!UnusedFrameworkOverrideMemberNames.Contains(candidate.Name)) - return false; - - return EndsWithAny(candidate.ContainerName, ["Stream", "TextReader", "TextWriter"]) - || EndsWithAny(candidate.ContainerQualifiedName, ["Stream", "TextReader", "TextWriter"]) - || ContainsAny(candidate.Signature, ["Stream", "TextReader", "TextWriter"]); - } - - private static bool IsMcpContractSurface(UnusedCandidateSymbol candidate) - { - return ContainsAny(candidate.Path, UnusedMcpPathMarkers) - || ContainsAny(candidate.ContainerName, ["Mcp", "JsonRpc"]) - || ContainsAny(candidate.ContainerQualifiedName, ["Mcp", "JsonRpc"]) - || ContainsAny(candidate.Signature, ["Mcp", "JsonRpc"]); - } - - private static bool IsLspContractSurface(UnusedCandidateSymbol candidate) - { - return ContainsAny(candidate.Path, UnusedLspPathMarkers) - || ContainsAny(candidate.ContainerName, ["Lsp", "LanguageServer"]) - || ContainsAny(candidate.ContainerQualifiedName, ["Lsp", "LanguageServer"]) - || ContainsAny(candidate.Signature, ["Lsp", "LanguageServer"]); - } - - private static bool IsCliContractSurface(UnusedCandidateSymbol candidate) - { - var hasCliContext = ContainsAny(candidate.Path, UnusedCliPathMarkers) - || ContainsAny(candidate.ContainerQualifiedName, ["Cli", "CommandLine"]) - || ContainsAny(candidate.Signature, ["CommandLine", "System.CommandLine", "Option<", "Argument<"]); - if (hasCliContext) - return true; - - return EndsWithAny(candidate.Name, ["Command", "Flag", "Flags", "Usage", "ExitCode", "ErrorCode"]) - || EndsWithAny(candidate.ContainerName, ["Command", "Flag", "Flags", "Usage", "ExitCode", "ErrorCode"]); - } - - private static bool IsConfigurationContractSurface(UnusedCandidateSymbol candidate, string kind, IReadOnlyList surfaceTags) - { - return HasUnusedSurfaceTag(surfaceTags, "config_or_metadata_surface") - || HasUnusedSurfaceTag(surfaceTags, "config_or_metadata_member") - || IsConfigOrManifestSurface(candidate, kind) - || IsConfigOrMetadataMember(candidate, kind); - } - - private static bool IsJsonContractSurface(UnusedCandidateSymbol candidate, IReadOnlyList surfaceTags) - { - return HasUnusedSurfaceTag(surfaceTags, "serialization_contract") - || HasUnusedSurfaceTag(surfaceTags, "contract_member") - || HasUnusedSurfaceTag(surfaceTags, "source_generated_json_context") - || ContainsAny(candidate.Path, UnusedContractPathSegments) - || EndsWithAny(candidate.Name, UnusedRecordContractSuffixes) - || EndsWithAny(candidate.ContainerName, UnusedRecordContractSuffixes) - || ContainsAny(candidate.Name, UnusedJsonContractTerms) - || ContainsAny(candidate.ContainerName, UnusedJsonContractTerms) - || ContainsAny(candidate.Signature, ["JsonProperty", "JsonInclude", "JsonSerializerContext", "DataContract", "DataMember", "XmlElement", "XmlAttribute", "YamlMember", "MessagePackObject", "ProtoContract"]); - } - - private static bool IsPrivateLikeVisibility(string? visibility) - { - return string.Equals(visibility, "private", StringComparison.OrdinalIgnoreCase) - || string.Equals(visibility, "fileprivate", StringComparison.OrdinalIgnoreCase); - } - - private static readonly string[] OrderedUnusedBuckets = - [ - UnusedBucketLikelyPrivate, - UnusedBucketMaybeNonPublic, - UnusedBucketPublicOrExported, - UnusedBucketReflectionOrConfig, - ]; } diff --git a/src/CodeIndex/Database/DbWriter.CSharpMetadataTargets.cs b/src/CodeIndex/Database/DbWriter.CSharpMetadataTargets.cs index 0b2890d9d7..a8f4ef5c8d 100644 --- a/src/CodeIndex/Database/DbWriter.CSharpMetadataTargets.cs +++ b/src/CodeIndex/Database/DbWriter.CSharpMetadataTargets.cs @@ -117,7 +117,7 @@ internal CSharpMetadataTargetResolutionStats ResolveCSharpMetadataTargetsCore( dependencies.Clear(); perFileImports.TryGetValue(row.FileId, out var fileImports); bool directTarget = CollectMetadataTargetDependencies( - ParseCSharpBaseIdentifiers(row.Signature), + CSharpBaseListParser.Parse(row.Signature, CSharpBaseListProjection.HeadIdentifier), GetEnclosingScope(row.QualifiedName, row.Name), scopeNameToIds, qualifiedToIds, @@ -904,147 +904,4 @@ private static void CollectNamespaceImportDependencies( } } - /// - /// Extract base-type head identifiers from a C# class signature, respecting generic depth - /// so that `Foo : IBase, IOther` yields ["IBase", "IOther"]. Stops at the - /// first `where` clause (generic constraints are not bases) and trims modifiers like - /// `public sealed`. - /// C# class signature から基底/インターフェース識別子の頭を抜き出す。`<...>` の depth を - /// 数えて generic argument 内の `,` を区切りに誤認しないようにし、`where` 制約は除外する。 - /// - internal static List ParseCSharpBaseIdentifiers(string? signature) - { - var result = new List(); - if (string.IsNullOrEmpty(signature)) - return result; - - int colonIdx = FindBaseListColon(signature); - if (colonIdx < 0) - return result; - - int start = colonIdx + 1; - int genericDepth = 0; - var current = new System.Text.StringBuilder(); - for (int i = start; i < signature.Length; i++) - { - char c = signature[i]; - if (c == '<') - { - genericDepth++; - current.Append(c); - continue; - } - if (c == '>') - { - if (genericDepth > 0) - genericDepth--; - current.Append(c); - continue; - } - if (c == '{') - break; - if (genericDepth == 0 && c == ',') - { - AddBaseIfPresent(result, current.ToString()); - current.Clear(); - continue; - } - // `where T : ...` ends the base list - if (genericDepth == 0 && (c == 'w' || c == 'W')) - { - if (LooksLikeWhereKeyword(signature, i)) - { - AddBaseIfPresent(result, current.ToString()); - current.Clear(); - return result; - } - } - current.Append(c); - } - AddBaseIfPresent(result, current.ToString()); - return result; - } - - private static int FindBaseListColon(string signature) - { - int genericDepth = 0; - int parenDepth = 0; - for (int i = 0; i < signature.Length; i++) - { - char c = signature[i]; - if (c == '<') { genericDepth++; continue; } - if (c == '>') { if (genericDepth > 0) genericDepth--; continue; } - if (c == '(') { parenDepth++; continue; } - if (c == ')') { if (parenDepth > 0) parenDepth--; continue; } - if (c == '{') - return -1; - // `class Foo where T : IBar {}` has no base list — only a generic constraint. - // If we reach a top-level `where` before finding `:`, treat that `:` as a - // constraint separator, not a base list opener. Issue #435 codex review. - // `class Foo where T : IBar {}` のように base list を持たない場合、ここで遭遇する - // `:` は generic constraint の区切りなので base list colon として採用しない。 - if (genericDepth == 0 && parenDepth == 0 && (c == 'w' || c == 'W') - && LooksLikeWhereKeyword(signature, i)) - { - return -1; - } - if (c == ':' && genericDepth == 0 && parenDepth == 0) - { - // Skip `::` namespace alias separator / `::` 名前空間エイリアスは除外 - if (i + 1 < signature.Length && signature[i + 1] == ':') - { - i++; - continue; - } - if (i > 0 && signature[i - 1] == ':') - continue; - return i; - } - } - return -1; - } - - private static bool LooksLikeWhereKeyword(string signature, int i) - { - if (i + 5 > signature.Length) - return false; - if (string.Compare(signature, i, "where", 0, 5, StringComparison.OrdinalIgnoreCase) != 0) - return false; - if (i > 0) - { - char prev = signature[i - 1]; - if (char.IsLetterOrDigit(prev) || prev == '_') - return false; - } - if (i + 5 < signature.Length) - { - char next = signature[i + 5]; - if (char.IsLetterOrDigit(next) || next == '_') - return false; - } - return true; - } - - private static void AddBaseIfPresent(List result, string raw) - { - var trimmed = raw.Trim(); - if (trimmed.Length == 0) - return; - // Take the head identifier (everything before `<` or whitespace) but preserve - // any namespace prefix so the caller can treat `System.Attribute` directly. - // `<` 以前と空白以前を頭とし、`System.Attribute` などの名前空間付きはそのまま残す。 - int cut = trimmed.Length; - for (int i = 0; i < trimmed.Length; i++) - { - char c = trimmed[i]; - if (c == '<' || char.IsWhiteSpace(c)) - { - cut = i; - break; - } - } - var head = trimmed.Substring(0, cut); - if (head.Length > 0) - result.Add(head); - } } diff --git a/src/CodeIndex/Database/DbWriter.TypeScriptAugmentationPlanning.cs b/src/CodeIndex/Database/DbWriter.TypeScriptAugmentationPlanning.cs new file mode 100644 index 0000000000..01a251eeb4 --- /dev/null +++ b/src/CodeIndex/Database/DbWriter.TypeScriptAugmentationPlanning.cs @@ -0,0 +1,326 @@ +using TypeScriptDeletedAugmentationReferences = System.Collections.Generic.List<( + long Id, + long FileId, + long? SourceId, + long? TargetId, + string? ContainerNameFolded, + string? SymbolNameFolded)>; + +namespace CodeIndex.Database; + +public partial class DbWriter +{ + private readonly record struct TypeScriptAugmentationScopePlan( + string[]? ScopedNames, + bool IncludeIndexedInterfaceMarkers); + + private readonly record struct TypeScriptInterfaceDeclaration( + long FileId, + string Path, + string Name, + int Line, + int Column, + string Signature, + string Kind, + string ContainerName, + string? Visibility); + + private TypeScriptAugmentationScopePlan BuildTypeScriptAugmentationScopePlan( + IReadOnlyCollection? dirtyNames, + CancellationToken cancellationToken) + { + if (dirtyNames == null) + return new TypeScriptAugmentationScopePlan(null, IncludeIndexedInterfaceMarkers: false); + + var uniqueNames = new HashSet(StringComparer.Ordinal); + var inspectedNameCount = 0; + foreach (var name in dirtyNames) + { + if (!string.IsNullOrEmpty(name)) + uniqueNames.Add(name); + if ((++inspectedNameCount & 1_023) == 0) + cancellationToken.ThrowIfCancellationRequested(); + } + cancellationToken.ThrowIfCancellationRequested(); + if (uniqueNames.Count > 1_024 + && ShouldUseFullTypeScriptAugmentationRebuild(uniqueNames.Count, cancellationToken)) + { + return new TypeScriptAugmentationScopePlan(null, IncludeIndexedInterfaceMarkers: false); + } + + var scopedNames = new string[uniqueNames.Count]; + uniqueNames.CopyTo(scopedNames); + Array.Sort(scopedNames, StringComparer.Ordinal); + return new TypeScriptAugmentationScopePlan(scopedNames, IncludeIndexedInterfaceMarkers: true); + } + + private int DeleteAndTrackTypeScriptAugmentationReferences( + TypeScriptAugmentationScopePlan scopePlan, + HashSet affectedFileIds, + CancellationToken cancellationToken) + { + var deletedReferences = new TypeScriptDeletedAugmentationReferences(); + if (scopePlan.ScopedNames == null) + { + DeleteAllTypeScriptAugmentationReferences( + affectedFileIds, + deletedReferences, + cancellationToken); + } + else + { + DeleteScopedTypeScriptAugmentationReferences( + scopePlan.ScopedNames, + affectedFileIds, + deletedReferences, + cancellationToken); + } + + TrackReferenceGraphDeletedReferences(deletedReferences); + return deletedReferences.Count; + } + + private void DeleteAllTypeScriptAugmentationReferences( + HashSet affectedFileIds, + TypeScriptDeletedAugmentationReferences deletedReferences, + CancellationToken cancellationToken) + { + var deleteCmd = RentCommand( + """ + DELETE FROM symbol_references + WHERE reference_kind = 'augmentation' + RETURNING id, + file_id, + source_symbol_id, + target_symbol_id, + container_name_folded, + symbol_name_folded + """, + static _ => { }); + try + { + using var reader = deleteCmd.ExecuteReader(); + ReadDeletedTypeScriptAugmentationReferences( + reader, + affectedFileIds, + deletedReferences, + cancellationToken); + cancellationToken.ThrowIfCancellationRequested(); + } + finally + { + ReleaseCommand(deleteCmd); + } + } + + private void DeleteScopedTypeScriptAugmentationReferences( + IReadOnlyList scopedNames, + HashSet affectedFileIds, + TypeScriptDeletedAugmentationReferences deletedReferences, + CancellationToken cancellationToken) + { + const int nameBatchSize = 900; + for (var offset = 0; offset < scopedNames.Count; offset += nameBatchSize) + { + cancellationToken.ThrowIfCancellationRequested(); + var count = Math.Min(nameBatchSize, scopedNames.Count - offset); + using (var deleteCmd = CreateTypeScriptAugmentationNameCommand( + scopedNames, + offset, + count, + """ + DELETE FROM symbol_references + WHERE reference_kind = 'augmentation' + AND symbol_name IN ({0}) + RETURNING id, + file_id, + source_symbol_id, + target_symbol_id, + container_name_folded, + symbol_name_folded + """)) + using (var reader = deleteCmd.ExecuteReader()) + { + ReadDeletedTypeScriptAugmentationReferences( + reader, + affectedFileIds, + deletedReferences, + cancellationToken); + } + TypeScriptAugmentationNameBatchForTesting?.Invoke((offset / nameBatchSize) + 1); + cancellationToken.ThrowIfCancellationRequested(); + } + } + + private static void ReadDeletedTypeScriptAugmentationReferences( + Microsoft.Data.Sqlite.SqliteDataReader reader, + HashSet affectedFileIds, + TypeScriptDeletedAugmentationReferences deletedReferences, + CancellationToken cancellationToken) + { + var deletedRowCount = 0; + while (reader.Read()) + { + var fileId = reader.GetInt64(1); + affectedFileIds.Add(fileId); + deletedReferences.Add(( + reader.GetInt64(0), + fileId, + ReadNullableInt64(reader, 2), + ReadNullableInt64(reader, 3), + ReadNullableString(reader, 4), + ReadNullableString(reader, 5))); + if ((++deletedRowCount & 255) == 0) + cancellationToken.ThrowIfCancellationRequested(); + } + } + + private List LoadTypeScriptInterfaceDeclarations( + TypeScriptAugmentationScopePlan scopePlan, + CancellationToken cancellationToken) + { + var declarations = new List(); + if (scopePlan.ScopedNames == null) + { + var command = RentCommand( + BuildTypeScriptInterfaceDeclarationSql(namePredicate: null), + static _ => { }); + try + { + using var reader = command.ExecuteReader(); + ReadTypeScriptInterfaceDeclarations(reader, declarations, cancellationToken); + } + finally + { + ReleaseCommand(command); + } + } + else + { + LoadScopedTypeScriptInterfaceDeclarations( + scopePlan.ScopedNames, + declarations, + cancellationToken); + } + return declarations; + } + + private void LoadScopedTypeScriptInterfaceDeclarations( + IReadOnlyList scopedNames, + List declarations, + CancellationToken cancellationToken) + { + const int nameBatchSize = 900; + for (var offset = 0; offset < scopedNames.Count; offset += nameBatchSize) + { + cancellationToken.ThrowIfCancellationRequested(); + var count = Math.Min(nameBatchSize, scopedNames.Count - offset); + using (var command = CreateTypeScriptAugmentationNameCommand( + scopedNames, + offset, + count, + BuildTypeScriptInterfaceDeclarationSql("s.name IN ({0})"))) + using (var reader = command.ExecuteReader()) + { + ReadTypeScriptInterfaceDeclarations(reader, declarations, cancellationToken); + } + TypeScriptAugmentationNameBatchForTesting?.Invoke((offset / nameBatchSize) + 1); + cancellationToken.ThrowIfCancellationRequested(); + } + } + + private static string BuildTypeScriptInterfaceDeclarationSql(string? namePredicate) => + @" + SELECT s.file_id, + f.path, + s.name, + s.line, + s.start_column, + s.signature, + s.kind, + s.container_name, + s.visibility + FROM symbols s" + + (namePredicate == null ? string.Empty : " INDEXED BY idx_symbols_name") + + @" + JOIN files f ON f.id = s.file_id + WHERE f.lang = 'typescript' + AND s.name IS NOT NULL + AND s.name <> '' + AND s.kind = 'interface'" + + (namePredicate == null ? string.Empty : "\n AND " + namePredicate) + + "\n ORDER BY s.name, s.file_id, s.line"; + + private static void ReadTypeScriptInterfaceDeclarations( + Microsoft.Data.Sqlite.SqliteDataReader reader, + List declarations, + CancellationToken cancellationToken) + { + while (reader.Read()) + { + declarations.Add(new TypeScriptInterfaceDeclaration( + reader.GetInt64(0), + reader.GetString(1), + reader.GetString(2), + reader.IsDBNull(3) ? 1 : Math.Max(1, reader.GetInt32(3)), + reader.IsDBNull(4) ? 1 : Math.Max(1, reader.GetInt32(4) + 1), + reader.IsDBNull(5) ? string.Empty : reader.GetString(5), + reader.IsDBNull(6) ? string.Empty : reader.GetString(6), + reader.IsDBNull(7) ? string.Empty : reader.GetString(7), + reader.IsDBNull(8) ? null : reader.GetString(8))); + if ((declarations.Count & 255) == 0) + cancellationToken.ThrowIfCancellationRequested(); + } + cancellationToken.ThrowIfCancellationRequested(); + } + + private Microsoft.Data.Sqlite.SqliteCommand CreateTypeScriptAugmentationNameCommand( + IReadOnlyList names, + int offset, + int count, + string sqlTemplate) + { + SqliteDynamicSql.EnsureParameterBudget(count, "TypeScript augmentation name batch"); + var command = _conn.CreateCommand(); + command.Transaction = _activeTransaction; + var parameterNames = new string[count]; + for (var index = 0; index < count; index++) + { + var parameterName = SqliteDynamicSql.BuildParameterName("augmentation_name", index); + parameterNames[index] = parameterName; + command.Parameters.Add(parameterName, Microsoft.Data.Sqlite.SqliteType.Text).Value = names[offset + index]; + } + command.CommandText = string.Format( + System.Globalization.CultureInfo.InvariantCulture, + sqlTemplate, + string.Join(", ", parameterNames)); + return command; + } + + private bool ShouldUseFullTypeScriptAugmentationRebuild( + int dirtyNameCount, + CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + var command = RentCommand( + @" + SELECT COUNT(*) + FROM symbols s INDEXED BY idx_symbols_kind + JOIN files f ON f.id = s.file_id + WHERE s.kind = 'interface' + AND f.lang = 'typescript'", + static _ => { }); + try + { + var declarationCount = Convert.ToInt64( + command.ExecuteScalar(), + System.Globalization.CultureInfo.InvariantCulture); + cancellationToken.ThrowIfCancellationRequested(); + return dirtyNameCount >= Math.Max(1_024L, (declarationCount + 1L) / 2L); + } + finally + { + ReleaseCommand(command); + } + } +} diff --git a/src/CodeIndex/Database/DbWriter.TypeScriptAugmentationProjection.cs b/src/CodeIndex/Database/DbWriter.TypeScriptAugmentationProjection.cs new file mode 100644 index 0000000000..9efed56057 --- /dev/null +++ b/src/CodeIndex/Database/DbWriter.TypeScriptAugmentationProjection.cs @@ -0,0 +1,134 @@ +using CodeIndex.Models; + +namespace CodeIndex.Database; + +public partial class DbWriter +{ + private List ProjectTypeScriptAugmentationReferences( + string? projectRoot, + TypeScriptAugmentationScopePlan scopePlan, + IReadOnlyList declarations, + CancellationToken cancellationToken) + { + var moduleFileIds = FindTypeScriptModuleFileIds( + projectRoot, + declarations, + scopePlan.IncludeIndexedInterfaceMarkers, + cancellationToken); + var groupIndexes = new Dictionary<(string Name, string ScopeKey), int>(declarations.Count); + var groups = new List<(int FirstDeclarationIndex, List? DeclarationIndexes)>(declarations.Count); + for (var declarationIndex = 0; declarationIndex < declarations.Count; declarationIndex++) + { + if ((declarationIndex & 1_023) == 0) + cancellationToken.ThrowIfCancellationRequested(); + var declaration = declarations[declarationIndex]; + var key = ( + declaration.Name, + BuildTypeScriptScopeKey( + declaration.FileId, + declaration.Path, + declaration.Signature, + declaration.ContainerName, + moduleFileIds)); + if (!groupIndexes.TryGetValue(key, out var groupIndex)) + { + groupIndexes.Add(key, groups.Count); + groups.Add((declarationIndex, null)); + continue; + } + + var group = groups[groupIndex]; + if (group.DeclarationIndexes == null) + group.DeclarationIndexes = new List(2) { group.FirstDeclarationIndex }; + group.DeclarationIndexes.Add(declarationIndex); + groups[groupIndex] = group; + } + + return ProjectMergedTypeScriptAugmentationGroups( + scopePlan, + declarations, + groups, + cancellationToken); + } + + private static List ProjectMergedTypeScriptAugmentationGroups( + TypeScriptAugmentationScopePlan scopePlan, + IReadOnlyList declarations, + IReadOnlyList<(int FirstDeclarationIndex, List? DeclarationIndexes)> groups, + CancellationToken cancellationToken) + { + var references = new List(); + var mergedGroupCount = 0; + var materializedDeclarationIndexCount = 0; + var mergedDeclarationCount = 0; + for (var groupIndex = 0; groupIndex < groups.Count; groupIndex++) + { + if ((groupIndex & 1_023) == 0) + cancellationToken.ThrowIfCancellationRequested(); + var group = groups[groupIndex]; + if (group.DeclarationIndexes == null) + continue; + + mergedGroupCount++; + materializedDeclarationIndexCount += group.DeclarationIndexes.Count; + foreach (var declarationIndex in group.DeclarationIndexes) + { + if ((mergedDeclarationCount++ & 1_023) == 0) + cancellationToken.ThrowIfCancellationRequested(); + var declaration = declarations[declarationIndex]; + references.Add(new ReferenceRecord + { + FileId = declaration.FileId, + SymbolName = declaration.Name, + ReferenceKind = "augmentation", + Line = declaration.Line, + Column = declaration.Column, + Context = declaration.Signature, + ContainerKind = declaration.Kind == "interface" ? "interface" : "type", + ContainerName = declaration.Name, + }); + } + } + TypeScriptAugmentationGroupingForTesting?.Invoke(new TypeScriptAugmentationGroupingStats( + declarations.Count, + groups.Count, + mergedGroupCount, + materializedDeclarationIndexCount, + scopePlan.ScopedNames?.Length)); + return references; + } + + private void ApplyTypeScriptAugmentationReferences( + List references, + int deletedReferenceCount, + HashSet affectedFileIds, + bool finalizeDeferredReferenceGraph, + ReferenceSecondaryIndexBulkLoadGuard? referenceSecondaryIndexBulkLoad, + CancellationToken cancellationToken) + { + InsertReferencesInAtomicFileScope( + references, + refreshMutualRecursionFlags: true, + cancellationToken, + referenceSecondaryIndexBulkLoad); + if (references.Count == 0 + && (deletedReferenceCount > 0 || finalizeDeferredReferenceGraph)) + { + // The insert helper intentionally no-ops for an empty batch. Augmentation + // rebuilds finalize only when they deleted synthetic edges or explicitly + // inherited a coalesced graph pass. Marker-only validation stays O(1) here. + // 空batchはedge削除または先行pass統合時だけgraphを確定し、marker検証だけなら省く。 + cancellationToken.ThrowIfCancellationRequested(); + RefreshMutualRecursionFlags( + cancellationToken, + referenceSecondaryIndexBulkLoad: referenceSecondaryIndexBulkLoad); + } + for (var referenceIndex = 0; referenceIndex < references.Count; referenceIndex++) + { + if ((referenceIndex & 1_023) == 0) + cancellationToken.ThrowIfCancellationRequested(); + affectedFileIds.Remove(references[referenceIndex].FileId); + } + RefreshHotspotReferenceCounts(affectedFileIds, cancellationToken); + } +} diff --git a/src/CodeIndex/Database/DbWriter.TypeScriptAugmentations.cs b/src/CodeIndex/Database/DbWriter.TypeScriptAugmentations.cs index 1c47e00d74..945c5acc20 100644 --- a/src/CodeIndex/Database/DbWriter.TypeScriptAugmentations.cs +++ b/src/CodeIndex/Database/DbWriter.TypeScriptAugmentations.cs @@ -166,17 +166,6 @@ internal static Action? TypeScriptAugmentationReadyCheckForTesting set => ScopedTypeScriptAugmentationReadyCheckForTesting.Value = value; } - private readonly record struct TypeScriptInterfaceDeclaration( - long FileId, - string Path, - string Name, - int Line, - int Column, - string Signature, - string Kind, - string ContainerName, - string? Visibility); - public int RebuildTypeScriptAugmentationReferences(string? projectRoot = null) => RebuildTypeScriptAugmentationReferencesCore( projectRoot, @@ -247,245 +236,25 @@ private int RebuildTypeScriptAugmentationReferencesCore( { cancellationToken.ThrowIfCancellationRequested(); _ = TryStartDeferredHotspotReferenceMutation(); - - string[]? scopedNames = null; - if (dirtyNames != null) - { - var uniqueNames = new HashSet(StringComparer.Ordinal); - var inspectedNameCount = 0; - foreach (var name in dirtyNames) - { - if (!string.IsNullOrEmpty(name)) - uniqueNames.Add(name); - if ((++inspectedNameCount & 1_023) == 0) - cancellationToken.ThrowIfCancellationRequested(); - } - cancellationToken.ThrowIfCancellationRequested(); - if (uniqueNames.Count > 1_024 - && ShouldUseFullTypeScriptAugmentationRebuild(uniqueNames.Count, cancellationToken)) - { - scopedNames = null; - } - else - { - scopedNames = [.. uniqueNames]; - Array.Sort(scopedNames, StringComparer.Ordinal); - } - } - + var scopePlan = BuildTypeScriptAugmentationScopePlan(dirtyNames, cancellationToken); var affectedFileIds = new HashSet(); - var deletedReferences = new List<( - long Id, - long FileId, - long? SourceId, - long? TargetId, - string? ContainerNameFolded, - string? SymbolNameFolded)>(); - if (scopedNames == null) - { - var deleteCmd = RentCommand( - """ - DELETE FROM symbol_references - WHERE reference_kind = 'augmentation' - RETURNING id, - file_id, - source_symbol_id, - target_symbol_id, - container_name_folded, - symbol_name_folded - """, - static _ => { }); - try - { - using var reader = deleteCmd.ExecuteReader(); - var deletedRowCount = 0; - while (reader.Read()) - { - var fileId = reader.GetInt64(1); - affectedFileIds.Add(fileId); - deletedReferences.Add(( - reader.GetInt64(0), - fileId, - ReadNullableInt64(reader, 2), - ReadNullableInt64(reader, 3), - ReadNullableString(reader, 4), - ReadNullableString(reader, 5))); - if ((++deletedRowCount & 255) == 0) - cancellationToken.ThrowIfCancellationRequested(); - } - cancellationToken.ThrowIfCancellationRequested(); - } - finally - { - ReleaseCommand(deleteCmd); - } - } - else - { - ForEachTypeScriptAugmentationNameBatch(scopedNames, cancellationToken, (names, offset, count) => - { - using var deleteCmd = CreateTypeScriptAugmentationNameCommand( - names, - offset, - count, - """ - DELETE FROM symbol_references - WHERE reference_kind = 'augmentation' - AND symbol_name IN ({0}) - RETURNING id, - file_id, - source_symbol_id, - target_symbol_id, - container_name_folded, - symbol_name_folded - """); - using var reader = deleteCmd.ExecuteReader(); - var deletedRowCount = 0; - while (reader.Read()) - { - var fileId = reader.GetInt64(1); - affectedFileIds.Add(fileId); - deletedReferences.Add(( - reader.GetInt64(0), - fileId, - ReadNullableInt64(reader, 2), - ReadNullableInt64(reader, 3), - ReadNullableString(reader, 4), - ReadNullableString(reader, 5))); - if ((++deletedRowCount & 255) == 0) - cancellationToken.ThrowIfCancellationRequested(); - } - }); - } - TrackReferenceGraphDeletedReferences(deletedReferences); - - var references = new List(); - var declarations = new List(); - if (scopedNames == null) - { - var cmd = RentCommand( - BuildTypeScriptInterfaceDeclarationSql(namePredicate: null), - static _ => { }); - try - { - using var reader = cmd.ExecuteReader(); - ReadTypeScriptInterfaceDeclarations(reader, declarations, cancellationToken); - } - finally - { - ReleaseCommand(cmd); - } - } - else - { - ForEachTypeScriptAugmentationNameBatch(scopedNames, cancellationToken, (names, offset, count) => - { - using var cmd = CreateTypeScriptAugmentationNameCommand( - names, - offset, - count, - BuildTypeScriptInterfaceDeclarationSql("s.name IN ({0})")); - using var reader = cmd.ExecuteReader(); - ReadTypeScriptInterfaceDeclarations(reader, declarations, cancellationToken); - }); - } - - var moduleFileIds = FindTypeScriptModuleFileIds( + var deletedReferenceCount = DeleteAndTrackTypeScriptAugmentationReferences( + scopePlan, + affectedFileIds, + cancellationToken); + var declarations = LoadTypeScriptInterfaceDeclarations(scopePlan, cancellationToken); + var references = ProjectTypeScriptAugmentationReferences( projectRoot, + scopePlan, declarations, - includeIndexedInterfaceMarkers: scopedNames != null, cancellationToken); - var groupIndexes = new Dictionary<(string Name, string ScopeKey), int>(declarations.Count); - var groups = new List<(int FirstDeclarationIndex, List? DeclarationIndexes)>(declarations.Count); - for (var declarationIndex = 0; declarationIndex < declarations.Count; declarationIndex++) - { - if ((declarationIndex & 1_023) == 0) - cancellationToken.ThrowIfCancellationRequested(); - var declaration = declarations[declarationIndex]; - var key = ( - declaration.Name, - BuildTypeScriptScopeKey( - declaration.FileId, - declaration.Path, - declaration.Signature, - declaration.ContainerName, - moduleFileIds)); - if (!groupIndexes.TryGetValue(key, out var groupIndex)) - { - groupIndexes.Add(key, groups.Count); - groups.Add((declarationIndex, null)); - continue; - } - - var group = groups[groupIndex]; - if (group.DeclarationIndexes == null) - group.DeclarationIndexes = new List(2) { group.FirstDeclarationIndex }; - group.DeclarationIndexes.Add(declarationIndex); - groups[groupIndex] = group; - } - - var mergedGroupCount = 0; - var materializedDeclarationIndexCount = 0; - var mergedDeclarationCount = 0; - for (var groupIndex = 0; groupIndex < groups.Count; groupIndex++) - { - if ((groupIndex & 1_023) == 0) - cancellationToken.ThrowIfCancellationRequested(); - var group = groups[groupIndex]; - if (group.DeclarationIndexes == null) - continue; - - mergedGroupCount++; - materializedDeclarationIndexCount += group.DeclarationIndexes.Count; - foreach (var declarationIndex in group.DeclarationIndexes) - { - if ((mergedDeclarationCount++ & 1_023) == 0) - cancellationToken.ThrowIfCancellationRequested(); - var declaration = declarations[declarationIndex]; - references.Add(new ReferenceRecord - { - FileId = declaration.FileId, - SymbolName = declaration.Name, - ReferenceKind = "augmentation", - Line = declaration.Line, - Column = declaration.Column, - Context = declaration.Signature, - ContainerKind = declaration.Kind == "interface" ? "interface" : "type", - ContainerName = declaration.Name, - }); - } - } - TypeScriptAugmentationGroupingForTesting?.Invoke(new TypeScriptAugmentationGroupingStats( - declarations.Count, - groups.Count, - mergedGroupCount, - materializedDeclarationIndexCount, - scopedNames?.Length)); - - InsertReferencesInAtomicFileScope( + ApplyTypeScriptAugmentationReferences( references, - refreshMutualRecursionFlags: true, - cancellationToken, - referenceSecondaryIndexBulkLoad); - if (references.Count == 0 - && (deletedReferences.Count > 0 || finalizeDeferredReferenceGraph)) - { - // The insert helper intentionally no-ops for an empty batch. Augmentation - // rebuilds finalize only when they deleted synthetic edges or explicitly - // inherited a coalesced graph pass. Marker-only validation stays O(1) here. - // 空batchはedge削除または先行pass統合時だけgraphを確定し、marker検証だけなら省く。 - cancellationToken.ThrowIfCancellationRequested(); - RefreshMutualRecursionFlags( - cancellationToken, - referenceSecondaryIndexBulkLoad: referenceSecondaryIndexBulkLoad); - } - for (var referenceIndex = 0; referenceIndex < references.Count; referenceIndex++) - { - if ((referenceIndex & 1_023) == 0) - cancellationToken.ThrowIfCancellationRequested(); - affectedFileIds.Remove(references[referenceIndex].FileId); - } - RefreshHotspotReferenceCounts(affectedFileIds, cancellationToken); + deletedReferenceCount, + affectedFileIds, + finalizeDeferredReferenceGraph, + referenceSecondaryIndexBulkLoad, + cancellationToken); cancellationToken.ThrowIfCancellationRequested(); MarkTypeScriptAugmentationReady(); ownedDeferredRefresh?.Complete(cancellationToken); @@ -503,116 +272,6 @@ AND symbol_name IN ({0}) } } - private static string BuildTypeScriptInterfaceDeclarationSql(string? namePredicate) => - @" - SELECT s.file_id, - f.path, - s.name, - s.line, - s.start_column, - s.signature, - s.kind, - s.container_name, - s.visibility - FROM symbols s" - + (namePredicate == null ? string.Empty : " INDEXED BY idx_symbols_name") - + @" - JOIN files f ON f.id = s.file_id - WHERE f.lang = 'typescript' - AND s.name IS NOT NULL - AND s.name <> '' - AND s.kind = 'interface'" - + (namePredicate == null ? string.Empty : "\n AND " + namePredicate) - + "\n ORDER BY s.name, s.file_id, s.line"; - - private static void ReadTypeScriptInterfaceDeclarations( - Microsoft.Data.Sqlite.SqliteDataReader reader, - List declarations, - CancellationToken cancellationToken) - { - while (reader.Read()) - { - declarations.Add(new TypeScriptInterfaceDeclaration( - reader.GetInt64(0), - reader.GetString(1), - reader.GetString(2), - reader.IsDBNull(3) ? 1 : Math.Max(1, reader.GetInt32(3)), - reader.IsDBNull(4) ? 1 : Math.Max(1, reader.GetInt32(4) + 1), - reader.IsDBNull(5) ? string.Empty : reader.GetString(5), - reader.IsDBNull(6) ? string.Empty : reader.GetString(6), - reader.IsDBNull(7) ? string.Empty : reader.GetString(7), - reader.IsDBNull(8) ? null : reader.GetString(8))); - if ((declarations.Count & 255) == 0) - cancellationToken.ThrowIfCancellationRequested(); - } - cancellationToken.ThrowIfCancellationRequested(); - } - - private Microsoft.Data.Sqlite.SqliteCommand CreateTypeScriptAugmentationNameCommand( - IReadOnlyList names, - int offset, - int count, - string sqlTemplate) - { - SqliteDynamicSql.EnsureParameterBudget(count, "TypeScript augmentation name batch"); - var command = _conn.CreateCommand(); - command.Transaction = _activeTransaction; - var parameterNames = new string[count]; - for (var index = 0; index < count; index++) - { - var parameterName = SqliteDynamicSql.BuildParameterName("augmentation_name", index); - parameterNames[index] = parameterName; - command.Parameters.Add(parameterName, Microsoft.Data.Sqlite.SqliteType.Text).Value = names[offset + index]; - } - command.CommandText = string.Format( - System.Globalization.CultureInfo.InvariantCulture, - sqlTemplate, - string.Join(", ", parameterNames)); - return command; - } - - private static void ForEachTypeScriptAugmentationNameBatch( - IReadOnlyList names, - CancellationToken cancellationToken, - Action, int, int> action) - { - const int nameBatchSize = 900; - for (var offset = 0; offset < names.Count; offset += nameBatchSize) - { - cancellationToken.ThrowIfCancellationRequested(); - action(names, offset, Math.Min(nameBatchSize, names.Count - offset)); - TypeScriptAugmentationNameBatchForTesting?.Invoke((offset / nameBatchSize) + 1); - cancellationToken.ThrowIfCancellationRequested(); - } - } - - private bool ShouldUseFullTypeScriptAugmentationRebuild( - int dirtyNameCount, - CancellationToken cancellationToken) - { - cancellationToken.ThrowIfCancellationRequested(); - var cmd = RentCommand( - @" - SELECT COUNT(*) - FROM symbols s INDEXED BY idx_symbols_kind - JOIN files f ON f.id = s.file_id - WHERE s.kind = 'interface' - AND f.lang = 'typescript'", - static _ => { }); - try - { - var declarationCount = Convert.ToInt64( - cmd.ExecuteScalar(), - System.Globalization.CultureInfo.InvariantCulture); - cancellationToken.ThrowIfCancellationRequested(); - return dirtyNameCount >= Math.Max(1_024L, (declarationCount + 1L) / 2L); - } - finally - { - ReleaseCommand(cmd); - } - } - private bool AddTypeScriptInterfaceNamesAtPath( string path, HashSet? destination, diff --git a/src/CodeIndex/Database/FindSnippetAssembler.cs b/src/CodeIndex/Database/FindSnippetAssembler.cs new file mode 100644 index 0000000000..e1668f2a55 --- /dev/null +++ b/src/CodeIndex/Database/FindSnippetAssembler.cs @@ -0,0 +1,102 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private static class FindSnippetAssembler + { + internal static void FlushReadyMatches( + FindCandidateFile file, + Queue pendingMatches, + Dictionary snippetLinesByNumber, + List results, + int maxLineWidth, + int availableThroughLine) + { + while (pendingMatches.Count > 0 + && pendingMatches.Peek().SnippetEnd <= availableThroughLine) + { + var pending = pendingMatches.Dequeue(); + var lineNumbers = Enumerable + .Range(pending.SnippetStart, pending.SnippetEnd - pending.SnippetStart + 1) + .Where(snippetLinesByNumber.ContainsKey) + .ToList(); + if (lineNumbers.Count == 0) + continue; + + var lines = lineNumbers.Select(line => snippetLinesByNumber[line]).ToList(); + var (snippet, truncationContext) = ClampSnippetLines( + lines, + maxLineWidth, + lineNumbers.IndexOf(pending.LineNumber), + pending.Column + 1, + pending.Length); + var matchLine = snippetLinesByNumber[pending.LineNumber]; + results.Add(new FileFindResult + { + Path = file.Path, + Lang = file.Lang, + Line = pending.LineNumber, + Column = pending.Column + 1, + Length = pending.Length, + OriginalLineLength = matchLine.Length, + StartLine = lineNumbers[0], + EndLine = lineNumbers[^1], + Snippet = snippet, + SnippetTruncated = truncationContext.LineCount > 0, + SnippetTruncationContext = truncationContext, + }); + } + } + + private static (string Text, FileFindSnippetTruncationContext Context) ClampSnippetLines( + IReadOnlyList lines, + int maxLineWidth, + int focusLineIndex, + int focusColumn, + int focusLength) + { + if (lines.Count == 0) + return (string.Empty, new FileFindSnippetTruncationContext()); + + var output = new string[lines.Count]; + var truncatedCharCounts = new List(); + for (var index = 0; index < lines.Count; index++) + { + var clamped = index == focusLineIndex + ? LineWidthFormatter.ClampLine(lines[index], maxLineWidth, focusColumn, focusLength) + : LineWidthFormatter.ClampLine(lines[index], maxLineWidth); + output[index] = clamped.Text; + if (clamped.Truncated) + truncatedCharCounts.Add(clamped.TruncatedCharCount); + } + + return ( + string.Join('\n', output), + new FileFindSnippetTruncationContext + { + LineCount = truncatedCharCounts.Count, + CharCounts = truncatedCharCounts, + TotalChars = truncatedCharCounts.Sum(), + Reason = truncatedCharCounts.Count > 0 ? "line_width" : null, + }); + } + + internal static void PruneWindow( + int currentLine, + int before, + Queue pendingMatches, + Queue snippetWindow, + Dictionary snippetLinesByNumber) + { + var minLineToKeep = currentLine - before; + if (pendingMatches.Count > 0) + minLineToKeep = Math.Min(minLineToKeep, pendingMatches.Peek().SnippetStart); + + while (snippetWindow.Count > 0 && snippetWindow.Peek().Number < minLineToKeep) + { + var removed = snippetWindow.Dequeue(); + snippetLinesByNumber.Remove(removed.Number); + } + } + } +} diff --git a/src/CodeIndex/Database/IndexedFindCollectors.cs b/src/CodeIndex/Database/IndexedFindCollectors.cs new file mode 100644 index 0000000000..6edf1c633d --- /dev/null +++ b/src/CodeIndex/Database/IndexedFindCollectors.cs @@ -0,0 +1,213 @@ +using System.Text; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class FindResultCollector( + IndexedFindListRequest request, + FindScanState state) : IFindScanCollector + { + private readonly List _results = []; + private readonly Queue _pendingMatches = []; + private readonly Queue _snippetWindow = []; + private readonly Dictionary _snippetLinesByNumber = []; + private int _acceptedMatches; + private int _matchesSkipped; + private bool _resumeMatchPending = request.Scan.Resume.MatchOrdinal.HasValue; + + public FindScanMode Mode => FindScanMode.Results; + public int ContextBefore => request.Before; + public bool NotifyLineScanned => true; + public bool ShouldStopBeforeCandidate + => !request.CaptureContinuation && _results.Count >= request.Limit; + public bool ShouldStopAfterFile + => state.Truncated + || state.ResultLimitReached + || !request.CaptureContinuation && _results.Count >= request.Limit; + + public void BeginFile(FindCandidateFile file) + { + _pendingMatches.Clear(); + _snippetWindow.Clear(); + _snippetLinesByNumber.Clear(); + _acceptedMatches = _results.Count; + } + + public void ObserveLine(IndexedLine line) + { + _snippetWindow.Enqueue(line); + _snippetLinesByNumber[line.Number] = line.Text; + } + + public bool ShouldEvaluateMatches(IndexedLine line, bool eligibleForMatch) + => !state.ResultLimitReached + && (_matchesSkipped < request.Offset + || request.CaptureContinuation && _acceptedMatches <= request.Limit + || !request.CaptureContinuation && _acceptedMatches < request.Limit) + && eligibleForMatch + && (!request.Scan.FocusLine.HasValue + || line.Number == request.Scan.FocusLine.Value); + + public bool AcceptMatch( + FindCandidateFile file, + IndexedLine line, + FindLineMatch match, + int matchOrdinal) + { + if (_resumeMatchPending && line.Number == file.FirstEligibleLine) + { + if (matchOrdinal < request.Scan.Resume.MatchOrdinal!.Value) + return false; + var boundaryByteOffset = Encoding.UTF8.GetByteCount( + line.Text.AsSpan(0, match.Column)); + if (matchOrdinal != request.Scan.Resume.MatchOrdinal.Value + || boundaryByteOffset != request.Scan.Resume.ByteOffset) + { + throw new FindContinuationException( + "cursor_malformed", + "find cursor match position is not a record boundary for the current query."); + } + _resumeMatchPending = false; + } + + if (_matchesSkipped < request.Offset) + { + _matchesSkipped++; + return false; + } + if (_acceptedMatches >= request.Limit) + { + state.SetResultLimit( + file.Path, + line.Number, + file.Ordinal, + matchOrdinal, + Encoding.UTF8.GetByteCount(line.Text.AsSpan(0, match.Column))); + return true; + } + + _pendingMatches.Enqueue(new PendingFileFindMatch( + line.Number, + match.Column, + match.Length, + Math.Max(1, line.Number - request.Before), + Math.Min(file.TotalLines, line.Number + request.After))); + _acceptedMatches++; + return false; + } + + public void CompleteLine(FindCandidateFile file, IndexedLine line) + { + FindSnippetAssembler.FlushReadyMatches( + file, + _pendingMatches, + _snippetLinesByNumber, + _results, + request.MaxLineWidth, + line.Number); + FindSnippetAssembler.PruneWindow( + line.Number, + request.Before, + _pendingMatches, + _snippetWindow, + _snippetLinesByNumber); + } + + public bool CanStopAfterLine(bool stopScanning) + => (stopScanning + || !request.CaptureContinuation && _results.Count >= request.Limit) + && _pendingMatches.Count == 0; + + public void CompleteFile(FindCandidateFile file) + => FindSnippetAssembler.FlushReadyMatches( + file, + _pendingMatches, + _snippetLinesByNumber, + _results, + request.MaxLineWidth, + int.MaxValue); + + public void ValidateCompletedScan() + { + if (_resumeMatchPending) + { + throw new FindContinuationException( + "cursor_malformed", + "find cursor position does not exist in the current result sequence."); + } + } + + internal FindResults CreateResult(FindSearchPlan searchPlan) + => new( + _results, + state.CreateSummary( + searchPlan, + request.Scan.MaxCandidateFiles, + request.Scan.MaxLinesScanned)); + } + + private sealed class FindCountCollector( + IndexedFindScanRequest request, + FindScanState state) : IFindScanCollector + { + private int _fileMatches; + private int _count; + private int _fileCount; + + public FindScanMode Mode => FindScanMode.Count; + public int ContextBefore => 0; + public bool NotifyLineScanned => false; + public bool ShouldStopBeforeCandidate => false; + public bool ShouldStopAfterFile => false; + + public void BeginFile(FindCandidateFile file) + => _fileMatches = 0; + + public void ObserveLine(IndexedLine line) + { + } + + public bool ShouldEvaluateMatches(IndexedLine line, bool eligibleForMatch) + => eligibleForMatch + && (!request.FocusLine.HasValue || line.Number == request.FocusLine.Value); + + public bool AcceptMatch( + FindCandidateFile file, + IndexedLine line, + FindLineMatch match, + int matchOrdinal) + { + _fileMatches++; + return false; + } + + public void CompleteLine(FindCandidateFile file, IndexedLine line) + { + } + + public bool CanStopAfterLine(bool stopScanning) + => stopScanning; + + public void CompleteFile(FindCandidateFile file) + { + if (_fileMatches <= 0) + return; + _count += _fileMatches; + _fileCount++; + } + + public void ValidateCompletedScan() + { + } + + internal FindCountResult CreateResult(FindSearchPlan searchPlan) + => new( + _count, + _fileCount, + state.CreateSummary( + searchPlan, + request.MaxCandidateFiles, + request.MaxLinesScanned)); + } +} diff --git a/src/CodeIndex/Database/IndexedFindModels.cs b/src/CodeIndex/Database/IndexedFindModels.cs new file mode 100644 index 0000000000..e115af40fb --- /dev/null +++ b/src/CodeIndex/Database/IndexedFindModels.cs @@ -0,0 +1,172 @@ +namespace CodeIndex.Database; + +public partial class DbReader +{ + private readonly record struct FindResumePosition( + string? Path, + int? Line, + int? FileOrdinal, + int? MatchOrdinal, + int? ByteOffset); + + private sealed record IndexedFindScanRequest( + string Query, + string? Lang, + IReadOnlyList? PathPatterns, + IReadOnlyList? ExcludePathPatterns, + bool ExcludeTests, + bool Exact, + int? FocusLine, + int? FocusColumn, + bool Regex, + int? MaxCandidateFiles, + int? MaxLinesScanned, + bool UseIndexedLiteralCandidates, + FindResumePosition Resume, + CancellationToken CancellationToken); + + private sealed record IndexedFindListRequest( + IndexedFindScanRequest Scan, + int Limit, + int Before, + int After, + int MaxLineWidth, + int Offset, + bool CaptureContinuation); + + private interface IFindScanCollector + { + FindScanMode Mode { get; } + int ContextBefore { get; } + bool NotifyLineScanned { get; } + bool ShouldStopBeforeCandidate { get; } + bool ShouldStopAfterFile { get; } + void BeginFile(FindCandidateFile file); + void ObserveLine(IndexedLine line); + bool ShouldEvaluateMatches(IndexedLine line, bool eligibleForMatch); + bool AcceptMatch( + FindCandidateFile file, + IndexedLine line, + FindLineMatch match, + int matchOrdinal); + void CompleteLine(FindCandidateFile file, IndexedLine line); + bool CanStopAfterLine(bool stopScanning); + void CompleteFile(FindCandidateFile file); + void ValidateCompletedScan(); + } + + private enum FindScanMode + { + Results, + Count, + } + + private readonly record struct FindSearchPlan( + string Strategy, + string? FallbackReason, + string? TrigramMatchExpression); + + private readonly record struct FindCandidateFile( + long Id, + string Path, + string? Lang, + int TotalLines, + int Ordinal, + int FirstEligibleLine); + + private readonly record struct IndexedLine(int Number, string Text); + + private readonly record struct FindLineMatch(int Column, int Length); + + private readonly record struct PendingFileFindMatch( + int LineNumber, + int Column, + int Length, + int SnippetStart, + int SnippetEnd); + + private sealed class FindScanState(FindResumePosition resume) + { + internal int CandidateFiles { get; set; } + internal int FilesScanned { get; set; } + internal int LinesScanned { get; set; } + internal int CandidateFileOrdinal { get; set; } = -1; + internal bool ResumePending { get; set; } = resume.Path is not null; + internal bool Truncated { get; private set; } + internal string? TruncationReason { get; private set; } + internal string? NextPath { get; private set; } + internal int? NextLine { get; private set; } + internal int? NextFileOrdinal { get; private set; } + internal int? NextMatchOrdinal { get; private set; } + internal int? NextByteOffset { get; private set; } + internal bool ResultLimitReached { get; private set; } + + internal void Truncate( + string reason, + string path, + int nextLine, + int fileOrdinal) + { + Truncated = true; + TruncationReason ??= reason; + NextPath = path; + NextLine = nextLine; + NextFileOrdinal = fileOrdinal; + NextByteOffset = 0; + } + + internal void TruncateIfUnset( + string reason, + string path, + int nextLine, + int fileOrdinal) + { + Truncated = true; + TruncationReason ??= reason; + if (NextPath != null) + return; + NextPath = path; + NextLine = nextLine; + NextFileOrdinal = fileOrdinal; + NextByteOffset = 0; + } + + internal void SetResultLimit( + string path, + int line, + int fileOrdinal, + int matchOrdinal, + int byteOffset) + { + NextPath = path; + NextLine = line; + NextFileOrdinal = fileOrdinal; + NextMatchOrdinal = matchOrdinal; + NextByteOffset = byteOffset; + ResultLimitReached = true; + } + + internal FindScanSummary CreateSummary( + FindSearchPlan searchPlan, + int? candidateFileLimit, + int? lineLimit) + => new( + CandidateFiles, + FilesScanned, + LinesScanned, + Truncated, + CapReached: Truncated, + TimedOut: false, + TruncationReason, + candidateFileLimit, + lineLimit, + searchPlan.Strategy, + searchPlan.FallbackReason, + NextPath, + NextLine, + NextFileOrdinal, + NextMatchOrdinal, + NextByteOffset, + ResultLimitReached); + } +} diff --git a/src/CodeIndex/Database/IndexedFindPipeline.cs b/src/CodeIndex/Database/IndexedFindPipeline.cs new file mode 100644 index 0000000000..7312977c0d --- /dev/null +++ b/src/CodeIndex/Database/IndexedFindPipeline.cs @@ -0,0 +1,320 @@ +using System.Text.RegularExpressions; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class IndexedFindPipeline(DbReader owner) + { + private readonly DbReader _owner = owner; + private readonly IndexedFindQuerySource _querySource = new(owner); + + internal FindResults Find(IndexedFindListRequest request) + { + var scan = request.Scan; + scan.CancellationToken.ThrowIfCancellationRequested(); + if (string.IsNullOrWhiteSpace(scan.Query) || request.Limit <= 0) + return new FindResults([], new FindScanSummary(0, 0, 0)); + ValidateContinuation(scan.Resume); + + var normalizedRequest = request with + { + Before = Math.Max(0, request.Before), + After = Math.Max(0, request.After), + MaxLineWidth = LineWidthFormatter.ClampMaxLineWidth(request.MaxLineWidth), + Offset = Math.Max(0, request.Offset), + }; + var state = new FindScanState(scan.Resume); + var collector = new FindResultCollector(normalizedRequest, state); + var searchPlan = ScanFiles(scan, collector, state); + ValidateCompletedScan(collector.Mode, state, collector); + return collector.CreateResult(searchPlan); + } + + internal FindCountResult Count(IndexedFindScanRequest request) + { + request.CancellationToken.ThrowIfCancellationRequested(); + if (string.IsNullOrWhiteSpace(request.Query)) + return new FindCountResult(0, 0, new FindScanSummary(0, 0, 0)); + ValidateContinuation(request.Resume); + if (request.Resume.MatchOrdinal.HasValue || request.Resume.ByteOffset is not (null or 0)) + { + throw new FindContinuationException( + "cursor_malformed", + "find count cursor must resume at a line boundary."); + } + + var state = new FindScanState(request.Resume); + var collector = new FindCountCollector(request, state); + var searchPlan = ScanFiles(request, collector, state); + ValidateCompletedScan(collector.Mode, state, collector); + return collector.CreateResult(searchPlan); + } + + private FindSearchPlan ScanFiles( + IndexedFindScanRequest request, + IFindScanCollector collector, + FindScanState state) + { + var comparison = request.Exact ? StringComparison.Ordinal : StringComparison.OrdinalIgnoreCase; + var regexMatcher = request.Regex + ? CreateFindRegexMatcher(request.Query, request.Exact) + : null; + var searchPlan = _querySource.CreateSearchPlan(request); + using var fileCommand = _querySource.CreateFileCommand(searchPlan, request); + state.CandidateFiles = _owner.CountFindCandidateFiles( + request.Lang, + request.PathPatterns, + request.ExcludePathPatterns, + request.ExcludeTests); + + using var fileReader = fileCommand.ExecuteTrackedReader(); + while (fileReader.TrackedRead()) + { + request.CancellationToken.ThrowIfCancellationRequested(); + if (collector.ShouldStopBeforeCandidate) + break; + + state.CandidateFileOrdinal++; + var file = ReadCandidateFile(fileReader, state.CandidateFileOrdinal); + if (!TryReachResumeFile(request.Resume, collector.Mode, file, state)) + continue; + if (ReachedCandidateFileLimit(request, file, state)) + break; + + var firstEligibleLine = string.Equals(file.Path, request.Resume.Path, StringComparison.Ordinal) + ? Math.Max(1, request.Resume.Line ?? 1) + : 1; + file = file with { FirstEligibleLine = firstEligibleLine }; + state.FilesScanned++; + if (file.TotalLines <= 0) + continue; + + collector.BeginFile(file); + var searchQuery = request.Exact && !request.Regex + ? ExactSourceSearchNormalizer.Normalize(request.Query, file.Lang) + : request.Query; + var stopScanning = ScanFileLines( + request, + collector, + state, + file, + searchQuery, + comparison, + regexMatcher); + collector.CompleteFile(file); + if (stopScanning || collector.ShouldStopAfterFile) + break; + } + + return searchPlan; + } + + private bool ScanFileLines( + IndexedFindScanRequest request, + IFindScanCollector collector, + FindScanState state, + FindCandidateFile file, + string searchQuery, + StringComparison comparison, + Regex? regexMatcher) + { + var firstContextLine = Math.Max(1, file.FirstEligibleLine - collector.ContextBefore); + var stopScanning = false; + foreach (var indexedLine in _querySource.EnumerateIndexedFileLines(file.Id)) + { + request.CancellationToken.ThrowIfCancellationRequested(); + if (indexedLine.Number > file.TotalLines) + break; + if (indexedLine.Number < firstContextLine) + continue; + + var eligibleForMatch = indexedLine.Number >= file.FirstEligibleLine; + if (ReachedLineScanLimit(request, file, indexedLine, state)) + { + stopScanning = true; + break; + } + RecordScannedLine(request, collector, state, eligibleForMatch); + collector.ObserveLine(indexedLine); + + if (collector.ShouldEvaluateMatches(indexedLine, eligibleForMatch)) + { + var matchOrdinal = 0; + foreach (var lineMatch in IndexedFindQuerySource.EnumerateLineMatches( + indexedLine.Text, + file.Lang, + searchQuery, + comparison, + regexMatcher, + request.Exact && !request.Regex, + request.FocusColumn)) + { + if (collector.AcceptMatch(file, indexedLine, lineMatch, matchOrdinal)) + { + stopScanning = true; + break; + } + matchOrdinal++; + } + } + + collector.CompleteLine(file, indexedLine); + if (collector.CanStopAfterLine(stopScanning)) + break; + } + return stopScanning; + } + + private static void RecordScannedLine( + IndexedFindScanRequest request, + IFindScanCollector collector, + FindScanState state, + bool eligibleForMatch) + { + if (!eligibleForMatch) + return; + + state.LinesScanned++; + if (!collector.NotifyLineScanned) + return; + FindLineScannedForTesting?.Invoke(); + request.CancellationToken.ThrowIfCancellationRequested(); + } + + private static FindCandidateFile ReadCandidateFile(SqliteDataReader reader, int ordinal) + => new( + reader.GetInt64(0), + reader.GetString(1), + GetNullableString(reader, 2), + reader.GetInt32(3), + ordinal, + FirstEligibleLine: 1); + + private static bool TryReachResumeFile( + FindResumePosition resume, + FindScanMode mode, + FindCandidateFile file, + FindScanState state) + { + if (!state.ResumePending) + return true; + + if (resume.FileOrdinal.HasValue) + { + if (file.Ordinal < resume.FileOrdinal.Value) + return false; + if (file.Ordinal != resume.FileOrdinal.Value + || !string.Equals(file.Path, resume.Path, StringComparison.Ordinal)) + { + throw new FindContinuationException( + "cursor_malformed", + mode == FindScanMode.Count + ? "find count cursor file position does not match the current candidate order." + : "find cursor file position does not match the current candidate order."); + } + } + else if (!string.Equals(file.Path, resume.Path, StringComparison.Ordinal)) + { + return false; + } + + if (resume.Line > Math.Max(1, file.TotalLines)) + { + throw new FindContinuationException( + "cursor_malformed", + mode == FindScanMode.Count + ? "find count cursor line position exceeds the selected file." + : "find cursor line position exceeds the selected file."); + } + state.ResumePending = false; + return true; + } + + private static bool ReachedCandidateFileLimit( + IndexedFindScanRequest request, + FindCandidateFile file, + FindScanState state) + { + if (!request.MaxCandidateFiles.HasValue + || state.FilesScanned < request.MaxCandidateFiles.Value) + { + return false; + } + + state.Truncate( + "candidate_file_limit", + file.Path, + nextLine: 1, + file.Ordinal); + return true; + } + + private static bool ReachedLineScanLimit( + IndexedFindScanRequest request, + FindCandidateFile file, + IndexedLine indexedLine, + FindScanState state) + { + if (!request.MaxLinesScanned.HasValue + || state.LinesScanned < request.MaxLinesScanned.Value) + { + return false; + } + + state.TruncateIfUnset( + "line_scan_limit", + file.Path, + indexedLine.Number, + file.Ordinal); + return true; + } + + private static void ValidateContinuation(FindResumePosition resume) + { + if (resume.Path is null) + { + if (resume.Line.HasValue + || resume.FileOrdinal.HasValue + || resume.MatchOrdinal.HasValue + || resume.ByteOffset.HasValue) + { + throw new FindContinuationException( + "cursor_malformed", + "find cursor continuation fields require a resume path."); + } + return; + } + + if (!resume.Line.HasValue + || resume.Line.Value <= 0 + || resume.FileOrdinal is < 0 + || resume.MatchOrdinal is < 0 + || resume.ByteOffset is < 0 + || resume.MatchOrdinal.HasValue && !resume.ByteOffset.HasValue + || !resume.MatchOrdinal.HasValue && resume.ByteOffset is not (null or 0)) + { + throw new FindContinuationException( + "cursor_malformed", + "find cursor continuation position is invalid."); + } + } + + private static void ValidateCompletedScan( + FindScanMode mode, + FindScanState state, + IFindScanCollector collector) + { + if (state.ResumePending) + { + throw new FindContinuationException( + "cursor_malformed", + mode == FindScanMode.Count + ? "find count cursor position does not exist in the current candidate sequence." + : "find cursor position does not exist in the current result sequence."); + } + collector.ValidateCompletedScan(); + } + } +} diff --git a/src/CodeIndex/Database/IndexedFindQuerySource.cs b/src/CodeIndex/Database/IndexedFindQuerySource.cs new file mode 100644 index 0000000000..79bf3151fb --- /dev/null +++ b/src/CodeIndex/Database/IndexedFindQuerySource.cs @@ -0,0 +1,193 @@ +using System.Text.RegularExpressions; +using Microsoft.Data.Sqlite; + +namespace CodeIndex.Database; + +public partial class DbReader +{ + private sealed class IndexedFindQuerySource(DbReader owner) + { + private readonly DbReader _owner = owner; + + internal FindSearchPlan CreateSearchPlan(IndexedFindScanRequest request) + { + if (!request.UseIndexedLiteralCandidates) + return new FindSearchPlan("line_scan", null, null); + if (request.Regex) + return new FindSearchPlan("line_scan", "regex", null); + if (request.Exact) + return new FindSearchPlan("line_scan", "exact_source_normalization", null); + if (request.Query.Length < 3) + return new FindSearchPlan("line_scan", "query_too_short", null); + if (request.Query.Any(character => character < ' ' || character > '~')) + return new FindSearchPlan("line_scan", "unsupported_query_characters", null); + if (!_owner.HasTable(DbContext.FtsChunksTrigramTableName)) + return new FindSearchPlan("line_scan", "trigram_index_unavailable", null); + if (DbWriter.IsFtsBulkLoadMarkerSet(_owner.GetMetaString(DbWriter.FtsBulkLoadInProgressMetaKey))) + return new FindSearchPlan("line_scan", "trigram_index_rebuilding", null); + if (!HasAllTrigramFtsSyncTriggers()) + return new FindSearchPlan("line_scan", "trigram_index_unsynchronized", null); + + var phrase = "\"" + request.Query.Replace("\"", "\"\"", StringComparison.Ordinal) + "\""; + return new FindSearchPlan("indexed_trigram", null, phrase); + } + + private bool HasAllTrigramFtsSyncTriggers() + { + using var command = _owner._conn.CreateCommand(); + command.CommandText = DbContext.CountFtsChunksTrigramSyncTriggersSql; + return SqliteCommandPolicy.ReadInt32Scalar( + command, + "find trigram FTS synchronization trigger count") == 3; + } + + internal SqliteCommand CreateFileCommand( + FindSearchPlan searchPlan, + IndexedFindScanRequest request) + { + var command = _owner._conn.CreateCommand(); + string sql; + if (searchPlan.TrigramMatchExpression != null) + { + sql = $""" + SELECT f.id, f.path, f.lang, f.lines + FROM ( + SELECT DISTINCT find_chunk.file_id + FROM {DbContext.FtsChunksTrigramTableName} + JOIN chunks find_chunk ON find_chunk.id = {DbContext.FtsChunksTrigramTableName}.rowid + WHERE {DbContext.FtsChunksTrigramTableName} MATCH @trigramQuery + ) find_candidate + JOIN files f ON f.id = find_candidate.file_id + WHERE 1=1 + """; + } + else + { + sql = "SELECT f.id, f.path, f.lang, f.lines FROM files f WHERE 1=1"; + } + if (request.Lang != null) + sql += " AND f.lang = @lang"; + AppendPathFilters( + ref sql, + request.PathPatterns, + request.ExcludePathPatterns, + request.ExcludeTests); + sql += $" ORDER BY {PathBucketOrder}, f.path"; + command.CommandText = sql; + if (searchPlan.TrigramMatchExpression != null) + SqliteCommandPolicy.AddText(command, "@trigramQuery", searchPlan.TrigramMatchExpression); + if (request.Lang != null) + SqliteCommandPolicy.Add(command, "@lang", request.Lang); + AddPathFilterParameters( + command, + request.PathPatterns, + request.ExcludePathPatterns); + return command; + } + + internal IEnumerable EnumerateIndexedFileLines(long fileId) + { + using var command = _owner._conn.CreateCommand(); + command.CommandText = @" + SELECT c.start_line, c.end_line, c.content + FROM chunks c + WHERE c.file_id = @fileId + ORDER BY c.start_line, c.chunk_index"; + SqliteCommandPolicy.Add(command, "@fileId", fileId); + + var lastEmittedLine = 0; + using var reader = command.ExecuteTrackedReader(); + while (reader.TrackedRead()) + { + var chunkStartLine = reader.GetInt32(0); + var chunkEndLine = reader.GetInt32(1); + var chunkLines = reader.GetString(2).Split('\n'); + var lineCount = chunkEndLine - chunkStartLine + 1; + + for (var index = 0; index < chunkLines.Length && index < lineCount; index++) + { + var absoluteLine = chunkStartLine + index; + if (absoluteLine <= lastEmittedLine) + continue; + + lastEmittedLine = absoluteLine; + yield return new IndexedLine(absoluteLine, chunkLines[index]); + } + } + } + + internal static IEnumerable EnumerateLineMatches( + string lineText, + string? fileLang, + string searchQuery, + StringComparison comparison, + Regex? regexMatcher, + bool normalizeExactSource, + int? focusColumn) + { + int[]? rawIndexMap = null; + var searchLine = normalizeExactSource + ? ExactSourceSearchNormalizer.Normalize(lineText, fileLang, out rawIndexMap) + : lineText; + + if (regexMatcher != null) + { + foreach (Match match in regexMatcher.Matches(searchLine)) + { + if (!match.Success) + continue; + if (TryCreateLineMatch( + match.Index, + match.Length, + rawIndexMap, + focusColumn, + out var lineMatch)) + { + yield return lineMatch; + } + } + yield break; + } + + for (var searchStart = 0; searchStart < searchLine.Length;) + { + var matchColumn = searchLine.IndexOf(searchQuery, searchStart, comparison); + if (matchColumn < 0) + break; + + if (TryCreateLineMatch( + matchColumn, + searchQuery.Length, + rawIndexMap, + focusColumn, + out var lineMatch)) + { + yield return lineMatch; + } + searchStart = matchColumn + 1; + } + } + + private static bool TryCreateLineMatch( + int matchColumn, + int matchLength, + int[]? rawIndexMap, + int? focusColumn, + out FindLineMatch lineMatch) + { + var rawMatchColumn = rawIndexMap == null ? matchColumn : rawIndexMap[matchColumn]; + var rawMatchLength = matchLength; + if (rawIndexMap != null && matchLength > 0) + { + var rawMatchEndIndex = rawIndexMap[matchColumn + matchLength - 1]; + rawMatchLength = rawMatchEndIndex - rawMatchColumn + 1; + } + + lineMatch = new FindLineMatch(rawMatchColumn, rawMatchLength); + var focusEndColumn = rawMatchColumn + Math.Max(1, rawMatchLength); + return !focusColumn.HasValue + || focusColumn.Value >= rawMatchColumn + 1 + && focusColumn.Value <= focusEndColumn; + } + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCSharpDocumentationLines.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCSharpDocumentationLines.cs new file mode 100644 index 0000000000..099237f4a0 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCSharpDocumentationLines.cs @@ -0,0 +1,78 @@ +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static void EmitCoreCSharpDocumentationReferences( + in CoreDocumentationLineContext line, + ref bool inDelimitedDocComment) + { + if (line.CSharpLinesInsideMultilineStringContent == null + || line.CSharpLinesInsideMultilineStringContent[line.LineIndex] + || !TryGetCSharpXmlDocCommentSpan( + line.OriginalLine, + inDelimitedDocComment, + line.CSharpLinesInsideBlockComment?[line.LineIndex] ?? false, + out var docCommentStartIndex, + out var docCommentEndExclusive, + out var nextDelimitedDocComment)) + { + return; + } + + var docCommentText = line.OriginalLine[docCommentStartIndex..docCommentEndExclusive]; + if (docCommentText.IndexOf("cref=\"", StringComparison.OrdinalIgnoreCase) >= 0) + { + EmitCoreCSharpDocCrefReferences( + in line, + docCommentText, + docCommentStartIndex, + docCommentEndExclusive, + nextDelimitedDocComment); + } + + inDelimitedDocComment = nextDelimitedDocComment; + } + + private static void EmitCoreCSharpDocCrefReferences( + in CoreDocumentationLineContext line, + string docCommentText, + int docCommentStartIndex, + int docCommentEndExclusive, + bool nextDelimitedDocComment) + { + var innermostContainer = line.ContainerResolver.Find(line.LineNumber); + var sameLineDeclarationStartColumn = GetCSharpSameLineDocumentedDeclarationStartColumn( + line.OriginalLine, + docCommentEndExclusive, + nextDelimitedDocComment); + var docContainer = FindDocumentedContainer( + line.ContainerCandidates, + line.StructuralLines[line.LineIndex], + line.PreparedLine, + line.CSharpAttributeRangesOnLine, + line.LineNumber, + sameLineDeclarationStartColumn); + if (docContainer == null + || (docContainer.StartLine != line.LineNumber + && !CanAttachCSharpXmlDocCommentToNextDeclaration( + innermostContainer, + line.Lookups.GetCSharpXmlDocAttachmentScopeCandidates(), + line.CSharpAttributeRanges, + line.PreparedLines, + line.LineNumber, + docContainer))) + { + return; + } + + CSharpReferenceExtractor.EmitDocCrefReferences( + docCommentText, + line.References, + line.Seen, + line.FileId, + docCommentStartIndex, + docCommentText.Trim(), + line.LineNumber, + docContainer); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallAdditionalReferences.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallAdditionalReferences.cs new file mode 100644 index 0000000000..7212a9fab8 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallAdditionalReferences.cs @@ -0,0 +1,261 @@ +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static void EmitRubyOrPerlCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + if (line.Language == "ruby") + { + var matchedCallIndices = GetOrCreateMatchedCallIndices(ref state); + var addCallLikeReference = GetOrCreateCallLikeReferenceEmitter( + in call, + ref state); + RubyReferenceExtractor.EmitAdditionalCallReferences( + line.PreparedLine, + line.OriginalLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + matchedCallIndices, + addCallLikeReference); + } + else if (line.Language is "perl" or "ambiguous_pl") + { + PerlReferenceExtractor.EmitAdditionalReferences( + line.Language == "ambiguous_pl" + ? state.CallScanLine + : line.PreparedLine, + line.OriginalLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + emitArrowCallReferences: line.Language != "ambiguous_pl" + || call.DynamicDeclarativeState?.HasPrologContainer( + line.LineNumber) != true); + } + } + + private static void EmitDynamicDeclarativeCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + if (call.DynamicDeclarativeState == null) + return; + + ref readonly var line = ref call.Line; + DynamicDeclarativeReferenceExtractor.EmitAdditionalReferences( + line.Language, + state.CallScanLine, + call.ReferenceStructuralLine, + call.DynamicDeclarativeState, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + GetOrCreateCallLikeReferenceEmitter(in call, ref state)); + } + + private static void EmitGoCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + if (line.Language != "go") + return; + + LanguageReferenceExtractionSupport.EmitGoBranchLabelReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + line.References); + } + + private static void EmitSwiftOrKotlinCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + if (line.Language == "swift") + { + SwiftReferenceExtractor.EmitTrailingClosureReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + line.References); + } + else if (line.Language == "kotlin") + { + var addCallLikeReference = GetOrCreateCallLikeReferenceEmitter( + in call, + ref state); + KotlinReferenceExtractor.EmitInfixCallReferences( + line.PreparedLine, + line.OriginalLine, + call.KotlinInfixFunctionNames!, + addCallLikeReference); + KotlinReferenceExtractor.EmitTrailingLambdaReferences( + line.PreparedLine, + addCallLikeReference, + line.References); + } + } + + private static void EmitFSharpCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + if (line.Language != "fsharp") + return; + + FSharpReferenceExtractor.EmitAdditionalCallReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + line.References); + } + + private static void EmitScalaOrGradleCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + if (line.Language == "scala") + { + var addCallLikeReference = GetOrCreateCallLikeReferenceEmitter( + in call, + ref state); + ScalaReferenceExtractor.EmitTrailingBlockCallReferences( + line.PreparedLine, + addCallLikeReference, + line.References); + ScalaReferenceExtractor.EmitAdditionalReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + addCallLikeReference); + } + else if (line.Language == "gradle") + { + GradleReferenceExtractor.EmitDslCallReferences( + line.PreparedLine, + CreateGradleDslReferenceEmitter(in line), + line.References); + } + } + + private static void EmitSecondaryLanguageCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + if (line.Language == "fortran") + { + FortranReferenceExtractor.EmitAdditionalCallReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state)); + } + else if (line.Language == "pascal") + { + PascalReferenceExtractor.EmitAdditionalCallReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + line.DefinitionNames); + } + else if (line.Language == "objc") + { + ObjectiveCReferenceExtractor.EmitAdditionalCallReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + } + else if (line.Language == "haskell") + { + HaskellReferenceExtractor.EmitAdditionalCallReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + line.DefinitionNames); + } + else if (line.Language == "elixir") + { + ElixirReferenceExtractor.EmitAdditionalCallReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + line.DefinitionNames); + } + else if (line.Language == "lua") + { + LuaReferenceExtractor.EmitAdditionalCallReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + line.DefinitionNames); + } + else if (line.Language == "smalltalk") + { + SmalltalkReferenceExtractor.EmitAdditionalCallReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + line.DefinitionNames); + } + else if (line.Language == "vb") + { + LanguageReferenceExtractionSupport.EmitAdditionalCallReferences( + "vb", + line.PreparedLine, + line.OriginalLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + line.DefinitionNames); + } + } + + private static void EmitRustPostCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + if (line.Language != "rust") + return; + + RustReferenceExtractor.EmitAdditionalCallReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state)); + RustReferenceExtractor.EmitAttributeReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.Container); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallClassification.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallClassification.cs index a32283f95d..dcb77e8fe6 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallClassification.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallClassification.cs @@ -5,217 +5,47 @@ namespace CodeIndex.Indexer; public static partial class ReferenceExtractor { private static bool TryAddCoreCallLikeReference( - CoreCallReferenceContext call, + in CoreCallReferenceContext call, string name, int callIndex, + bool isScientificNativeLanguage, string? targetQualifier = null) { - var line = call.Line; - var normalizedName = - line.Language == "fsharp" - && FSharpReferenceExtractor.IsOperatorCallName(name) - ? $"operator {name}" - : line.Language == "rust" - ? RustReferenceExtractor.NormalizeIdentifier(name) - : NormalizeAtPrefixedIdentifier(name); - - // In tuple-return declarations such as `private static (int Value, string Error) - // Resolve(...)`, CallRegex sees the modifier token as `static(`. It is a C# keyword, - // never a callable identifier, so suppress the phantom edge before graph ingestion. - // `private static (int Value, string Error) Resolve(...)` のような tuple return 宣言では - // CallRegex が modifier を `static(` と誤認する。C# keyword は呼び出し対象にならないため、 - // graph に入る前に phantom edge を除外する。 - if (line.Language == "csharp" && name == "static") - return false; - - if (line.Language == "rust" - && RustReferenceExtractor.IsFunctionDeclarationCallSite( - line.PreparedLine, - callIndex)) - { - return false; - } - if (line.Language == "rust" - && RustReferenceExtractor.IsDeriveAttributeCallSite( - line.PreparedLine, + ref readonly var line = ref call.Line; + var normalizedName = NormalizeCoreCallName( + line.Language, + name); + if (ShouldSuppressCoreCallCandidate( + in call, + name, normalizedName, callIndex)) { return false; } - if (line.Language == "wgsl" && name.StartsWith('@')) - return false; - if (line.Language == "kotlin" - && KotlinReferenceExtractor.IsInfixFunctionDeclarationSite( - line.PreparedLine, - callIndex)) - { - return false; - } - - // Suppress the same-line Java ctor declarator's self-call. CallRegex matches - // `CtorName(` at the declarator once per same-line ctor, but it is a declaration - // site — not a call — so attributing it to `class:CtorName` produces a phantom - // `CtorName|call|class|CtorName` edge. `line.DefinitionNames` does not cover this - // because same-line ctors do not appear in the symbol table. - // 同一行 ctor の宣言子 `CtorName(` は呼び出しではないため CallRegex の対象から除外する。 - if (call.JavaSameLineCtor != null - && callIndex == call.JavaSameLineCtor.Value.NameIndex - && string.Equals( - normalizedName, - call.JavaSameLineCtor.Value.Synthetic.Name, - StringComparison.Ordinal)) - { - return false; - } - // C# positional patterns such as `case Point(var x, var y):` are type-pattern - // heads, not calls. `CallRegex` still sees `Point(` and would otherwise emit a - // phantom `call` edge alongside the real `type_reference`. - // C# の positional pattern (`case Point(var x, var y):`) は型パターンの先頭であり、 - // 呼び出しではない。`CallRegex` が `Point(` を拾ってしまうため、そのままだと - // 本物の `type_reference` に加えて phantom な `call` エッジが出る。 - if (line.Language == "csharp" - && CSharpReferenceExtractor.IsPatternHeadCallSite( - line.PreparedLines, - line.LineIndex, - line.PreparedLine, - callIndex)) - { - return false; - } - if (line.Language == "typescript" - && TypeScriptReferenceExtractor.IsSatisfiesTypeOperand( - line.PreparedLine, - callIndex)) - { - return false; - } - if (call.Definitions.ShouldSuppressDefinitionCall( - normalizedName, + var callContainer = line.ResolveContainerForCall(callIndex); + if (TryHandleCSharpValueReceiverCall( + in call, name, - callIndex)) + normalizedName, + callIndex, + callContainer)) { return false; } - var callContainer = line.ResolveContainerForCall(callIndex); - if (line.Language == "csharp" - && callIndex + name.Length < line.PreparedLine.Length - && line.PreparedLine.AsSpan(callIndex + name.Length) - .TrimStart() - .StartsWith(".", StringComparison.Ordinal)) - { - var receiverLookups = call.Lookups.GetCSharpValueReceiverLookups(); - if (HasCSharpValueReceiverConflict( - normalizedName, - normalizedName, - line.LineNumber, - callIndex, - callContainer, - receiverLookups.ByContainingType, - receiverLookups.ByFunctionStartLine)) - { - var containingType = - GetContainingTypeQualifiedName(callContainer); - if (containingType != null - && receiverLookups.ByContainingType.TryGetValue( - containingType, - out var receiverNames) - && (receiverNames.InstanceNames.Contains(normalizedName) - || receiverNames.StaticNames.Contains(normalizedName)) - && call.Lookups.HasCSharpFieldOrPropertyMember( - containingType, - normalizedName)) - { - line.References.RemoveAll(reference => - reference.FileId == line.FileId - && reference.Line == line.LineNumber - && reference.Column == callIndex + 1 - && reference.ReferenceKind == "type_reference" - && string.Equals( - reference.SymbolName, - normalizedName, - StringComparison.Ordinal)); - AddReference( - line.References, - line.Seen, - line.FileId, - $"{containingType}.{normalizedName}", - callIndex, - "reference", - line.Context, - line.LineNumber, - callContainer, - line.Language, - sourceLength: name.Length); - } - - return false; - } - } - if (IsConstructorCallName( - line.Language, - line.PreparedLine, - callIndex)) - { - AddReference( - line.References, - line.Seen, - line.FileId, - normalizedName, - callIndex, - "instantiate", - line.Context, - line.LineNumber, - callContainer, - line.Language, - targetQualifier, - name.Length); - return true; - } - if (line.Language == "rust" - && RustReferenceExtractor.IsLikelyInstantiationCallName( + if (TryAddCoreInstantiationReference( + in call, name, normalizedName, - line.PreparedLine, - callIndex)) - { - AddReference( - line.References, - line.Seen, - line.FileId, - normalizedName, callIndex, - "instantiate", - line.Context, - line.LineNumber, callContainer, - line.Language, - sourceLength: name.Length); - return true; - } - if (line.Language == "python" - && TryGetKnownPythonTypeCall( - call, - normalizedName, - callIndex, - out var pythonTypeName)) + targetQualifier)) { - AddReference( - line.References, - line.Seen, - line.FileId, - pythonTypeName, - callIndex, - "instantiate", - line.Context, - line.LineNumber, - callContainer, - line.Language, - sourceLength: name.Length); return true; } + if (IsIgnoredCallName(line.Language, name) && !(line.Language == "scala" && string.Equals(name, "foreach", StringComparison.Ordinal))) @@ -223,113 +53,55 @@ private static bool TryAddCoreCallLikeReference( return false; } - var insideCSharpAttributeRange = call.CSharpAttributeRanges != null - && IsInsideCSharpAttributeRange( - call.CSharpAttributeRanges, - callIndex); - var metadataKind = TryClassifyMetadataReference( - line.Language, - line.PreparedLine, - callIndex, - insideCSharpAttributeRange); - if (metadataKind != null) - { - AddReference( - line.References, - line.Seen, - line.FileId, + if (TryAddCoreMetadataReference( + in call, + name, normalizedName, callIndex, - metadataKind, - line.Context, - line.LineNumber, - callContainer, - line.Language, - sourceLength: name.Length); - if (line.Language == "csharp" - && metadataKind == "attribute" - && CSharpReferenceExtractor.TryGetCallerInfoAttributeTypeName( - name, - line.PreparedLine, - callIndex) is { } callerInfoAttributeTypeName) - { - AddReference( - line.References, - line.Seen, - line.FileId, - callerInfoAttributeTypeName, - callIndex, - "type_reference", - line.Context, - line.LineNumber, - callContainer, - line.Language, - sourceLength: name.Length); - } - return true; - } - - if (line.Language == "kotlin" - && KotlinReferenceExtractor.IsConstructorCallName( - normalizedName, - call.KotlinConstructorTypeNames!)) + callContainer)) { - AddReference( - line.References, - line.Seen, - line.FileId, - normalizedName, - callIndex, - "instantiate", - line.Context, - line.LineNumber, - callContainer, - sourceLength: name.Length); return true; } - if (line.Language is "javascript" or "typescript" - && SymbolExtractor.IsJavaScriptTypeScriptReactHookName( - normalizedName)) - { - AddReference( - line.References, - line.Seen, - line.FileId, + if (TryAddSpecialCoreCallReference( + in call, + name, normalizedName, callIndex, - "consumes_hook", - line.Context, - line.LineNumber, - callContainer, - sourceLength: name.Length); + callContainer)) + { return true; } - AddReference( - line.References, - line.Seen, - line.FileId, + AddCoreClassifiedReference( + in call, normalizedName, callIndex, "call", - line.Context, - line.LineNumber, callContainer, - ScientificNativeReferenceExtractor.Supports(line.Language) - ? line.Language - : null, - targetQualifier: targetQualifier, - sourceLength: name.Length); + isScientificNativeLanguage ? line.Language : null, + targetQualifier, + name.Length); return true; } + private static string NormalizeCoreCallName( + string language, + string name) => + language == "fsharp" + && FSharpReferenceExtractor.IsOperatorCallName(name) + ? $"operator {name}" + : language == "rust" + ? RustReferenceExtractor.NormalizeIdentifier(name) + : NormalizeAtPrefixedIdentifier(name); + private static bool TryGetKnownPythonTypeCall( - CoreCallReferenceContext call, + in CoreCallReferenceContext call, string candidate, int callIndex, out string canonicalName) { + ref readonly var line = ref call.Line; canonicalName = candidate; var separator = candidate.LastIndexOf('.'); var leaf = separator >= 0 ? candidate[(separator + 1)..] : candidate; @@ -341,7 +113,7 @@ private static bool TryGetKnownPythonTypeCall( return PythonImportBindingResolver.TryResolveImportedTypeCall( candidate, - call.Line.PreparedLine, + line.PreparedLine, callIndex, call.Lookups.GetPythonImportedTypeCallLookup(), out canonicalName); diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallDispatch.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallDispatch.cs new file mode 100644 index 0000000000..28a20c2c79 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallDispatch.cs @@ -0,0 +1,148 @@ +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static void EmitCoreCallPrelude( + in CoreCallReferenceContext call) + { + ref readonly var line = ref call.Line; + if (line.Language is "javascript" or "typescript") + { + JavaScriptReferenceExtractor.EmitOptionalMemberChainReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + + JavaScriptReferenceExtractor.EmitDiscriminantStringGuardReferences( + call.ReferenceStructuralLine, + line.OriginalLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + + JavaScriptReferenceExtractor.EmitParenlessConstructorReferences( + line.PreparedLine, + line.PreparedLines, + line.LineIndex, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + } + + if (line.Language == "batch") + { + BatchReferenceExtractor.EmitJumpTargetReferences( + line.OriginalLine, + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + } + + if (line.Language == "assembly") + { + AssemblyReferenceExtractor.EmitInstructionTargetReferences( + line.OriginalLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + } + } + + private static void EmitCoreLanguageCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + switch (line.Language) + { + case "commonlisp": + case "racket": + EmitLispCallReferences(in call); + break; + case "powershell": + EmitPowerShellCallReferences(in call, ref state); + break; + case "shell": + EmitShellCallReferences(in call, ref state); + break; + case "assembly": + // Assembly references are operand-driven, not `name(...)` call syntax. + break; + default: + EmitGeneralCoreCallReferences(in call, ref state); + break; + } + } + + private static void EmitLispCallReferences( + in CoreCallReferenceContext call) + { + ref readonly var line = ref call.Line; + LispReferenceExtractor.EmitReferences( + line.Language, + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + line.DefinitionNames); + } + + private static void EmitPowerShellCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + PowerShellReferenceExtractor.EmitCallReferences( + line.PreparedLine, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + line.References); + if (ReferenceLimitReached(line.References)) + return; + + PowerShellReferenceExtractor.EmitSplatParameterReferences( + line.PreparedLine, + call.Lookups.GetPowerShellSplatAssignments, + line.LineNumber, + CreatePowerShellParameterReferenceEmitter(in line), + line.References); + } + + private static void EmitShellCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + ShellReferenceExtractor.EmitReferences( + line.PreparedLine, + line.OriginalLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + call.ShellCallableNames, + call.ShellGlobalAliasNames, + line.ResolveContainerForCall, + GetOrCreateCallLikeReferenceEmitter(in call, ref state)); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallKinds.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallKinds.cs new file mode 100644 index 0000000000..92bf5b09dc --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallKinds.cs @@ -0,0 +1,193 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static bool TryAddCoreInstantiationReference( + in CoreCallReferenceContext call, + string name, + string normalizedName, + int callIndex, + SymbolRecord? callContainer, + string? targetQualifier) + { + ref readonly var line = ref call.Line; + if (IsConstructorCallName( + line.Language, + line.PreparedLine, + callIndex)) + { + AddCoreClassifiedReference( + in call, + normalizedName, + callIndex, + "instantiate", + callContainer, + line.Language, + targetQualifier, + name.Length); + return true; + } + + if (line.Language == "rust" + && RustReferenceExtractor.IsLikelyInstantiationCallName( + name, + normalizedName, + line.PreparedLine, + callIndex)) + { + AddCoreClassifiedReference( + in call, + normalizedName, + callIndex, + "instantiate", + callContainer, + line.Language, + targetQualifier: null, + name.Length); + return true; + } + + if (line.Language == "python" + && TryGetKnownPythonTypeCall( + in call, + normalizedName, + callIndex, + out var pythonTypeName)) + { + AddCoreClassifiedReference( + in call, + pythonTypeName, + callIndex, + "instantiate", + callContainer, + line.Language, + targetQualifier: null, + name.Length); + return true; + } + + return false; + } + + private static bool TryAddCoreMetadataReference( + in CoreCallReferenceContext call, + string name, + string normalizedName, + int callIndex, + SymbolRecord? callContainer) + { + ref readonly var line = ref call.Line; + var insideCSharpAttributeRange = call.CSharpAttributeRanges != null + && IsInsideCSharpAttributeRange( + call.CSharpAttributeRanges, + callIndex); + var metadataKind = TryClassifyMetadataReference( + line.Language, + line.PreparedLine, + callIndex, + insideCSharpAttributeRange); + if (metadataKind == null) + return false; + + AddCoreClassifiedReference( + in call, + normalizedName, + callIndex, + metadataKind, + callContainer, + line.Language, + targetQualifier: null, + name.Length); + if (line.Language == "csharp" + && metadataKind == "attribute" + && CSharpReferenceExtractor.TryGetCallerInfoAttributeTypeName( + name, + line.PreparedLine, + callIndex) is { } callerInfoAttributeTypeName) + { + AddCoreClassifiedReference( + in call, + callerInfoAttributeTypeName, + callIndex, + "type_reference", + callContainer, + line.Language, + targetQualifier: null, + name.Length); + } + + return true; + } + + private static bool TryAddSpecialCoreCallReference( + in CoreCallReferenceContext call, + string name, + string normalizedName, + int callIndex, + SymbolRecord? callContainer) + { + ref readonly var line = ref call.Line; + if (line.Language == "kotlin" + && KotlinReferenceExtractor.IsConstructorCallName( + normalizedName, + call.KotlinConstructorTypeNames!)) + { + AddCoreClassifiedReference( + in call, + normalizedName, + callIndex, + "instantiate", + callContainer, + referenceLanguage: null, + targetQualifier: null, + name.Length); + return true; + } + + if (line.Language is "javascript" or "typescript" + && SymbolExtractor.IsJavaScriptTypeScriptReactHookName( + normalizedName)) + { + AddCoreClassifiedReference( + in call, + normalizedName, + callIndex, + "consumes_hook", + callContainer, + referenceLanguage: null, + targetQualifier: null, + name.Length); + return true; + } + + return false; + } + + private static void AddCoreClassifiedReference( + in CoreCallReferenceContext call, + string symbolName, + int callIndex, + string referenceKind, + SymbolRecord? callContainer, + string? referenceLanguage, + string? targetQualifier, + int sourceLength) + { + ref readonly var line = ref call.Line; + AddReference( + line.References, + line.Seen, + line.FileId, + symbolName, + callIndex, + referenceKind, + line.Context, + line.LineNumber, + callContainer, + referenceLanguage, + targetQualifier, + sourceLength); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallReferences.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallReferences.cs index 8d5b90f73a..1001018329 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallReferences.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallReferences.cs @@ -1,4 +1,3 @@ -using System.Text.RegularExpressions; using CodeIndex.Models; namespace CodeIndex.Indexer; @@ -6,7 +5,6 @@ namespace CodeIndex.Indexer; public static partial class ReferenceExtractor { private readonly record struct CoreCallReferenceContext( - CoreReferenceLineContext Line, CoreExtractionLookups Lookups, (SymbolRecord Synthetic, int NameIndex, int OpenBraceIndex, int CloseBraceIndex)? JavaSameLineCtor, List<(int start, int end)>? CSharpAttributeRanges, @@ -20,18 +18,68 @@ private readonly record struct CoreCallReferenceContext( Action? ReportDiagnostic, HashSet? SqlSuppressedCallIndices, HashSet<(int LineNumber, int ColumnIndex)>? SqlWindowFunctionCallSiteSuppressions, - CoreLineDefinitionState Definitions); + CoreLineDefinitionState Definitions) + { + public readonly CoreReferenceLineContext Line; + + public CoreCallReferenceContext( + in CoreReferenceLineContext line, + CoreExtractionLookups lookups, + (SymbolRecord Synthetic, int NameIndex, int OpenBraceIndex, int CloseBraceIndex)? javaSameLineCtor, + List<(int start, int end)>? cSharpAttributeRanges, + HashSet? kotlinConstructorTypeNames, + HashSet? kotlinInfixFunctionNames, + HashSet? shellCallableNames, + HashSet? shellGlobalAliasNames, + DynamicDeclarativeReferenceExtractor.ExtractionState? dynamicDeclarativeState, + string referenceStructuralLine, + int scientificNativeDependencyLimit, + Action? reportDiagnostic, + HashSet? sqlSuppressedCallIndices, + HashSet<(int LineNumber, int ColumnIndex)>? sqlWindowFunctionCallSiteSuppressions, + CoreLineDefinitionState definitions) + : this( + lookups, + javaSameLineCtor, + cSharpAttributeRanges, + kotlinConstructorTypeNames, + kotlinInfixFunctionNames, + shellCallableNames, + shellGlobalAliasNames, + dynamicDeclarativeState, + referenceStructuralLine, + scientificNativeDependencyLimit, + reportDiagnostic, + sqlSuppressedCallIndices, + sqlWindowFunctionCallSiteSuppressions, + definitions) + { + Line = line; + } + } + + private struct CoreCallEmissionState( + string callScanLine, + bool isScientificNativeLanguage) + { + public string CallScanLine { get; } = callScanLine; + public bool IsScientificNativeLanguage { get; } = isScientificNativeLanguage; + public Action? CallLikeReferenceEmitter { get; set; } + public HashSet? MatchedCallIndices { get; set; } + } private static Action CreateCallLikeReferenceEmitter( - CoreCallReferenceContext call) => + CoreCallReferenceContext call, + bool isScientificNativeLanguage) => (name, callIndex) => { - var line = call.Line; + ref readonly var line = ref call.Line; _ = TryAddCoreCallLikeReference( - call, + in call, name, callIndex, - ScientificNativeReferenceExtractor.Supports(line.Language) + isScientificNativeLanguage, + isScientificNativeLanguage ? ScientificNativeReferenceExtractor.GetParenthesizedCallTargetQualifier( line.Language, line.PreparedLine, @@ -39,471 +87,91 @@ private static Action CreateCallLikeReferenceEmitter( : null); }; + private static Action GetOrCreateCallLikeReferenceEmitter( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) => + state.CallLikeReferenceEmitter ??= + CreateCallLikeReferenceEmitter(call, state.IsScientificNativeLanguage); + + private static HashSet GetOrCreateMatchedCallIndices( + ref CoreCallEmissionState state) => + state.MatchedCallIndices ??= []; + private static Action CreatePowerShellParameterReferenceEmitter( - CoreReferenceLineContext line) => - (name, callIndex) => + in CoreReferenceLineContext line) + { + var references = line.References; + var seen = line.Seen; + var fileId = line.FileId; + var context = line.Context; + var lineNumber = line.LineNumber; + var language = line.Language; + var resolveContainerForCall = line.ResolveContainerForCall; + return (name, callIndex) => { - var callContainer = line.ResolveContainerForCall(callIndex); + var callContainer = resolveContainerForCall(callIndex); AddReference( - line.References, - line.Seen, - line.FileId, + references, + seen, + fileId, name, callIndex, "parameter", - line.Context, - line.LineNumber, + context, + lineNumber, callContainer, - line.Language, + language, sourceLength: name.Length); }; + } private static Action CreateGradleDslReferenceEmitter( - CoreReferenceLineContext line) => - (name, callIndex) => + in CoreReferenceLineContext line) + { + var references = line.References; + var seen = line.Seen; + var fileId = line.FileId; + var context = line.Context; + var lineNumber = line.LineNumber; + var language = line.Language; + var resolveContainerForCall = line.ResolveContainerForCall; + return (name, callIndex) => { var normalizedName = NormalizeAtPrefixedIdentifier(name); - var callContainer = line.ResolveContainerForCall(callIndex); + var callContainer = resolveContainerForCall(callIndex); AddReference( - line.References, - line.Seen, - line.FileId, + references, + seen, + fileId, normalizedName, callIndex, "call", - line.Context, - line.LineNumber, + context, + lineNumber, callContainer, - line.Language, + language, sourceLength: name.Length); }; + } - private static void EmitCoreCallReferences(CoreCallReferenceContext call) + private static void EmitCoreCallReferences( + in CoreCallReferenceContext call) { - var line = call.Line; - Action? addCallLikeReference = null; - if (line.Language is "javascript" or "typescript") - { - JavaScriptReferenceExtractor.EmitOptionalMemberChainReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - - JavaScriptReferenceExtractor.EmitDiscriminantStringGuardReferences( - call.ReferenceStructuralLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); + ref readonly var line = ref call.Line; + EmitCoreCallPrelude(in call); - JavaScriptReferenceExtractor.EmitParenlessConstructorReferences( - line.PreparedLine, - line.PreparedLines, - line.LineIndex, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - } - - if (line.Language is "batch") - BatchReferenceExtractor.EmitJumpTargetReferences( - line.OriginalLine, - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - - if (line.Language is "assembly") - AssemblyReferenceExtractor.EmitInstructionTargetReferences( - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - - HashSet? matchedCallIndices = null; var callScanLine = call.DynamicDeclarativeState?.GetCallScanLine( line.Language, line.LineNumber, line.PreparedLine) ?? line.PreparedLine; - - if (line.Language is "commonlisp" or "racket") - { - LispReferenceExtractor.EmitReferences( - line.Language, - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - line.DefinitionNames); - } - else if (line.Language is "powershell") - { - PowerShellReferenceExtractor.EmitCallReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.References); - if (!ReferenceLimitReached(line.References)) - { - PowerShellReferenceExtractor.EmitSplatParameterReferences( - line.PreparedLine, - call.Lookups.GetPowerShellSplatAssignments, - line.LineNumber, - CreatePowerShellParameterReferenceEmitter(line), - line.References); - } - } - else if (line.Language is "shell") - { - ShellReferenceExtractor.EmitReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - call.ShellCallableNames, - call.ShellGlobalAliasNames, - line.ResolveContainerForCall, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call)); - } - else if (line.Language is "assembly") - { - // Assembly line.References are operand-driven, not `name(...)` call syntax. - } - else - { - IReadOnlyList? - dTemplateArgumentCallSpans = null; - if (ScientificNativeReferenceExtractor.Supports(line.Language)) - { - dTemplateArgumentCallSpans = ScientificNativeReferenceExtractor.EmitReferences( - line.Language, - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - call.ScientificNativeDependencyLimit, - call.ReportDiagnostic); - } - - var dTemplateArgumentCallSpanIndex = 0; - if (line.Language is not ("tcl" or "prolog") - && callScanLine.IndexOf('(') >= 0) - { - foreach (Match match in EnumerateReferenceMatches(CallRegex, callScanLine, line.References)) - { - var name = match.Groups["name"].Value; - var callIndex = match.Groups["name"].Index; - if (line.Language == "rust" && RustReferenceExtractor.IsRawIdentifierPrefix(line.PreparedLine, callIndex)) - continue; - if (line.Language == "d" - && ScientificNativeReferenceExtractor.IsDTemplateArgumentCall( - dTemplateArgumentCallSpans, - ref dTemplateArgumentCallSpanIndex, - callIndex)) - { - continue; - } - if (line.Language == "ada" - && callIndex > 0 - && line.PreparedLine[callIndex - 1] == '\'') - { - continue; - } - if (line.Language == "objc" && IsObjCSelectorLiteralCall(line.PreparedLine, name, callIndex)) - continue; - if (call.SqlSuppressedCallIndices != null && call.SqlSuppressedCallIndices.Contains(callIndex)) - continue; - if (call.SqlWindowFunctionCallSiteSuppressions != null - && call.SqlWindowFunctionCallSiteSuppressions.Contains((line.LineNumber, callIndex))) - continue; - if (DynamicDeclarativeReferenceExtractor.ShouldSuppressGenericCall( - line.Language, - callScanLine, - name, - callIndex, - line.LineNumber, - call.DynamicDeclarativeState, - line.Language == "groovy" - ? line.ResolveContainerForCall(callIndex) - : null)) - { - continue; - } - (matchedCallIndices ??= []).Add(callIndex); - if (TryAddCoreCallLikeReference( - call, - name, - callIndex, - ScientificNativeReferenceExtractor.Supports(line.Language) - ? ScientificNativeReferenceExtractor.GetParenthesizedCallTargetQualifier( - line.Language, - line.PreparedLine, - callIndex) - : null)) - { - EmitGenericInvocationTypeArgumentReferences( - line.Language, - line.PreparedLine, - callIndex, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall(callIndex)); - } - if (line.Language == "ruby") - RubyReferenceExtractor.EmitCommandTargetReferences( - name, - callIndex, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - } - } - - if (line.Language == "ruby") - { - RubyReferenceExtractor.EmitAdditionalCallReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - matchedCallIndices ??= [], - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call)); - } - else if (line.Language is "perl" or "ambiguous_pl") - { - PerlReferenceExtractor.EmitAdditionalReferences( - line.Language == "ambiguous_pl" ? callScanLine : line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - emitArrowCallReferences: line.Language != "ambiguous_pl" - || call.DynamicDeclarativeState?.HasPrologContainer(line.LineNumber) != true); - } - - if (call.DynamicDeclarativeState != null) - { - DynamicDeclarativeReferenceExtractor.EmitAdditionalReferences( - line.Language, - callScanLine, - call.ReferenceStructuralLine, - call.DynamicDeclarativeState, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call)); - } - - if (line.Language == "go") - LanguageReferenceExtractionSupport.EmitGoBranchLabelReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.References); - - if (line.Language == "swift") - SwiftReferenceExtractor.EmitTrailingClosureReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.References); - else if (line.Language == "kotlin") - { - KotlinReferenceExtractor.EmitInfixCallReferences( - line.PreparedLine, - line.OriginalLine, - call.KotlinInfixFunctionNames!, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call)); - KotlinReferenceExtractor.EmitTrailingLambdaReferences( - line.PreparedLine, - addCallLikeReference, - line.References); - } - - if (line.Language == "fsharp") - { - FSharpReferenceExtractor.EmitAdditionalCallReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.References); - } - - if (line.Language == "scala") - { - ScalaReferenceExtractor.EmitTrailingBlockCallReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.References); - ScalaReferenceExtractor.EmitAdditionalReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - addCallLikeReference); - } - else if (line.Language == "gradle") - { - GradleReferenceExtractor.EmitDslCallReferences( - line.PreparedLine, - CreateGradleDslReferenceEmitter(line), - line.References); - } - - if (line.Language == "fortran") - FortranReferenceExtractor.EmitAdditionalCallReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call)); - else if (line.Language == "pascal") - PascalReferenceExtractor.EmitAdditionalCallReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.DefinitionNames); - else if (line.Language == "objc") - ObjectiveCReferenceExtractor.EmitAdditionalCallReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - else if (line.Language == "haskell") - HaskellReferenceExtractor.EmitAdditionalCallReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.DefinitionNames); - else if (line.Language == "elixir") - ElixirReferenceExtractor.EmitAdditionalCallReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.DefinitionNames); - else if (line.Language == "lua") - LuaReferenceExtractor.EmitAdditionalCallReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - line.DefinitionNames); - else if (line.Language == "smalltalk") - SmalltalkReferenceExtractor.EmitAdditionalCallReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.DefinitionNames); - else if (line.Language == "vb") - LanguageReferenceExtractionSupport.EmitAdditionalCallReferences( - "vb", - line.PreparedLine, - line.OriginalLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call), - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - line.DefinitionNames); - - // The flat CallRegex misses nested generic tails like `>>(` because `<[^>\n]+>` - // stops at the first `>`. Add a depth-aware fallback so `Foo>()` and - // `new Dict>()` still emit call/instantiate rows. See issue #263. - // 平坦な CallRegex は `<[^>\n]+>` が最初の `>` で止まるため `>>(` 形を取りこぼす。 - // depth-aware な fallback を足し、`Foo>()` や `new Dict>()` でも - // `call` / `instantiate` を発行する。issue #263 参照。 - if (line.Language is not ("tcl" or "prolog" or "ambiguous_pl") - && line.PreparedLine.IndexOf('(') >= 0 - && MayContainNestedGenericSyntax(line.PreparedLine)) - { - foreach (var candidate in EnumerateNestedGenericCallCandidates(line.PreparedLine, matchedCallIndices ?? EmptyMatchedIndices)) - { - if (ReferenceLimitReached(line.References)) - break; - - if (TryAddCoreCallLikeReference( - call, - candidate.Name, - candidate.NameIndex)) - { - EmitGenericInvocationTypeArgumentReferences( - line.Language, - line.PreparedLine, - candidate.NameIndex, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall(candidate.NameIndex)); - } - } - } - } + var state = new CoreCallEmissionState( + callScanLine, + ScientificNativeReferenceExtractor.Supports(line.Language)); + EmitCoreLanguageCallReferences(in call, ref state); if (ReferenceLimitReached(line.References)) return; - if (line.Language == "rust") - { - RustReferenceExtractor.EmitAdditionalCallReferences( - line.PreparedLine, - addCallLikeReference ??= CreateCallLikeReferenceEmitter(call)); - RustReferenceExtractor.EmitAttributeReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - } + EmitRustPostCallReferences(in call, ref state); } } diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallScanning.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallScanning.cs new file mode 100644 index 0000000000..06824d7368 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallScanning.cs @@ -0,0 +1,242 @@ +using System.Text.RegularExpressions; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static void EmitGeneralCoreCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + var dTemplateArgumentCallSpans = EmitScientificNativeCallReferences( + in call, + ref state); + EmitParenthesizedCoreCallReferences( + in call, + dTemplateArgumentCallSpans, + ref state); + EmitRubyOrPerlCallReferences(in call, ref state); + EmitDynamicDeclarativeCallReferences(in call, ref state); + EmitGoCallReferences(in call, ref state); + EmitSwiftOrKotlinCallReferences(in call, ref state); + EmitFSharpCallReferences(in call, ref state); + EmitScalaOrGradleCallReferences(in call, ref state); + EmitSecondaryLanguageCallReferences(in call, ref state); + EmitNestedGenericCallReferences(in call, ref state); + } + + private static IReadOnlyList? + EmitScientificNativeCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + if (!state.IsScientificNativeLanguage) + return null; + + ref readonly var line = ref call.Line; + return ScientificNativeReferenceExtractor.EmitReferences( + line.Language, + line.PreparedLine, + line.OriginalLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + GetOrCreateCallLikeReferenceEmitter(in call, ref state), + call.ScientificNativeDependencyLimit, + call.ReportDiagnostic); + } + + private static void EmitParenthesizedCoreCallReferences( + in CoreCallReferenceContext call, + IReadOnlyList? + dTemplateArgumentCallSpans, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + if (line.Language is "tcl" or "prolog" + || state.CallScanLine.IndexOf('(') < 0) + { + return; + } + + var dTemplateArgumentCallSpanIndex = 0; + foreach (Match match in EnumerateReferenceMatches( + CallRegex, + state.CallScanLine, + line.References)) + { + var name = match.Groups["name"].Value; + var callIndex = match.Groups["name"].Index; + if (ShouldSuppressParenthesizedCoreCall( + in call, + state.CallScanLine, + name, + callIndex, + dTemplateArgumentCallSpans, + ref dTemplateArgumentCallSpanIndex)) + { + continue; + } + + GetOrCreateMatchedCallIndices(ref state).Add(callIndex); + EmitParenthesizedCoreCallMatch( + in call, + name, + callIndex, + ref state); + } + } + + private static bool ShouldSuppressParenthesizedCoreCall( + in CoreCallReferenceContext call, + string callScanLine, + string name, + int callIndex, + IReadOnlyList? + dTemplateArgumentCallSpans, + ref int dTemplateArgumentCallSpanIndex) + { + ref readonly var line = ref call.Line; + if (line.Language == "rust" + && RustReferenceExtractor.IsRawIdentifierPrefix( + line.PreparedLine, + callIndex)) + { + return true; + } + + if (line.Language == "d" + && ScientificNativeReferenceExtractor.IsDTemplateArgumentCall( + dTemplateArgumentCallSpans, + ref dTemplateArgumentCallSpanIndex, + callIndex)) + { + return true; + } + + if (line.Language == "ada" + && callIndex > 0 + && line.PreparedLine[callIndex - 1] == '\'') + { + return true; + } + + if (line.Language == "objc" + && IsObjCSelectorLiteralCall(line.PreparedLine, name, callIndex)) + { + return true; + } + + if (call.SqlSuppressedCallIndices?.Contains(callIndex) == true) + return true; + + if (call.SqlWindowFunctionCallSiteSuppressions?.Contains( + (line.LineNumber, callIndex)) == true) + { + return true; + } + + return DynamicDeclarativeReferenceExtractor.ShouldSuppressGenericCall( + line.Language, + callScanLine, + name, + callIndex, + line.LineNumber, + call.DynamicDeclarativeState, + line.Language == "groovy" + ? line.ResolveContainerForCall(callIndex) + : null); + } + + private static void EmitParenthesizedCoreCallMatch( + in CoreCallReferenceContext call, + string name, + int callIndex, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + var targetQualifier = state.IsScientificNativeLanguage + ? ScientificNativeReferenceExtractor.GetParenthesizedCallTargetQualifier( + line.Language, + line.PreparedLine, + callIndex) + : null; + if (TryAddCoreCallLikeReference( + in call, + name, + callIndex, + state.IsScientificNativeLanguage, + targetQualifier)) + { + EmitGenericInvocationTypeArgumentReferences( + line.Language, + line.PreparedLine, + callIndex, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall(callIndex)); + } + + if (line.Language == "ruby") + { + RubyReferenceExtractor.EmitCommandTargetReferences( + name, + callIndex, + line.OriginalLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + } + } + + private static void EmitNestedGenericCallReferences( + in CoreCallReferenceContext call, + ref CoreCallEmissionState state) + { + ref readonly var line = ref call.Line; + // CallRegex stops at the first `>`. Keep the depth-aware fallback for + // nested tails such as `Foo>()` (issue #263). + if (line.Language is "tcl" or "prolog" or "ambiguous_pl" + || line.PreparedLine.IndexOf('(') < 0 + || !MayContainNestedGenericSyntax(line.PreparedLine)) + { + return; + } + + var matchedCallIndices = state.MatchedCallIndices ?? EmptyMatchedIndices; + foreach (var candidate in EnumerateNestedGenericCallCandidates( + line.PreparedLine, + matchedCallIndices)) + { + if (ReferenceLimitReached(line.References)) + break; + + if (TryAddCoreCallLikeReference( + in call, + candidate.Name, + candidate.NameIndex, + state.IsScientificNativeLanguage)) + { + EmitGenericInvocationTypeArgumentReferences( + line.Language, + line.PreparedLine, + candidate.NameIndex, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall(candidate.NameIndex)); + } + } + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallSuppressions.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallSuppressions.cs new file mode 100644 index 0000000000..e7e24efad7 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreCallSuppressions.cs @@ -0,0 +1,149 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static bool ShouldSuppressCoreCallCandidate( + in CoreCallReferenceContext call, + string name, + string normalizedName, + int callIndex) + { + ref readonly var line = ref call.Line; + + // In tuple-return declarations, CallRegex can see the modifier in + // `static (int Value, ...)` as a call. A verbatim `@static()` remains valid. + if (line.Language == "csharp" && name == "static") + return true; + + if (line.Language == "rust" + && RustReferenceExtractor.IsFunctionDeclarationCallSite( + line.PreparedLine, + callIndex)) + { + return true; + } + + if (line.Language == "rust" + && RustReferenceExtractor.IsDeriveAttributeCallSite( + line.PreparedLine, + normalizedName, + callIndex)) + { + return true; + } + + if (line.Language == "wgsl" && name.StartsWith('@')) + return true; + + if (line.Language == "kotlin" + && KotlinReferenceExtractor.IsInfixFunctionDeclarationSite( + line.PreparedLine, + callIndex)) + { + return true; + } + + // A same-line Java constructor declarator is a declaration, not a call. + if (call.JavaSameLineCtor is { } javaSameLineCtor + && callIndex == javaSameLineCtor.NameIndex + && string.Equals( + normalizedName, + javaSameLineCtor.Synthetic.Name, + StringComparison.Ordinal)) + { + return true; + } + + if (line.Language == "csharp" + && CSharpReferenceExtractor.IsPatternHeadCallSite( + line.PreparedLines, + line.LineIndex, + line.PreparedLine, + callIndex)) + { + return true; + } + + if (line.Language == "typescript" + && TypeScriptReferenceExtractor.IsSatisfiesTypeOperand( + line.PreparedLine, + callIndex)) + { + return true; + } + + return call.Definitions.ShouldSuppressDefinitionCall( + normalizedName, + name, + callIndex); + } + + private static bool TryHandleCSharpValueReceiverCall( + in CoreCallReferenceContext call, + string name, + string normalizedName, + int callIndex, + SymbolRecord? callContainer) + { + ref readonly var line = ref call.Line; + var afterNameIndex = callIndex + name.Length; + if (line.Language != "csharp" + || afterNameIndex >= line.PreparedLine.Length + || !line.PreparedLine.AsSpan(afterNameIndex) + .TrimStart() + .StartsWith(".", StringComparison.Ordinal)) + { + return false; + } + + var receiverLookups = call.Lookups.GetCSharpValueReceiverLookups(); + if (!HasCSharpValueReceiverConflict( + normalizedName, + normalizedName, + line.LineNumber, + callIndex, + callContainer, + receiverLookups.ByContainingType, + receiverLookups.ByFunctionStartLine)) + { + return false; + } + + var containingType = GetContainingTypeQualifiedName(callContainer); + if (containingType != null + && receiverLookups.ByContainingType.TryGetValue( + containingType, + out var receiverNames) + && (receiverNames.InstanceNames.Contains(normalizedName) + || receiverNames.StaticNames.Contains(normalizedName)) + && call.Lookups.HasCSharpFieldOrPropertyMember( + containingType, + normalizedName)) + { + var fileId = line.FileId; + var lineNumber = line.LineNumber; + line.References.RemoveAll(reference => + reference.FileId == fileId + && reference.Line == lineNumber + && reference.Column == callIndex + 1 + && reference.ReferenceKind == "type_reference" + && string.Equals( + reference.SymbolName, + normalizedName, + StringComparison.Ordinal)); + AddCoreClassifiedReference( + in call, + $"{containingType}.{normalizedName}", + callIndex, + "reference", + callContainer, + line.Language, + targetQualifier: null, + name.Length); + } + + return true; + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreContainerResolution.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreContainerResolution.cs new file mode 100644 index 0000000000..7cf40728c7 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreContainerResolution.cs @@ -0,0 +1,170 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private sealed class CoreReferenceLineContainerResolver + { + private readonly CoreReferenceLoopContext _loop; + private readonly int _lineIndex; + private readonly int _lineNumber; + private readonly SymbolRecord? _container; + private readonly bool _csharpLineHasWhereClause; + private readonly ( + SymbolRecord Synthetic, + int NameIndex, + int OpenBraceIndex, + int CloseBraceIndex)? _javaSameLineCtor; + + internal CoreReferenceLineContainerResolver( + CoreReferenceLoopContext loop, + int lineIndex, + int lineNumber, + SymbolRecord? container, + bool csharpLineHasWhereClause, + (SymbolRecord Synthetic, int NameIndex, int OpenBraceIndex, + int CloseBraceIndex)? javaSameLineCtor) + { + _loop = loop; + _lineIndex = lineIndex; + _lineNumber = lineNumber; + _container = container; + _csharpLineHasWhereClause = csharpLineHasWhereClause; + _javaSameLineCtor = javaSameLineCtor; + ResolveContainerForCall = ResolveContainer; + ResolveSwiftPropertyContainerForCall = + ResolveSwiftPropertyContainer; + } + + internal Func ResolveContainerForCall { get; } + + internal Func ResolveSwiftPropertyContainerForCall + { + get; + } + + private SymbolRecord? ResolveContainer(int column) + { + var language = _loop.Request.Language; + if (language == "csharp") + { + var recordContainer = ResolveCSharpRecordContainer(column); + if (recordContainer.IsResolved) + return recordContainer.Container; + } + + if (_javaSameLineCtor != null) + { + var info = _javaSameLineCtor.Value; + if (info.CloseBraceIndex >= 0 + && column > info.OpenBraceIndex + && column < info.CloseBraceIndex) + { + return info.Synthetic; + } + } + + if (language == "csharp") + { + if (_csharpLineHasWhereClause) + { + var declarationRangeContainer = + FindInnermostCSharpDeclarationRangeContainer( + _loop.ContainerCandidates, + _loop.Preparation.StructuralLines[_lineIndex], + _lineNumber, + column); + if (declarationRangeContainer != null) + return declarationRangeContainer; + } + + var sameLineContainer = + FindInnermostSameLineCSharpContainer( + _loop.Lookups + .GetCSharpSameLineContainerCandidatesByLine(), + _loop.Preparation.StructuralLines[_lineIndex], + _lineNumber, + column); + if (sameLineContainer != null) + return sameLineContainer; + + if (_csharpLineHasWhereClause + && _container?.Kind == "function" + && _container.StartLine == _lineNumber + && (!TryFindCSharpFunctionNameColumn( + _loop.Preparation.StructuralLines[_lineIndex], + _container.Name, + out var containerNameColumn) + || column < containerNameColumn)) + { + return null; + } + } + + return _loop.DynamicDeclarativeState?.ResolveContainer( + _lineNumber, + column, + _container) ?? _container; + } + + private (bool IsResolved, SymbolRecord? Container) + ResolveCSharpRecordContainer(int column) + { + SymbolRecord? primaryCtorOwner = null; + foreach (var ( + rangeStart, + rangeStartColumn, + rangeEnd, + rangeEndColumn, + syntheticRecordCtor, + owner) in + _loop.Lookups.GetRecordPrimaryCtorRanges()) + { + if (ReferenceEquals(_container, syntheticRecordCtor) + || (_container?.Kind == "function" + && _container.FileId == syntheticRecordCtor.FileId + && _container.StartLine == syntheticRecordCtor.StartLine + && (_container.StartLine < rangeEnd + || (_container.StartColumn + is int containerStartColumn + && containerStartColumn < rangeEndColumn)) + && string.Equals( + _container.Name, + syntheticRecordCtor.Name, + StringComparison.Ordinal))) + { + primaryCtorOwner ??= owner; + } + if (_lineNumber < rangeStart || _lineNumber > rangeEnd) + continue; + if (_lineNumber == rangeStart && column < rangeStartColumn) + return (true, owner); + if (_lineNumber == rangeEnd && column >= rangeEndColumn) + continue; + return (true, syntheticRecordCtor); + } + + return primaryCtorOwner != null + ? (true, primaryCtorOwner) + : (false, null); + } + + private SymbolRecord? ResolveSwiftPropertyContainer(int column) + { + if (_loop.SwiftPropertyDefinitionsByLine != null + && _loop.SwiftPropertyDefinitionsByLine.TryGetValue( + _lineNumber, + out var sameLineProperties)) + { + foreach (var property in sameLineProperties) + { + if ((property.StartColumn ?? 0) <= column) + return property; + } + } + + return ResolveContainer(column); + } + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreDocumentationLines.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreDocumentationLines.cs index 8df7c39ba4..d3c642254a 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreDocumentationLines.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreDocumentationLines.cs @@ -22,146 +22,77 @@ private readonly record struct CoreDocumentationLineContext( bool[]? CSharpLinesInsideMultilineStringContent, bool[]? CSharpLinesInsideBlockComment, List<(int start, int end)>? CSharpAttributeRangesOnLine, - List<(int start, int end)>?[]? CSharpAttributeRanges, - Func? GetPhpLineContainer); + List<(int start, int end)>?[]? CSharpAttributeRanges); - private static void EmitCoreDocumentationReferences( - CoreDocumentationLineContext line, - ref bool csharpInDelimitedDocComment, - ref bool jvmInDelimitedDocComment, - ref bool phpInDocblock, - ref SymbolRecord? phpDocblockContainer, - ref HashSet? phpDocblockPropertyNames) + private struct CoreDocumentationState + { + internal bool CSharpInDelimitedDocComment; + internal bool JvmInDelimitedDocComment; + internal PhpDocumentationState Php; + } + + private static void EmitCoreLanguageDocumentationReferences( + CoreReferenceLoopContext loop, + CoreReferenceLoopState state, + int lineIndex, + int lineNumber, + string originalLine, + string preparedLine, + List<(int start, int end)>? csharpAttributeRangesOnLine) { - if (line.Language == "csharp" - && line.CSharpLinesInsideMultilineStringContent != null - && !(line.CSharpLinesInsideMultilineStringContent?[line.LineIndex] ?? false) - && TryGetCSharpXmlDocCommentSpan( - line.OriginalLine, - csharpInDelimitedDocComment, - line.CSharpLinesInsideBlockComment?[line.LineIndex] ?? false, - out var csharpDocCommentStartIndex, - out var csharpDocCommentEndExclusive, - out var nextCsharpDelimitedDocComment)) + var request = loop.Request; + if (request.Language == "php") { - var csharpDocCommentText = line.OriginalLine[csharpDocCommentStartIndex..csharpDocCommentEndExclusive]; - if (csharpDocCommentText.IndexOf("cref=\"", StringComparison.OrdinalIgnoreCase) >= 0) - { - var innermostContainer = line.ContainerResolver.Find(line.LineNumber); - var sameLineDeclarationStartColumn = GetCSharpSameLineDocumentedDeclarationStartColumn( - line.OriginalLine, - csharpDocCommentEndExclusive, - nextCsharpDelimitedDocComment); - var docContainer = FindDocumentedContainer( - line.ContainerCandidates, - line.StructuralLines[line.LineIndex], - line.PreparedLine, - line.CSharpAttributeRangesOnLine, - line.LineNumber, - sameLineDeclarationStartColumn); - if (docContainer != null - && (docContainer.StartLine == line.LineNumber - || CanAttachCSharpXmlDocCommentToNextDeclaration( - innermostContainer, - line.Lookups.GetCSharpXmlDocAttachmentScopeCandidates(), - line.CSharpAttributeRanges, - line.PreparedLines, - line.LineNumber, - docContainer))) - { - CSharpReferenceExtractor.EmitDocCrefReferences( - csharpDocCommentText, - line.References, - line.Seen, - line.FileId, - csharpDocCommentStartIndex, - csharpDocCommentText.Trim(), - line.LineNumber, - docContainer); - } - } - csharpInDelimitedDocComment = nextCsharpDelimitedDocComment; + var phpLine = new CorePhpDocumentationLineContext( + request.FileId, + originalLine, + lineNumber, + loop.References, + loop.Seen, + loop.ContainerResolver); + EmitPhpDocumentationReferences( + in phpLine, + ref state.Documentation.Php); + return; } - else if (line.Language is "java" or "kotlin" - && TryGetJvmDocCommentSpan( - line.OriginalLine, - jvmInDelimitedDocComment, - out var jvmDocCommentStartIndex, - out var jvmDocCommentEndExclusive, - out var jvmSameLineDeclarationStartColumn, - out var nextJvmDelimitedDocComment)) - { - if (jvmDocCommentEndExclusive > jvmDocCommentStartIndex) - { - var docContainer = FindJvmDocumentedContainer( - line.ContainerCandidates, - line.Lines, - line.StructuralLines[line.LineIndex], - line.LineNumber, - jvmSameLineDeclarationStartColumn); - if (docContainer != null) - { - var docText = line.OriginalLine[jvmDocCommentStartIndex..jvmDocCommentEndExclusive]; - EmitJvmDocLinkReferences( - line.Language, - docText, - line.References, - line.Seen, - line.FileId, - jvmDocCommentStartIndex, - docText.Trim(), - line.LineNumber, - docContainer); - } - } - jvmInDelimitedDocComment = nextJvmDelimitedDocComment; - } + if (request.Language is not ("csharp" or "java" or "kotlin" or "r")) + return; - if (line.Language == "r") - { - var roxygenContext = line.OriginalLine.Trim(); - if (roxygenContext.Length > 0) - { - RReferenceExtractor.EmitRoxygenImportFromReferences( - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - roxygenContext, - line.LineNumber, - container: null); - RReferenceExtractor.EmitRoxygenImportReferences( - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - roxygenContext, - line.LineNumber, - container: null); - RReferenceExtractor.EmitRoxygenMethodReferences( - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - roxygenContext, - line.LineNumber, - container: null); - } - } + var input = loop.Preparation; + var documentationLine = new CoreDocumentationLineContext( + request.FileId, + request.Language, + input.Lines, + input.PreparedLines, + input.StructuralLines, + lineIndex, + lineNumber, + originalLine, + preparedLine, + loop.References, + loop.Seen, + loop.ContainerCandidates, + loop.ContainerResolver, + loop.Lookups, + input.CSharpLinesInsideMultilineStringContent, + input.CSharpLinesInsideBlockComment, + csharpAttributeRangesOnLine, + loop.CSharpAttributeRanges); + EmitCoreDocumentationReferences( + in documentationLine, + ref state.Documentation); + } - if (line.Language == "php") - { - EmitPhpLinePreambleReferences( - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.LineNumber, - line.GetPhpLineContainer!, - ref phpInDocblock, - ref phpDocblockContainer, - ref phpDocblockPropertyNames); - } + private static void EmitCoreDocumentationReferences( + in CoreDocumentationLineContext line, + ref CoreDocumentationState state) + { + if (line.Language == "csharp") + EmitCoreCSharpDocumentationReferences(in line, ref state.CSharpInDelimitedDocComment); + else if (line.Language is "java" or "kotlin") + EmitCoreJvmDocumentationReferences(in line, ref state.JvmInDelimitedDocComment); + else if (line.Language == "r") + EmitRDocumentationReferences(in line); } } diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Context.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Context.cs new file mode 100644 index 0000000000..268da7abd5 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Context.cs @@ -0,0 +1,87 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static CoreReferenceLoopContext CreateCoreReferenceLoopContext( + CoreExtractionPreparation preparation, + CorePreSolidityLookups preSolidityLookups) + { + var request = preparation.Request; + var lines = preparation.Lines.Lines; + var postSolidityLookups = BuildCorePostSolidityLookups( + preparation, + preSolidityLookups); + var references = CreateReferenceList( + request.MaxReferenceCount, + EstimateReferenceListInitialCapacity(lines.Length)); + var seen = CreateReferenceSeenSet(lines.Length); + + return new CoreReferenceLoopContext + { + Request = request, + Preparation = preparation.Lines, + IsJsxFile = preparation.IsJsxFile, + IsRazorFile = preparation.IsRazorFile, + XamlReferenceEnabled = preparation.XamlReferenceEnabled, + ScientificNativeDependencyLimit = + preSolidityLookups.ScientificNativeDependencyLimit, + CSharpAttributeRanges = + preSolidityLookups.CSharpAttributes.Ranges, + CSharpAttributeTopLevelRanges = + preSolidityLookups.CSharpAttributes.TopLevelRanges, + DefinitionNamesComparer = + preSolidityLookups.Definitions.Comparer, + DefinitionNamesByLine = + preSolidityLookups.Definitions.NamesByLine, + ScientificDefinitionNameIndicesByLine = + preSolidityLookups.Definitions.ScientificNameIndicesByLine, + AllDefinitionNames = + preSolidityLookups.Definitions.AllNames, + FileDefinitionNames = + preSolidityLookups.Definitions.FileNames, + SqlDefinitionLeafSpansByLine = + preSolidityLookups.Sql.DefinitionLeafSpansByLine, + SqlWindowFunctionCallSiteSuppressions = + preSolidityLookups.Sql.WindowFunctionCallSiteSuppressions, + CobolCallableSymbols = preSolidityLookups.CobolCallableSymbols, + ContainerCandidates = preSolidityLookups.Containers.Candidates, + ContainerResolver = preSolidityLookups.Containers.Resolver, + SwiftPropertyDefinitionsByLine = + postSolidityLookups.Language.SwiftPropertyDefinitionsByLine, + CSharpQualifiedEnumMemberLookup = + postSolidityLookups.CSharp.QualifiedEnumMembers, + CSharpQualifiedConstantPatternMemberLookup = + postSolidityLookups.CSharp.QualifiedConstantPatternMembers, + CSharpQualifiedTypePatternLookup = + postSolidityLookups.CSharp.QualifiedTypePatterns, + KotlinConstructorTypeNames = + postSolidityLookups.Language.KotlinConstructorTypeNames, + KotlinInfixFunctionNames = + postSolidityLookups.Language.KotlinInfixFunctionNames, + CallableDefinitionNames = + postSolidityLookups.CSharp.CallableDefinitionNames, + StylusVariableDefinitionNames = + postSolidityLookups.Language.StylusVariableDefinitionNames, + DockerfileStageNames = + postSolidityLookups.Language.DockerfileStageNames, + DockerfileVariableNames = + postSolidityLookups.Language.DockerfileVariableNames, + ShellCallableNames = + postSolidityLookups.Language.ShellCallableNames, + ShellGlobalAliasNames = + postSolidityLookups.Language.ShellGlobalAliasNames, + DynamicDeclarativeState = + postSolidityLookups.DynamicDeclarativeState, + CSharpUsingAliases = postSolidityLookups.CSharp.UsingAliases, + CSharpUsingStatics = postSolidityLookups.CSharp.UsingStatics, + Lookups = postSolidityLookups.Extraction, + TypeScriptTypeAliases = + preSolidityLookups.TypeScriptTypeAliases, + SwiftTypeAliases = preSolidityLookups.SwiftTypeAliases, + References = references, + Seen = seen, + }; + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Execution.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Execution.cs new file mode 100644 index 0000000000..7a303ba54f --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Execution.cs @@ -0,0 +1,51 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static List ExtractCoreSolidityReferences( + CoreExtractionPreparation preparation, + CorePreSolidityLookups preSolidityLookups) + => ExtractSolidityReferences( + preparation.Request.FileId, + preparation.Lines.Lines, + preparation.Lines.PreparedLines, + preSolidityLookups.Containers.Resolver); + + private static void EmitCoreExtractionPrelude( + CoreReferenceLoopContext loop) + { + var request = loop.Request; + if (request.Language == "csharp") + { + EmitCSharpAsyncIteratorReferences( + request.FileId, + loop.Preparation.Lines, + loop.Preparation.StructuralLines, + request.Symbols, + loop.References, + loop.Seen); + EmitCSharpStaticInterfaceMemberImplementationReferences( + request.FileId, + loop.Preparation.Lines, + loop.Preparation.StructuralLines, + request.Symbols, + request.WorkspaceSymbols ?? request.Symbols, + request.CSharpStaticInterfaceMemberLookups, + loop.References, + loop.Seen); + } + else if (request.Language == "rust") + { + RustReferenceExtractor.EmitMultilineAttributeReferences( + loop.Preparation.PreparedLines, + loop.References, + loop.Seen, + request.FileId, + (lineNumber, _) => FindInnermostContainer( + loop.ContainerCandidates, + lineNumber)); + } + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Finalization.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Finalization.cs new file mode 100644 index 0000000000..aeb3fce56c --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Finalization.cs @@ -0,0 +1,131 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static List FinalizeCoreExtraction( + CoreReferenceLoopContext loop, + CSharpMultiLineTypePatternState pendingCSharpMultiLineTypePattern) + { + var request = loop.Request; + var references = loop.References; + if (!ReferenceLimitReached(references) + && request.Language == "csharp") + { + CSharpReferenceExtractor.EmitSwitchExpressionTypePatternReferences( + loop.Preparation.Lines, + loop.Preparation.PreparedLines, + loop.ContainerCandidates, + loop.CSharpQualifiedConstantPatternMemberLookup, + loop.CSharpQualifiedTypePatternLookup, + loop.CSharpUsingAliases, + loop.CSharpUsingStatics, + loop.Lookups.HasActiveSameFileCSharpTypeCandidate, + references, + loop.Seen, + request.FileId); + + CSharpReferenceExtractor + .FlushPendingMultiLineTypePatternReference( + ref pendingCSharpMultiLineTypePattern, + loop.CSharpQualifiedConstantPatternMemberLookup, + loop.CSharpUsingAliases, + loop.CSharpUsingStatics, + loop.Lookups.HasActiveSameFileCSharpTypeCandidate, + references, + loop.Seen, + request.FileId); + } + + if (request.Language == "csharp") + { + RewriteCSharpPropertyReceiverReferences( + loop.Preparation.PreparedLines, + references, + loop.Lookups); + RemoveCSharpCallsDuplicatedByMemberReads(references); + } + + loop.Lookups.ApplyCSharpUsingAliasReferenceNames(references); + if (!ReferenceLimitReached(references)) + { + loop.Lookups.EmitCSharpBclRegexWithoutTimeoutReferences( + references, + loop.Seen); + } + MarkMutualRecursionReferences(references); + return references; + } + + private static void RemoveCSharpCallsDuplicatedByMemberReads( + List references) + { + var memberReadSites = references + .Where(reference => reference.ReferenceKind == "member_read") + .Select(reference => ( + reference.FileId, + reference.Line, + reference.Column, + reference.SymbolName, + reference.ContainerKind, + reference.ContainerName)) + .ToHashSet(); + if (memberReadSites.Count == 0) + return; + + references.RemoveAll(reference => + reference.ReferenceKind == "call" + && memberReadSites.Contains(( + reference.FileId, + reference.Line, + reference.Column, + reference.SymbolName, + reference.ContainerKind, + reference.ContainerName))); + } + + private static void RewriteCSharpPropertyReceiverReferences( + IReadOnlyList preparedLines, + List references, + CoreExtractionLookups lookups) + { + foreach (var reference in references) + { + if (reference.ReferenceKind != "type_reference" + || reference.Line <= 0 + || reference.Line > preparedLines.Count + || reference.Column <= 0) + { + continue; + } + + var line = preparedLines[reference.Line - 1]; + var tokenEnd = + reference.Column - 1 + reference.SymbolName.Length; + if (tokenEnd >= line.Length + || !line.AsSpan(tokenEnd) + .TrimStart() + .StartsWith(".", StringComparison.Ordinal)) + { + continue; + } + + var owner = lookups.FindCSharpContainerCandidate( + reference.ContainerName, + reference.Line); + var containingType = GetContainingTypeQualifiedName(owner); + if (containingType == null + || !lookups.HasCSharpFieldOrPropertyMember( + containingType, + reference.SymbolName)) + { + continue; + } + + reference.SymbolName = + $"{containingType}.{reference.SymbolName}"; + reference.ReferenceKind = "reference"; + } + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Lookups.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Lookups.cs new file mode 100644 index 0000000000..cb46af6815 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Lookups.cs @@ -0,0 +1,298 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private readonly record struct CoreCSharpAttributeLookups( + List<(int start, int end)>?[]? Ranges, + List<(int start, int end)>?[]? TopLevelRanges); + + private readonly record struct CoreDefinitionLookups( + StringComparer Comparer, + IReadOnlyDictionary> NamesByLine, + IReadOnlyDictionary>>? ScientificNameIndicesByLine, + IReadOnlySet? AllNames, + IReadOnlySet? FileNames); + + private readonly record struct CoreSqlLookups( + Dictionary>? DefinitionLeafSpansByLine, + HashSet<(int LineNumber, int ColumnIndex)>? WindowFunctionCallSiteSuppressions); + + private readonly record struct CoreContainerLookups( + IReadOnlyList Candidates, + InnermostContainerResolver Resolver); + + private readonly record struct CorePreSolidityLookups( + int ScientificNativeDependencyLimit, + IReadOnlyList? TypeScriptTypeAliases, + IReadOnlyList? SwiftTypeAliases, + CoreCSharpAttributeLookups CSharpAttributes, + CoreDefinitionLookups Definitions, + CoreSqlLookups Sql, + IReadOnlyList? CobolCallableSymbols, + CoreContainerLookups Containers); + + private readonly record struct CoreCSharpPatternLookups( + IReadOnlySet KnownTypeNames, + IReadOnlyDictionary> QualifiedEnumMembers, + IReadOnlyDictionary> QualifiedConstantPatternMembers, + IReadOnlyDictionary> QualifiedTypePatterns); + + private readonly record struct CoreCSharpLoopLookups( + IReadOnlySet KnownTypeNames, + IReadOnlyDictionary> QualifiedEnumMembers, + IReadOnlyDictionary> QualifiedConstantPatternMembers, + IReadOnlyDictionary> QualifiedTypePatterns, + IReadOnlyList UsingAliases, + IReadOnlyList UsingNamespaces, + IReadOnlyList UsingStatics, + HashSet? CallableDefinitionNames); + + private readonly record struct CoreLanguageLoopLookups( + IReadOnlyDictionary? SwiftPropertyDefinitionsByLine, + HashSet? KotlinConstructorTypeNames, + HashSet? KotlinInfixFunctionNames, + HashSet? StylusVariableDefinitionNames, + HashSet? DockerfileStageNames, + HashSet? DockerfileVariableNames, + HashSet? ShellCallableNames, + HashSet? ShellGlobalAliasNames); + + private readonly record struct CorePostSolidityLookups( + CoreCSharpLoopLookups CSharp, + CoreLanguageLoopLookups Language, + DynamicDeclarativeReferenceExtractor.ExtractionState? DynamicDeclarativeState, + CoreExtractionLookups Extraction); + + private static CorePreSolidityLookups BuildCorePreSolidityLookups( + CoreExtractionPreparation preparation) + { + var request = preparation.Request; + var language = request.Language; + var lines = preparation.Lines.Lines; + var structuralLines = preparation.Lines.StructuralLines; + var preparedLines = preparation.Lines.PreparedLines; + var scientificNativeDependencyLimit = + ScientificNativeReferenceExtractor.Supports(language) + ? GetSafetyLimits().MaxNamesPerLine + : 0; + var typeScriptTypeAliases = language == "typescript" + ? TypeScriptReferenceExtractor.BuildTypeAliasTargets(preparedLines) + : null; + var swiftTypeAliases = language == "swift" + ? SwiftReferenceExtractor.BuildTypeAliasTargets(preparedLines) + : null; + + var csharpAttributeTables = language == "csharp" + && preparation.Lines.Content.Contains('[', StringComparison.Ordinal) + ? BuildCSharpAttributeRanges(preparedLines) + : (null, null); + var definitionNamesComparer = GetDefinitionNamesComparer(language); + var definitionNamesByLine = BuildDefinitionNamesByLine( + language, + request.Symbols, + request.ReportDiagnostic); + var scientificDefinitionNameIndicesByLine = + BuildScientificDefinitionNameIndicesByLine( + language, + lines, + request.Symbols, + definitionNamesByLine); + var allDefinitionNames = language == "stylus" + ? BuildAllDefinitionNames( + language, + request.Symbols, + request.ReportDiagnostic) + : null; + var fileDefinitionNames = preparation.IsRazorFile + ? BuildFileDefinitionNames(request.Symbols) + : null; + var sqlDefinitionLeafSpansByLine = language == "sql" + ? SqlReferenceExtractor.BuildDefinitionLeafSpansByLine( + lines, + request.Symbols) + : null; + var sqlWindowFunctionCallSiteSuppressions = language == "sql" + ? SqlReferenceExtractor + .BuildWindowFunctionCallSiteSuppressions(structuralLines) + : null; + var cobolCallableSymbols = language == "cobol" + ? BuildCobolCallableSymbols(request.Symbols) + : null; + var containerCandidates = BuildReferenceContainerCandidates( + language, + request.Symbols, + request.ReportDiagnostic); + var containerResolver = new InnermostContainerResolver( + containerCandidates, + preferCallable: language == "csharp"); + + return new CorePreSolidityLookups( + scientificNativeDependencyLimit, + typeScriptTypeAliases, + swiftTypeAliases, + new CoreCSharpAttributeLookups( + csharpAttributeTables.Item1, + csharpAttributeTables.Item2), + new CoreDefinitionLookups( + definitionNamesComparer, + definitionNamesByLine, + scientificDefinitionNameIndicesByLine, + allDefinitionNames, + fileDefinitionNames), + new CoreSqlLookups( + sqlDefinitionLeafSpansByLine, + sqlWindowFunctionCallSiteSuppressions), + cobolCallableSymbols, + new CoreContainerLookups( + containerCandidates, + containerResolver)); + } + + private static CoreCSharpPatternLookups BuildCoreCSharpPatternLookups( + CoreExtractionPreparation preparation) + { + var request = preparation.Request; + if (request.Language != "csharp") + { + return new CoreCSharpPatternLookups( + EmptyCSharpStringSet, + EmptyCSharpQualifiedEnumMemberLookup, + EmptyCSharpQualifiedPatternLookup, + EmptyCSharpQualifiedPatternLookup); + } + + var typeNameSets = BuildCSharpTypeNameSets( + request.Language, + request.Symbols); + var localQualifiedPatternLookups = + BuildCSharpQualifiedPatternLookups( + request.Symbols, + typeNameSets.NonEnumTypeNames); + var workspaceQualifiedPatternLookups = + request.CSharpQualifiedPatternLookups + ?? (request.WorkspaceSymbols is { Count: > 0 } + ? BuildCSharpQualifiedPatternLookups(request.WorkspaceSymbols) + : localQualifiedPatternLookups); + var qualifiedPatternLookups = new CSharpQualifiedPatternLookups( + workspaceQualifiedPatternLookups.EnumMemberLookup, + workspaceQualifiedPatternLookups.ConstantPatternMemberLookup, + localQualifiedPatternLookups.TypePatternLookup); + return new CoreCSharpPatternLookups( + typeNameSets.KnownTypeNames, + qualifiedPatternLookups.EnumMemberLookup, + qualifiedPatternLookups.ConstantPatternMemberLookup, + qualifiedPatternLookups.TypePatternLookup); + } + + private static CorePostSolidityLookups BuildCorePostSolidityLookups( + CoreExtractionPreparation preparation, + CorePreSolidityLookups preSolidityLookups) + { + var request = preparation.Request; + var language = request.Language; + var lines = preparation.Lines.Lines; + var structuralLines = preparation.Lines.StructuralLines; + var referenceStructuralLines = + preparation.Lines.ReferenceStructuralLines; + var preparedLines = preparation.Lines.PreparedLines; + + var swiftPropertyDefinitionsByLine = language == "swift" + ? BuildSwiftPropertyDefinitionsByLine( + language, + request.Symbols, + request.ReportDiagnostic) + : null; + var csharpPatterns = BuildCoreCSharpPatternLookups(preparation); + + HashSet? kotlinConstructorTypeNames = null; + HashSet? kotlinInfixFunctionNames = null; + if (language == "kotlin") + { + var kotlinNameSets = KotlinReferenceExtractor.BuildNameSets( + language, + request.Symbols); + kotlinConstructorTypeNames = kotlinNameSets.ConstructorTypeNames; + kotlinInfixFunctionNames = kotlinNameSets.InfixFunctionNames; + KotlinReferenceExtractor.AddDeclaredInfixFunctionNames( + lines, + kotlinInfixFunctionNames); + } + + var callableDefinitionNames = language == "csharp" + ? BuildCallableDefinitionNames(language, request.Symbols) + : null; + var stylusVariableDefinitionNames = language == "stylus" + ? CssReferenceExtractor.BuildStylusVariableDefinitionNames(lines) + : null; + var dockerfileNameSets = language == "dockerfile" + ? DockerfileReferenceExtractor.BuildNameSets( + language, + request.Symbols) + : default; + var shellNameSets = language == "shell" + ? ShellReferenceExtractor.BuildNameSets( + language, + request.Symbols) + : default; + var dynamicDeclarativeState = + DynamicDeclarativeReferenceExtractor.CreateState( + language, + preparedLines, + referenceStructuralLines, + request.Symbols); + IReadOnlyList<(int StartLine, int EndLine)> csharpNamespaceScopes = + language == "csharp" + ? BuildCSharpNamespaceScopes(request.Symbols) + : Array.Empty<(int StartLine, int EndLine)>(); + var csharpUsingImports = language == "csharp" + ? BuildCSharpUsingImports( + language, + request.Symbols, + csharpPatterns.KnownTypeNames, + csharpNamespaceScopes, + lines, + structuralLines) + : ( + Aliases: Array.Empty(), + Namespaces: Array.Empty(), + Statics: Array.Empty()); + var csharpLookups = new CoreCSharpLoopLookups( + csharpPatterns.KnownTypeNames, + csharpPatterns.QualifiedEnumMembers, + csharpPatterns.QualifiedConstantPatternMembers, + csharpPatterns.QualifiedTypePatterns, + csharpUsingImports.Aliases, + csharpUsingImports.Namespaces, + csharpUsingImports.Statics, + callableDefinitionNames); + var languageLookups = new CoreLanguageLoopLookups( + swiftPropertyDefinitionsByLine, + kotlinConstructorTypeNames, + kotlinInfixFunctionNames, + stylusVariableDefinitionNames, + dockerfileNameSets.StageNames, + dockerfileNameSets.VariableNames, + shellNameSets.CallableNames, + shellNameSets.GlobalAliasNames); + var extractionLookups = new CoreExtractionLookups( + request, + language, + request.Symbols, + preSolidityLookups.Containers.Candidates, + preparation.Lines.CSharpLinesInsideMultilineStringContent, + preparedLines, + structuralLines, + lines, + csharpLookups.KnownTypeNames, + csharpLookups.UsingAliases, + csharpLookups.UsingNamespaces); + + return new CorePostSolidityLookups( + csharpLookups, + languageLookups, + dynamicDeclarativeState, + extractionLookups); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Preparation.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Preparation.cs new file mode 100644 index 0000000000..93cfb36087 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.Preparation.cs @@ -0,0 +1,87 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private readonly record struct CoreExtractionPreparation( + ReferenceExtractionContext Request, + ReferenceLinePreparation Lines, + bool IsJsxFile, + bool IsRazorFile, + bool XamlReferenceEnabled); + + private readonly record struct CoreExtractionPreparationOutcome( + bool IsComplete, + List? References, + CoreExtractionPreparation? Preparation) + { + internal static CoreExtractionPreparationOutcome Complete( + List references) + => new(true, references, null); + + internal static CoreExtractionPreparationOutcome Continue( + CoreExtractionPreparation preparation) + => new(false, null, preparation); + } + + private static CoreExtractionPreparationOutcome PrepareCoreExtraction( + ReferenceExtractionContext request) + { + request.CancellationToken.ThrowIfCancellationRequested(); + var language = request.Language; + var isJsxFile = IsJsxFilePath(request.Path); + var isRazorFile = IsRazorFilePath(request.Path) + || request.RequestedLanguage is "razor" or "blazor" or "cshtml"; + + if (language == "ambiguous_m") + return CoreExtractionPreparationOutcome.Complete( + ExtractAmbiguousMReferences(request)); + if (language is "clojure" or "erlang" or "ocaml" or "raku") + return CoreExtractionPreparationOutcome.Complete( + ExtractFunctionalLanguageReferences(request)); + + if (TryExtractStructuralMetadataReferences( + request.FileId, + language, + request.Content, + request.Symbols, + request.Path, + request.ContentIsNormalized, + request.HasOversizeLine, + request.ConflictMarkerLine, + request.MaxReferenceCount, + request.CancellationToken, + out var structuralMetadataReferences)) + { + return CoreExtractionPreparationOutcome.Complete( + structuralMetadataReferences); + } + + if (!TryPrepareReferenceLines( + language, + request.Content, + isRazorFile, + request.ContentIsNormalized, + request.HasOversizeLine, + request.ConflictMarkerLine, + out var preparedInput)) + { + return CoreExtractionPreparationOutcome.Complete([]); + } + request.CancellationToken.ThrowIfCancellationRequested(); + + var xamlReferenceEnabled = language == "xml" + && XamlReferenceExtractor.IsXaml(preparedInput.Lines); + if (language == "xml" && !xamlReferenceEnabled) + return CoreExtractionPreparationOutcome.Complete([]); + + return CoreExtractionPreparationOutcome.Continue( + new CoreExtractionPreparation( + request, + preparedInput, + isJsxFile, + isRazorFile, + xamlReferenceEnabled)); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.cs index 7639cf8468..53aae9bf88 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreExtraction.cs @@ -7,441 +7,18 @@ public static partial class ReferenceExtractor internal static List ExtractCore( ReferenceExtractionContext request) { - request.CancellationToken.ThrowIfCancellationRequested(); - var fileId = request.FileId; - var language = request.Language; - var content = request.Content; - var symbols = request.Symbols; - var path = request.Path; - var workspaceSymbols = request.WorkspaceSymbols; - var requestedLanguage = request.RequestedLanguage; - var isJsxFile = IsJsxFilePath(path); - var isRazorFile = IsRazorFilePath(path) - || requestedLanguage is "razor" or "blazor" or "cshtml"; - - if (language == "ambiguous_m") - return ExtractAmbiguousMReferences(request); - if (language is "clojure" or "erlang" or "ocaml" or "raku") - return ExtractFunctionalLanguageReferences(request); - - if (TryExtractStructuralMetadataReferences( - fileId, - language, - content, - symbols, - path, - request.ContentIsNormalized, - request.HasOversizeLine, - request.ConflictMarkerLine, - request.MaxReferenceCount, - request.CancellationToken, - out var structuralMetadataReferences)) - { - return structuralMetadataReferences; - } - - if (!TryPrepareReferenceLines( - language, - content, - isRazorFile, - request.ContentIsNormalized, - request.HasOversizeLine, - request.ConflictMarkerLine, - out var preparedInput)) - { - return []; - } - request.CancellationToken.ThrowIfCancellationRequested(); - - content = preparedInput.Content; - var lines = preparedInput.Lines; - var xamlReferenceEnabled = language == "xml" - && XamlReferenceExtractor.IsXaml(lines); - if (language == "xml" && !xamlReferenceEnabled) - return []; - - var structuralLines = preparedInput.StructuralLines; - var referenceStructuralLines = - preparedInput.ReferenceStructuralLines; - var preparedLines = preparedInput.PreparedLines; - var scientificNativeDependencyLimit = - ScientificNativeReferenceExtractor.Supports(language) - ? GetSafetyLimits().MaxNamesPerLine - : 0; - var typeScriptTypeAliases = language == "typescript" - ? TypeScriptReferenceExtractor.BuildTypeAliasTargets(preparedLines) - : null; - var swiftTypeAliases = language == "swift" - ? SwiftReferenceExtractor.BuildTypeAliasTargets(preparedLines) - : null; - - var csharpAttrTables = language == "csharp" - && content.Contains('[', StringComparison.Ordinal) - ? BuildCSharpAttributeRanges(preparedLines) - : (null, null); - var csharpAttrRanges = csharpAttrTables.Item1; - var csharpAttrTopLevelRanges = csharpAttrTables.Item2; - var definitionNamesComparer = - GetDefinitionNamesComparer(language); - var definitionNamesByLine = BuildDefinitionNamesByLine( - language, - symbols, - request.ReportDiagnostic); - var scientificDefinitionNameIndicesByLine = - BuildScientificDefinitionNameIndicesByLine( - language, - lines, - symbols, - definitionNamesByLine); - var allDefinitionNames = language == "stylus" - ? BuildAllDefinitionNames( - language, - symbols, - request.ReportDiagnostic) - : null; - var fileDefinitionNames = isRazorFile - ? BuildFileDefinitionNames(symbols) - : null; - var sqlDefinitionLeafSpansByLine = language == "sql" - ? SqlReferenceExtractor.BuildDefinitionLeafSpansByLine( - lines, - symbols) - : null; - var sqlWindowFunctionCallSiteSuppressions = language == "sql" - ? SqlReferenceExtractor - .BuildWindowFunctionCallSiteSuppressions(structuralLines) - : null; - var cobolCallableSymbols = language == "cobol" - ? BuildCobolCallableSymbols(symbols) - : null; - var containerCandidates = BuildReferenceContainerCandidates( - language, - symbols, - request.ReportDiagnostic); - var containerResolver = - new InnermostContainerResolver( - containerCandidates, - preferCallable: language == "csharp"); - if (language == "solidity") - { - return ExtractSolidityReferences( - fileId, - lines, - preparedLines, - containerResolver); - } - - var swiftPropertyDefinitionsByLine = language == "swift" - ? BuildSwiftPropertyDefinitionsByLine( - language, - symbols, - request.ReportDiagnostic) - : null; - var csharpTypeNameSets = language == "csharp" - ? BuildCSharpTypeNameSets(language, symbols) - : ( - KnownTypeNames: EmptyCSharpStringSet, - NonEnumTypeNames: EmptyCSharpStringSet); - var csharpKnownTypeNames = csharpTypeNameSets.KnownTypeNames; - var localCSharpQualifiedPatternLookups = language == "csharp" - ? BuildCSharpQualifiedPatternLookups( - symbols, - csharpTypeNameSets.NonEnumTypeNames) - : null; - var workspaceCSharpQualifiedPatternLookups = language == "csharp" - ? request.CSharpQualifiedPatternLookups - ?? (workspaceSymbols is { Count: > 0 } - ? BuildCSharpQualifiedPatternLookups(workspaceSymbols) - : localCSharpQualifiedPatternLookups) - : null; - var csharpQualifiedPatternLookups = language == "csharp" - ? new CSharpQualifiedPatternLookups( - workspaceCSharpQualifiedPatternLookups!.EnumMemberLookup, - workspaceCSharpQualifiedPatternLookups.ConstantPatternMemberLookup, - localCSharpQualifiedPatternLookups!.TypePatternLookup) - : new CSharpQualifiedPatternLookups( - EmptyCSharpQualifiedEnumMemberLookup, - EmptyCSharpQualifiedPatternLookup, - EmptyCSharpQualifiedPatternLookup); - var csharpQualifiedEnumMemberLookup = - csharpQualifiedPatternLookups.EnumMemberLookup; - var csharpQualifiedConstantPatternMemberLookup = - csharpQualifiedPatternLookups.ConstantPatternMemberLookup; - var csharpQualifiedTypePatternLookup = - csharpQualifiedPatternLookups.TypePatternLookup; - - HashSet? kotlinConstructorTypeNames = null; - HashSet? kotlinInfixFunctionNames = null; - if (language == "kotlin") - { - var kotlinNameSets = - KotlinReferenceExtractor.BuildNameSets(language, symbols); - kotlinConstructorTypeNames = - kotlinNameSets.ConstructorTypeNames; - kotlinInfixFunctionNames = kotlinNameSets.InfixFunctionNames; - KotlinReferenceExtractor.AddDeclaredInfixFunctionNames( - lines, - kotlinInfixFunctionNames); - } - - var callableDefinitionNames = language == "csharp" - ? BuildCallableDefinitionNames(language, symbols) - : null; - var stylusVariableDefinitionNames = language == "stylus" - ? CssReferenceExtractor.BuildStylusVariableDefinitionNames(lines) - : null; - var dockerfileNameSets = language == "dockerfile" - ? DockerfileReferenceExtractor.BuildNameSets(language, symbols) - : default; - var shellNameSets = language == "shell" - ? ShellReferenceExtractor.BuildNameSets(language, symbols) - : default; - var dynamicDeclarativeState = - DynamicDeclarativeReferenceExtractor.CreateState( - language, - preparedLines, - referenceStructuralLines, - symbols); - IReadOnlyList<(int StartLine, int EndLine)> csharpNamespaceScopes = - language == "csharp" - ? BuildCSharpNamespaceScopes(symbols) - : Array.Empty<(int StartLine, int EndLine)>(); - var csharpUsingImports = language == "csharp" - ? BuildCSharpUsingImports( - language, - symbols, - csharpKnownTypeNames, - csharpNamespaceScopes, - lines, - structuralLines) - : ( - Aliases: Array.Empty(), - Namespaces: Array.Empty(), - Statics: Array.Empty()); - var csharpUsingAliases = csharpUsingImports.Aliases; - var csharpUsingStatics = csharpUsingImports.Statics; - var lookups = new CoreExtractionLookups( - request, - language, - symbols, - containerCandidates, - preparedInput.CSharpLinesInsideMultilineStringContent, - preparedLines, - structuralLines, - lines, - csharpKnownTypeNames, - csharpUsingAliases, - csharpUsingImports.Namespaces); - - var references = CreateReferenceList( - request.MaxReferenceCount, - EstimateReferenceListInitialCapacity(lines.Length)); - var seen = CreateReferenceSeenSet(lines.Length); - if (language == "csharp") - { - EmitCSharpAsyncIteratorReferences( - fileId, - lines, - structuralLines, - symbols, - references, - seen); - EmitCSharpStaticInterfaceMemberImplementationReferences( - fileId, - lines, - structuralLines, - symbols, - workspaceSymbols ?? symbols, - request.CSharpStaticInterfaceMemberLookups, - references, - seen); - } - else if (language == "rust") - { - RustReferenceExtractor.EmitMultilineAttributeReferences( - preparedLines, - references, - seen, - fileId, - (lineNumber, _) => - FindInnermostContainer( - containerCandidates, - lineNumber)); - } - - var pendingCSharpMultiLineTypePattern = - EmitCoreReferenceLines( - new CoreReferenceLoopContext - { - Request = request, - Preparation = preparedInput, - IsJsxFile = isJsxFile, - IsRazorFile = isRazorFile, - XamlReferenceEnabled = xamlReferenceEnabled, - ScientificNativeDependencyLimit = - scientificNativeDependencyLimit, - CSharpAttributeRanges = csharpAttrRanges, - CSharpAttributeTopLevelRanges = - csharpAttrTopLevelRanges, - DefinitionNamesComparer = definitionNamesComparer, - DefinitionNamesByLine = definitionNamesByLine, - ScientificDefinitionNameIndicesByLine = - scientificDefinitionNameIndicesByLine, - AllDefinitionNames = allDefinitionNames, - FileDefinitionNames = fileDefinitionNames, - SqlDefinitionLeafSpansByLine = - sqlDefinitionLeafSpansByLine, - SqlWindowFunctionCallSiteSuppressions = - sqlWindowFunctionCallSiteSuppressions, - CobolCallableSymbols = cobolCallableSymbols, - ContainerCandidates = containerCandidates, - ContainerResolver = containerResolver, - SwiftPropertyDefinitionsByLine = - swiftPropertyDefinitionsByLine, - CSharpQualifiedEnumMemberLookup = - csharpQualifiedEnumMemberLookup, - CSharpQualifiedConstantPatternMemberLookup = - csharpQualifiedConstantPatternMemberLookup, - CSharpQualifiedTypePatternLookup = - csharpQualifiedTypePatternLookup, - KotlinConstructorTypeNames = - kotlinConstructorTypeNames, - KotlinInfixFunctionNames = kotlinInfixFunctionNames, - CallableDefinitionNames = callableDefinitionNames, - StylusVariableDefinitionNames = - stylusVariableDefinitionNames, - DockerfileStageNames = - dockerfileNameSets.StageNames, - DockerfileVariableNames = - dockerfileNameSets.VariableNames, - ShellCallableNames = shellNameSets.CallableNames, - ShellGlobalAliasNames = - shellNameSets.GlobalAliasNames, - DynamicDeclarativeState = dynamicDeclarativeState, - CSharpUsingAliases = csharpUsingAliases, - CSharpUsingStatics = csharpUsingStatics, - Lookups = lookups, - TypeScriptTypeAliases = typeScriptTypeAliases, - SwiftTypeAliases = swiftTypeAliases, - References = references, - Seen = seen, - }); - - if (!ReferenceLimitReached(references) && language == "csharp") - { - CSharpReferenceExtractor.EmitSwitchExpressionTypePatternReferences( - lines, - preparedLines, - containerCandidates, - csharpQualifiedConstantPatternMemberLookup, - csharpQualifiedTypePatternLookup, - csharpUsingAliases, - csharpUsingStatics, - lookups.HasActiveSameFileCSharpTypeCandidate, - references, - seen, - fileId); - - CSharpReferenceExtractor - .FlushPendingMultiLineTypePatternReference( - ref pendingCSharpMultiLineTypePattern, - csharpQualifiedConstantPatternMemberLookup, - csharpUsingAliases, - csharpUsingStatics, - lookups.HasActiveSameFileCSharpTypeCandidate, - references, - seen, - fileId); - } - - if (language == "csharp") - { - RewriteCSharpPropertyReceiverReferences( - preparedLines, - references, - lookups); - RemoveCSharpCallsDuplicatedByMemberReads(references); - } - - lookups.ApplyCSharpUsingAliasReferenceNames(references); - if (!ReferenceLimitReached(references)) - { - lookups.EmitCSharpBclRegexWithoutTimeoutReferences( - references, - seen); - } - MarkMutualRecursionReferences(references); - return references; - } - - private static void RemoveCSharpCallsDuplicatedByMemberReads( - List references) - { - var memberReadSites = references - .Where(reference => reference.ReferenceKind == "member_read") - .Select(reference => ( - reference.FileId, - reference.Line, - reference.Column, - reference.SymbolName, - reference.ContainerKind, - reference.ContainerName)) - .ToHashSet(); - if (memberReadSites.Count == 0) - return; - - references.RemoveAll(reference => - reference.ReferenceKind == "call" - && memberReadSites.Contains(( - reference.FileId, - reference.Line, - reference.Column, - reference.SymbolName, - reference.ContainerKind, - reference.ContainerName))); - } - - private static void RewriteCSharpPropertyReceiverReferences( - IReadOnlyList preparedLines, - List references, - CoreExtractionLookups lookups) - { - foreach (var reference in references) - { - if (reference.ReferenceKind != "type_reference" - || reference.Line <= 0 - || reference.Line > preparedLines.Count - || reference.Column <= 0) - { - continue; - } - - var line = preparedLines[reference.Line - 1]; - var tokenEnd = - reference.Column - 1 + reference.SymbolName.Length; - if (tokenEnd >= line.Length - || !line.AsSpan(tokenEnd) - .TrimStart() - .StartsWith(".", StringComparison.Ordinal)) - { - continue; - } - - var owner = lookups.FindCSharpContainerCandidate( - reference.ContainerName, - reference.Line); - var containingType = GetContainingTypeQualifiedName(owner); - if (containingType == null - || !lookups.HasCSharpFieldOrPropertyMember( - containingType, - reference.SymbolName)) - { - continue; - } - - reference.SymbolName = - $"{containingType}.{reference.SymbolName}"; - reference.ReferenceKind = "reference"; - } + var preparationOutcome = PrepareCoreExtraction(request); + if (preparationOutcome.IsComplete) + return preparationOutcome.References!; + + var preparation = preparationOutcome.Preparation!.Value; + var preSolidityLookups = BuildCorePreSolidityLookups(preparation); + if (request.Language == "solidity") + return ExtractCoreSolidityReferences(preparation, preSolidityLookups); + + var loop = CreateCoreReferenceLoopContext(preparation, preSolidityLookups); + EmitCoreExtractionPrelude(loop); + var pendingCSharpMultiLineTypePattern = EmitCoreReferenceLines(loop); + return FinalizeCoreExtraction(loop, pendingCSharpMultiLineTypePattern); } } diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreJvmDocumentationLines.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreJvmDocumentationLines.cs new file mode 100644 index 0000000000..13440c4dc9 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreJvmDocumentationLines.cs @@ -0,0 +1,59 @@ +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static void EmitCoreJvmDocumentationReferences( + in CoreDocumentationLineContext line, + ref bool inDelimitedDocComment) + { + if (!TryGetJvmDocCommentSpan( + line.OriginalLine, + inDelimitedDocComment, + out var docCommentStartIndex, + out var docCommentEndExclusive, + out var sameLineDeclarationStartColumn, + out var nextDelimitedDocComment)) + { + return; + } + + if (docCommentEndExclusive > docCommentStartIndex) + { + EmitCoreJvmDocLinkReferences( + in line, + docCommentStartIndex, + docCommentEndExclusive, + sameLineDeclarationStartColumn); + } + + inDelimitedDocComment = nextDelimitedDocComment; + } + + private static void EmitCoreJvmDocLinkReferences( + in CoreDocumentationLineContext line, + int docCommentStartIndex, + int docCommentEndExclusive, + int sameLineDeclarationStartColumn) + { + var docContainer = FindJvmDocumentedContainer( + line.ContainerCandidates, + line.Lines, + line.StructuralLines[line.LineIndex], + line.LineNumber, + sameLineDeclarationStartColumn); + if (docContainer == null) + return; + + var docText = line.OriginalLine[docCommentStartIndex..docCommentEndExclusive]; + EmitJvmDocLinkReferences( + line.Language, + docText, + line.References, + line.Seen, + line.FileId, + docCommentStartIndex, + docText.Trim(), + line.LineNumber, + docContainer); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreLanguageLines.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreLanguageLines.cs index 7f178eb883..0948d4ec3e 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreLanguageLines.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreLanguageLines.cs @@ -23,7 +23,7 @@ private readonly record struct CoreReferenceLineContext( Func IsIgnoredCallName); private static void EmitJavaScriptTaggedTemplateReferences( - CoreReferenceLineContext line, + in CoreReferenceLineContext line, IReadOnlyList tagHitsOnLine) { foreach (var hit in tagHitsOnLine) @@ -59,7 +59,7 @@ private static void EmitJavaScriptTaggedTemplateReferences( } private static void EmitMetadataLineReferences( - CoreReferenceLineContext line, + in CoreReferenceLineContext line, List<(int start, int end)>? csharpAttrTopLevelOnLine) { if (line.Language == "csharp" && csharpAttrTopLevelOnLine != null && csharpAttrTopLevelOnLine.Count > 0) @@ -153,454 +153,4 @@ private static void EmitMetadataLineReferences( } } } - - private static Func CreatePythonDefinitionContainerResolver( - CoreReferenceLineContext line, - CoreExtractionLookups lookups, - SymbolRecord? headerContainer, - string definitionKind) => - column => - { - if (headerContainer != null) - return headerContainer; - - var container = line.ResolveContainerForCall(column); - if (container != null) - return container; - - var definitionContainers = - lookups.GetPythonDefinitionContainersByLineAndKind(); - if (definitionContainers == null) - return null; - return definitionContainers.TryGetValue( - (line.LineNumber, definitionKind), - out var symbol) - ? symbol - : null; - }; - - private static void EmitPythonLineReferences( - CoreReferenceLineContext line, - CoreExtractionLookups lookups) - { - - var pythonPreparedLine = line.PreparedLine; - var pythonHeaderMap = default(PythonLogicalHeaderReferenceLine?); - SymbolRecord? pythonHeaderSymbol = null; - lookups.GetPythonHeaderSymbolsByLine()?.TryGetValue(line.LineNumber, out pythonHeaderSymbol); - if (pythonHeaderSymbol?.Signature != null - && TryBuildPythonLogicalHeaderReferenceLine(line.Lines, line.LineIndex, pythonHeaderSymbol.StartColumn ?? 0, out var builtPythonHeaderMap)) - { - pythonPreparedLine = builtPythonHeaderMap.Text; - pythonHeaderMap = builtPythonHeaderMap; - } - var pythonTypeFactoryLine = line.PreparedLine; - var pythonTypeFactoryMap = default(PythonLogicalHeaderReferenceLine?); - if (line.PreparedLine.Contains("TypeVar", StringComparison.Ordinal) - || line.PreparedLine.Contains("ParamSpec", StringComparison.Ordinal)) - { - var typeFactoryStartColumn = line.OriginalLine.IndexOfAny(['T', 'P']); - if (typeFactoryStartColumn < 0) - typeFactoryStartColumn = 0; - if (TryBuildPythonLogicalStatementReferenceLine(line.Lines, line.LineIndex, typeFactoryStartColumn, out var builtPythonTypeFactoryMap)) - { - pythonTypeFactoryLine = builtPythonTypeFactoryMap.Text; - pythonTypeFactoryMap = builtPythonTypeFactoryMap; - } - } - var pythonHeaderContainer = pythonHeaderSymbol ?? line.Container; - var resolvePythonClassContainer = - pythonPreparedLine.IndexOf("class", StringComparison.Ordinal) >= 0 - ? CreatePythonDefinitionContainerResolver( - line, - lookups, - pythonHeaderContainer, - "class") - : line.ResolveContainerForCall; - var resolvePythonFunctionContainer = - pythonPreparedLine.IndexOf("def", StringComparison.Ordinal) >= 0 - ? CreatePythonDefinitionContainerResolver( - line, - lookups, - pythonHeaderContainer, - "function") - : line.ResolveContainerForCall; - - var pythonReferenceStart = line.References.Count; - PythonReferenceExtractor.EmitDecoratorReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.DefinitionNames, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitRaiseReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitExceptReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitIsInstanceReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitIsSubclassReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitCastReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitAssertTypeReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitClassBaseReferences( - pythonPreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - pythonHeaderContainer, - resolvePythonClassContainer, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitFunctionReturnReferences( - pythonPreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - pythonHeaderContainer, - resolvePythonFunctionContainer, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitFunctionParameterReferences( - pythonPreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - pythonHeaderContainer, - resolvePythonFunctionContainer, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitVariableAnnotationReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitTypeAliasReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitNewTypeReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - var pythonTypeFactoryReferenceStart = line.References.Count; - PythonReferenceExtractor.EmitTypeVarBoundReferences( - pythonTypeFactoryLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitTypeVarConstraintReferences( - pythonTypeFactoryLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitGetTypeHintsReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitDataclassesFieldsReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitDataclassFieldReferences( - line.PreparedLines, - line.Lines, - line.LineIndex, - line.References, - line.Seen, - line.FileId, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitAttrsFieldsReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitPydanticTypeAdapterReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitPytestRaisesReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - PythonReferenceExtractor.EmitContextlibSuppressReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.IsIgnoredCallName); - - if (pythonTypeFactoryMap.HasValue) - RemapPythonLogicalHeaderReferences(line.References, pythonTypeFactoryReferenceStart, pythonTypeFactoryMap.Value, line.Lines); - PythonReferenceExtractor.EmitDynamicImportReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - if (pythonHeaderMap.HasValue) - RemapPythonLogicalHeaderReferences(line.References, pythonReferenceStart, pythonHeaderMap.Value, line.Lines); - } - - private static void EmitRLineReferences(CoreReferenceLineContext line) - { - - RReferenceExtractor.EmitNamespaceReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.DefinitionNames); - RReferenceExtractor.EmitNamespaceDirectiveReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - RReferenceExtractor.EmitS4DispatchReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - RReferenceExtractor.EmitBacktickCallReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.DefinitionNames); - RReferenceExtractor.EmitInfixOperatorCallReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.DefinitionNames); - RReferenceExtractor.EmitSourceFileReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - RReferenceExtractor.EmitLoadAllReferences( - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - RReferenceExtractor.EmitDataCallReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - RReferenceExtractor.EmitSystemFileReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - RReferenceExtractor.EmitVignetteReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - RReferenceExtractor.EmitHelpExampleReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - RReferenceExtractor.EmitInstallPackagesReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - RReferenceExtractor.EmitNamespacePackageInstallReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - RReferenceExtractor.EmitGitHubPackageInstallReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - RReferenceExtractor.EmitDollarMemberReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.DefinitionNames); - RReferenceExtractor.EmitBracketMemberReferences( - line.PreparedLine, - line.OriginalLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.DefinitionNames); - RReferenceExtractor.EmitSlotMemberReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - line.DefinitionNames); - } } diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CorePhpDocumentationLines.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CorePhpDocumentationLines.cs new file mode 100644 index 0000000000..2b965e078f --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CorePhpDocumentationLines.cs @@ -0,0 +1,217 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private readonly record struct CorePhpDocumentationLineContext( + long FileId, + string OriginalLine, + int LineNumber, + List References, + ReferenceDedupeSet Seen, + InnermostContainerResolver ContainerResolver); + + private struct PhpDocumentationState + { + internal bool InDocblock; + internal SymbolRecord? DocblockContainer; + internal HashSet? DocblockPropertyNames; + } + + private struct PhpLineContainerCache + { + private readonly InnermostContainerResolver _resolver; + private readonly int _lineNumber; + private bool _resolved; + private SymbolRecord? _container; + + internal PhpLineContainerCache( + InnermostContainerResolver resolver, + int lineNumber) + { + _resolver = resolver; + _lineNumber = lineNumber; + } + + internal SymbolRecord? Resolve() + { + if (!_resolved) + { + _container = _resolver.Find(_lineNumber); + _resolved = true; + } + + return _container; + } + } + + private static void EmitPhpDocumentationReferences( + in CorePhpDocumentationLineContext line, + ref PhpDocumentationState state) + { + var containerCache = new PhpLineContainerCache( + line.ContainerResolver, + line.LineNumber); + EmitPhpAttributeDocumentationReferences(in line, ref containerCache); + BeginPhpDocblockIfNeeded(in line, ref state, ref containerCache); + + var context = line.OriginalLine.Trim(); + if (context.Length > 0) + { + EmitPhpDocblockSignatureReferences(in line, context, ref state, ref containerCache); + EmitPhpDocblockInheritanceReferences(in line, context, ref state, ref containerCache); + EmitPhpDocblockPropertyReferences(in line, context, ref state, ref containerCache); + EmitPhpDocblockMethodReferences(in line, context, ref state, ref containerCache); + EmitPhpDocblockTemplateAndAliasReferences(in line, context, ref state, ref containerCache); + } + + EndPhpDocblockIfNeeded(in line, ref state); + } + + private static void EmitPhpAttributeDocumentationReferences( + in CorePhpDocumentationLineContext line, + ref PhpLineContainerCache containerCache) + { + if (!line.OriginalLine.Contains("#[", StringComparison.Ordinal)) + return; + + var context = line.OriginalLine.Trim(); + if (context.Length == 0) + return; + + PhpReferenceExtractor.EmitAttributeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, containerCache.Resolve()); + } + + private static void BeginPhpDocblockIfNeeded( + in CorePhpDocumentationLineContext line, + ref PhpDocumentationState state, + ref PhpLineContainerCache containerCache) + { + if (line.OriginalLine.IndexOf("/**", StringComparison.Ordinal) < 0) + return; + + state.InDocblock = true; + state.DocblockContainer = containerCache.Resolve(); + state.DocblockPropertyNames = null; + } + + private static void EmitPhpDocblockSignatureReferences( + in CorePhpDocumentationLineContext line, + string context, + ref PhpDocumentationState state, + ref PhpLineContainerCache containerCache) + { + if (line.OriginalLine.Contains("param", StringComparison.OrdinalIgnoreCase)) + PhpReferenceExtractor.EmitDocblockParamTypeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + if (line.OriginalLine.Contains("return", StringComparison.OrdinalIgnoreCase)) + PhpReferenceExtractor.EmitDocblockReturnTypeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + if (line.OriginalLine.Contains("var", StringComparison.OrdinalIgnoreCase)) + PhpReferenceExtractor.EmitDocblockVarTypeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + if (line.OriginalLine.Contains("@throws", StringComparison.OrdinalIgnoreCase)) + PhpReferenceExtractor.EmitDocblockThrowsTypeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + } + + private static void EmitPhpDocblockInheritanceReferences( + in CorePhpDocumentationLineContext line, + string context, + ref PhpDocumentationState state, + ref PhpLineContainerCache containerCache) + { + if (line.OriginalLine.Contains("extends", StringComparison.OrdinalIgnoreCase)) + PhpReferenceExtractor.EmitDocblockExtendsTypeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + if (line.OriginalLine.Contains("implements", StringComparison.OrdinalIgnoreCase)) + PhpReferenceExtractor.EmitDocblockImplementsTypeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + if (line.OriginalLine.Contains("@mixin", StringComparison.OrdinalIgnoreCase)) + PhpReferenceExtractor.EmitDocblockMixinTypeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + } + + private static void EmitPhpDocblockPropertyReferences( + in CorePhpDocumentationLineContext line, + string context, + ref PhpDocumentationState state, + ref PhpLineContainerCache containerCache) + { + if (!line.OriginalLine.Contains("property", StringComparison.OrdinalIgnoreCase)) + return; + + PhpReferenceExtractor.EmitDocblockPropertyTypeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache), + state.InDocblock, ref state.DocblockPropertyNames); + } + + private static void EmitPhpDocblockMethodReferences( + in CorePhpDocumentationLineContext line, + string context, + ref PhpDocumentationState state, + ref PhpLineContainerCache containerCache) + { + if (!line.OriginalLine.Contains("@method", StringComparison.OrdinalIgnoreCase)) + return; + + PhpReferenceExtractor.EmitDocblockMethodReturnTypeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + PhpReferenceExtractor.EmitDocblockMethodParameterTypeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + } + + private static void EmitPhpDocblockTemplateAndAliasReferences( + in CorePhpDocumentationLineContext line, + string context, + ref PhpDocumentationState state, + ref PhpLineContainerCache containerCache) + { + if (line.OriginalLine.Contains("@template", StringComparison.OrdinalIgnoreCase)) + PhpReferenceExtractor.EmitDocblockTemplateBoundTypeReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + if (!line.OriginalLine.Contains("type", StringComparison.OrdinalIgnoreCase)) + return; + + PhpReferenceExtractor.EmitDocblockTypeAliasTargetReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + PhpReferenceExtractor.EmitDocblockImportTypeSourceReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, ResolvePhpDocblockContainer(ref state, ref containerCache)); + } + + private static SymbolRecord? ResolvePhpDocblockContainer( + ref PhpDocumentationState state, + ref PhpLineContainerCache containerCache) + => state.InDocblock ? state.DocblockContainer : containerCache.Resolve(); + + private static void EndPhpDocblockIfNeeded( + in CorePhpDocumentationLineContext line, + ref PhpDocumentationState state) + { + if (!state.InDocblock + || line.OriginalLine.IndexOf("*/", StringComparison.Ordinal) < 0) + { + return; + } + + state.InDocblock = false; + state.DocblockContainer = null; + state.DocblockPropertyNames = null; + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CorePhpLines.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CorePhpLines.cs new file mode 100644 index 0000000000..f17ba44961 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CorePhpLines.cs @@ -0,0 +1,62 @@ +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static void EmitPhpLineReferences(in CoreReferenceLineContext line) + { + if (line.Language != "php") + return; + + EmitPhpTypeLineReferences(in line); + EmitPhpImportLineReferences(in line); + EmitPhpMemberLineReferences(in line); + } + + private static void EmitPhpTypeLineReferences( + in CoreReferenceLineContext line) + { + PhpReferenceExtractor.EmitStaticAccessReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + PhpReferenceExtractor.EmitInstanceofReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + PhpReferenceExtractor.EmitCatchTypeReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + PhpReferenceExtractor.EmitReturnTypeReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + PhpReferenceExtractor.EmitParameterTypeReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + PhpReferenceExtractor.EmitPropertyTypeReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + PhpReferenceExtractor.EmitInheritanceTypeReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + } + + private static void EmitPhpImportLineReferences( + in CoreReferenceLineContext line) + { + PhpReferenceExtractor.EmitUseTypeReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + PhpReferenceExtractor.EmitUseFunctionReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + PhpReferenceExtractor.EmitUseConstReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + } + + private static void EmitPhpMemberLineReferences( + in CoreReferenceLineContext line) + { + PhpReferenceExtractor.EmitObjectMemberAccessReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CorePythonLines.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CorePythonLines.cs new file mode 100644 index 0000000000..0d929277a5 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CorePythonLines.cs @@ -0,0 +1,228 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private readonly record struct CorePythonReferenceLinePreparation( + string HeaderLine, + PythonLogicalHeaderReferenceLine? HeaderMap, + SymbolRecord? HeaderContainer, + Func ResolveClassContainer, + Func ResolveFunctionContainer, + string TypeFactoryLine, + PythonLogicalHeaderReferenceLine? TypeFactoryMap); + + private static Func CreatePythonDefinitionContainerResolver( + in CoreReferenceLineContext line, + CoreExtractionLookups lookups, + SymbolRecord? headerContainer, + string definitionKind) + { + var resolveContainerForCall = line.ResolveContainerForCall; + var lineNumber = line.LineNumber; + return column => + { + if (headerContainer != null) + return headerContainer; + + var container = resolveContainerForCall(column); + if (container != null) + return container; + + var definitionContainers = + lookups.GetPythonDefinitionContainersByLineAndKind(); + if (definitionContainers == null) + return null; + return definitionContainers.TryGetValue( + (lineNumber, definitionKind), + out var symbol) + ? symbol + : null; + }; + } + + private static CorePythonReferenceLinePreparation PrepareCorePythonReferenceLine( + in CoreReferenceLineContext line, + CoreExtractionLookups lookups) + { + var headerLine = line.PreparedLine; + var headerMap = default(PythonLogicalHeaderReferenceLine?); + SymbolRecord? headerSymbol = null; + lookups.GetPythonHeaderSymbolsByLine()?.TryGetValue(line.LineNumber, out headerSymbol); + if (headerSymbol?.Signature != null + && TryBuildPythonLogicalHeaderReferenceLine( + line.Lines, + line.LineIndex, + headerSymbol.StartColumn ?? 0, + out var builtHeaderMap)) + { + headerLine = builtHeaderMap.Text; + headerMap = builtHeaderMap; + } + + var typeFactoryLine = line.PreparedLine; + var typeFactoryMap = default(PythonLogicalHeaderReferenceLine?); + if (line.PreparedLine.Contains("TypeVar", StringComparison.Ordinal) + || line.PreparedLine.Contains("ParamSpec", StringComparison.Ordinal)) + { + var startColumn = line.OriginalLine.IndexOfAny(['T', 'P']); + if (startColumn < 0) + startColumn = 0; + if (TryBuildPythonLogicalStatementReferenceLine( + line.Lines, + line.LineIndex, + startColumn, + out var builtTypeFactoryMap)) + { + typeFactoryLine = builtTypeFactoryMap.Text; + typeFactoryMap = builtTypeFactoryMap; + } + } + + var headerContainer = headerSymbol ?? line.Container; + var resolveClassContainer = headerLine.IndexOf("class", StringComparison.Ordinal) >= 0 + ? CreatePythonDefinitionContainerResolver( + in line, + lookups, + headerContainer, + "class") + : line.ResolveContainerForCall; + var resolveFunctionContainer = headerLine.IndexOf("def", StringComparison.Ordinal) >= 0 + ? CreatePythonDefinitionContainerResolver( + in line, + lookups, + headerContainer, + "function") + : line.ResolveContainerForCall; + return new CorePythonReferenceLinePreparation( + headerLine, + headerMap, + headerContainer, + resolveClassContainer, + resolveFunctionContainer, + typeFactoryLine, + typeFactoryMap); + } + + private static void EmitPythonLineReferences( + in CoreReferenceLineContext line, + CoreExtractionLookups lookups) + { + var preparation = PrepareCorePythonReferenceLine(in line, lookups); + var referenceStart = line.References.Count; + EmitPythonRuntimeTypeReferences(in line); + EmitPythonDeclarationTypeReferences(in line, in preparation); + + var typeFactoryReferenceStart = line.References.Count; + EmitPythonTypingAndFrameworkReferences( + in line, + preparation.TypeFactoryLine); + if (preparation.TypeFactoryMap.HasValue) + { + RemapPythonLogicalHeaderReferences( + line.References, + typeFactoryReferenceStart, + preparation.TypeFactoryMap.Value, + line.Lines); + } + + PythonReferenceExtractor.EmitDynamicImportReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container); + if (preparation.HeaderMap.HasValue) + { + RemapPythonLogicalHeaderReferences( + line.References, + referenceStart, + preparation.HeaderMap.Value, + line.Lines); + } + } + + private static void EmitPythonRuntimeTypeReferences( + in CoreReferenceLineContext line) + { + PythonReferenceExtractor.EmitDecoratorReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.DefinitionNames, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitRaiseReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitExceptReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitIsInstanceReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitIsSubclassReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitCastReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitAssertTypeReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + } + + private static void EmitPythonDeclarationTypeReferences( + in CoreReferenceLineContext line, + in CorePythonReferenceLinePreparation preparation) + { + PythonReferenceExtractor.EmitClassBaseReferences( + preparation.HeaderLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, preparation.HeaderContainer, preparation.ResolveClassContainer, + line.IsIgnoredCallName); + PythonReferenceExtractor.EmitFunctionReturnReferences( + preparation.HeaderLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, preparation.HeaderContainer, preparation.ResolveFunctionContainer, + line.IsIgnoredCallName); + PythonReferenceExtractor.EmitFunctionParameterReferences( + preparation.HeaderLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, preparation.HeaderContainer, preparation.ResolveFunctionContainer, + line.IsIgnoredCallName); + PythonReferenceExtractor.EmitVariableAnnotationReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitTypeAliasReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitNewTypeReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + } + + private static void EmitPythonTypingAndFrameworkReferences( + in CoreReferenceLineContext line, + string typeFactoryLine) + { + PythonReferenceExtractor.EmitTypeVarBoundReferences( + typeFactoryLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitTypeVarConstraintReferences( + typeFactoryLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitGetTypeHintsReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitDataclassesFieldsReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitDataclassFieldReferences( + line.PreparedLines, line.Lines, line.LineIndex, line.References, line.Seen, + line.FileId, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitAttrsFieldsReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitPydanticTypeAdapterReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitPytestRaisesReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + PythonReferenceExtractor.EmitContextlibSuppressReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.IsIgnoredCallName); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreRLines.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreRLines.cs new file mode 100644 index 0000000000..49e597d7a7 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreRLines.cs @@ -0,0 +1,101 @@ +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static void EmitRDocumentationReferences( + in CoreDocumentationLineContext line) + { + var context = line.OriginalLine.Trim(); + if (context.Length == 0) + return; + + RReferenceExtractor.EmitRoxygenImportFromReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, container: null); + RReferenceExtractor.EmitRoxygenImportReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, container: null); + RReferenceExtractor.EmitRoxygenMethodReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, context, + line.LineNumber, container: null); + } + + private static void EmitRLineReferences(in CoreReferenceLineContext line) + { + EmitRNamespaceAndDispatchReferences(in line); + EmitRWorkspaceAndDataReferences(in line); + EmitRHelpAndPackageReferences(in line); + EmitRMemberReferences(in line); + } + + private static void EmitRNamespaceAndDispatchReferences( + in CoreReferenceLineContext line) + { + RReferenceExtractor.EmitNamespaceReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.DefinitionNames); + RReferenceExtractor.EmitNamespaceDirectiveReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container); + RReferenceExtractor.EmitS4DispatchReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container); + RReferenceExtractor.EmitBacktickCallReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.DefinitionNames); + RReferenceExtractor.EmitInfixOperatorCallReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.DefinitionNames); + } + + private static void EmitRWorkspaceAndDataReferences( + in CoreReferenceLineContext line) + { + RReferenceExtractor.EmitSourceFileReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container); + RReferenceExtractor.EmitLoadAllReferences( + line.OriginalLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + RReferenceExtractor.EmitDataCallReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container); + RReferenceExtractor.EmitSystemFileReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container); + RReferenceExtractor.EmitVignetteReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container); + } + + private static void EmitRHelpAndPackageReferences( + in CoreReferenceLineContext line) + { + RReferenceExtractor.EmitHelpExampleReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container); + RReferenceExtractor.EmitInstallPackagesReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container); + RReferenceExtractor.EmitNamespacePackageInstallReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container); + RReferenceExtractor.EmitGitHubPackageInstallReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container); + } + + private static void EmitRMemberReferences(in CoreReferenceLineContext line) + { + RReferenceExtractor.EmitDollarMemberReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.DefinitionNames); + RReferenceExtractor.EmitBracketMemberReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, + line.FileId, line.Context, line.LineNumber, line.Container, + line.DefinitionNames); + RReferenceExtractor.EmitSlotMemberReferences( + line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container, line.DefinitionNames); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLine.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLine.cs new file mode 100644 index 0000000000..d658e877c5 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLine.cs @@ -0,0 +1,283 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static CoreReferenceLineFlow ProcessCoreReferenceLine( + CoreReferenceLoopContext loop, + CoreReferenceLoopState state, + int lineIndex) + { + if (!TryPrepareCoreReferenceLine( + loop, + state, + lineIndex, + out var prepared)) + { + return CoreReferenceLineFlow.LineConsumed; + } + + return EmitCoreOrderedReferencePhases(loop, state, in prepared); + } + + private static bool TryPrepareCoreReferenceLine( + CoreReferenceLoopContext loop, + CoreReferenceLoopState state, + int lineIndex, + out CorePreparedReferenceLine prepared) + { + var request = loop.Request; + var input = loop.Preparation; + var language = request.Language; + var lineNumber = lineIndex + 1; + var originalLine = input.Lines[lineIndex]; + var languageLines = PrepareCoreLanguageLine( + loop, + state, + lineIndex, + originalLine); + var preparedLine = languageLines.PreparedLine; + var preparedLineIsWhiteSpace = + string.IsNullOrWhiteSpace(preparedLine); + var csharpAttributeRanges = loop.CSharpAttributeRanges?[lineIndex]; + var csharpAttributeTopLevelRanges = + loop.CSharpAttributeTopLevelRanges?[lineIndex]; + + if (EmitCoreDocumentationAndSpecialLineReferences( + loop, + state, + lineIndex, + originalLine, + preparedLine, + preparedLineIsWhiteSpace, + csharpAttributeRanges, + out var sourceContext)) + { + prepared = default; + return false; + } + + if (preparedLineIsWhiteSpace) + { + if (language == "csharp" + && (state.PendingCSharpMultiLineTypePattern.WaitingForHead + || state.PendingCSharpMultiLineTypePattern + .PendingTypeExpression != null)) + { + prepared = default; + return false; + } + + if (language == "csharp") + { + CSharpReferenceExtractor + .FlushPendingMultiLineTypePatternReference( + ref state.PendingCSharpMultiLineTypePattern, + loop.CSharpQualifiedConstantPatternMemberLookup, + loop.CSharpUsingAliases, + loop.CSharpUsingStatics, + loop.Lookups.HasActiveSameFileCSharpTypeCandidate, + loop.References, + loop.Seen, + request.FileId); + } + + prepared = default; + return false; + } + + if (sourceContext.Length == 0) + { + prepared = default; + return false; + } + + var definitionNames = + loop.DefinitionNamesByLine.TryGetValue( + lineNumber, + out var namesOnLine) + ? namesOnLine + : null; + Dictionary>? + scientificDefinitionNameIndices = null; + loop.ScientificDefinitionNameIndicesByLine?.TryGetValue( + lineNumber, + out scientificDefinitionNameIndices); + List? + sqlDefinitionLeafSpans = null; + if (language == "sql") + { + loop.SqlDefinitionLeafSpansByLine?.TryGetValue( + lineNumber, + out sqlDefinitionLeafSpans); + } + + var container = loop.ContainerResolver.Find(lineNumber); + var definitionState = new CoreLineDefinitionState( + language, + sourceContext, + preparedLine, + definitionNames, + loop.DefinitionNamesComparer, + scientificDefinitionNameIndices, + sqlDefinitionLeafSpans); + var csharpLineHasWhereClause = language == "csharp" + && preparedLine.IndexOf("where", StringComparison.Ordinal) >= 0 + && CSharpWhereClauseRegex.IsMatch(preparedLine); + + (SymbolRecord Synthetic, int NameIndex, int OpenBraceIndex, + int CloseBraceIndex)? javaSameLineCtor = null; + if (language == "java") + { + javaSameLineCtor = + JavaReferenceExtractor.TryBuildSameLineCtorSpan( + preparedLine, + lineNumber, + loop.Lookups.GetEnclosingTypeCandidates); + } + + var containerResolver = new CoreReferenceLineContainerResolver( + loop, + lineIndex, + lineNumber, + container, + csharpLineHasWhereClause, + javaSameLineCtor); + var line = new CoreReferenceLineContext( + request.FileId, + language, + input.Lines, + input.PreparedLines, + lineIndex, + preparedLine, + originalLine, + sourceContext, + lineNumber, + loop.References, + loop.Seen, + container, + definitionNames, + containerResolver.ResolveContainerForCall, + state.IsIgnoredCallName); + prepared = new CorePreparedReferenceLine( + in line, + languageLines.OriginalLineForLanguage, + csharpAttributeRanges, + csharpAttributeTopLevelRanges, + definitionState, + javaSameLineCtor, + containerResolver); + return true; + } + + private static ( + string PreparedLine, + string OriginalLineForLanguage) PrepareCoreLanguageLine( + CoreReferenceLoopContext loop, + CoreReferenceLoopState state, + int lineIndex, + string originalLine) + { + var input = loop.Preparation; + var preparedLine = input.LuaPreparedLines?[lineIndex] + ?? input.LispReferenceLines?[lineIndex] + ?? input.PreparedLines[lineIndex]; + var originalLineForLanguage = originalLine; + if (loop.Request.Language == "sass") + { + preparedLine = CssReferenceExtractor.MaskSassBlockCommentLine( + preparedLine, + state.SassPreparedCommentState!); + originalLineForLanguage = + CssReferenceExtractor.MaskSassBlockCommentLine( + originalLine, + state.SassOriginalCommentState!); + } + else if (loop.Request.Language == "stylus") + { + preparedLine = + CssReferenceExtractor.MaskSassStylusBlockCommentLine( + preparedLine, + ref state.SassStylusPreparedInBlockComment); + originalLineForLanguage = + CssReferenceExtractor.MaskSassStylusBlockCommentLine( + originalLine, + ref state.SassStylusOriginalInBlockComment); + } + + return (preparedLine, originalLineForLanguage); + } + + private static bool EmitCoreDocumentationAndSpecialLineReferences( + CoreReferenceLoopContext loop, + CoreReferenceLoopState state, + int lineIndex, + string originalLine, + string preparedLine, + bool preparedLineIsWhiteSpace, + List<(int start, int end)>? csharpAttributeRangesOnLine, + out string sourceContext) + { + var request = loop.Request; + var lineNumber = lineIndex + 1; + EmitCoreLanguageDocumentationReferences( + loop, + state, + lineIndex, + lineNumber, + originalLine, + preparedLine, + csharpAttributeRangesOnLine); + + if (preparedLineIsWhiteSpace + && request.Language + is not ("cmake" + or "justfile" + or "makefile" + or "msbuild" + or "graphql" + or "html" + or "markdown")) + { + sourceContext = string.Empty; + return false; + } + + sourceContext = originalLine.Trim(); + if (request.Language + is "cmake" or "justfile" or "makefile" or "msbuild" + && sourceContext.Length > 0) + { + BuildAutomationReferenceExtractor.EmitReferences( + request.Language, + originalLine, + sourceContext, + lineNumber, + loop.References, + loop.Seen, + request.FileId, + loop.ContainerResolver.Find(lineNumber)); + return true; + } + + if (request.Language is not ("graphql" or "html" or "markdown") + || sourceContext.Length == 0) + { + return false; + } + + MarkupSchemaReferenceExtractor.EmitReferences( + request.Language, + originalLine, + sourceContext, + lineNumber, + loop.References, + loop.Seen, + request.FileId, + loop.ContainerResolver.Find(lineNumber), + state.MarkupSchemaState); + return true; + } + +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs index c738ab4927..4fb6dfe7a5 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferenceLoop.cs @@ -4,6 +4,13 @@ namespace CodeIndex.Indexer; public static partial class ReferenceExtractor { + private enum CoreReferenceLineFlow + { + Continue, + LineConsumed, + StopExtraction, + } + private sealed class CoreReferenceLoopContext { internal required ReferenceExtractionContext Request { get; init; } @@ -46,25 +53,49 @@ private sealed class CoreReferenceLoopContext internal required ReferenceDedupeSet Seen { get; init; } } - private sealed class CoreReferenceLoopMutableState + private sealed class CoreReferenceLoopState { - internal bool CSharpInDelimitedDocComment; - internal bool JvmInDelimitedDocComment; - internal bool PhpInDocblock; - internal SymbolRecord? PhpDocblockContainer; - internal HashSet? PhpDocblockPropertyNames; - internal MarkupSchemaReferenceExtractor.MarkupState? MarkupSchemaState; - internal CssReferenceExtractor.SassLoudCommentState? SassPreparedCommentState; - internal CssReferenceExtractor.SassLoudCommentState? SassOriginalCommentState; + internal CSharpMultiLineTypePatternState PendingCSharpMultiLineTypePattern; + internal readonly CSharpWhereConstraintState? PendingCSharpWhereConstraint; + internal readonly Dictionary>? CSharpLocalNamesByFunction; + internal readonly SqlReferenceExtractor.State? SqlState; + internal bool XamlInXmlComment; + internal readonly XamlReferenceExtractor.BindingPropertyElementState? XamlBindingPropertyElementState; + internal readonly XamlReferenceExtractor.BindingMarkupExtensionState? XamlBindingMarkupExtensionState; + internal CoreDocumentationState Documentation; + internal readonly MarkupSchemaReferenceExtractor.MarkupState? MarkupSchemaState; + internal readonly CssReferenceExtractor.SassLoudCommentState? SassPreparedCommentState; + internal readonly CssReferenceExtractor.SassLoudCommentState? SassOriginalCommentState; internal bool SassStylusPreparedInBlockComment; internal bool SassStylusOriginalInBlockComment; + internal readonly ShaderReferenceExtractor.State? ShaderState; + internal readonly Func IsIgnoredCallName; - internal CoreReferenceLoopMutableState(string language, string[] lines) + internal CoreReferenceLoopState(CoreReferenceLoopContext loop) { + var language = loop.Request.Language; + var preparation = loop.Preparation; + PendingCSharpWhereConstraint = language == "csharp" + ? new CSharpWhereConstraintState() + : null; + CSharpLocalNamesByFunction = language == "csharp" + ? new Dictionary>(StringComparer.Ordinal) + : null; + SqlState = language == "sql" + ? SqlReferenceExtractor.CreateState() + : null; + XamlBindingPropertyElementState = language == "xml" + ? new XamlReferenceExtractor.BindingPropertyElementState() + : null; + XamlBindingMarkupExtensionState = language == "xml" + ? new XamlReferenceExtractor.BindingMarkupExtensionState() + : null; if (language is "graphql" or "html" or "markdown") { MarkupSchemaState = - MarkupSchemaReferenceExtractor.CreateState(language, lines); + MarkupSchemaReferenceExtractor.CreateState( + language, + preparation.Lines); } if (language == "sass") { @@ -73,710 +104,67 @@ internal CoreReferenceLoopMutableState(string language, string[] lines) SassOriginalCommentState = new CssReferenceExtractor.SassLoudCommentState(); } - } - } - - private static CSharpMultiLineTypePatternState EmitCoreReferenceLines( - CoreReferenceLoopContext loop) - { - var request = loop.Request; - var fileId = request.FileId; - var language = request.Language; - var symbols = request.Symbols; - var workspaceSymbols = request.WorkspaceSymbols; - var input = loop.Preparation; - var lines = input.Lines; - var preparedLines = input.PreparedLines; - var structuralLines = input.StructuralLines; - var referenceStructuralLines = input.ReferenceStructuralLines; - var references = loop.References; - var seen = loop.Seen; - var containerCandidates = loop.ContainerCandidates; - var containerResolver = loop.ContainerResolver; - var lookups = loop.Lookups; - var csharpAttrRanges = loop.CSharpAttributeRanges; - var csharpAttrTopLevelRanges = loop.CSharpAttributeTopLevelRanges; - var csharpUsingAliases = loop.CSharpUsingAliases; - var csharpUsingStatics = loop.CSharpUsingStatics; - var dynamicDeclarativeState = loop.DynamicDeclarativeState; - Func isIgnoredCallName = - name => IsIgnoredCallName(language, name); - var pendingCSharpMultiLineTypePattern = - default(CSharpMultiLineTypePatternState); - var pendingCSharpWhereConstraint = language == "csharp" - ? new CSharpWhereConstraintState() - : null; - var csharpLocalNamesByFunction = language == "csharp" - ? new Dictionary>(StringComparer.Ordinal) - : null; - var sqlState = language == "sql" - ? SqlReferenceExtractor.CreateState() - : null; - var xamlInXmlComment = false; - var xamlBindingPropertyElementState = language == "xml" - ? new XamlReferenceExtractor.BindingPropertyElementState() - : null; - var xamlBindingMarkupExtensionState = language == "xml" - ? new XamlReferenceExtractor.BindingMarkupExtensionState() - : null; - var mutableState = new CoreReferenceLoopMutableState(language, lines); - var shaderState = ShaderReferenceExtractor.CreateState( - language, - preparedLines, - symbols, - workspaceSymbols, - request.ReportDiagnostic); - - for (var i = 0; i < lines.Length; i++) - { - if (ReferenceLimitReached(references)) - break; - - if ((i & 0x3f) == 0) - request.CancellationToken.ThrowIfCancellationRequested(); - - var lineNumber = i + 1; - var originalLine = lines[i]; - var languageLines = PrepareCoreLanguageLine( - loop, - mutableState, - i, - originalLine); - var preparedLine = languageLines.PreparedLine; - var preparedLineIsWhiteSpace = - string.IsNullOrWhiteSpace(preparedLine); - var originalLineForLanguage = - languageLines.OriginalLineForLanguage; - var csharpAttrRangesOnLine = csharpAttrRanges?[i]; - var csharpAttrTopLevelOnLine = csharpAttrTopLevelRanges?[i]; - if (EmitCoreDocumentationAndSpecialLineReferences( - loop, - mutableState, - i, - originalLine, - preparedLine, - preparedLineIsWhiteSpace, - csharpAttrRangesOnLine, - out var sourceContext)) - { - continue; - } - - if (preparedLineIsWhiteSpace) - { - if (language == "csharp" - && (pendingCSharpMultiLineTypePattern.WaitingForHead - || pendingCSharpMultiLineTypePattern - .PendingTypeExpression != null)) - { - continue; - } - - if (language == "csharp") - { - CSharpReferenceExtractor - .FlushPendingMultiLineTypePatternReference( - ref pendingCSharpMultiLineTypePattern, - loop.CSharpQualifiedConstantPatternMemberLookup, - csharpUsingAliases, - csharpUsingStatics, - lookups.HasActiveSameFileCSharpTypeCandidate, - references, - seen, - fileId); - } - continue; - } - - if (sourceContext.Length == 0) - continue; - - var definitionNames = - loop.DefinitionNamesByLine.TryGetValue( - lineNumber, - out var namesOnLine) - ? namesOnLine - : null; - Dictionary>? - scientificDefinitionNameIndices = null; - loop.ScientificDefinitionNameIndicesByLine?.TryGetValue( - lineNumber, - out scientificDefinitionNameIndices); - List? - sqlDefinitionLeafSpans = null; - if (language == "sql") - { - loop.SqlDefinitionLeafSpansByLine?.TryGetValue( - lineNumber, - out sqlDefinitionLeafSpans); - } - var container = containerResolver.Find(lineNumber); - var definitionState = new CoreLineDefinitionState( - language, - sourceContext, - preparedLine, - definitionNames, - loop.DefinitionNamesComparer, - scientificDefinitionNameIndices, - sqlDefinitionLeafSpans); - var csharpLineHasWhereClause = language == "csharp" - && preparedLine.IndexOf( - "where", - StringComparison.Ordinal) >= 0 - && CSharpWhereClauseRegex.IsMatch(preparedLine); - - (SymbolRecord Synthetic, int NameIndex, int OpenBraceIndex, - int CloseBraceIndex)? javaSameLineCtor = null; - if (language == "java") - { - javaSameLineCtor = - JavaReferenceExtractor.TryBuildSameLineCtorSpan( - preparedLine, - lineNumber, - lookups.GetEnclosingTypeCandidates); - } - - SymbolRecord? ResolveContainerForCall(int column) - { - if (language == "csharp") - { - SymbolRecord? primaryCtorOwner = null; - foreach (var ( - rangeStart, - rangeStartColumn, - rangeEnd, - rangeEndColumn, - syntheticRecordCtor, - owner) in - lookups.GetRecordPrimaryCtorRanges()) - { - if (ReferenceEquals(container, syntheticRecordCtor) - || (container?.Kind == "function" - && container.FileId == syntheticRecordCtor.FileId - && container.StartLine == syntheticRecordCtor.StartLine - && (container.StartLine < rangeEnd - || (container.StartColumn is int containerStartColumn - && containerStartColumn < rangeEndColumn)) - && string.Equals( - container.Name, - syntheticRecordCtor.Name, - StringComparison.Ordinal))) - { - primaryCtorOwner ??= owner; - } - if (lineNumber < rangeStart || lineNumber > rangeEnd) - continue; - if (lineNumber == rangeStart - && column < rangeStartColumn) - { - return owner; - } - if (lineNumber == rangeEnd && column >= rangeEndColumn) - continue; - return syntheticRecordCtor; - } - - if (primaryCtorOwner != null) - return primaryCtorOwner; - } - - if (javaSameLineCtor != null) - { - var info = javaSameLineCtor.Value; - if (info.CloseBraceIndex >= 0 - && column > info.OpenBraceIndex - && column < info.CloseBraceIndex) - { - return info.Synthetic; - } - } - - if (language == "csharp") - { - if (csharpLineHasWhereClause) - { - var declarationRangeContainer = - FindInnermostCSharpDeclarationRangeContainer( - containerCandidates, - structuralLines[i], - lineNumber, - column); - if (declarationRangeContainer != null) - return declarationRangeContainer; - } - - var sameLineContainer = - FindInnermostSameLineCSharpContainer( - lookups - .GetCSharpSameLineContainerCandidatesByLine(), - structuralLines[i], - lineNumber, - column); - if (sameLineContainer != null) - return sameLineContainer; - - if (csharpLineHasWhereClause - && container?.Kind == "function" - && container.StartLine == lineNumber - && (!TryFindCSharpFunctionNameColumn( - structuralLines[i], - container.Name, - out var containerNameColumn) - || column < containerNameColumn)) - { - return null; - } - } - - return dynamicDeclarativeState?.ResolveContainer( - lineNumber, - column, - container) ?? container; - } - SymbolRecord? ResolveSwiftPropertyContainerForCall(int column) - { - if (loop.SwiftPropertyDefinitionsByLine != null - && loop.SwiftPropertyDefinitionsByLine.TryGetValue( - lineNumber, - out var sameLineProperties)) - { - foreach (var property in sameLineProperties) - { - if ((property.StartColumn ?? 0) <= column) - return property; - } - } - - return ResolveContainerForCall(column); - } - - var lineContext = new CoreReferenceLineContext( - fileId, + ShaderState = ShaderReferenceExtractor.CreateState( language, - lines, - preparedLines, - i, - preparedLine, - originalLine, - sourceContext, - lineNumber, - references, - seen, - container, - definitionNames, - ResolveContainerForCall, - isIgnoredCallName); - - if (shaderState is not null) - { - ShaderReferenceExtractor.EmitLineReferences( - shaderState, - preparedLine, - originalLine, - references, - seen, - fileId, - sourceContext, - lineNumber, - ResolveContainerForCall); - } - - if (ReferenceLimitReached(references)) - break; - - if (loop.IsJsxFile - && language is "javascript" or "typescript") - { - EmitJsxElementReferences(lineContext); - } - - if (ReferenceLimitReached(references)) - break; - - var typeContext = new CoreTypeReferenceContext( - lineContext, - lookups, - containerCandidates, - symbols, - structuralLines, - loop.CSharpQualifiedConstantPatternMemberLookup, - loop.CSharpQualifiedTypePatternLookup, - csharpUsingAliases, - csharpUsingStatics, - csharpLocalNamesByFunction, - pendingCSharpWhereConstraint, - loop.KotlinConstructorTypeNames, - input.TypeScriptNamespaceAliases, - loop.TypeScriptTypeAliases, - loop.SwiftTypeAliases, - ResolveSwiftPropertyContainerForCall, - input.GoImportBlockLines, - input.LuaReferenceLines, - originalLineForLanguage, - loop.AllDefinitionNames, - loop.StylusVariableDefinitionNames, - loop.XamlReferenceEnabled, - xamlBindingPropertyElementState, - xamlBindingMarkupExtensionState); - if (EmitCoreTypeReferences( - typeContext, - ref pendingCSharpMultiLineTypePattern, - ref xamlInXmlComment)) - { - continue; - } - - if (ReferenceLimitReached(references)) - break; - - EmitInfrastructureLineReferences( - lineContext, - loop.DockerfileStageNames, - loop.DockerfileVariableNames, - loop.CobolCallableSymbols); - - if (ReferenceLimitReached(references)) - break; - - var sqlSuppressedCallIndices = EmitSqlLineReferences( - lineContext, - structuralLines[i], - sqlState, - definitionState); - - if (ReferenceLimitReached(references)) - break; - - if (language is "csharp" or "java") - EmitParenlessInitializerReferences(lineContext); - - EmitPhpAndScssLineReferences(lineContext); - - if (ReferenceLimitReached(references)) - break; - - var callContext = new CoreCallReferenceContext( - lineContext, - lookups, - javaSameLineCtor, - csharpAttrRangesOnLine, - loop.KotlinConstructorTypeNames, - loop.KotlinInfixFunctionNames, - loop.ShellCallableNames, - loop.ShellGlobalAliasNames, - dynamicDeclarativeState, - referenceStructuralLines[i], - loop.ScientificNativeDependencyLimit, - request.ReportDiagnostic, - sqlSuppressedCallIndices, - loop.SqlWindowFunctionCallSiteSuppressions, - definitionState); - EmitCoreCallReferences(callContext); - - if (ReferenceLimitReached(references)) - break; - - EmitCoreMethodAndMemberReferences( - loop, - lineContext, - preparedLine, - csharpAttrRangesOnLine, - ResolveContainerForCall); - - if (ReferenceLimitReached(references)) - break; - - if (input.JsTaggedTemplatesByLine != null - && input.JsTaggedTemplatesByLine.TryGetValue( - lineNumber, - out var tagHitsOnLine)) - { - EmitJavaScriptTaggedTemplateReferences( - lineContext, - tagHitsOnLine); - } - - if (ReferenceLimitReached(references)) - break; - - EmitMetadataLineReferences( - lineContext, - csharpAttrTopLevelOnLine); - - if (ReferenceLimitReached(references)) - break; - - if (loop.IsRazorFile && language == "csharp") - { - RazorReferenceExtractor.EmitReferences( - input.RazorReferenceLines?[i] ?? originalLine, - references, - seen, - fileId, - sourceContext, - lineNumber, - ResolveContainerForCall, - definitionNames, - loop.FileDefinitionNames, - input.RazorImplementedTypeNames); - } - - if (ReferenceLimitReached(references)) - break; - - if (language == "python") - { - EmitPythonLineReferences( - lineContext, - lookups); - } - if (language == "r") - EmitRLineReferences(lineContext); - } - - return pendingCSharpMultiLineTypePattern; - } - - private static ( - string PreparedLine, - string OriginalLineForLanguage) PrepareCoreLanguageLine( - CoreReferenceLoopContext loop, - CoreReferenceLoopMutableState state, - int lineIndex, - string originalLine) - { - var input = loop.Preparation; - var preparedLine = input.LuaPreparedLines?[lineIndex] - ?? input.LispReferenceLines?[lineIndex] - ?? input.PreparedLines[lineIndex]; - var originalLineForLanguage = originalLine; - if (loop.Request.Language == "sass") - { - preparedLine = CssReferenceExtractor.MaskSassBlockCommentLine( - preparedLine, - state.SassPreparedCommentState!); - originalLineForLanguage = - CssReferenceExtractor.MaskSassBlockCommentLine( - originalLine, - state.SassOriginalCommentState!); - } - else if (loop.Request.Language == "stylus") - { - preparedLine = - CssReferenceExtractor.MaskSassStylusBlockCommentLine( - preparedLine, - ref state.SassStylusPreparedInBlockComment); - originalLineForLanguage = - CssReferenceExtractor.MaskSassStylusBlockCommentLine( - originalLine, - ref state.SassStylusOriginalInBlockComment); + preparation.PreparedLines, + loop.Request.Symbols, + loop.Request.WorkspaceSymbols, + loop.Request.ReportDiagnostic); + IsIgnoredCallName = + name => ReferenceExtractor.IsIgnoredCallName(language, name); } - - return (preparedLine, originalLineForLanguage); } - private static bool EmitCoreDocumentationAndSpecialLineReferences( - CoreReferenceLoopContext loop, - CoreReferenceLoopMutableState state, - int lineIndex, - string originalLine, - string preparedLine, - bool preparedLineIsWhiteSpace, - List<(int start, int end)>? csharpAttributeRangesOnLine, - out string sourceContext) + private readonly record struct CorePreparedReferenceLine( + string OriginalLineForLanguage, + List<(int start, int end)>? CSharpAttributeRanges, + List<(int start, int end)>? CSharpAttributeTopLevelRanges, + CoreLineDefinitionState DefinitionState, + (SymbolRecord Synthetic, int NameIndex, int OpenBraceIndex, int CloseBraceIndex)? JavaSameLineCtor, + CoreReferenceLineContainerResolver ContainerResolver) { - var request = loop.Request; - var input = loop.Preparation; - var lineNumber = lineIndex + 1; - if (request.Language is "csharp" or "java" or "kotlin" or "r" or "php") - { - Func? getPhpLineContainer = null; - if (request.Language == "php") - { - SymbolRecord? phpLineContainer = null; - var phpLineContainerResolved = false; - - SymbolRecord? GetPhpLineContainer() - { - if (!phpLineContainerResolved) - { - phpLineContainer = - loop.ContainerResolver.Find(lineNumber); - phpLineContainerResolved = true; - } - - return phpLineContainer; - } - - getPhpLineContainer = GetPhpLineContainer; - } - - var documentationLine = new CoreDocumentationLineContext( - request.FileId, - request.Language, - input.Lines, - input.PreparedLines, - input.StructuralLines, - lineIndex, - lineNumber, - originalLine, - preparedLine, - loop.References, - loop.Seen, - loop.ContainerCandidates, - loop.ContainerResolver, - loop.Lookups, - input.CSharpLinesInsideMultilineStringContent, - input.CSharpLinesInsideBlockComment, - csharpAttributeRangesOnLine, - loop.CSharpAttributeRanges, - getPhpLineContainer); - EmitCoreDocumentationReferences( - documentationLine, - ref state.CSharpInDelimitedDocComment, - ref state.JvmInDelimitedDocComment, - ref state.PhpInDocblock, - ref state.PhpDocblockContainer, - ref state.PhpDocblockPropertyNames); - } - - if (preparedLineIsWhiteSpace - && request.Language - is not ("cmake" - or "justfile" - or "makefile" - or "msbuild" - or "graphql" - or "html" - or "markdown")) - { - sourceContext = string.Empty; - return false; - } - - sourceContext = originalLine.Trim(); - if (request.Language - is "cmake" or "justfile" or "makefile" or "msbuild" - && sourceContext.Length > 0) - { - BuildAutomationReferenceExtractor.EmitReferences( - request.Language, - originalLine, - sourceContext, - lineNumber, - loop.References, - loop.Seen, - request.FileId, - loop.ContainerResolver.Find(lineNumber)); - return true; - } - - if (request.Language is not ("graphql" or "html" or "markdown") - || sourceContext.Length == 0) + public readonly CoreReferenceLineContext Line; + + public CorePreparedReferenceLine( + in CoreReferenceLineContext line, + string originalLineForLanguage, + List<(int start, int end)>? cSharpAttributeRanges, + List<(int start, int end)>? cSharpAttributeTopLevelRanges, + CoreLineDefinitionState definitionState, + (SymbolRecord Synthetic, int NameIndex, int OpenBraceIndex, int CloseBraceIndex)? javaSameLineCtor, + CoreReferenceLineContainerResolver containerResolver) + : this( + originalLineForLanguage, + cSharpAttributeRanges, + cSharpAttributeTopLevelRanges, + definitionState, + javaSameLineCtor, + containerResolver) { - return false; + Line = line; } - - MarkupSchemaReferenceExtractor.EmitReferences( - request.Language, - originalLine, - sourceContext, - lineNumber, - loop.References, - loop.Seen, - request.FileId, - loop.ContainerResolver.Find(lineNumber), - state.MarkupSchemaState); - return true; } - private static void EmitCoreMethodAndMemberReferences( - CoreReferenceLoopContext loop, - CoreReferenceLineContext line, - string preparedLine, - List<(int start, int end)>? csharpAttributeRanges, - Func resolveContainerForCall) + private static CSharpMultiLineTypePatternState EmitCoreReferenceLines( + CoreReferenceLoopContext loop) { - var language = line.Language; - if (language == "csharp") - { - EmitMethodGroupReferences( - language, - preparedLine, - loop.CallableDefinitionNames, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - resolveContainerForCall); - } - else if (language == "java") - { - JavaReferenceExtractor.EmitMethodReferenceReferences( - preparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - resolveContainerForCall); - } - else if (language == "kotlin") + var state = new CoreReferenceLoopState(loop); + for (var lineIndex = 0; + lineIndex < loop.Preparation.Lines.Length; + lineIndex++) { - KotlinReferenceExtractor.EmitMethodReferenceReferences( - preparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - resolveContainerForCall); - } - else if (language == "scala") - { - ScalaReferenceExtractor.EmitMethodReferenceReferences( - preparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - resolveContainerForCall); - } + if (ReferenceLimitReached(loop.References)) + break; - if (language == "csharp") - { - CSharpReferenceExtractor.EmitStaticMemberQualifierReferences( - preparedLine, - csharpAttributeRanges, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - resolveContainerForCall); - } + if ((lineIndex & 0x3f) == 0) + loop.Request.CancellationToken.ThrowIfCancellationRequested(); - if (language != "csharp" - || loop.CSharpQualifiedEnumMemberLookup.Count == 0) - { - return; + var flow = ProcessCoreReferenceLine(loop, state, lineIndex); + if (flow == CoreReferenceLineFlow.StopExtraction) + break; } - CSharpReferenceExtractor.EmitQualifiedEnumMemberReferences( - preparedLine, - loop.CSharpQualifiedEnumMemberLookup, - csharpAttributeRanges, - loop.CSharpUsingAliases, - loop.Lookups.GetCSharpValueReceiverNames, - loop.Lookups.GetCSharpFunctionValueReceiverNames, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - resolveContainerForCall); + return state.PendingCSharpMultiLineTypePattern; } } diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferencePhases.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferencePhases.cs new file mode 100644 index 0000000000..d10e1a60d6 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreReferencePhases.cs @@ -0,0 +1,292 @@ +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static CoreReferenceLineFlow EmitCoreOrderedReferencePhases( + CoreReferenceLoopContext loop, + CoreReferenceLoopState state, + in CorePreparedReferenceLine prepared) + { + var initialFlow = EmitCoreInitialReferencePhases( + loop, + state, + in prepared); + if (initialFlow != CoreReferenceLineFlow.Continue) + return initialFlow; + + return EmitCoreRemainingReferencePhases(loop, state, in prepared); + } + + private static CoreReferenceLineFlow EmitCoreInitialReferencePhases( + CoreReferenceLoopContext loop, + CoreReferenceLoopState state, + in CorePreparedReferenceLine prepared) + { + var request = loop.Request; + var input = loop.Preparation; + ref readonly var line = ref prepared.Line; + if (state.ShaderState is not null) + { + ShaderReferenceExtractor.EmitLineReferences( + state.ShaderState, + line.PreparedLine, + line.OriginalLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + } + + if (ReferenceLimitReached(line.References)) + return CoreReferenceLineFlow.StopExtraction; + + if (loop.IsJsxFile + && line.Language is "javascript" or "typescript") + { + EmitJsxElementReferences(in line); + } + + if (ReferenceLimitReached(line.References)) + return CoreReferenceLineFlow.StopExtraction; + + var typeContext = new CoreTypeReferenceContext( + in line, + loop.Lookups, + loop.ContainerCandidates, + request.Symbols, + input.StructuralLines, + loop.CSharpQualifiedConstantPatternMemberLookup, + loop.CSharpQualifiedTypePatternLookup, + loop.CSharpUsingAliases, + loop.CSharpUsingStatics, + state.CSharpLocalNamesByFunction, + state.PendingCSharpWhereConstraint, + loop.KotlinConstructorTypeNames, + input.TypeScriptNamespaceAliases, + loop.TypeScriptTypeAliases, + loop.SwiftTypeAliases, + prepared.ContainerResolver + .ResolveSwiftPropertyContainerForCall, + input.GoImportBlockLines, + input.LuaReferenceLines, + prepared.OriginalLineForLanguage, + loop.AllDefinitionNames, + loop.StylusVariableDefinitionNames, + loop.XamlReferenceEnabled, + state.XamlBindingPropertyElementState, + state.XamlBindingMarkupExtensionState); + if (EmitCoreTypeReferences( + in typeContext, + ref state.PendingCSharpMultiLineTypePattern, + ref state.XamlInXmlComment)) + { + return CoreReferenceLineFlow.LineConsumed; + } + + return ReferenceLimitReached(line.References) + ? CoreReferenceLineFlow.StopExtraction + : CoreReferenceLineFlow.Continue; + } + + private static CoreReferenceLineFlow EmitCoreRemainingReferencePhases( + CoreReferenceLoopContext loop, + CoreReferenceLoopState state, + in CorePreparedReferenceLine prepared) + { + var request = loop.Request; + var input = loop.Preparation; + ref readonly var line = ref prepared.Line; + EmitInfrastructureLineReferences( + in line, + loop.DockerfileStageNames, + loop.DockerfileVariableNames, + loop.CobolCallableSymbols); + + if (ReferenceLimitReached(line.References)) + return CoreReferenceLineFlow.StopExtraction; + + var sqlSuppressedCallIndices = EmitSqlLineReferences( + in line, + input.StructuralLines[line.LineIndex], + state.SqlState, + prepared.DefinitionState); + + if (ReferenceLimitReached(line.References)) + return CoreReferenceLineFlow.StopExtraction; + + if (line.Language is "csharp" or "java") + EmitParenlessInitializerReferences(in line); + + EmitScssLineReferences(in line); + EmitPhpLineReferences(in line); + + if (ReferenceLimitReached(line.References)) + return CoreReferenceLineFlow.StopExtraction; + + var callContext = new CoreCallReferenceContext( + in line, + loop.Lookups, + prepared.JavaSameLineCtor, + prepared.CSharpAttributeRanges, + loop.KotlinConstructorTypeNames, + loop.KotlinInfixFunctionNames, + loop.ShellCallableNames, + loop.ShellGlobalAliasNames, + loop.DynamicDeclarativeState, + input.ReferenceStructuralLines[line.LineIndex], + loop.ScientificNativeDependencyLimit, + request.ReportDiagnostic, + sqlSuppressedCallIndices, + loop.SqlWindowFunctionCallSiteSuppressions, + prepared.DefinitionState); + EmitCoreCallReferences(in callContext); + + if (ReferenceLimitReached(line.References)) + return CoreReferenceLineFlow.StopExtraction; + + EmitCoreMethodAndMemberReferences( + loop, + in line, + prepared.CSharpAttributeRanges, + prepared.ContainerResolver.ResolveContainerForCall); + + if (ReferenceLimitReached(line.References)) + return CoreReferenceLineFlow.StopExtraction; + + if (input.JsTaggedTemplatesByLine != null + && input.JsTaggedTemplatesByLine.TryGetValue( + line.LineNumber, + out var tagHitsOnLine)) + { + EmitJavaScriptTaggedTemplateReferences(in line, tagHitsOnLine); + } + + if (ReferenceLimitReached(line.References)) + return CoreReferenceLineFlow.StopExtraction; + + EmitMetadataLineReferences( + in line, + prepared.CSharpAttributeTopLevelRanges); + + if (ReferenceLimitReached(line.References)) + return CoreReferenceLineFlow.StopExtraction; + + if (loop.IsRazorFile && line.Language == "csharp") + { + RazorReferenceExtractor.EmitReferences( + input.RazorReferenceLines?[line.LineIndex] + ?? line.OriginalLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + line.DefinitionNames, + loop.FileDefinitionNames, + input.RazorImplementedTypeNames); + } + + if (ReferenceLimitReached(line.References)) + return CoreReferenceLineFlow.StopExtraction; + + if (line.Language == "python") + EmitPythonLineReferences(in line, loop.Lookups); + if (line.Language == "r") + EmitRLineReferences(in line); + + return CoreReferenceLineFlow.Continue; + } + + private static void EmitCoreMethodAndMemberReferences( + CoreReferenceLoopContext loop, + in CoreReferenceLineContext line, + List<(int start, int end)>? csharpAttributeRanges, + Func resolveContainerForCall) + { + var language = line.Language; + if (language == "csharp") + { + EmitMethodGroupReferences( + language, + line.PreparedLine, + loop.CallableDefinitionNames, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + resolveContainerForCall); + } + else if (language == "java") + { + JavaReferenceExtractor.EmitMethodReferenceReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + resolveContainerForCall); + } + else if (language == "kotlin") + { + KotlinReferenceExtractor.EmitMethodReferenceReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + resolveContainerForCall); + } + else if (language == "scala") + { + ScalaReferenceExtractor.EmitMethodReferenceReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + resolveContainerForCall); + } + + if (language == "csharp") + { + CSharpReferenceExtractor.EmitStaticMemberQualifierReferences( + line.PreparedLine, + csharpAttributeRanges, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + resolveContainerForCall); + } + + if (language != "csharp" + || loop.CSharpQualifiedEnumMemberLookup.Count == 0) + { + return; + } + + CSharpReferenceExtractor.EmitQualifiedEnumMemberReferences( + line.PreparedLine, + loop.CSharpQualifiedEnumMemberLookup, + csharpAttributeRanges, + loop.CSharpUsingAliases, + loop.Lookups.GetCSharpValueReceiverNames, + loop.Lookups.GetCSharpFunctionValueReceiverNames, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + resolveContainerForCall); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreSpecializedLines.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreSpecializedLines.cs index 173aecd025..68ee9bd9db 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreSpecializedLines.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreSpecializedLines.cs @@ -5,7 +5,7 @@ namespace CodeIndex.Indexer; public static partial class ReferenceExtractor { - private static void EmitJsxElementReferences(CoreReferenceLineContext line) + private static void EmitJsxElementReferences(in CoreReferenceLineContext line) { if (line.PreparedLine.IndexOf('<') < 0) return; @@ -77,7 +77,7 @@ private static void EmitJsxElementReferences(CoreReferenceLineContext line) } private static void EmitInfrastructureLineReferences( - CoreReferenceLineContext line, + in CoreReferenceLineContext line, HashSet? dockerfileStageNames, HashSet? dockerfileVariableNames, IReadOnlyList? cobolCallableSymbols) @@ -133,7 +133,7 @@ private static void EmitInfrastructureLineReferences( } private static HashSet? EmitSqlLineReferences( - CoreReferenceLineContext line, + in CoreReferenceLineContext line, string structuralLine, SqlReferenceExtractor.State? sqlState, CoreLineDefinitionState definitionState) @@ -158,7 +158,8 @@ private static void EmitInfrastructureLineReferences( callIndex)); } - private static void EmitParenlessInitializerReferences(CoreReferenceLineContext line) + private static void EmitParenlessInitializerReferences( + in CoreReferenceLineContext line) { if (line.PreparedLine.IndexOf("new", StringComparison.Ordinal) < 0) return; @@ -274,120 +275,19 @@ private static void EmitParenlessInitializerReferences(CoreReferenceLineContext } } - private static void EmitPhpAndScssLineReferences(CoreReferenceLineContext line) + private static void EmitScssLineReferences( + in CoreReferenceLineContext line) { - if (line.Language == "css") - { - CssReferenceExtractor.EmitScss( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - } - - if (line.Language == "php") - { - PhpReferenceExtractor.EmitStaticAccessReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - - PhpReferenceExtractor.EmitInstanceofReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - - PhpReferenceExtractor.EmitCatchTypeReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - - PhpReferenceExtractor.EmitReturnTypeReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - - PhpReferenceExtractor.EmitParameterTypeReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - - PhpReferenceExtractor.EmitPropertyTypeReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - - PhpReferenceExtractor.EmitInheritanceTypeReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - - PhpReferenceExtractor.EmitUseTypeReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - - PhpReferenceExtractor.EmitUseFunctionReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - - PhpReferenceExtractor.EmitUseConstReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); + if (line.Language != "css") + return; - PhpReferenceExtractor.EmitObjectMemberAccessReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - } + CssReferenceExtractor.EmitScss( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.Container); } } diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeCSharpReferences.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeCSharpReferences.cs new file mode 100644 index 0000000000..75b2aac8e8 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeCSharpReferences.cs @@ -0,0 +1,173 @@ +using System.Text.RegularExpressions; +using CodeIndex.Models; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static void EmitCoreCSharpTypeReferences( + in CoreTypeReferenceContext type, + ref CSharpMultiLineTypePatternState pendingCSharpMultiLineTypePattern) + { + ref readonly var line = ref type.Line; + if (line.Language is not "csharp") + return; + + if (line.PreparedLine.IndexOf("=>", StringComparison.Ordinal) >= 0) + { + EmitCSharpLambdaCaptureReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.Container, + type.CSharpLocalNamesByFunction); + } + + CSharpReferenceExtractor.EmitTypePositionReferences( + line.PreparedLine, + line.OriginalLine, + type.CSharpQualifiedConstantPatternMemberLookup, + type.CSharpQualifiedTypePatternLookup, + type.CSharpUsingAliases, + type.CSharpUsingStatics, + type.Lookups.HasActiveSameFileCSharpTypeCandidate, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + line.Container, + type.PendingCSharpWhereConstraint!, + ref pendingCSharpMultiLineTypePattern); + + if (CSharpReferenceExtractor.HasTrailingIsAsTypePatternIntro(line.PreparedLine, line.OriginalLine)) + { + CSharpReferenceExtractor.StartWaitingForMultiLineTypePatternHead( + ref pendingCSharpMultiLineTypePattern); + } + + if (CSharpReferenceExtractor.HasTrailingCaseTypePatternIntro(line.PreparedLine, line.OriginalLine)) + { + CSharpReferenceExtractor.StartWaitingForMultiLineTypePatternHead( + ref pendingCSharpMultiLineTypePattern); + } + + TrackCSharpLocalDeclarations( + line.PreparedLine, + line.Container, + type.CSharpLocalNamesByFunction); + } + + private static void EmitCSharpLambdaCaptureReferences( + string preparedLine, + List references, + ReferenceDedupeSet seen, + long fileId, + string context, + int lineNumber, + SymbolRecord? container, + Dictionary>? localNamesByFunction) + { + if (container?.Kind != "function" + || localNamesByFunction == null + || !localNamesByFunction.TryGetValue(GetCSharpContainerLocalScopeKey(container), out var localNames) + || localNames.Count == 0) + { + return; + } + + foreach (Match lambda in BoundedRegex.EnumerateMatches(CSharpLambdaRegex, preparedLine)) + { + var body = lambda.Groups["body"].Value; + if (string.IsNullOrWhiteSpace(body)) + continue; + + var parameterNames = CollectCSharpLambdaParameterNames(lambda.Groups["params"].Value); + foreach (var localName in localNames) + { + if (parameterNames.Contains(localName)) + continue; + if (!ContainsCSharpIdentifier(body, localName, out var bodyRelativeIndex)) + continue; + + AddReference( + references, + seen, + fileId, + localName, + lambda.Groups["body"].Index + bodyRelativeIndex, + "capture", + context, + lineNumber, + container, + "csharp"); + } + } + } + + private static HashSet CollectCSharpLambdaParameterNames(string parameterText) + { + var names = new HashSet(StringComparer.Ordinal); + foreach (Match match in BoundedRegex.EnumerateMatches(parameterText, CSharpIdentifierPattern)) + { + var name = NormalizeAtPrefixedIdentifier(match.Value); + if (!IsIgnoredCallName("csharp", name)) + names.Add(name); + } + + return names; + } + + private static bool ContainsCSharpIdentifier(string text, string name, out int index) + { + index = -1; + var normalizedName = NormalizeAtPrefixedIdentifier(name); + foreach (Match match in BoundedRegex.EnumerateMatches(text, CSharpIdentifierPattern)) + { + if (string.Equals( + NormalizeAtPrefixedIdentifier(match.Value), + normalizedName, + StringComparison.Ordinal)) + { + index = match.Index; + return true; + } + } + + return false; + } + + private static void TrackCSharpLocalDeclarations( + string preparedLine, + SymbolRecord? container, + Dictionary>? localNamesByFunction) + { + if (container?.Kind != "function" || localNamesByFunction == null) + return; + if (preparedLine.Contains("=>", StringComparison.Ordinal)) + return; + + foreach (Match match in BoundedRegex.EnumerateMatches(CSharpLocalDeclarationRegex, preparedLine)) + { + var name = NormalizeAtPrefixedIdentifier(match.Groups["name"].Value); + if (IsIgnoredCallName("csharp", name)) + continue; + + var scopeKey = GetCSharpContainerLocalScopeKey(container); + if (!localNamesByFunction.TryGetValue(scopeKey, out var localNames)) + { + localNames = new HashSet(StringComparer.Ordinal); + localNamesByFunction[scopeKey] = localNames; + } + + localNames.Add(name); + } + } + + private static string GetCSharpContainerLocalScopeKey(SymbolRecord container) + => $"{container.Kind}:{container.ContainerQualifiedName}:{container.ContainerKind}:{container.ContainerName}:{container.Name}:{container.StartLine}:{container.EndLine}:{container.BodyStartLine}:{container.BodyEndLine}:{container.StartColumn}"; +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeConsumedLanguageReferences.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeConsumedLanguageReferences.cs new file mode 100644 index 0000000000..04e82d7346 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeConsumedLanguageReferences.cs @@ -0,0 +1,71 @@ +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static bool EmitCoreConsumedLanguageTypeReferences( + in CoreTypeReferenceContext type, + ref bool xamlInXmlComment) + { + ref readonly var line = ref type.Line; + switch (line.Language) + { + case "css": + CssReferenceExtractor.EmitCss( + line.PreparedLine, + line.OriginalLine, + line.Context, + line.LineNumber, + line.References, + line.Seen, + line.FileId, + line.DefinitionNames, + line.Container); + return false; + case "sass": + CssReferenceExtractor.EmitSass( + line.PreparedLine, + type.OriginalLineForLanguage, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.Container); + return true; + case "stylus": + CssReferenceExtractor.EmitStylus( + line.PreparedLine, + type.OriginalLineForLanguage, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + type.AllDefinitionNames, + type.StylusVariableDefinitionNames, + line.Container); + return true; + case "xml": + if (type.XamlReferenceEnabled) + { + var xamlLine = XamlReferenceExtractor.StripXmlComments( + line.OriginalLine, + ref xamlInXmlComment); + XamlReferenceExtractor.Emit( + xamlLine, + line.Context, + line.LineNumber, + line.References, + line.Seen, + line.FileId, + line.Container, + type.XamlBindingPropertyElementState!, + type.XamlBindingMarkupExtensionState!); + } + + return true; + default: + return false; + } + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypePreludeReferences.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypePreludeReferences.cs new file mode 100644 index 0000000000..7c08fa15ba --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypePreludeReferences.cs @@ -0,0 +1,176 @@ +using System.Text.RegularExpressions; + +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static void EmitCoreTypePreludeReferences( + in CoreTypeReferenceContext type, + ref CSharpMultiLineTypePatternState pendingCSharpMultiLineTypePattern) + { + AdvanceCoreCSharpMultiLineTypePattern(in type, ref pendingCSharpMultiLineTypePattern); + EmitCoreCSharpEventReferences(in type); + EmitCoreConstructorChainReferences(in type); + EmitCoreCompileTimeTypeReferences(in type); + + ref readonly var line = ref type.Line; + if (line.Language is "csharp" or "java" or "kotlin") + { + EmitCatchTypeReferences( + line.Language, + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + } + } + + private static void AdvanceCoreCSharpMultiLineTypePattern( + in CoreTypeReferenceContext type, + ref CSharpMultiLineTypePatternState pendingCSharpMultiLineTypePattern) + { + ref readonly var line = ref type.Line; + if (line.Language is not "csharp") + return; + + CSharpReferenceExtractor.AdvanceMultiLineTypePatternState( + line.PreparedLine, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + type.CSharpQualifiedConstantPatternMemberLookup, + type.CSharpUsingAliases, + type.CSharpUsingStatics, + type.Lookups.HasActiveSameFileCSharpTypeCandidate, + line.References, + line.Seen, + line.FileId, + ref pendingCSharpMultiLineTypePattern); + } + + private static void EmitCoreCSharpEventReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + if (line.Language is not "csharp" + || (line.PreparedLine.IndexOf("+=", StringComparison.Ordinal) < 0 + && line.PreparedLine.IndexOf("-=", StringComparison.Ordinal) < 0)) + { + return; + } + + foreach (Match match in BoundedRegex.EnumerateMatches(EventSubscriptionRegex, line.PreparedLine)) + { + if (ReferenceLimitReached(line.References)) + break; + var eventContainer = line.ResolveContainerForCall(match.Groups["name"].Index); + AddReference( + line.References, + line.Seen, + line.FileId, + match, + "subscribe", + line.Context, + line.LineNumber, + eventContainer); + } + } + + private static void EmitCoreConstructorChainReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + switch (line.Language) + { + case "csharp": + CSharpReferenceExtractor.EmitCtorChainReferences( + line.PreparedLine, type.Lookups.GetEnclosingTypeCandidates, type.ContainerCandidates, + type.StructuralLines, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + break; + case "java": + JavaReferenceExtractor.EmitCtorChainReferences( + line.PreparedLine, type.Lookups.GetEnclosingTypeCandidates, type.Symbols, + type.StructuralLines, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + break; + case "kotlin": + KotlinReferenceExtractor.EmitCtorDelegationReferences( + line.PreparedLine, type.Lookups.GetEnclosingTypeCandidates, type.Symbols, + type.StructuralLines, line.References, line.Seen, line.FileId, line.Context, + line.LineNumber, line.Container); + break; + } + } + + private static void EmitCoreCompileTimeTypeReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + switch (line.Language) + { + case "csharp": + EmitCoreCSharpCompileTimeTypeReferences(in type); + break; + case "java": + JavaReferenceExtractor.EmitDotClassTypeLiteralReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.Container); + break; + case "kotlin": + KotlinReferenceExtractor.EmitClassLiteralReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.Container); + KotlinReferenceExtractor.EmitBacktickConstructorReferences( + line.PreparedLine, + type.KotlinConstructorTypeNames!, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + break; + } + } + + private static void EmitCoreCSharpCompileTimeTypeReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + var hasTypeKeywordIntro = line.PreparedLine.IndexOf('(') >= 0 + && (line.PreparedLine.IndexOf("nameof", StringComparison.Ordinal) >= 0 + || line.PreparedLine.IndexOf("typeof", StringComparison.Ordinal) >= 0 + || line.PreparedLine.IndexOf("sizeof", StringComparison.Ordinal) >= 0 + || line.PreparedLine.IndexOf("default", StringComparison.Ordinal) >= 0); + if (hasTypeKeywordIntro) + { + var genericParameterNames = CollectCSharpGenericParameterNamesForDeclaration(line.PreparedLine); + foreach (Match match in BoundedRegex.EnumerateMatches(CSharpTypeKeywordIntroRegex, line.PreparedLine)) + { + if (ReferenceLimitReached(line.References)) + break; + int parenIndex = match.Index + match.Length - 1; + ExtractCSharpTypeKeywordSegments( + line.References, line.Seen, line.FileId, line.PreparedLine, parenIndex + 1, + line.Context, line.LineNumber, line.Container, line.Language, genericParameterNames); + } + } + + if (line.OriginalLine.IndexOf('"') >= 0) + { + ExtractCSharpReflectionNameLiteralReferences( + line.References, line.Seen, line.FileId, line.PreparedLine, line.OriginalLine, + line.Context, line.LineNumber, line.Container); + } + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypePrimaryLanguageReferences.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypePrimaryLanguageReferences.cs new file mode 100644 index 0000000000..06c71f1ebe --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypePrimaryLanguageReferences.cs @@ -0,0 +1,143 @@ +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static void EmitCorePrimaryLanguageTypeReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + switch (line.Language) + { + case "java": + EmitCoreJavaTypeReferences(in type); + break; + case "typescript": + EmitCoreTypeScriptTypeReferences(in type); + break; + case "kotlin": + EmitCoreKotlinTypeReferences(in type); + break; + case "swift": + EmitCoreSwiftTypeReferences(in type); + break; + case "rust": + EmitCoreRustTypeReferences(in type); + break; + } + } + + private static void EmitCoreJavaTypeReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + JavaReferenceExtractor.EmitModuleDirectiveReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + + JavaReferenceExtractor.EmitTypePositionReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + line.Container); + } + + private static void EmitCoreTypeScriptTypeReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + TypeScriptReferenceExtractor.EmitTypePositionReferences( + line.PreparedLines, + line.Lines, + line.LineIndex, + line.PreparedLine, + line.Lines[line.LineIndex], + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + type.TypeScriptNamespaceAliases); + + TypeScriptReferenceExtractor.EmitDeclarationTypeReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + + TypeScriptReferenceExtractor.EmitAliasTargetReferences( + line.PreparedLine, + type.TypeScriptTypeAliases!, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + } + + private static void EmitCoreKotlinTypeReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + KotlinReferenceExtractor.EmitTypePositionReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + } + + private static void EmitCoreSwiftTypeReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + SwiftReferenceExtractor.EmitTypePositionReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + type.ResolveSwiftPropertyContainerForCall); + SwiftReferenceExtractor.EmitAliasTargetReferences( + line.PreparedLine, + type.SwiftTypeAliases!, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + } + + private static void EmitCoreRustTypeReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + var enumCandidates = type.Lookups.GetRustEnumCandidates(); + var enumContainer = enumCandidates != null + ? FindInnermostContainer(enumCandidates, line.LineNumber) + : null; + var typePositionLine = RustReferenceExtractor.MaskAttributeBodies(line.PreparedLine); + RustReferenceExtractor.EmitTypePositionReferences( + typePositionLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall, + line.Container, + enumContainer); + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeReferences.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeReferences.cs index 9694250f2b..dea06d47cb 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeReferences.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeReferences.cs @@ -1,4 +1,3 @@ -using System.Text.RegularExpressions; using CodeIndex.Models; namespace CodeIndex.Indexer; @@ -6,7 +5,6 @@ namespace CodeIndex.Indexer; public static partial class ReferenceExtractor { private readonly record struct CoreTypeReferenceContext( - CoreReferenceLineContext Line, CoreExtractionLookups Lookups, IReadOnlyList ContainerCandidates, IReadOnlyList Symbols, @@ -35,365 +33,79 @@ private readonly record struct CoreTypeReferenceContext( HashSet? StylusVariableDefinitionNames, bool XamlReferenceEnabled, XamlReferenceExtractor.BindingPropertyElementState? XamlBindingPropertyElementState, - XamlReferenceExtractor.BindingMarkupExtensionState? XamlBindingMarkupExtensionState); + XamlReferenceExtractor.BindingMarkupExtensionState? XamlBindingMarkupExtensionState) + { + public readonly CoreReferenceLineContext Line; + + public CoreTypeReferenceContext( + in CoreReferenceLineContext line, + CoreExtractionLookups lookups, + IReadOnlyList containerCandidates, + IReadOnlyList symbols, + string[] structuralLines, + IReadOnlyDictionary< + string, + List<(string ContainerName, string? QualifiedContainerName, bool AllowShortNameFallback)>> + cSharpQualifiedConstantPatternMemberLookup, + IReadOnlyDictionary< + string, + List<(string ContainerName, string? QualifiedContainerName, bool AllowShortNameFallback)>> + cSharpQualifiedTypePatternLookup, + IReadOnlyList cSharpUsingAliases, + IReadOnlyList cSharpUsingStatics, + Dictionary>? cSharpLocalNamesByFunction, + CSharpWhereConstraintState? pendingCSharpWhereConstraint, + HashSet? kotlinConstructorTypeNames, + IReadOnlyList typeScriptNamespaceAliases, + IReadOnlyList? typeScriptTypeAliases, + IReadOnlyList? swiftTypeAliases, + Func resolveSwiftPropertyContainerForCall, + bool[]? goImportBlockLines, + string[]? luaReferenceLines, + string originalLineForLanguage, + IReadOnlySet? allDefinitionNames, + HashSet? stylusVariableDefinitionNames, + bool xamlReferenceEnabled, + XamlReferenceExtractor.BindingPropertyElementState? xamlBindingPropertyElementState, + XamlReferenceExtractor.BindingMarkupExtensionState? xamlBindingMarkupExtensionState) + : this( + lookups, + containerCandidates, + symbols, + structuralLines, + cSharpQualifiedConstantPatternMemberLookup, + cSharpQualifiedTypePatternLookup, + cSharpUsingAliases, + cSharpUsingStatics, + cSharpLocalNamesByFunction, + pendingCSharpWhereConstraint, + kotlinConstructorTypeNames, + typeScriptNamespaceAliases, + typeScriptTypeAliases, + swiftTypeAliases, + resolveSwiftPropertyContainerForCall, + goImportBlockLines, + luaReferenceLines, + originalLineForLanguage, + allDefinitionNames, + stylusVariableDefinitionNames, + xamlReferenceEnabled, + xamlBindingPropertyElementState, + xamlBindingMarkupExtensionState) + { + Line = line; + } + } private static bool EmitCoreTypeReferences( - CoreTypeReferenceContext type, + in CoreTypeReferenceContext type, ref CSharpMultiLineTypePatternState pendingCSharpMultiLineTypePattern, ref bool xamlInXmlComment) { - var line = type.Line; - if (line.Language == "csharp") - { - CSharpReferenceExtractor.AdvanceMultiLineTypePatternState( - line.PreparedLine, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - type.CSharpQualifiedConstantPatternMemberLookup, - type.CSharpUsingAliases, - type.CSharpUsingStatics, - type.Lookups.HasActiveSameFileCSharpTypeCandidate, - line.References, - line.Seen, - line.FileId, - ref pendingCSharpMultiLineTypePattern); - } - - // Event subscription/unsubscription (C#) / イベント購読・解除 (C#) - if (line.Language is "csharp" - && (line.PreparedLine.IndexOf("+=", StringComparison.Ordinal) >= 0 - || line.PreparedLine.IndexOf("-=", StringComparison.Ordinal) >= 0)) - { - foreach (Match match in BoundedRegex.EnumerateMatches(EventSubscriptionRegex, line.PreparedLine)) - { - if (ReferenceLimitReached(line.References)) - break; - var eventContainer = line.ResolveContainerForCall(match.Groups["name"].Index); - AddReference(line.References, line.Seen, line.FileId, match, "subscribe", line.Context, line.LineNumber, eventContainer); - } - } - - // Constructor chain-call rewrites: C# `: this(...)` / `: base(...)`, Java `this(...)` / `super(...)`, - // and Kotlin `constructor(...) : this(...)` / `: super(...)`. - // コンストラクタ連鎖呼び出しの書き換え - if (line.Language is "csharp") - { - CSharpReferenceExtractor.EmitCtorChainReferences( - line.PreparedLine, type.Lookups.GetEnclosingTypeCandidates, type.ContainerCandidates, - type.StructuralLines, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.Container); - } - else if (line.Language is "java") - { - JavaReferenceExtractor.EmitCtorChainReferences( - line.PreparedLine, type.Lookups.GetEnclosingTypeCandidates, type.Symbols, type.StructuralLines, - line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.Container); - } - else if (line.Language is "kotlin") - { - KotlinReferenceExtractor.EmitCtorDelegationReferences( - line.PreparedLine, type.Lookups.GetEnclosingTypeCandidates, type.Symbols, type.StructuralLines, - line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.Container); - } - - // Compile-time type/member line.References that CallRegex cannot see because the - // argument has no trailing `(` of its own. See issue #253. - // 末尾の `(` を持たず CallRegex では取れないコンパイル時の型/メンバ参照。issue #253 参照。 - if (line.Language is "csharp") - { - var hasTypeKeywordIntro = line.PreparedLine.IndexOf('(') >= 0 - && (line.PreparedLine.IndexOf("nameof", StringComparison.Ordinal) >= 0 - || line.PreparedLine.IndexOf("typeof", StringComparison.Ordinal) >= 0 - || line.PreparedLine.IndexOf("sizeof", StringComparison.Ordinal) >= 0 - || line.PreparedLine.IndexOf("default", StringComparison.Ordinal) >= 0); - if (hasTypeKeywordIntro) - { - var csharpGenericParameterNames = CollectCSharpGenericParameterNamesForDeclaration(line.PreparedLine); - foreach (Match match in BoundedRegex.EnumerateMatches(CSharpTypeKeywordIntroRegex, line.PreparedLine)) - { - if (ReferenceLimitReached(line.References)) - break; - int parenIndex = match.Index + match.Length - 1; // position of '(' / '(' の位置 - ExtractCSharpTypeKeywordSegments( - line.References, line.Seen, line.FileId, line.PreparedLine, parenIndex + 1, - line.Context, line.LineNumber, line.Container, line.Language, csharpGenericParameterNames); - } - } - - if (line.OriginalLine.IndexOf('"') >= 0) - { - ExtractCSharpReflectionNameLiteralReferences( - line.References, line.Seen, line.FileId, line.PreparedLine, line.OriginalLine, line.Context, line.LineNumber, line.Container); - } - } - else if (line.Language is "java") - { - JavaReferenceExtractor.EmitDotClassTypeLiteralReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - } - else if (line.Language is "kotlin") - { - KotlinReferenceExtractor.EmitClassLiteralReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container); - KotlinReferenceExtractor.EmitBacktickConstructorReferences( - line.PreparedLine, - type.KotlinConstructorTypeNames!, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - } - - // Type-position line.References without an introducing keyword-call: base lists, - // declaration types, generic constraints, throws clauses, type tests, and - // XML-doc crefs. These are dependency edges for `line.References` / `impact`, but - // not invocation edges for default `callers` / `callees`. See issue #256. - // キーワード呼び出しの外にある型位置参照（継承リスト、宣言型、generic 制約、 - // throws、型テスト、XML doc cref）。`line.References` / `impact` では依存として扱うが、 - // 既定の `callers` / `callees` では呼び出しエッジではない。issue #256 参照。 - if (line.Language is "csharp" or "java" or "kotlin") - { - EmitCatchTypeReferences( - line.Language, - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - } - - if (line.Language == "csharp") - { - if (line.PreparedLine.IndexOf("=>", StringComparison.Ordinal) >= 0) - { - EmitCSharpLambdaCaptureReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.Container, - type.CSharpLocalNamesByFunction); - } - - CSharpReferenceExtractor.EmitTypePositionReferences( - line.PreparedLine, - line.OriginalLine, - type.CSharpQualifiedConstantPatternMemberLookup, - type.CSharpQualifiedTypePatternLookup, - type.CSharpUsingAliases, - type.CSharpUsingStatics, - type.Lookups.HasActiveSameFileCSharpTypeCandidate, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - line.Container, - type.PendingCSharpWhereConstraint!, - ref pendingCSharpMultiLineTypePattern); - - if (CSharpReferenceExtractor.HasTrailingIsAsTypePatternIntro(line.PreparedLine, line.OriginalLine)) - { - CSharpReferenceExtractor.StartWaitingForMultiLineTypePatternHead(ref pendingCSharpMultiLineTypePattern); - } - - if (CSharpReferenceExtractor.HasTrailingCaseTypePatternIntro(line.PreparedLine, line.OriginalLine)) - { - CSharpReferenceExtractor.StartWaitingForMultiLineTypePatternHead(ref pendingCSharpMultiLineTypePattern); - } - - TrackCSharpLocalDeclarations(line.PreparedLine, line.Container, type.CSharpLocalNamesByFunction); - } - else if (line.Language == "java") - { - JavaReferenceExtractor.EmitModuleDirectiveReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - - JavaReferenceExtractor.EmitTypePositionReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - line.Container); - } - else if (line.Language == "typescript") - { - TypeScriptReferenceExtractor.EmitTypePositionReferences( - line.PreparedLines, - line.Lines, - line.LineIndex, - line.PreparedLine, - line.Lines[line.LineIndex], - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - type.TypeScriptNamespaceAliases); - - TypeScriptReferenceExtractor.EmitDeclarationTypeReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - - TypeScriptReferenceExtractor.EmitAliasTargetReferences( - line.PreparedLine, - type.TypeScriptTypeAliases!, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - } - else if (line.Language == "kotlin") - { - KotlinReferenceExtractor.EmitTypePositionReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - } - else if (line.Language == "swift") - { - SwiftReferenceExtractor.EmitTypePositionReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - type.ResolveSwiftPropertyContainerForCall); - SwiftReferenceExtractor.EmitAliasTargetReferences( - line.PreparedLine, - type.SwiftTypeAliases!, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - } - else if (line.Language == "rust") - { - var rustEnumCandidatesForLine = type.Lookups.GetRustEnumCandidates(); - var rustEnumContainer = rustEnumCandidatesForLine != null - ? FindInnermostContainer(rustEnumCandidatesForLine, line.LineNumber) - : null; - var rustTypePositionLine = RustReferenceExtractor.MaskAttributeBodies(line.PreparedLine); - RustReferenceExtractor.EmitTypePositionReferences( - rustTypePositionLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall, - line.Container, - rustEnumContainer); - } - else if (line.Language == "c") - CReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.OriginalLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall); - else if (line.Language == "cpp") - CppReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.OriginalLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall); - else if (line.Language == "go") - { - GoReferenceExtractor.EmitConcurrencyReferences( - line.PreparedLine, - line.References, - line.Seen, - line.FileId, - line.Context, - line.LineNumber, - line.ResolveContainerForCall); - GoReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.OriginalLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall, type.GoImportBlockLines?[line.LineIndex] == true); - } - else if (line.Language == "dart") - DartReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall); - else if (line.Language == "vb") - VisualBasicReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall); - else if (line.Language == "fortran") - FortranReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.OriginalLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall, line.Container); - else if (line.Language == "pascal") - PascalReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall, line.Container); - else if (line.Language == "objc") - ObjectiveCReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall, line.Container); - else if (line.Language == "haskell") - HaskellReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.Container); - else if (line.Language == "elixir") - ElixirReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.Container); - else if (line.Language == "lua") - LuaReferenceExtractor.EmitTypePositionReferences(type.LuaReferenceLines?[line.LineIndex] ?? line.OriginalLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.Container); - else if (line.Language == "css") - { - CssReferenceExtractor.EmitCss( - line.PreparedLine, - line.OriginalLine, - line.Context, - line.LineNumber, - line.References, - line.Seen, - line.FileId, - line.DefinitionNames, - line.Container); - } - else if (line.Language == "sass") - { - CssReferenceExtractor.EmitSass(line.PreparedLine, type.OriginalLineForLanguage, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.Container); - return true; - } - else if (line.Language == "stylus") - { - CssReferenceExtractor.EmitStylus(line.PreparedLine, type.OriginalLineForLanguage, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, type.AllDefinitionNames, type.StylusVariableDefinitionNames, line.Container); - return true; - } - else if (line.Language == "xml" && type.XamlReferenceEnabled) - { - var xamlLine = XamlReferenceExtractor.StripXmlComments(line.OriginalLine, ref xamlInXmlComment); - XamlReferenceExtractor.Emit(xamlLine, line.Context, line.LineNumber, line.References, line.Seen, line.FileId, line.Container, type.XamlBindingPropertyElementState!, type.XamlBindingMarkupExtensionState!); - return true; - } - else if (line.Language == "xml") - { - return true; - } - return false; + EmitCoreTypePreludeReferences(in type, ref pendingCSharpMultiLineTypePattern); + EmitCoreCSharpTypeReferences(in type, ref pendingCSharpMultiLineTypePattern); + EmitCorePrimaryLanguageTypeReferences(in type); + EmitCoreSecondaryLanguageTypeReferences(in type); + return EmitCoreConsumedLanguageTypeReferences(in type, ref xamlInXmlComment); } } diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeSecondaryLanguageReferences.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeSecondaryLanguageReferences.cs new file mode 100644 index 0000000000..c75f4fa4c8 --- /dev/null +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.CoreTypeSecondaryLanguageReferences.cs @@ -0,0 +1,74 @@ +namespace CodeIndex.Indexer; + +public static partial class ReferenceExtractor +{ + private static void EmitCoreSecondaryLanguageTypeReferences(in CoreTypeReferenceContext type) + { + EmitCoreCAndGoTypeReferences(in type); + EmitCoreAdditionalLanguageTypeReferences(in type); + } + + private static void EmitCoreCAndGoTypeReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + switch (line.Language) + { + case "c": + CReferenceExtractor.EmitTypePositionReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, line.FileId, + line.Context, line.LineNumber, line.ResolveContainerForCall); + break; + case "cpp": + CppReferenceExtractor.EmitTypePositionReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, line.FileId, + line.Context, line.LineNumber, line.ResolveContainerForCall); + break; + case "go": + GoReferenceExtractor.EmitConcurrencyReferences( + line.PreparedLine, + line.References, + line.Seen, + line.FileId, + line.Context, + line.LineNumber, + line.ResolveContainerForCall); + GoReferenceExtractor.EmitTypePositionReferences( + line.PreparedLine, line.OriginalLine, line.References, line.Seen, line.FileId, + line.Context, line.LineNumber, line.ResolveContainerForCall, + type.GoImportBlockLines?[line.LineIndex] == true); + break; + } + } + + private static void EmitCoreAdditionalLanguageTypeReferences(in CoreTypeReferenceContext type) + { + ref readonly var line = ref type.Line; + switch (line.Language) + { + case "dart": + DartReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall); + break; + case "vb": + VisualBasicReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall); + break; + case "fortran": + FortranReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.OriginalLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall, line.Container); + break; + case "pascal": + PascalReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall, line.Container); + break; + case "objc": + ObjectiveCReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.ResolveContainerForCall, line.Container); + break; + case "haskell": + HaskellReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.Container); + break; + case "elixir": + ElixirReferenceExtractor.EmitTypePositionReferences(line.PreparedLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.Container); + break; + case "lua": + LuaReferenceExtractor.EmitTypePositionReferences(type.LuaReferenceLines?[line.LineIndex] ?? line.OriginalLine, line.References, line.Seen, line.FileId, line.Context, line.LineNumber, line.Container); + break; + } + } +} diff --git a/src/CodeIndex/Indexer/References/ReferenceExtractor.ReferenceRecords.cs b/src/CodeIndex/Indexer/References/ReferenceExtractor.ReferenceRecords.cs index 92fa1b3554..8cdd08dad1 100644 --- a/src/CodeIndex/Indexer/References/ReferenceExtractor.ReferenceRecords.cs +++ b/src/CodeIndex/Indexer/References/ReferenceExtractor.ReferenceRecords.cs @@ -8,208 +8,6 @@ namespace CodeIndex.Indexer; public static partial class ReferenceExtractor { - private static void EmitPhpLinePreambleReferences( - string originalLine, - List references, - ReferenceDedupeSet seen, - long fileId, - int lineNumber, - Func getLineContainer, - ref bool inDocblock, - ref SymbolRecord? docblockContainer, - ref HashSet? docblockPropertyNames) - { - if (originalLine.Contains("#[", StringComparison.Ordinal)) - { - var attributeContext = originalLine.Trim(); - if (attributeContext.Length > 0) - { - PhpReferenceExtractor.EmitAttributeReferences( - originalLine, - references, - seen, - fileId, - attributeContext, - lineNumber, - getLineContainer()); - } - } - - if (originalLine.IndexOf("/**", StringComparison.Ordinal) >= 0) - { - inDocblock = true; - docblockContainer = getLineContainer(); - docblockPropertyNames = null; - } - - var docblockContext = originalLine.Trim(); - if (docblockContext.Length > 0) - { - if (originalLine.Contains("param", StringComparison.OrdinalIgnoreCase)) - { - PhpReferenceExtractor.EmitDocblockParamTypeReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - } - - if (originalLine.Contains("return", StringComparison.OrdinalIgnoreCase)) - { - PhpReferenceExtractor.EmitDocblockReturnTypeReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - } - - if (originalLine.Contains("var", StringComparison.OrdinalIgnoreCase)) - { - PhpReferenceExtractor.EmitDocblockVarTypeReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - } - - if (originalLine.Contains("@throws", StringComparison.OrdinalIgnoreCase)) - { - PhpReferenceExtractor.EmitDocblockThrowsTypeReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - } - - if (originalLine.Contains("extends", StringComparison.OrdinalIgnoreCase)) - { - PhpReferenceExtractor.EmitDocblockExtendsTypeReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - } - - if (originalLine.Contains("implements", StringComparison.OrdinalIgnoreCase)) - { - PhpReferenceExtractor.EmitDocblockImplementsTypeReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - } - - if (originalLine.Contains("@mixin", StringComparison.OrdinalIgnoreCase)) - { - PhpReferenceExtractor.EmitDocblockMixinTypeReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - } - - if (originalLine.Contains("property", StringComparison.OrdinalIgnoreCase)) - { - PhpReferenceExtractor.EmitDocblockPropertyTypeReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer), - inDocblock, - ref docblockPropertyNames); - } - - if (originalLine.Contains("@method", StringComparison.OrdinalIgnoreCase)) - { - PhpReferenceExtractor.EmitDocblockMethodReturnTypeReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - PhpReferenceExtractor.EmitDocblockMethodParameterTypeReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - } - - if (originalLine.Contains("@template", StringComparison.OrdinalIgnoreCase)) - { - PhpReferenceExtractor.EmitDocblockTemplateBoundTypeReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - } - - if (originalLine.Contains("type", StringComparison.OrdinalIgnoreCase)) - { - PhpReferenceExtractor.EmitDocblockTypeAliasTargetReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - PhpReferenceExtractor.EmitDocblockImportTypeSourceReferences( - originalLine, - references, - seen, - fileId, - docblockContext, - lineNumber, - ResolvePhpDocblockContainer(inDocblock, docblockContainer, getLineContainer)); - } - } - - if (inDocblock && originalLine.IndexOf("*/", StringComparison.Ordinal) >= 0) - { - inDocblock = false; - docblockContainer = null; - docblockPropertyNames = null; - } - } - - private static SymbolRecord? ResolvePhpDocblockContainer( - bool inDocblock, - SymbolRecord? docblockContainer, - Func getLineContainer) - => inDocblock ? docblockContainer : getLineContainer(); - internal static void AddReference( List references, ReferenceDedupeSet seen, @@ -360,112 +158,6 @@ internal static void CompactCSharpUsingAliasReferences(List ref references.RemoveRange(writeIndex, referenceCount - writeIndex); } - private static void EmitCSharpLambdaCaptureReferences( - string preparedLine, - List references, - ReferenceDedupeSet seen, - long fileId, - string context, - int lineNumber, - SymbolRecord? container, - Dictionary>? localNamesByFunction) - { - if (container?.Kind != "function" - || localNamesByFunction == null - || !localNamesByFunction.TryGetValue(GetCSharpContainerLocalScopeKey(container), out var localNames) - || localNames.Count == 0) - { - return; - } - - foreach (Match lambda in BoundedRegex.EnumerateMatches(CSharpLambdaRegex, preparedLine)) - { - var body = lambda.Groups["body"].Value; - if (string.IsNullOrWhiteSpace(body)) - continue; - - var parameterNames = CollectCSharpLambdaParameterNames(lambda.Groups["params"].Value); - foreach (var localName in localNames) - { - if (parameterNames.Contains(localName)) - continue; - if (!ContainsCSharpIdentifier(body, localName, out var bodyRelativeIndex)) - continue; - - AddReference( - references, - seen, - fileId, - localName, - lambda.Groups["body"].Index + bodyRelativeIndex, - "capture", - context, - lineNumber, - container, - "csharp"); - } - } - } - - private static HashSet CollectCSharpLambdaParameterNames(string parameterText) - { - var names = new HashSet(StringComparer.Ordinal); - foreach (Match match in BoundedRegex.EnumerateMatches(parameterText, CSharpIdentifierPattern)) - { - var name = NormalizeAtPrefixedIdentifier(match.Value); - if (!IsIgnoredCallName("csharp", name)) - names.Add(name); - } - - return names; - } - - private static bool ContainsCSharpIdentifier(string text, string name, out int index) - { - index = -1; - var normalizedName = NormalizeAtPrefixedIdentifier(name); - foreach (Match match in BoundedRegex.EnumerateMatches(text, CSharpIdentifierPattern)) - { - if (string.Equals(NormalizeAtPrefixedIdentifier(match.Value), normalizedName, StringComparison.Ordinal)) - { - index = match.Index; - return true; - } - } - - return false; - } - - private static void TrackCSharpLocalDeclarations( - string preparedLine, - SymbolRecord? container, - Dictionary>? localNamesByFunction) - { - if (container?.Kind != "function" || localNamesByFunction == null) - return; - if (preparedLine.Contains("=>", StringComparison.Ordinal)) - return; - - foreach (Match match in BoundedRegex.EnumerateMatches(CSharpLocalDeclarationRegex, preparedLine)) - { - var name = NormalizeAtPrefixedIdentifier(match.Groups["name"].Value); - if (IsIgnoredCallName("csharp", name)) - continue; - - var scopeKey = GetCSharpContainerLocalScopeKey(container); - if (!localNamesByFunction.TryGetValue(scopeKey, out var localNames)) - { - localNames = new HashSet(StringComparer.Ordinal); - localNamesByFunction[scopeKey] = localNames; - } - - localNames.Add(name); - } - } - - private static string GetCSharpContainerLocalScopeKey(SymbolRecord container) - => $"{container.Kind}:{container.ContainerQualifiedName}:{container.ContainerKind}:{container.ContainerName}:{container.Name}:{container.StartLine}:{container.EndLine}:{container.BodyStartLine}:{container.BodyEndLine}:{container.StartColumn}"; - internal static void MarkMutualRecursionReferences(List references) { var edges = new HashSet<(string Caller, string Callee)>(); diff --git a/src/CodeIndex/Indexer/References/Support/StructuralLineMasker.JvmAndSwiftStrings.cs b/src/CodeIndex/Indexer/References/Support/StructuralLineMasker.JvmAndSwiftStrings.cs index 2564a95e97..d52d03169d 100644 --- a/src/CodeIndex/Indexer/References/Support/StructuralLineMasker.JvmAndSwiftStrings.cs +++ b/src/CodeIndex/Indexer/References/Support/StructuralLineMasker.JvmAndSwiftStrings.cs @@ -127,348 +127,6 @@ private static int SkipJsSingleLineStringContinuation(string line, int startInde return p; } - // Kotlin multi-line raw string literals: """...""". - // Body is raw (no backslash escape processing). Interpolation: $identifier and - // ${expression}. Only ${expr} hole contents are preserved so downstream reference - // extraction still sees real call edges; $ident is a bare identifier that cannot - // be a call by itself, so masking the surrounding body is safe. - // Regression target: issue #385. - // Kotlin の複数行 raw 文字列 """...""" を扱う。本文は raw（\ エスケープなし）。 - // 補間は $identifier と ${expression}。${expr} ホール内の本物の呼び出しを - // 参照抽出に残すため、ホール内は保存する。$ident は単独識別子で call にならないため - // 周囲本体と一緒にマスクしてよい。回帰対象: issue #385。 - private static void MaskKotlinTripleStringContents(string[] lines) - { - var insideTriple = false; - var blockCommentDepth = 0; - // Hole state persists across lines so multi-line ${ ... } bodies keep real - // call edges and do not accidentally close at the wrong `}`. - // -1 when outside a hole, >=0 = nested `{` depth inside the hole (0 = top). - // ホール状態は行をまたいで保持する。ホール外は -1、ホール内は `{` 深さ（0 が最上位）。 - var holeBraceDepth = -1; - // Persistent across lines: a nested `"""..."""` literal opened inside the - // current `${ ... }` hole. While true, the nested literal acts like its own - // mini triple body — `${...}` holes inside it still preserve real call - // edges (closes #996), but body chars between holes are masked through to - // the next `"""` closer so call-shaped identifiers cannot leak (closes #992). - // ホール内に開いた nested triple-quoted string の状態。nested literal 内も - // 自身の `${...}` ホールでは本物の call を残しつつ、本文は次の `"""` まで - // 空白化して phantom call の漏れを防ぐ。 - var nestedTripleOpen = false; - // -1 when not inside a nested-triple ${...} hole, >=0 = brace depth of that - // inner hole. The inner hole preserves real call edges inside the nested - // triple-quoted literal. - // nested triple 内 `${...}` ホールの brace 深さ。-1 はホール外。 - var nestedHoleBraceDepth = -1; - // Defensive depth tracking for triple-quoted literals opened 3+ levels deep - // (i.e. inside the nested triple's own `${...}` hole). >0 = current 3+ deep - // body. While >0, every char is masked and `"""` toggles depth so phantom - // calls cannot leak. Real calls 4+ levels deep are not preserved — full - // stack tracking would be needed for that — but masking soundness is. - // 3 段以上のネスト triple に対する防御的な深さ追跡。> 0 の間は本文をマスクし、 - // 4 段以上の本物の call は保持しないが、phantom の漏れは防ぐ。 - var deepNestedTripleDepth = 0; - var deepNestedTripleHashCounts = new Stack(); - - for (int i = 0; i < lines.Length; i++) - { - var line = lines[i]; - if (line.Length == 0) - continue; - - char[]? masked = null; - char[] GetMaskedLine() => masked ??= line.ToCharArray(); - var pos = 0; - - while (pos < line.Length) - { - if (blockCommentDepth > 0) - { - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '*') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - blockCommentDepth++; - pos += 2; - continue; - } - if (pos + 1 < line.Length && line[pos] == '*' && line[pos + 1] == '/') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - blockCommentDepth--; - pos += 2; - continue; - } - GetMaskedLine()[pos] = ' '; - pos++; - continue; - } - - if (insideTriple) - { - if (holeBraceDepth >= 0) - { - // Inside ${expr} hole: preserve body. Block comments and line - // comments must be recognized first so a legal `/* } */` inside - // the hole does not close the hole at the comment body's `}`. - // Nested single-line strings and char literals are also skipped - // so their `}` does not close the hole, and nested `{` / `}` - // are tracked for lambdas / object literals. - // ${expr} ホール内: 本文を保存。block / line コメントを先に - // 認識して `/* } */` のようなコメント内 `}` でホールを早閉じ - // しないようにする。単行文字列・char リテラルも同様にスキップし、 - // lambda / object literal 用のネスト `{` / `}` を追跡する。 - if (nestedTripleOpen) - { - if (nestedHoleBraceDepth >= 0) - { - // Inside the nested triple's own ${expr} hole: preserve - // body chars so real call edges land in the reference - // graph. Closes #996. - // nested triple 内の `${expr}` ホール内: 本文を保存し、 - // 本物の call が reference graph に届くようにする。 - if (deepNestedTripleDepth > 0) - { - // 3+ level deep triple body: keep masking through - // nested open/close pairs so a 4th opener cannot - // unwind the 3-deep frame early. - // 3 段以上深い triple 本文: ネスト open/close を - // 追跡し、4 段目の opener で 3 段深い frame が - // 早抜けしないようにする。 - if (pos + 2 < line.Length - && line[pos] == '"' && line[pos + 1] == '"' && line[pos + 2] == '"') - { - var looksLikeNestedOpen = LooksLikeDeepTripleOpenerContext(lines, i, pos, 3); - if (looksLikeNestedOpen) - { - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - deepNestedTripleDepth++; - deepNestedTripleHashCounts.Push(0); - continue; - } - - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - deepNestedTripleDepth--; - if (deepNestedTripleHashCounts.Count > 0) - deepNestedTripleHashCounts.Pop(); - continue; - } - GetMaskedLine()[pos] = ' '; - pos++; - continue; - } - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '/') - { - ReplaceWithSpaces(GetMaskedLine(), pos, line.Length - pos); - pos = line.Length; - continue; - } - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '*') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - blockCommentDepth = 1; - pos += 2; - continue; - } - // 3rd-level triple opener inside the inner hole. - // Detect before the single-line-string skipper so the - // leading `"` does not advance us into the literal - // body via SkipJsSingleLineString and break paren / brace - // counting. - // 3 段目の triple opener。先頭 `"` が単行スキッパーへ - // 渡って literal 本体に進まないよう先に検知する。 - if (pos + 2 < line.Length - && line[pos] == '"' && line[pos + 1] == '"' && line[pos + 2] == '"') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - deepNestedTripleDepth = 1; - deepNestedTripleHashCounts.Push(0); - continue; - } - if (line[pos] == '"' || line[pos] == '\'') - { - pos = SkipJsSingleLineString(line, pos); - continue; - } - if (line[pos] == '{') - { - nestedHoleBraceDepth++; - pos++; - continue; - } - if (line[pos] == '}') - { - if (nestedHoleBraceDepth == 0) - { - GetMaskedLine()[pos] = ' '; - nestedHoleBraceDepth = -1; - pos++; - continue; - } - nestedHoleBraceDepth--; - pos++; - continue; - } - pos++; - continue; - } - - // Inside a nested `"""..."""` literal opened earlier in this - // outer hole. Recognize a closing `"""`, an opening `${...}` - // hole inside the nested literal (so real calls inside it - // still reach the reference graph), and otherwise mask. - // 外側ホール内で開いた nested triple 本体。閉じ `"""`、内側 - // `${...}` ホール、それ以外は body としてマスク。 - if (pos + 2 < line.Length - && line[pos] == '"' && line[pos + 1] == '"' && line[pos + 2] == '"') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - nestedTripleOpen = false; - nestedHoleBraceDepth = -1; - deepNestedTripleDepth = 0; - deepNestedTripleHashCounts.Clear(); - continue; - } - if (pos + 1 < line.Length && line[pos] == '$' && line[pos + 1] == '{') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - nestedHoleBraceDepth = 0; - pos += 2; - continue; - } - GetMaskedLine()[pos] = ' '; - pos++; - continue; - } - - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '/') - { - ReplaceWithSpaces(GetMaskedLine(), pos, line.Length - pos); - pos = line.Length; - continue; - } - - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '*') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - blockCommentDepth = 1; - pos += 2; - continue; - } - - // Nested `"""..."""` literal opener inside the hole. Detect - // before the single-line-string skipper so the first `"` does - // not advance us into the literal body via `SkipJsSingleLineString`. - // ホール内で開く nested `"""..."""` の opener。先頭 `"` が単行 - // 文字列スキッパーに渡って literal 本体へ進まないよう先に検知する。 - if (pos + 2 < line.Length - && line[pos] == '"' && line[pos + 1] == '"' && line[pos + 2] == '"') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - nestedTripleOpen = true; - nestedHoleBraceDepth = -1; - continue; - } - - if (line[pos] == '"' || line[pos] == '\'') - { - pos = SkipJsSingleLineString(line, pos); - continue; - } - - if (line[pos] == '{') - { - holeBraceDepth++; - pos++; - continue; - } - - if (line[pos] == '}') - { - if (holeBraceDepth == 0) - { - GetMaskedLine()[pos] = ' '; - holeBraceDepth = -1; - pos++; - continue; - } - - holeBraceDepth--; - pos++; - continue; - } - - pos++; - continue; - } - - if (pos + 2 < line.Length - && line[pos] == '"' && line[pos + 1] == '"' && line[pos + 2] == '"') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - insideTriple = false; - // Defensive: any open nested-triple state is owned by the just- - // closed outer triple, so reset it as well. - // 防御的に、外側 triple を閉じた時点で nested-triple 状態も解除する。 - nestedTripleOpen = false; - nestedHoleBraceDepth = -1; - deepNestedTripleDepth = 0; - deepNestedTripleHashCounts.Clear(); - continue; - } - - if (pos + 1 < line.Length && line[pos] == '$' && line[pos + 1] == '{') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - holeBraceDepth = 0; - pos += 2; - continue; - } - - GetMaskedLine()[pos] = ' '; - pos++; - continue; - } - - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '/') - break; - - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '*') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - blockCommentDepth = 1; - pos += 2; - continue; - } - - if (pos + 2 < line.Length - && line[pos] == '"' && line[pos + 1] == '"' && line[pos + 2] == '"') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - insideTriple = true; - continue; - } - - if (line[pos] == '"' || line[pos] == '\'') - { - pos = SkipJsSingleLineString(line, pos); - continue; - } - - pos++; - } - - if (masked is not null) - lines[i] = new string(masked); - } - } - // Swift multi-line string literals: """...""" and extended """#"""..."""# forms. // Plain form supports \(expr) interpolation; N-hash extended form needs \#(expr) // (matching hash count). Interpolation hole contents are preserved so downstream diff --git a/src/CodeIndex/Indexer/References/Support/StructuralLineMasker.JvmTripleStrings.cs b/src/CodeIndex/Indexer/References/Support/StructuralLineMasker.JvmTripleStrings.cs new file mode 100644 index 0000000000..a3d3a69ad1 --- /dev/null +++ b/src/CodeIndex/Indexer/References/Support/StructuralLineMasker.JvmTripleStrings.cs @@ -0,0 +1,392 @@ +namespace CodeIndex.Indexer; + +internal static partial class StructuralLineMasker +{ + private enum JvmTripleStringPolicy + { + Kotlin, + Scala, + } + + private static void MaskKotlinTripleStringContents(string[] lines) + { + var scanner = new JvmTripleStringScanner( + lines, + JvmTripleStringPolicy.Kotlin); + scanner.MaskLines(); + } + + private static void MaskScalaTripleStringContents(string[] lines) + { + var scanner = new JvmTripleStringScanner( + lines, + JvmTripleStringPolicy.Scala); + scanner.MaskLines(); + } + + private struct JvmTripleStringScanner( + string[] lines, + JvmTripleStringPolicy policy) + { + private readonly Stack _deepTripleHashCounts = new(); + private bool _insideTriple; + private bool _tripleInterpolates; + private int _blockCommentDepth; + private int _holeBraceDepth = -1; + private bool _nestedTripleOpen; + private bool _nestedTripleInterpolates; + private int _nestedHoleBraceDepth = -1; + private int _deepTripleDepth; + + internal void MaskLines() + { + for (var lineIndex = 0; lineIndex < lines.Length; lineIndex++) + { + if (lines[lineIndex].Length == 0) + continue; + + ScanLine(lineIndex); + } + } + + private void ScanLine(int lineIndex) + { + var line = new JvmTripleStringLine(lines[lineIndex]); + while (line.Position < line.Text.Length) + { + if (_blockCommentDepth > 0) + { + ScanBlockComment(ref line); + continue; + } + + if (!_insideTriple) + { + ScanCode(ref line); + continue; + } + + if (_holeBraceDepth < 0) + { + ScanTripleBody(ref line); + continue; + } + + if (!_nestedTripleOpen) + { + ScanInterpolationHole(ref line); + continue; + } + + if (_nestedHoleBraceDepth < 0) + { + ScanNestedTripleBody(ref line); + continue; + } + + if (_deepTripleDepth > 0) + ScanOpaqueDeepTriple(ref line, lineIndex); + else + ScanNestedInterpolationHole(ref line); + } + + if (line.WasMasked) + lines[lineIndex] = line.CreateMaskedText(); + } + + private void ScanBlockComment(ref JvmTripleStringLine line) + { + if (StartsWith(line.Text, line.Position, "/*")) + { + line.MaskAndAdvance(2); + _blockCommentDepth++; + return; + } + + if (StartsWith(line.Text, line.Position, "*/")) + { + line.MaskAndAdvance(2); + _blockCommentDepth--; + return; + } + + line.MaskAndAdvance(); + } + + private void ScanCode(ref JvmTripleStringLine line) + { + if (StartsWith(line.Text, line.Position, "//")) + { + line.MoveToEnd(); + return; + } + + if (StartsWith(line.Text, line.Position, "/*")) + { + line.MaskAndAdvance(2); + _blockCommentDepth = 1; + return; + } + + if (IsTripleQuoteAt(line.Text, line.Position)) + { + _tripleInterpolates = + TripleAtPositionInterpolates(line.Text, line.Position); + line.MaskAndAdvance(3); + _insideTriple = true; + return; + } + + if (line.Text[line.Position] is '"' or '\'') + { + line.Position = + SkipJsSingleLineString(line.Text, line.Position); + return; + } + + line.Position++; + } + + private void ScanTripleBody(ref JvmTripleStringLine line) + { + if (IsTripleQuoteAt(line.Text, line.Position)) + { + line.MaskAndAdvance(3); + _insideTriple = false; + _tripleInterpolates = false; + ResetNestedTripleState(); + return; + } + + if (_tripleInterpolates + && StartsWith(line.Text, line.Position, "${")) + { + line.MaskAndAdvance(2); + _holeBraceDepth = 0; + return; + } + + line.MaskAndAdvance(); + } + + private void ScanInterpolationHole(ref JvmTripleStringLine line) + { + if (TryScanHoleComment(ref line)) + return; + + if (IsTripleQuoteAt(line.Text, line.Position)) + { + _nestedTripleInterpolates = + TripleAtPositionInterpolates(line.Text, line.Position); + line.MaskAndAdvance(3); + _nestedTripleOpen = true; + _nestedHoleBraceDepth = -1; + return; + } + + if (TrySkipQuotedHoleLiteral(ref line)) + return; + + if (line.Text[line.Position] == '{') + { + _holeBraceDepth++; + line.Position++; + return; + } + + if (line.Text[line.Position] == '}') + { + if (_holeBraceDepth == 0) + { + line.MaskAndAdvance(); + _holeBraceDepth = -1; + return; + } + + _holeBraceDepth--; + } + + line.Position++; + } + + private void ScanNestedTripleBody(ref JvmTripleStringLine line) + { + if (IsTripleQuoteAt(line.Text, line.Position)) + { + line.MaskAndAdvance(3); + ResetNestedTripleState(); + return; + } + + if (_nestedTripleInterpolates + && StartsWith(line.Text, line.Position, "${")) + { + line.MaskAndAdvance(2); + _nestedHoleBraceDepth = 0; + return; + } + + line.MaskAndAdvance(); + } + + private void ScanNestedInterpolationHole( + ref JvmTripleStringLine line) + { + if (TryScanHoleComment(ref line)) + return; + + if (IsTripleQuoteAt(line.Text, line.Position)) + { + line.MaskAndAdvance(3); + _deepTripleDepth = 1; + _deepTripleHashCounts.Push(0); + return; + } + + if (TrySkipQuotedHoleLiteral(ref line)) + return; + + if (line.Text[line.Position] == '{') + { + _nestedHoleBraceDepth++; + line.Position++; + return; + } + + if (line.Text[line.Position] == '}') + { + if (_nestedHoleBraceDepth == 0) + { + line.MaskAndAdvance(); + _nestedHoleBraceDepth = -1; + return; + } + + _nestedHoleBraceDepth--; + } + + line.Position++; + } + + private void ScanOpaqueDeepTriple( + ref JvmTripleStringLine line, + int lineIndex) + { + var hashCount = policy == JvmTripleStringPolicy.Scala + ? CountRun(line.Text, line.Position, '#') + : 0; + var quoteIndex = line.Position + hashCount; + if (!IsTripleQuoteAt(line.Text, quoteIndex)) + { + line.MaskAndAdvance(); + return; + } + + var delimiterLength = hashCount + 3; + if (LooksLikeDeepTripleOpenerContext( + lines, + lineIndex, + line.Position, + delimiterLength)) + { + line.MaskAndAdvance(delimiterLength); + _deepTripleDepth++; + _deepTripleHashCounts.Push(hashCount); + return; + } + + var currentHashCount = _deepTripleHashCounts.Count > 0 + ? _deepTripleHashCounts.Peek() + : 0; + if (hashCount != currentHashCount) + { + line.MaskAndAdvance(); + return; + } + + line.MaskAndAdvance(delimiterLength); + _deepTripleDepth--; + if (_deepTripleHashCounts.Count > 0) + _deepTripleHashCounts.Pop(); + } + + private bool TryScanHoleComment(ref JvmTripleStringLine line) + { + if (StartsWith(line.Text, line.Position, "//")) + { + line.MaskToEnd(); + return true; + } + + if (!StartsWith(line.Text, line.Position, "/*")) + return false; + + line.MaskAndAdvance(2); + _blockCommentDepth = 1; + return true; + } + + private static bool TrySkipQuotedHoleLiteral( + ref JvmTripleStringLine line) + { + if (line.Text[line.Position] is not ('"' or '\'')) + return false; + + line.Position = + SkipJsSingleLineString(line.Text, line.Position); + return true; + } + + private bool TripleAtPositionInterpolates( + string line, + int quoteIndex) => + policy == JvmTripleStringPolicy.Kotlin + || (quoteIndex > 0 && IsIdentifierPart(line[quoteIndex - 1])); + + private void ResetNestedTripleState() + { + _nestedTripleOpen = false; + _nestedTripleInterpolates = false; + _nestedHoleBraceDepth = -1; + _deepTripleDepth = 0; + _deepTripleHashCounts.Clear(); + } + + private static bool IsTripleQuoteAt(string line, int position) => + position + 2 < line.Length + && line[position] == '"' + && line[position + 1] == '"' + && line[position + 2] == '"'; + } + + private struct JvmTripleStringLine(string text) + { + private char[]? _masked; + + internal string Text { get; } = text; + + internal int Position { get; set; } + + internal readonly bool WasMasked => _masked is not null; + + internal void MaskAndAdvance(int length = 1) + { + _masked ??= Text.ToCharArray(); + ReplaceWithSpaces(_masked, Position, length); + Position += length; + } + + internal void MaskToEnd() + { + MaskAndAdvance(Text.Length - Position); + } + + internal void MoveToEnd() + { + Position = Text.Length; + } + + internal readonly string CreateMaskedText() => + new(_masked!); + } +} diff --git a/src/CodeIndex/Indexer/References/Support/StructuralLineMasker.ScalaTripleStrings.cs b/src/CodeIndex/Indexer/References/Support/StructuralLineMasker.ScalaTripleStrings.cs deleted file mode 100644 index c882a497d1..0000000000 --- a/src/CodeIndex/Indexer/References/Support/StructuralLineMasker.ScalaTripleStrings.cs +++ /dev/null @@ -1,371 +0,0 @@ -namespace CodeIndex.Indexer; - -internal static partial class StructuralLineMasker -{ - private static void MaskScalaTripleStringContents(string[] lines) - { - var insideTriple = false; - // Whether the currently-open triple is an interpolator form (prefixed by an - // identifier): only interpolators recognize ${expr} holes. Plain `"""..."""` - // has no interpolation. - // 現在開いている triple が interpolator 形式か。interpolator のみ ${expr} - // を補間として扱う。プレーン `"""..."""` は補間なし。 - var isInterpolator = false; - var blockCommentDepth = 0; - var holeBraceDepth = -1; - // Persistent across lines: a nested `"""..."""` literal opened inside the - // current `${ ... }` hole. While true, the nested literal acts like its own - // mini triple body — interpolator-prefixed nested triples (`s"""`, `f"""`, - // `raw"""`, ...) keep `${expr}` holes alive so real call edges still reach - // the reference graph (closes #996), while plain nested `"""..."""` masks - // everything (closes #992). - // ホール内で開いた nested triple-quoted string の状態。interpolator 付きの - // nested triple は内部の `${expr}` ホールを保存して real call を残し、 - // プレーンな nested triple は全文を masking する。 - var nestedTripleOpen = false; - // Whether the nested triple-quoted literal in the current hole was opened - // with an identifier prefix (interpolator form). - // ホール内で開いた nested triple が interpolator 形式かどうか。 - var nestedTripleIsInterpolator = false; - // -1 when not inside a nested-triple ${...} hole, >=0 = brace depth of that - // inner hole. The inner hole preserves real call edges inside the nested - // triple-quoted literal. - // nested triple 内 `${...}` ホールの brace 深さ。-1 はホール外。 - var nestedHoleBraceDepth = -1; - // Defensive depth tracking for triple-quoted literals opened 3+ levels deep - // (i.e. inside the nested triple's own `${...}` hole). >0 = current 3+ deep - // body. While >0, every char is masked and `"""` toggles depth so phantom - // calls cannot leak. Real calls 4+ levels deep are not preserved — full - // stack tracking would be needed for that — but masking soundness is. - // 3 段以上のネスト triple に対する防御的な深さ追跡。> 0 の間は本文をマスクし、 - // 4 段以上の本物の call は保持しないが、phantom の漏れは防ぐ。 - var deepNestedTripleDepth = 0; - var deepNestedTripleHashCounts = new Stack(); - - for (int i = 0; i < lines.Length; i++) - { - var line = lines[i]; - if (line.Length == 0) - continue; - - char[]? masked = null; - char[] GetMaskedLine() => masked ??= line.ToCharArray(); - var pos = 0; - - while (pos < line.Length) - { - if (blockCommentDepth > 0) - { - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '*') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - blockCommentDepth++; - pos += 2; - continue; - } - if (pos + 1 < line.Length && line[pos] == '*' && line[pos + 1] == '/') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - blockCommentDepth--; - pos += 2; - continue; - } - GetMaskedLine()[pos] = ' '; - pos++; - continue; - } - - if (insideTriple) - { - if (holeBraceDepth >= 0) - { - // Inside ${expr} hole: preserve body. Block comments and line - // comments must be recognized first so a legal `/* } */` inside - // the hole does not close the hole at the comment body's `}`. - // ${expr} ホール内: 本文を保存。block / line コメントを先に - // 認識して `/* } */` のようなコメント内 `}` でホールを早閉じ - // しないようにする。 - if (nestedTripleOpen) - { - if (nestedHoleBraceDepth >= 0) - { - // Inside the interpolator-prefixed nested triple's own - // ${expr} hole: preserve body chars so real call edges - // land in the reference graph. Closes #996. - // interpolator 付き nested triple 内の `${expr}` ホール - // 内: 本物の call を残す。 - if (deepNestedTripleDepth > 0) - { - // 3+ level deep triple body: keep masking through - // nested open/close pairs so a 4th opener cannot - // unwind the 3-deep frame early. - // 3 段以上深い triple 本文: ネスト open/close を - // 追跡し、4 段目の opener で 3 段深い frame が - // 早抜けしないようにする。 - var deepHashes = CountRun(line, pos, '#'); - if (pos + deepHashes + 2 < line.Length - && line[pos + deepHashes] == '"' - && line[pos + deepHashes + 1] == '"' - && line[pos + deepHashes + 2] == '"') - { - var looksLikeNestedOpen = LooksLikeDeepTripleOpenerContext(lines, i, pos, deepHashes + 3); - if (looksLikeNestedOpen) - { - ReplaceWithSpaces(GetMaskedLine(), pos, deepHashes + 3); - pos += deepHashes + 3; - deepNestedTripleDepth++; - deepNestedTripleHashCounts.Push(deepHashes); - continue; - } - - var currentDeepHashCount = deepNestedTripleHashCounts.Count > 0 - ? deepNestedTripleHashCounts.Peek() - : 0; - if (deepHashes == currentDeepHashCount) - { - ReplaceWithSpaces(GetMaskedLine(), pos, 3 + deepHashes); - pos += 3 + deepHashes; - deepNestedTripleDepth--; - if (deepNestedTripleHashCounts.Count > 0) - deepNestedTripleHashCounts.Pop(); - continue; - } - } - - GetMaskedLine()[pos] = ' '; - pos++; - continue; - } - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '/') - { - ReplaceWithSpaces(GetMaskedLine(), pos, line.Length - pos); - pos = line.Length; - continue; - } - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '*') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - blockCommentDepth = 1; - pos += 2; - continue; - } - // 3rd-level triple opener inside the inner hole. - // Detect before the single-line-string skipper so the - // leading `"` does not advance us into the literal - // body via SkipJsSingleLineString and break brace counting. - // 3 段目の triple opener。先頭 `"` が単行スキッパーへ - // 渡って literal 本体に進まないよう先に検知する。 - if (pos + 2 < line.Length - && line[pos] == '"' && line[pos + 1] == '"' && line[pos + 2] == '"') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - deepNestedTripleDepth = 1; - deepNestedTripleHashCounts.Push(0); - continue; - } - if (line[pos] == '"' || line[pos] == '\'') - { - pos = SkipJsSingleLineString(line, pos); - continue; - } - if (line[pos] == '{') - { - nestedHoleBraceDepth++; - pos++; - continue; - } - if (line[pos] == '}') - { - if (nestedHoleBraceDepth == 0) - { - GetMaskedLine()[pos] = ' '; - nestedHoleBraceDepth = -1; - pos++; - continue; - } - nestedHoleBraceDepth--; - pos++; - continue; - } - pos++; - continue; - } - - // Inside a nested `"""..."""` literal opened earlier in this - // outer hole. Recognize a closing `"""`; only interpolator- - // prefixed nested triples honor `${...}` holes, plain ones - // mask everything. - // 外側ホール内で開いた nested triple 本体。閉じ `"""`、 - // interpolator 付きでは `${...}` を内部ホールとして開く、 - // それ以外は body としてマスク。 - if (pos + 2 < line.Length - && line[pos] == '"' && line[pos + 1] == '"' && line[pos + 2] == '"') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - nestedTripleOpen = false; - nestedTripleIsInterpolator = false; - nestedHoleBraceDepth = -1; - deepNestedTripleDepth = 0; - deepNestedTripleHashCounts.Clear(); - continue; - } - if (nestedTripleIsInterpolator - && pos + 1 < line.Length - && line[pos] == '$' && line[pos + 1] == '{') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - nestedHoleBraceDepth = 0; - pos += 2; - continue; - } - GetMaskedLine()[pos] = ' '; - pos++; - continue; - } - - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '/') - { - ReplaceWithSpaces(GetMaskedLine(), pos, line.Length - pos); - pos = line.Length; - continue; - } - - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '*') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - blockCommentDepth = 1; - pos += 2; - continue; - } - - // Nested `"""..."""` literal opener inside the hole. Detect - // before the single-line-string skipper so the first `"` does - // not advance us into the literal body via `SkipJsSingleLineString`. - // ホール内で開く nested `"""..."""` の opener。先頭 `"` が単行 - // 文字列スキッパーに渡って literal 本体へ進まないよう先に検知する。 - if (pos + 2 < line.Length - && line[pos] == '"' && line[pos + 1] == '"' && line[pos + 2] == '"') - { - // Interpolator detection: an identifier character immediately - // before the nested `"""` marks this as a prefixed form - // (s""", f""", raw""", or user-defined). - // interpolator 判定: 直前が識別子文字なら prefix 付き。 - var nestedPrefixIsInterpolator = pos > 0 && IsIdentifierPart(line[pos - 1]); - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - nestedTripleOpen = true; - nestedTripleIsInterpolator = nestedPrefixIsInterpolator; - nestedHoleBraceDepth = -1; - continue; - } - - if (line[pos] == '"' || line[pos] == '\'') - { - pos = SkipJsSingleLineString(line, pos); - continue; - } - - if (line[pos] == '{') - { - holeBraceDepth++; - pos++; - continue; - } - - if (line[pos] == '}') - { - if (holeBraceDepth == 0) - { - GetMaskedLine()[pos] = ' '; - holeBraceDepth = -1; - pos++; - continue; - } - - holeBraceDepth--; - pos++; - continue; - } - - pos++; - continue; - } - - if (pos + 2 < line.Length - && line[pos] == '"' && line[pos + 1] == '"' && line[pos + 2] == '"') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - insideTriple = false; - isInterpolator = false; - // Defensive: outer triple owns any nested-triple state from a - // hole, so reset it as well when the outer literal closes. - // 防御的に、外側 triple が閉じた時点で nested-triple 状態も解除する。 - nestedTripleOpen = false; - nestedTripleIsInterpolator = false; - nestedHoleBraceDepth = -1; - deepNestedTripleDepth = 0; - deepNestedTripleHashCounts.Clear(); - continue; - } - - if (isInterpolator - && pos + 1 < line.Length - && line[pos] == '$' && line[pos + 1] == '{') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - holeBraceDepth = 0; - pos += 2; - continue; - } - - GetMaskedLine()[pos] = ' '; - pos++; - continue; - } - - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '/') - break; - - if (pos + 1 < line.Length && line[pos] == '/' && line[pos + 1] == '*') - { - ReplaceWithSpaces(GetMaskedLine(), pos, 2); - blockCommentDepth = 1; - pos += 2; - continue; - } - - if (pos + 2 < line.Length - && line[pos] == '"' && line[pos + 1] == '"' && line[pos + 2] == '"') - { - // Interpolator detection: an identifier character immediately before - // `"""` marks this as a prefixed form (s""", f""", raw""", or a - // user-defined interpolator). Only those forms honor ${expr} holes. - // interpolator 判定: `"""` の直前が識別子文字なら prefix 付き - // （`s"""` / `f"""` / `raw"""` / ユーザー定義）で、${expr} ホールを有効化する。 - var prefixIsInterpolator = pos > 0 && IsIdentifierPart(line[pos - 1]); - ReplaceWithSpaces(GetMaskedLine(), pos, 3); - pos += 3; - insideTriple = true; - isInterpolator = prefixIsInterpolator; - continue; - } - - if (line[pos] == '"' || line[pos] == '\'') - { - pos = SkipJsSingleLineString(line, pos); - continue; - } - - pos++; - } - - if (masked is not null) - lines[i] = new string(masked); - } - } - - -} diff --git a/src/CodeIndex/Indexer/Scanning/FileContentLoader.Decoding.cs b/src/CodeIndex/Indexer/Scanning/FileContentLoader.Decoding.cs index 197136321b..de3afcca82 100644 --- a/src/CodeIndex/Indexer/Scanning/FileContentLoader.Decoding.cs +++ b/src/CodeIndex/Indexer/Scanning/FileContentLoader.Decoding.cs @@ -11,12 +11,38 @@ internal sealed partial class FileContentLoader private static readonly UnicodeEncoding Utf16BeBomEncoding = new(bigEndian: true, byteOrderMark: true, throwOnInvalidBytes: false); private static readonly UnicodeEncoding Utf16BeNoBomEncoding = new(bigEndian: true, byteOrderMark: false, throwOnInvalidBytes: false); - private (string Content, string? Warning, FileContentInspection Inspection, bool HadInvalidUtf8Replacement) DecodeIndexableContent( + private readonly record struct DecodedFileContent( + string Content, + string? Warning, + FileContentInspection Inspection, + bool HadInvalidUtf8Replacement); + + private readonly record struct ContentDecodePreparation( + bool IsUtf16Encoded, + bool Utf16BigEndian, + bool HasUtf16Bom, + bool RawByteContentInspected, + FileContentInspection Inspection); + + private DecodedFileContent DecodeIndexableContent( byte[] bytes, string relativePath, bool inspectRawByteContent = true) { - var isUtf16Encoded = TryDetectUtf16Encoding(bytes, allowHeuristic: true, out var utf16BigEndian, out var hasUtf16Bom); + var preparation = PrepareContentDecode(bytes, inspectRawByteContent); + ThrowIfBinaryContent(bytes, relativePath, preparation); + return DecodePreparedContent(bytes, relativePath, preparation); + } + + private static ContentDecodePreparation PrepareContentDecode( + byte[] bytes, + bool inspectRawByteContent) + { + var isUtf16Encoded = TryDetectUtf16Encoding( + bytes, + allowHeuristic: true, + out var utf16BigEndian, + out var hasUtf16Bom); var rawByteContentInspected = !isUtf16Encoded && inspectRawByteContent; var rawByteContent = rawByteContentInspected ? RawByteContentInspection.Inspect(bytes) @@ -28,36 +54,70 @@ internal sealed partial class FileContentLoader HasUtf16Bom: hasUtf16Bom, RawByteContent: rawByteContent); - if (!isUtf16Encoded && TryFindIndexBlockingNullByte( - bytes, - rawByteContent, + return new ContentDecodePreparation( + isUtf16Encoded, + utf16BigEndian, + hasUtf16Bom, rawByteContentInspected, + inspection); + } + + private static void ThrowIfBinaryContent( + byte[] bytes, + string relativePath, + ContentDecodePreparation preparation) + { + if (preparation.IsUtf16Encoded + || !TryFindIndexBlockingNullByte( + bytes, + preparation.Inspection.RawByteContent, + preparation.RawByteContentInspected, out var nullByteOffset)) - throw new FileIndexer.BinaryFileSkippedException( - relativePath, - nullByteOffset, - $"{relativePath}: binary file skipped because it contains NULL byte at byte offset {nullByteOffset}"); + { + return; + } + + throw new FileIndexer.BinaryFileSkippedException( + relativePath, + nullByteOffset, + $"{relativePath}: binary file skipped because it contains NULL byte at byte offset {nullByteOffset}"); + } - if (isUtf16Encoded) + private static DecodedFileContent DecodePreparedContent( + byte[] bytes, + string relativePath, + ContentDecodePreparation preparation) + { + if (preparation.IsUtf16Encoded) { - var content = GetUtf16Encoding(utf16BigEndian, hasUtf16Bom).GetString(bytes); - var warning = hasUtf16Bom + var content = GetUtf16Encoding( + preparation.Utf16BigEndian, + preparation.HasUtf16Bom).GetString(bytes); + var warning = preparation.HasUtf16Bom ? null - : $"{relativePath}: decoded as {(utf16BigEndian ? "UTF-16BE" : "UTF-16LE")} without BOM by NUL-byte heuristic"; - return (content, warning, inspection, HadInvalidUtf8Replacement: false); + : $"{relativePath}: decoded as {(preparation.Utf16BigEndian ? "UTF-16BE" : "UTF-16LE")} without BOM by NUL-byte heuristic"; + return new DecodedFileContent( + content, + warning, + preparation.Inspection, + HadInvalidUtf8Replacement: false); } try { - return (StrictUtf8Encoding.GetString(bytes), null, inspection, HadInvalidUtf8Replacement: false); + return new DecodedFileContent( + StrictUtf8Encoding.GetString(bytes), + null, + preparation.Inspection, + HadInvalidUtf8Replacement: false); } catch (DecoderFallbackException) { var content = LenientUtf8Encoding.GetString(bytes); - return ( + return new DecodedFileContent( content, $"{relativePath}: contains invalid UTF-8 bytes (replaced with U+FFFD)", - inspection, + preparation.Inspection, HadInvalidUtf8Replacement: true); } } diff --git a/src/CodeIndex/Indexer/Scanning/FileContentLoader.GitLfs.cs b/src/CodeIndex/Indexer/Scanning/FileContentLoader.GitLfs.cs new file mode 100644 index 0000000000..95297a9f67 --- /dev/null +++ b/src/CodeIndex/Indexer/Scanning/FileContentLoader.GitLfs.cs @@ -0,0 +1,111 @@ +namespace CodeIndex.Indexer; + +internal sealed partial class FileContentLoader +{ + private const int GitLfsPointerMaxBytes = 1024; + private static ReadOnlySpan GitLfsPointerPrefix + => "version https://git-lfs.github.com/spec/v1"u8; + private static ReadOnlySpan GitLfsExtensionPrefix => "ext-"u8; + private static ReadOnlySpan GitLfsSha256OidPrefix => "oid sha256:"u8; + private static ReadOnlySpan GitLfsSizePrefix => "size "u8; + + internal static bool IsGitLfsPointer(byte[] rawBytes) + { + if (rawBytes.Length == 0 || rawBytes.Length >= GitLfsPointerMaxBytes) + return false; + + ReadOnlySpan remaining = rawBytes; + if (!remaining.StartsWith(GitLfsPointerPrefix)) + return false; + + if (!TryReadGitLfsLine(ref remaining, out var line) + || !line.SequenceEqual(GitLfsPointerPrefix)) + { + return false; + } + + if (!TryReadGitLfsLine(ref remaining, out line)) + return false; + while (line.StartsWith(GitLfsExtensionPrefix)) + { + if (!TryReadGitLfsLine(ref remaining, out line)) + return false; + } + + if (!IsGitLfsSha256OidLine(line)) + return false; + if (!TryReadGitLfsLine(ref remaining, out line) + || !IsGitLfsSizeLine(line)) + { + return false; + } + + return remaining.IsEmpty; + } + + private static bool TryReadGitLfsLine( + ref ReadOnlySpan remaining, + out ReadOnlySpan line) + { + if (remaining.IsEmpty) + { + line = default; + return false; + } + + var newlineIndex = remaining.IndexOfAny((byte)'\r', (byte)'\n'); + if (newlineIndex < 0) + { + line = remaining; + remaining = ReadOnlySpan.Empty; + return true; + } + + line = remaining[..newlineIndex]; + var nextIndex = newlineIndex + 1; + if (remaining[newlineIndex] == (byte)'\r' + && nextIndex < remaining.Length + && remaining[nextIndex] == (byte)'\n') + { + nextIndex++; + } + + remaining = remaining[nextIndex..]; + return true; + } + + private static bool IsGitLfsSha256OidLine(ReadOnlySpan line) + { + if (!line.StartsWith(GitLfsSha256OidPrefix)) + return false; + + var hash = line[GitLfsSha256OidPrefix.Length..]; + if (hash.Length != 64) + return false; + foreach (var value in hash) + { + if (!((value >= (byte)'0' && value <= (byte)'9') + || (value >= (byte)'a' && value <= (byte)'f'))) + { + return false; + } + } + return true; + } + + private static bool IsGitLfsSizeLine(ReadOnlySpan line) + { + if (!line.StartsWith(GitLfsSizePrefix)) + return false; + + var size = line[GitLfsSizePrefix.Length..]; + if (size.Length == 0) + return false; + foreach (var value in size) + { + if (value < (byte)'0' || value > (byte)'9') + return false; + } + return true; + } +} diff --git a/src/CodeIndex/Indexer/Scanning/FileContentLoader.Loading.cs b/src/CodeIndex/Indexer/Scanning/FileContentLoader.Loading.cs new file mode 100644 index 0000000000..65ec6f9098 --- /dev/null +++ b/src/CodeIndex/Indexer/Scanning/FileContentLoader.Loading.cs @@ -0,0 +1,42 @@ +namespace CodeIndex.Indexer; + +internal sealed partial class FileContentLoader +{ + private static LoadedFileContent BuildGitLfsPointerContent( + RawFileSnapshot rawFile) + { + return new LoadedFileContent( + string.Empty, + rawFile.Bytes, + rawFile.SizeBytes, + rawFile.ModifiedUtc, + NormalizedContentFacts.Empty, + ComputeChecksum(rawFile.Bytes), + null, + FileContentInspection.GitLfsPointer()); + } + + private static LoadedFileContent BuildLoadedFileContent( + RawFileSnapshot rawFile, + DecodedFileContent decoded, + NormalizedIndexableContent normalized) + { + var checksum = CanReuseRawBytesForNormalizedChecksum( + decoded.Content, + decoded.Warning, + decoded.Inspection, + normalized) + ? ComputeRawChecksum(rawFile.Bytes) + : ComputeChecksumFromNormalizedContent(normalized.Content); + + return new LoadedFileContent( + normalized.Content, + rawFile.Bytes, + rawFile.SizeBytes, + rawFile.ModifiedUtc, + normalized.Facts, + checksum, + decoded.Warning, + decoded.Inspection); + } +} diff --git a/src/CodeIndex/Indexer/Scanning/FileContentLoader.Normalization.cs b/src/CodeIndex/Indexer/Scanning/FileContentLoader.Normalization.cs new file mode 100644 index 0000000000..12c1292231 --- /dev/null +++ b/src/CodeIndex/Indexer/Scanning/FileContentLoader.Normalization.cs @@ -0,0 +1,494 @@ +using System.Text; + +namespace CodeIndex.Indexer; + +internal sealed partial class FileContentLoader +{ + internal readonly record struct NormalizedIndexableContent( + string Content, + NormalizedContentFacts Facts) + { + internal int LineCount => Facts.LineCount; + internal bool HasOversizeLine => Facts.HasOversizeLine; + internal int ConflictMarkerLine => Facts.ConflictMarkerLine; + } + + internal static string NormalizeLineEndings(string content) + { + var firstCarriageReturn = content.IndexOf('\r'); + if (firstCarriageReturn < 0) + return content; + + var builder = new StringBuilder(content.Length); + builder.Append(content, 0, firstCarriageReturn); + + for (var index = firstCarriageReturn; index < content.Length; index++) + { + if (content[index] != '\r') + { + builder.Append(content[index]); + continue; + } + + builder.Append('\n'); + if (index + 1 < content.Length && content[index + 1] == '\n') + index++; + } + + return builder.ToString(); + } + + internal static string StripLineLeadingInvisibles(string content) + { + if (string.IsNullOrEmpty(content)) + return content; + + var firstStripIndex = FindFirstLineLeadingInvisible(content); + if (firstStripIndex < 0) + return content; + + var builder = new StringBuilder(content.Length - 1); + if (firstStripIndex > 0) + builder.Append(content, 0, firstStripIndex); + var atLineStart = true; + for (var index = firstStripIndex + 1; index < content.Length; index++) + { + var current = content[index]; + if (IsLineLeadingInvisible(current) && atLineStart) + continue; + builder.Append(current); + atLineStart = current == '\n'; + } + return builder.ToString(); + } + + private static int FindFirstLineLeadingInvisible(string content) + { + var searchOffset = 0; + while (searchOffset < content.Length) + { + var relativeIndex = content.AsSpan(searchOffset).IndexOfAny('\uFEFF', '\u200B'); + if (relativeIndex < 0) + return -1; + + var index = searchOffset + relativeIndex; + if (index == 0 || content[index - 1] == '\n') + return index; + + searchOffset = index + 1; + } + + return -1; + } + + private static bool IsLineLeadingInvisible(char current) + => current is '\uFEFF' or '\u200B'; + + internal static NormalizedIndexableContent NormalizeForIndexing( + string content, + bool discardReplacementLinesWhenNonUtf8Likely = false) + { + if (content.Length == 0) + return new NormalizedIndexableContent(content, NormalizedContentFacts.Empty); + + var scanner = new IndexingNormalizationScanner( + content, + discardReplacementLinesWhenNonUtf8Likely); + return scanner.Normalize(); + } + + internal static string NormalizeContentForPrepass(string content) + { + if (content.Length == 0) + return content; + var firstNormalizationIndex = FindFirstPrepassNormalizationIndex(content); + if (firstNormalizationIndex < 0) + return content; + + StringBuilder? builder = null; + var atLineStart = firstNormalizationIndex == 0 || content[firstNormalizationIndex - 1] == '\n'; + + StringBuilder EnsureBuilder(int sourceIndex) + { + builder ??= new StringBuilder(content.Length).Append(content, 0, sourceIndex); + return builder; + } + + for (var index = firstNormalizationIndex; index < content.Length; index++) + { + var current = content[index]; + if (IsLineLeadingInvisible(current) && atLineStart) + { + EnsureBuilder(index); + continue; + } + + if (current == '\r') + { + EnsureBuilder(index).Append('\n'); + if (index + 1 < content.Length && content[index + 1] == '\n') + index++; + atLineStart = true; + continue; + } + + builder?.Append(current); + atLineStart = current == '\n'; + } + + return builder?.ToString() ?? content; + } + + private static int FindFirstPrepassNormalizationIndex(string content) + { + var searchOffset = 0; + while (searchOffset < content.Length) + { + var relativeIndex = content.AsSpan(searchOffset).IndexOfAny('\r', '\uFEFF', '\u200B'); + if (relativeIndex < 0) + return -1; + + var index = searchOffset + relativeIndex; + if (content[index] == '\r' || index == 0 || content[index - 1] == '\n') + return index; + + searchOffset = index + 1; + } + + return -1; + } + + private struct IndexingNormalizationScanner + { + private readonly string _content; + private readonly bool _discardReplacementLinesWhenNonUtf8Likely; + private readonly bool _trackFtsTokens; + private StringBuilder? _builder; + private int _outputLength; + private int _lineCount; + private int _currentLineLength; + private int _firstOversizeLine; + private int _conflictMarkerLine; + private int _conflictScanByteCount; + private bool _conflictScanComplete; + private int _replacementCharacterCount; + private List? _replacementCharacterLines; + private bool _retainReplacementCharacterLines; + private int _firstOversizeFtsTokenLine; + private int _ftsTokenLength; + private int _pendingChunkStartOffset; + private int _pendingFullChunkEndOffset; + private List? _additionalChunkStartOffsets; + private List? _fullChunkEndOffsets; + private bool _trackChunkSlices; + private bool _previousOutputWasLineBreak; + private bool _atLineStart; + + internal IndexingNormalizationScanner( + string content, + bool discardReplacementLinesWhenNonUtf8Likely) + { + this = default; + _content = content; + _discardReplacementLinesWhenNonUtf8Likely = + discardReplacementLinesWhenNonUtf8Likely; + _trackFtsTokens = content.Length + > CodeIndex.Database.DbReader.FtsUnicode61MaxTokenLength; + _retainReplacementCharacterLines = true; + _trackChunkSlices = true; + _atLineStart = true; + } + + internal NormalizedIndexableContent Normalize() + { + ScanContent(); + return BuildResult(); + } + + private void ScanContent() + { + for (var index = 0; index < _content.Length; index++) + { + var current = _content[index]; + if (IsLineLeadingInvisible(current) && _atLineStart) + { + EnsureBuilder(index); + continue; + } + + if (current == '\r') + { + NormalizeCarriageReturn(ref index); + continue; + } + + if (char.IsHighSurrogate(current) + && index + 1 < _content.Length + && char.IsLowSurrogate(_content[index + 1])) + { + NormalizeSurrogatePair(ref index, current); + continue; + } + + NormalizeSingleCharacter(index, current); + } + } + + private void NormalizeCarriageReturn(ref int sourceIndex) + { + EnsureBuilder(sourceIndex).Append('\n'); + BeginOutputUnit(); + TrackConflictBytes(1); + FinishOutputLineBreak(); + if (sourceIndex + 1 < _content.Length + && _content[sourceIndex + 1] == '\n') + { + sourceIndex++; + } + } + + private void NormalizeSurrogatePair(ref int sourceIndex, char highSurrogate) + { + var lowSurrogate = _content[++sourceIndex]; + _builder?.Append(highSurrogate).Append(lowSurrogate); + BeginOutputUnit(); + var rune = new Rune(highSurrogate, lowSurrogate); + TrackConflictBytes(rune.Utf8SequenceLength); + if (_trackFtsTokens) + TrackFtsRune(rune); + FinishOutputChars(2); + } + + private void NormalizeSingleCharacter(int sourceIndex, char current) + { + _builder?.Append(current); + BeginOutputUnit(); + if (char.IsSurrogate(current)) + { + TrackConflictBytes(3); + if (_trackFtsTokens) + TrackInvalidFtsRune(); + FinishOutputChars(1); + return; + } + + var utf8ByteLength = current <= '\u007F' + ? 1 + : new Rune(current).Utf8SequenceLength; + TrackConflictBytes(utf8ByteLength, current, sourceIndex); + TrackReplacementCharacter(current); + if (current == '\n') + { + FinishOutputLineBreak(); + return; + } + + if (_trackFtsTokens) + TrackFtsRune(new Rune(current)); + FinishOutputChars(1); + } + + private StringBuilder EnsureBuilder(int sourceIndex) + { + _builder ??= new StringBuilder(_content.Length) + .Append(_content, 0, sourceIndex); + return _builder; + } + + private void BeginOutputUnit() + { + if (_outputLength == 0) + { + _lineCount = 1; + } + else if (_previousOutputWasLineBreak) + { + _lineCount++; + + if (!_trackChunkSlices) + return; + + var chunkStep = ChunkSplitter.ChunkSize - ChunkSplitter.Overlap; + if (_lineCount > ChunkSplitter.ChunkSize + && (_lineCount - ChunkSplitter.ChunkSize - 1) % chunkStep == 0) + { + (_additionalChunkStartOffsets ??= []).Add(_pendingChunkStartOffset); + (_fullChunkEndOffsets ??= []).Add(_pendingFullChunkEndOffset); + } + + if (_lineCount > 1 + && (_lineCount - 1) % chunkStep == 0) + { + _pendingChunkStartOffset = _outputLength; + } + } + } + + private void TrackConflictBytes( + int utf8ByteLength, + char firstChar = '\0', + int sourceIndex = -1) + { + if (_conflictMarkerLine > 0 || _conflictScanComplete) + return; + + _conflictScanByteCount += utf8ByteLength; + if (_conflictScanByteCount > FileIndexer.ConflictMarkerScanLimitBytes) + { + _conflictScanComplete = true; + return; + } + + if (_atLineStart + && sourceIndex >= 0 + && firstChar is '<' or '>' + && FileIndexer.IsConflictMarkerLineStart(_content.AsSpan(sourceIndex))) + { + _conflictMarkerLine = _lineCount; + } + } + + private void TrackReplacementCharacter(char current) + { + if (current != '\uFFFD') + return; + + _replacementCharacterCount++; + if (_discardReplacementLinesWhenNonUtf8Likely + && FileIndexer.MeetsNonUtf8LikelyReplacementThreshold( + _replacementCharacterCount, + _content.Length)) + { + _replacementCharacterLines = null; + _retainReplacementCharacterLines = false; + return; + } + + if (!_retainReplacementCharacterLines) + return; + + if (_replacementCharacterLines is null + || _replacementCharacterLines[^1] != _lineCount) + { + (_replacementCharacterLines ??= []).Add(_lineCount); + } + } + + private void TrackFtsRune(Rune rune) + { + if (_firstOversizeFtsTokenLine > 0) + return; + + var isTokenRune = rune.Value <= '\u007F' + ? FileIndexer.IsLikelyUnicode61AsciiTokenChar((char)rune.Value) + : FileIndexer.IsLikelyUnicode61TokenRune(rune); + if (isTokenRune) + { + _ftsTokenLength++; + if (_ftsTokenLength > CodeIndex.Database.DbReader.FtsUnicode61MaxTokenLength) + _firstOversizeFtsTokenLine = _lineCount; + } + else + { + _ftsTokenLength = 0; + } + } + + private void TrackInvalidFtsRune() + { + if (_firstOversizeFtsTokenLine == 0) + _ftsTokenLength = 0; + } + + private void FinishOutputChars(int charCount) + { + _currentLineLength += charCount; + if (_firstOversizeLine == 0 + && _currentLineLength > ChunkSplitter.MaxLineLength) + { + _firstOversizeLine = _lineCount; + _trackChunkSlices = false; + _additionalChunkStartOffsets = null; + _fullChunkEndOffsets = null; + } + + _outputLength += charCount; + _previousOutputWasLineBreak = false; + _atLineStart = false; + } + + private void FinishOutputLineBreak() + { + if (_trackChunkSlices + && _lineCount >= ChunkSplitter.ChunkSize + && (_lineCount - ChunkSplitter.ChunkSize) + % (ChunkSplitter.ChunkSize - ChunkSplitter.Overlap) == 0) + { + _pendingFullChunkEndOffset = _outputLength; + } + + _outputLength++; + _currentLineLength = 0; + _ftsTokenLength = 0; + _previousOutputWasLineBreak = true; + _atLineStart = true; + } + + private NormalizedIndexableContent BuildResult() + { + var normalized = _builder?.ToString() ?? _content; + var replacementLines = _discardReplacementLinesWhenNonUtf8Likely + && FileIndexer.MeetsNonUtf8LikelyReplacementThreshold( + _replacementCharacterCount, + normalized.Length) + ? null + : _replacementCharacterLines?.ToArray(); + return new NormalizedIndexableContent( + normalized, + new NormalizedContentFacts( + _lineCount, + _firstOversizeLine, + _conflictMarkerLine, + _replacementCharacterCount, + replacementLines, + _firstOversizeFtsTokenLine, + BuildChunkSlices())); + } + + private NormalizedChunkSlice[]? BuildChunkSlices() + { + if (_outputLength == 0 + || _firstOversizeLine > 0 + || _lineCount <= ChunkSplitter.ChunkSize) + { + return null; + } + + var chunkStep = ChunkSplitter.ChunkSize - ChunkSplitter.Overlap; + var chunkCount = 1 + + (_lineCount - ChunkSplitter.ChunkSize + chunkStep - 1) / chunkStep; + var slices = new NormalizedChunkSlice[chunkCount]; + var effectiveContentLength = _previousOutputWasLineBreak + ? _outputLength - 1 + : _outputLength; + for (var chunkIndex = 0; chunkIndex < chunkCount; chunkIndex++) + { + var startOffset = chunkIndex == 0 + ? 0 + : _additionalChunkStartOffsets![chunkIndex - 1]; + var startLineIndex = chunkIndex * chunkStep; + var endLineIndex = Math.Min( + startLineIndex + ChunkSplitter.ChunkSize, + _lineCount); + var endOffset = endLineIndex < _lineCount + ? _fullChunkEndOffsets![chunkIndex] + : effectiveContentLength; + slices[chunkIndex] = new NormalizedChunkSlice( + startOffset, + endOffset - startOffset); + } + + return slices; + } + } +} diff --git a/src/CodeIndex/Indexer/Scanning/FileContentLoader.RawBytes.cs b/src/CodeIndex/Indexer/Scanning/FileContentLoader.RawBytes.cs index 876e4d1017..d2409d3017 100644 --- a/src/CodeIndex/Indexer/Scanning/FileContentLoader.RawBytes.cs +++ b/src/CodeIndex/Indexer/Scanning/FileContentLoader.RawBytes.cs @@ -8,7 +8,12 @@ internal sealed partial class FileContentLoader internal delegate bool RawByteChunkPredicate(ReadOnlySpan bytes); - private (byte[] Bytes, long SizeBytes, DateTime ModifiedUtc) ReadRawBytesWithSizeLimit( + private readonly record struct RawFileSnapshot( + byte[] Bytes, + long SizeBytes, + DateTime ModifiedUtc); + + private RawFileSnapshot ReadRawBytesWithSizeLimit( string absolutePath, string normalizedRelativePath, CancellationToken cancellationToken) @@ -37,12 +42,9 @@ internal sealed partial class FileContentLoader { modifiedBeforeRead = File.GetLastWriteTimeUtc(stream.SafeFileHandle); var initialLength = stream.Length; - if (initialLength > maxFileSizeBytes) - throw new FileIndexer.FileTooLargeSkippedException( - normalizedRelativePath, - initialLength, - maxFileSizeBytes, - BuildFileTooLargeMessage(initialLength, grewDuringRead: false)); + ThrowIfInitialLengthExceedsMaxFileSize( + normalizedRelativePath, + initialLength); (bytes, sizeBytes) = ReadStreamBytesWithKnownInitialLength( stream, @@ -56,7 +58,7 @@ internal sealed partial class FileContentLoader break; } - return (bytes, sizeBytes, modifiedUtc); + return new RawFileSnapshot(bytes, sizeBytes, modifiedUtc); } internal bool RawByteChunksMayMatch( @@ -76,12 +78,9 @@ internal bool RawByteChunksMayMatch( { modifiedBeforeRead = File.GetLastWriteTimeUtc(stream.SafeFileHandle); var initialLength = stream.Length; - if (initialLength > maxFileSizeBytes) - throw new FileIndexer.FileTooLargeSkippedException( - normalizedRelativePath, - initialLength, - maxFileSizeBytes, - BuildFileTooLargeMessage(initialLength, grewDuringRead: false)); + ThrowIfInitialLengthExceedsMaxFileSize( + normalizedRelativePath, + initialLength); matched = RawByteChunksMayMatch( stream, @@ -323,6 +322,20 @@ private void ThrowIfReadExceedsMaxFileSize(string normalizedRelativePath, long t BuildFileTooLargeMessage(total, grewDuringRead: true)); } + private void ThrowIfInitialLengthExceedsMaxFileSize( + string normalizedRelativePath, + long initialLength) + { + if (initialLength <= maxFileSizeBytes) + return; + + throw new FileIndexer.FileTooLargeSkippedException( + normalizedRelativePath, + initialLength, + maxFileSizeBytes, + BuildFileTooLargeMessage(initialLength, grewDuringRead: false)); + } + private static int GetReadLengthWithinLimit(long total, long maxBytes, int bufferLength) { var remaining = maxBytes - total; diff --git a/src/CodeIndex/Indexer/Scanning/FileContentLoader.cs b/src/CodeIndex/Indexer/Scanning/FileContentLoader.cs index 2c432e20a0..4adbbb9fe8 100644 --- a/src/CodeIndex/Indexer/Scanning/FileContentLoader.cs +++ b/src/CodeIndex/Indexer/Scanning/FileContentLoader.cs @@ -1,5 +1,3 @@ -using System.Text; - namespace CodeIndex.Indexer; internal sealed partial class FileContentLoader( @@ -17,20 +15,6 @@ internal sealed partial class FileContentLoader( bindReadToFileSystemIdentity; private readonly Action? _validateResolvedFileReadPath = validateResolvedFileReadPath; - private const int GitLfsPointerMaxBytes = 1024; - private static ReadOnlySpan GitLfsPointerPrefix => "version https://git-lfs.github.com/spec/v1"u8; - private static ReadOnlySpan GitLfsExtensionPrefix => "ext-"u8; - private static ReadOnlySpan GitLfsSha256OidPrefix => "oid sha256:"u8; - private static ReadOnlySpan GitLfsSizePrefix => "size "u8; - - internal readonly record struct NormalizedIndexableContent( - string Content, - NormalizedContentFacts Facts) - { - internal int LineCount => Facts.LineCount; - internal bool HasOversizeLine => Facts.HasOversizeLine; - internal int ConflictMarkerLine => Facts.ConflictMarkerLine; - } internal readonly record struct CSharpPrepassCandidateContent( string Content, @@ -42,41 +26,18 @@ internal LoadedFileContent Load( string relativePath, CancellationToken cancellationToken) { - var (bytes, sizeBytes, modifiedUtc) = ReadRawBytesWithSizeLimit( + var rawFile = ReadRawBytesWithSizeLimit( absolutePath, normalizedRelativePath, cancellationToken); - if (IsGitLfsPointer(bytes)) - { - var lfsInspection = FileContentInspection.GitLfsPointer(); - return new LoadedFileContent( - string.Empty, - bytes, - sizeBytes, - modifiedUtc, - NormalizedContentFacts.Empty, - ComputeChecksum(bytes), - null, - lfsInspection); - } + if (IsGitLfsPointer(rawFile.Bytes)) + return BuildGitLfsPointerContent(rawFile); - var (content, warning, inspection, hadInvalidUtf8Replacement) = DecodeIndexableContent(bytes, relativePath); + var decoded = DecodeIndexableContent(rawFile.Bytes, relativePath); var normalized = NormalizeForIndexing( - content, - discardReplacementLinesWhenNonUtf8Likely: hadInvalidUtf8Replacement); - var checksum = CanReuseRawBytesForNormalizedChecksum(content, warning, inspection, normalized) - ? ComputeRawChecksum(bytes) - : ComputeChecksumFromNormalizedContent(normalized.Content); - - return new LoadedFileContent( - normalized.Content, - bytes, - sizeBytes, - modifiedUtc, - normalized.Facts, - checksum, - warning, - inspection); + decoded.Content, + discardReplacementLinesWhenNonUtf8Likely: decoded.HadInvalidUtf8Replacement); + return BuildLoadedFileContent(rawFile, decoded, normalized); } internal static bool CanReuseRawBytesForNormalizedChecksum( @@ -96,15 +57,18 @@ internal string LoadNormalizedContentForPrepass( string relativePath, CancellationToken cancellationToken) { - var (bytes, _, _) = ReadRawBytesWithSizeLimit( + var rawFile = ReadRawBytesWithSizeLimit( absolutePath, normalizedRelativePath, cancellationToken); - if (IsGitLfsPointer(bytes)) + if (IsGitLfsPointer(rawFile.Bytes)) return string.Empty; - var (content, _, _, _) = DecodeIndexableContent(bytes, relativePath, inspectRawByteContent: false); - return NormalizeContentForPrepass(content); + var decoded = DecodeIndexableContent( + rawFile.Bytes, + relativePath, + inspectRawByteContent: false); + return NormalizeContentForPrepass(decoded.Content); } internal (CSharpPrepassCandidateContent? Content, bool RequiresRetry) @@ -128,12 +92,9 @@ internal string LoadNormalizedContentForPrepass( { modifiedBeforeRead = File.GetLastWriteTimeUtc(stream.SafeFileHandle); var initialLength = stream.Length; - if (initialLength > maxFileSizeBytes) - throw new FileIndexer.FileTooLargeSkippedException( - normalizedRelativePath, - initialLength, - maxFileSizeBytes, - BuildFileTooLargeMessage(initialLength, grewDuringRead: false)); + ThrowIfInitialLengthExceedsMaxFileSize( + normalizedRelativePath, + initialLength); var probe = CSharpStaticInterfacePrepass.CreateRawByteContractProbe(); var rawCandidate = RawByteChunksMayMatch( @@ -163,19 +124,22 @@ internal string LoadNormalizedContentForPrepass( if (retryOnMutation && (modifiedAfterRead != modifiedBeforeRead || lengthChanged || pathIdentityChanged)) + { return (null, RequiresRetry: true); + } + if (bytes is null || IsGitLfsPointer(bytes)) return (null, RequiresRetry: false); - var (content, warning, inspection, _) = DecodeIndexableContent( + var decoded = DecodeIndexableContent( bytes, relativePath, inspectRawByteContent: false); - var normalized = NormalizeContentForPrepass(content); + var normalized = NormalizeContentForPrepass(decoded.Content); var checksum = includeChecksum - ? warning is null - && !inspection.IsUtf16 - && ReferenceEquals(normalized, content) + ? decoded.Warning is null + && !decoded.Inspection.IsUtf16 + && ReferenceEquals(normalized, decoded.Content) ? ComputeRawChecksum(bytes) : ComputeChecksumFromNormalizedContent(normalized) : string.Empty; @@ -183,516 +147,6 @@ internal string LoadNormalizedContentForPrepass( new CSharpPrepassCandidateContent(normalized, checksum), RequiresRetry: false); } - - internal static string NormalizeLineEndings(string content) - { - var firstCarriageReturn = content.IndexOf('\r'); - if (firstCarriageReturn < 0) - return content; - - var builder = new StringBuilder(content.Length); - builder.Append(content, 0, firstCarriageReturn); - - for (var index = firstCarriageReturn; index < content.Length; index++) - { - if (content[index] != '\r') - { - builder.Append(content[index]); - continue; - } - - builder.Append('\n'); - if (index + 1 < content.Length && content[index + 1] == '\n') - index++; - } - - return builder.ToString(); - } - - internal static string StripLineLeadingInvisibles(string content) - { - if (string.IsNullOrEmpty(content)) - return content; - - var firstStripIndex = FindFirstLineLeadingInvisible(content); - if (firstStripIndex < 0) - return content; - - var sb = new StringBuilder(content.Length - 1); - if (firstStripIndex > 0) - sb.Append(content, 0, firstStripIndex); - var atLineStart = true; - for (var i = firstStripIndex + 1; i < content.Length; i++) - { - var c = content[i]; - if (IsLineLeadingInvisible(c) && atLineStart) - continue; - sb.Append(c); - atLineStart = c == '\n'; - } - return sb.ToString(); - } - - private static int FindFirstLineLeadingInvisible(string content) - { - var searchOffset = 0; - while (searchOffset < content.Length) - { - var relativeIndex = content.AsSpan(searchOffset).IndexOfAny('\uFEFF', '\u200B'); - if (relativeIndex < 0) - return -1; - - var index = searchOffset + relativeIndex; - if (index == 0 || content[index - 1] == '\n') - return index; - - searchOffset = index + 1; - } - - return -1; - } - - private static bool IsLineLeadingInvisible(char c) => c is '\uFEFF' or '\u200B'; - - internal static NormalizedIndexableContent NormalizeForIndexing( - string content, - bool discardReplacementLinesWhenNonUtf8Likely = false) - { - if (content.Length == 0) - return new NormalizedIndexableContent(content, NormalizedContentFacts.Empty); - - StringBuilder? builder = null; - var outputLength = 0; - var lineCount = 0; - var currentLineLength = 0; - var firstOversizeLine = 0; - var conflictMarkerLine = 0; - var conflictScanByteCount = 0; - var conflictScanComplete = false; - var replacementCharacterCount = 0; - List? replacementCharacterLines = null; - var retainReplacementCharacterLines = true; - var firstOversizeFtsTokenLine = 0; - var ftsTokenLength = 0; - var trackFtsTokens = content.Length - > CodeIndex.Database.DbReader.FtsUnicode61MaxTokenLength; - var pendingChunkStartOffset = 0; - var pendingFullChunkEndOffset = 0; - List? additionalChunkStartOffsets = null; - List? fullChunkEndOffsets = null; - var trackChunkSlices = true; - var previousOutputWasLineBreak = false; - var atLineStart = true; - - StringBuilder EnsureBuilder(int sourceIndex) - { - builder ??= new StringBuilder(content.Length).Append(content, 0, sourceIndex); - return builder; - } - - void BeginOutputUnit() - { - if (outputLength == 0) - { - lineCount = 1; - } - else if (previousOutputWasLineBreak) - { - lineCount++; - - if (!trackChunkSlices) - return; - - var chunkStep = ChunkSplitter.ChunkSize - ChunkSplitter.Overlap; - if (lineCount > ChunkSplitter.ChunkSize - && (lineCount - ChunkSplitter.ChunkSize - 1) % chunkStep == 0) - { - (additionalChunkStartOffsets ??= []).Add(pendingChunkStartOffset); - (fullChunkEndOffsets ??= []).Add(pendingFullChunkEndOffset); - } - - if (lineCount > 1 - && (lineCount - 1) % chunkStep == 0) - { - pendingChunkStartOffset = outputLength; - } - } - } - - void TrackConflictBytes(int utf8ByteLength, char firstChar = '\0', int sourceIndex = -1) - { - if (conflictMarkerLine > 0 || conflictScanComplete) - return; - - conflictScanByteCount += utf8ByteLength; - if (conflictScanByteCount > FileIndexer.ConflictMarkerScanLimitBytes) - { - conflictScanComplete = true; - return; - } - - if (atLineStart - && sourceIndex >= 0 - && firstChar is '<' or '>' - && FileIndexer.IsConflictMarkerLineStart(content.AsSpan(sourceIndex))) - { - conflictMarkerLine = lineCount; - } - } - - void TrackReplacementCharacter(char c) - { - if (c != '\uFFFD') - return; - - replacementCharacterCount++; - if (discardReplacementLinesWhenNonUtf8Likely - && FileIndexer.MeetsNonUtf8LikelyReplacementThreshold( - replacementCharacterCount, - content.Length)) - { - replacementCharacterLines = null; - retainReplacementCharacterLines = false; - return; - } - - if (!retainReplacementCharacterLines) - return; - - if (replacementCharacterLines is null || replacementCharacterLines[^1] != lineCount) - (replacementCharacterLines ??= []).Add(lineCount); - } - - void TrackFtsRune(Rune rune) - { - if (firstOversizeFtsTokenLine > 0) - return; - - var isTokenRune = rune.Value <= '\u007F' - ? FileIndexer.IsLikelyUnicode61AsciiTokenChar((char)rune.Value) - : FileIndexer.IsLikelyUnicode61TokenRune(rune); - if (isTokenRune) - { - ftsTokenLength++; - if (ftsTokenLength > CodeIndex.Database.DbReader.FtsUnicode61MaxTokenLength) - firstOversizeFtsTokenLine = lineCount; - } - else - { - ftsTokenLength = 0; - } - } - - void TrackInvalidFtsRune() - { - if (firstOversizeFtsTokenLine == 0) - ftsTokenLength = 0; - } - - void FinishOutputChars(int charCount) - { - currentLineLength += charCount; - if (firstOversizeLine == 0 - && currentLineLength > ChunkSplitter.MaxLineLength) - { - firstOversizeLine = lineCount; - trackChunkSlices = false; - additionalChunkStartOffsets = null; - fullChunkEndOffsets = null; - } - - outputLength += charCount; - previousOutputWasLineBreak = false; - atLineStart = false; - } - - void FinishOutputLineBreak() - { - if (trackChunkSlices - && lineCount >= ChunkSplitter.ChunkSize - && (lineCount - ChunkSplitter.ChunkSize) % (ChunkSplitter.ChunkSize - ChunkSplitter.Overlap) == 0) - { - pendingFullChunkEndOffset = outputLength; - } - - outputLength++; - currentLineLength = 0; - ftsTokenLength = 0; - previousOutputWasLineBreak = true; - atLineStart = true; - } - - NormalizedChunkSlice[]? BuildChunkSlices() - { - if (outputLength == 0 - || firstOversizeLine > 0 - || lineCount <= ChunkSplitter.ChunkSize) - { - return null; - } - - var chunkCount = 1 + (lineCount - ChunkSplitter.ChunkSize + (ChunkSplitter.ChunkSize - ChunkSplitter.Overlap) - 1) - / (ChunkSplitter.ChunkSize - ChunkSplitter.Overlap); - var slices = new NormalizedChunkSlice[chunkCount]; - var effectiveContentLength = previousOutputWasLineBreak - ? outputLength - 1 - : outputLength; - for (var chunkIndex = 0; chunkIndex < chunkCount; chunkIndex++) - { - var startOffset = chunkIndex == 0 - ? 0 - : additionalChunkStartOffsets![chunkIndex - 1]; - var startLineIndex = chunkIndex * (ChunkSplitter.ChunkSize - ChunkSplitter.Overlap); - var endLineIndex = Math.Min(startLineIndex + ChunkSplitter.ChunkSize, lineCount); - var endOffset = endLineIndex < lineCount - ? fullChunkEndOffsets![chunkIndex] - : effectiveContentLength; - slices[chunkIndex] = new NormalizedChunkSlice( - startOffset, - endOffset - startOffset); - } - - return slices; - } - - for (var i = 0; i < content.Length; i++) - { - var c = content[i]; - if (IsLineLeadingInvisible(c) && atLineStart) - { - EnsureBuilder(i); - continue; - } - - if (c == '\r') - { - EnsureBuilder(i).Append('\n'); - BeginOutputUnit(); - TrackConflictBytes(1); - FinishOutputLineBreak(); - if (i + 1 < content.Length && content[i + 1] == '\n') - i++; - continue; - } - - if (char.IsHighSurrogate(c) - && i + 1 < content.Length - && char.IsLowSurrogate(content[i + 1])) - { - var lowSurrogate = content[++i]; - builder?.Append(c).Append(lowSurrogate); - BeginOutputUnit(); - var rune = new Rune(c, lowSurrogate); - TrackConflictBytes(rune.Utf8SequenceLength); - if (trackFtsTokens) - TrackFtsRune(rune); - FinishOutputChars(2); - continue; - } - - builder?.Append(c); - BeginOutputUnit(); - if (char.IsSurrogate(c)) - { - TrackConflictBytes(3); - if (trackFtsTokens) - TrackInvalidFtsRune(); - FinishOutputChars(1); - continue; - } - - var utf8ByteLength = c <= '\u007F' - ? 1 - : new Rune(c).Utf8SequenceLength; - TrackConflictBytes(utf8ByteLength, c, i); - TrackReplacementCharacter(c); - if (c == '\n') - { - FinishOutputLineBreak(); - continue; - } - - if (trackFtsTokens) - TrackFtsRune(new Rune(c)); - FinishOutputChars(1); - } - - var normalized = builder?.ToString() ?? content; - var replacementLines = discardReplacementLinesWhenNonUtf8Likely - && FileIndexer.MeetsNonUtf8LikelyReplacementThreshold( - replacementCharacterCount, - normalized.Length) - ? null - : replacementCharacterLines?.ToArray(); - return new NormalizedIndexableContent( - normalized, - new NormalizedContentFacts( - lineCount, - firstOversizeLine, - conflictMarkerLine, - replacementCharacterCount, - replacementLines, - firstOversizeFtsTokenLine, - BuildChunkSlices())); - } - - internal static string NormalizeContentForPrepass(string content) - { - if (content.Length == 0) - return content; - var firstNormalizationIndex = FindFirstPrepassNormalizationIndex(content); - if (firstNormalizationIndex < 0) - return content; - - StringBuilder? builder = null; - var atLineStart = firstNormalizationIndex == 0 || content[firstNormalizationIndex - 1] == '\n'; - - StringBuilder EnsureBuilder(int sourceIndex) - { - builder ??= new StringBuilder(content.Length).Append(content, 0, sourceIndex); - return builder; - } - - for (var i = firstNormalizationIndex; i < content.Length; i++) - { - var c = content[i]; - if (IsLineLeadingInvisible(c) && atLineStart) - { - EnsureBuilder(i); - continue; - } - - if (c == '\r') - { - EnsureBuilder(i).Append('\n'); - if (i + 1 < content.Length && content[i + 1] == '\n') - i++; - atLineStart = true; - continue; - } - - builder?.Append(c); - atLineStart = c == '\n'; - } - - return builder?.ToString() ?? content; - } - - private static int FindFirstPrepassNormalizationIndex(string content) - { - var searchOffset = 0; - while (searchOffset < content.Length) - { - var relativeIndex = content.AsSpan(searchOffset).IndexOfAny('\r', '\uFEFF', '\u200B'); - if (relativeIndex < 0) - return -1; - - var index = searchOffset + relativeIndex; - if (content[index] == '\r' || index == 0 || content[index - 1] == '\n') - return index; - - searchOffset = index + 1; - } - - return -1; - } - - internal static bool IsGitLfsPointer(byte[] rawBytes) - { - if (rawBytes.Length == 0 || rawBytes.Length >= GitLfsPointerMaxBytes) - return false; - - ReadOnlySpan remaining = rawBytes; - if (!remaining.StartsWith(GitLfsPointerPrefix)) - return false; - - if (!TryReadGitLfsLine(ref remaining, out var line) - || !line.SequenceEqual(GitLfsPointerPrefix)) - return false; - - if (!TryReadGitLfsLine(ref remaining, out line)) - return false; - while (line.StartsWith(GitLfsExtensionPrefix)) - { - if (!TryReadGitLfsLine(ref remaining, out line)) - return false; - } - - if (!IsGitLfsSha256OidLine(line)) - return false; - if (!TryReadGitLfsLine(ref remaining, out line) - || !IsGitLfsSizeLine(line)) - { - return false; - } - - return remaining.IsEmpty; - } - - private static bool TryReadGitLfsLine(ref ReadOnlySpan remaining, out ReadOnlySpan line) - { - if (remaining.IsEmpty) - { - line = default; - return false; - } - - var newlineIndex = remaining.IndexOfAny((byte)'\r', (byte)'\n'); - if (newlineIndex < 0) - { - line = remaining; - remaining = ReadOnlySpan.Empty; - return true; - } - - line = remaining[..newlineIndex]; - var nextIndex = newlineIndex + 1; - if (remaining[newlineIndex] == (byte)'\r' - && nextIndex < remaining.Length - && remaining[nextIndex] == (byte)'\n') - { - nextIndex++; - } - - remaining = remaining[nextIndex..]; - return true; - } - - private static bool IsGitLfsSha256OidLine(ReadOnlySpan line) - { - if (!line.StartsWith(GitLfsSha256OidPrefix)) - return false; - - var hash = line[GitLfsSha256OidPrefix.Length..]; - if (hash.Length != 64) - return false; - foreach (var value in hash) - { - if (!((value >= (byte)'0' && value <= (byte)'9') - || (value >= (byte)'a' && value <= (byte)'f'))) - { - return false; - } - } - return true; - } - - private static bool IsGitLfsSizeLine(ReadOnlySpan line) - { - if (!line.StartsWith(GitLfsSizePrefix)) - return false; - - var size = line[GitLfsSizePrefix.Length..]; - if (size.Length == 0) - return false; - foreach (var value in size) - { - if (value < (byte)'0' || value > (byte)'9') - return false; - } - return true; - } - } internal readonly record struct LoadedFileContent( diff --git a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractCore.cs b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractCore.cs index b9bb4fae95..d5949483d6 100644 --- a/src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractCore.cs +++ b/src/CodeIndex/Indexer/Symbols/SymbolExtractor.ExtractCore.cs @@ -56,1330 +56,25 @@ private static List ExtractCore( return specializedSymbols; } - // Normalize CRLF / CR to LF first so direct callers that bypass FileIndexer - // still present a `\n`-only content stream, and then strip line-leading - // UTF-8 BOM (U+FEFF) defensively so `^\s*`-anchored patterns match on - // line 1 and on any mid-file line that begins with a BOM (e.g. from file - // concatenation or tool insertion). StripLineLeadingBom assumes `\n` is - // the sole line separator, so the CRLF pass must come first. Non-line- - // leading U+FEFF is preserved so content with intentional ZWNBSP inside - // a string literal stays verbatim. Closes #183. - // まず CRLF / CR を LF に正規化する。StripLineLeadingBom は `\n` を唯一の - // 行区切りとして行頭判定するので、FileIndexer を経由しない direct call - // でも CRLF 正規化を済ませてから呼ばないと mid-file の行頭 BOM を剥がし - // 損なう。続いて行頭 U+FEFF のみ剥がし、1 行目と mid-file の行頭 BOM 両方 - // で `^\s*` 固定パターンを成立させる。行頭以外の U+FEFF (文字列リテラル中 - // の意図的な ZWNBSP 等) はそのまま保持する。Closes #183. - List? patterns = null; - var usesLineBasedExtractor = lang is "commonlisp" or "racket" or "solidity" or "html" or "assembly" - || (lang is not null && PatternCache.TryGetValue(lang, out patterns)); - if (!usesLineBasedExtractor) - return []; - - var lines = SplitContentLines(content); - cancellationToken.ThrowIfCancellationRequested(); - if (lang is "commonlisp" or "racket") - return ExtractLispSymbols(fileId, lang, lines); - - if (lang == "solidity") - return ExtractSoliditySymbols(fileId, lines); - - // HTML has no brace/indent-scoped bodies, so the generic pattern loop's - // "first match per line" semantics drop every additional symbol on the - // same line. HTML also needs cross-line masking of `` and - // raw-text children of `