Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 83 additions & 23 deletions .dev-loop/INGEST_REPORT.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,95 @@
# Knowledge flush1 insight
# Consolidated reviewknowledge PRs #6–#13

Source: RNR-3440 (사내 잠재매물 주간 추출 스크립트 메모리 피크 저감). Candidate:
"QueryPie 프록시 경유로 대용량 결과를 스트리밍할 때 server-side named cursor 대신
일반 커서 + `fetchmany` + openpyxl `write_only`."
Eight fork PRs (`dch0202-rsquare`, 2026-07-28 → 2026-08-02) were reviewed together
against `AGENTS.md`. Each PR was audited by an independent reviewer (format rules,
sources, vague-qualifier ban, ≤120 body lines, index/log invariants), then
cross-compared to catch duplication the per-PR flushes could not see — they branched
independently off the same main and rewrote the same shared index/log files. Fork
branches can't be edited from here and several PRs needed content changes (drop a
duplicate, merge a colliding page), so this branch carries the reconciled end-state
rather than merging each PR as-is (which would import the duplicates).

## Verified best-practice

**Claim 1 — psycopg2 server-side (named) cursor requires a transaction; fails under autocommit.**
- Source: psycopg2 usage docs — `https://github.com/psycopg/psycopg2/blob/master/doc/src/usage.rst` (via Context7). Quote: "Named cursors are typically created 'WITHOUT HOLD', meaning they exist only within the current transaction. Attempting to fetch from them after a commit or in autocommit mode raises an exception."
- Matches my live repro (`can't use a named cursor outside of transactions`). → **verified**
Sources are per-page and were live-verified in each originating PR's flush; the
independent re-reviews re-checked them. Landed pages and their evidence base:

**Claim 2 — a client-side (default) cursor pulls the whole result set to the client on execute; `fetchmany` only caps the Python-list explosion.**
- Source: psycopg2 cursor/usage docs + FAQ (named-cursor advantage = "data is fetched in chunks … minimal client memory"). By contrast the default cursor buffers the full result in libpq. → **verified**
| Page | Confidence | Source basis |
|------|-----------|--------------|
| backend/common/llm/completion-response-validation | verified | OpenAI reasoning guide + chat `object` spec (5 `finish_reason` values), vLLM/LiteLLM reasoning fields; field incident (200/`length`/empty content/8,173-char reasoning) |
| backend/common/llm/context-window-budget | verified | Claude context-window docs, LiteLLM exception mapping, vLLM/Claude Code env-var docs |
| backend/common/integrations/externally-owned-defaults | verified | OpenAI deprecations (notice windows) + models `list`, LiteLLM model_discovery; field incident (alias removed between PR verify and review → 400) |
| backend/common/storage/object-key-persistence | verified | AWS S3 CompleteMultipartUpload + managed-upload API/source, aws-sdk-js issues #1158/#5656 |
| infrastructure/containers/host-cgroup-visibility | field-tested | cgroup_namespaces(7), Docker `--cgroupns=host`, nsenter, k8s #103363; OrbStack repro |
| infrastructure/observability/missing-container-metrics | verified/field-tested | k8s resource-metrics-pipeline docs, kube-prometheus-stack values, kubernetes-mixin; OrbStack #2217 repro |
| platforms/environment/unicode-text-matching | verified | UAX #15, Unicode core §3.12, APFS FAQ, POSIX grep; local repro (macOS 15/APFS, grep 2.6.0-FreeBSD, Python 3.13) |
| platforms/shells/command-text-inspected-before-execution | verified | Claude Code hooks docs, POSIX shell §2.6; local reproduction |
| platforms/processes/non-interactive-cli-invocation | verified | GNU nohup, OpenBSD ssh/ssh_config, git, timeout man pages; no-request-in-gateway-log field incident |
| qa/document-verification/spec-document-gates | field-tested | ESLint, Google mutation testing, RFC 2119, Vale, markdownlint; 32/32 mutant / 62/62 intact RFC sessions |
| qa/document-verification/editing-a-gated-document | field-tested | pgrep, Vale, markdownlint; in-house editing methodology |
| testing/quality/checks-that-cannot-pass | verified | James Shore AoAD2, POSIX grep exit status, Semgrep rule-testing, pytest exit codes; BSD/ugrep measurement |
| testing/quality/spec-artifact-checks | verified | JSON Schema, ESLint RuleTester, pitest, GFM table spec; local cell-count repro + GitHub renderer cross-check |
| testing/quality/harness-reverse-controls | verified | mutation-testing + CI-control sources; field repro (re-fetched all cited URLs, PASS) |

**Claim 3 — openpyxl `write_only` gives near-constant memory (<10 MB); one save only; lxml is for serialization speed, not the memory saving.**
- Source: openpyxl Optimised Modes — `https://openpyxl.readthedocs.io/en/stable/optimized.html` (via WebSearch). "keeping memory usage under 10Mb"; "A write-only workbook can only be saved once"; "make sure you have lxml installed" for large dumps (speed).
- This **corrects** the raw candidate's "lxml unnecessary" → precise form: unnecessary *for the memory win*, recommended *for large-dump speed*. Confirmed by my server test (write-only worked with lxml absent). → **verified**

**Claim 4 — QueryPie blocks `BEGIN`, so server-side cursor is impossible there.**
- Environment-specific, no external source. Live repro in gui context: `autocommit=False` + named cursor → `[ENGINE] No permission to execute BEGIN statement`. → **field-tested**. Generalized in the page to "a read-only access-control proxy that blocks transaction control", with QueryPie as the concrete example (not a product-specific page).
- Memory figure 838 MB → 38 MB (300k synthetic rows) is my RNR-3440 measurement (`ru_maxrss`, separate processes).
Three pages were reconciled from two overlapping PR versions each, keeping the more
complete/better-sourced body and folding in the other's unique cases:
- **completion-response-validation** — #12 body (all five `finish_reason` values,
`tool_calls`/`function_call` carve-out, streaming, Responses API, "reasoning is
scratch, not deliverable") kept in `llm/` (coherent with #6/#13); folded in #6's
DeepSeek first-party edge + the field incident.
- **externally-owned-defaults** — #12 generalized body (any repo-external resource)
in `integrations/`; folded in #6's alias-removed field incident + the
gateway-config-vs-live-upstream nuance.
- **non-interactive-cli-invocation** — #12 body (GNU-nohup extension precision,
ssh -n stdin-detach vs BatchMode, pre-log DNS/TLS/proxy + `curl -v`) kept; folded
in #11's DEBIAN_FRONTEND, pager/color TTY case, wrapper-CLI case, field incident.

## Existing-layer check

- Pages read: `databases/index.md`, `databases/query-optimization/keyset-pagination.md`, `backend/python/index.md`.
- Overlap: keyset-pagination is the nearest neighbor (both handle large result sets) but a **distinct** topic — pagination splits the read into many bounded queries; this page streams a *single* query's result in chunks. Not a duplicate → new page + **bidirectional `related` link** added to both.
- backend/python has no DB-cursor page; the psycopg2/openpyxl specifics live as concrete examples inside the databases page rather than a separate python page (no duplication).
- Conflicts: none found.
Cross-PR and against-main duplication was the focus. Findings and resolutions:

- **spec-artifact-checks (#8) ≡ document-conformance-checks (#9)** — same case
(coverage-vs-validity split, per-check negative controls, GFM pipe parsing,
ESLint/Semgrep/mutation examples). #9's report predated awareness of #8. →
**#8 kept canonical; #9's page dropped, `testing/docs-as-spec` category not created.**
- **completion-response-validation (#6) ≈ llm-response-completeness (#12)** — ~95%
same case (HTTP 200 ≠ usable output; `length`/blank/reasoning-budget). →
**merged into one `llm/` page; #12's `integrations/` copy dropped.**
- **gateway-model-alias-defaults (#6) ≈ externally-owned-defaults (#12)** — ~80%;
#12 generalizes the model-alias case to any external resource. →
**kept the general `integrations/` page; #6's LLM-only page dropped.**
- **non-interactive-cli-invocation** — created by BOTH #11 and #12 (file collision).
→ **single reconciled page.**
- Distinct (no overlap, all landed): checks-that-cannot-pass, harness-reverse-controls,
spec-document-gates, editing-a-gated-document, unicode-text-matching,
command-text-inspected-before-execution, object-key-persistence, context-window-budget,
host-cgroup-visibility, missing-container-metrics.
- Reciprocal `related:` links added on existing pages (tests-that-cannot-fail,
timeouts-and-retries, environment-config, release-gates, background-services,
portable-shell-scripts, timezone-and-locale, paths-case-and-line-endings,
acceptance-criteria, resource-limits-and-probes, logs-metrics-signals,
minimum-case-set). A dropped-page backlink (#6 → gateway-model-alias-defaults on
environment-config and release-gates) was retargeted to externally-owned-defaults.
- Invariants verified programmatically: all `related:`/inline `[id]` references
resolve, every page listed in its domain index, no duplicate ids, no page >120
body lines.

## Routing decision

- Target: **`databases/query-optimization/streaming-large-result-sets.md`** (new page).
- Category `query-optimization` fits (memory-bounding how a query's result is pulled into the app is query-execution optimization); no new category needed.
- Registered in `databases/index.md` (query-optimization section) and appended to `log.md`.
- `backend/common/llm/` (new) — LLM-specific server concerns: completion-response-validation,
context-window-budget. Coherent home shared by #6 and #13.
- `backend/common/integrations/` (new) — general repo-external-dependency concern:
externally-owned-defaults. Kept separate from `llm/` because its scope is any
external resource (bucket/queue/index), not LLM-only.
- `backend/common/storage/` (new) — object-key-persistence.
- `qa/document-verification/` (new) — spec-document-gates, editing-a-gated-document.
Introduced by both #10 and #11; unified into one index section.
- `testing/quality/` (existing) — checks-that-cannot-pass, spec-artifact-checks,
harness-reverse-controls (test/check-authoring discipline, distinct from
qa/document-verification which is release-process gate design).
- `platforms/{environment,shells,processes}/` (existing) — unicode-text-matching,
command-text-inspected-before-execution, non-interactive-cli-invocation.
- `infrastructure/{containers,observability}/` (existing) — host-cgroup-visibility,
missing-container-metrics.

Source PRs #6–#13 are closed with a disposition comment crediting the author.
6 changes: 3 additions & 3 deletions INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,14 @@ follow the cross-pointers in their index or take the next matching seeded domain
| Domain | Status | Route here when |
|--------|--------|-----------------|
| [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior |
| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
| [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility |
| [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) |
| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) |
| [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing (writing automated test code → testing) |
| [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing, automated verification of document deliverables (spec/RFC gates) (writing automated test code → testing) |
| [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) |
| [security](wiki/security/index.md) | **seeded** | Trust-boundary decisions: input validation, session-vs-token auth choice, per-resource authorization (IDOR), secrets hygiene, dependency trust, PII handling (XSS rendering → frontend; CI secrets → infrastructure; JWT implementation → backend/frontend auth) |
| [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, background services/cron, toolchain version pinning |
| [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, Unicode normalization in text/file-name matching, commands inspected before execution, background services/cron, invoking prompt-capable CLIs non-interactively, toolchain version pinning |
| [mobile](wiki/mobile/index.md) | **seeded** | App-side iOS/Android/cross-platform: process death/state survival, offline-first sync, mobile-network calls, store rollout/hotfix strategy, startup time |

All ten domains are seeded. New categories grow via `skills/wiki-ingest/SKILL.md`.
3 changes: 3 additions & 0 deletions log.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,6 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
## [2026-07-12] revise | security/secrets-in-code +1 edge case: third-party HTTP client (httpx/requests) logs the full request URL — including a query-param API key — at INFO, so root/DEBUG logging leaks it; keep the client logger above INFO. Found when a standalone sync process set logging.basicConfig(INFO) and httpx wrote the data.go.kr serviceKey to the log file. last_verified bumped to 2026-07-12.
## [2026-07-13] ingest | databases +1 (query-optimization): streaming-large-result-sets — memory-bounded export of a huge single-query result. Client-side cursor pulls the whole set to libpq on execute (fetchmany caps only the Python-list explosion); only a server-side/named cursor truly streams but needs a transaction, so it fails under autocommit or a proxy that blocks BEGIN → fall back to client-side fetchmany + disk spool + openpyxl write_only (measured 300k rows 838MB→38MB). Derived from RNR-3440 (potential-listing weekly extract memory peak); QueryPie BEGIN-block generalized to "read-only access proxy", field-tested. Sources: psycopg2 usage/cursor docs (named-cursor WITHOUT HOLD + autocommit exception), openpyxl optimized-modes (write-only near-constant memory, lxml=speed-not-memory).
## [2026-07-23] ingest | databases +2: schema-design/online-schema-changes (ACCESS EXCLUSIVE lock avoidance — non-volatile default fast path, ADD CONSTRAINT NOT VALID + VALIDATE at SHARE UPDATE EXCLUSIVE, CHECK-NOT-NULL trick, CREATE INDEX CONCURRENTLY, expand-and-contract to decouple DB migration from app deploy, lock_timeout for lock-queue pile-up) + operations/autovacuum-and-wraparound (NEW category operations: per-table scale_factor/cost_limit tuning for hot tables, age(datfrozenxid)/relfrozenxid + n_dead_tup monitoring, wraparound read-only cliff and superuser VACUUM recovery, VACUUM FULL vs pg_repack). Derived from the Hatchet "Postgres survival guide"; both cross-checked against PostgreSQL official docs (sql-altertable, routine-vacuuming).
## [2026-08-03] ingest | Consolidated review of knowledge PRs #6–#13 (8 fork PRs) into 12 pages. New: backend/common/llm (completion-response-validation, context-window-budget), backend/common/integrations (externally-owned-defaults), backend/common/storage (object-key-persistence), infrastructure/containers/host-cgroup-visibility, infrastructure/observability/missing-container-metrics, platforms/environment/unicode-text-matching, platforms/shells/command-text-inspected-before-execution, platforms/processes/non-interactive-cli-invocation, qa/document-verification (spec-document-gates, editing-a-gated-document), testing/quality (checks-that-cannot-pass, spec-artifact-checks, harness-reverse-controls). All cited URLs are per-PR live-verified; three pages were reconciled from two overlapping PR versions each (see revise/dedup entries below).
## [2026-08-03] revise | Reconciled 3 pages from overlapping PR pairs, taking the more complete/better-sourced body and folding in the other's unique cases: backend/common/llm/completion-response-validation (#12 body — tool_calls/function_call carve-out, streaming, Responses API status==incomplete, "reasoning is scratch, not deliverable" — kept in llm/ per #6/#13 category, folded in #6's DeepSeek-first-party edge + the 8,173-char reasoning_content field incident); backend/common/integrations/externally-owned-defaults (#12 generalized body — any repo-external resource — folded in #6's LiteLLM-alias-removed field incident + gateway-config-vs-live-upstream nuance); platforms/processes/non-interactive-cli-invocation (#12 body — GNU-nohup extension precision, ssh -n stdin-detach vs BatchMode, pre-log DNS/TLS/proxy + curl -v — folded in #11's DEBIAN_FRONTEND, pager/color TTY case, wrapper-CLI case, and the no-request-in-gateway-log field incident).
## [2026-08-03] dedup | Dropped 3 candidate pages as duplicates/superseded during the #6–#13 consolidation: testing/docs-as-spec/document-conformance-checks (#9 — same case as testing/quality/spec-artifact-checks from #8: coverage-vs-validity split, per-check negative controls, GFM pipe parsing; #8 kept as canonical, docs-as-spec category not created); backend/common/llm/gateway-model-alias-defaults (#6 — subsumed by the generalized integrations/externally-owned-defaults; the model-alias case is one instance); backend/common/integrations/llm-response-completeness (#12 — folded into llm/completion-response-validation, kept in llm/ for category coherence with context-window-budget).
Loading
Loading