diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 2ff3871..c375750 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,35 +1,95 @@ -# Knowledge flush — 1 insight +# Consolidated review — knowledge PRs #6–#13 -Source: RNR-3440 (사내 잠재매물 주간 추출 스크립트 메모리 피크 저감). Candidate: -"QueryPie 프록시 경유로 대용량 결과를 스트리밍할 때 server-side named cursor 대신 -일반 커서 + `fetchmany` + openpyxl `write_only`." +Eight fork PRs (`dch0202-rsquare`, 2026-07-28 → 2026-08-02) were reviewed together +against `AGENTS.md`. Each PR was audited by an independent reviewer (format rules, +sources, vague-qualifier ban, ≤120 body lines, index/log invariants), then +cross-compared to catch duplication the per-PR flushes could not see — they branched +independently off the same main and rewrote the same shared index/log files. Fork +branches can't be edited from here and several PRs needed content changes (drop a +duplicate, merge a colliding page), so this branch carries the reconciled end-state +rather than merging each PR as-is (which would import the duplicates). ## Verified best-practice -**Claim 1 — psycopg2 server-side (named) cursor requires a transaction; fails under autocommit.** -- Source: psycopg2 usage docs — `https://github.com/psycopg/psycopg2/blob/master/doc/src/usage.rst` (via Context7). Quote: "Named cursors are typically created 'WITHOUT HOLD', meaning they exist only within the current transaction. Attempting to fetch from them after a commit or in autocommit mode raises an exception." -- Matches my live repro (`can't use a named cursor outside of transactions`). → **verified** +Sources are per-page and were live-verified in each originating PR's flush; the +independent re-reviews re-checked them. Landed pages and their evidence base: -**Claim 2 — a client-side (default) cursor pulls the whole result set to the client on execute; `fetchmany` only caps the Python-list explosion.** -- Source: psycopg2 cursor/usage docs + FAQ (named-cursor advantage = "data is fetched in chunks … minimal client memory"). By contrast the default cursor buffers the full result in libpq. → **verified** +| Page | Confidence | Source basis | +|------|-----------|--------------| +| backend/common/llm/completion-response-validation | verified | OpenAI reasoning guide + chat `object` spec (5 `finish_reason` values), vLLM/LiteLLM reasoning fields; field incident (200/`length`/empty content/8,173-char reasoning) | +| backend/common/llm/context-window-budget | verified | Claude context-window docs, LiteLLM exception mapping, vLLM/Claude Code env-var docs | +| backend/common/integrations/externally-owned-defaults | verified | OpenAI deprecations (notice windows) + models `list`, LiteLLM model_discovery; field incident (alias removed between PR verify and review → 400) | +| backend/common/storage/object-key-persistence | verified | AWS S3 CompleteMultipartUpload + managed-upload API/source, aws-sdk-js issues #1158/#5656 | +| infrastructure/containers/host-cgroup-visibility | field-tested | cgroup_namespaces(7), Docker `--cgroupns=host`, nsenter, k8s #103363; OrbStack repro | +| infrastructure/observability/missing-container-metrics | verified/field-tested | k8s resource-metrics-pipeline docs, kube-prometheus-stack values, kubernetes-mixin; OrbStack #2217 repro | +| platforms/environment/unicode-text-matching | verified | UAX #15, Unicode core §3.12, APFS FAQ, POSIX grep; local repro (macOS 15/APFS, grep 2.6.0-FreeBSD, Python 3.13) | +| platforms/shells/command-text-inspected-before-execution | verified | Claude Code hooks docs, POSIX shell §2.6; local reproduction | +| platforms/processes/non-interactive-cli-invocation | verified | GNU nohup, OpenBSD ssh/ssh_config, git, timeout man pages; no-request-in-gateway-log field incident | +| qa/document-verification/spec-document-gates | field-tested | ESLint, Google mutation testing, RFC 2119, Vale, markdownlint; 32/32 mutant / 62/62 intact RFC sessions | +| qa/document-verification/editing-a-gated-document | field-tested | pgrep, Vale, markdownlint; in-house editing methodology | +| testing/quality/checks-that-cannot-pass | verified | James Shore AoAD2, POSIX grep exit status, Semgrep rule-testing, pytest exit codes; BSD/ugrep measurement | +| testing/quality/spec-artifact-checks | verified | JSON Schema, ESLint RuleTester, pitest, GFM table spec; local cell-count repro + GitHub renderer cross-check | +| testing/quality/harness-reverse-controls | verified | mutation-testing + CI-control sources; field repro (re-fetched all cited URLs, PASS) | -**Claim 3 — openpyxl `write_only` gives near-constant memory (<10 MB); one save only; lxml is for serialization speed, not the memory saving.** -- Source: openpyxl Optimised Modes — `https://openpyxl.readthedocs.io/en/stable/optimized.html` (via WebSearch). "keeping memory usage under 10Mb"; "A write-only workbook can only be saved once"; "make sure you have lxml installed" for large dumps (speed). -- This **corrects** the raw candidate's "lxml unnecessary" → precise form: unnecessary *for the memory win*, recommended *for large-dump speed*. Confirmed by my server test (write-only worked with lxml absent). → **verified** - -**Claim 4 — QueryPie blocks `BEGIN`, so server-side cursor is impossible there.** -- Environment-specific, no external source. Live repro in gui context: `autocommit=False` + named cursor → `[ENGINE] No permission to execute BEGIN statement`. → **field-tested**. Generalized in the page to "a read-only access-control proxy that blocks transaction control", with QueryPie as the concrete example (not a product-specific page). -- Memory figure 838 MB → 38 MB (300k synthetic rows) is my RNR-3440 measurement (`ru_maxrss`, separate processes). +Three pages were reconciled from two overlapping PR versions each, keeping the more +complete/better-sourced body and folding in the other's unique cases: +- **completion-response-validation** — #12 body (all five `finish_reason` values, + `tool_calls`/`function_call` carve-out, streaming, Responses API, "reasoning is + scratch, not deliverable") kept in `llm/` (coherent with #6/#13); folded in #6's + DeepSeek first-party edge + the field incident. +- **externally-owned-defaults** — #12 generalized body (any repo-external resource) + in `integrations/`; folded in #6's alias-removed field incident + the + gateway-config-vs-live-upstream nuance. +- **non-interactive-cli-invocation** — #12 body (GNU-nohup extension precision, + ssh -n stdin-detach vs BatchMode, pre-log DNS/TLS/proxy + `curl -v`) kept; folded + in #11's DEBIAN_FRONTEND, pager/color TTY case, wrapper-CLI case, field incident. ## Existing-layer check -- Pages read: `databases/index.md`, `databases/query-optimization/keyset-pagination.md`, `backend/python/index.md`. -- Overlap: keyset-pagination is the nearest neighbor (both handle large result sets) but a **distinct** topic — pagination splits the read into many bounded queries; this page streams a *single* query's result in chunks. Not a duplicate → new page + **bidirectional `related` link** added to both. -- backend/python has no DB-cursor page; the psycopg2/openpyxl specifics live as concrete examples inside the databases page rather than a separate python page (no duplication). -- Conflicts: none found. +Cross-PR and against-main duplication was the focus. Findings and resolutions: + +- **spec-artifact-checks (#8) ≡ document-conformance-checks (#9)** — same case + (coverage-vs-validity split, per-check negative controls, GFM pipe parsing, + ESLint/Semgrep/mutation examples). #9's report predated awareness of #8. → + **#8 kept canonical; #9's page dropped, `testing/docs-as-spec` category not created.** +- **completion-response-validation (#6) ≈ llm-response-completeness (#12)** — ~95% + same case (HTTP 200 ≠ usable output; `length`/blank/reasoning-budget). → + **merged into one `llm/` page; #12's `integrations/` copy dropped.** +- **gateway-model-alias-defaults (#6) ≈ externally-owned-defaults (#12)** — ~80%; + #12 generalizes the model-alias case to any external resource. → + **kept the general `integrations/` page; #6's LLM-only page dropped.** +- **non-interactive-cli-invocation** — created by BOTH #11 and #12 (file collision). + → **single reconciled page.** +- Distinct (no overlap, all landed): checks-that-cannot-pass, harness-reverse-controls, + spec-document-gates, editing-a-gated-document, unicode-text-matching, + command-text-inspected-before-execution, object-key-persistence, context-window-budget, + host-cgroup-visibility, missing-container-metrics. +- Reciprocal `related:` links added on existing pages (tests-that-cannot-fail, + timeouts-and-retries, environment-config, release-gates, background-services, + portable-shell-scripts, timezone-and-locale, paths-case-and-line-endings, + acceptance-criteria, resource-limits-and-probes, logs-metrics-signals, + minimum-case-set). A dropped-page backlink (#6 → gateway-model-alias-defaults on + environment-config and release-gates) was retargeted to externally-owned-defaults. +- Invariants verified programmatically: all `related:`/inline `[id]` references + resolve, every page listed in its domain index, no duplicate ids, no page >120 + body lines. ## Routing decision -- Target: **`databases/query-optimization/streaming-large-result-sets.md`** (new page). -- Category `query-optimization` fits (memory-bounding how a query's result is pulled into the app is query-execution optimization); no new category needed. -- Registered in `databases/index.md` (query-optimization section) and appended to `log.md`. +- `backend/common/llm/` (new) — LLM-specific server concerns: completion-response-validation, + context-window-budget. Coherent home shared by #6 and #13. +- `backend/common/integrations/` (new) — general repo-external-dependency concern: + externally-owned-defaults. Kept separate from `llm/` because its scope is any + external resource (bucket/queue/index), not LLM-only. +- `backend/common/storage/` (new) — object-key-persistence. +- `qa/document-verification/` (new) — spec-document-gates, editing-a-gated-document. + Introduced by both #10 and #11; unified into one index section. +- `testing/quality/` (existing) — checks-that-cannot-pass, spec-artifact-checks, + harness-reverse-controls (test/check-authoring discipline, distinct from + qa/document-verification which is release-process gate design). +- `platforms/{environment,shells,processes}/` (existing) — unicode-text-matching, + command-text-inspected-before-execution, non-interactive-cli-invocation. +- `infrastructure/{containers,observability}/` (existing) — host-cgroup-visibility, + missing-container-metrics. + +Source PRs #6–#13 are closed with a disposition comment crediting the author. diff --git a/INDEX.md b/INDEX.md index 05789bd..d6d4238 100644 --- a/INDEX.md +++ b/INDEX.md @@ -10,14 +10,14 @@ follow the cross-pointers in their index or take the next matching seeded domain | Domain | Status | Route here when | |--------|--------|-----------------| | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior | -| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | +| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility | | [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) | | [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) | -| [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing (writing automated test code → testing) | +| [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing, automated verification of document deliverables (spec/RFC gates) (writing automated test code → testing) | | [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) | | [security](wiki/security/index.md) | **seeded** | Trust-boundary decisions: input validation, session-vs-token auth choice, per-resource authorization (IDOR), secrets hygiene, dependency trust, PII handling (XSS rendering → frontend; CI secrets → infrastructure; JWT implementation → backend/frontend auth) | -| [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, background services/cron, toolchain version pinning | +| [platforms](wiki/platforms/index.md) | **seeded** | OS-level differences breaking code across macOS/Linux/Windows: shell portability, BSD-vs-GNU CLI, filesystem case/line endings, Unicode normalization in text/file-name matching, commands inspected before execution, background services/cron, invoking prompt-capable CLIs non-interactively, toolchain version pinning | | [mobile](wiki/mobile/index.md) | **seeded** | App-side iOS/Android/cross-platform: process death/state survival, offline-first sync, mobile-network calls, store rollout/hotfix strategy, startup time | All ten domains are seeded. New categories grow via `skills/wiki-ingest/SKILL.md`. diff --git a/log.md b/log.md index 587a5dc..1c6293b 100644 --- a/log.md +++ b/log.md @@ -34,3 +34,6 @@ Append-only. Format: `## [YYYY-MM-DD] window (Claude 4.5 and newer) | The request succeeds and generation stops with `stop_reason: "model_context_window_exceeded"` — branch on the stop reason, because a truncated answer is not an error and arrives as a normal 200 | +| Extended thinking is on | Thinking tokens are a subset of `max_tokens` and billed as output — raise the reservation for thinking instead of assuming it is free, then re-derive step 2 | +| The gateway advertises a window that differs from the server's | Trust the serving engine's configured length (the value that rejects the request), and correct the gateway's model config so its fallbacks compute against the same number | +| Prompt caching is enabled | Cached prefixes still occupy the window — caching changes billing, not occupancy, so the input side of step 2 is unchanged | +| The client is Claude Code and the base URL is not `api.anthropic.com` | MCP tool search is disabled by default (`ENABLE_TOOL_SEARCH=true` when the proxy forwards `tool_reference` blocks) and Remote Control is off as of v2.1.196 — budget the tool definitions as always-present input | +| The window is large enough but responses truncate mid-structure | The cap, not the window, is the limit — raise `max_output` toward the step-2 ceiling ([backend-common-api-design-error-responses] for surfacing the truncation to callers) | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Retry or raise the timeout after a context-window 400 | Recompute the cap from the error's own three numbers and resend once | The rejection is arithmetic, not transient; every retry fails identically ([backend-common-reliability-timeouts-and-retries] owns what is retryable) | +| Keep the client's default output cap when swapping in a smaller model | Set the cap from the new model's window before the first request | The default is sized for the vendor's largest window, so the smaller model fails on the first full-context turn rather than degrading later | +| Set the base URL to `http://host:4000/v1` | Set it to `http://host:4000` | The client appends `/v1/messages` itself, so the duplicated prefix 404s while looking like an auth or routing fault | +| Lower the cap until requests stop failing | Compute `window − worst_case_input` once and set that | Trial-and-error lands on a number that holds for today's history length and breaks as the conversation grows | + +## Sources + +- https://platform.claude.com/docs/en/build-with-claude/context-windows — the window "holds the conversation history plus the new output"; "Everything in the request counts toward the context window: the system prompt, every message in `messages` (including tool results, images, and documents), and your tool definitions"; overflow behavior: input alone over the window → 400 `invalid_request_error`, while on Claude 4.5+ input + `max_tokens` over the window is accepted and stops with `stop_reason: "model_context_window_exceeded"` (earlier models return a validation error); thinking tokens "are a subset of your `max_tokens` parameter"; "Cached prompt prefixes still occupy the context window" +- https://docs.litellm.ai/docs/exception_mapping — "400 | ContextWindowExceededError | litellm.BadRequestError | Special error type for context window exceeded error messages - enables context window fallbacks" +- https://docs.litellm.ai/docs/anthropic_unified — LiteLLM serves the Anthropic-format `/v1/messages` endpoint for "All LiteLLM supported providers" (openai, bedrock, vertex_ai, gemini, azure …), which is what lets an Anthropic-protocol client sit in front of an OpenAI-compatible model +- https://code.claude.com/docs/en/env-vars — `ANTHROPIC_BASE_URL`: "Override the API endpoint to route requests through a proxy or gateway. When set to a non-first-party host, MCP tool search is disabled by default. Set `ENABLE_TOOL_SEARCH=true` if your proxy forwards `tool_reference` blocks"; Remote Control disabled for non-`api.anthropic.com` hosts as of v2.1.196 +- https://docs.vllm.ai/en/stable/serving/integrations/claude_code/ — `ANTHROPIC_BASE_URL=http://localhost:8000` "Points to your vLLM server (default port is 8000)" — the root, with no `/v1` suffix +- Field reproduction 2026-07-31 (Claude Code 2.1.220 → LiteLLM → 128k-window OpenAI-compatible model): default output cap + 99,073 input tokens exceeded the 131,072-token window and returned 400 on the first request; an 8,192-token cap ran the same session through tool calls. `CLAUDE_CODE_MAX_OUTPUT_TOKENS` and `CLAUDE_CODE_MAX_CONTEXT_TOKENS` confirmed present in the shipped binary, and absent from the published env-vars reference diff --git a/wiki/backend/common/reliability/timeouts-and-retries.md b/wiki/backend/common/reliability/timeouts-and-retries.md index b64e69a..df130d6 100644 --- a/wiki/backend/common/reliability/timeouts-and-retries.md +++ b/wiki/backend/common/reliability/timeouts-and-retries.md @@ -9,7 +9,7 @@ sources: - https://sre.google/sre-book/addressing-cascading-failures/ - https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ last_verified: 2026-07-10 -related: [backend-common-api-design-idempotency] +related: [backend-common-api-design-idempotency, backend-common-llm-completion-response-validation] --- # Calling Another Service over the Network: Timeouts, Retries, Backoff diff --git a/wiki/backend/common/storage/object-key-persistence.md b/wiki/backend/common/storage/object-key-persistence.md new file mode 100644 index 0000000..c756cd7 --- /dev/null +++ b/wiki/backend/common/storage/object-key-persistence.md @@ -0,0 +1,75 @@ +--- +id: backend-common-storage-object-key-persistence +domain: backend +category: storage +applies_to: [aws-s3, aws-sdk-js-v2, aws-sdk-js-v3] +confidence: verified +sources: + - https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html + - https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html + - https://github.com/aws/aws-sdk-js/blob/master/lib/s3/managed_upload.js + - https://github.com/aws/aws-sdk-js/issues/1158 + - https://github.com/aws/aws-sdk-js-v3/issues/5656 +last_verified: 2026-07-30 +related: [backend-common-api-design-idempotency, backend-node-boundaries-runtime-validation] +--- + +# Persisting the Result of an Object-Storage Upload + +## When this applies + +You are writing the result of a managed/multipart object-storage upload +(`s3.upload()`, `@aws-sdk/lib-storage` `Upload`, an equivalent transfer manager) +into a database column, a message, or any store read back later — and you are +choosing which response field to persist. + +## Do this + +1. **Persist the raw object key (`Key`) plus the bucket, and derive URLs at read + time.** `Key` is the field the service documents as "the object key of the + newly created object" and is what every later call (`GetObject`, + `getSignedUrl`, `HeadObject`, delete) takes as input. +2. **Treat `Location` as display-only.** Its encoding and host are not a + documented contract, and the value comes from a different producer on each + upload path: + +| Upload path | Taken when | Who produces `Location` | Observed form | +|-------------|------------|-------------------------|---------------| +| Single part (`PutObject`) | Body ≤ the managed uploader's part size — 5 MiB (5,242,880 B) by default in aws-sdk-js | The SDK, joining `protocol + host + httpRequest.path` | Path-style percent-encoding: space → `%20` | +| Multipart (`CompleteMultipartUpload`) | Body over that threshold | The S3 response, passed through after the SDK rewrites `%2F` back to `/` | Service encoding with only slashes repaired: space → `+` survives | + +3. **Build the read path from `Key`.** Sign or fetch with the stored key exactly + as stored; apply no decode step. A stored key needs no repair, so no decode + rule has to be guessed later. +4. **When a column already holds URLs, migrate by extracting the key** — reverse + both encodings by upload size, verify each candidate with `HeadObject`, and + write `Key` back. Add the `Key` column before the code switch so old and new + rows are both readable ([backend-common-api-design-idempotency] for the + backfill's retry behavior). + +## Edge cases + +| Case | Then | +|------|------| +| A CDN or custom domain fronts the bucket | Store `Key` and compose `cdnBase + encodeURI(key)` at read time; in aws-sdk-js-v3 the multipart path returns the origin host in `Location` while the single-part path returns the custom domain (issue #5656) | +| The key contains `+` as a literal character | Storing `Key` keeps it literal; a URL round-trip cannot distinguish it from an encoded space, which is why the encoded form is not a safe identifier | +| A bug report says "only large files 404" | Read it as the single-part/multipart split, not as intermittency: the threshold is a deterministic byte boundary, so reproduce at part-size ± 1 byte | +| Only `Location` was ever stored and the objects must be found now | Try both decodings per row (`+` → space and `+` literal), confirm with `HeadObject`, and record which rows stayed ambiguous | +| `partSize` is configured above the default | The boundary moves to that value; read it from the uploader config rather than assuming 5 MiB | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Save `uploadRes.Location` as the object's identifier | Save `uploadRes.Key` (with `Bucket`) and build URLs on read | `Location` is produced by two different code paths with two different encodings, so one column ends up holding both | +| Add a `decodeURIComponent` on read to fix the broken rows | Store the raw key so the read path needs no decode | Standard URL decoding leaves `+` as a literal, so it repairs the single-part rows and leaves the multipart rows 404-ing | +| Chase the mismatch as a nondeterministic uploader bug | Compare file sizes against the uploader's part size and reproduce at the boundary | The behaviour is decided by one byte-size comparison; naming it "intermittent" sends the fix to the upload layer instead of the persistence layer | +| Normalize the encoding inside the upload wrapper | Change what the caller persists | The wrapper cannot fix rows already written, and the service keeps returning its own `Location` regardless | + +## Sources + +- https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html — response elements: `Key` is "the object key of the newly created object"; `Location` is documented only as "the URI that identifies the newly created object", with no encoding guarantee +- https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html — callback data carries `Location`, `ETag`, `Bucket`, `Key`; `AWS.S3.ManagedUpload.minPartSize = 1024 * 1024 * 5` and the default `partSize` is 5 MB +- https://github.com/aws/aws-sdk-js/blob/master/lib/s3/managed_upload.js — `finishSinglePart` builds `data.Location` from `endpoint.protocol + '//' + endpoint.host + httpReq.path` and sets `data.Key` from `params.Key`; `finishMultiPart` takes the service's `Location` and applies only `.replace(/%2F/g, '/')` +- https://github.com/aws/aws-sdk-js/issues/1158 — reported inconsistency: multipart returns `…/stream-uploads%2Fkokoko.gif` where single-part returns `…/stream-uploads/kokoko.gif`; `Key` is identical in both responses +- https://github.com/aws/aws-sdk-js-v3/issues/5656 — same split in v3 `lib-storage`: `__uploadUsingPut` constructs `Location` client-side, `CompleteMultipartUploadCommand` does not diff --git a/wiki/backend/index.md b/wiki/backend/index.md index e991b04..9e683bd 100644 --- a/wiki/backend/index.md +++ b/wiki/backend/index.md @@ -5,7 +5,7 @@ three stack subtrees — route by concern first, stack second: | Subtree | Route there when | |---------|------------------| -| [common](#common-language-agnostic) (below) | The concern is language-agnostic: API contracts, idempotency, JWT issuance, outbound calls, caching, jobs, transactions in app code, shared state/pools, exception structure | +| [common](#common-language-agnostic) (below) | The concern is language-agnostic: API contracts, idempotency, JWT issuance, outbound calls, caching, jobs, transactions in app code, shared state/pools, exception structure, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references | | [java](java/index.md) | You are writing/reviewing JVM backend code (Java/Kotlin, Spring, JPA/Hibernate) and the concern is stack-specific: entity mapping, persistence context, proxy pitfalls, JVM threads/memory | | [node](node/index.md) | You are writing/reviewing Node.js/TypeScript backend code: event-loop blocking, promise error handling, runtime validation at boundaries, graceful shutdown | | [python](python/index.md) | You are writing/reviewing Python backend code: GIL/concurrency model, pydantic validation, WSGI/ASGI workers, language traps | @@ -70,3 +70,22 @@ Match your situation to a "load when" line; load only matching pages. |------|-----------| | [shared-state-and-pools](common/concurrency/shared-state-and-pools.md) | Request handlers share in-process mutable state — concurrency-safe structures/confinement vs shared store in multi-instance deployments; sizing thread/connection pools; same-pool nested-acquisition deadlock; bounding queues; debugging deadlock or starvation under load | | [distributed-locks](common/concurrency/distributed-locks.md) | Only one instance may perform an action at a time — Redis-style lock with owner token and TTL/watchdog, safe release, when a DB constraint/advisory lock suffices instead; debugging locks released by the wrong holder or work done twice despite a lock | + +### llm + +| Page | Load when | +|------|-----------| +| [completion-response-validation](common/llm/completion-response-validation.md) | Consuming OpenAI-compatible `/chat/completions` output as a final artifact (summary, document, notification); LLM responses coming back empty or truncated while HTTP status is 200; a reasoning-family model may be routed onto the alias you call | +| [context-window-budget](common/llm/context-window-budget.md) | Repointing an LLM client or agent CLI at a different model, a self-hosted server (vLLM/Ollama), or a gateway (LiteLLM); setting `max_tokens` for a client whose default was sized for a larger model; the first request after such a switch returns 400 with a context-window error; deciding where to set the cap (request body vs client env var vs gateway config) and how to point the base URL at a proxy; handling truncation that arrives as a normal 200 | + +### integrations + +| Page | Load when | +|------|-----------| +| [externally-owned-defaults](common/integrations/externally-owned-defaults.md) | A code/config default names a resource the repo does not own (model alias, endpoint, bucket, queue, index) — reviewing or merging a PR that claims that default works, adding a startup check that the name still resolves, or diagnosing a default path that broke with no code change | + +### storage + +| Page | Load when | +|------|-----------| +| [object-key-persistence](common/storage/object-key-persistence.md) | Persisting the result of an object-storage upload (`s3.upload()`, `lib-storage` `Upload`, a transfer manager) — choosing which response field goes in the DB column; building the read/signing path from a stored reference; migrating a column that holds URLs to keys; only large uploads 404 on read | diff --git a/wiki/infrastructure/config/environment-config.md b/wiki/infrastructure/config/environment-config.md index 8964889..3b0bf72 100644 --- a/wiki/infrastructure/config/environment-config.md +++ b/wiki/infrastructure/config/environment-config.md @@ -9,7 +9,7 @@ sources: - https://12factor.net/build-release-run - https://12factor.net/dev-prod-parity last_verified: 2026-07-10 -related: [infrastructure-deploy-rollout-and-rollback, infrastructure-ci-cd-secrets-handling, backend-node-boundaries-runtime-validation, backend-python-boundaries-runtime-validation] +related: [infrastructure-deploy-rollout-and-rollback, infrastructure-ci-cd-secrets-handling, backend-node-boundaries-runtime-validation, backend-python-boundaries-runtime-validation, backend-common-integrations-externally-owned-defaults] --- # Configuration That Differs Per Environment @@ -61,6 +61,7 @@ files, and env vars; reviewing how a service gets its settings. |------|------| | A key is only meaningful in prd (e.g. a payments endpoint) | Declare it required in every environment and give dev/stg a working sandbox value — an optional-in-dev key hides a missing-prd-value crash until the prd deploy | | A value must change without a redeploy (kill switch, tuning knob) | Use a runtime flag ([infrastructure-deploy-rollout-and-rollback] config/flag row); flag names and allowed values still belong in the schema inventory | +| A required value names a resource the repo does not own (model alias, endpoint, bucket, queue) | Startup validation must resolve the name against the owner's catalog, not just check that the string is present — [backend-common-integrations-externally-owned-defaults] owns the review-time and startup checks | | Config service or mounted config unreachable at boot | Crash and let the orchestrator restart/retry; starting with fallback values means each instance runs config you cannot account for | | Legacy code full of `if (env === 'prod')` branches | On each touch, replace the branch you are editing with a named config value; record the remaining branches as inventory gaps | diff --git a/wiki/infrastructure/containers/host-cgroup-visibility.md b/wiki/infrastructure/containers/host-cgroup-visibility.md new file mode 100644 index 0000000..0f06bcc --- /dev/null +++ b/wiki/infrastructure/containers/host-cgroup-visibility.md @@ -0,0 +1,67 @@ +--- +id: infrastructure-containers-host-cgroup-visibility +domain: infrastructure +category: containers +applies_to: [kubernetes, docker, cgroup-v2] +confidence: field-tested +sources: + - https://man7.org/linux/man-pages/man7/cgroup_namespaces.7.html + - https://docs.docker.com/reference/cli/docker/container/run/ + - https://man7.org/linux/man-pages/man1/nsenter.1.html + - https://github.com/kubernetes/kubernetes/issues/103363 +last_verified: 2026-07-28 +related: [infrastructure-containers-resource-limits-and-probes, infrastructure-observability-missing-container-metrics] +--- + +# Reading Other Pods' cgroup v2 Stats from Inside a Container + +## When this applies + +A container needs to read the host's full cgroup v2 hierarchy — other pods' +CPU/memory/PSI stats under `kubepods/` — via hostPath or `docker -v` mounting +of `/sys/fs/cgroup`. The mount succeeds but the `kubepods` subtree is missing, +with no error (empty or partial listing, silently wrong results). + +## Do this + +Mounting the path is not sufficient; the cgroup namespace decides the view. +On cgroup v2, container runtimes default to a **private cgroup namespace** +(dockerd `--default-cgroupns-mode` default: `private`), and the cgroup2 +filesystem mounted for the container in that namespace shows only the +container's own subtree — the host hierarchy is simply absent. + +| Case | Do | +|------|----| +| Docker/standalone container | Run with `--cgroupns=host` in addition to the `-v /sys/fs/cgroup:...` mount — the container then gets the host's cgroup namespace and the full hierarchy | +| Kubernetes pod | There is no first-class host-cgroupns pod option (kubernetes#103363, open); use a privileged pod, or `hostPID: true` + `nsenter -t 1 -C` (`--cgroup`) to run the reading command inside PID 1's cgroup namespace | +| Verifying before trusting the data | List the mount for `kubepods/` (or the equivalent top-level slice) explicitly; treat its absence as a namespace problem, never as "those pods have no stats" | + +Mechanism: cgroup namespaces virtualize the cgroup view — `/proc/[pid]/cgroup` +paths and the cgroupfs contents are shown relative to the namespace's root, +fixed by the cgroup namespace in effect when that cgroup2 filesystem mount was +created (cgroup_namespaces(7)). The runtime creates the container's cgroup +mount inside the private namespace, so what looks like "the host's +/sys/fs/cgroup" is a namespaced view rooted at the container's own cgroup. + +## Edge cases + +| Case | Then | +|------|------| +| Files are visible but writes fail (`cgroup.procs`: no such file / EPERM) | A different restriction: the `nsdelegate` mount option (systemd default) blocks cross-namespace-boundary writes even when reads work — reading stats is fine, migrating processes is not | +| Reading only the container's OWN limits/usage | No host namespace needed — the default namespaced view is exactly right for self-monitoring | +| Sizing decisions based on the stats you read | Limits/QoS interpretation: [infrastructure-containers-resource-limits-and-probes] | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Conclude "cgroup data isn't there" from an empty mounted directory | Check the cgroup namespace first (`--cgroupns=host` / `nsenter -t 1 -C`), then re-list | Measured (OrbStack VM, 2026-07-22): identical `-v /sys/fs/cgroup` mount showed no `kubepods` without `--cgroupns=host` and the full `kubepods/burstable/pod/` tree with it — the failure mode is silent | +| Parse per-pod stats from inside a default (non-privileged) pod | Read via a host-namespace-capable agent (privileged/hostPID DaemonSet) or consume kubelet metrics endpoints instead: [infrastructure-observability-missing-container-metrics] | The default pod cgroupns cannot see sibling pods at all | + +## Sources + +- https://man7.org/linux/man-pages/man7/cgroup_namespaces.7.html — cgroup namespaces virtualize the cgroup view; cgroupfs contents depend on the namespace of the mount's creator; nsdelegate write restrictions +- https://docs.docker.com/reference/cli/docker/container/run/ — `--cgroupns=host` runs the container in the host's cgroup namespace; private is the cgroup v2 default +- https://man7.org/linux/man-pages/man1/nsenter.1.html — `-C/--cgroup` enters the target process's cgroup namespace +- https://github.com/kubernetes/kubernetes/issues/103363 — no first-class host cgroupns for pods; nsenter-based workaround pattern (as used by Cilium) +- Field context: 2026-07-22 OrbStack VM measurement — `docker run -v /sys/fs/cgroup:/hostcg alpine`: `kubepods` absent; adding `--cgroupns=host`: full `kubepods/burstable/pod/` hierarchy visible diff --git a/wiki/infrastructure/containers/resource-limits-and-probes.md b/wiki/infrastructure/containers/resource-limits-and-probes.md index d76c93b..1e11b65 100644 --- a/wiki/infrastructure/containers/resource-limits-and-probes.md +++ b/wiki/infrastructure/containers/resource-limits-and-probes.md @@ -9,7 +9,7 @@ sources: - https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ - https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ last_verified: 2026-07-10 -related: [infrastructure-deploy-rollout-and-rollback, backend-java-runtime-threads-and-memory] +related: [infrastructure-deploy-rollout-and-rollback, backend-java-runtime-threads-and-memory, infrastructure-containers-host-cgroup-visibility, infrastructure-observability-missing-container-metrics] --- # Resource Limits and Health Probes in Deployment Manifests diff --git a/wiki/infrastructure/index.md b/wiki/infrastructure/index.md index 4940602..2bff59c 100644 --- a/wiki/infrastructure/index.md +++ b/wiki/infrastructure/index.md @@ -25,6 +25,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| +| [host-cgroup-visibility](containers/host-cgroup-visibility.md) | A container must read the host's full cgroup v2 hierarchy (other pods' CPU/memory stats) via a hostPath/`-v` mount of `/sys/fs/cgroup`; the mounted directory is missing the `kubepods` subtree with no error | | [image-builds](containers/image-builds.md) | Writing or reviewing a Dockerfile; images rebuild everything on small changes, build slowly, or are too large; choosing an image tagging scheme | | [resource-limits-and-probes](containers/resource-limits-and-probes.md) | Writing or reviewing Kubernetes-style deployment manifests; pods OOMKilled, evicted, or CPU-throttled; a dependency outage triggered a restart storm; traffic hitting pods that are not ready | @@ -45,4 +46,5 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [logs-metrics-signals](observability/logs-metrics-signals.md) | Instrumenting a new or existing service (logs, metrics, correlation ids); an incident revealed you couldn't see what happened; choosing between a log line and a metric; a metric label would carry unbounded values (user ids/UUIDs) | +| [missing-container-metrics](observability/missing-container-metrics.md) | Prometheus `container_*` CPU/memory series are empty or pod dashboards blank while kubelet scrape targets all report up (common on embedded/VM Kubernetes like OrbStack); deciding between cAdvisor and kubelet `/metrics/resource` scraping | | [alerting](observability/alerting.md) | Creating or reviewing alerts; the team ignores a noisy pager; deciding whether a condition pages, tickets, or stays on a dashboard | diff --git a/wiki/infrastructure/observability/logs-metrics-signals.md b/wiki/infrastructure/observability/logs-metrics-signals.md index 87e9108..2d11dbb 100644 --- a/wiki/infrastructure/observability/logs-metrics-signals.md +++ b/wiki/infrastructure/observability/logs-metrics-signals.md @@ -9,7 +9,7 @@ sources: - https://prometheus.io/docs/practices/naming/ - https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html last_verified: 2026-07-10 -related: [infrastructure-observability-alerting] +related: [infrastructure-observability-alerting, infrastructure-observability-missing-container-metrics] --- # Instrumenting a Service with Logs and Metrics diff --git a/wiki/infrastructure/observability/missing-container-metrics.md b/wiki/infrastructure/observability/missing-container-metrics.md new file mode 100644 index 0000000..5127022 --- /dev/null +++ b/wiki/infrastructure/observability/missing-container-metrics.md @@ -0,0 +1,80 @@ +--- +id: infrastructure-observability-missing-container-metrics +domain: infrastructure +category: observability +applies_to: [kubernetes, kube-prometheus-stack] +confidence: verified +sources: + - https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/ + - https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml + - https://raw.githubusercontent.com/kubernetes-monitoring/kubernetes-mixin/master/dashboards/resources/queries/pod.libsonnet + - https://github.com/orbstack/orbstack/issues/2217 +last_verified: 2026-07-28 +related: [infrastructure-observability-logs-metrics-signals, infrastructure-containers-resource-limits-and-probes] +--- + +# Container Metrics Empty While Kubelet Scrape Targets Report Up + +## When this applies + +Monitoring pod CPU/memory with Prometheus (e.g. kube-prometheus-stack) and +`container_*` series are empty or dashboards show nothing, while every kubelet +scrape target is healthy ("up"). Common on embedded/VM Kubernetes distributions +(observed on OrbStack) whose kubelet cAdvisor endpoint emits only `machine_*` +series. + +## Do this + +1. Diagnose by series presence, not target health — a successful scrape of an + endpoint that emits no `container_*` series still shows "up": + +``` +kubectl get --raw /api/v1/nodes//proxy/metrics/cadvisor | grep -c '^container_' +kubectl get --raw /api/v1/nodes//proxy/metrics/resource | grep -c '^container_' +``` + +2. When cAdvisor emits no `container_*` series but `/metrics/resource` does, + scrape the kubelet resource endpoint instead. It serves + `container_cpu_usage_seconds_total` and `container_memory_working_set_bytes` + (defined in kubelet's resource-metrics collector). In kube-prometheus-stack + values: + +```yaml +kubelet: + serviceMonitor: + resource: true + resourcePath: "/metrics/resource" +``` + + Both lines are required: the chart's default `resourcePath` is still + `/metrics/resource/v1alpha1` (renamed in Kubernetes 1.18), which 404s on + modern kubelets — enabling `resource: true` alone produces a down target. + +3. Expect the bundled "Compute Resources" dashboards to stay empty even after + the data arrives: kubernetes-mixin dashboard queries and recording rules + filter with `image!=""`, and `/metrics/resource` series carry no `image` + label (only container/pod/namespace). Build a custom dashboard or recording + rules against the resource-endpoint series directly. + +## Edge cases + +| Case | Then | +|------|------| +| You need per-container filesystem, network, or throttling metrics | `/metrics/resource` carries only CPU and memory; those richer series exist only in cAdvisor — fix or replace the runtime's cAdvisor support instead | +| Dashboards empty but PromQL on the raw series returns data | You are hitting the `image!=""` filter, not a collection gap — adjust the queries, not the scrape | +| Verifying the fix | Query `container_memory_working_set_bytes{container!=""}` directly in Prometheus and compare one pod against `kubectl top pod` | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Conclude "metrics are being collected" from all-green scrape targets | Count the specific series you need at the source endpoint | Scrape health checks the HTTP exchange, not whether the series you consume exist | +| Enable `kubelet.serviceMonitor.resource: true` and stop | Also set `resourcePath: "/metrics/resource"` | The chart default still points at the pre-1.18 `v1alpha1` path, which 404s | + +## Sources + +- https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/ — kubelet `/metrics/resource` endpoint in the resource metrics pipeline +- https://raw.githubusercontent.com/kubernetes/kubernetes/master/pkg/kubelet/metrics/collectors/resource_metrics.go — the endpoint's exact series: `container_cpu_usage_seconds_total`, `container_memory_working_set_bytes` +- https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml — `kubelet.serviceMonitor.resource` (default false) and `resourcePath` (default `/metrics/resource/v1alpha1`, comment: renamed in k8s 1.18) +- https://raw.githubusercontent.com/kubernetes-monitoring/kubernetes-mixin/master/dashboards/resources/queries/pod.libsonnet — `container_memory_working_set_bytes{..., image!=""}` filter; same pattern in `rules/apps.libsonnet` recording rules +- https://github.com/orbstack/orbstack/issues/2217 — corroborates node-level metrics working while container-level silently missing on OrbStack embedded k8s (the machine_*-only cAdvisor observation itself is field-tested, 2026-07-15) diff --git a/wiki/platforms/environment/timezone-and-locale.md b/wiki/platforms/environment/timezone-and-locale.md index a4927ef..729de7c 100644 --- a/wiki/platforms/environment/timezone-and-locale.md +++ b/wiki/platforms/environment/timezone-and-locale.md @@ -14,7 +14,7 @@ sources: - https://unicode.org/faq/casemap_charprop.html - https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap07.html last_verified: 2026-07-10 -related: [databases-schema-design-column-data-types, platforms-processes-background-services] +related: [databases-schema-design-column-data-types, platforms-processes-background-services, platforms-environment-unicode-text-matching] --- # Timezone and Locale as Hidden Inputs to Date and Text Code diff --git a/wiki/platforms/environment/unicode-text-matching.md b/wiki/platforms/environment/unicode-text-matching.md new file mode 100644 index 0000000..cccb83b --- /dev/null +++ b/wiki/platforms/environment/unicode-text-matching.md @@ -0,0 +1,68 @@ +--- +id: platforms-environment-unicode-text-matching +domain: platforms +category: environment +applies_to: [general] +confidence: verified +sources: + - https://unicode.org/reports/tr15/ + - https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/ + - https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html + - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html +last_verified: 2026-07-30 +related: [platforms-environment-timezone-and-locale, platforms-filesystems-paths-case-and-line-endings, qa-document-verification-spec-document-gates] +--- + +# Matching Non-ASCII Text with grep and Regex + +## When this applies + +You are writing a grep/regex pattern that must match non-ASCII text (Korean, +Japanese, accented Latin, emoji) in files, log lines, or file names; a pattern that +looks correct returns zero hits on text you can see on screen; a search works on one +machine and misses on another after the file crossed an OS, archive, or editor boundary. + +## Do this + +| Case | Do | +|------|----| +| Writing a literal pattern | Copy the exact substring out of the file rather than typing a stem you expect to be a prefix. In a precomposed script each syllable is a single code point: `아니` (`U+C544 U+B2C8`) is not contained in `아닌` (`U+C544 U+B2CC`), so a stem-prefix pattern returns 0 hits on a line that plainly contains the word | +| The word occurs in several inflected forms | Enumerate the forms as alternatives — `(아닌\|아니다\|아니라)` — one alternative per surface form present in the text | +| Pattern and data can come from different producers (editor, export, archive, another OS) | Normalize both sides to the same form before comparing (NFC for stored text), because `grep` matches code-unit sequences and applies no canonical equivalence: an NFC pattern scores 0 against the same word stored as NFD | +| The match result gates a build, release, or verification checklist | Assert an expected non-zero count rather than "no error", so a normalization or stem mistake surfaces as a failed count ([qa-document-verification-spec-document-gates]) | +| Comparing file names collected on macOS and Linux | Normalize both name lists in your code before diffing: APFS preserves the normalization it was given and looks up either form, HFS+ stores its own normalized form, and Linux filesystems store the bytes given — so the same name reaches your comparison in different forms | + +Measured on macOS 15 / APFS, 2026-07-30 (`grep` 2.6.0-FreeBSD, Python 3.13): + +``` +grep -c '아니' → 0 # same line contains 아닌 +grep -c '아닌' → 1 +grep -c -f → 0 # no normalization by grep +grep -c -f → 1 +len('아닌') NFC = 2 code points, NFD = 5 # jamo L+V+T decomposition +``` + +## Edge cases + +| Case | Then | +|------|------| +| The pattern must survive both normalization forms | Match on a substring that contains no combining sequence (an ASCII token, an id, a number), or normalize the input through a filter before grep | +| A pattern with a character class or quantifier over non-ASCII text | Test the exact pattern against a known-matching line first: BSD and GNU regex engines differ in multi-byte class handling, so a class that works on one userland can misfire on the other ([platforms-tools-bsd-vs-gnu-cli]) | +| Zero hits and the cause is unclear | Print the code points of both the pattern and the target line (`python3 -c "print([hex(ord(c)) for c in open(f).read()])"`) before concluding the text is missing — it separates "word absent" from "different code points" | +| The text is user-supplied and used as a key or a dedup identifier | Normalize to NFC at the trust boundary on write, so later equality and search compare one form | +| The search happens inside a database rather than a file | Normalization is applied by the writer, not the engine — same rule: normalize on write, search the stored form | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Write a stem prefix (`아니`) expecting it to match its inflections | Copy the literal form present in the text, or enumerate the alternatives | Precomposed syllables are distinct code points, so the stem is not a substring of the inflected word | +| Read a 0-hit grep as "the requirement is absent from the document" | Compare the code points of the pattern and the line before acting | 0 hits also means "different normalization form" or "different syllable" — an absence conclusion from that is a false negative | +| Compare two file-name lists byte-for-byte across machines | Normalize both lists to NFC in code, then diff | Producers store different forms of the same name; APFS lookup hides this locally but a byte diff does not | + +## Sources + +- https://unicode.org/reports/tr15/ — normalization forms; Hangul syllables have special full-decomposition rules; canonically equivalent strings have different binary representations unless normalized +- https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/ — §3.12 Conjoining Jamo Behavior: 11,172 precomposed Hangul syllables from `SBase = U+AC00` decompose algorithmically into L/V/T jamo +- https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html — APFS preserves the file name's normalization and is normalization-insensitive via hashes of the normalized form; HFS+ stores the normalized form +- https://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html — grep matches patterns against input lines by the specified regular-expression rules; no canonical-equivalence folding is specified diff --git a/wiki/platforms/filesystems/paths-case-and-line-endings.md b/wiki/platforms/filesystems/paths-case-and-line-endings.md index e9a209e..0cf95c1 100644 --- a/wiki/platforms/filesystems/paths-case-and-line-endings.md +++ b/wiki/platforms/filesystems/paths-case-and-line-endings.md @@ -9,7 +9,7 @@ sources: - https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file - https://git-scm.com/docs/gitattributes last_verified: 2026-07-10 -related: [platforms-shells-portable-shell-scripts] +related: [platforms-shells-portable-shell-scripts, platforms-environment-unicode-text-matching] --- # Files That Break When a Repo Moves Between macOS, Windows, and Linux diff --git a/wiki/platforms/index.md b/wiki/platforms/index.md index dccb400..b97fa71 100644 --- a/wiki/platforms/index.md +++ b/wiki/platforms/index.md @@ -3,8 +3,9 @@ Route here for: OS-level differences that break code and scripts moving between macOS, Linux, and Windows — shell portability, BSD-vs-GNU CLI flags, filesystem case/line-ending/path behavior, file permissions and exec bits across -git/archives/containers, hidden environment inputs (timezone/locale, per-context -PATH resolution), keeping processes alive as services or scheduled jobs, and +git/archives/containers, hidden environment inputs (timezone/locale, Unicode +normalization form in text and file names, per-context PATH resolution), keeping +processes alive as services or scheduled jobs, and pinning toolchain versions across machines. Application logic stays in backend; SQL stays in databases. @@ -15,6 +16,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [portable-shell-scripts](shells/portable-shell-scripts.md) | Writing a shell script that must run on more than one machine/OS/shell or in CI; a script that works locally fails elsewhere; choosing a shebang (bash vs sh); a bash script misbehaves in zsh or vice versa (unquoted vars, `=word`, array indexing); deciding how `set -euo pipefail` protects (and doesn't); building argument lists safely | +| [command-text-inspected-before-execution](shells/command-text-inspected-before-execution.md) | A hook, policy gate, allow-list, or audit rule blocked a command that is correct as written; composing a command that must satisfy such a gate first try; deciding whether to write a path literally or as `"$VAR"` in an inspected argument; a gate reports an argument missing or a file nonexistent though both are right; a gate must read a file your command creates; prose containing a dangerous-looking command (release notes, docs, fixtures) trips a text scanner | ## tools @@ -27,6 +29,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [timezone-and-locale](environment/timezone-and-locale.md) | Date/time or text-processing code behaves differently across machines (passes locally, fails in CI or vice versa); a cron/scheduled job fires at the wrong hour or double-fires/skips around DST; reviewing code that formats, parses, or compares dates or strings; writing tests that touch time; building case-insensitive keys, sorted output, or number parsing that must agree across machines | +| [unicode-text-matching](environment/unicode-text-matching.md) | A grep/regex pattern over non-ASCII text (Korean/Japanese/accented Latin/emoji) returns zero hits on text you can see; writing a pattern that must match an inflected or precomposed word; a search or name comparison works on one machine and misses after the file crossed an OS/archive/editor boundary; deciding where to normalize (NFC/NFD) user-supplied text used as a key | | [path-resolution](environment/path-resolution.md) | "command not found" though the tool is installed; a different version runs than the one installed; sudo/CI/cron/GUI apps/ssh can't find a command the interactive shell finds; two installations of the same tool conflict; deciding how a script should locate its correctness-critical tools | ## filesystems @@ -41,6 +44,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [background-services](processes/background-services.md) | Something must run persistently or on a schedule on a dev machine or server (daemon, watcher, cron-style job); a "started" process dies when the terminal/SSH/agent session ends; choosing nohup vs LaunchAgent vs systemd unit vs cron/timer; a job works in the terminal but fails under cron/launchd (minimal environment); wiring service logs and restart policy | +| [non-interactive-cli-invocation](processes/non-interactive-cli-invocation.md) | Calling a tool that can prompt (agent CLI, ssh, git, package manager) from a script, CI step, hook, or agent session, including with its own `-p`/`--print`/`--yes` flag; such a call produced no output and never returned; deciding whether a hang belongs to the client, the network, or the far-side service; choosing the stdin/timeout/fail-fast switches for an unattended call; a TTY-detecting tool changes its output format under automation | ## toolchains diff --git a/wiki/platforms/processes/background-services.md b/wiki/platforms/processes/background-services.md index f6e9503..22b1e9a 100644 --- a/wiki/platforms/processes/background-services.md +++ b/wiki/platforms/processes/background-services.md @@ -11,7 +11,7 @@ sources: - https://man7.org/linux/man-pages/man1/nohup.1.html - https://man7.org/linux/man-pages/man1/loginctl.1.html last_verified: 2026-07-10 -related: [platforms-toolchains-version-management, platforms-shells-portable-shell-scripts] +related: [platforms-toolchains-version-management, platforms-shells-portable-shell-scripts, platforms-processes-non-interactive-cli-invocation] --- # Keeping a Process Running Beyond the Terminal Session diff --git a/wiki/platforms/processes/non-interactive-cli-invocation.md b/wiki/platforms/processes/non-interactive-cli-invocation.md new file mode 100644 index 0000000..da7927c --- /dev/null +++ b/wiki/platforms/processes/non-interactive-cli-invocation.md @@ -0,0 +1,83 @@ +--- +id: platforms-processes-non-interactive-cli-invocation +domain: platforms +category: processes +applies_to: [macos, linux] +confidence: verified +sources: + - https://www.gnu.org/software/coreutils/manual/html_node/nohup-invocation.html + - https://man.openbsd.org/ssh + - https://man.openbsd.org/ssh_config + - https://git-scm.com/docs/git + - https://man7.org/linux/man-pages/man1/timeout.1.html +last_verified: 2026-07-31 +related: [platforms-processes-background-services, platforms-tools-bsd-vs-gnu-cli, platforms-shells-portable-shell-scripts, debugging-methodology-hypothesis-testing] +--- + +# Invoking a Prompt-Capable CLI from a Script or Agent Harness + +## When this applies + +Calling a CLI that is able to prompt (agent CLIs, `ssh`, `git`, package managers) +from a script, CI step, hook, or agent harness — including when you passed its own +non-interactive flag (`-p`, `--print`, `--yes`). Also when such a call hangs with no +output and no error and you must decide whether the client, the network, or the +remote service is at fault. + +## Do this + +1. **Detach fd 0 at the call site**: `cmd /dev/null >"$LOG" 2>&1 &` — the stdin redirect is a GNU extension | +| Tool detects a TTY and changes output (color codes, progress bars, a pager) | Closing stdin is not enough — also disable the pager/color (`GIT_PAGER=cat`, `--no-color`) so parsers see stable text | +| Wrapper CLI shells out to a second binary that prompts | Redirecting the wrapper's stdin covers the child only if the child inherits it — confirm with a timeout run before trusting the wrapper's own flags | +| Still hangs with stdin closed and nothing in the server log | Look for a non-stdin block: a lockfile, a keychain/credential prompt that only resolves in a GUI session, or a missing binary ([platforms-environment-path-resolution]) | +| The process must outlive the session | [platforms-processes-background-services] owns detach/supervision choice | +| Output arrives only after the process exits | The tool buffers when fd 1 is not a TTY — read the log after exit, or use the tool's line-buffered/streaming flag | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Raise the client timeout because the call "is just slow" | Re-run with `/dev/null`. The terminal precondition is why the explicit redirect is the guarantee +- https://man.openbsd.org/ssh — `-n` "Redirects stdin from /dev/null (actually, prevents reading from stdin). This must be used when ssh is run in the background"; the same entry notes it "does not work if ssh needs to ask for a password or passphrase" — a stdin detach, not a prompt suppressor +- https://man.openbsd.org/ssh_config — `BatchMode=yes`: "user interaction such as password prompts and host key confirmation requests will be disabled"; `StrictHostKeyChecking=accept-new` adds new host keys without permitting changed ones +- https://git-scm.com/docs/git — `GIT_TERMINAL_PROMPT`: "If this Boolean environment variable is set to false, git will not prompt on the terminal (e.g., when asking for HTTP authentication)" +- https://man7.org/linux/man-pages/man1/timeout.1.html — "Start COMMAND, and kill it if still running after DURATION"; exit status 124 "if COMMAND times out" +- Field context: a 2026-07 session ran an agent CLI with its non-interactive `--tools` flag in the foreground; two runs hung (300 s, 150 s) with zero output. The gateway access log showed **no request from that host** in either window (ruling out the model and gateway); the same command with stdin taken from `/dev/null` completed immediately diff --git a/wiki/platforms/shells/command-text-inspected-before-execution.md b/wiki/platforms/shells/command-text-inspected-before-execution.md new file mode 100644 index 0000000..e69daff --- /dev/null +++ b/wiki/platforms/shells/command-text-inspected-before-execution.md @@ -0,0 +1,94 @@ +--- +id: platforms-shells-command-text-inspected-before-execution +domain: platforms +category: shells +applies_to: [bash, zsh, posix-sh] +confidence: verified +sources: + - https://code.claude.com/docs/en/hooks + - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html +last_verified: 2026-07-30 +related: [platforms-shells-portable-shell-scripts, platforms-environment-path-resolution] +--- + +# Commands Read as Text by a Gate Before the Shell Runs Them + +## When this applies + +A hook, policy gate, allow-list, or audit rule inspects your command line and +decides whether it may run (agent PreToolUse hook, commit gate, sudo command +pattern, CI policy check); the gate blocked or mis-parsed a command that is +correct as written; or you are composing a command that must satisfy such a gate +on the first attempt. + +## Do this + +1. **Write values the gate reads as literal text.** The gate is handed the command + as an unexecuted string — a Claude Code `PreToolUse` hook receives + `tool_input.command` (`{"tool_name":"Bash","tool_input":{"command":"npm test"}}`) + and runs "Before a tool call executes". Nothing has expanded yet: `"$VAR"` is + four literal characters plus a name, and the quote marks are part of the text. + Shell expansion is defined to happen when the shell processes the line, which is + after the gate has already decided. + +2. **Know which of the two failure modes you are in — the error message tells you.** + Both a quoted path and an unexpanded variable break a gate that extracts a file + argument, but for different reasons and with different symptoms: + +| What you wrote | What the gate extracts | How it fails | +|----------------|------------------------|--------------| +| `--body-file /abs/path/REPORT.md` | `/abs/path/REPORT.md` | Passes | +| `--body-file="/abs/path/REPORT.md"` or `--body-file "$REPO/REPORT.md"` | nothing — the extraction pattern excludes the quote character, so the match fails outright | Gate reports the argument as **missing** ("no `--body-file` found"), which reads as a malformed command | +| `--body-file $REPO/REPORT.md` (unquoted variable) | the literal string `$REPO/REPORT.md` | Gate reports the file as **nonexistent**, which reads as a missing deliverable | + +3. **Create the file a gate will read in an earlier, separate command.** The gate + runs before this command executes, so a file produced by a heredoc inside the + same command does not exist yet at inspection time and the gate fails closed. + Write the file in one call, reference it by literal path in the next. + +4. **When content must contain patterns the gate treats as dangerous, put the + content in a file with a non-shell tool and pass the path.** Release notes, + docs, or fixtures containing `curl … | sh` or `rm -rf` are data, but a + text-scanning gate cannot tell data from an invocation. A file written by an + editor/Write tool is never scanned as a command; `--notes-file` / `--body-file` + then carries it. + +5. **Read the gate's own extraction pattern when a correct command is refused.** + The pattern is the specification of what the gate can see. Reproduce it against + your exact command string before rewriting anything else — one run tells you + whether you are in the missing-argument or nonexistent-file mode above. + +## Edge cases + +| Case | Then | +|------|------| +| Blocking feedback appears without the gate's message | Exit code 2 sends the reason to **stderr**, not stdout; read stderr for the actual cause | +| The gate matches an intended-as-prose mention of a dangerous command (in a commit message, doc, or test fixture) | Move the text into a file and pass it by path (step 4) rather than reshaping the sentence | +| Path contains a space, so quoting is unavoidable | Relocate or symlink the target to a space-free path for gated commands; a gate that excludes quote characters cannot receive a quoted path at all | +| The gate needs `~` expanded | Write the absolute path; a gate that resolves `~` itself is doing so on the literal tilde, which only works if it implements the expansion | +| The same command must also be portable/robust as a script | Keep the gate-read argument literal and leave the rest of the script quoted normally ([platforms-shells-portable-shell-scripts]) — this page narrows one argument, it does not license unquoted expansions elsewhere | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Interpolate `"$VAR/file"` into an argument a gate inspects | Write the resolved absolute path literally | The gate sees pre-expansion text; the quote character can defeat its extractor entirely and the variable name never resolves for it | +| Assume a blocked command means the deliverable is wrong | Reproduce the gate's extraction pattern against your literal command string first | A quoting-level extraction failure and a genuinely incomplete deliverable produce the same refusal, so fixing content wastes the round | +| Build the file the gate checks with a heredoc in the same command | Write it in a prior command and reference the path | The gate is evaluated before execution, so the file is absent at decision time | +| Reword prose to get a dangerous-looking string past a scanner | Put the prose in a file and pass `--notes-file`/`--body-file` | Editing meaning to satisfy a text scanner degrades the artifact; a file is not scanned as a command | + +## Sources + +- https://code.claude.com/docs/en/hooks — `PreToolUse` runs "Before a tool call executes. Can block it"; the hook's stdin JSON carries `tool_input.command` — the unexecuted Bash command string. Exit 2 blocks and "stderr text is fed back to Claude as an error message" +- https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html — the shell's order of word expansion (tilde, parameter, command substitution, field splitting, quote removal) is performed by the shell as it processes the command, so an external reader of the command text sees none of it applied + +## Field context + +Reproduced against the extraction pattern of this repo's own flush gate +(``--body-file[= ]+[^ '"`]+``, `hooks/pre-flush-pr-gate.sh`) on 2026-07-30: five +variants run through that pattern gave `--body-file "$REPO/…"` → empty (blocked as +missing), `--body-file $REPO/…` → literal `$REPO/…` (blocked as nonexistent), +`--body-file "/abs/…"` → empty even with a literal path, while +`--body-file /abs/…` and `--body-file=/abs/…` extracted correctly. A same-command +heredoc body-file was separately blocked as not-yet-existing until moved to a +preceding call. diff --git a/wiki/platforms/shells/portable-shell-scripts.md b/wiki/platforms/shells/portable-shell-scripts.md index dcab402..23eed9b 100644 --- a/wiki/platforms/shells/portable-shell-scripts.md +++ b/wiki/platforms/shells/portable-shell-scripts.md @@ -11,7 +11,7 @@ sources: - https://google.github.io/styleguide/shellguide.html - https://www.shellcheck.net/ last_verified: 2026-07-10 -related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management] +related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management, platforms-shells-command-text-inspected-before-execution] --- # Shell Scripts That Must Run on More Than One Machine or Shell diff --git a/wiki/qa/document-verification/editing-a-gated-document.md b/wiki/qa/document-verification/editing-a-gated-document.md new file mode 100644 index 0000000..a3e0387 --- /dev/null +++ b/wiki/qa/document-verification/editing-a-gated-document.md @@ -0,0 +1,107 @@ +--- +id: qa-document-verification-editing-a-gated-document +domain: qa +category: document-verification +applies_to: [general] +confidence: field-tested +sources: + - https://man7.org/linux/man-pages/man1/pgrep.1.html + - https://docs.vale.sh/topics/scopes.md + - https://docs.vale.sh/checks/existence + - https://github.com/DavidAnson/markdownlint/blob/main/doc/md013.md +last_verified: 2026-07-30 +related: [qa-process-acceptance-criteria, qa-process-regression-scope, testing-quality-tests-that-cannot-fail] +--- + +# Editing a Document That Automated Text Gates Check + +## When this applies + +You are adding to or rewording a document (RFC, spec, plan, audit record) that +grep/regex gates or a lint config run over; you are recording an audit result +*inside* the document the audit examined; or a check that passed before your edit +now fails on wording whose meaning did not change. + +## Do this + +1. **Inventory the anchors before editing.** Grep the gate definitions for patterns + naming this file and treat every hit as a constraint on the edit. Anchors come in + kinds that break differently: + +| Anchor kind | Example | How an edit breaks it | +|-------------|---------|-----------------------| +| Phrase or vocabulary | a pattern requiring/forbidding certain verbs near a term | Reworded sentence stops matching, or starts matching a forbidden pattern | +| Line or row count | "file has 444 lines", "table has 5 rows" | Any insertion shifts the count | +| Substring | a section title or term quoted verbatim | Retitling or translating the term silently drops it | +| Quoted original | a check that requires an upstream sentence to appear verbatim | Paraphrasing removes it while preserving meaning | + +2. **State facts about an upstream document as observations, not definitions.** A + lexical gate that means "do not redefine the upstream contract" can only + approximate that with vocabulary: Vale's `existence` check "looks for the + 'existence' of particular tokens" and evaluates no meaning. So a true, purely + descriptive sentence trips it when it uses a defining verb. Write the observable + shape instead — "the contract table has two rows, `arena` and `pool`, and no + `heap` row" — which carries the same information and matches no definition + pattern. + +3. **Scope a check outside the region that quotes it.** When the document contains + the check command or its regex, the pattern matches its own quotation. Bound the + judgment to the normative region (`awk '/^### /{exit}'`, a line + range, or a section selector) rather than running it over the whole file. This is + standard practice in the tools themselves: Vale scopes are markup-aware and "Any + scope prefaced with `~` is negated", and markdownlint rules take + `code_blocks: false` to exclude quoted code from prose rules. `pgrep` hard-codes + the same defense — "The running pgrep, pkill, or pidwait process will never + report itself as a match." + +4. **Record verdicts as scoped conditions, never as a global count.** Write "0 + matches outside §Audit" or "every match is inside a fenced block", not "1 match + found". A fixed number is invalidated by the next edit — including the edit that + quotes the finding — so a count-based verdict decays into a false statement while + looking precise. + +5. **Re-run the whole gate set after the edit and compare against the pre-edit + pass count.** Your edit can break a check owned by a different section or a + different task ([qa-process-regression-scope]). A baseline number (`60 → 61`) is + what distinguishes "my change fixed one" from "my change fixed one and broke two". + +## Edge cases + +| Case | Then | +|------|------| +| The quoted pattern must stay in the document (it is the audit record) | Keep the quote and narrow the check's scope; the document's job is to be readable, the check's job is to be scoped | +| Fixing the report by quoting the offending match adds another match | Stop counting globally and switch to the scoped condition in step 4 — each quote-to-fix round otherwise raises the count and re-falsifies the statement | +| Gate anchors on a line count you must change | Update the gate and the document in the same commit, and say so in the PR ([qa-process-acceptance-criteria]) | +| The failure surfaces in another task's or another agent's check log | Attribute before repairing: identify which file the failing pattern targets, since a cross-file gate makes your edit look like their regression | +| The gate is genuinely wrong (it forbids a correct sentence with no replacement available) | Change the gate, with a control proving it still catches the defect it owns — do not reword a correct document into a worse one | +| You are authoring the gate rather than the document | This page covers the author's side; gate construction and its controls are a separate concern → [testing-quality-tests-that-cannot-fail] | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Edit a gated document and run only the check you were fixing | Grep the gates for anchors on that file first, then re-run the full set against a baseline count | Line-count and substring anchors break silently, and the breakage lands in a check you never looked at | +| Write "X defines/mandates two variants" when describing an upstream contract | Write the observable shape: "the table has two rows and no Y row" | A vocabulary gate cannot separate describing from redefining, so the defining verb fails a sentence whose content is correct | +| Run an audit pattern over the whole document that quotes it | Bound it to the normative region by heading range or line range | The quotation is a match, so the whole-file run measures the document's prose about itself | +| Record "grep found 0 hits" as the audit result | Record the scoped condition that must hold ("no hits outside §Audit") | The bare count is true only for the file revision that produced it and silently becomes false on the next edit | + +## Sources + +- https://docs.vale.sh/checks/existence — the check "looks for the 'existence' of particular tokens", transformed into a word-bounded non-capturing group: a lexical gate matches patterns, not intent +- https://docs.vale.sh/topics/scopes.md — scopes restrict where a rule applies via markup-aware selectors; "Any scope prefaced with `~` is negated" and scopes can be chained, so checks can be kept off regions like code examples +- https://github.com/DavidAnson/markdownlint/blob/main/doc/md013.md — rules expose `code_blocks`, `tables`, `headings` booleans (default `true`) so quoted code can be excluded from a prose rule +- https://man7.org/linux/man-pages/man1/pgrep.1.html — "The running pgrep, pkill, or pidwait process will never report itself as a match" — self-exclusion is designed in because self-matching is the expected failure + +## Field context + +Distilled from 2026-07 RFC/plan-authoring sessions in this repo. A vague-word audit +of `docs/ROADMAP.md` reported 1 global hit — the audit command's own quoted pattern — +and quoting the finding to fix it raised the count to 3, while an `awk`-scoped run +over the normative region reported 0; the same self-reference appeared twice more +(a negative-control example that always matched, and a `docs/*.md` glob matching its +own document 45 times). Separately, adding the sentence "defines only the two +variants arena and pool" to an RFC failed a sibling task's vocabulary gate +(`(arena|pool)…(재정의|정의한다|규정한다)`); rephrasing to "the contract table has +two rows and no heap row" restored the suite from 60 to 61 passing, and a pre-edit +anchor survey of that file found three further anchors — a line count, a section +substring, and a verbatim upstream sentence — all preserved by the same edit. diff --git a/wiki/qa/document-verification/spec-document-gates.md b/wiki/qa/document-verification/spec-document-gates.md new file mode 100644 index 0000000..c2e3959 --- /dev/null +++ b/wiki/qa/document-verification/spec-document-gates.md @@ -0,0 +1,95 @@ +--- +id: qa-document-verification-spec-document-gates +domain: qa +category: document-verification +applies_to: [general] +confidence: field-tested +sources: + - https://eslint.org/docs/latest/extend/custom-rule-tutorial + - https://eslint.org/docs/latest/integrate/nodejs-api + - https://testing.googleblog.com/2021/04/mutation-testing.html + - https://www.rfc-editor.org/rfc/rfc2119 + - https://docs.vale.sh/checks/conditional + - https://docs.vale.sh/checks/occurrence + - https://github.com/DavidAnson/markdownlint/blob/main/doc/md056.md + - https://github.com/DavidAnson/markdownlint/issues/1206 +last_verified: 2026-07-30 +related: [qa-process-acceptance-criteria, testing-quality-tests-that-cannot-fail, platforms-environment-unicode-text-matching] +--- + +# Automated Gates on a Specification Document + +## When this applies + +You are writing or reviewing automated checks (grep/script) that decide whether a +written deliverable — RFC, API spec, schema doc, design doc — satisfies its stated +requirements; a document passed its checklist but a reviewer still found the +requirement unmet; you are fixing gate patterns for a document that is not written yet. + +## Do this + +1. **Prove the gate on a conforming sample before adopting it.** Run each pattern + against a sibling document that already satisfies the same spec and require the + exact expected count (`7` sections, `2` signature mentions). A run against the + not-yet-written target only shows the file is absent — a typo'd or mis-anchored + pattern produces the identical result. ESLint's `RuleTester` encodes this pairing: + it "requires that at least one valid and one invalid test scenario be present." +2. **Prove the gate on a mutated copy.** Plant the exact defect the gate claims to + catch, require FAIL, and keep the mutant as a permanent self-test — this is + mutation testing applied to the checker instead of the code. +3. **Check on four axes.** Token existence alone passes documents that violate the spec: + +| Axis | What the check does | Defect that a token-existence check misses | +|------|---------------------|--------------------------------------------| +| Structure | Parse the table and assert rows, columns, and non-empty cells | The whole table is deleted while the token survives in a nearby paragraph | +| Modality and polarity | Within one sentence scope, assert the requirement is neither negated nor demoted (MUST→SHOULD, 필수→권장/원칙적으로) | "X is not required" and "X is recommended" both contain every keyword | +| Set completeness | Assert the exact member count of a closed set (`enum has exactly 5 rows`) | One enum row is deleted; the token count is still ≥ 1 | +| Cross-reference | Assert that a statement in one section implies its counterpart elsewhere, and recompute a derived value from its inputs | Two sections disagree, or an `Examples` block silently stands in for the deleted normative rule | + +4. **Fail closed when the anchor is missing.** When the section heading, table, or + derivation input a check needs cannot be located, report FAIL. A check that + reports "nothing to check" disappears the moment someone rewords the sentence it + keyed on. +5. **Write patterns against the literal text in the document**, not a remembered + stem — for non-ASCII text apply [platforms-environment-unicode-text-matching]. + +## Edge cases + +| Case | Then | +|------|------| +| No conforming sibling exists (first document of its kind) | Author a minimal conforming fixture, run the gate against it, require PASS, and keep the fixture next to the gate | +| The gate must exist before the document (plan-first workflow) | Take the positive control from the sibling or fixture; treat the target's failing run as evidence of absence only | +| The `Examples` section satisfies the check while the normative section does not | Scope the check to the normative section (heading range), so examples cannot stand in for the rule | +| The check counts delimiters (`\|`) as a stand-in for parsing | Use a Markdown parser — markdownlint's own MD056 misreports when a pipe appears inside backticks (issue #1206) | +| The document deliberately relaxes a requirement | Change the gate and the acceptance criteria in the same commit, and record the relaxation in the PR ([qa-process-acceptance-criteria]) | +| The deliverable is code, not a document | Apply [testing-quality-tests-that-cannot-fail] — same red-run proof, expressed as tests | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Adopt a pattern because it exits non-zero against the unwritten target | Run it against a conforming sibling and require the expected count | An absent file fails every pattern; the red run cannot distinguish a correct pattern from a mistyped one | +| Treat "the keyword is present" as the requirement being met | Add the sentence-scoped polarity and modality check | RFC 2119 makes MUST and SHOULD different requirement levels, so a demotion keeps every keyword while dropping the obligation | +| Assert a token appears at least once for a closed enum | Parse the table and assert the exact row count | Deleting one member leaves the count ≥ 1, so the gate stays green on an incomplete set | +| Skip a check whose anchor sentence was not found | Report FAIL and name the missing anchor | A skipped check is indistinguishable from a passed one in the summary line | +| Verify a cross-section value by matching the number as written | Recompute it from its inputs and compare | Matching the written number passes when both sections were edited to the same wrong value | + +## Sources + +- https://eslint.org/docs/latest/extend/custom-rule-tutorial — "RuleTester requires that at least one valid and one invalid test scenario be present" — a checker is proved by code that must pass plus code that must fail +- https://eslint.org/docs/latest/integrate/nodejs-api — `RuleTester#run()` takes `valid` and `invalid` case arrays; invalid cases assert the expected error count +- https://testing.googleblog.com/2021/04/mutation-testing.html — inserting faults and requiring failure measures whether checks detect defects; coverage alone does not +- https://www.rfc-editor.org/rfc/rfc2119 — MUST is "an absolute requirement"; SHOULD permits ignoring the item with justification — distinct requirement levels behind the same vocabulary +- https://docs.vale.sh/checks/conditional — "Ensures that the existence of 'first' implies the existence of 'second'" — the cross-reference axis as a first-class check +- https://docs.vale.sh/checks/occurrence — enforces the minimum/maximum number of times a token appears "in a given scope" — count and scope, not bare existence +- https://github.com/DavidAnson/markdownlint/blob/main/doc/md056.md — MD056 flags tables whose rows disagree with the header's column count (structural table checking) +- https://github.com/DavidAnson/markdownlint/issues/1206 — MD056 counts pipes inside backticks as separators: a delimiter count is not a parse + +## Field context + +Distilled from RFC-authoring sessions in this repo (2026-07): an independent +auditor built 8 tampered copies of an RFC that a 16-gate token-existence +checklist accepted — 5 passed all 16 gates, including a copy whose 5-row `type` +enum had a row removed. After the four axes above plus fail-closed anchors were +added, all 32 tampered copies failed their designated check while the intact +document passed 62/62. diff --git a/wiki/qa/index.md b/wiki/qa/index.md index 92b7105..c80140c 100644 --- a/wiki/qa/index.md +++ b/wiki/qa/index.md @@ -2,7 +2,8 @@ Route here for: release-quality process — acceptance criteria, gates, regression scoping, test-environment parity, post-release verification, bug -reports, manual/exploratory testing. Writing automated test code → wiki/testing/; +reports, manual/exploratory testing, and automated verification of document +deliverables (specs/RFCs). Writing automated test code → wiki/testing/; rollout/canary/rollback mechanics → wiki/infrastructure/. Match your situation to a "load when" line; load only matching pages. @@ -17,6 +18,13 @@ Match your situation to a "load when" line; load only matching pages. | [severity-and-priority](process/severity-and-priority.md) | Triaging a bug — deciding how bad it is and when it gets fixed; a triage stalled on a severity debate | | [post-release-verification](process/post-release-verification.md) | A release just deployed to production; defining what "released safely" means; an incident revealed a release was broken for hours before anyone noticed | +## document-verification + +| Page | Load when | +|------|-----------| +| [spec-document-gates](document-verification/spec-document-gates.md) | Writing or reviewing automated checks (grep/script) that decide whether a spec/RFC/schema document meets its requirements; a document passed its checklist but the requirement is still unmet; choosing what a doc gate must assert beyond keyword presence (table structure, MUST-vs-SHOULD demotion, closed-set completeness, cross-section consistency); validating a gate pattern for a document that does not exist yet | +| [editing-a-gated-document](document-verification/editing-a-gated-document.md) | Editing or rewording a document that grep/regex gates or a lint config check; a gate fails on wording whose meaning did not change; describing what an upstream spec says without tripping a "do not redefine it" gate; a check matches the pattern your own document quotes; recording an audit verdict inside the document that was audited; deciding which checks to re-run after editing a gated document | + ## environments | Page | Load when | diff --git a/wiki/qa/process/acceptance-criteria.md b/wiki/qa/process/acceptance-criteria.md index c1704c5..d72562b 100644 --- a/wiki/qa/process/acceptance-criteria.md +++ b/wiki/qa/process/acceptance-criteria.md @@ -10,7 +10,7 @@ sources: - https://www.agilealliance.org/glossary/acceptance/ - https://www.agilealliance.org/glossary/definition-of-ready/ last_verified: 2026-07-10 -related: [qa-process-regression-scope, qa-process-release-gates] +related: [qa-process-regression-scope, qa-process-release-gates, qa-document-verification-spec-document-gates] --- # Writing Acceptance Criteria That Settle "Done" Before Development diff --git a/wiki/qa/process/release-gates.md b/wiki/qa/process/release-gates.md index 332660e..015c488 100644 --- a/wiki/qa/process/release-gates.md +++ b/wiki/qa/process/release-gates.md @@ -7,7 +7,7 @@ confidence: field-tested sources: - https://sre.google/sre-book/release-engineering/ last_verified: 2026-07-10 -related: [qa-process-regression-scope, qa-exploratory-exploratory-sessions] +related: [qa-process-regression-scope, qa-exploratory-exploratory-sessions, backend-common-integrations-externally-owned-defaults] --- # Deciding Whether a Build Is Ready to Ship diff --git a/wiki/testing/index.md b/wiki/testing/index.md index 78418d7..2cbad76 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -2,7 +2,8 @@ Route here for: writing or structuring automated tests — choosing the test level, selecting cases and assertions, test data and isolation, mock/fake -decisions, fixing flaky tests, verifying tests can actually fail, testing +decisions, fixing flaky tests, verifying tests can actually fail, validating a +check before its target exists, testing async code (promises/timers/events), and browser E2E selector/wait/setup strategy. Release-process quality (gates, manual testing, bug triage) → wiki/qa/. @@ -22,6 +23,9 @@ Match your situation to a "load when" line; load only matching pages. | [minimum-case-set](quality/minimum-case-set.md) | Writing tests for a function/endpoint/change and choosing which cases to cover; reviewing whether coverage suffices; picking boundary values by input type; adding a regression test for a bug fix | | [behavior-not-implementation](quality/behavior-not-implementation.md) | Deciding what a test should assert; a behavior-preserving refactor broke tests; tempted to expose privates for testing; deciding whether a snapshot test is appropriate | | [tests-that-cannot-fail](quality/tests-that-cannot-fail.md) | Reviewing tests that always pass; a bug shipped through an area the suite reported as covered; auditing a suspiciously green suite; judging whether an assertion, error-path test, or mock-based test can actually detect a defect | +| [checks-that-cannot-pass](quality/checks-that-cannot-pass.md) | Authoring a check whose target does not exist yet (grep/regex gate on an unwritten file or doc section, lint/scan rule, schema assertion on an unbuilt endpoint, a plan's verification command) and it has only ever been observed failing; reviewing a plan's gates before adopting them; separating "target missing" from "content missing" in a gate's exit status | +| [spec-artifact-checks](quality/spec-artifact-checks.md) | Writing or reviewing an automated check that a mapping table covers every rule/field/enum case, or that ids resolve across documents; deciding whether a green check earned "verified" or only "present"; designing one negative control per check in a multi-check harness; parsing Markdown table rows programmatically in a doc-as-spec repo | +| [harness-reverse-controls](quality/harness-reverse-controls.md) | You built a harness that scores how well something is verified (mutation run, doc/spec gate suite, CI check matrix) and are about to cite its score in a commit, PR, README, or report; its verdicts come out uniform (every case caught, or every case green); deciding what control run proves the harness discriminates, how to score errored/never-ran cases, and what the harness's isolated working tree must contain | ## data diff --git a/wiki/testing/quality/checks-that-cannot-pass.md b/wiki/testing/quality/checks-that-cannot-pass.md new file mode 100644 index 0000000..e371894 --- /dev/null +++ b/wiki/testing/quality/checks-that-cannot-pass.md @@ -0,0 +1,96 @@ +--- +id: testing-quality-checks-that-cannot-pass +domain: testing +category: quality +applies_to: [general] +confidence: verified +sources: + - https://www.jamesshore.com/v2/books/aoad2/test-driven_development + - https://pubs.opengroup.org/onlinepubs/9799919799/utilities/grep.html + - https://docs.semgrep.dev/writing-rules/testing-rules + - https://docs.pytest.org/en/stable/reference/exit-codes.html +last_verified: 2026-07-29 +related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set] +--- + +# Validating a Check Whose Target Does Not Exist Yet + +## When this applies + +You are authoring a check that will decide pass/fail for work not yet done — +a grep/regex gate on an unwritten file or doc section, a lint/scan rule, a +schema assertion against an unbuilt endpoint, a plan's verification command — +and the only run you have observed is a failing one. Also applies when +reviewing a plan whose gates have never been seen to pass. + +## Do this + +1. **Run the check against a known-good input before adopting it**, and require + the exact expected result. A failing run against an absent target is produced + by every pattern, correct or mistyped alike, so it is not evidence the check + is right. The known-good input removes the absence variable and tests only + the check. + +| What the check is | Known-good input to run it against | Result to require | +|-------------------|------------------------------------|-------------------| +| Regex/grep gate on a file or section not yet written | An existing sibling artifact built to the same template or spec | The exact count (`grep -c` = the number the spec implies), not merely "nonzero" | +| Lint/scan rule (ESLint, Semgrep, custom AST rule) | One fixture that must match plus one that must not | Match on the first, silence on the second — both directions, per Semgrep's `ruleid:`/`ok:` split | +| Schema/contract assertion on an unbuilt endpoint | A shipped endpoint with the same response envelope | Assertion passes unmodified | +| Cross-artifact signature check (doc A must quote the signature in doc B) | Both existing documents of the same family | The expected match count in each | + +2. **Predict the failure mode before running the check, then compare.** State the + expected exit status and message. When the observed failure differs from the + prediction, the check is what is unknown — fix it before it becomes a gate. + +3. **Make "target missing" a distinct outcome from "content missing."** POSIX + `grep` already separates them; keep that distinction instead of collapsing it + into a boolean: + +| Outcome | POSIX grep exit | What it means for the gate | +|---------|-----------------|----------------------------| +| One or more lines selected | 0 | The check passes | +| No lines selected | 1 | Target readable; content absent **or** pattern wrong — indistinguishable | +| Error (unreadable path, invalid regex) | >1 (2 in practice) | The check could not run at all | + +4. **Give each of the three outcomes its own exit code and message**, so the + gate's failure names its own cause. Keep grep's status before defaulting the + count — `|| n=0` alone reports an invalid pattern as a count mismatch, which + is the conflation this page exists to prevent: + +```sh +[ -f "$f" ] || { echo "gate: target missing: $f" >&2; exit 3; } +n=$(grep -cE "$pat" "$f" 2>/dev/null); rc=$? +[ "$rc" -le 1 ] || { echo "gate: check could not run on $f (grep exit $rc)" >&2; exit 4; } +[ "${n:-0}" -eq "$expected" ] || { echo "gate: $f matched ${n:-0}, expected $expected" >&2; exit 1; } +``` + +Verified 2026-07-29 against four inputs: known-good file + correct pattern → 0; +known-good file + wrong anchor → 1; missing target → 3; invalid regex → 4. + +## Edge cases + +| Case | Then | +|------|------| +| No sibling artifact exists (first of its kind) | Hand-write a throwaway stub that satisfies the spec, run the check against it, require the expected result, delete the stub. The stub is the positive control | +| The gate globs paths that include the not-yet-created target, using `-q` | Drop `-q` and check each path separately: with `-q`, exit status is 0 when a line is selected **even if an error occurred** (POSIX), so a matching sibling hides the missing target. Measured 2026-07-29 (BSD grep 2.6.0-FreeBSD, ugrep 7.5.0): `grep -q pat missing.md matching.md` → exit 0 | +| The check's output is piped (`grep -c … \| tail -1`) | Set `set -o pipefail`, or capture into a variable and compare. Without it the pipeline reports the last command's status, so grep's error 2 becomes exit 0 — the absent target reads as a pass. Measured on the same runs | +| A count is compared without a value (`[ "$n" -eq 7 ]`, `$n` empty) | Default the capture (`n=${n:-0}`) after the existence guard: `grep -c` on an unreadable path writes nothing to stdout, and the empty comparison raises a shell error whose message hides the real cause | +| The check is a test for behavior you are about to implement | Red is the expected state; require that it fails with the assertion the behavior owns, not with a collection/import error. `pytest` exit 5 means "no tests were collected" — a selector typo, not a failing test | +| The check is already adopted and has never been observed passing | Run it against a known-good input now; when the expected result does not appear, treat the gate as defective rather than the work as incomplete | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Adopt a gate after seeing it fail on the missing target | Run it on an artifact that already satisfies the spec and require the exact expected count | Absence makes every pattern fail; the failure proves the target is missing, not that the pattern is correct | +| Write the gate as `grep -q pat docs/*.md` | Assert the specific path exists, then match it with an exact count | `-q` returns 0 on a match even when another path errored, so a sibling's match masks the missing target | +| Treat any nonzero exit as "not done yet" | Branch on 1 versus >1: content-absent versus check-could-not-run | An invalid regex and an unwritten file give different statuses; conflating them leaves a mistyped gate failing forever after the work is complete | +| Leave the expected failure unstated and just run the check | Predict the exit status and message first, then compare | An unpredicted failure that matches nothing you expected means the check, not the code, is the unknown | + +## Sources + +- https://www.jamesshore.com/v2/books/aoad2/test-driven_development — "Don't just predict that it will fail, though; predict *how* it will fail"; if it "fails in a different way than you expected, you're no longer in control of your code" +- https://pubs.opengroup.org/onlinepubs/9799919799/utilities/grep.html — EXIT STATUS 0 = lines selected, 1 = no lines selected, >1 = an error occurred; with `-q` "the exit status shall be zero if an input line is selected, even if an error was detected" +- https://docs.semgrep.dev/writing-rules/testing-rules — rule tests annotate `ruleid:` lines "for protecting against false negatives" and `ok:` lines "for protecting against false positives"; a rule is validated against inputs that must match and inputs that must not +- https://docs.pytest.org/en/stable/reference/exit-codes.html — exit code 5 = "No tests were collected", distinct from 1 = tests ran and failed +- Field reproduction 2026-07-29 (BSD grep 2.6.0-FreeBSD, ugrep 7.5.0, macOS): missing path → exit 2; wrong-but-valid pattern on a present file → exit 1; `grep -q` over missing + matching paths → exit 0; unpiped `grep -c missing | tail -1` → exit 0 without `pipefail` diff --git a/wiki/testing/quality/harness-reverse-controls.md b/wiki/testing/quality/harness-reverse-controls.md new file mode 100644 index 0000000..088268a --- /dev/null +++ b/wiki/testing/quality/harness-reverse-controls.md @@ -0,0 +1,98 @@ +--- +id: testing-quality-harness-reverse-controls +domain: testing +category: quality +applies_to: [general] +confidence: verified +sources: + - https://pitest.org/quickstart/basic_concepts/ + - https://stryker-mutator.io/docs/stryker-js/configuration/ + - https://stryker-mutator.io/docs/stryker-js/troubleshooting/ + - https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/ + - https://stryker-mutator.io/docs/mutation-testing-elements/equivalent-mutants/ + - https://testing.googleblog.com/2021/04/mutation-testing.html +last_verified: 2026-08-02 +related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set] +--- + +# Citing a Verification Harness's Own Score + +## When this applies + +You built a harness that reports how well something is verified — a mutation run, a +doc/spec gate suite, a matrix of CI checks — and you are about to cite its score as +evidence in a commit message, PR body, README, or report. Also when its verdicts +come out uniform across every case: every mutant caught, or every one surviving. + +## Do this + +1. **Run a case whose correct verdict is the opposite of the failure you are + hunting, and require that verdict, before citing any score.** For a mutation + harness the control is a semantics-preserving change (reformat, rename a local, + edit a comment or docstring). That is an *equivalent mutation* — PIT's term for a + mutant that "behaves in exactly the same way as the original" — so no correct + test can kill it. Require **survived**. When the harness reports it caught, stop + and report that nothing was measured: the cases are failing before the rule under + test ever runs. +2. **Read a uniform verdict as a property of the harness, not of the code:** + +| Observed | Read it as | Do | +|----------|------------|-----| +| Mixed verdicts, and the no-op control survived | The harness discriminates | Cite the score together with the control's result | +| Every case caught / red, including the no-op control | Cases die before the rule executes — a broken isolated environment (missing input files, absent dependency, wrong working directory) | Fix the environment, then re-run the control; a 100% catch rate here is a 0% detection rate | +| Every case survives / green | The harness never applied the mutation or never reached the rule — Stryker's troubleshooting carries two distinct "All mutants survive" sections whose documented causes are both sandbox mechanics, not weak tests (the Jest runner cannot run in a hidden temp directory; sandboxing does not support `module-alias/register`) | Verify one mutation reaches the artifact by hand before adjusting the rules | + +3. **Observe the harness produce a verdict on the unmutated artifact first.** Stryker + makes this a named phase — "Initial test run fails" is its own documented failure + mode, and `dryRunOnly` ("Execute the initial test run only without doing actual + mutation testing") runs the phase alone; the run's timing (`netTimeMs`, + `overheadMs`) is derived from it. A harness never seen reporting the unmutated + state has no reference point. +4. **Give the harness a working tree equivalent to the real runner's.** When + isolating into a temp directory, copy the whole repository rather than the + directory under test — a partial copy silently removes fixtures, data files, and + path anchors that tests resolve relative to the repo root. +5. **Make "the case never ran" a distinct outcome from "the case ran and passed."** + Count executed cases and fail the harness when the count is zero. Stryker's Vitest + runner does exactly this — "No tests were executed. Stryker will exit prematurely. + Please check your configuration." — and PIT separates **no coverage** from + **survived** ("the same as Survived except there were no tests that exercised the + line of code where the mutation was created"). +6. **Score `detected / valid`, keeping errored cases out of the numerator *and* the + denominator.** Stryker models `Killed` and `Survived` alongside `Timeout`, + `Runtime error` and `Compile error`, and computes the score as + "detected / valid * 100". A case that blew up before reaching the rule is invalid, + not a detection — folding the two together is the arithmetic that turns a broken + environment into a perfect score. +7. **Publish the score with the control alongside it** — "34/36 caught; no-op + control survived" — so the number carries its own proof of discrimination. + +## Edge cases + +| Case | Then | +|------|------| +| A no-op change is impossible to construct (fully generated artifact) | Use a whitespace- or comment-only edit of the generator's input, and require the same survived verdict | +| The no-op control is legitimately caught | An assertion is pinned to formatting rather than behavior — narrow that assertion, then re-run; the control is measuring the right thing and finding a real over-specification ([testing-quality-tests-that-cannot-fail]) | +| The harness genuinely catches every real mutant (small rule set, exhaustive cases) | The no-op control is then the only evidence separating that from a broken harness — report it explicitly rather than the bare percentage | +| A mutation changes behavior in a way outside what the suite is meant to cover | PIT's second undetectable class (it excludes logging code for this reason) — exclude that region from the mutation set instead of adding a test to chase it | +| The score is already published and cited | Re-run with the control before defending the number, and correct the citation when the control fails | +| Individual checks each have a negative control already | Add the harness-level control too — a per-check control asks "can this check go red", the harness control asks "can this harness go green"; the second failure mode survives the first | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Cite "N/N caught" as proof the rules are enforced | Run the no-op control first and cite the score with its result | A harness whose cases all die before the rule runs reports every mutant as caught while detecting nothing | +| Read a uniform 100% detection rate as strength | Treat uniformity as the fault signal and run the control | Discriminating measurement produces mixed results; a single verdict for every input is what a constant function looks like | +| Tighten the rules when every case comes back red | Verify one case reaches the rule, then re-run the control | Rules are not what fails when the environment is missing the inputs the cases need | +| Copy only the directory under test into the harness's temp tree | Copy the repository, or fail the case on a missing input | Tests that resolve paths from the repo root read as detections when they die on a missing file | + +## Sources + +- https://pitest.org/quickstart/basic_concepts/ — "not all mutations will behave differently than the unmutated class. These mutants are referred to as **equivalent mutations**"; "The resulting mutant behaves in exactly the same way as the original"; a second undetectable class "behaves differently but in a way that is outside the scope of testing" (PIT excludes logging code); "**No coverage** is the same as **Survived** except there were no tests that exercised the line of code where the mutation was created" +- https://stryker-mutator.io/docs/stryker-js/configuration/ — the initial test run is a distinct phase with its own options: `dryRunOnly` "Execute the initial test run only without doing actual mutation testing", `dryRunTimeoutMinutes`; run timing (`netTimeMs`/`overheadMs`) is calculated during it +- https://stryker-mutator.io/docs/stryker-js/troubleshooting/ — section headings "Initial test run fails", "All mutants survive - Jest runner" (cause: Jest "doesn't support running in a hidden directory on windows") and "All mutants survive - module-alias" (cause: "StrykerJS's sandboxing does not support alias imports like `module-alias/register`") — both sandbox mechanics rather than test weakness; the Vitest-runner example "No tests were executed. Stryker will exit prematurely. Please check your configuration." +- https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/ — the state set (`Killed`, `Survived`, `No coverage`, `Timeout`, `Runtime error`, `Compile error`, `Ignored`) and the score as "detected / valid * 100", so errored cases leave the denominator rather than counting as catches +- https://stryker-mutator.io/docs/mutation-testing-elements/equivalent-mutants/ — an equivalent mutant cannot be killed and "There is no definitive way for Stryker to find and ignore them", which is why a surviving no-op is the correct control verdict +- https://testing.googleblog.com/2021/04/mutation-testing.html — inserting faults and requiring test failure is what measures detection, as opposed to coverage +- Field reproduction 2026-07-31 (Python rule-conformance harness): the harness copied only the implementation directory into its temp tree while the tests resolved `examples/*.json` from the repo root, so all 105 tests died on `FileNotFoundError` and every mutation reported as caught — "36/36" was cited in five commits and a README. A docstring-capitalization no-op reproduced the red verdict and exposed it; copying the full repository plus adding the no-op control moved the score to 34/36 and surfaced two rules no test asserted diff --git a/wiki/testing/quality/minimum-case-set.md b/wiki/testing/quality/minimum-case-set.md index 6adbd68..8b1287e 100644 --- a/wiki/testing/quality/minimum-case-set.md +++ b/wiki/testing/quality/minimum-case-set.md @@ -9,7 +9,7 @@ sources: - https://abseil.io/resources/swe-book/html/ch12.html - https://martinfowler.com/bliki/TestDrivenDevelopment.html last_verified: 2026-07-10 -related: [testing-strategy-test-level-choice, testing-quality-behavior-not-implementation] +related: [testing-strategy-test-level-choice, testing-quality-behavior-not-implementation, testing-quality-checks-that-cannot-pass] --- # Selecting the Minimum Case Set for a Function or Endpoint diff --git a/wiki/testing/quality/spec-artifact-checks.md b/wiki/testing/quality/spec-artifact-checks.md new file mode 100644 index 0000000..d754f6e --- /dev/null +++ b/wiki/testing/quality/spec-artifact-checks.md @@ -0,0 +1,105 @@ +--- +id: testing-quality-spec-artifact-checks +domain: testing +category: quality +applies_to: [general] +confidence: verified +sources: + - https://json-schema.org/draft/2020-12/json-schema-validation + - https://eslint.org/docs/latest/integrate/nodejs-api + - https://pitest.org/ + - https://github.github.com/gfm/ +last_verified: 2026-07-29 +related: [testing-quality-tests-that-cannot-fail] +--- + +# Checks That Verify a Spec or Mapping Artifact + +## When this applies + +You are writing or reviewing an automated check that an artifact conforms to a +source of truth: a mapping table that must hold a row per rule, a key per field, +or a case per enum; ids that must resolve across documents; required sections in +an RFC. Includes doc-as-spec repos where the artifact is Markdown. + +## Do this + +1. **Split coverage from value validity into two named checks.** Coverage asks + "is every required row/key/case present". Value validity asks "does each + cell's value exist in the canonical enum, schema, or id set". A set-equality + check on row *names* is blind to cell contents, so a table reaches 100% + coverage while a cell holds a typo'd, empty, or swapped value. This is the + split JSON Schema draws between `required` — "every item in the array is the + name of a property in the instance" — and `enum`, where the instance's *value* + must equal one of the listed elements. + +2. **Give each check its own negative control, mutating only what that check + owns.** Rerun the whole harness per mutation and require exactly the owning + check to turn red while the others hold their prior verdict: + +| Check | Mutation that must redden it | Other checks must hold at | +|-------|------------------------------|---------------------------| +| Coverage — every required row/key/case present | Delete one required row | Their prior verdict | +| Value validity — each cell in the canonical set | Replace one cell with a value absent from that set | Coverage stays green | +| Cross-document id resolution | Repoint one id at a target that does not exist | Coverage and validity stay green | + + Seeding a fault and requiring a failure is the mutation-testing mechanic — a + mutant is killed when a test fails. Requiring *which* check fails is a + convention this page adds on top of it, so that each verdict names a distinct + property. Pair every mutation with a must-pass input, the form an ESLint rule + test takes when its `invalid` cases declare the errors they expect. + +3. **Print which check caught each seeded fault**, so a green run states what was + proven rather than only that something ran. + +4. **Report the verdict in the words the checks earned.** When only coverage ran, + the result is "mapping present"; "mapping verified" requires the validity + check to have run and to have its own negative control. + +5. **Parse Markdown tables by splitting on unescaped pipes, then unescape each + cell**, before asserting cell counts or per-column values: + +```python +row = re.sub(r"(?