diff --git a/.agents/skills/pgpm-projections/SKILL.md b/.agents/skills/pgpm-projections/SKILL.md new file mode 100644 index 000000000..d8b21d4c1 --- /dev/null +++ b/.agents/skills/pgpm-projections/SKILL.md @@ -0,0 +1,105 @@ +--- +name: pgpm-projections +description: The pgpm projections pipeline — normalize any PostgreSQL schema source (module, .sql file, dump, live database) to an identity-keyed semantic model and project it into any representation. Use when asked to "import a schema", "introspect a database", "generate a migration", "diff two schemas", "change migration granularity", "split changes per column", "convert to one big migration", "restructure a pgpm module", "emit linear SQL", "partition a module", "verify a migration against a database", or when working with pgpm import/transform/diff, --granularity, --change-granularity, --partition, --emit-migration, --emit-sql, or --emit-bundle. +compatibility: pgpm CLI, PostgreSQL 14+, Node.js 22+ +metadata: + author: constructive-io + version: "1.0.0" +--- + +# pgpm projections + +One canonical semantic model, many representations. pgpm normalizes any schema +source to an **identity-keyed object set** (objects keyed by kind/schema/name, +ASTs canonicalized — whitespace, statement order, constraint placement, and +authoring granularity all wash out). Everything downstream is a **projection**: +it changes the representation, never the meaning. + +``` +source (module dir | .sql file | dump | live DB) + → parse/classify → identity-keyed semantic model + → projections (granularity, change granularity, naming, partition) + → outputs (pgpm module | pgpm changes | linear .sql | bundle) + → deploy / verify / revert +``` + +## The dials + +| Dial | Values | What it shapes | +| --- | --- | --- | +| `--granularity` | `atomic` \| `object` \| `consolidated` | SQL statement shape *within* a change: one `ALTER` per column/constraint vs one statement per object vs maximally combined | +| `--change-granularity` | `alteration` \| `object` (default) \| `single` | plan-entry distribution *across* changes: one change per column/constraint vs one per object vs one big change | +| `--naming` | `directory` (default) \| `flat` | change path style | +| `--partition ` | JSON rules | which objects land in which package (cross-package requires derived) | + +The two granularity axes are orthogonal: `--granularity` controls statement +shape, `--change-granularity` controls how statements are distributed across +plan entries. Every combination is semantically invariant — same identity-keyed +model, same deployed catalog. + +With `--change-granularity alteration`, every `ADD COLUMN` / `ADD CONSTRAINT` +becomes its own change with its own deploy/revert/verify and graph-derived +requires (paths like `schemas/app/tables/users/columns/email/column`, +`.../constraints/users_pkey/constraint`); unnamed constraints are auto-named +with their Postgres default (`{table}_pkey`, `{table}_{cols}_key`, +`{table}_{cols}_fkey`, `{table}_{col}_check`) so each is independently +revertible. With `single`, the whole module becomes one plan entry +(`module/init` by default). + +## Commands + +```bash +# schema.sql (or a pg_dump) -> pgpm module +pgpm import schema.sql --pkg myapp --out ./out \ + --granularity object --change-granularity object + +# re-dial an existing module (writes sibling -) +cd out/myapp +pgpm transform --granularity atomic --change-granularity alteration --out ../alt +pgpm transform --granularity consolidated --change-granularity single --out ../one + +# prove a transform is lossless against a real catalog (scratch DBs) +pgpm transform --granularity atomic --check + +# semantic diff + migration generation; sides can be a module dir, a .sql +# file, or a live database (db: or a postgres:// DSN — dumped, never +# held open) +pgpm diff --emit-migration ./out --pkg my-migration \ + --granularity atomic --change-granularity alteration \ + --emit-sql ./out/migration.sql --emit-bundle ./out/migration.tar.gz + +# append the delta into an existing module's plan instead of a new package +pgpm diff --append-module ./existing-module + +# oracle: deploy A + emitted migration into a scratch DB, assert catalog +# equivalence with B deployed fresh +pgpm diff --verify +``` + +## Library seams (for programmatic use) + +- `@pgpmjs/transform` — the engine: `restructureChanges({ granularity, changeGranularity, singleChangeName })`, + `restructureExportRows`, `diffChangeSets`, `subObjectIdentityOf` (recovers + column/constraint identity from the raw parse node), `nameUnnamedConstraints`, + `defaultConstraintName`, `snapshotCatalog`/`diffCatalogSnapshots`. +- `@pgpmjs/naming-spec` — pure identity → path projection (`pathFor`), including + `column` and `constraint` kinds. +- `@pgpmjs/import` / `@pgpmjs/diff` — source loading (`importDumpRows`, + `loadDiffSideFromDisk`). +- Output projections (module / linear SQL / bundle) live in the pgpm CLI + (`pgpm/cli/src/utils/module-projections.ts`) because they depend on + `@pgpmjs/core`. + +## Invariants to preserve (and test against) + +- Any dial combination normalizes back to the identical identity-keyed model + (`diffChangeSets(a, b).identical === true`). +- Any dial combination deploys to the identical Postgres catalog. +- Generated migrations never use `CREATE OR REPLACE`; changed functions emit + `DROP` + `CREATE`. +- Every emitted change has deploy, revert, and verify; non-derivable reverts + get a `-- revert not derivable` comment plus a warning, never silence. + +Reference example with executable proofs: `examples/pgpm-projections` +(README + database-free jest suite that runs the real CLI). Live-Postgres +coverage: `pgpm/cli/__tests__/transform-e2e.test.ts` and `diff-e2e.test.ts`. diff --git a/examples/pgpm-projections/README.md b/examples/pgpm-projections/README.md index 3b1224ddd..b4a33a6ae 100644 --- a/examples/pgpm-projections/README.md +++ b/examples/pgpm-projections/README.md @@ -24,7 +24,8 @@ one canonical model: | Axis | Projections | What changes | What stays the same | | --- | --- | --- | --- | -| **granularity** | `atomic` · `object` · `consolidated` | how finely a change is split (one `ALTER` per column vs one statement per object vs the whole thing) | the resulting catalog | +| **granularity** | `atomic` · `object` · `consolidated` | the SQL statement shape (one `ALTER` per column vs one statement per object vs the whole thing) | the schema | +| **change granularity** | `alteration` · `object` · `single` | the plan-entry shape (one change per column/constraint vs one per object vs one big change, each with its own deploy/revert/verify) | the schema | | **partition** | one module vs many | which objects live in which package | the combined schema | | **diff** | `v1 → v2` migration | — | derived, never hand-written | | **output** | pgpm module · linear `.sql` · bundle | the artifact format | the migration | @@ -48,6 +49,12 @@ cd out/blog pgpm transform --granularity atomic pgpm transform --granularity consolidated +# 2b. The fourth dial: one change PER ALTERATION — every column and constraint +# becomes its own plan entry with its own deploy/revert/verify + requires — +# or ONE BIG CHANGE for the whole module +pgpm transform --granularity atomic --change-granularity alteration --out ../alteration +pgpm transform --granularity consolidated --change-granularity single --out ../single + # 3. Partition the source into app + security modules cd ../.. pgpm import schema/schema.sql --pkg blog-part --partition schema/partition.json --out ./out @@ -74,8 +81,12 @@ The suite ([`__tests__/projections.e2e.test.ts`](./__tests__/projections.e2e.tes runs the real CLI and then compares the emitted artifacts **semantically** — no database needed, because equivalence is checked at the identity-keyed model: -- **granularity-invariant** — `object` and `consolidated` normalize to the same - schema (empty diff). +- **granularity-invariant** — `atomic`, `object`, and `consolidated` all + normalize to the same schema (empty diff), including standalone + `ADD CONSTRAINT` placement. +- **change-granularity-invariant** — one change per alteration (per column / + per constraint plan entries) and one big change for the whole module both + still normalize to the same schema (empty diff). - **partition-invariant** — the `blog-core` + `blog-security` modules recombine to the same schema as the single module (empty diff). - **diff is exact** — `schema.sql → schema-v2.sql` derives precisely the real @@ -92,11 +103,4 @@ covered against live Postgres by the engine's own suites (`pgpm/cli` `transform-e2e` "dial parity" and `diff-e2e`). This example is deliberately database-free and asserts the model-level invariants above. -> Note: `atomic` authorship emits standalone `ALTER TABLE … ADD CONSTRAINT` -> statements. Because Postgres names such constraints at deploy time, the -> semantic normalizer does not yet fold every standalone constraint back into -> its owning table object, so `atomic` is guaranteed **catalog-equivalent** -> rather than AST-identical to `object`/`consolidated`. Tightening that -> normalization is tracked in `constructive-planning`. - It's a normal workspace package, so `pnpm install` at the repo root wires it up. diff --git a/examples/pgpm-projections/__tests__/projections.e2e.test.ts b/examples/pgpm-projections/__tests__/projections.e2e.test.ts index 5a15fa321..318987cf6 100644 --- a/examples/pgpm-projections/__tests__/projections.e2e.test.ts +++ b/examples/pgpm-projections/__tests__/projections.e2e.test.ts @@ -3,19 +3,21 @@ // One SQL schema (schema/schema.sql) is the source of truth. We normalize it // to an identity-keyed object set and then *project* it into many shapes: // -// granularity object | consolidated (atomic too — see the note below) -// partition one module vs app + security modules -// diff schema.sql -> schema-v2.sql as a generated migration -// output pgpm module | linear .sql +// granularity atomic | object | consolidated (statement shape) +// change granularity alteration | object | single (plan-entry shape) +// partition one module vs app + security modules +// diff schema.sql -> schema-v2.sql as a generated migration +// output pgpm module | linear .sql // // The headline guarantee: a projection changes the *representation*, never the // *meaning*. We prove it WITHOUT a database by normalizing each projection back // to its object set and asserting the diff is empty — authoring granularity, -// naming, partitioning, ordering, and whitespace all wash out. +// change granularity, naming, partitioning, ordering, and whitespace all wash +// out. // -// (Deploy-level catalog equivalence for every projection, including `atomic`, -// is proven against live Postgres by the engine's own suites — pgpm/cli -// transform-e2e "dial parity" and diff-e2e. See README.) +// (Deploy-level catalog equivalence for every projection is also proven +// against live Postgres by the engine's own suites — pgpm/cli transform-e2e +// "dial parity" and diff-e2e. See README.) // // This suite is intentionally database-free: it only runs the CLI and compares // the emitted artifacts semantically. @@ -50,6 +52,17 @@ describe('pgpm projections: one schema, every shape, one meaning', () => { pgpm(work, ['transform', '--granularity', 'atomic', '--cwd', path.join(work, 'blog')]); pgpm(work, ['transform', '--granularity', 'consolidated', '--cwd', path.join(work, 'blog')]); + // The fourth dial: one change per alteration (per column / per constraint), + // or the whole module as one big change. + pgpm(work, [ + 'transform', '--granularity', 'atomic', '--change-granularity', 'alteration', + '--cwd', path.join(work, 'blog'), '--out', path.join(work, 'alteration') + ]); + pgpm(work, [ + 'transform', '--granularity', 'consolidated', '--change-granularity', 'single', + '--cwd', path.join(work, 'blog'), '--out', path.join(work, 'single') + ]); + // Partition the same source into app + security modules. pgpm(work, [ 'import', path.join(SCHEMA_DIR, 'schema.sql'), @@ -93,19 +106,37 @@ describe('pgpm projections: one schema, every shape, one meaning', () => { expect(diffChangeSets(object, partitioned).identical).toBe(true); }); - it('emits the atomic projection as a deployable module covering the same schemas', () => { - // atomic explodes objects into per-column / per-constraint statements. It - // deploys to the same catalog as the other granularities (proven against - // live Postgres by pgpm/cli transform-e2e "dial parity"); here we assert it - // is a well-formed module spanning the same schemas. - const atomicDir = path.join(work, 'blog-atomic'); - expect(fs.existsSync(path.join(atomicDir, 'pgpm.plan'))).toBe(true); - const atomicChanges = changesOf(atomicDir); - expect(atomicChanges.length).toBeGreaterThan(0); - - const schemasOf = (changes: ChangeSet): string[] => - [...new Set(changes.map(c => c.name.split('/')[1]).filter(Boolean))].sort(); - expect(schemasOf(atomicChanges)).toEqual(schemasOf(changesOf(path.join(work, 'blog')))); + it('is granularity-invariant: atomic normalizes to the same schema too', () => { + // atomic explodes objects into per-column / per-constraint statements, yet + // constraint-placement normalization folds them back — full semantic + // identity, not just catalog equivalence. + const object = changesOf(path.join(work, 'blog')); + const atomic = changesOf(path.join(work, 'blog-atomic')); + expect(diffChangeSets(object, atomic).identical).toBe(true); + }); + + it('is change-granularity-invariant: one change per alteration, same schema', () => { + const alterationDir = path.join(work, 'alteration', 'blog-atomic'); + const plan = fs.readFileSync(path.join(alterationDir, 'pgpm.plan'), 'utf-8'); + // every column and constraint is its own plan entry... + expect(plan).toMatch(/\/columns\/[a-z_]+\/column/); + expect(plan).toMatch(/\/constraints\/[a-z_]+\/constraint/); + // ...and the meaning is untouched. + const object = changesOf(path.join(work, 'blog')); + const alteration = changesOf(alterationDir); + expect(diffChangeSets(object, alteration).identical).toBe(true); + }); + + it('is change-granularity-invariant: one big change, same schema', () => { + const singleDir = path.join(work, 'single', 'blog-consolidated'); + const plan = fs.readFileSync(path.join(singleDir, 'pgpm.plan'), 'utf-8'); + const entries = plan.split('\n').filter(l => l.trim() && !l.startsWith('%') && !l.startsWith('#')); + // the whole module is one plan entry... + expect(entries).toHaveLength(1); + // ...and the meaning is untouched. + const object = changesOf(path.join(work, 'blog')); + const single = changesOf(singleDir); + expect(diffChangeSets(object, single).identical).toBe(true); }); it('derives the v1 -> v2 migration: exactly the real changes, nothing guessed', () => { diff --git a/pgpm/cli/__tests__/diff-e2e.test.ts b/pgpm/cli/__tests__/diff-e2e.test.ts index ea1ab7028..aaf3c491d 100644 --- a/pgpm/cli/__tests__/diff-e2e.test.ts +++ b/pgpm/cli/__tests__/diff-e2e.test.ts @@ -324,6 +324,33 @@ describe('pgpm diff e2e', () => { } ); + it('emits an alteration-change-granularity migration that reaches the same v2 catalog', async () => { + const pkg = 'diff-mig-alteration'; + await fixture.runTerminalCommands( + ` + cd ${WS} + pgpm diff diff-v1 diff-v2 --emit-migration . --pkg ${pkg} --granularity atomic --change-granularity alteration + `, + {} + ); + expect(fs.existsSync(path.join(wsDir, pkg, 'pgpm.plan'))).toBe(true); + + const testDb = await fixture.setupTestDatabase(); + await fixture.runTerminalCommands( + ` + cd ${WS}/diff-v1 + pgpm deploy --database ${testDb.name} --package diff-v1 --yes + cd ../${pkg} + pgpm deploy --database ${testDb.name} --package ${pkg} --yes + `, + { database: testDb.name } + ); + + const snapMigrated = await snapshotCatalog(testDb); + const snapV2 = await snapshotCatalog(v2Db); + expect(diffCatalogSnapshots(withoutColumnOrder(snapMigrated), withoutColumnOrder(snapV2))).toEqual([]); + }); + it('--verify proves the emitted migration is a catalog-level oracle match', async () => { await fixture.runTerminalCommands( ` diff --git a/pgpm/cli/__tests__/transform-e2e.test.ts b/pgpm/cli/__tests__/transform-e2e.test.ts index cfeab8d76..e7e15dd8a 100644 --- a/pgpm/cli/__tests__/transform-e2e.test.ts +++ b/pgpm/cli/__tests__/transform-e2e.test.ts @@ -215,6 +215,97 @@ describe('pgpm transform e2e', () => { } ); + it('--change-granularity alteration splits per column/constraint, deploys equivalently, verifies, and reverts clean', async () => { + await fixture.runTerminalCommands( + ` + cd ${WS}/${MODULE_NAME} + pgpm transform --granularity atomic --change-granularity alteration --out ../alteration-out + `, + {} + ); + + const outDir = path.join(wsDir, 'alteration-out', `${MODULE_NAME}-atomic`); + const plan = fs.readFileSync(path.join(outDir, 'pgpm.plan'), 'utf-8'); + expect(plan).toContain('schemas/tfx_app/tables/users/columns/id/column'); + expect(plan).toContain('schemas/tfx_app/tables/users/columns/email/column'); + expect(plan).toContain('schemas/tfx_app/tables/users/constraints/users_pkey/constraint'); + expect(plan).toContain('schemas/tfx_sec/tables/audit/constraints/audit_user_id_fkey/constraint'); + + // Each alteration has its own deploy/revert/verify triple. + const columnChange = 'schemas/tfx_app/tables/users/columns/email/column'; + expect(fs.readFileSync(path.join(outDir, 'deploy', `${columnChange}.sql`), 'utf-8')).toContain('ADD COLUMN email'); + expect(fs.readFileSync(path.join(outDir, 'revert', `${columnChange}.sql`), 'utf-8')).toContain('DROP COLUMN email'); + expect(fs.readFileSync(path.join(outDir, 'verify', `${columnChange}.sql`), 'utf-8')).toContain('information_schema.columns'); + + const testDb = await fixture.setupTestDatabase(); + await fixture.runTerminalCommands( + ` + cd ${WS}/alteration-out/${MODULE_NAME}-atomic + pgpm deploy --database ${testDb.name} --package ${MODULE_NAME}-atomic --yes + `, + { database: testDb.name } + ); + + const snapOriginal = await snapshotCatalog(originalDb); + const snapTransformed = await snapshotCatalog(testDb); + expect(diffCatalogSnapshots(snapOriginal, snapTransformed)).toEqual([]); + + await fixture.runTerminalCommands( + ` + cd ${WS}/alteration-out/${MODULE_NAME}-atomic + pgpm verify --database ${testDb.name} --package ${MODULE_NAME}-atomic --yes + pgpm revert --database ${testDb.name} --package ${MODULE_NAME}-atomic --yes + `, + { database: testDb.name } + ); + expect(await testDb.exists('schema', 'tfx_app')).toBe(false); + }); + + it('--change-granularity single collapses to one change, deploys equivalently, verifies, and reverts clean', async () => { + await fixture.runTerminalCommands( + ` + cd ${WS}/${MODULE_NAME} + pgpm transform --granularity consolidated --change-granularity single --out ../single-out + `, + {} + ); + + const outDir = path.join(wsDir, 'single-out', `${MODULE_NAME}-consolidated`); + const plan = fs.readFileSync(path.join(outDir, 'pgpm.plan'), 'utf-8'); + const entries = plan.split('\n').filter(l => l.trim() && !l.startsWith('%') && !l.startsWith('#')); + expect(entries).toHaveLength(1); + expect(plan).toContain('module/init'); + + const deploy = fs.readFileSync(path.join(outDir, 'deploy', 'module/init.sql'), 'utf-8'); + expect(deploy).toContain('CREATE SCHEMA tfx_app'); + expect(deploy).toContain('CREATE TABLE tfx_sec.audit'); + expect(fs.readFileSync(path.join(outDir, 'revert', 'module/init.sql'), 'utf-8')).toContain('DROP SCHEMA tfx_app'); + expect(fs.readFileSync(path.join(outDir, 'verify', 'module/init.sql'), 'utf-8')).toContain('information_schema'); + + const testDb = await fixture.setupTestDatabase(); + await fixture.runTerminalCommands( + ` + cd ${WS}/single-out/${MODULE_NAME}-consolidated + pgpm deploy --database ${testDb.name} --package ${MODULE_NAME}-consolidated --yes + `, + { database: testDb.name } + ); + + const snapOriginal = await snapshotCatalog(originalDb); + const snapTransformed = await snapshotCatalog(testDb); + expect(diffCatalogSnapshots(snapOriginal, snapTransformed)).toEqual([]); + + await fixture.runTerminalCommands( + ` + cd ${WS}/single-out/${MODULE_NAME}-consolidated + pgpm verify --database ${testDb.name} --package ${MODULE_NAME}-consolidated --yes + pgpm revert --database ${testDb.name} --package ${MODULE_NAME}-consolidated --yes + `, + { database: testDb.name } + ); + expect(await testDb.exists('schema', 'tfx_app')).toBe(false); + }); + it('rewrites an existing output when --write is passed', async () => { // The refusal path (no --write) exits the process, so it is covered by the // checkOverwrite unit tests; here we prove --write re-generates in place. diff --git a/pgpm/cli/src/commands/diff.ts b/pgpm/cli/src/commands/diff.ts index af8c777b0..9688bbc76 100644 --- a/pgpm/cli/src/commands/diff.ts +++ b/pgpm/cli/src/commands/diff.ts @@ -8,9 +8,12 @@ import { import { Logger } from '@pgpmjs/logger'; import { appendModule, + CHANGE_GRANULARITIES, + ChangeGranularity, diffChangeSets, EXPORT_GRANULARITIES, ExportGranularity, + isChangeGranularity, isExportGranularity, loadModule, SemanticDiffResult, @@ -70,6 +73,9 @@ Options: --pkg Emitted migration package name (default: diff-migration) --granularity Granularity for emitted changes: atomic | object | consolidated (default: object) + --change-granularity + Change-level distribution for emitted changes: + alteration | object | single (default: object) --naming