feat(pgpm): change-granularity 'single' level + pgpm-projections skill

.agents/skills/pgpm-projections/SKILL.md

@@ -0,0 +1,105 @@
+---
+name: pgpm-projections
+description: The pgpm projections pipeline — normalize any PostgreSQL schema source (module, .sql file, dump, live database) to an identity-keyed semantic model and project it into any representation. Use when asked to "import a schema", "introspect a database", "generate a migration", "diff two schemas", "change migration granularity", "split changes per column", "convert to one big migration", "restructure a pgpm module", "emit linear SQL", "partition a module", "verify a migration against a database", or when working with pgpm import/transform/diff, --granularity, --change-granularity, --partition, --emit-migration, --emit-sql, or --emit-bundle.
+compatibility: pgpm CLI, PostgreSQL 14+, Node.js 22+
+metadata:
+  author: constructive-io
+  version: "1.0.0"
+---
+
+# pgpm projections
+
+One canonical semantic model, many representations. pgpm normalizes any schema
+source to an **identity-keyed object set** (objects keyed by kind/schema/name,
+ASTs canonicalized — whitespace, statement order, constraint placement, and
+authoring granularity all wash out). Everything downstream is a **projection**:
+it changes the representation, never the meaning.
+
+```
+source (module dir | .sql file | dump | live DB)
+  → parse/classify → identity-keyed semantic model
+  → projections (granularity, change granularity, naming, partition)
+  → outputs (pgpm module | pgpm changes | linear .sql | bundle)
+  → deploy / verify / revert
+```
+
+## The dials
+
+| Dial | Values | What it shapes |
+| --- | --- | --- |
+| `--granularity` | `atomic` \| `object` \| `consolidated` | SQL statement shape *within* a change: one `ALTER` per column/constraint vs one statement per object vs maximally combined |
+| `--change-granularity` | `alteration` \| `object` (default) \| `single` | plan-entry distribution *across* changes: one change per column/constraint vs one per object vs one big change |
+| `--naming` | `directory` (default) \| `flat` | change path style |
+| `--partition <file>` | JSON rules | which objects land in which package (cross-package requires derived) |
+
+The two granularity axes are orthogonal: `--granularity` controls statement
+shape, `--change-granularity` controls how statements are distributed across
+plan entries. Every combination is semantically invariant — same identity-keyed
+model, same deployed catalog.
+
+With `--change-granularity alteration`, every `ADD COLUMN` / `ADD CONSTRAINT`
+becomes its own change with its own deploy/revert/verify and graph-derived
+requires (paths like `schemas/app/tables/users/columns/email/column`,
+`.../constraints/users_pkey/constraint`); unnamed constraints are auto-named
+with their Postgres default (`{table}_pkey`, `{table}_{cols}_key`,
+`{table}_{cols}_fkey`, `{table}_{col}_check`) so each is independently
+revertible. With `single`, the whole module becomes one plan entry
+(`module/init` by default).
+
+## Commands
+
+```bash
+# schema.sql (or a pg_dump) -> pgpm module
+pgpm import schema.sql --pkg myapp --out ./out \
+  --granularity object --change-granularity object
+
+# re-dial an existing module (writes sibling <name>-<granularity>)
+cd out/myapp
+pgpm transform --granularity atomic --change-granularity alteration --out ../alt
+pgpm transform --granularity consolidated --change-granularity single --out ../one
+
+# prove a transform is lossless against a real catalog (scratch DBs)
+pgpm transform --granularity atomic --check
+
+# semantic diff + migration generation; sides can be a module dir, a .sql
+# file, or a live database (db:<name> or a postgres:// DSN — dumped, never
+# held open)
+pgpm diff <A> <B> --emit-migration ./out --pkg my-migration \
+  --granularity atomic --change-granularity alteration \
+  --emit-sql ./out/migration.sql --emit-bundle ./out/migration.tar.gz
+
+# append the delta into an existing module's plan instead of a new package
+pgpm diff <A> <B> --append-module ./existing-module
+
+# oracle: deploy A + emitted migration into a scratch DB, assert catalog
+# equivalence with B deployed fresh
+pgpm diff <A> <B> --verify
+```
+
+## Library seams (for programmatic use)
+
+- `@pgpmjs/transform` — the engine: `restructureChanges({ granularity, changeGranularity, singleChangeName })`,
+  `restructureExportRows`, `diffChangeSets`, `subObjectIdentityOf` (recovers
+  column/constraint identity from the raw parse node), `nameUnnamedConstraints`,
+  `defaultConstraintName`, `snapshotCatalog`/`diffCatalogSnapshots`.
+- `@pgpmjs/naming-spec` — pure identity → path projection (`pathFor`), including
+  `column` and `constraint` kinds.
+- `@pgpmjs/import` / `@pgpmjs/diff` — source loading (`importDumpRows`,
+  `loadDiffSideFromDisk`).
+- Output projections (module / linear SQL / bundle) live in the pgpm CLI
+  (`pgpm/cli/src/utils/module-projections.ts`) because they depend on
+  `@pgpmjs/core`.
+
+## Invariants to preserve (and test against)
+
+- Any dial combination normalizes back to the identical identity-keyed model
+  (`diffChangeSets(a, b).identical === true`).
+- Any dial combination deploys to the identical Postgres catalog.
+- Generated migrations never use `CREATE OR REPLACE`; changed functions emit
+  `DROP` + `CREATE`.
+- Every emitted change has deploy, revert, and verify; non-derivable reverts
+  get a `-- revert not derivable` comment plus a warning, never silence.
+
+Reference example with executable proofs: `examples/pgpm-projections`
+(README + database-free jest suite that runs the real CLI). Live-Postgres
+coverage: `pgpm/cli/__tests__/transform-e2e.test.ts` and `diff-e2e.test.ts`.

examples/pgpm-projections/README.md

@@ -24,7 +24,8 @@ one canonical model:
 
 | Axis | Projections | What changes | What stays the same |
 | --- | --- | --- | --- |
-| **granularity** | `atomic` · `object` · `consolidated` | how finely a change is split (one `ALTER` per column vs one statement per object vs the whole thing) | the resulting catalog |
+| **granularity** | `atomic` · `object` · `consolidated` | the SQL statement shape (one `ALTER` per column vs one statement per object vs the whole thing) | the schema |
+| **change granularity** | `alteration` · `object` · `single` | the plan-entry shape (one change per column/constraint vs one per object vs one big change, each with its own deploy/revert/verify) | the schema |
 | **partition** | one module vs many | which objects live in which package | the combined schema |
 | **diff** | `v1 → v2` migration | — | derived, never hand-written |
 | **output** | pgpm module · linear `.sql` · bundle | the artifact format | the migration |
@@ -48,6 +49,12 @@ cd out/blog
 pgpm transform --granularity atomic
 pgpm transform --granularity consolidated
 
+# 2b. The fourth dial: one change PER ALTERATION — every column and constraint
+#     becomes its own plan entry with its own deploy/revert/verify + requires —
+#     or ONE BIG CHANGE for the whole module
+pgpm transform --granularity atomic --change-granularity alteration --out ../alteration
+pgpm transform --granularity consolidated --change-granularity single --out ../single
+
 # 3. Partition the source into app + security modules
 cd ../..
 pgpm import schema/schema.sql --pkg blog-part --partition schema/partition.json --out ./out
@@ -74,8 +81,12 @@ The suite ([`__tests__/projections.e2e.test.ts`](./__tests__/projections.e2e.tes
 runs the real CLI and then compares the emitted artifacts **semantically** — no
 database needed, because equivalence is checked at the identity-keyed model:
 
-- **granularity-invariant** — `object` and `consolidated` normalize to the same
-  schema (empty diff).
+- **granularity-invariant** — `atomic`, `object`, and `consolidated` all
+  normalize to the same schema (empty diff), including standalone
+  `ADD CONSTRAINT` placement.
+- **change-granularity-invariant** — one change per alteration (per column /
+  per constraint plan entries) and one big change for the whole module both
+  still normalize to the same schema (empty diff).
 - **partition-invariant** — the `blog-core` + `blog-security` modules recombine
   to the same schema as the single module (empty diff).
 - **diff is exact** — `schema.sql → schema-v2.sql` derives precisely the real
@@ -92,11 +103,4 @@ covered against live Postgres by the engine's own suites
 (`pgpm/cli` `transform-e2e` "dial parity" and `diff-e2e`). This example is
 deliberately database-free and asserts the model-level invariants above.
 
-> Note: `atomic` authorship emits standalone `ALTER TABLE … ADD CONSTRAINT`
-> statements. Because Postgres names such constraints at deploy time, the
-> semantic normalizer does not yet fold every standalone constraint back into
-> its owning table object, so `atomic` is guaranteed **catalog-equivalent**
-> rather than AST-identical to `object`/`consolidated`. Tightening that
-> normalization is tracked in `constructive-planning`.
-
 It's a normal workspace package, so `pnpm install` at the repo root wires it up.

examples/pgpm-projections/__tests__/projections.e2e.test.ts

@@ -3,19 +3,21 @@
 // One SQL schema (schema/schema.sql) is the source of truth. We normalize it
 // to an identity-keyed object set and then *project* it into many shapes:
 //
-//   granularity   object | consolidated  (atomic too — see the note below)
-//   partition     one module vs app + security modules
-//   diff          schema.sql -> schema-v2.sql as a generated migration
-//   output        pgpm module | linear .sql
+//   granularity          atomic | object | consolidated  (statement shape)
+//   change granularity   alteration | object | single    (plan-entry shape)
+//   partition            one module vs app + security modules
+//   diff                 schema.sql -> schema-v2.sql as a generated migration
+//   output               pgpm module | linear .sql
 //
 // The headline guarantee: a projection changes the *representation*, never the
 // *meaning*. We prove it WITHOUT a database by normalizing each projection back
 // to its object set and asserting the diff is empty — authoring granularity,
-// naming, partitioning, ordering, and whitespace all wash out.
+// change granularity, naming, partitioning, ordering, and whitespace all wash
+// out.
 //
-// (Deploy-level catalog equivalence for every projection, including `atomic`,
-// is proven against live Postgres by the engine's own suites — pgpm/cli
-// transform-e2e "dial parity" and diff-e2e. See README.)
+// (Deploy-level catalog equivalence for every projection is also proven
+// against live Postgres by the engine's own suites — pgpm/cli transform-e2e
+// "dial parity" and diff-e2e. See README.)
 //
 // This suite is intentionally database-free: it only runs the CLI and compares
 // the emitted artifacts semantically.
@@ -50,6 +52,17 @@ describe('pgpm projections: one schema, every shape, one meaning', () => {
     pgpm(work, ['transform', '--granularity', 'atomic', '--cwd', path.join(work, 'blog')]);
     pgpm(work, ['transform', '--granularity', 'consolidated', '--cwd', path.join(work, 'blog')]);
 
+    // The fourth dial: one change per alteration (per column / per constraint),
+    // or the whole module as one big change.
+    pgpm(work, [
+      'transform', '--granularity', 'atomic', '--change-granularity', 'alteration',
+      '--cwd', path.join(work, 'blog'), '--out', path.join(work, 'alteration')
+    ]);
+    pgpm(work, [
+      'transform', '--granularity', 'consolidated', '--change-granularity', 'single',
+      '--cwd', path.join(work, 'blog'), '--out', path.join(work, 'single')
+    ]);
+
     // Partition the same source into app + security modules.
     pgpm(work, [
       'import', path.join(SCHEMA_DIR, 'schema.sql'),
@@ -93,19 +106,37 @@ describe('pgpm projections: one schema, every shape, one meaning', () => {
     expect(diffChangeSets(object, partitioned).identical).toBe(true);
   });
 
-  it('emits the atomic projection as a deployable module covering the same schemas', () => {
-    // atomic explodes objects into per-column / per-constraint statements. It
-    // deploys to the same catalog as the other granularities (proven against
-    // live Postgres by pgpm/cli transform-e2e "dial parity"); here we assert it
-    // is a well-formed module spanning the same schemas.
-    const atomicDir = path.join(work, 'blog-atomic');
-    expect(fs.existsSync(path.join(atomicDir, 'pgpm.plan'))).toBe(true);
-    const atomicChanges = changesOf(atomicDir);
-    expect(atomicChanges.length).toBeGreaterThan(0);
-
-    const schemasOf = (changes: ChangeSet): string[] =>
-      [...new Set(changes.map(c => c.name.split('/')[1]).filter(Boolean))].sort();
-    expect(schemasOf(atomicChanges)).toEqual(schemasOf(changesOf(path.join(work, 'blog'))));
+  it('is granularity-invariant: atomic normalizes to the same schema too', () => {
+    // atomic explodes objects into per-column / per-constraint statements, yet
+    // constraint-placement normalization folds them back — full semantic
+    // identity, not just catalog equivalence.
+    const object = changesOf(path.join(work, 'blog'));
+    const atomic = changesOf(path.join(work, 'blog-atomic'));
+    expect(diffChangeSets(object, atomic).identical).toBe(true);
+  });
+
+  it('is change-granularity-invariant: one change per alteration, same schema', () => {
+    const alterationDir = path.join(work, 'alteration', 'blog-atomic');
+    const plan = fs.readFileSync(path.join(alterationDir, 'pgpm.plan'), 'utf-8');
+    // every column and constraint is its own plan entry...
+    expect(plan).toMatch(/\/columns\/[a-z_]+\/column/);
+    expect(plan).toMatch(/\/constraints\/[a-z_]+\/constraint/);
+    // ...and the meaning is untouched.
+    const object = changesOf(path.join(work, 'blog'));
+    const alteration = changesOf(alterationDir);
+    expect(diffChangeSets(object, alteration).identical).toBe(true);
+  });
+
+  it('is change-granularity-invariant: one big change, same schema', () => {
+    const singleDir = path.join(work, 'single', 'blog-consolidated');
+    const plan = fs.readFileSync(path.join(singleDir, 'pgpm.plan'), 'utf-8');
+    const entries = plan.split('\n').filter(l => l.trim() && !l.startsWith('%') && !l.startsWith('#'));
+    // the whole module is one plan entry...
+    expect(entries).toHaveLength(1);
+    // ...and the meaning is untouched.
+    const object = changesOf(path.join(work, 'blog'));
+    const single = changesOf(singleDir);
+    expect(diffChangeSets(object, single).identical).toBe(true);
   });
 
   it('derives the v1 -> v2 migration: exactly the real changes, nothing guessed', () => {

pgpm/cli/__tests__/diff-e2e.test.ts

@@ -324,6 +324,33 @@ describe('pgpm diff e2e', () => {
     }
   );
 
+  it('emits an alteration-change-granularity migration that reaches the same v2 catalog', async () => {
+    const pkg = 'diff-mig-alteration';
+    await fixture.runTerminalCommands(
+      `
+      cd ${WS}
+      pgpm diff diff-v1 diff-v2 --emit-migration . --pkg ${pkg} --granularity atomic --change-granularity alteration
+      `,
+      {}
+    );
+    expect(fs.existsSync(path.join(wsDir, pkg, 'pgpm.plan'))).toBe(true);
+
+    const testDb = await fixture.setupTestDatabase();
+    await fixture.runTerminalCommands(
+      `
+      cd ${WS}/diff-v1
+      pgpm deploy --database ${testDb.name} --package diff-v1 --yes
+      cd ../${pkg}
+      pgpm deploy --database ${testDb.name} --package ${pkg} --yes
+      `,
+      { database: testDb.name }
+    );
+
+    const snapMigrated = await snapshotCatalog(testDb);
+    const snapV2 = await snapshotCatalog(v2Db);
+    expect(diffCatalogSnapshots(withoutColumnOrder(snapMigrated), withoutColumnOrder(snapV2))).toEqual([]);
+  });
+
   it('--verify proves the emitted migration is a catalog-level oracle match', async () => {
     await fixture.runTerminalCommands(
       `

pgpm/cli/__tests__/transform-e2e.test.ts

@@ -215,6 +215,97 @@ describe('pgpm transform e2e', () => {
     }
   );
 
+  it('--change-granularity alteration splits per column/constraint, deploys equivalently, verifies, and reverts clean', async () => {
+    await fixture.runTerminalCommands(
+      `
+      cd ${WS}/${MODULE_NAME}
+      pgpm transform --granularity atomic --change-granularity alteration --out ../alteration-out
+      `,
+      {}
+    );
+
+    const outDir = path.join(wsDir, 'alteration-out', `${MODULE_NAME}-atomic`);
+    const plan = fs.readFileSync(path.join(outDir, 'pgpm.plan'), 'utf-8');
+    expect(plan).toContain('schemas/tfx_app/tables/users/columns/id/column');
+    expect(plan).toContain('schemas/tfx_app/tables/users/columns/email/column');
+    expect(plan).toContain('schemas/tfx_app/tables/users/constraints/users_pkey/constraint');
+    expect(plan).toContain('schemas/tfx_sec/tables/audit/constraints/audit_user_id_fkey/constraint');
+
+    // Each alteration has its own deploy/revert/verify triple.
+    const columnChange = 'schemas/tfx_app/tables/users/columns/email/column';
+    expect(fs.readFileSync(path.join(outDir, 'deploy', `${columnChange}.sql`), 'utf-8')).toContain('ADD COLUMN email');
+    expect(fs.readFileSync(path.join(outDir, 'revert', `${columnChange}.sql`), 'utf-8')).toContain('DROP COLUMN email');
+    expect(fs.readFileSync(path.join(outDir, 'verify', `${columnChange}.sql`), 'utf-8')).toContain('information_schema.columns');
+
+    const testDb = await fixture.setupTestDatabase();
+    await fixture.runTerminalCommands(
+      `
+      cd ${WS}/alteration-out/${MODULE_NAME}-atomic
+      pgpm deploy --database ${testDb.name} --package ${MODULE_NAME}-atomic --yes
+      `,
+      { database: testDb.name }
+    );
+
+    const snapOriginal = await snapshotCatalog(originalDb);
+    const snapTransformed = await snapshotCatalog(testDb);
+    expect(diffCatalogSnapshots(snapOriginal, snapTransformed)).toEqual([]);
+
+    await fixture.runTerminalCommands(
+      `
+      cd ${WS}/alteration-out/${MODULE_NAME}-atomic
+      pgpm verify --database ${testDb.name} --package ${MODULE_NAME}-atomic --yes
+      pgpm revert --database ${testDb.name} --package ${MODULE_NAME}-atomic --yes
+      `,
+      { database: testDb.name }
+    );
+    expect(await testDb.exists('schema', 'tfx_app')).toBe(false);
+  });
+
+  it('--change-granularity single collapses to one change, deploys equivalently, verifies, and reverts clean', async () => {
+    await fixture.runTerminalCommands(
+      `
+      cd ${WS}/${MODULE_NAME}
+      pgpm transform --granularity consolidated --change-granularity single --out ../single-out
+      `,
+      {}
+    );
+
+    const outDir = path.join(wsDir, 'single-out', `${MODULE_NAME}-consolidated`);
+    const plan = fs.readFileSync(path.join(outDir, 'pgpm.plan'), 'utf-8');
+    const entries = plan.split('\n').filter(l => l.trim() && !l.startsWith('%') && !l.startsWith('#'));
+    expect(entries).toHaveLength(1);
+    expect(plan).toContain('module/init');
+
+    const deploy = fs.readFileSync(path.join(outDir, 'deploy', 'module/init.sql'), 'utf-8');
+    expect(deploy).toContain('CREATE SCHEMA tfx_app');
+    expect(deploy).toContain('CREATE TABLE tfx_sec.audit');
+    expect(fs.readFileSync(path.join(outDir, 'revert', 'module/init.sql'), 'utf-8')).toContain('DROP SCHEMA tfx_app');
+    expect(fs.readFileSync(path.join(outDir, 'verify', 'module/init.sql'), 'utf-8')).toContain('information_schema');
+
+    const testDb = await fixture.setupTestDatabase();
+    await fixture.runTerminalCommands(
+      `
+      cd ${WS}/single-out/${MODULE_NAME}-consolidated
+      pgpm deploy --database ${testDb.name} --package ${MODULE_NAME}-consolidated --yes
+      `,
+      { database: testDb.name }
+    );
+
+    const snapOriginal = await snapshotCatalog(originalDb);
+    const snapTransformed = await snapshotCatalog(testDb);
+    expect(diffCatalogSnapshots(snapOriginal, snapTransformed)).toEqual([]);
+
+    await fixture.runTerminalCommands(
+      `
+      cd ${WS}/single-out/${MODULE_NAME}-consolidated
+      pgpm verify --database ${testDb.name} --package ${MODULE_NAME}-consolidated --yes
+      pgpm revert --database ${testDb.name} --package ${MODULE_NAME}-consolidated --yes
+      `,
+      { database: testDb.name }
+    );
+    expect(await testDb.exists('schema', 'tfx_app')).toBe(false);
+  });
+
   it('rewrites an existing output when --write is passed', async () => {
     // The refusal path (no --write) exits the process, so it is covered by the
     // checkOverwrite unit tests; here we prove --write re-generates in place.