Skip to content

make test / installcheck never fails the build, even in CI — .IGNORE: installcheck swallows all pg_regress failures #49

Description

@jnasbyupgrade

`base.mk` has:

```make

Don't have installcheck bomb on error

.IGNORE: installcheck
installcheck: $(TEST_RESULT_FILES) $(TEST_SQL_FILES) | ...

test: $(TEST_DEPS)
@if [ -r $(TESTOUT)/regression.diffs ]; then cat $(TESTOUT)/regression.diffs; fi
```

`.IGNORE: installcheck` makes `make` treat `installcheck` as always successful
regardless of `pg_regress`'s actual exit status, and `test`'s recipe only
`cat`s `regression.diffs` for visibility — it never checks whether that file
is non-empty or whether `installcheck` "failed." So `make test` always exits
0, even when every single test fails.

This isn't just a local annoyance — it's silently defeating CI. Confirmed on
two real PRs in extension_tools (Postgres-Extensions/extension_tools#6 and #7):
every `pg_regress` test failed (`not ok 1`, `not ok 2`, `not ok 3`) in every
PG-version job's logs, yet every job and the overall check both reported
`"conclusion":"success"`. A completely broken extension merges with a green
checkmark.

The `verify-results` safeguard added in 2.0.0 doesn't help here — it only
guards `make results` (`results: verify-results`), preventing you from
blessing failing output as new expected output. It's never invoked by
`make test`/`installcheck` itself, so CI remains blind.

Suggest: make `test`/`installcheck` actually fail (exit non-zero) when
`regression.diffs` is non-empty or `pg_regress`'s own exit status is
non-zero, while still printing the diff for visibility. If `.IGNORE` is there
specifically to let `test`'s recipe run and print diffs even after a
failure, that's fine — just make the recipe itself `exit 1` afterward when a
real failure occurred, similar to how `verify-results`'s non-pgtap branch
already detects this.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions