Skip to content

Fix the three defects blocking natspec spec publication - #148

Open
shellygr wants to merge 5 commits into
masterfrom
shelly/natspec-conf-path
Open

Fix the three defects blocking natspec spec publication#148
shellygr wants to merge 5 commits into
masterfrom
shelly/natspec-conf-path

Conversation

@shellygr

@shellygr shellygr commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Problem

Greenfield tui-natspec cannot publish a spec, and never could. publish is gated on a passing typecheck, and three separate defects sat in that gate, each hidden behind the one before it. The symptom is the same every time: the authoring agents write complete CVL, the feedback judge approves it, every publish attempt is rejected, the agents burn their remaining turns and give up — and the pipeline still writes the interface and the stub, so the run looks like it merely failed to produce specs.

1 & 2 — doubled certora/ segments

temp_certora_file yields a path already relative to the project root, so it carries the certora/ segment; its docstring says callers use it verbatim, "no certora/ prefixing". Two ConfigurationBuilder methods prefixed it again:

yield path / "certora" / basename                # -> <root>/certora/certora/run_<uid>.conf
verify=f"{main_contract}:certora/{spec_file}"    # -> <C>:certora/certora/generated_<uid>.spec

typecheck.py runs the CLI with cwd at the project root and validate_readable_file resolves against the CWD, so both paths were simply missing:

read_from_conf_file: /tmp/tmpXXXXXXXX/certora/certora/run_YYYY.conf: not found
attribute/flag 'verify': file certora/certora/generated_YYYY.spec not found

They mask each other — fixing the conf location just moves the failure to the second message.

3 — interfaces in the conf's files list

With the paths fixed, the CLI got far enough to assemble a scene and refused:

Contract IFoo has no bytecode. It may be caused because the contract is abstract,
or is missing constructor code.

Every entry in files must compile to bytecode. The pipeline registers only stubs, but the authoring agent also registered the interface — reasonably, since its spec references it and register_verification_file invited "any contract source the spec references". There is no unregister tool, so the session poisoned itself irrecoverably. The agent's own diagnosis, after its sixth rejected publish: "blocking error is scene assembly of the pre-registered interface-only file."

The registration also outlives the run that made it: FILES_NS is keyed by document digest, not by cache namespace, so re-running the same document under a fresh --cache-ns reuses the poisoned entry.

Fix

  1. Join the project-root-relative paths to the root without re-prefixing, in both places.
  2. FileRegistry learns which paths are interfaces: register refuses them with an explanation the agent can act on, and read_all filters them so entries persisted before this commit stay out of the conf. The tool docstring now says not to register interfaces and why. Interfaces reach the scene regardless, through the stubs' imports.

Tests

  • tests/test_natspec_conf_path.py mirrors typecheck.run_typecheck — materialize a spec, build a conf around it — and asserts the invariant both path bugs broke: every path the conf hands to the CLI must resolve, from the project root, to a file on disk.
  • tests/test_file_registry_non_units.py covers both guards, including the persisted-entry case.

Verified against each code state in the AutoProver container:

Code state Result
origin/master conf not written at yielded path: …/certora/certora/run_9c1cb9dc.conf
conf location fixed verify points at a file that was never written: certora/certora/generated_b80726c8.spec
both paths fixed scene assembly fails: Contract IFoo has no bytecode
all three fixed passes

The last row is an end-to-end run of the real typecheck gate — a generated interface + stub and rule sanity { assert true; } through typecheck_spec — which now returns clean. pytest tests/test_natspec_conf_path.py tests/test_file_registry_non_units.py → 3 passed.

Still draft pending a full greenfield pipeline run.

`temp_certora_file` yields a path already relative to the project root, so it
carries the `certora/` segment (its docstring: "callers use it verbatim (no
`certora/` prefixing)"). `ConfigurationBuilder._build_to` prefixed it a second
time and yielded `<root>/certora/certora/run_<uid>.conf`, a path nothing ever
wrote to, so every natspec typecheck died with:

    read_from_conf_file: /tmp/tmpXXXXXXXX/certora/certora/run_YYYY.conf: not found

`publish` is gated on a passing typecheck, so greenfield natspec runs could
never emit a spec: the authoring agents produced complete, judge-approved CVL,
burned their remaining turns retrying, and gave up. The pipeline still wrote
the interface and the stub, which made the failure look like missing specs
rather than a broken backend.

The sibling call in the same `with` block, `with_verify(spec_file=...)`, already
uses the yielded path verbatim.

Adds a regression test asserting `build_to` yields a path that exists and sits
under exactly one `certora` segment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@shellygr
shellygr requested a review from jtoman August 10, 2026 23:02
`with_verify` re-prefixed `certora/` onto a spec path that already carried it,
so the conf's `verify` attribute read

    <Contract>:certora/certora/generated_<uid>.spec

The Certora CLI validates that path against the process CWD (which typecheck.py
sets to the project root), so it rejected every spec with

    attribute/flag 'verify': file certora/certora/generated_<uid>.spec not found

This is the same mistake as the conf-location bug in the previous commit, one
layer up: fixing only that one moved the failure from "conf not found" to
"spec not found" without unblocking publish.

Widens the regression test to assert the invariant both bugs broke — every path
the conf hands to the CLI must resolve, from the project root, to a file on
disk. The test now fails on either bug alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@shellygr shellygr changed the title Fix doubled certora/ segment in the natspec typecheck conf path Fix doubled certora/ segments in the natspec typecheck paths Aug 11, 2026
Certora's scene assembly requires every entry in the conf's `files` to compile
to bytecode. An interface never does, so one interface entry fails the build
for every spec authored in that session:

    Contract IFoo has no bytecode. It may be caused because the contract is
    abstract, or is missing constructor code.

The pipeline registers only stubs, but the CVL-authoring agent registered the
interface too — reasonably, since its spec references it and
`register_verification_file` invited "any contract source the spec references".
There is no unregister tool, so a session poisoned itself irrecoverably: the
agent's own diagnosis after its sixth rejected publish was "blocking error is
scene assembly of the pre-registered interface-only file".

The registration also outlived the run that made it. FILES_NS is keyed by
document digest and not by cache namespace, so re-running the same document
under a fresh `--cache-ns` reused the poisoned entry. Hence two guards:
`register` refuses these paths, and `read_all` filters them so entries written
before this commit stay out of the conf.

The interfaces are in the scene regardless, via the stubs' imports. Verified by
running the real typecheck gate against a generated interface + stub with
`rule sanity { assert true; }`: with the interface excluded from `files`, the
gate passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@shellygr shellygr changed the title Fix doubled certora/ segments in the natspec typecheck paths Fix the three defects blocking natspec spec publication Aug 11, 2026
shellygr and others added 2 commits August 12, 2026 01:40
`_compile_stub` gates every stub update and trusted solc's exit status. An
`abstract contract` satisfies that happily: solc returns 0 and emits no
bytecode. Certora's scene assembly then rejects the verification unit —

    Contract StreamShareSplitter has no bytecode. It may be caused because the
    contract is abstract, or is missing constructor code.

— failing every subsequent typecheck in the session. `publish` is gated on the
typecheck, so nothing the CVL author does afterwards can recover: the bad stub
is already in the VFS and the spec is not what is wrong.

This is not hypothetical. A registry agent asked for storage fields and got
back `abstract contract StreamShareSplitter is IStreamShareSplitter`, which the
validator accepted. All four components then produced judge-approved specs that
could never be published, while the artifact dumped at the end was the pristine
concrete stub from generation — so the failure was invisible in the output.

Ask solc for the bytecode via --combined-json and require it to be non-empty.
The rejection message names the two ways a contract ends up without bytecode so
the agent can act on it within its retry loop.

Note the neighbouring case is already safe: a contract that inherits a function
it does not implement is a hard solc error ("should be marked as abstract"), so
the exit-status path catches that one. Only explicit `abstract` slipped through.
Both are pinned by tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant