Skip to content

regression/cbmc/array-constraint/test_json: un-KNOWNBUG, fix expectations#9018

Open
tautschnig wants to merge 1 commit into
diffblue:developfrom
tautschnig:array-constraint-test
Open

regression/cbmc/array-constraint/test_json: un-KNOWNBUG, fix expectations#9018
tautschnig wants to merge 1 commit into
diffblue:developfrom
tautschnig:array-constraint-test

Conversation

@tautschnig
Copy link
Copy Markdown
Collaborator

test_json.desc was added as KNOWNBUG in 2020 (commit ecb0a5d) asserting an aspirational JSON shape that cbmc never produced. Two flaws:

  1. The exit code assertion '^EXIT=0$' did not match cbmc's actual exit. main.c contains 'assert(a[index] == 1)' with a non-deterministic 'index', which fails verification — cbmc exits with 10. (The two sibling tests, test.desc and test_xml.desc, exercise '--show-array- constraints' without '--json-ui' / with '--xml-ui'; cbmc rejects those combinations and exits 1, which they correctly assert.)

  2. The regex asserted list-shaped JSON values, e.g. '"arrayWith": [.*]', but the implementation has always emitted scalar counts: '"arrayWith": 2'. The test was therefore unmatchable on day one.

Fix both: assert '^EXIT=10$' and rewrite the regex to match scalar counts ('\d+'). Drop the KNOWNBUG tag so the JSON contract for '--show-array-constraints' is now pinned by an actually-running test.

The test only applies to the default SAT backend: under '--cprover-smt2' or '--incremental-smt2-solver z3' cbmc bypasses the array theory entirely and emits no 'arrayConstraints' JSON block (and '--cprover-smt2' returns a different exit code). Tag the test 'broken-cprover-smt-backend no-new-smt' so it is skipped under those profiles, matching the convention used by other arrayst-dependent tests such as Array_operations2 / Bitfields1 / Computed-Goto1.

  • Each commit message has a non-empty body, explaining why the change was made.
  • n/a Methods or procedures I have added are documented, following the guidelines provided in CODING_STANDARD.md.
  • n/a The feature or user visible behaviour I have added or modified has been documented in the User Guide in doc/cprover-manual/
  • Regression or unit tests are included, or existing tests cover the modified code (in this case I have detailed which ones those are in the commit message).
  • n/a My commit message includes data points confirming performance improvements (if claimed).
  • My PR is restricted to a single feature or bugfix.
  • n/a White-space or formatting changes outside the feature-related changed lines are in commits of their own.

…ions

test_json.desc was added as KNOWNBUG in 2020 (commit ecb0a5d)
asserting an aspirational JSON shape that cbmc never produced. Two
flaws:

  1. The exit code assertion '^EXIT=0$' did not match cbmc's actual
     exit. main.c contains 'assert(a[index] == 1)' with a non-deterministic
     'index', which fails verification — cbmc exits with 10. (The two
     sibling tests, test.desc and test_xml.desc, exercise '--show-array-
     constraints' without '--json-ui' / with '--xml-ui'; cbmc rejects
     those combinations and exits 1, which they correctly assert.)

  2. The regex asserted list-shaped JSON values, e.g.
     '"arrayWith": [.*]', but the implementation has always emitted
     scalar counts: '"arrayWith": 2'. The test was therefore
     unmatchable on day one.

Fix both: assert '^EXIT=10$' and rewrite the regex to match scalar
counts ('\\d+'). Drop the KNOWNBUG tag so the JSON contract for
'--show-array-constraints' is now pinned by an actually-running test.

The test only applies to the default SAT backend: under
'--cprover-smt2' or '--incremental-smt2-solver z3' cbmc bypasses the
array theory entirely and emits no 'arrayConstraints' JSON block (and
'--cprover-smt2' returns a different exit code). Tag the test
'broken-cprover-smt-backend no-new-smt' so it is skipped under those
profiles, matching the convention used by other arrayst-dependent
tests such as Array_operations2 / Bitfields1 / Computed-Goto1.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
@tautschnig tautschnig self-assigned this May 27, 2026
Copilot AI review requested due to automatic review settings May 27, 2026 21:39
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR corrects the array-constraint JSON regression test so it matches CBMC’s actual behavior and can run as a normal CORE test (instead of an unmatchable KNOWNBUG), while being skipped under SMT-backend profiles where the output/exit code differs.

Changes:

  • Remove KNOWNBUG and add solver-profile skip tags (broken-cprover-smt-backend, broken-z3-smt-backend, no-new-smt).
  • Update expected exit code from 0 to 10 (verification failure).
  • Fix the expected JSON shape to match scalar counts (e.g., "arrayWith": \d+) instead of list values.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.61%. Comparing base (79f0f3a) to head (0402bc0).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9018      +/-   ##
===========================================
+ Coverage    80.59%   80.61%   +0.01%     
===========================================
  Files         1711     1711              
  Lines       189454   189454              
  Branches        73       73              
===========================================
+ Hits        152694   152727      +33     
+ Misses       36760    36727      -33     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants