Skip to content

fix(testcomp): the smoke test requires evidence that each engine ran - #87

Merged
hbgit merged 1 commit into
developfrom
fix/testcomp27-engine-evidence
Oct 25, 2026
Merged

hbgit merged 1 commit into
developfrom
fix/testcomp27-engine-evidence

Conversation

@GuilhermeBn198

Copy link
Copy Markdown
Collaborator

Summary

Addresses the review of #86: the Executing AFL++ with map2check / Executing Klee with map2check banners are logged before the engine is launched (caller.cpp), so they also appear when the engine fails to start. Since the other engine can still produce the suite, the smoke test could pass with AFL++ or KLEE unavailable. Reproduced it: with an afl-fuzz that exits at once, KLEE found Cover-Error's 42 and the old check passed.

The frontend now logs, after each engine, what the engine itself wrote:

  • AFL++ ran: N executions, M inputs in the corpus, read from execs_done and corpus_count in afl-fuzz's fuzzer_stats. With no executions it logs a warning with the last lines of fuzzer.output, which would otherwise be lost when the scratch directory is cleaned up.
  • KLEE ran: N tests generated, counted from the .ktest files in klee-last.

smoketest.sh now requires those lines with a positive count instead of the banners: AFL++ in both examples, KLEE in Cover-Branches. The README of the archive documents the two lines.

These are log lines only; no engine behaviour changes.

Test plan

Archive rebuilt from this branch (693 files, license coverage passes):

  • ubuntu:24.04 with python3 gcc-multilib: smoke test passes
  • ubuntu:26.04 (the FM-Tools CI image) with python3 gcc-multilib: smoke test passes
  • ubuntu:24.04 with only python3: fails, AFL++ did not run
  • afl-fuzz replaced by a script that exits 1 at once: fails, AFL++ did not run; the log carries AFL++ recorded no executions and the fake's own message (this scenario passed before)
  • klee replaced by a script that exits 1 at once: fails, KLEE did not run (KLEE generated no tests in the log)
  • unit tests 14/14

🤖 Generated with Claude Code

Review of #86: the "Executing AFL++/Klee with map2check" banners are logged
before the engine is launched, so they also appear when it fails to start,
and the other engine can still produce the suite the smoke test asks for.
With afl-fuzz unable to start, KLEE found Cover-Error's 42 and the check
passed.

The frontend now logs, after each engine, what the engine itself wrote:
- "AFL++ ran: N executions, M inputs in the corpus", from execs_done and
  corpus_count in afl-fuzz's fuzzer_stats; with no executions, a warning
  carrying the last lines of fuzzer.output, which the scratch cleanup
  would otherwise remove;
- "KLEE ran: N tests generated", from the .ktest files in klee-last.

smoketest.sh requires those lines with a positive count instead of the
banners: AFL++ in both examples, KLEE in Cover-Branches.

Checked on the archive rebuilt from this branch: the smoke test passes on
ubuntu:24.04 and 26.04 with python3 and gcc-multilib, and fails with only
python3, with an afl-fuzz that exits at once ("AFL++ did not run", its
output in the log) and with a klee that exits at once ("KLEE did not run").
Unit tests: 14/14.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused logging and smoke-test changes address false passes from engine startup failures, with no unresolved findings.

No open issues 🎉

What changed in this PR

Strengthens the Test-Comp archive smoke test so it verifies that each required engine actually ran, rather than relying on pre-launch banners.

Changes:

  • Logs AFL++ execution and corpus counts and KLEE-generated test counts after execution.
  • Requires positive engine counts in the smoke test and preserves diagnostic output when AFL++ records no executions.
  • Documents the new evidence lines without changing engine behavior.
File Description
utils/​testcomp-archive/​smoketest.sh Requires positive execution or test counts instead of launch banners.
utils/​testcomp-archive/​README.txt Documents engine evidence lines and smoke-test requirements.
modules/​frontend/​caller.cpp Reports engine output counts and AFL++ failure diagnostics.

Review effort: Balanced

@hbgit
hbgit merged commit cb5a8f8 into develop Oct 25, 2026
10 checks passed
@GuilhermeBn198
GuilhermeBn198 deleted the fix/testcomp27-engine-evidence branch October 25, 2026 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants