Repository navigation
fix(testcomp): the smoke test requires evidence that each engine ran - #87
Merged
Merged
Conversation
Review of #86: the "Executing AFL++/Klee with map2check" banners are logged before the engine is launched, so they also appear when it fails to start, and the other engine can still produce the suite the smoke test asks for. With afl-fuzz unable to start, KLEE found Cover-Error's 42 and the check passed. The frontend now logs, after each engine, what the engine itself wrote: - "AFL++ ran: N executions, M inputs in the corpus", from execs_done and corpus_count in afl-fuzz's fuzzer_stats; with no executions, a warning carrying the last lines of fuzzer.output, which the scratch cleanup would otherwise remove; - "KLEE ran: N tests generated", from the .ktest files in klee-last. smoketest.sh requires those lines with a positive count instead of the banners: AFL++ in both examples, KLEE in Cover-Branches. Checked on the archive rebuilt from this branch: the smoke test passes on ubuntu:24.04 and 26.04 with python3 and gcc-multilib, and fails with only python3, with an afl-fuzz that exits at once ("AFL++ did not run", its output in the log) and with a klee that exits at once ("KLEE did not run"). Unit tests: 14/14. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
The focused logging and smoke-test changes address false passes from engine startup failures, with no unresolved findings.
No open issues 🎉
What changed in this PR
Strengthens the Test-Comp archive smoke test so it verifies that each required engine actually ran, rather than relying on pre-launch banners.
Changes:
- Logs AFL++ execution and corpus counts and KLEE-generated test counts after execution.
- Requires positive engine counts in the smoke test and preserves diagnostic output when AFL++ records no executions.
- Documents the new evidence lines without changing engine behavior.
| File | Description |
|---|---|
utils/testcomp-archive/smoketest.sh |
Requires positive execution or test counts instead of launch banners. |
utils/testcomp-archive/README.txt |
Documents engine evidence lines and smoke-test requirements. |
modules/frontend/caller.cpp |
Reports engine output counts and AFL++ failure diagnostics. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses the review of #86: the
Executing AFL++ with map2check/Executing Klee with map2checkbanners are logged before the engine is launched (caller.cpp), so they also appear when the engine fails to start. Since the other engine can still produce the suite, the smoke test could pass with AFL++ or KLEE unavailable. Reproduced it: with anafl-fuzzthat exits at once, KLEE found Cover-Error's42and the old check passed.The frontend now logs, after each engine, what the engine itself wrote:
AFL++ ran: N executions, M inputs in the corpus, read fromexecs_doneandcorpus_countin afl-fuzz'sfuzzer_stats. With no executions it logs a warning with the last lines offuzzer.output, which would otherwise be lost when the scratch directory is cleaned up.KLEE ran: N tests generated, counted from the.ktestfiles inklee-last.smoketest.shnow requires those lines with a positive count instead of the banners: AFL++ in both examples, KLEE in Cover-Branches. The README of the archive documents the two lines.These are log lines only; no engine behaviour changes.
Test plan
Archive rebuilt from this branch (693 files, license coverage passes):
ubuntu:24.04withpython3 gcc-multilib: smoke test passesubuntu:26.04(the FM-Tools CI image) withpython3 gcc-multilib: smoke test passesubuntu:24.04with onlypython3: fails,AFL++ did not runafl-fuzzreplaced by a script that exits 1 at once: fails,AFL++ did not run; the log carriesAFL++ recorded no executionsand the fake's own message (this scenario passed before)kleereplaced by a script that exits 1 at once: fails,KLEE did not run(KLEE generated no testsin the log)🤖 Generated with Claude Code