Skip to content

[TS] Measure source-replayed branch coverage for unknown-call policies #414

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
CaelmBleidd merged 2 commits into from
Oct 13, 2026
Merged

[TS] Measure source-replayed branch coverage for unknown-call policies #414

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0eab090
Collect and replay coverage candidates through current Calls preparation
CaelmBleidd Oct 12, 2026
19b1184
Bind coverage replay to frozen source and pre-call inputs
CaelmBleidd Oct 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
216 changes: 216 additions & 0 deletions usvm-ts-calls/src/main/kotlin/org/usvm/ts/calls/CallsCoverageCell.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
package org.usvm.ts.calls

import kotlinx.serialization.Serializable
import kotlinx.serialization.decodeFromString
import org.usvm.ts.pbt.model.encodeToUtf8SafeString
import java.nio.ByteBuffer
import java.nio.channels.FileChannel
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.StandardCopyOption
import java.nio.file.StandardOpenOption
import kotlin.time.Duration.Companion.milliseconds

@Serializable
internal data class CallsCoverageCellIdentity(
val schemaVersion: Int,
val toolRevision: String,
val nativeFrontendRevision: String,
val sourceManifest: String,
val sourceIdentity: CallsCoverageSourceIdentity,
val projectIndex: Int,
val functionIndex: Int,
val profile: CallsExperimentProfile,
val seed: Long,
val budgetMillis: Long,
val solverLimitMillis: Long,
val candidateCap: Int,
)

@Serializable
internal data class CallsCoverageReplayEvent(
val index: Int,
val replay: CallsCoverageReplayResult? = null,
val error: String? = null,
)

@Serializable
internal data class CallsCoverageCheckpoint(
val searchBudgetMillis: Long,
val emittedCandidates: Int,
val replayedCandidates: Int,
val coveredStatements: Int,
val coveredIfArms: Int,
)

@Serializable
internal data class CallsCoverageCellResult(
val identity: CallsCoverageCellIdentity,
val search: CallsCoverageSearchResult,
val replayErrors: Int,
val completionMismatches: Int,
val checkpoints: List<CallsCoverageCheckpoint>,
)

internal fun runCoverageCell(

Check warning

Code scanning / detekt

One method should have one responsibility. Long methods tend to handle many things at once. Prefer smaller methods to make them easier to understand. Warning

The function runCoverageCell is too long (125). The maximum length is 100.
args: List<String>,
runtimeToolRevision: String = CallsBuildIdentity.toolRevision,
runtimeNativeFrontendRevision: String = CallsBuildIdentity.nativeFrontendRevision,
searchFunction: (CallsCoverageSearchRequest) -> CallsCoverageSearchResult = CurrentTsCallsCoverageEngine()::search,
) {
require(args.size == 9) { "coverage-cell requires 9 arguments" }

Check warning

Code scanning / detekt

Report magic numbers. Magic number is a numeric literal that is not defined as a constant and hence it's unclear what the purpose of this number is. It's better to declare such numbers as constants and give them a proper name. By default, -1, 0, 1, and 2 are not considered to be magic numbers. Warning

This expression contains a magic number. Consider defining it to a well named constant.
val projectIndex = args[1].toInt()
val functionIndex = args[2].toInt()
val source = CallsCoverageSource.load(Path.of(args[0]), projectIndex, functionIndex)
val manifestPath = source.manifestPath
val manifest = source.manifest
val profile = CallsExperimentProfile.valueOf(args[3])

Check warning

Code scanning / detekt

Report magic numbers. Magic number is a numeric literal that is not defined as a constant and hence it's unclear what the purpose of this number is. It's better to declare such numbers as constants and give them a proper name. By default, -1, 0, 1, and 2 are not considered to be magic numbers. Warning

This expression contains a magic number. Consider defining it to a well named constant.
val seed = args[4].toLong()

Check warning

Code scanning / detekt

Report magic numbers. Magic number is a numeric literal that is not defined as a constant and hence it's unclear what the purpose of this number is. It's better to declare such numbers as constants and give them a proper name. By default, -1, 0, 1, and 2 are not considered to be magic numbers. Warning

This expression contains a magic number. Consider defining it to a well named constant.
val output = Path.of(args[5]).toAbsolutePath().normalize()

Check warning

Code scanning / detekt

Report magic numbers. Magic number is a numeric literal that is not defined as a constant and hence it's unclear what the purpose of this number is. It's better to declare such numbers as constants and give them a proper name. By default, -1, 0, 1, and 2 are not considered to be magic numbers. Warning

This expression contains a magic number. Consider defining it to a well named constant.
val budgetMillis = args[6].toLong()

Check warning

Code scanning / detekt

Report magic numbers. Magic number is a numeric literal that is not defined as a constant and hence it's unclear what the purpose of this number is. It's better to declare such numbers as constants and give them a proper name. By default, -1, 0, 1, and 2 are not considered to be magic numbers. Warning

This expression contains a magic number. Consider defining it to a well named constant.
val solverLimitMillis = args[7].toLong()

Check warning

Code scanning / detekt

Report magic numbers. Magic number is a numeric literal that is not defined as a constant and hence it's unclear what the purpose of this number is. It's better to declare such numbers as constants and give them a proper name. By default, -1, 0, 1, and 2 are not considered to be magic numbers. Warning

This expression contains a magic number. Consider defining it to a well named constant.
val candidateCap = args[8].toInt()

Check warning

Code scanning / detekt

Report magic numbers. Magic number is a numeric literal that is not defined as a constant and hence it's unclear what the purpose of this number is. It's better to declare such numbers as constants and give them a proper name. By default, -1, 0, 1, and 2 are not considered to be magic numbers. Warning

This expression contains a magic number. Consider defining it to a well named constant.
val project = source.project
val function = source.function
require(seed in manifest.seeds) { "Seed is outside the frozen source manifest" }
require(budgetMillis > 0 && solverLimitMillis in 1..budgetMillis && candidateCap > 0)
require(System.getenv("ETS_FRONTEND_SCRIPT") == null) { "ETS_FRONTEND_SCRIPT must be unset" }

val identity = CallsCoverageCellIdentity(
schemaVersion = 2,
toolRevision = runtimeToolRevision,
nativeFrontendRevision = runtimeNativeFrontendRevision,
sourceManifest = manifestPath.toString(),
sourceIdentity = source.identity,
projectIndex = projectIndex,
functionIndex = functionIndex,
profile = profile,
seed = seed,
budgetMillis = budgetMillis,
solverLimitMillis = solverLimitMillis,
candidateCap = candidateCap,
)
require(Regex("[0-9a-f]{40}").matches(identity.toolRevision)) { "Coverage build must be clean" }

Files.createDirectories(output)
val identityPath = output.resolve("identity.json")
val expectedIdentity = encode(identity)
if (Files.exists(identityPath)) {
require(Files.readString(identityPath).trim() == expectedIdentity) { "Cell identity changed" }
} else {
require(Files.list(output).use { entries -> entries.findAny().isEmpty }) { "Output directory is not empty" }
atomicWrite(identityPath, expectedIdentity)
}
require(!Files.exists(output.resolve("result.json"))) { "Cell already completed" }

val candidatePath = output.resolve("candidates.jsonl")
val searchPath = output.resolve("search.json")
val search = if (Files.exists(searchPath)) {
CallsExperimentJson.json.decodeFromString<CallsCoverageSearchResult>(Files.readString(searchPath))
} else {
require(!Files.exists(candidatePath)) { "Interrupted search requires investigation before restart" }
source.verifyUnchanged()
val request = CallsCoverageSearchRequest(
sourceRoot = source.sourceRoot,
project = project,
function = function,
profile = profile,
frozenModelIds = manifest.modelSet.ids,
expectedNativeFrontendRevision = identity.nativeFrontendRevision,
seed = seed,
budget = budgetMillis.milliseconds,
solverQueryLimit = solverLimitMillis.milliseconds,
candidateCap = candidateCap,
onCandidate = { candidate -> appendDurably(candidatePath, encode(candidate)) },
)
val result = searchFunction(request)
source.verifyUnchanged()
val persistedCandidates = readLines<CallsCoverageCandidate>(candidatePath)
require(result.candidates == persistedCandidates) { "Persisted candidate journal differs from search result" }
atomicWrite(searchPath, encode(result))
result
}

val replayPath = output.resolve("replays.jsonl")
val events = readLines<CallsCoverageReplayEvent>(replayPath).toMutableList()
require(events.map(CallsCoverageReplayEvent::index) == events.indices.toList()) {
"Replay journal has missing or duplicate indices"
}
source.verifyUnchanged()
if (search.candidates.isNotEmpty()) {
OriginalTypeScriptCoverageReplayer(
sourceRoot = source.sourceRoot,
function = function,
).use { replayer ->
search.candidates.indices.drop(events.size).forEach { index ->
val candidate = search.candidates[index]
source.verifyUnchanged()
val replay = runCatching {
replayer.replay(inputs = candidate.inputs, timeoutMillis = REPLAY_TIMEOUT_MILLIS)
}
source.verifyUnchanged()
val event = CallsCoverageReplayEvent(
index = index,
replay = replay.getOrNull(),
error = replay.exceptionOrNull()?.let { error -> error.message ?: error::class.java.name },
)
appendDurably(replayPath, encode(event))
events += event
}
}
}

source.verifyUnchanged()
val checkpoints = listOf(30_000L, 60_000L, 120_000L).filter { it <= budgetMillis }.map { checkpoint ->

Check warning

Code scanning / detekt

Report magic numbers. Magic number is a numeric literal that is not defined as a constant and hence it's unclear what the purpose of this number is. It's better to declare such numbers as constants and give them a proper name. By default, -1, 0, 1, and 2 are not considered to be magic numbers. Warning

This expression contains a magic number. Consider defining it to a well named constant.

Check warning

Code scanning / detekt

Report magic numbers. Magic number is a numeric literal that is not defined as a constant and hence it's unclear what the purpose of this number is. It's better to declare such numbers as constants and give them a proper name. By default, -1, 0, 1, and 2 are not considered to be magic numbers. Warning

This expression contains a magic number. Consider defining it to a well named constant.

Check warning

Code scanning / detekt

Report magic numbers. Magic number is a numeric literal that is not defined as a constant and hence it's unclear what the purpose of this number is. It's better to declare such numbers as constants and give them a proper name. By default, -1, 0, 1, and 2 are not considered to be magic numbers. Warning

This expression contains a magic number. Consider defining it to a well named constant.
val indices = search.candidates.indices.filter { index ->
search.candidates[index].emittedAtMillis <= checkpoint
}
val replayed = indices.mapNotNull { index -> events[index].replay }
CallsCoverageCheckpoint(
searchBudgetMillis = checkpoint,
emittedCandidates = indices.size,
replayedCandidates = replayed.size,
coveredStatements = replayed.flatMap { it.coveredStatementKeys }.toSet().size,
coveredIfArms = replayed.flatMap { it.coveredIfArmKeys }.toSet().size,
)
}
val result = CallsCoverageCellResult(
identity = identity,
search = search,
replayErrors = events.count { it.error != null },
completionMismatches = events.indices.count { index ->
events[index].replay?.completion != null &&
events[index].replay?.completion != search.candidates[index].completion
},
checkpoints = checkpoints,
)
atomicWrite(output.resolve("result.json"), encode(result))
println(encode(result))

Check warning

Code scanning / detekt

Mark forbidden methods. A forbidden method could be an invocation of an unstable / experimental method and hence you might want to mark it as forbidden in order to get warned about the usage. Warning

The method kotlin.io.println has been forbidden: println does not allow you to configure the output stream. Use a logger instead.
}

private inline fun <reified T> readLines(path: Path): List<T> = if (Files.exists(path)) {
Files.readAllLines(path).filter(String::isNotBlank).map { line ->
CallsExperimentJson.json.decodeFromString<T>(line)
}
} else {
emptyList()
}

private inline fun <reified T> encode(value: T): String = CallsExperimentJson.json.encodeToUtf8SafeString(value)

private fun appendDurably(path: Path, line: String) {
val bytes = ByteBuffer.wrap((line + "\n").toByteArray(Charsets.UTF_8))
FileChannel.open(path, StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.APPEND).use { channel ->

Check warning

Code scanning / detekt

Reports lines with exceeded length Warning

Exceeded max line length (120)

Check warning

Code scanning / detekt

Line detected, which is longer than the defined maximum line length in the code style. Warning

Line detected, which is longer than the defined maximum line length in the code style.

Check warning

Code scanning / detekt

Reports incorrect argument list wrapping Warning

Argument should be on a separate line (unless all arguments can fit a single line)

Check warning

Code scanning / detekt

Reports incorrect argument list wrapping Warning

Argument should be on a separate line (unless all arguments can fit a single line)

Check warning

Code scanning / detekt

Reports incorrect argument list wrapping Warning

Argument should be on a separate line (unless all arguments can fit a single line)

Check warning

Code scanning / detekt

Reports incorrect argument list wrapping Warning

Argument should be on a separate line (unless all arguments can fit a single line)

Check warning

Code scanning / detekt

Reports incorrect argument list wrapping Warning

Missing newline before ")"
while (bytes.hasRemaining()) channel.write(bytes)
channel.force(true)
}
}

private fun atomicWrite(path: Path, contents: String) {
val temporary = path.resolveSibling("${path.fileName}.tmp")
Files.writeString(temporary, contents + "\n")
FileChannel.open(temporary, StandardOpenOption.READ).use { it.force(true) }
Files.move(temporary, path, StandardCopyOption.ATOMIC_MOVE)
}

private const val REPLAY_TIMEOUT_MILLIS = 20_000L
Loading
Loading