From c67a10bdd06ecf0b04a9e1d3b7efb109140639e7 Mon Sep 17 00:00:00 2001 From: Rittick Basak <49360849+r0ny123@users.noreply.github.com> Date: Mon, 26 Oct 2026 10:17:27 +0000 Subject: [PATCH 1/5] labels: audit and harden Delphi recovery --- CHANGELOG.md | 14 + profiling/delphi_audit/README.md | 249 ++++ profiling/delphi_audit/audit.py | 306 +++++ profiling/delphi_audit/manifest.json | 81 ++ src/smda/SmdaConfig.py | 1 + src/smda/common/FunctionCandidateManager.py | 3 +- src/smda/common/LanguageAnalyzer.py | 14 +- src/smda/common/RecursiveDisassembler.py | 15 +- .../labelprovider/DelphiPythiaProvider.py | 180 ++- .../labelprovider/DelphiReSymProvider.py | 808 ++++--------- .../common/labelprovider/DelphiVmtScanner.py | 33 + src/smda/intel/X86Backend.py | 44 + tests/delphi_corpus/INNO-LICENSE.txt | 32 + tests/delphi_corpus/README.md | 25 + .../delphi_corpus/inno-7.1.0-finally-32.json | 11 + .../delphi_corpus/inno-7.1.0-tobject-32.json | 948 +++++++++++++++ .../delphi_corpus/inno-7.1.0-tobject-64.json | 1043 +++++++++++++++++ tests/testDelphiFinallyContinuation.py | 96 ++ tests/testDelphiReSymProvider.py | 554 +++++++++ tests/testLabelProviderBoundary.py | 5 +- 20 files changed, 3782 insertions(+), 680 deletions(-) create mode 100644 profiling/delphi_audit/README.md create mode 100644 profiling/delphi_audit/audit.py create mode 100644 profiling/delphi_audit/manifest.json create mode 100644 src/smda/common/labelprovider/DelphiVmtScanner.py create mode 100644 tests/delphi_corpus/INNO-LICENSE.txt create mode 100644 tests/delphi_corpus/README.md create mode 100644 tests/delphi_corpus/inno-7.1.0-finally-32.json create mode 100644 tests/delphi_corpus/inno-7.1.0-tobject-32.json create mode 100644 tests/delphi_corpus/inno-7.1.0-tobject-64.json create mode 100644 tests/testDelphiFinallyContinuation.py create mode 100644 tests/testDelphiReSymProvider.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 517465db..88585058 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,10 +79,24 @@ past roughly six lines it belongs in the PR the entry links. ### Removed ### Fixed +- **(labels)** Repair Delphi class RTTI offsets, published/extended method-table traversal, record bounds, + Unicode names, executable-target validation and Pythia table bounds; preserve Delphi labels when + supplementing a PDB, and prefer explicit PDB names over generic PE entry-point labels. Cache shared + legacy tables, share provider results and use a dependency-free pointer-suffix scan. *Measured on the supplied + `1d4e5452…5681c` sample against `04652e6`:* 292 usable labels become 7,969; five-run median provider + time falls from 1.100s to 0.084s, with lower peak Python allocation. Real metadata excerpts and + malformed-input tests are bundled; the full binary is private. See `profiling/delphi_audit/`. +- **(intel)** Recover normal-flow continuations from the validated 32-bit Delphi finally-dispatch idiom. + Constructor tails previously ended at `pop eax; jmp eax`; match the saved continuation and linked + handler thunk before following the missing edge. Real and generated regression fixtures are bundled. ### Security ### Compatibility +- **(labels)** Delphi names gain correct unit/class qualification and validated signatures. +- **(labels)** An explicit PDB symbol now takes precedence over PE's synthetic `original_entry_point` label + at the same address. +- **(intel)** Delphi finally continuations extend owning function bodies and can change CFG fingerprints. ## [v5.0.1] - 2026-10-20 diff --git a/profiling/delphi_audit/README.md b/profiling/delphi_audit/README.md new file mode 100644 index 00000000..47cb7661 --- /dev/null +++ b/profiling/delphi_audit/README.md @@ -0,0 +1,249 @@ +# Delphi integration audit — 2026-10-26 + +Baseline: SMDA `04652e6` (5.0.1). Upstream: DelphiReSym +`22ad5f7cbe2b5f07487df96a9dfec1566e9a96b5`. Independent reference: the official +Ghidra **12.2** release, with its published ZIP SHA-256 verified. Java 25 and +PyGhidra 3.2.0 were used. None of the input executables were executed. + +## Integration and provenance + +`e1ca7b1` imported the algorithm in November 2025. `02be949` added the namespace +parser and purported upstream RTTI validation in January 2026. That adaptation +invented a byte between the class name and the first class-RTTI pointer. +`0d7dc3c` subsequently reused that shifted pointer offset to locate the namespace, +turning a pointer-validation error into a namespace-decoding error as well. +The original upstream fix, `45f9cb1`, caught inaccessible RTTI and removed the +associated false-positive VMT; it did not add this byte. `e6b4176` only regrouped +the arithmetic, without correcting the layout. + +SMDA maps PE sections into an RVA-indexed buffer before invoking the providers. +Subtracting `base_addr` is correct for this buffer, but it does **not** convert a +VA into an on-disk file offset. The provider now documents this distinction. +`DelphiReSymProvider` supplies names and extended method-table entry points; +`DelphiPythiaProvider` also supplies legacy, fixed, virtual, dynamic and interface +method candidates. The recursive engine and language analyzer previously created +separate instances and repeated both Delphi scans. The analyzer now shares the +engine's updated providers, while retaining standalone operation. + +Ghidra classes, namespaces, data types, VMT structures and signature objects are +not SMDA report objects. Porting their mutation code would not preserve those +semantics in SMDA. Qualified names and signatures remain flat symbol labels; +function bodies, block edges and fingerprints remain the recursive engine's job. + +## Confirmed problems and fixes + +| Problem | Origin and evidence | Resolution | +|---|---|---| +| Class RTTI pointers and unit namespace shifted by one byte | Port regression; thesis table 8, upstream traversal and both real architecture fixtures agree | Use the encoded name's byte length, then the two packed pointers and property-count word | +| Mostly unusable labels despite apparently successful extraction | Shifted namespace decoded binary data into strings; on the uploaded sample only 292 of 7,955 raw entries survived `getSymbol()` | Reject corrupt components before inserting symbols; preserve printable names | +| UTF-8 names corrupt following field offsets | Upstream reads individual bytes as characters; decoded character count differs from encoded byte count | Decode UTF-8, with Windows-1252 fallback; retain byte lengths for all cursor arithmetic | +| Published entries misread as an extended-reference count/table | Upstream and port assume the first method-table count is zero | Traverse length-delimited published entries before reading the extended count | +| Parsing beyond a method's declared length | Both implementations ignore the method-length boundary | Bound names, return metadata and parameters to that record; retain a valid method name if optional signature data is corrupt | +| Non-executable method addresses become symbol candidates | Neither original extractor validates every recovered method target | Validate against all executable ranges; reject data targets, including upstream's `TCustomDecompressor.Reset` entry in both Inno 7.1 stubs | +| First executable region treated as the entire image | Upstream `.text` assumption and port's `code_areas[0]` shortcut | Scan mapped metadata independently of code permissions; use merged code ranges only for function targets | +| MDT must follow its VMT and lie in code | Inherited heuristic, not a format requirement | Accept mapped metadata before or after the VMT; validate VMT class identity and method targets instead | +| Unreadable VMT fields silently omitted from validation | Inherited permissive validation | Require a complete header, class name, fixed method slots and consistent class RTTI when present | +| External ancestor or optional table pointer discards otherwise usable VMT methods | Port-specific ReSym check and inherited Pythia metadata check | Do not dereference external pointers; keep fixed in-image slots and parse only bounded mapped tables | +| Shared legacy tables are reparsed for every class; malformed method names can cross table records | Pythia port; especially costly on aliases and unsafe on damaged metadata | Cache table walks for one provider update and bound each name to its declared record | +| Pythia layout heuristics can run on non-x86 images; interface pointer walks have no bound | Port-specific architecture and hostile-input gaps | Gate to Intel binaries and cap interface walks at 4,096 entries | +| Empty parameter list deletes an entire upstream class | Upstream `if not params: del ...`; observed additional static `TThread` methods | Preserve zero-parameter methods; reject only the defective optional signature | +| Failed non-null return-type lookup represented as `void` | Port fallback loses the distinction between absent and unresolved type information | Use `?` for an unresolved return type | +| Duplicate names/corrupt parameter sets | Upstream issues #6/#8; invalid Delphi signatures | Drop the defective signature, preserve the method and continue the binary | +| Valid named `tkUnknown` types rejected | Official `TTypeKind` includes zero | Preserve their available names without guessing a layout | +| Hostile counts and repeated references can multiply work | Count fields and aliases are attacker-controlled | Bound record traversal and cache repeated type lookups; configurable `MAX_DELPHI_METADATA_ENTRIES`, default 1,000,000; zero disables extraction | +| Partial results lost when the metadata budget expires | Budget enforcement must bound parsing, not discard completed records | Preserve already parsed method names and stop subsequent traversal | +| Shared code acquires a later derived class's name | Multiple metadata names can identify one address | Deterministic first-qualified-name precedence, preferring available signature information; one address still has one report label | +| PDB supplementation clears Delphi and other non-PDB providers | Engine routes the empty PDB-only buffer to every provider | Route PDB updates only to the PDB provider; test Delphi label survival | +| Valid ReSym-only classes do not enable Delphi-specific recovery | Language evidence previously consulted only Pythia | Reuse validated ReSym evidence when the legacy scan finds nothing | +| Missing normal-flow continuation after a Delphi finally block | Independently exposed by Ghidra constructor bodies and checked in machine code | In 32-bit structurally identified Delphi, match the saved continuation, `pop eax; jmp eax`, handler thunk and cleanup back-edge; recover the continuation in its owning function | + +The finally resolver deliberately requires a local saved continuation and rejects +explicit stack mutations inside the matched cleanup. It does not invent exception +edges, absorb the separate handler thunk into its caller, or claim to reconstruct +all Delphi EH personalities. Runtime or imported calls with unknown stack effects +remain a limitation of static CFG reconstruction. + +## Upstream issues and pull requests + +All 12 issue/PR records, including open and closed entries and their available +comments, were reviewed on the audit date. Closed PR patches were also compared +against repository history. Numbers link to the upstream discussion. + +| Record | Finding and applicability | +|---|---| +| [#1](https://github.com/WenzWenzWenz/DelphiReSym/pull/1), closed | Cancellation/types. Ghidra monitor integration is not an SMDA API. SMDA's parsing now has its own bounded metadata work. | +| [#2](https://github.com/WenzWenzWenz/DelphiReSym/issues/2), open | Ghidra template display simplification. No SMDA display setting to port; complete generic names are preserved. | +| [#3](https://github.com/WenzWenzWenz/DelphiReSym/pull/3), closed | Type/VT construction collaboration. Ghidra data-type behavior is not represented by SMDA's flat labels. | +| [#4](https://github.com/WenzWenzWenz/DelphiReSym/pull/4), closed | Shifted VT pointer types. SMDA does not create these Ghidra pointer types. | +| [#5](https://github.com/WenzWenzWenz/DelphiReSym/issues/5), closed | Delphi 7 uses `CODE`, not `.text`, and older tables. Its attached binary was included in the audit; SMDA retains legacy recovery through Pythia. | +| [#6](https://github.com/WenzWenzWenz/DelphiReSym/issues/6), closed | Malformed duplicate parameters abort Ghidra signature application. Structural record checks and per-method degradation address the parser side. | +| [#7](https://github.com/WenzWenzWenz/DelphiReSym/issues/7), closed | Per-byte JPype crossings dominate scanning. SMDA has no JPype crossing, but had its own per-byte Python reader overhead. | +| [#8](https://github.com/WenzWenzWenz/DelphiReSym/pull/8), merged | Catch `DuplicateNameException`. No equivalent Ghidra exception is needed in SMDA. | +| [#9](https://github.com/WenzWenzWenz/DelphiReSym/pull/9), merged | Bulk NumPy scan, limited to 32-bit, with bytewise fallback. Replaced in SMDA by a dependency-free exact candidate filter for both widths. | +| [#10](https://github.com/WenzWenzWenz/DelphiReSym/issues/10), open | Non-ASCII/misdecoded names abort renaming. Decode names appropriately and preserve Unicode in SMDA; bound and validate encoded strings. | +| [#11](https://github.com/WenzWenzWenz/DelphiReSym/issues/11), open | Final x64 VT pointer represented as four bytes in Ghidra. This concerns Ghidra structure construction, which SMDA does not port. | +| [#12](https://github.com/WenzWenzWenz/DelphiReSym/pull/12), open | Proposed ASCII-only cleanup removes underscores, generic syntax and non-ASCII characters. Not adopted: it destroys valid names and can cause collisions. | + +The upstream Ghidra type table also maps `UInt64` to a four-byte +`UnsignedIntegerDataType`, and maps Delphi `string` to a pointer to `Char` despite +the Unicode string representation. SMDA does not inherit these errors because it +does not construct Ghidra data types; its output records the parsed type name only. + +## Papers and technical evidence + +Read Wenz's *Towards Ghidra Reverse Engineering Tools for Delphi Executables* +(thesis, November 22, 2025), including format tables 2–10, methodology, tool design, +evaluation and limitations, and the associated August 18, 2025 presentation. +The Git repository contains Git LFS pointers for both PDFs; the actual PDF objects +were retrieved, rather than treating those pointer files as documents. + +- [Thesis](https://github.com/WenzWenzWenz/DelphiReSym/blob/main/Academic_work_v2.pdf), + SHA-256 `9ecb96009a6c136328a38ae94bbb4d53f4a9bcd4983f338ac3126ba4a1f5f080`. +- [Slides](https://github.com/WenzWenzWenz/DelphiReSym/blob/main/Presentation_slides_v1.pdf). +- [Official TTypeKind documentation](https://docwiki.embarcadero.com/Libraries/Athens/en/System.TypInfo.TTypeKind). +- Tagged Inno Setup build READMEs establish Delphi 2009 Update 3, 10.3.3 Rio and + 12.3 Athens compiler provenance; see the manifest's evidence URLs. + +The thesis explicitly calls its evaluation **transformation coverage, not semantic +correctness**. Its approximately 31–54% name-recovery figures are not precision or +recall against source-level ground truth. Its claim that all Delphi 2010–12 layouts +are compatible also rests on sampled versions, with missing 2009/2010 binaries. +Neither claim substitutes for validation of this port. + +## Empirical results + +The eight-binary corpus includes 32/64-bit release executables, real VCL and RTL +metadata, legacy `CODE` images, signed installers with overlays, and the supplied +sample. Public source/download URLs, executable SHA-256 values and archive hashes +are in [manifest.json](manifest.json). The uploaded binary and complete third-party +executables are not redistributed in this repository. + +The uploaded sample is SHA-256 +`1d4e54529feef53850f97f39029a906d53f3d4b2aea8373e27c413324a55681c`: + +- Original raw metadata entries: **7,955**, with only **292 usable labels**. +- Revised metadata entries: **7,969**, all printable and usable; every entry is + recovered as an SMDA function start. +- Unmodified upstream extractor in Ghidra: **7,911** named entries. All are retained + by SMDA, with zero namespace disagreements on the common entries. SMDA adds 58. +- Stock Ghidra 12.2: **8,336** functions. Metadata gives independent evidence for + functions absent from its default recursive analysis. +- Metadata fixes alone preserve all **20,486** original SMDA functions and CFGs. + The finally-continuation correction subsequently recovers additional reachable + code, producing **20,950** functions. This is a deliberate CFG/fingerprint change. + +The Inno Setup 5.6.1 executable is tagged as Delphi 2009 Update 3, but only 84,992 +of its 1,963,136 bytes map into the PE image. Its code section is named `CODE`; +SMDA scans it and rejects three self-pointer lookalikes because their VMT fields +are machine code, recovering no Delphi labels. Upstream stops earlier at its +`.text`-only lookup. This checks the section-name edge case from issue #5; the +small mapped stub does not establish Delphi 2009 method-table coverage. + +The Delphi 7 executable attached to upstream issue #5 also has a `CODE` section +and no `.text`; the unmodified extractor exits with `.text segment not found`. +SMDA's legacy Pythia path yields 3,873 method candidates, 224 with usable names, +while ReSym yields none on this older layout. Ghidra 12.2 reports 6,537 functions. +This exercises the legacy recovery path on the actual issue binary. + +The upstream comparison invokes its unmodified extraction functions inside Ghidra; +it does **not** count partially successful GUI renaming, overwrite types, or treat +Ghidra's process exit status as proof that a script succeeded. The upstream README +asks users to run its mutating script twice; this audit's extraction comparison +requires no second mutation pass. + +Machine-readable counts and disagreement examples are in [results.json](results.json). +Function-count differences are **not** automatically false positives or negatives. +For example, the supplied sample's compressed switch tables contain real cases +missing from Ghidra's recovered bodies; conversely, Ghidra exposed the constructor +continuation bug fixed here. The saved-continuation pattern also establishes +normal-flow blocks absent from Ghidra's analysis, so raw agreement can decrease +when SMDA becomes more complete. No source-ground-truth precision percentage is +claimed. + +## Performance + +The original ReSym scanner read and unpacked every byte position. The optimized scanner +partitions candidate addresses into windows within which the upper pointer bytes +are constant. Native `bytes.find` locates that necessary suffix, after which the +entire pointer and VMT are checked. Carry boundaries, the final readable pointer, +unaligned candidates and pointer-width overflow are covered by bytewise-oracle tests. +There is no NumPy dependency or input-sized temporary vector. + +Other measured changes: native `unpack_from` avoids pointer slices; entire VMT +headers are unpacked together; string printability is checked in C; repeated RTTI +lookups are cached only for the current update; temporary records use slots; +redundant namespace validation/parser construction and an unused inheritance graph +are removed; legacy metadata tables are deduplicated; and engine/analyzer scans +share provider results. Long-lived per-method/table caches were tried and removed +after profiling showed a memory regression. + +On five-run medians in this environment, the supplied sample's ReSym provider pass +fell from **1.100 s to 0.084 s**, and 64-bit HeidiSQL from **5.467 s to 0.209 s**. +These are provider timings, not whole-disassembler speedups, and include accuracy +fixes. A separate alternating experiment against the previous corrected scanner +asserted identical symbol dictionaries: the new scan reduced that revision's total +provider time by approximately 2.8x on the upload and 6x on 64-bit HeidiSQL. +Pythia's complete provider pass fell from 0.129 s to 0.056 s on the upload, +0.342 s to 0.148 s on 64-bit HeidiSQL, and 0.075 s to 0.027 s on the Delphi 7 +binary. All eight maps match the original implementation exactly after the +record-boundary and table-cache changes. Candidate counts include unnamed method +addresses: the upload has 4,840 Pythia candidates, of which 14 carry names; ReSym +recovers 7,969 named methods there. + +Tracemalloc/cProfile on the upload measured peak Python allocation of approximately +**2.69 MB**, versus **2.87 MB** in the original provider. Its per-byte reader had +3.11 million calls; the final implementation makes about **51,600** pointer-reader +calls after the native suffix filter. RTTI traversal +calls fell from 22,567 to 2,989. Profiling overhead is excluded from timing medians. +These are measured improvements, not a proof of globally optimal performance. + +## Reproduction + +Activate SMDA's `.venv`. The harness executes each input in a fresh worker so large +reports and JVM state do not accumulate across the corpus. Profile mode reports +both ReSym and Pythia calls and peak Python allocation. It verifies downloaded +archive and executable digests and never executes an input binary. + +```bash +python profiling/delphi_audit/audit.py --download --corpus /tmp/delphi-corpus --output /tmp/delphi-provider --repeat 5 +python profiling/delphi_audit/audit.py --corpus /tmp/delphi-corpus --output /tmp/delphi-baseline --source /path/to/baseline/src --repeat 5 +python profiling/delphi_audit/audit.py --corpus /tmp/delphi-corpus --output /tmp/delphi-native --mode native +python profiling/delphi_audit/audit.py --corpus /tmp/delphi-corpus --output /tmp/delphi-profile --mode profile +``` + +Install `pyghidra` separately and provide Java 25 plus the verified Ghidra 12.2 +installation for independent analysis. NumPy is optional only for reproducing the +upstream extractor's own 32-bit optimization. + +```bash +JAVA_TOOL_OPTIONS=-Xmx4g python profiling/delphi_audit/audit.py --corpus /tmp/delphi-corpus --output /tmp/delphi-ghidra --mode ghidra --ghidra-install /path/to/ghidra_12.2_PUBLIC +JAVA_TOOL_OPTIONS=-Xmx4g python profiling/delphi_audit/audit.py --corpus /tmp/delphi-corpus --output /tmp/delphi-upstream --mode upstream --ghidra-install /path/to/ghidra_12.2_PUBLIC --upstream-script /path/to/DelphiReSym.py +JAVA_TOOL_OPTIONS=-Xmx6g python profiling/delphi_audit/audit.py --corpus /tmp/delphi-corpus --output /tmp/heidi64-ghidra.json --mode ghidra --ghidra-install /path/to/ghidra_12.2_PUBLIC --ghidra-disable-analyzer 'Create Address Tables' --worker /tmp/delphi-corpus/HeidiSQL_12.10_64_Portable.exe +``` + +Use `--include-private` only after placing the supplied executable under its +manifest filename in the local corpus directory. Ghidra and the full slow test +suite can each use several GB; run those sequentially in an 8 GB environment. +Stock 12.2 analysis of HeidiSQL 64-bit exhausted a 6 GB heap in `Create Address +Tables`. Its completed comparison disables only that analyzer; the result records +the setting and is not presented as stock output. The other seven Ghidra snapshots +use stock analysis. Function data is captured before Ghidra saves or closes its +temporary project, which avoids a separate high-memory database commit. + +## Scope and remaining evidence limits + +Sparse real metadata fixtures, a real finally-dispatch excerpt, and generated +malformed/partially stripped variants are permanent tests. They cover both pointer +widths, names/signatures, VMT/RTTI consistency, virtual/fixed method survival after +extended metadata removal, duplicate parameters, entry budgets, code-range handling, +Unicode byte lengths, provider reuse and PDB supplementation. + +This is not an exhaustive compiler/optimization matrix. Release binaries do not +supply independently verified optimization settings; Delphi 2010, every XE release, +and Delphi 13 were not all rebuilt with licensed compilers. Pythia's conservative +interface-table recovery remains 32-bit only and is capped at 4,096 methods. Fully removed RTTI, +packing, imported runtime-package metadata, all EH personalities, class properties, +and complete unit/type hierarchies cannot be reconstructed from these method tables +alone. Damaged optional metadata degrades to known names or unknown types rather +than fabricated signatures. No claim is made that all possible correctness or +performance issues have been eliminated. diff --git a/profiling/delphi_audit/audit.py b/profiling/delphi_audit/audit.py new file mode 100644 index 00000000..7c3e7f27 --- /dev/null +++ b/profiling/delphi_audit/audit.py @@ -0,0 +1,306 @@ +"""Reproduce Delphi metadata, native CFG and independent Ghidra measurements.""" + +import argparse +import cProfile +import hashlib +import io +import json +import os +import statistics +import subprocess +import sys +import time +import tracemalloc +import types +import urllib.request +import zipfile +from pathlib import Path + +ROOT = Path(__file__).resolve().parent + + +def mapped_info(path): + from smda.common.BinaryInfo import BinaryInfo + from smda.utility.PeFileLoader import PeFileLoader + + raw = path.read_bytes() + info = BinaryInfo(PeFileLoader.mapBinary(raw)) + info.raw_data = raw + info.base_addr = PeFileLoader.getBaseAddress(raw) + info.bitness = PeFileLoader.getBitness(raw) + info.architecture = "intel" + info.code_areas = PeFileLoader.getCodeAreas(raw) + return info + + +def provider_result(path, repeat=1): + from smda.common.labelprovider.DelphiReSymProvider import DelphiReSymProvider + from smda.SmdaConfig import SmdaConfig + + provider = DelphiReSymProvider(SmdaConfig()) + info = mapped_info(path) + times = [] + for _ in range(repeat): + started = time.perf_counter() + provider.update(info) + times.append(time.perf_counter() - started) + return { + "seconds": statistics.median(times), + "runs_seconds": times, + "symbols": provider.getFunctionSymbols(), + "usable_labels": sum(bool(provider.getSymbol(addr)) for addr in provider.getFunctionSymbols()), + } + + +def provider_profile(path, provider_type): + from smda.SmdaConfig import SmdaConfig + + info = mapped_info(path) + provider = provider_type(SmdaConfig()) + profiler = cProfile.Profile() + tracemalloc.start() + started = time.perf_counter() + profiler.enable() + provider.update(info) + profiler.disable() + seconds = time.perf_counter() - started + peak_python_bytes = tracemalloc.get_traced_memory()[1] + tracemalloc.stop() + profile = sorted( + ( + { + "function": entry.code.co_name, + "calls": entry.callcount, + "seconds": entry.totaltime, + } + for entry in profiler.getstats() + if isinstance(entry.code, types.CodeType) + and "Delphi" in entry.code.co_filename + ), + key=lambda entry: entry["seconds"], + reverse=True, + )[:40] + return { + "seconds": seconds, + "symbols": len(provider.getFunctionSymbols()), + "usable_labels": sum(bool(provider.getSymbol(addr)) for addr in provider.getFunctionSymbols()), + "peak_python_bytes": peak_python_bytes, + "profile": profile, + } + + +def native_result(path): + from smda.Disassembler import Disassembler + + report = Disassembler().disassembleFile(str(path)) + functions = {} + for function in report.getFunctions(): + blocks = [] + for block in function.getBlocks(): + blocks.append([block.offset, [(i.offset, i.bytes) for i in block.getInstructions()]]) + cfg = json.dumps([blocks, function.blockrefs], sort_keys=True).encode() + functions[function.offset] = { + "name": function.function_name, + "cfg_sha256": hashlib.sha256(cfg).hexdigest(), + "pic_hash": function.pic_hash, + } + return {"status": report.status, "functions": functions} + + +def java_iter(iterator): + while iterator.hasNext(): + yield iterator.next() + + +def ghidra_result(path, install, disabled_analyzers=()): + import pyghidra + + pyghidra.start(install_dir=install) + from tempfile import TemporaryDirectory + + from ghidra.app.plugin.core.analysis import AutoAnalysisManager + from ghidra.app.script import GhidraScriptUtil + from ghidra.framework import Application + from ghidra.program.flatapi import FlatProgramAPI + from ghidra.program.model.block import BasicBlockModel + from ghidra.program.model.listing import Program + from ghidra.util.task import TaskMonitor + from pyghidra.api import transaction + from pyghidra.core import _analyze_program, _setup_project + + with TemporaryDirectory() as project_location: + _project, program = _setup_project(path, project_location=project_location) + GhidraScriptUtil.acquireBundleHostReference() + try: + flat = FlatProgramAPI(program) + if disabled_analyzers: + manager = AutoAnalysisManager.getAnalysisManager(program) + manager.registerOptions() + options = program.getOptions(Program.ANALYSIS_PROPERTIES) + with transaction(program, "Set Ghidra audit analysis options"): + for analyzer in disabled_analyzers: + options.setBoolean(analyzer, False) + manager.initializeOptions() + _analyze_program(flat, program) + model = BasicBlockModel(program) + functions = {} + for function in program.getFunctionManager().getFunctions(True): + blocks = [] + for block in model.getCodeBlocksContaining(function.getBody(), TaskMonitor.DUMMY): + edges = [ + int(edge.getDestinationAddress().getOffset()) + for edge in java_iter(block.getDestinations(TaskMonitor.DUMMY)) + if not edge.getFlowType().isCall() + ] + blocks.append( + [int(block.getFirstStartAddress().getOffset()), int(block.getMaxAddress().getOffset()) + 1, edges] + ) + functions[int(function.getEntryPoint().getOffset())] = { + "name": str(function.getName(True)), + "ranges": [ + [int(r.getMinAddress().getOffset()), int(r.getMaxAddress().getOffset()) + 1] + for r in function.getBody().getAddressRanges() + ], + "blocks": blocks, + } + return { + "functions": functions, + "ghidra_version": str(Application.getApplicationVersion()), + "disabled_analyzers": list(disabled_analyzers), + } + finally: + GhidraScriptUtil.releaseBundleHostReference() + + +def download(entry, corpus): + with urllib.request.urlopen(entry["source_url"], timeout=120) as response: + raw = response.read() + if hashlib.sha256(raw).hexdigest() != entry["source_sha256"]: + raise ValueError(f"Source digest mismatch: {entry['name']}") + if entry.get("archive_member"): + with zipfile.ZipFile(io.BytesIO(raw)) as archive: + raw = archive.read(entry["archive_member"]) + if hashlib.sha256(raw).hexdigest() != entry["sha256"]: + raise ValueError(f"Binary digest mismatch: {entry['name']}") + (corpus / entry["name"]).write_bytes(raw) + + +def upstream_result(path, install, source): + import pyghidra + + pyghidra.start(install_dir=install) + from tempfile import TemporaryDirectory + + from ghidra.util.task import TaskMonitor + + text = source.read_text().rsplit("if pyghidra.started():", 1)[0] + with TemporaryDirectory() as project, pyghidra.open_program(path, project_location=project, analyze=False) as flat: + module = types.ModuleType("delphi_upstream_audit") + sys.modules[module.__name__] = module + module.__dict__.update(currentProgram=flat.getCurrentProgram(), monitor=TaskMonitor.DUMMY, toAddr=flat.toAddr) + exec(compile(text, str(source), "exec"), module.__dict__) + module.__dict__["VERBOSE_INFO"] = False + settings = module.get_architecture_settings(module.get_text_section(flat.getCurrentProgram().getMemory())) + vmts = module.find_vmts(settings) + mdts = module.get_vmt_field_addresses(vmts, settings, settings.mdt_offset) + rttis = module.get_vmt_field_addresses(vmts, settings, settings.rtti_offset) + entries = module.traverse_mdt_top_level(mdts, settings) + entries = module.traverse_method_entries(entries, settings) + entries = module.add_namespace_information(rttis, entries, settings) + symbols = {} + for cls in entries.entries.values(): + for method in cls.method_entries.values(): + if method.function_name: + symbols[int(method.function_entry_point.getOffset())] = { + "name": method.function_name, + "namespace": cls.namespace, + "return_type": method.return_type_str, + "parameters": [(p.parameter_name, p.rtti_namespace) for p in method.parameter_entries.values()], + } + return {"vmts": len(vmts), "symbols": symbols} + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--corpus", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--mode", choices=("provider", "profile", "native", "ghidra", "upstream"), default="provider") + parser.add_argument("--source", type=Path, help="Alternate SMDA src directory, e.g. a baseline checkout") + parser.add_argument("--ghidra-install", type=Path) + parser.add_argument("--ghidra-disable-analyzer", action="append", default=[]) + parser.add_argument("--upstream-script", type=Path) + parser.add_argument("--include-private", action="store_true") + parser.add_argument("--repeat", type=int, default=1) + parser.add_argument("--download", action="store_true") + parser.add_argument("--worker", type=Path, help=argparse.SUPPRESS) + args = parser.parse_args() + if args.repeat < 1: + parser.error("--repeat must be positive") + if args.worker: + started = time.perf_counter() + if args.mode == "provider": + result = provider_result(args.worker, args.repeat) + elif args.mode == "profile": + from smda.common.labelprovider.DelphiPythiaProvider import DelphiPythiaProvider + from smda.common.labelprovider.DelphiReSymProvider import DelphiReSymProvider + + result = { + "providers": { + "resym": provider_profile(args.worker, DelphiReSymProvider), + "pythia": provider_profile(args.worker, DelphiPythiaProvider), + } + } + elif args.mode == "native": + result = native_result(args.worker) + elif args.mode == "ghidra": + result = ghidra_result(args.worker, args.ghidra_install, args.ghidra_disable_analyzer) + else: + try: + result = upstream_result(args.worker, args.ghidra_install, args.upstream_script) + except Exception as exc: + result = {"error": f"{type(exc).__name__}: {exc}"} + result = {**result, "binary": args.worker.name, "total_seconds": time.perf_counter() - started} + args.output.write_text(json.dumps(result, indent=2) + "\n") + return + args.corpus.mkdir(parents=True, exist_ok=True) + args.output.mkdir(parents=True, exist_ok=True) + manifest = json.loads((ROOT / "manifest.json").read_text()) + entries = manifest["binaries"] + (manifest["private_binaries"] if args.include_private else []) + for entry in entries: + path = args.corpus / entry["name"] + if args.download and not path.exists() and "source_url" in entry: + download(entry, args.corpus) + if not path.exists(): + raise FileNotFoundError(path) + if hashlib.sha256(path.read_bytes()).hexdigest() != entry["sha256"]: + raise ValueError(f"Binary digest mismatch: {path}") + output = args.output / f"{path.stem}.{args.mode}.json" + command = [ + sys.executable, + str(Path(__file__).resolve()), + "--corpus", + str(args.corpus), + "--output", + str(output), + "--mode", + args.mode, + "--worker", + str(path), + "--repeat", + str(args.repeat), + ] + if args.ghidra_install: + command.extend(["--ghidra-install", str(args.ghidra_install)]) + for analyzer in args.ghidra_disable_analyzer: + command.extend(["--ghidra-disable-analyzer", analyzer]) + if args.upstream_script: + command.extend(["--upstream-script", str(args.upstream_script)]) + env = os.environ.copy() + if args.source: + env["PYTHONPATH"] = str(args.source.resolve()) + subprocess.run(command, env=env, check=True) + print(output, flush=True) + + +if __name__ == "__main__": + main() diff --git a/profiling/delphi_audit/manifest.json b/profiling/delphi_audit/manifest.json new file mode 100644 index 00000000..462469ed --- /dev/null +++ b/profiling/delphi_audit/manifest.json @@ -0,0 +1,81 @@ +{ + "smda_baseline": "04652e6", + "delphiresym_commit": "22ad5f7cbe2b5f07487df96a9dfec1566e9a96b5", + "ghidra_release": "12.2", + "ghidra_sha256": "2c2cd776db3c1f6a097ba418922378ac3649cadd5e844a6f5f088ae628a7be67", + "binaries": [ + { + "name": "HeidiSQL_12.10_32_Portable.exe", + "sha256": "cb1c66ff68230f08e07bb8018d1c8c91cc8b389d1aee8ed6ba2441caf4d51a56", + "size": 18209768, + "source_url": "https://www.heidisql.com/downloads/releases/HeidiSQL_12.10_32_Portable.zip", + "compiler": "modern Delphi; exact build compiler not independently established", + "archive_member": "heidisql.exe", + "source_sha256": "670558b198f6f89c1fffa31b00c21de34e0c9bd2c0b1b4b9909a708509744ca6" + }, + { + "name": "HeidiSQL_12.10_64_Portable.exe", + "sha256": "c17982e753d24fb9f90212e2d11e01e08cca0641372e146b1e32d087fb9a9659", + "size": 25056744, + "source_url": "https://www.heidisql.com/downloads/releases/HeidiSQL_12.10_64_Portable.zip", + "compiler": "modern Delphi; exact build compiler not independently established", + "archive_member": "heidisql.exe", + "source_sha256": "a7b0e40acc85900ff5ca63769e702acaa06e81ae658b7958e1bc794ae635fd7a" + }, + { + "name": "UPSPowerMTR_V2.03_C.exe", + "sha256": "738c6a3b0b70e7de8494cf28a78254e0dfe4d644901a1c55cfcb8b1868af058e", + "size": 2948608, + "source_url": "https://github.com/user-attachments/files/24620994/UPSPowerMTR_V2.03_C.zip", + "archive_member": "UPSPowerMTR_V2.03_C.exe", + "compiler": "Delphi 7, identified by upstream maintainer in issue #5", + "source_sha256": "a4b4c43a011b94a6f570f5e95f033ee90e45964e53f7999b9edbdf0bff94c92d" + }, + { + "name": "innosetup-5.6.1.exe", + "sha256": "96fd6a5eaab473c61a19affff89618764b940ee3f15837c2944a5595aed5fde6", + "size": 1963136, + "source_url": "https://files.jrsoftware.org/is/5/innosetup-5.6.1.exe", + "compiler": "Delphi 2009 Update 3 (Unicode builds), per tagged upstream README", + "compiler_evidence": "https://github.com/jrsoftware/issrc/blob/is-5_6_1/README.md", + "source_sha256": "96fd6a5eaab473c61a19affff89618764b940ee3f15837c2944a5595aed5fde6" + }, + { + "name": "innosetup-6.2.2.exe", + "sha256": "8117d10d00a2ad33a1390978ea3872861c330e087914410a6377b22c4c5b8563", + "size": 4722512, + "source_url": "https://github.com/jrsoftware/issrc/releases/download/is-6_2_2/innosetup-6.2.2.exe", + "compiler": "Delphi 10.3.3 Rio, per tagged upstream README", + "compiler_evidence": "https://github.com/jrsoftware/issrc/blob/is-6_2_2/README.md", + "source_sha256": "8117d10d00a2ad33a1390978ea3872861c330e087914410a6377b22c4c5b8563" + }, + { + "name": "innosetup-7.1.0-x64.exe", + "sha256": "0362a383ed217d4c4239b5933866dd96d3eb2102737da92f80f6057a4b40df2f", + "size": 14304168, + "source_url": "https://github.com/jrsoftware/issrc/releases/download/is-7_1_0/innosetup-7.1.0-x64.exe", + "compiler": "Delphi 12.3 Athens with May Patch, per tagged upstream README", + "compiler_evidence": "https://github.com/jrsoftware/issrc/blob/is-7_1_0/README.md", + "source_sha256": "0362a383ed217d4c4239b5933866dd96d3eb2102737da92f80f6057a4b40df2f" + }, + { + "name": "innosetup-7.1.0-x86.exe", + "sha256": "f9671174e0d15ba9b4f6b56564c6aed32ea8db9c3cb9bf6f2af0850fe7894f60", + "size": 13119232, + "source_url": "https://github.com/jrsoftware/issrc/releases/download/is-7_1_0/innosetup-7.1.0-x86.exe", + "compiler": "Delphi 12.3 Athens with May Patch, per tagged upstream README", + "compiler_evidence": "https://github.com/jrsoftware/issrc/blob/is-7_1_0/README.md", + "source_sha256": "f9671174e0d15ba9b4f6b56564c6aed32ea8db9c3cb9bf6f2af0850fe7894f60" + } + ], + "private_binaries": [ + { + "name": "1d4e54529feef53850f97f39029a906d53f3d4b2aea8373e27c413324a55681c.exe", + "sha256": "1d4e54529feef53850f97f39029a906d53f3d4b2aea8373e27c413324a55681c", + "source": "user upload, not publicly redistributed", + "archive_sha256": "68882c848f75ac8e81fb0c4d0a7cdcec28f8aad22e37842db8373f987910b2af", + "size": 3447808, + "compiler": "modern 32-bit Delphi; exact compiler version unconfirmed" + } + ] +} diff --git a/src/smda/SmdaConfig.py b/src/smda/SmdaConfig.py index d4425f3b..e02abccb 100644 --- a/src/smda/SmdaConfig.py +++ b/src/smda/SmdaConfig.py @@ -263,6 +263,7 @@ class SmdaConfig: # backstops against memory usage explosion during candidate identification on pathological/junk samples # maximum number of function candidates to track; 0 == unlimited. Normal binaries have orders of magnitude fewer. MAX_FUNCTION_CANDIDATES = 200000 + MAX_DELPHI_METADATA_ENTRIES = 1000000 # maximum number of inbound call references tracked per candidate; 0 == unlimited. Bounds set growth and rescoring. MAX_CALL_REFS_PER_CANDIDATE = 2000 # backstops for batch mode, where peak worker RSS scales with input size. The factor is diff --git a/src/smda/common/FunctionCandidateManager.py b/src/smda/common/FunctionCandidateManager.py index 4f28ad8e..1259d2d5 100644 --- a/src/smda/common/FunctionCandidateManager.py +++ b/src/smda/common/FunctionCandidateManager.py @@ -35,6 +35,7 @@ class FunctionCandidateManager: def __init__(self, config): self.config = config self.lang_analyzer = None + self.delphi_resolvers = None self.disassembly = None self.bitness = None self._code_areas = [] @@ -86,7 +87,7 @@ def init(self, disassembly, cbAnalysisTimeout=None): self._code_areas = disassembly.binary_info.code_areas self.disassembly = disassembly self._cb_analysis_timeout = cbAnalysisTimeout - self.lang_analyzer = LanguageAnalyzer(disassembly) + self.lang_analyzer = LanguageAnalyzer(disassembly, self.delphi_resolvers) self.disassembly.language = self.lang_analyzer.identify() self.disassembly.language_guess = self.lang_analyzer.getGuess() self.bitness = disassembly.binary_info.bitness diff --git a/src/smda/common/LanguageAnalyzer.py b/src/smda/common/LanguageAnalyzer.py index 60c6e115..49304e35 100644 --- a/src/smda/common/LanguageAnalyzer.py +++ b/src/smda/common/LanguageAnalyzer.py @@ -21,13 +21,15 @@ class LanguageAnalyzer: - def __init__(self, disassembly): + def __init__(self, disassembly, delphi_resolvers=None): self.disassembly = disassembly + delphi_resolvers = delphi_resolvers or {} + self._updated_resolvers = set(delphi_resolvers.values()) self.go_resolver = GoSymbolProvider(None) self.rust_resolver = RustSymbolProvider(None) self.delphi_kb_resolver = DelphiKbSymbolProvider(None) - self.delphi_pythia_resolver = DelphiPythiaProvider(None) - self.delphi_resym_resolver = DelphiReSymProvider(None) + self.delphi_pythia_resolver = delphi_resolvers.get(DelphiPythiaProvider) or DelphiPythiaProvider(None) + self.delphi_resym_resolver = delphi_resolvers.get(DelphiReSymProvider) or DelphiReSymProvider(None) self.strings = None self._cpp_symbol_count = 0 self._guess = None @@ -161,6 +163,8 @@ def _computeDelphiScore(self): delphi_score = max(delphi_score, 0.9) elif delphi_objects: delphi_score = max(delphi_score, 0.6) + if delphi_score < 0.6 and self.getDelphiReSymObjects(): + delphi_score = 0.6 return delphi_score def getGoScore(self): @@ -257,8 +261,12 @@ def parseDelphiString(self, buffer): return parsed_string def _safeResolverUpdate(self, resolver): + if resolver in self._updated_resolvers: + return try: resolver.update(self.disassembly.binary_info) + if isinstance(resolver, (DelphiReSymProvider, DelphiPythiaProvider)): + self._updated_resolvers.add(resolver) except Exception as exc: reraise_non_operational_exception(exc) LOGGER.error("Symbol resolver %s failed to update: %r", resolver.__class__.__name__, exc) diff --git a/src/smda/common/RecursiveDisassembler.py b/src/smda/common/RecursiveDisassembler.py index 7d9a6fe8..055d1368 100644 --- a/src/smda/common/RecursiveDisassembler.py +++ b/src/smda/common/RecursiveDisassembler.py @@ -76,11 +76,11 @@ def _addLabelProviders(self): self._registerLabelProvider(DelphiKbSymbolProvider(self.config)) self._registerLabelProvider(DelphiReSymProvider(self.config)) self._registerLabelProvider(DelphiPythiaProvider(self.config)) + self._registerLabelProvider(PdbSymbolProvider(self.config)) # Generic binary format providers (fallback) self._registerLabelProvider(ElfSymbolProvider(self.config)) self._registerLabelProvider(PeSymbolProvider(self.config)) self._registerLabelProvider(MachoSymbolProvider(self.config)) - self._registerLabelProvider(PdbSymbolProvider(self.config)) def _registerLabelProvider(self, provider): self.label_providers.append(provider) @@ -89,8 +89,10 @@ def _registerLabelProvider(self, provider): if provider.isSymbolProvider(): self.symbol_providers.append(provider) - def _runLabelProviderUpdate(self, binary_info): + def _runLabelProviderUpdate(self, binary_info, pdb_only=False): for provider in self.label_providers: + if pdb_only and not isinstance(provider, PdbSymbolProvider): + continue try: provider.update(binary_info) except Exception as exc: @@ -103,7 +105,7 @@ def _updateLabelProviders(self, binary_info): # analysis would be wiped by this pass; re-apply it afterwards instead pdb_info = getattr(self, "_pdb_info", None) if pdb_info is not None: - self._runLabelProviderUpdate(pdb_info) + self._runLabelProviderUpdate(pdb_info, pdb_only=True) self.active_api_providers = [p for p in self.api_providers if p.is_active()] self.active_symbol_providers = [p for p in self.symbol_providers if p.is_active()] @@ -114,7 +116,7 @@ def addPdbFile(self, binary_info, pdb_path): pdb_info.file_path = pdb_path pdb_info.base_addr = binary_info.base_addr self._pdb_info = pdb_info - self._runLabelProviderUpdate(pdb_info) + self._runLabelProviderUpdate(pdb_info, pdb_only=True) self.active_api_providers = [p for p in self.api_providers if p.is_active()] self.active_symbol_providers = [p for p in self.symbol_providers if p.is_active()] @@ -566,6 +568,11 @@ def analyzeBuffer(self, binary_info, cbAnalysisTimeout=None): self.jumptable_analyzer = self.backend.createJumpTableAnalyzer(self) self.fc_manager = self.backend.createCandidateManager(self.config) + self.fc_manager.delphi_resolvers = { + type(provider): provider + for provider in self.symbol_providers + if isinstance(provider, (DelphiReSymProvider, DelphiPythiaProvider)) + } if self.config.USE_SYMBOLS_AS_CANDIDATES: self.fc_manager.symbol_addresses = self.getSymbolCandidates() # once we are initialized, add OEP diff --git a/src/smda/common/labelprovider/DelphiPythiaProvider.py b/src/smda/common/labelprovider/DelphiPythiaProvider.py index 1c6b53b6..2d7bc22d 100644 --- a/src/smda/common/labelprovider/DelphiPythiaProvider.py +++ b/src/smda/common/labelprovider/DelphiPythiaProvider.py @@ -18,12 +18,12 @@ * the instance size must be reasonable, * the class name pointer must decode to a plausible Pascal string, * the fixed VMT method slots must point into executable code areas, - * metadata pointers must stay inside the mapped image. -4. Extract the class name, parent VMT pointer, method table, dynamic table, - and virtual method slots directly from the mapped image. -5. Compare child and parent VMT slots to identify inherited vs. overridden - methods. SMDA's public API still only returns ``{address: function_name}``, - but the provider keeps this relationship internally for future use. + * mapped metadata is read only within image bounds; external optional tables + do not invalidate otherwise usable VMT methods. +4. Extract method table, dynamic table, interface methods and virtual method + slots directly from the mapped image. +5. Merge recovered addresses and names into SMDA's public + ``{address: function_name}`` contract. Compared to the historic implementation, this is significantly faster on large inputs because it scans only code sections, advances in pointer-sized steps, and @@ -36,13 +36,12 @@ import logging import struct from dataclasses import dataclass, field -from itertools import compress -from operator import eq, itemgetter from typing import Dict, Iterable, List, Optional, Sequence, Tuple import lief from .AbstractLabelProvider import AbstractLabelProvider +from .DelphiVmtScanner import iter_vmt_self_references LOGGER = logging.getLogger(__name__) @@ -50,9 +49,10 @@ VMT_HEADER_FIELD_COUNT = 11 MAX_INSTANCE_SIZE = 1024 * 500 MAX_VIRTUAL_SLOT_SCAN = 512 +MAX_INTERFACE_METHOD_SCAN = 4096 -@dataclass(frozen=True) +@dataclass(frozen=True, slots=True) class DelphiVmtProfile: """Static layout information for a Delphi VMT profile.""" @@ -81,10 +81,6 @@ def class_name_field_offset(self) -> int: def instance_size_field_offset(self) -> int: return self.ptr_size * 9 - @property - def parent_field_offset(self) -> int: - return self.ptr_size * 10 - @property def interface_table_field_offset(self) -> int: return self.ptr_size @@ -98,21 +94,14 @@ def dynamic_table_field_offset(self) -> int: return self.ptr_size * 7 -@dataclass +@dataclass(slots=True) class DelphiClassInfo: - """Internal representation of a Delphi class recovered from a VMT.""" + """Metadata tables and code slots recovered from a Delphi VMT.""" - profile_name: str - candidate_offset: int - vmt_offset: int - class_name: str - parent_vmt_addr: int = 0 method_table_addr: int = 0 dynamic_table_addr: int = 0 interface_table_addr: int = 0 method_slots: List[int] = field(default_factory=list) - inherited_slots: List[int] = field(default_factory=list) - overridden_slots: List[int] = field(default_factory=list) class DelphiPythiaProvider(AbstractLabelProvider): @@ -143,7 +132,6 @@ def __init__(self, config): self._bitness = 0 self._scan_ranges: List[Tuple[int, int]] = [] self._func_symbols: Dict[int, str] = {} - self._classes: Dict[int, DelphiClassInfo] = {} def update(self, binary_info): """Parse Delphi metadata from the given binary info.""" @@ -152,9 +140,12 @@ def update(self, binary_info): self._base_addr = binary_info.base_addr self._bitness = binary_info.bitness self._func_symbols = {} - self._classes = {} - if not self._binary or self._bitness not in self._profiles: + if ( + not self._binary + or self._bitness not in self._profiles + or getattr(binary_info, "architecture", "") not in ("", "intel") + ): return # Cheap Delphi signature check @@ -170,19 +161,17 @@ def update(self, binary_info): def _parse_delphi_objects(self) -> None: function_offsets = set() name_mapping: Dict[int, str] = {} - classes_by_vmt_addr: Dict[int, DelphiClassInfo] = {} + seen_dynamic_tables = set() + seen_method_tables = set() + seen_interface_tables = set() for profile in self._profiles.get(self._bitness, ()): for start_offset, end_offset in self._scan_ranges: for class_info in self._scan_code_range(profile, start_offset, end_offset): - classes_by_vmt_addr[self._offset_to_addr(class_info.vmt_offset)] = class_info function_offsets.update(class_info.method_slots) - self._extract_dynamic_methods(class_info, function_offsets) - self._extract_interface_methods(class_info, function_offsets) - self._extract_method_table_symbols(class_info, function_offsets, name_mapping) - - self._classes = classes_by_vmt_addr - self._resolve_inheritance() + self._extract_dynamic_methods(class_info, function_offsets, seen_dynamic_tables) + self._extract_interface_methods(class_info, function_offsets, seen_interface_tables) + self._extract_method_table_symbols(class_info, function_offsets, name_mapping, seen_method_tables) for function_addr in sorted(function_offsets): if self._binary_info.isInCodeAreas(function_addr): @@ -235,39 +224,19 @@ def _scan_code_range( ) -> Iterable[DelphiClassInfo]: """Yield validated VMTs from a single code range.""" ptr_size = profile.ptr_size - first_offset = self._align_offset(start_offset, ptr_size) - scan_end = first_offset + max(0, min(end_offset, len(self._binary)) - first_offset) // ptr_size * ptr_size - # a slot is a candidate when it holds its own address plus the self-pointer distance. - # The slots are compared in C: one Python-level read per slot made this scan dominate - # the analysis of large images that are not Delphi but contain the string "TObject". - slot_values = map( - itemgetter(0), - struct.iter_unpack(" int: - """Align the given offset upwards.""" - remainder = offset % alignment - return offset if remainder == 0 else offset + (alignment - remainder) - def _build_class_info( self, profile: DelphiVmtProfile, candidate_offset: int, range_end_offset: int ) -> Optional[DelphiClassInfo]: """Validate a VMT candidate and extract its metadata.""" class_name_addr = self._read_ptr(candidate_offset + profile.class_name_field_offset, profile.ptr_size) or 0 instance_size = self._read_ptr(candidate_offset + profile.instance_size_field_offset, profile.ptr_size) - parent_vmt_addr = self._read_ptr(candidate_offset + profile.parent_field_offset, profile.ptr_size) or 0 method_table_addr = self._read_ptr(candidate_offset + profile.method_table_field_offset, profile.ptr_size) or 0 dynamic_table_addr = ( self._read_ptr(candidate_offset + profile.dynamic_table_field_offset, profile.ptr_size) or 0 @@ -286,16 +255,6 @@ def _build_class_info( if not self._is_probable_class_name(class_name): return None - metadata_addresses = ( - class_name_addr, - parent_vmt_addr, - method_table_addr, - dynamic_table_addr, - interface_table_addr, - ) - if not self._validate_metadata_addresses(metadata_addresses): - return None - fixed_methods = self._read_fixed_methods(profile, candidate_offset) if not self._validate_fixed_methods(profile, fixed_methods): return None @@ -305,11 +264,6 @@ def _build_class_info( return None class_info = DelphiClassInfo( - profile_name=profile.name, - candidate_offset=candidate_offset, - vmt_offset=vmt_offset, - class_name=class_name, - parent_vmt_addr=parent_vmt_addr, method_table_addr=method_table_addr, dynamic_table_addr=dynamic_table_addr, interface_table_addr=interface_table_addr, @@ -317,10 +271,6 @@ def _build_class_info( ) return class_info - def _validate_metadata_addresses(self, addresses: Sequence[int]) -> bool: - """Ensure optional metadata pointers remain inside the mapped image.""" - return all(not (address and not self._is_mapped_address(address)) for address in addresses) - def _read_fixed_methods(self, profile: DelphiVmtProfile, candidate_offset: int) -> List[int]: """Read the fixed compiler-emitted method slots preceding vmtSelfPtr.""" methods = [] @@ -368,10 +318,16 @@ def _read_virtual_methods( slots_read += 1 return methods - def _extract_dynamic_methods(self, class_info: DelphiClassInfo, function_offsets: set) -> None: + def _extract_dynamic_methods( + self, class_info: DelphiClassInfo, function_offsets: set, seen_tables: Optional[set] = None + ) -> None: """Recover method pointers from a dynamic table when present.""" if not class_info.dynamic_table_addr: return + if seen_tables is not None: + if class_info.dynamic_table_addr in seen_tables: + return + seen_tables.add(class_info.dynamic_table_addr) dynamic_offset = self._addr_to_offset(class_info.dynamic_table_addr) table_length = self._read_word(dynamic_offset) if table_length is None or table_length == 0: @@ -386,11 +342,15 @@ def _extract_dynamic_methods(self, class_info: DelphiClassInfo, function_offsets function_offset += self._ptr_size def _extract_method_table_symbols( - self, class_info: DelphiClassInfo, function_offsets: set, name_mapping: Dict[int, str] + self, class_info: DelphiClassInfo, function_offsets: set, name_mapping: Dict[int, str], seen_tables: Optional[set] = None ) -> None: """Parse the classic Delphi method table and recover symbol names.""" if not class_info.method_table_addr: return + if seen_tables is not None: + if class_info.method_table_addr in seen_tables: + return + seen_tables.add(class_info.method_table_addr) entry_offset = self._addr_to_offset(class_info.method_table_addr) entry_count = self._read_word(entry_offset) if entry_count is None or entry_count == 0: @@ -398,46 +358,47 @@ def _extract_method_table_symbols( entry_offset += 2 for _ in range(entry_count): entry_size = self._read_word(entry_offset) + minimum_entry_size = 2 + self._ptr_size + 1 + if ( + entry_size is None + or entry_size < minimum_entry_size + or entry_offset + entry_size > len(self._binary) + ): + break method_addr = self._read_ptr(entry_offset + 2, self._ptr_size) - method_name = self._read_pascal_string(entry_offset + 2 + self._ptr_size) + method_name = self._read_pascal_string( + entry_offset + 2 + self._ptr_size, entry_offset + entry_size + ) if method_addr and self._binary_info.isInCodeAreas(method_addr): function_offsets.add(method_addr) if method_name: name_mapping[method_addr] = method_name - if not entry_size or entry_size < 2 + self._ptr_size: - break entry_offset += entry_size - def _extract_interface_methods(self, class_info: DelphiClassInfo, function_offsets: set) -> None: + def _extract_interface_methods( + self, class_info: DelphiClassInfo, function_offsets: set, seen_tables: Optional[set] = None + ) -> None: """Parse interface tables conservatively for classic 32-bit Delphi layouts.""" if self._bitness != 32 or not class_info.interface_table_addr: return + if seen_tables is not None: + if class_info.interface_table_addr in seen_tables: + return + seen_tables.add(class_info.interface_table_addr) interface_offset = self._addr_to_offset(class_info.interface_table_addr) start_interface_addr = self._read_ptr(interface_offset + 20, self._ptr_size) if not start_interface_addr: return current_offset = self._addr_to_offset(start_interface_addr) - while 0 <= current_offset + self._ptr_size <= len(self._binary): + for _ in range(MAX_INTERFACE_METHOD_SCAN): + if not 0 <= current_offset + self._ptr_size <= len(self._binary): + break method_addr = self._read_ptr(current_offset, self._ptr_size) if not method_addr or not self._binary_info.isInCodeAreas(method_addr): break function_offsets.add(method_addr) current_offset += self._ptr_size - def _resolve_inheritance(self) -> None: - """Mark inherited versus overridden VMT slots using Delphi's single inheritance.""" - for class_info in self._classes.values(): - parent = self._classes.get(class_info.parent_vmt_addr) - if not parent: - continue - for index, method_addr in enumerate(class_info.method_slots): - if index >= len(parent.method_slots): - break - if method_addr == parent.method_slots[index]: - class_info.inherited_slots.append(index) - else: - class_info.overridden_slots.append(index) - @property def _ptr_size(self) -> int: return 4 if self._bitness == 32 else 8 @@ -447,38 +408,43 @@ def _read_ptr(self, offset: int, ptr_size: int) -> Optional[int]: if offset < 0 or offset + ptr_size > len(self._binary): return None if ptr_size == 4: - return struct.unpack(" Optional[int]: """Read a little-endian 16-bit value from the mapped image.""" if offset < 0 or offset + 2 > len(self._binary): return None - return struct.unpack(" str: + def _read_pascal_string(self, offset: int, end: Optional[int] = None) -> str: """Read a Pascal-style string from the mapped image.""" if offset < 0 or offset >= len(self._binary): return "" length = self._binary[offset] end_offset = offset + 1 + length - if length == 0 or end_offset > len(self._binary): + limit = len(self._binary) if end is None else min(end, len(self._binary)) + if length == 0 or end_offset > limit: return "" - return self._binary[offset + 1 : end_offset].decode("latin-1", errors="ignore") + data = self._binary[offset + 1 : end_offset] + try: + value = data.decode("utf-8") + except UnicodeDecodeError: + try: + value = data.decode("cp1252") + except UnicodeDecodeError: + value = data.decode("latin-1") + return value if value.isprintable() else "" @staticmethod def _is_probable_class_name(class_name: str) -> bool: """Heuristic rejection for obviously invalid class names.""" if not class_name or len(class_name) > 255: return False - if not all(character.isprintable() for character in class_name): + if not class_name.isprintable(): return False return any(character.isalpha() for character in class_name) - def _is_mapped_address(self, address: int) -> bool: - """Check whether an address can be resolved into the mapped image.""" - return self._base_addr <= address < self._base_addr + len(self._binary) - def _offset_to_addr(self, offset: int) -> int: """Convert mapped-image offset to absolute address.""" return self._base_addr + offset diff --git a/src/smda/common/labelprovider/DelphiReSymProvider.py b/src/smda/common/labelprovider/DelphiReSymProvider.py index 9b374b9e..e31301f1 100644 --- a/src/smda/common/labelprovider/DelphiReSymProvider.py +++ b/src/smda/common/labelprovider/DelphiReSymProvider.py @@ -3,22 +3,46 @@ DelphiReSymProvider - Delphi Symbol Recovery for SMDA Adapted from DelphiReSym by Lukas Wenz (https://github.com/WenzWenzWenz/DelphiReSym) Parses Delphi metadata structures (VMT, MDT, RTTI) to extract function symbols from Delphi binaries. -Supports Delphi versions 2010 through 13 Florence. +Uses the modern Windows Delphi VMT layout documented for Delphi 2010 through 13. -Changes integrated from upstream (January 2026): -- Added RecursiveDescentParser for template-aware namespace parsing -- Fixed false positive VMT detection during RTTI object traversal -- Enhanced robustness with improved exception handling +BSD 3-Clause License + +Copyright (c) 2025, Lukas Wenz + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. """ import logging import struct +from bisect import bisect_right from dataclasses import dataclass, field -from typing import List, Optional, Tuple - -from smda.common.ExceptionHandling import reraise_non_operational_exception +from typing import Iterator, List, Optional, Tuple from .AbstractLabelProvider import AbstractLabelProvider +from .DelphiVmtScanner import iter_vmt_self_references LOGGER = logging.getLogger(__name__) @@ -42,147 +66,7 @@ PARAM_ENTRY_PADDING = 3 -class RecursiveDescentParser: - """ - Parser for Delphi fully-qualified namespace strings with template support. - - Handles complex generic type names like: - - Simple: "System.Classes.TList" - - Templates: "System.Generics.Collections.TList" - - Nested: "TDictionary>" - - Grammar: - FQN := Namespace ('.' Namespace)* ('<' TemplateArgs '>')? - TemplateArgs := FQN (',' FQN)* - Namespace := [a-zA-Z0-9_()]+ - - Ported from upstream DelphiReSym (Oct 2025). - """ - - def __init__(self, input_string: str): - self.string = input_string - self.pos = 0 - - def _peek(self) -> Optional[str]: - """Returns current character without consuming it.""" - if self.pos < len(self.string): - return self.string[self.pos] - return None - - def _consume(self, expected_char: Optional[str] = None) -> str: - """ - Advances position and returns the consumed character. - If expected_char is provided, validates the character matches. - """ - if self.pos >= len(self.string): - raise ValueError("Unexpected end of input") - char = self.string[self.pos] - if expected_char is not None and char != expected_char: - raise ValueError(f"Expected '{expected_char}' but got '{char}' at position {self.pos}") - self.pos += 1 - return char - - def _parse_namespace_component(self) -> str: - """ - Parses a single namespace component (identifier). - Valid characters: letters, digits, underscores, parentheses. - """ - result = [] - while True: - char = self._peek() - if char is None: - break - # Valid namespace characters: alphanumeric, underscore, parentheses - if char.isalnum() or char in "_()": - result.append(self._consume()) - else: - break - return "".join(result) - - def _parse_template_args(self) -> str: - """ - Parses template arguments between angle brackets. - Handles nested templates recursively. - Returns the complete template string including brackets. - """ - self._consume("<") - result = ["<"] - depth = 1 - - while depth > 0 and self._peek() is not None: - char = self._consume() - result.append(char) - if char == "<": - depth += 1 - elif char == ">": - depth -= 1 - if depth > 0: - raise ValueError("Unterminated template arguments") - - return "".join(result) - - def parse_fqn(self, trim_mode: bool = False) -> Tuple[List[str], str]: - """ - Parses a fully-qualified name (FQN) with optional template arguments. - - Args: - trim_mode: If True, stops parsing at template start and returns - remaining string. Used for extracting class name from FQN. - - Returns: - Tuple of (namespace_components, remaining_string) - - namespace_components: List of namespace parts (e.g., ["System", "Classes", "TList"]) - - remaining_string: Any unparsed portion (templates if trim_mode=True) - """ - components = [] - - while True: - component = self._parse_namespace_component() - if not component: - break - components.append(component) - - # Check for dot (namespace separator) or template start - if self._peek() == ".": - self._consume(".") - elif self._peek() == "<": - if trim_mode: - # In trim mode, stop at template and return remainder - return components, self.string[self.pos :] - # Parse and append template arguments - template = self._parse_template_args() - if components: - components[-1] += template - break - else: - break - - remaining = self.string[self.pos :] - return components, remaining - - def get_class_name(self) -> str: - """ - Extracts the class name (last component) from an FQN. - For "System.Classes.TList", returns "TList". - """ - components, remainder = self.parse_fqn() - if components: - return components[-1] + remainder - return remainder - - def get_namespace(self) -> str: - """ - Extracts the namespace (all but last component) from an FQN. - For "System.Classes.TList", returns "System.Classes". - """ - # Use trim_mode to get components without template - components, _ = self.parse_fqn(trim_mode=True) - if len(components) > 1: - return ".".join(components[:-1]) - return "" - - -@dataclass +@dataclass(slots=True) class ArchitectureSettings: """Architecture-specific settings for VMT/MDT parsing.""" @@ -200,7 +84,7 @@ def rtti_offset(self) -> int: return self.ptr_size * VMT_RTTI_FIELD_INDEX -@dataclass +@dataclass(slots=True) class ParameterInfo: """Information about a function parameter.""" @@ -208,7 +92,7 @@ class ParameterInfo: parameter_name: str = "" -@dataclass +@dataclass(slots=True) class MethodInfo: """Information about a method extracted from MDT.""" @@ -219,479 +103,281 @@ class MethodInfo: class DelphiReSymProvider(AbstractLabelProvider): - """ - Label provider for extracting Delphi symbols from modern Delphi executables. - Parses Virtual Method Tables (VMT), Method Definition Tables (MDT), and RTTI metadata. + """Recover modern Delphi method names from a mapped PE image. + + Offsets in this provider are RVAs into BinaryInfo.binary, never raw-file offsets. + Code ranges constrain function targets; metadata may reside in any mapped region. """ def __init__(self, config): self._config = config - self._func_symbols = {} + self._func_symbols: dict[int, str] = {} self._binary: bytes = b"" self._base_addr: int = 0 - self._bitness = 32 - self._code_start: int = 0 - self._code_end: int = 0 - self._settings = None + self._settings = ArchitectureSettings(4, VMT_JUMP_DIST_32BIT) + self._code_ranges: List[Tuple[int, int]] = [] + self._code_starts: List[int] = [] + self._type_cache: dict[int, Optional[str]] = {} def update(self, binary_info): - """Parse Delphi metadata from the given binary.""" self._func_symbols = {} self._binary = binary_info.binary self._base_addr = binary_info.base_addr - self._bitness = binary_info.bitness - - # Only process PE files with .text sections - if not self._is_compatible(): - return - - # Cheap Delphi signature check - if b"TObject" not in self._binary: + self._settings = ArchitectureSettings(4, VMT_JUMP_DIST_32BIT) + self._code_ranges = [] + self._code_starts = [] + self._type_cache = {} + self._remaining_entries = getattr(self._config, "MAX_DELPHI_METADATA_ENTRIES", 1000000) + self._budget_logged = False + if ( + getattr(binary_info, "architecture", "") not in ("", "intel") + or not self._is_compatible() + or b"TObject" not in self._binary + ): return - - # Determine code areas - code_areas = binary_info.code_areas - if not code_areas: + if not binary_info.code_areas: LOGGER.warning("No code areas found, skipping DelphiReSym parsing") return - - # Code areas are virtual addresses - convert to file offsets for scanning - # Use the first code area (typically .text section) - code_area_start_va = code_areas[0][0] - code_area_end_va = code_areas[0][1] - - # Convert virtual addresses to file offsets - self._code_start = code_area_start_va - self._base_addr - self._code_end = code_area_end_va - self._base_addr - - # Validate the offsets are within binary bounds - if self._code_start < 0 or self._code_end > len(self._binary): - LOGGER.warning( - f"Code area offsets out of bounds: {self._code_start}-{self._code_end}, binary size: {len(self._binary)}" - ) + if self._remaining_entries <= 0: return - - # Set up architecture-specific settings - if self._bitness == 32: - self._settings = ArchitectureSettings(ptr_size=4, jump_dist=VMT_JUMP_DIST_32BIT) - elif self._bitness == 64: - self._settings = ArchitectureSettings(ptr_size=8, jump_dist=VMT_JUMP_DIST_64BIT) + ranges = [] + for start, end in sorted(binary_info.code_areas): + start, end = start - self._base_addr, end - self._base_addr + if not 0 <= start < end <= len(self._binary): + LOGGER.warning("Code area offsets out of bounds: %s-%s, binary size: %s", start, end, len(self._binary)) + continue + if ranges and start <= ranges[-1][1]: + ranges[-1] = (ranges[-1][0], max(end, ranges[-1][1])) + else: + ranges.append((start, end)) + self._code_ranges = ranges + self._code_starts = [start for start, _ in ranges] + if not ranges: + return + if binary_info.bitness == 32: + self._settings = ArchitectureSettings(4, VMT_JUMP_DIST_32BIT) + elif binary_info.bitness == 64: + self._settings = ArchitectureSettings(8, VMT_JUMP_DIST_64BIT) else: - LOGGER.warning(f"Unsupported bitness: {self._bitness}") return - try: - LOGGER.debug("Starting DelphiReSym symbol extraction...") self._parse_delphi_symbols() - if self._func_symbols: - LOGGER.info(f"Extracted {len(self._func_symbols)} Delphi symbols") - else: - LOGGER.debug("No Delphi symbols extracted") - except Exception as e: - reraise_non_operational_exception(e) - LOGGER.warning(f"Error during DelphiReSym parsing: {e}") + finally: + self._type_cache.clear() def _is_compatible(self): - """Check if binary is a PE file.""" - if len(self._binary) < 0x40: - return False - return self._binary[:2] == b"MZ" + return len(self._binary) >= 0x40 and self._binary[:2] == b"MZ" def _read_ptr(self, offset: int) -> Optional[int]: - """Read a pointer at the given offset.""" - if offset < 0: - return None - try: - if self._settings.ptr_size == 4: - return struct.unpack(" Optional[int]: - """Read a byte at the given offset.""" - if 0 <= offset < len(self._binary): - return self._binary[offset] - return None + return self._binary[offset] if 0 <= offset < len(self._binary) else None def _read_short(self, offset: int) -> Optional[int]: - """Read a 16-bit value at the given offset.""" - if offset < 0: - return None - try: - return struct.unpack(" str: - """ - Read a Pascal-style string (length byte followed by characters). - Returns empty string on error. - """ + def _read_pascal_string(self, offset: int, end: Optional[int] = None) -> str: + length = self._read_byte(offset) + limit = len(self._binary) if end is None else min(end, len(self._binary)) + if not length or offset + 1 + length > limit: + return "" + data = self._binary[offset + 1 : offset + 1 + length] try: - length = self._read_byte(offset) - if length is None or length == 0: + value = data.decode("utf-8") + except UnicodeDecodeError: + try: + value = data.decode("cp1252") + except UnicodeDecodeError: return "" - - string_data = self._binary[offset + 1 : offset + 1 + length] - if len(string_data) < length: - return "" - - return string_data.decode("latin-1", errors="ignore") - except (IndexError, UnicodeDecodeError): - return "" + return value if value.isprintable() else "" def _offset_in_code_area(self, offset: int) -> bool: - """Check if offset is within code area.""" - return self._code_start <= offset < self._code_end + index = bisect_right(self._code_starts, offset) - 1 + return index >= 0 and offset < self._code_ranges[index][1] def _addr_to_offset(self, addr: int) -> int: - """Convert virtual address to file offset.""" return addr - self._base_addr def _offset_to_addr(self, offset: int) -> int: - """Convert file offset to virtual address.""" return offset + self._base_addr def _check_vmt_candidate(self, vmt_offset: int, next_struct_offset: int) -> bool: - """ - Perform sanity checks on a VMT candidate. - Returns True if the candidate passes all checks. - - Bug fix (Jan 2026): Added RTTI validation to prevent false positives - during VMT scanning when RTTI objects have invalid pointers. - """ - addresses_to_check = [next_struct_offset] - - # Check MDT pointer - mdt_offset_field = vmt_offset + self._settings.mdt_offset - mdt_addr = self._read_ptr(mdt_offset_field) - if mdt_addr is not None: - mdt_offset = self._addr_to_offset(mdt_addr) - addresses_to_check.append(mdt_offset) - # MDTs should be at higher addresses than VMTs - if mdt_offset <= vmt_offset: + size = self._settings.ptr_size + if not 0 <= vmt_offset < next_struct_offset < len(self._binary): + return False + if vmt_offset + self._settings.jump_dist > len(self._binary): + return False + fields = struct.unpack_from("<22I" if size == 4 else "<22Q", self._binary, vmt_offset) + class_addr = fields[8] + if not class_addr or not self._read_pascal_string(self._addr_to_offset(class_addr)): + return False + for field_num in (VMT_MDT_FIELD_INDEX, VMT_RTTI_FIELD_INDEX): + addr = fields[field_num] + if addr and not 0 <= self._addr_to_offset(addr) < len(self._binary): return False - - # Bug fix: Validate RTTI pointer to prevent false positives - # The RTTI field should point to a valid RTTI object - rtti_offset_field = vmt_offset + self._settings.rtti_offset - rtti_addr = self._read_ptr(rtti_offset_field) - if rtti_addr is not None and rtti_addr != 0: - rtti_offset = self._addr_to_offset(rtti_addr) - # RTTI should be within binary bounds - if rtti_offset < 0 or rtti_offset >= len(self._binary): + rtti_addr = fields[VMT_RTTI_FIELD_INDEX] + if rtti_addr: + rtti = self._addr_to_offset(rtti_addr) + name = self._read_pascal_string(rtti + 1) + if self._read_byte(rtti) != RTTI_CLASS_MAGIC_BYTE or not name: return False - # Validate RTTI object structure (magic byte check) - magic_byte = self._read_byte(rtti_offset) - if magic_byte is None or magic_byte > RTTI_MAX_MAGIC_BYTE: + if self._read_ptr(rtti + (self._read_byte(rtti + 1) or 0) + 2) != self._offset_to_addr(next_struct_offset): return False - - # Check virtual method pointers (fields 11-22, excluding SafeCallException which is optional) for field_num in range(VMT_VMETHOD_START_INDEX, VMT_VMETHOD_END_INDEX): - if field_num != VMT_SAFECALL_EXCEPTION_INDEX: - field_offset = vmt_offset + (self._settings.ptr_size * field_num) - method_addr = self._read_ptr(field_offset) - if method_addr is not None: - method_offset = self._addr_to_offset(method_addr) - addresses_to_check.append(method_offset) - - # All addresses should be within code area - return all(self._offset_in_code_area(off) for off in addresses_to_check) - - def _find_vmts(self) -> list: - """ - Scan for Virtual Method Tables in the code section. - Uses a sliding window approach to find VMT structures. - """ - vmt_offsets = [] - current_offset = self._code_start - - LOGGER.debug(f"Scanning for VMTs from 0x{current_offset:x} to 0x{self._code_end:x}") - - while current_offset < self._code_end - self._settings.ptr_size: - ptr_value = self._read_ptr(current_offset) - if ptr_value is None: - current_offset += 1 + addr = fields[field_num] + if field_num == VMT_SAFECALL_EXCEPTION_INDEX and addr == 0: continue + if not addr or not self._offset_in_code_area(self._addr_to_offset(addr)): + return False + return True - target_offset = self._addr_to_offset(ptr_value) - distance = target_offset - current_offset - - # Check if this matches the expected VMT jump distance - if distance == self._settings.jump_dist and self._check_vmt_candidate(current_offset, target_offset): - vmt_offsets.append(current_offset) - LOGGER.debug(f"Found VMT at offset 0x{current_offset:x}") - - current_offset += 1 - - return vmt_offsets + def _find_vmts(self) -> Iterator[int]: + jump = self._settings.jump_dist + for offset in iter_vmt_self_references( + self._binary, self._base_addr, self._settings.ptr_size, jump, 0, len(self._binary) + ): + if self._check_vmt_candidate(offset, offset + jump): + yield offset def _traverse_rtti_object(self, rtti_offset: int, validate_pointers: bool = True) -> Optional[str]: - """ - Traverse an RTTI object to extract type information. - Returns the type name (with namespace for RTTI_Class types). - - Bug fix (Jan 2026): Added validation for pointers within RTTI_Class - to prevent false positives during VMT scanning. - - Args: - rtti_offset: File offset of the RTTI object - validate_pointers: If True, validate pointer fields in RTTI_Class - to prevent false positives - """ - try: - magic_byte = self._read_byte(rtti_offset) - if magic_byte is None or magic_byte > RTTI_MAX_MAGIC_BYTE: - return None - - # Read object name (Pascal string at offset +1) - object_name = self._read_pascal_string(rtti_offset + 1) - if not object_name: - return None - - # RTTI_Class contains namespace information - # RTTI_Class structure layout: - # MagicByte(1) + ObjectName(pascal) + Unknown(1) + Pointers(2*ptr_size) + Unknown(2) + Namespace(pascal) - if magic_byte == RTTI_CLASS_MAGIC_BYTE: - # Calculate pointer field offsets for validation - ptr1_offset = rtti_offset + 1 + (1 + len(object_name)) + 1 - ptr2_offset = ptr1_offset + self._settings.ptr_size - - # Bug fix: Validate that pointer fields point to reasonable addresses - # This prevents false positive VMT detection when scanning - if validate_pointers: - ptr1 = self._read_ptr(ptr1_offset) - ptr2 = self._read_ptr(ptr2_offset) - # Both pointers should either be null or point within binary - for ptr in (ptr1, ptr2): - if ptr is not None and ptr != 0: - ptr_file_offset = self._addr_to_offset(ptr) - if ptr_file_offset < 0 or ptr_file_offset >= len(self._binary): - return None - - namespace_offset = ptr2_offset + self._settings.ptr_size + 2 - namespace = self._read_pascal_string(namespace_offset) - if namespace: - full_fqn = f"{namespace}.{object_name}" - # Use RecursiveDescentParser for template-aware FQN handling - try: - # Use RecursiveDescentParser for template-aware FQN handling. - # This is mainly for validation; we return the original string regardless. - parser = RecursiveDescentParser(full_fqn) - parser.parse_fqn() - except (ValueError, IndexError) as e: - # Fallback to simple concatenation if parsing fails - LOGGER.debug(f"Failed to parse FQN '{full_fqn}': {e}. Falling back to simple concatenation.") - return full_fqn - - return object_name - - except Exception as e: - reraise_non_operational_exception(e) - LOGGER.debug(f"Error traversing RTTI object at 0x{rtti_offset:x}: {e}") + kind = self._read_byte(rtti_offset) + if kind is None or kind > RTTI_MAX_MAGIC_BYTE: return None - - def _resolve_type_from_double_ptr(self, ptr_field_offset: int) -> Optional[str]: - """ - Resolve a type name from a double-dereferenced pointer to an RTTI object. - Used for both return types and parameter types. - - Enhanced with bounds checking for improved robustness. - """ - try: - ptr_addr = self._read_ptr(ptr_field_offset) - if ptr_addr is None or ptr_addr == 0: - return None - - ptr_offset = self._addr_to_offset(ptr_addr) - # Bounds check for first dereference - if ptr_offset < 0 or ptr_offset >= len(self._binary): - return None - - rtti_addr = self._read_ptr(ptr_offset) - if rtti_addr is None or rtti_addr == 0: - return None - - rtti_offset = self._addr_to_offset(rtti_addr) - # Bounds check for second dereference - if rtti_offset < 0 or rtti_offset >= len(self._binary): + name = self._read_pascal_string(rtti_offset + 1) + if not name: + return None + if kind != RTTI_CLASS_MAGIC_BYTE: + return name + ptr_offset = rtti_offset + (self._read_byte(rtti_offset + 1) or 0) + 2 + if validate_pointers: + ptr = self._read_ptr(ptr_offset) + if ptr is None or (ptr and not 0 <= self._addr_to_offset(ptr) < len(self._binary)): return None + namespace = self._read_pascal_string(ptr_offset + 2 * self._settings.ptr_size + 2) + return f"{namespace}.{name}" if namespace else name - # Don't validate pointers for type resolution (they may be external) - return self._traverse_rtti_object(rtti_offset, validate_pointers=False) - - except Exception as e: - reraise_non_operational_exception(e) - LOGGER.debug(f"Error resolving type from double ptr at 0x{ptr_field_offset:x}: {e}") + def _resolve_type_from_double_ptr(self, ptr_field_offset: int) -> Optional[str]: + addr = self._read_ptr(ptr_field_offset) + if not addr: return None + cache = self._type_cache + if addr not in cache: + rtti_addr = self._read_ptr(self._addr_to_offset(addr)) + cache[addr] = self._traverse_rtti_object(self._addr_to_offset(rtti_addr), False) if rtti_addr else None + return cache[addr] + + def _take_entry(self) -> bool: + remaining = getattr(self, "_remaining_entries", 1000000) + if remaining <= 0: + if not getattr(self, "_budget_logged", False): + LOGGER.warning("Delphi metadata entry budget exhausted") + self._budget_logged = True + return False + self._remaining_entries = remaining - 1 + return True def _extract_method_info(self, method_entry_offset: int) -> Optional[MethodInfo]: - """Extract detailed information about a method from its MethodEntry structure.""" - if method_entry_offset < 0: + size = self._settings.ptr_size + length = self._read_short(method_entry_offset) + if length is None or length < size + 3 or method_entry_offset + length > len(self._binary): return None - try: - # Function entry point (offset +2) - func_addr = self._read_ptr(method_entry_offset + 2) - if func_addr is None: - return None - func_offset = self._addr_to_offset(func_addr) - - # Function name (Pascal string at offset +ptr_size+2) - func_name_offset = method_entry_offset + self._settings.ptr_size + 2 - func_name = self._read_pascal_string(func_name_offset) - if not func_name: - return None - - method_info = MethodInfo(function_offset=func_offset, function_name=func_name) - - # Return type (at offset +function_name_len+ptr_size+4) - return_type_field_offset = method_entry_offset + len(func_name) + 1 + self._settings.ptr_size + 4 - return_type_name = self._resolve_type_from_double_ptr(return_type_field_offset) - if return_type_name: - method_info.return_type = return_type_name - - # Parameter count (at offset +function_name_len+2*ptr_size+6) - param_count_offset = method_entry_offset + len(func_name) + 1 + 2 * self._settings.ptr_size + 6 - param_count = self._read_byte(param_count_offset) - - if param_count is not None and param_count > 0: - # Parse parameters - param_offset = param_count_offset + 2 - for _ in range(param_count): - # Parameter RTTI (double-dereferenced pointer) - param_type_name = self._resolve_type_from_double_ptr(param_offset) - - # Parameter name (Pascal string at offset +ptr_size+2) - param_name_offset = param_offset + self._settings.ptr_size + 2 - param_name = self._read_pascal_string(param_name_offset) - - method_info.parameters.append(ParameterInfo(type_name=param_type_name, parameter_name=param_name)) - - # Move to next parameter entry - param_offset = param_name_offset + len(param_name) + 1 + PARAM_ENTRY_PADDING - - return method_info - - except Exception as e: - reraise_non_operational_exception(e) - LOGGER.debug(f"Error extracting method info at 0x{method_entry_offset:x}: {e}") + end = method_entry_offset + length + func_addr = self._read_ptr(method_entry_offset + 2) + if not func_addr or not self._offset_in_code_area(self._addr_to_offset(func_addr)): + return None + name_offset = method_entry_offset + size + 2 + name = self._read_pascal_string(name_offset, end) + if not name: return None + method = MethodInfo(self._addr_to_offset(func_addr), name) + signature = name_offset + (self._read_byte(name_offset) or 0) + 1 + param_count_offset = signature + size + 4 + if param_count_offset + 2 > end or self._read_byte(signature) != 3: + return method + return_field = signature + 2 + return_type = self._resolve_type_from_double_ptr(return_field) if self._read_ptr(return_field) else "void" + parameters = [] + parameter_names = set() + offset = param_count_offset + 2 + for _ in range(self._read_byte(param_count_offset) or 0): + if not self._take_entry(): + return method + param_name_offset = offset + size + 2 + param_name = self._read_pascal_string(param_name_offset, end) + if not param_name: + return method + if param_name.casefold() in parameter_names: + return method + parameter_names.add(param_name.casefold()) + parameters.append(ParameterInfo(self._resolve_type_from_double_ptr(offset), param_name)) + offset = param_name_offset + (self._read_byte(param_name_offset) or 0) + 1 + PARAM_ENTRY_PADDING + method.return_type = return_type or "?" + method.parameters = parameters + return method def _parse_mdt(self, mdt_offset: int) -> list: - """Parse a Method Definition Table to extract method information.""" methods = [] - - # Read number of method entries (short at offset +2) - num_entries = self._read_short(mdt_offset + 2) - if num_entries is None or num_entries == 0: + count = self._read_short(mdt_offset) + if count is None: return methods - - # Method entry references start at offset +4 - me_refs_start = mdt_offset + 4 - - for i in range(num_entries): - me_ref_offset = me_refs_start + i * (self._settings.ptr_size + 4) - me_addr = self._read_ptr(me_ref_offset) - - if me_addr is None: + cursor = mdt_offset + 2 + for _ in range(count): + if not self._take_entry(): + return methods + length = self._read_short(cursor) + if not length or length < self._settings.ptr_size + 3 or cursor + length > len(self._binary): + return methods + method = self._extract_method_info(cursor) + if method: + methods.append(method) + cursor += length + count = self._read_short(cursor) + if count is None: + return methods + cursor += 2 + method_cache = {} + for _ in range(count): + addr = self._read_ptr(cursor) + if addr is None: break - - me_offset = self._addr_to_offset(me_addr) - method_info = self._extract_method_info(me_offset) - - if method_info: - methods.append(method_info) - + if not self._take_entry() or cursor + self._settings.ptr_size + 4 > len(self._binary): + break + offset = self._addr_to_offset(addr) + if offset not in method_cache: + method_cache[offset] = self._extract_method_info(offset) + method = method_cache[offset] + if method: + methods.append(method) + cursor += self._settings.ptr_size + 4 return methods def _parse_delphi_symbols(self): - """ - Main parsing routine to extract Delphi symbols. - - Pipeline: - 1. Scan code section for VMT structures using heuristics - 2. For each VMT, extract MDT and RTTI information - 3. Parse all methods in each MDT - 4. Build fully-qualified symbol names with templates - - Statistics are logged at debug level. - """ - # Step 1: Find all VMT structures - vmt_offsets = self._find_vmts() - LOGGER.debug(f"Found {len(vmt_offsets)} VMT candidates") - - if not vmt_offsets: - return - - # Statistics tracking - vmts_with_mdt = 0 - vmts_with_rtti = 0 - total_methods = 0 - - # Step 2: For each VMT, extract MDT and parse methods - for vmt_offset in vmt_offsets: - # Get MDT address from VMT - mdt_field_offset = vmt_offset + self._settings.mdt_offset - mdt_addr = self._read_ptr(mdt_field_offset) - - if mdt_addr is None: - continue - - mdt_offset = self._addr_to_offset(mdt_addr) - # Validate MDT offset is within bounds - if mdt_offset < 0 or mdt_offset >= len(self._binary): + for vmt in self._find_vmts(): + if not self._take_entry(): + break + mdt_addr = self._read_ptr(vmt + self._settings.mdt_offset) + if not mdt_addr: continue - - vmts_with_mdt += 1 - - # Get RTTI namespace for this VMT - rtti_field_offset = vmt_offset + self._settings.rtti_offset - rtti_addr = self._read_ptr(rtti_field_offset) - namespace = None - if rtti_addr is not None and rtti_addr != 0: - rtti_offset = self._addr_to_offset(rtti_addr) - if 0 <= rtti_offset < len(self._binary): - namespace = self._traverse_rtti_object(rtti_offset) - if namespace: - vmts_with_rtti += 1 - - # Parse all methods in this MDT - methods = self._parse_mdt(mdt_offset) - total_methods += len(methods) - - # Store function symbols - for method in methods: - func_addr = self._offset_to_addr(method.function_offset) - - # Build function name with namespace if available - if namespace: - # Use RecursiveDescentParser to validate namespace format - try: - parser = RecursiveDescentParser(namespace) - # Validate parsing succeeds (catches malformed namespaces) - parser.parse_fqn() - except (ValueError, IndexError) as e: - LOGGER.debug(f"Malformed namespace '{namespace}' encountered: {e}. Using as-is.") - # Invalid namespace format, use as-is - full_name = f"{namespace}.{method.function_name}" - else: - full_name = method.function_name - - # Add return type and parameters to make it more informative + rtti_addr = self._read_ptr(vmt + self._settings.rtti_offset) + namespace = self._traverse_rtti_object(self._addr_to_offset(rtti_addr)) if rtti_addr else None + if not namespace: + class_addr = self._read_ptr(vmt + 8 * self._settings.ptr_size) + namespace = self._read_pascal_string(self._addr_to_offset(class_addr)) if class_addr else "" + for method in self._parse_mdt(self._addr_to_offset(mdt_addr)): + name = f"{namespace}.{method.function_name}" if namespace else method.function_name if method.return_type != "void" or method.parameters: - param_str = ", ".join([f"{p.parameter_name}: {p.type_name or '?'}" for p in method.parameters]) - full_name = f"{full_name}({param_str}): {method.return_type}" - - self._func_symbols[func_addr] = full_name - - # Log statistics - LOGGER.debug( - f"DelphiReSym statistics: VMTs={len(vmt_offsets)}, " - f"with MDT={vmts_with_mdt}, with RTTI={vmts_with_rtti}, " - f"methods={total_methods}" - ) + params = ", ".join(f"{p.parameter_name}: {p.type_name or '?'}" for p in method.parameters) + name = f"{name}({params}): {method.return_type}" + addr = self._offset_to_addr(method.function_offset) + previous = self._func_symbols.get(addr, "") + if not previous or ("(" not in previous and "(" in name): + self._func_symbols[addr] = name def isSymbolProvider(self): return True @@ -703,11 +389,7 @@ def getApi(self, to_addr, absolute_addr=None): return None def getSymbol(self, address): - # sanitize output because the extractor may produce non-printable characters - symbol = self._func_symbols.get(address, "") - if not all(c.isprintable() or c.isspace() for c in symbol): - symbol = "" - return symbol + return self._func_symbols.get(address, "") def getFunctionSymbols(self): return self._func_symbols diff --git a/src/smda/common/labelprovider/DelphiVmtScanner.py b/src/smda/common/labelprovider/DelphiVmtScanner.py new file mode 100644 index 00000000..52a3c3cb --- /dev/null +++ b/src/smda/common/labelprovider/DelphiVmtScanner.py @@ -0,0 +1,33 @@ +"""Exact, bounded-memory candidate discovery for Delphi VMT self pointers.""" + +import struct +from typing import Iterator + + +def iter_vmt_self_references( + binary: bytes, base: int, ptr_size: int, distance: int, start: int, end: int, alignment: int = 1 +) -> Iterator[int]: + """Yield RVAs whose complete pointer equals base + RVA + distance. + + Within each address window the pointer's upper bytes are constant. Searching + for that necessary suffix in C avoids unpacking every byte or aligned slot. + Full-pointer verification preserves exactness, including carries and overlaps. + The exclusive end bounds the bytes available to read, not just pointer starts. + """ + limit = min(end, len(binary)) - ptr_size + 1 + max_pointer = (1 << (8 * ptr_size)) - 1 + start = max(start, 0, -base - distance) + unpack = struct.Struct(" max_pointer: + break + window_end = min(limit, start + 0x10000 - (expected & 0xFFFF)) + needle = expected.to_bytes(ptr_size, "little")[2:] + position = binary.find(needle, start + 2, window_end + ptr_size - 1) + while position != -1: + offset = position - 2 + if offset % alignment == 0 and unpack(binary, offset)[0] == base + offset + distance: + yield offset + position = binary.find(needle, position + 1, window_end + ptr_size - 1) + start = window_end diff --git a/src/smda/intel/X86Backend.py b/src/smda/intel/X86Backend.py index 24d6b475..1c1f32be 100644 --- a/src/smda/intel/X86Backend.py +++ b/src/smda/intel/X86Backend.py @@ -545,6 +545,13 @@ def _analyzeJmpInstruction(self, d, i, state): state.addBlockToQueue(jump_destination) state.addCodeRef(i_address, jump_destination, by_jump=True) else: + continuation = self._resolveDelphiFinallyContinuation(d, i, state) if i_op_str == "eax" else None + if continuation is not None: + state.addBlockToQueue(continuation) + state.addCodeRef(i_address, continuation, by_jump=True) + state.setNextInstructionReachable(False) + state.setBlockEndingInstruction(True) + return self._collectMemRegSlot(state, i_address, i_op_str) jumptable_targets = d.jumptable_analyzer.getJumpTargets(i, state) for target in jumptable_targets: @@ -554,6 +561,43 @@ def _analyzeJmpInstruction(self, d, i, state): state.setNextInstructionReachable(False) state.setBlockEndingInstruction(True) + @staticmethod + def _resolveDelphiFinallyContinuation(d, instruction, state): + """Recover the saved continuation past a Delphi HandleFinally thunk.""" + address, size, _, operand = instruction + if ( + operand != "eax" + or d.disassembly.binary_info.bitness != 32 + or not getattr(d.fc_manager, "_delphi_detected", False) + ): + return None + block = state.current_block + if not block or block[-1][2:4] != ("pop", "eax") or block[-1][0] + block[-1][1] != address: + return None + handler = d.disassembly.getBytes(address + size, 7) + if not handler or len(handler) != 7 or handler[0] != 0xE9 or handler[5] != 0xEB: + return None + handler_target = (address + size + 5 + int.from_bytes(handler[1:5], "little", signed=True)) & 0xFFFFFFFF + if not d.disassembly.binary_info.isInCodeAreas(handler_target): + return None + continuation = address + size + 7 + displacement = handler[6] - 256 if handler[6] >= 128 else handler[6] + cleanup = continuation + displacement + for index in range(len(block) - 2, -1, -1): + previous = block[index] + if previous[0] >= cleanup: + if previous[2] in ("push", "pop", "enter", "leave") or "esp" in previous[3]: + return None + continue + if previous[0] + previous[1] != cleanup: + return None + if previous[2] != "push" or previous[1] != 5 or previous[3] != hex(continuation): + return None + if d.disassembly.binary_info.isInCodeAreas(continuation): + return continuation + return None + return None + @staticmethod def _handleApiJumpTarget(d, state, instruction_addr, import_slot, dereferenced): resolved_api = d._handleApiTarget(instruction_addr, import_slot, dereferenced, slot=import_slot) diff --git a/tests/delphi_corpus/INNO-LICENSE.txt b/tests/delphi_corpus/INNO-LICENSE.txt new file mode 100644 index 00000000..3f3deb35 --- /dev/null +++ b/tests/delphi_corpus/INNO-LICENSE.txt @@ -0,0 +1,32 @@ +Inno Setup License +================== + +Except where otherwise noted, all of the documentation and software included in the Inno +Setup package is copyrighted by Jordan Russell. + +Copyright (C) 1997-2026 Jordan Russell. All rights reserved. +Portions Copyright (C) 2000-2026 Martijn Laan. All rights reserved. + +This software is provided "as-is," without any express or implied warranty. In no event shall +the author be held liable for any damages arising from the use of this software. + +Permission is granted to anyone to use this software for any purpose, including commercial +applications, and to alter and redistribute it, provided that the following conditions are met: + +1. All redistributions of source code files must retain all copyright notices that are currently + in place, and this list of conditions without modification. + +2. All redistributions in binary form must retain all occurrences of the above copyright notice + and web site addresses that are currently in place (for example, in the About boxes). + +3. The origin of this software must not be misrepresented; you must not claim that you wrote + the original software. If you use this software to distribute a product, an acknowledgment + in the product documentation would be appreciated but is not required. + +4. Modified versions in source or binary form must be plainly marked as such, and must not + be misrepresented as being the original software. + + +Jordan Russell +jr-2020 AT jrsoftware.org +https://jrsoftware.org/ \ No newline at end of file diff --git a/tests/delphi_corpus/README.md b/tests/delphi_corpus/README.md new file mode 100644 index 00000000..e70eaa7e --- /dev/null +++ b/tests/delphi_corpus/README.md @@ -0,0 +1,25 @@ +# Delphi recovery fixtures + +The two TObject JSON fixtures are **modified, sparse metadata excerpts**, not runnable copies of Inno Setup. +They retain original RVAs and pointer values from the official Inno Setup 7.1.0 installers, including +TObject's VMT, method entries, class RTTI, indirect type references, parameter records and short code +excerpts. The third JSON fixture is a short, unmodified finally-dispatch code excerpt. Unrecorded bytes +reconstruct as zero. Each fixture records its source URL and SHA-256. +Expected names are checked against the metadata layout in Lukas Wenz's thesis, tables 2–8. + +Inno Setup's tagged README identifies its build compiler as Delphi 12.3 Athens with the May Patch: +https://github.com/jrsoftware/issrc/blob/is-7_1_0/README.md + +Original software: Copyright (C) 1997–2026 Jordan Russell; portions Copyright (C) 2000–2026 +Martijn Laan. https://jrsoftware.org/ . See [INNO-LICENSE.txt](INNO-LICENSE.txt). +The SMDA project did not write the original software. The excerpts are modified for parser testing. + +`testDelphiReSymProvider.py` reconstructs these images and verifies the entire recovered symbol map. +Additional generated layouts cover published entries preceding extended RTTI, metadata outside code, +unaligned VMTs, multiple code regions, external ancestors, corrupt lengths/counts/pointers, invalid +names and bounded alias expansion. Pythia tests cover shared-table caching, record-bounded names, +external parents and x86 gating. These layouts are format tests, not evidence of compiler-version or +optimization-level coverage. + +The full audit corpus is described in `profiling/delphi_audit/manifest.json`; full installers and the +third-party upstream issue attachment are deliberately not bundled. Never execute those binaries. diff --git a/tests/delphi_corpus/inno-7.1.0-finally-32.json b/tests/delphi_corpus/inno-7.1.0-finally-32.json new file mode 100644 index 00000000..5613b32d --- /dev/null +++ b/tests/delphi_corpus/inno-7.1.0-finally-32.json @@ -0,0 +1,11 @@ +{ + "source_url": "https://github.com/jrsoftware/issrc/releases/download/is-7_1_0/innosetup-7.1.0-x86.exe", + "source_sha256": "f9671174e0d15ba9b4f6b56564c6aed32ea8db9c3cb9bf6f2af0850fe7894f60", + "base_addr": 4194304, + "entry": 4218732, + "continuation": 4218853, + "jump": 4218844, + "bytes": "558bec6a0053568bda8bf033c05568de5f400064ff306489208d55fc8bc6e8f17e00008bc3e806240000508bc3e8d6290000508b45fce8f5230000508b45fce8c4290000506a01a188794b0050e8cad8ffff83f8020f94c08bd833c05a595964891068e55f40008d45fce8e91e000058ffe0e9ad140000ebee8bc35e5b595dc3", + "handler_target": 4224144, + "handler_bytes": "8b442404f74004060000000f848f0000005331db56575555681075400064ff33" +} diff --git a/tests/delphi_corpus/inno-7.1.0-tobject-32.json b/tests/delphi_corpus/inno-7.1.0-tobject-32.json new file mode 100644 index 00000000..766beaed --- /dev/null +++ b/tests/delphi_corpus/inno-7.1.0-tobject-32.json @@ -0,0 +1,948 @@ +{ + "source_url": "https://github.com/jrsoftware/issrc/releases/download/is-7_1_0/innosetup-7.1.0-x86.exe", + "source_sha256": "f9671174e0d15ba9b4f6b56564c6aed32ea8db9c3cb9bf6f2af0850fe7894f60", + "bitness": 32, + "base_addr": 4194304, + "size": 26300, + "code_areas": [ + [ + 4219248, + 4220604 + ] + ], + "vmt_offset": 5868, + "segments": [ + [ + 0, + "4d5a50000200000004000f00ffff0000b80000000000000040001a00000000000000000000000000000000000000000000000000000000000000000000010000" + ], + [ + 4096, + "041040000307426f6f6c65616e" + ], + [ + 4252, + "a01040000107496e7465676572" + ], + [ + 4352, + "041140001407506f696e746572" + ], + [ + 4524, + "b0114000050b53686f7274537472696e67" + ], + [ + 4736, + "841240001206737472696e67" + ], + [ + 4828, + "e0124000130654436c617373" + ], + [ + 4848, + "f4124000010748524553554c54" + ], + [ + 4872, + "0c1340000e055447554944" + ], + [ + 5224, + "6c144000140f50496e74657266616365456e747279" + ], + [ + 5868, + "44174000000000000000000000000000681f40000000000044174000000000005a1840000800000000000000fc62400004634000f8644000f06440001065400014654000186540000c65400090614000a8614000dc6140000000220062184000" + ], + [ + 5968, + "88184000" + ], + [ + 5976, + "ac184000" + ], + [ + 5984, + "d5184000" + ], + [ + 5992, + "13194000" + ], + [ + 6000, + "42194000" + ], + [ + 6008, + "6b194000" + ], + [ + 6016, + "9f194000" + ], + [ + 6024, + "d8194000" + ], + [ + 6032, + "031a4000" + ], + [ + 6040, + "2c1a4000" + ], + [ + 6048, + "581a4000" + ], + [ + 6056, + "941a4000" + ], + [ + 6064, + "cf1a4000" + ], + [ + 6072, + "0a1b4000" + ], + [ + 6080, + "501b4000" + ], + [ + 6088, + "8d1b4000" + ], + [ + 6096, + "c71b4000" + ], + [ + 6104, + "011c4000" + ], + [ + 6112, + "471c4000" + ], + [ + 6120, + "851c4000" + ], + [ + 6128, + "b61c4000" + ], + [ + 6136, + "e91c4000" + ], + [ + 6144, + "1d1d4000" + ], + [ + 6152, + "501d4000" + ], + [ + 6160, + "7b1d4000" + ], + [ + 6168, + "ae1d4000" + ], + [ + 6176, + "091e4000" + ], + [ + 6184, + "3a1e4000" + ], + [ + 6192, + "6b1e4000" + ], + [ + 6200, + "a41e4000" + ], + [ + 6208, + "e31e4000" + ], + [ + 6216, + "0e1f4000" + ], + [ + 6224, + "3a1f4000" + ], + [ + 6234, + "07544f626a6563742600bc6140000643726561746503000000000008000108641f400000000453656c66020002002400ec614000044672656503" + ], + [ + 6293, + "00000000" + ], + [ + 6299, + "01" + ], + [ + 6301, + "641f4000" + ], + [ + 6307, + "0453656c66" + ], + [ + 6316, + "2900183c4b00" + ], + [ + 6357, + "3e00f86140000c496e6974496e7374616e636503" + ], + [ + 6378, + "641f4000" + ], + [ + 6384, + "02" + ], + [ + 6386, + "00000000" + ], + [ + 6392, + "0453656c66" + ], + [ + 6400, + "00114000" + ], + [ + 6406, + "08496e7374616e6365" + ], + [ + 6419, + "2f00986240000f436c65616e7570496e7374616e636503" + ], + [ + 6443, + "00000000" + ], + [ + 6449, + "01" + ], + [ + 6451, + "641f4000" + ], + [ + 6457, + "0453656c66" + ], + [ + 6466, + "2900183c4b00" + ], + [ + 6507, + "34007061400009436c6173734e616d6503" + ], + [ + 6525, + "80124000" + ], + [ + 6531, + "02" + ], + [ + 6533, + "00000000" + ], + [ + 6539, + "0453656c66" + ], + [ + 6553, + "0101" + ], + [ + 6559, + "3900846140000b436c6173734e616d65497303" + ], + [ + 6579, + "00104000" + ], + [ + 6585, + "02" + ], + [ + 6587, + "00000000" + ], + [ + 6593, + "0453656c66" + ], + [ + 6601, + "80124000" + ], + [ + 6607, + "044e616d65" + ], + [ + 6616, + "2b00183c4b00" + ], + [ + 6659, + "2900183c4b00" + ], + [ + 6700, + "2c00183c4b00" + ], + [ + 6744, + "3c00dc6440000c496e68657269747346726f6d03" + ], + [ + 6765, + "00104000" + ], + [ + 6771, + "02" + ], + [ + 6773, + "00000000" + ], + [ + 6779, + "0453656c66" + ], + [ + 6787, + "dc124000" + ], + [ + 6793, + "0641436c617373" + ], + [ + 6804, + "3b00446540000d4d6574686f644164647265737303" + ], + [ + 6826, + "00114000" + ], + [ + 6832, + "02" + ], + [ + 6834, + "00000000" + ], + [ + 6840, + "0453656c66" + ], + [ + 6848, + "ac114000" + ], + [ + 6854, + "044e616d65" + ], + [ + 6863, + "3b00b46540000d4d6574686f644164647265737303" + ], + [ + 6885, + "00114000" + ], + [ + 6891, + "02" + ], + [ + 6893, + "00000000" + ], + [ + 6899, + "0453656c66" + ], + [ + 6907, + "80124000" + ], + [ + 6913, + "044e616d65" + ], + [ + 6922, + "4600dc6540000a4d6574686f644e616d6503" + ], + [ + 6941, + "80124000" + ], + [ + 6947, + "03" + ], + [ + 6949, + "00000000" + ], + [ + 6955, + "0453656c66" + ], + [ + 6963, + "00114000" + ], + [ + 6969, + "0741646472657373" + ], + [ + 6986, + "0101" + ], + [ + 6992, + "3d00183c4b00" + ], + [ + 7053, + "3a00286640000c4669656c644164647265737303" + ], + [ + 7074, + "00114000" + ], + [ + 7080, + "02" + ], + [ + 7082, + "641f4000" + ], + [ + 7088, + "0453656c66" + ], + [ + 7096, + "ac114000" + ], + [ + 7102, + "044e616d65" + ], + [ + 7111, + "3a009c6640000c4669656c644164647265737303" + ], + [ + 7132, + "00114000" + ], + [ + 7138, + "02" + ], + [ + 7140, + "641f4000" + ], + [ + 7146, + "0453656c66" + ], + [ + 7154, + "80124000" + ], + [ + 7160, + "044e616d65" + ], + [ + 7169, + "46000c6340000c476574496e7465726661636503" + ], + [ + 7190, + "00104000" + ], + [ + 7196, + "03" + ], + [ + 7198, + "641f4000" + ], + [ + 7204, + "0453656c66" + ], + [ + 7212, + "08134000" + ], + [ + 7218, + "03494944" + ], + [ + 7225, + "00000000" + ], + [ + 7231, + "034f626a" + ], + [ + 7239, + "3e007063400011476574496e74657266616365456e74727903" + ], + [ + 7265, + "68144000" + ], + [ + 7271, + "02" + ], + [ + 7273, + "00000000" + ], + [ + 7279, + "0453656c66" + ], + [ + 7287, + "08134000" + ], + [ + 7293, + "03494944" + ], + [ + 7301, + "3100183c4b00" + ], + [ + 7350, + "3300b463400008556e69744e616d6503" + ], + [ + 7367, + "80124000" + ], + [ + 7373, + "02" + ], + [ + 7375, + "00000000" + ], + [ + 7381, + "0453656c66" + ], + [ + 7395, + "0101" + ], + [ + 7401, + "3400183c4b00" + ], + [ + 7453, + "3300fc62400006457175616c7303" + ], + [ + 7468, + "00104000" + ], + [ + 7474, + "02" + ], + [ + 7476, + "641f4000" + ], + [ + 7482, + "0453656c66" + ], + [ + 7490, + "641f4000" + ], + [ + 7496, + "034f626a" + ], + [ + 7504, + "2b00046340000b47657448617368436f646503" + ], + [ + 7524, + "9c104000" + ], + [ + 7530, + "01" + ], + [ + 7532, + "641f4000" + ], + [ + 7538, + "0453656c66" + ], + [ + 7547, + "3300f864400008546f537472696e6703" + ], + [ + 7564, + "80124000" + ], + [ + 7570, + "02" + ], + [ + 7572, + "641f4000" + ], + [ + 7578, + "0453656c66" + ], + [ + 7592, + "0101" + ], + [ + 7598, + "5b00f0644000115361666543616c6c457863657074696f6e03" + ], + [ + 7624, + "f0124000" + ], + [ + 7630, + "03" + ], + [ + 7632, + "641f4000" + ], + [ + 7638, + "0453656c66" + ], + [ + 7646, + "641f4000" + ], + [ + 7652, + "0c4578636570744f626a656374" + ], + [ + 7668, + "00114000" + ], + [ + 7674, + "0a45786365707441646472" + ], + [ + 7689, + "310010654000114166746572436f6e737472756374696f6e03" + ], + [ + 7715, + "00000000" + ], + [ + 7721, + "01" + ], + [ + 7723, + "641f4000" + ], + [ + 7729, + "0453656c66" + ], + [ + 7738, + "310014654000114265666f72654465737472756374696f6e03" + ], + [ + 7764, + "00000000" + ], + [ + 7770, + "01" + ], + [ + 7772, + "641f4000" + ], + [ + 7778, + "0453656c66" + ], + [ + 7787, + "39001865400008446973706174636803" + ], + [ + 7804, + "00000000" + ], + [ + 7810, + "02" + ], + [ + 7812, + "641f4000" + ], + [ + 7818, + "0453656c66" + ], + [ + 7826, + "00000000" + ], + [ + 7832, + "074d657373616765" + ], + [ + 7844, + "3f000c6540000e44656661756c7448616e646c657203" + ], + [ + 7867, + "00000000" + ], + [ + 7873, + "02" + ], + [ + 7875, + "641f4000" + ], + [ + 7881, + "0453656c66" + ], + [ + 7889, + "00000000" + ], + [ + 7895, + "074d657373616765" + ], + [ + 7907, + "2b00906140000b4e6577496e7374616e636503" + ], + [ + 7927, + "641f4000" + ], + [ + 7933, + "01" + ], + [ + 7935, + "00000000" + ], + [ + 7941, + "0453656c66" + ], + [ + 7950, + "2c00a86140000c46726565496e7374616e636503" + ], + [ + 7971, + "00000000" + ], + [ + 7977, + "01" + ], + [ + 7979, + "641f4000" + ], + [ + 7985, + "0453656c66" + ], + [ + 7994, + "2700dc6140000744657374726f7903" + ], + [ + 8010, + "00000000" + ], + [ + 8016, + "01" + ], + [ + 8018, + "641f4000" + ], + [ + 8024, + "0453656c66" + ], + [ + 8036, + "681f40000707544f626a65637444174000" + ], + [ + 8059, + "0653797374656d" + ], + [ + 24944, + "53568bf28bd88bd68b43c8e8e4feffff5e5bc3908b40c8e814ffffffc38d4000538bd88b43cce81df2ffff8bd08bc3e8540000005bc38bc0538bd88bc3e8e60000008bc3e81bf2ffff5bc39084d2740883c4f0e8fc04000084d2740fe84b050000648f050000000083c40cc3e88b05000084d27e05e82a050000c39085c07407b2018b08ff51fcc353565789c389d789d08b53cc31c9e8b9f9ffff891f89f889e28b4bac85c97401" + ], + [ + 25240, + "53565789c389c689c7e8ae78000089f08b368b56b48b76d085d27407e8373000" + ], + [ + 25340, + "3bc20f94c0c38bc08bd0c1ea0433c2c35356518bd98914248bf033c089038b14248b06e84c00000085c074298b501485" + ], + [ + 25456, + "535689c38b43ac85c074298b0883c0048b323b3075188b72043b700475108b72" + ], + [ + 25524, + "53568bda8b40b885c074428bd08bca83c102420fb60203c88d710a8bd38bc6e8" + ], + [ + 25820, + "3bc2740b8b40d085c074068b00ebf1b001c38bc0b8ffff0080c38bc053568bf28bd88bd68b03e869fcffff5e5bc38bc0c38d4000c38d4000c38d400052668b126609d274196681fa00c07312508b00e85cffffff89c15885c974035a" + ], + [ + 25924, + "53565731c931ff8a1aeb028b008b70c085f67419668b3e85ff741283c6028a4e" + ], + [ + 26036, + "535681c400ffffff8bf28bd88bd48bc6e82f7800008bd48bc3e872ffffff81c4" + ], + [ + 26076, + "56575389cf31db31c9eb028b008b70c085f67417668b0e85c9741083c6023b56" + ], + [ + 26152, + "53565731c931ff8a1a508b008b70bc85f67418668b3e85ff741183c6068a4e06" + ], + [ + 26268, + "535681c400ffffff8bf28bd88bd48bc6e8477700008bd48bc3e86effffff81c4" + ] + ], + "symbols": { + "4219324": "System.TObject.Create(Self: System.TObject): void", + "4219372": "System.TObject.Free(Self: System.TObject): void", + "4219384": "System.TObject.InitInstance(Self: ?, Instance: Pointer): System.TObject", + "4219544": "System.TObject.CleanupInstance(Self: System.TObject): void", + "4219248": "System.TObject.ClassName", + "4219268": "System.TObject.ClassNameIs(Self: ?, Name: string): Boolean", + "4220124": "System.TObject.InheritsFrom(Self: ?, AClass: TClass): Boolean", + "4220228": "System.TObject.MethodAddress(Self: ?, Name: ShortString): Pointer", + "4220340": "System.TObject.MethodAddress(Self: ?, Name: string): Pointer", + "4220380": "System.TObject.MethodName", + "4220456": "System.TObject.FieldAddress(Self: System.TObject, Name: ShortString): Pointer", + "4220572": "System.TObject.FieldAddress(Self: System.TObject, Name: string): Pointer", + "4219660": "System.TObject.GetInterface(Self: System.TObject, IID: TGUID, Obj: ?): Boolean", + "4219760": "System.TObject.GetInterfaceEntry(Self: ?, IID: TGUID): PInterfaceEntry", + "4219828": "System.TObject.UnitName", + "4219644": "System.TObject.Equals(Self: System.TObject, Obj: System.TObject): Boolean", + "4219652": "System.TObject.GetHashCode(Self: System.TObject): Integer", + "4220152": "System.TObject.ToString", + "4220144": "System.TObject.SafeCallException(Self: System.TObject, ExceptObject: System.TObject, ExceptAddr: Pointer): HRESULT", + "4220176": "System.TObject.AfterConstruction(Self: System.TObject): void", + "4220180": "System.TObject.BeforeDestruction(Self: System.TObject): void", + "4220184": "System.TObject.Dispatch(Self: System.TObject, Message: ?): void", + "4220172": "System.TObject.DefaultHandler(Self: System.TObject, Message: ?): void", + "4219280": "System.TObject.NewInstance(Self: ?): System.TObject", + "4219304": "System.TObject.FreeInstance(Self: System.TObject): void", + "4219356": "System.TObject.Destroy(Self: System.TObject): void" + } +} diff --git a/tests/delphi_corpus/inno-7.1.0-tobject-64.json b/tests/delphi_corpus/inno-7.1.0-tobject-64.json new file mode 100644 index 00000000..08be6e02 --- /dev/null +++ b/tests/delphi_corpus/inno-7.1.0-tobject-64.json @@ -0,0 +1,1043 @@ +{ + "source_url": "https://github.com/jrsoftware/issrc/releases/download/is-7_1_0/innosetup-7.1.0-x64.exe", + "source_sha256": "0362a383ed217d4c4239b5933866dd96d3eb2102737da92f80f6057a4b40df2f", + "bitness": 64, + "base_addr": 4194304, + "size": 39808, + "code_areas": [ + [ + 4231632, + 4234112 + ] + ], + "vmt_offset": 6448, + "segments": [ + [ + 0, + "4d5a50000200000004000f00ffff0000b80000000000000040001a00000000000000000000000000000000000000000000000000000000000000000000010000" + ], + [ + 4096, + "08104000000000000307426f6f6c65616e" + ], + [ + 4280, + "c0104000000000000107496e7465676572" + ], + [ + 4408, + "40114000000000001407506f696e746572" + ], + [ + 4640, + "2812400000000000050b53686f7274537472696e67" + ], + [ + 4896, + "28134000000000001206737472696e67" + ], + [ + 5016, + "a013400000000000130654436c617373" + ], + [ + 5048, + "c013400000000000010748524553554c54" + ], + [ + 5080, + "e0134000000000000e055447554944" + ], + [ + 5552, + "b815400000000000140f50496e74657266616365456e747279" + ], + [ + 6448, + "f81940000000000000000000000000000000000000000000000000000000000058254000000000000000000000000000f8194000000000000000000000000000ba1b400000000000100000000000000000000000000000009095400000000000a095400000000000b098400000000000a098400000000000e098400000000000f0984000000000000099400000000000d09840000000000020924000000000005092400000000000309340000000000070924000000000008092400000000000909240000000000000002500c21b400000000000" + ], + [ + 6664, + "f41b400000000000" + ], + [ + 6676, + "241c400000000000" + ], + [ + 6688, + "591c400000000000" + ], + [ + 6700, + "a71c400000000000" + ], + [ + 6712, + "e21c400000000000" + ], + [ + 6724, + "171d400000000000" + ], + [ + 6736, + "5b1d400000000000" + ], + [ + 6748, + "a41d400000000000" + ], + [ + 6760, + "db1d400000000000" + ], + [ + 6772, + "101e400000000000" + ], + [ + 6784, + "481e400000000000" + ], + [ + 6796, + "941e400000000000" + ], + [ + 6808, + "df1e400000000000" + ], + [ + 6820, + "2a1f400000000000" + ], + [ + 6832, + "841f400000000000" + ], + [ + 6844, + "d11f400000000000" + ], + [ + 6856, + "1b20400000000000" + ], + [ + 6868, + "6520400000000000" + ], + [ + 6880, + "bf20400000000000" + ], + [ + 6892, + "0d21400000000000" + ], + [ + 6904, + "4a21400000000000" + ], + [ + 6916, + "8d21400000000000" + ], + [ + 6928, + "d121400000000000" + ], + [ + 6940, + "1422400000000000" + ], + [ + 6952, + "4b22400000000000" + ], + [ + 6964, + "8e22400000000000" + ], + [ + 6976, + "fd22400000000000" + ], + [ + 6988, + "3a23400000000000" + ], + [ + 7000, + "7723400000000000" + ], + [ + 7012, + "c023400000000000" + ], + [ + 7024, + "0f24400000000000" + ], + [ + 7036, + "4624400000000000" + ], + [ + 7048, + "7e24400000000000" + ], + [ + 7060, + "b124400000000000" + ], + [ + 7072, + "e624400000000000" + ], + [ + 7084, + "1b25400000000000" + ], + [ + 7098, + "07544f626a6563743200a092400000000000064372656174650300000000000000000020000108502540000000000000000453656c660200020030006093400000000000044672656503" + ], + [ + 7173, + "0000000000000000" + ], + [ + 7183, + "01" + ], + [ + 7185, + "5025400000000000" + ], + [ + 7195, + "0453656c66" + ], + [ + 7204, + "35000091500000000000" + ], + [ + 7257, + "4e0000944000000000000c496e6974496e7374616e636503" + ], + [ + 7282, + "5025400000000000" + ], + [ + 7292, + "02" + ], + [ + 7294, + "0000000000000000" + ], + [ + 7304, + "0453656c66" + ], + [ + 7312, + "3811400000000000" + ], + [ + 7322, + "08496e7374616e6365" + ], + [ + 7335, + "3b00c0944000000000000f436c65616e7570496e7374616e636503" + ], + [ + 7363, + "0000000000000000" + ], + [ + 7373, + "01" + ], + [ + 7375, + "5025400000000000" + ], + [ + 7385, + "0453656c66" + ], + [ + 7394, + "35000091500000000000" + ], + [ + 7447, + "4400d09140000000000009436c6173734e616d6503" + ], + [ + 7469, + "2013400000000000" + ], + [ + 7479, + "02" + ], + [ + 7481, + "0000000000000000" + ], + [ + 7491, + "0453656c66" + ], + [ + 7509, + "0101" + ], + [ + 7515, + "490000924000000000000b436c6173734e616d65497303" + ], + [ + 7539, + "0010400000000000" + ], + [ + 7549, + "02" + ], + [ + 7551, + "0000000000000000" + ], + [ + 7561, + "0453656c66" + ], + [ + 7569, + "2013400000000000" + ], + [ + 7579, + "044e616d65" + ], + [ + 7588, + "37000091500000000000" + ], + [ + 7643, + "35000091500000000000" + ], + [ + 7696, + "38000091500000000000" + ], + [ + 7752, + "4c0080984000000000000c496e68657269747346726f6d03" + ], + [ + 7777, + "0010400000000000" + ], + [ + 7787, + "02" + ], + [ + 7789, + "0000000000000000" + ], + [ + 7799, + "0453656c66" + ], + [ + 7807, + "9813400000000000" + ], + [ + 7817, + "0641436c617373" + ], + [ + 7828, + "4b0070994000000000000d4d6574686f644164647265737303" + ], + [ + 7854, + "3811400000000000" + ], + [ + 7864, + "02" + ], + [ + 7866, + "0000000000000000" + ], + [ + 7876, + "0453656c66" + ], + [ + 7884, + "2012400000000000" + ], + [ + 7894, + "044e616d65" + ], + [ + 7903, + "4b00009a4000000000000d4d6574686f644164647265737303" + ], + [ + 7929, + "3811400000000000" + ], + [ + 7939, + "02" + ], + [ + 7941, + "0000000000000000" + ], + [ + 7951, + "0453656c66" + ], + [ + 7959, + "2013400000000000" + ], + [ + 7969, + "044e616d65" + ], + [ + 7978, + "5a00309a4000000000000a4d6574686f644e616d6503" + ], + [ + 8001, + "2013400000000000" + ], + [ + 8011, + "03" + ], + [ + 8013, + "0000000000000000" + ], + [ + 8023, + "0453656c66" + ], + [ + 8031, + "3811400000000000" + ], + [ + 8041, + "0741646472657373" + ], + [ + 8062, + "0101" + ], + [ + 8068, + "4d000091500000000000" + ], + [ + 8145, + "4a00b09a4000000000000c4669656c644164647265737303" + ], + [ + 8170, + "3811400000000000" + ], + [ + 8180, + "02" + ], + [ + 8182, + "5025400000000000" + ], + [ + 8192, + "0453656c66" + ], + [ + 8200, + "2012400000000000" + ], + [ + 8210, + "044e616d65" + ], + [ + 8219, + "4a00609b4000000000000c4669656c644164647265737303" + ], + [ + 8244, + "3811400000000000" + ], + [ + 8254, + "02" + ], + [ + 8256, + "5025400000000000" + ], + [ + 8266, + "0453656c66" + ], + [ + 8274, + "2013400000000000" + ], + [ + 8284, + "044e616d65" + ], + [ + 8293, + "5a00b0954000000000000c476574496e7465726661636503" + ], + [ + 8318, + "0010400000000000" + ], + [ + 8328, + "03" + ], + [ + 8330, + "5025400000000000" + ], + [ + 8340, + "0453656c66" + ], + [ + 8348, + "d813400000000000" + ], + [ + 8358, + "03494944" + ], + [ + 8365, + "0000000000000000" + ], + [ + 8375, + "034f626a" + ], + [ + 8383, + "4e00509640000000000011476574496e74657266616365456e74727903" + ], + [ + 8413, + "b015400000000000" + ], + [ + 8423, + "02" + ], + [ + 8425, + "0000000000000000" + ], + [ + 8435, + "0453656c66" + ], + [ + 8443, + "d813400000000000" + ], + [ + 8453, + "03494944" + ], + [ + 8461, + "3d000091500000000000" + ], + [ + 8522, + "4300d09640000000000008556e69744e616d6503" + ], + [ + 8543, + "2013400000000000" + ], + [ + 8553, + "02" + ], + [ + 8555, + "0000000000000000" + ], + [ + 8565, + "0453656c66" + ], + [ + 8583, + "0101" + ], + [ + 8589, + "44000091500000000000" + ], + [ + 8657, + "4300909540000000000006457175616c7303" + ], + [ + 8676, + "0010400000000000" + ], + [ + 8686, + "02" + ], + [ + 8688, + "5025400000000000" + ], + [ + 8698, + "0453656c66" + ], + [ + 8706, + "5025400000000000" + ], + [ + 8716, + "034f626a" + ], + [ + 8724, + "3700a0954000000000000b47657448617368436f646503" + ], + [ + 8748, + "b810400000000000" + ], + [ + 8758, + "01" + ], + [ + 8760, + "5025400000000000" + ], + [ + 8770, + "0453656c66" + ], + [ + 8779, + "4300b09840000000000008546f537472696e6703" + ], + [ + 8800, + "2013400000000000" + ], + [ + 8810, + "02" + ], + [ + 8812, + "5025400000000000" + ], + [ + 8822, + "0453656c66" + ], + [ + 8840, + "0101" + ], + [ + 8846, + "6f00a098400000000000115361666543616c6c457863657074696f6e03" + ], + [ + 8876, + "b813400000000000" + ], + [ + 8886, + "03" + ], + [ + 8888, + "5025400000000000" + ], + [ + 8898, + "0453656c66" + ], + [ + 8906, + "5025400000000000" + ], + [ + 8916, + "0c4578636570744f626a656374" + ], + [ + 8932, + "3811400000000000" + ], + [ + 8942, + "0a45786365707441646472" + ], + [ + 8957, + "3d00e098400000000000114166746572436f6e737472756374696f6e03" + ], + [ + 8987, + "0000000000000000" + ], + [ + 8997, + "01" + ], + [ + 8999, + "5025400000000000" + ], + [ + 9009, + "0453656c66" + ], + [ + 9018, + "3d00f098400000000000114265666f72654465737472756374696f6e03" + ], + [ + 9048, + "0000000000000000" + ], + [ + 9058, + "01" + ], + [ + 9060, + "5025400000000000" + ], + [ + 9070, + "0453656c66" + ], + [ + 9079, + "4900009940000000000008446973706174636803" + ], + [ + 9100, + "0000000000000000" + ], + [ + 9110, + "02" + ], + [ + 9112, + "5025400000000000" + ], + [ + 9122, + "0453656c66" + ], + [ + 9130, + "0000000000000000" + ], + [ + 9140, + "074d657373616765" + ], + [ + 9152, + "4f00d0984000000000000e44656661756c7448616e646c657203" + ], + [ + 9179, + "0000000000000000" + ], + [ + 9189, + "02" + ], + [ + 9191, + "5025400000000000" + ], + [ + 9201, + "0453656c66" + ], + [ + 9209, + "0000000000000000" + ], + [ + 9219, + "074d657373616765" + ], + [ + 9231, + "370020924000000000000b4e6577496e7374616e636503" + ], + [ + 9255, + "5025400000000000" + ], + [ + 9265, + "01" + ], + [ + 9267, + "0000000000000000" + ], + [ + 9277, + "0453656c66" + ], + [ + 9286, + "380050924000000000000c46726565496e7374616e636503" + ], + [ + 9311, + "0000000000000000" + ], + [ + 9321, + "01" + ], + [ + 9323, + "5025400000000000" + ], + [ + 9333, + "0453656c66" + ], + [ + 9342, + "330030934000000000000744657374726f7903" + ], + [ + 9362, + "0000000000000000" + ], + [ + 9372, + "01" + ], + [ + 9374, + "5025400000000000" + ], + [ + 9384, + "0453656c66" + ], + [ + 9393, + "35007092400000000000094350505f4142495f3103" + ], + [ + 9415, + "0000000000000000" + ], + [ + 9425, + "01" + ], + [ + 9427, + "5025400000000000" + ], + [ + 9437, + "0453656c66" + ], + [ + 9446, + "35008092400000000000094350505f4142495f3203" + ], + [ + 9468, + "0000000000000000" + ], + [ + 9478, + "01" + ], + [ + 9480, + "5025400000000000" + ], + [ + 9490, + "0453656c66" + ], + [ + 9499, + "35009092400000000000094350505f4142495f3303" + ], + [ + 9521, + "0000000000000000" + ], + [ + 9531, + "01" + ], + [ + 9533, + "5025400000000000" + ], + [ + 9543, + "0453656c66" + ], + [ + 9552, + "58254000000000000707544f626a656374f819400000000000" + ], + [ + 9587, + "0653797374656d" + ], + [ + 37328, + "534883ec204889d34889d8488b9178ffffff4889c1e8b6fdffff4889d84883c4" + ], + [ + 37376, + "4883ec28488b8978ffffffe8f0fdffff4883c428c3cccccccccccccccccccccc534883ec204889cb48634b80e8afdcffff4889d94889c2e8c40100004883c420" + ], + [ + 37456, + "534883ec204889cb4889d9e8600200004889d9e8a8dcffff4883c4205bc3ccccc3ccccccccccccccccccccccccccccccc3ccccccccccccccccccccccccccccccc3cccccccccccccccccccccccccccccc55534883ec38488bec48896d2848894d50885558807d58007412488b4d50480f" + ], + [ + 37680, + "56534883ec284889cb4089d64889d94089f2e8e90800004084f67e084889d9e8" + ], + [ + 37728, + "534883ec204885c9740e4889c8b201488b194889c1ff53e04883c4205bc3cccc" + ], + [ + 37888, + "555756534883ec28488bec4889cb4889d64889df4889f1486357804d33c0e85d" + ], + [ + 38080, + "56534883ec284889cb4889de4889f1e8acb40000488b064889c6488b8650ffff" + ], + [ + 38288, + "483bd10f94c0c3cccccccccccccccccc488bc148c1e80448c1e92033c1c3cccc555756534883ec38488bec4889cb4889d64c89c748c70700000000488b0b4889" + ], + [ + 38480, + "554883ec10488bec4c8b8140ffffff4d85c0744d498d40084d33c9458b104183" + ], + [ + 38608, + "56534883ec284889d3488b8158ffffff4885c07442480fb64801488d74081448" + ], + [ + 39040, + "4889c8483bc2740e488b40884885c07408488b00ebed90b001c3ccccccccccccb8ffff0080c3cccccccccccccccccccc534883ec204889d3488b094889dae80df9ffff4889d84883c4205bc3ccccccccc3ccccccccccccccccccccccccccccccc3ccccccccccccccccccccccccccccccc3cccccccccccccccccccccccccccccc555756534883ec38488bec4889cb4889d6480fb7066685c0743a663d00c07334" + ], + [ + 39280, + "41555756534883ec28488954245833db4989cd498b8568ffffff4885c0740a48" + ], + [ + 39424, + "55534881ec28010000488bec4889cb488d4d20e808b300004889d9488d5520e8" + ], + [ + 39472, + "5756534883ec204889cb4889d64c89c74889f1e8482a00004889da488b8268ff" + ], + [ + 39600, + "415641555756534883ec3048894c2428488954246833db488b4424284c8b3049" + ], + [ + 39776, + "55534881ec28010000488bec4889cb488d4d20e8a8b100004889d9488d5520e8" + ] + ], + "symbols": { + "4231840": "System.TObject.Create(Self: System.TObject): void", + "4232032": "System.TObject.Free(Self: System.TObject): void", + "4232192": "System.TObject.InitInstance(Self: ?, Instance: Pointer): System.TObject", + "4232384": "System.TObject.CleanupInstance(Self: System.TObject): void", + "4231632": "System.TObject.ClassName", + "4231680": "System.TObject.ClassNameIs(Self: ?, Name: string): Boolean", + "4233344": "System.TObject.InheritsFrom(Self: ?, AClass: TClass): Boolean", + "4233584": "System.TObject.MethodAddress(Self: ?, Name: ShortString): Pointer", + "4233728": "System.TObject.MethodAddress(Self: ?, Name: string): Pointer", + "4233776": "System.TObject.MethodName", + "4233904": "System.TObject.FieldAddress(Self: System.TObject, Name: ShortString): Pointer", + "4234080": "System.TObject.FieldAddress(Self: System.TObject, Name: string): Pointer", + "4232624": "System.TObject.GetInterface(Self: System.TObject, IID: TGUID, Obj: ?): Boolean", + "4232784": "System.TObject.GetInterfaceEntry(Self: ?, IID: TGUID): PInterfaceEntry", + "4232912": "System.TObject.UnitName", + "4232592": "System.TObject.Equals(Self: System.TObject, Obj: System.TObject): Boolean", + "4232608": "System.TObject.GetHashCode(Self: System.TObject): Integer", + "4233392": "System.TObject.ToString", + "4233376": "System.TObject.SafeCallException(Self: System.TObject, ExceptObject: System.TObject, ExceptAddr: Pointer): HRESULT", + "4233440": "System.TObject.AfterConstruction(Self: System.TObject): void", + "4233456": "System.TObject.BeforeDestruction(Self: System.TObject): void", + "4233472": "System.TObject.Dispatch(Self: System.TObject, Message: ?): void", + "4233424": "System.TObject.DefaultHandler(Self: System.TObject, Message: ?): void", + "4231712": "System.TObject.NewInstance(Self: ?): System.TObject", + "4231760": "System.TObject.FreeInstance(Self: System.TObject): void", + "4231984": "System.TObject.Destroy(Self: System.TObject): void", + "4231792": "System.TObject.CPP_ABI_1(Self: System.TObject): void", + "4231808": "System.TObject.CPP_ABI_2(Self: System.TObject): void", + "4231824": "System.TObject.CPP_ABI_3(Self: System.TObject): void" + } +} diff --git a/tests/testDelphiFinallyContinuation.py b/tests/testDelphiFinallyContinuation.py new file mode 100644 index 00000000..57ba7c8d --- /dev/null +++ b/tests/testDelphiFinallyContinuation.py @@ -0,0 +1,96 @@ +from types import SimpleNamespace + +import pytest + +from smda.Disassembler import Disassembler +from smda.intel.X86Backend import X86Backend +from smda.SmdaConfig import SmdaConfig + +CODE = bytes.fromhex("68171040008d45f8e8f300000058ffe0e9eb000000ebee8bc6c3") +BASE = 0x401000 + + +def resolver(code=CODE, bitness=32, delphi=True, pushed_target=BASE + 0x17): + block = [ + (BASE, 5, "push", hex(pushed_target)), + (BASE + 5, 3, "lea", "eax, [ebp - 8]"), + (BASE + 8, 5, "call", "0x401100"), + (BASE + 13, 1, "pop", "eax"), + ] + d = SimpleNamespace( + fc_manager=SimpleNamespace(_delphi_detected=delphi), + disassembly=SimpleNamespace( + binary_info=SimpleNamespace(bitness=bitness, isInCodeAreas=lambda addr: BASE <= addr < BASE + 0x200), + getBytes=lambda addr, size: code[addr - BASE : addr - BASE + size], + ), + ) + state = SimpleNamespace(current_block=block) + return X86Backend._resolveDelphiFinallyContinuation(d, (BASE + 14, 2, "jmp", "eax"), state) + + +def test_resolves_delphi_finally_stack_continuation(): + assert resolver() == BASE + 0x17 + + +@pytest.mark.parametrize("bitness,delphi", [(64, True), (32, False)]) +def test_requires_32_bit_delphi_evidence(bitness, delphi): + assert resolver(bitness=bitness, delphi=delphi) is None + + +@pytest.mark.parametrize("offset,value", [(16, 0x90), (21, 0x90), (22, 0xEF)]) +def test_rejects_broken_handler_or_cleanup_link(offset, value): + code = bytearray(CODE) + code[offset] = value + assert resolver(bytes(code)) is None + + +def test_rejects_mismatched_saved_continuation(): + assert resolver(pushed_target=BASE + 0x18) is None + + +def test_continuation_is_booked_in_the_owning_function(): + from tests.testDelphiReSymProvider import image + + info, layout = image(vmt=0x200, method_target=0x1000) + layout.data[0x1000 : 0x1000 + len(CODE)] = CODE + layout.data[0x1100] = 0xC3 + info.binary = bytes(layout.data) + info.code_areas = [(info.base_addr + 0x200, info.base_addr + 0x1800)] + config = SmdaConfig() + config.USE_ALIGNMENT = False + report = Disassembler(config).disassembleBuffer( + info.binary, info.base_addr, bitness=info.bitness, code_areas=info.code_areas + ) + function = report.getFunction(BASE) + instructions = {ins.offset for block in function.getBlocks() for ins in block.getInstructions()} + assert BASE + 0x17 in instructions + assert BASE + 0x19 in instructions + assert BASE + 0x17 in function.blockrefs[BASE] + assert report.getFunction(BASE + 0x17) is None + + +def test_real_inno_cleanup_dispatch(): + import json + from pathlib import Path + + from smda.common.BinaryInfo import BinaryInfo + from smda.DisassemblyResult import DisassemblyResult + + fixture = json.loads((Path(__file__).parent / "delphi_corpus/inno-7.1.0-finally-32.json").read_text()) + code = bytes.fromhex(fixture["bytes"]) + info = BinaryInfo(code) + info.base_addr = fixture["entry"] + info.bitness = 32 + info.code_areas = [ + (fixture["entry"], fixture["entry"] + len(code)), + (fixture["handler_target"], fixture["handler_target"] + len(bytes.fromhex(fixture["handler_bytes"]))), + ] + result = DisassemblyResult() + result.setBinaryInfo(info) + md = X86Backend().createCapstone(32) + block = list(md.disasm_lite(code[: fixture["jump"] - fixture["entry"]], fixture["entry"])) + d = SimpleNamespace(disassembly=result, fc_manager=SimpleNamespace(_delphi_detected=True)) + target = X86Backend._resolveDelphiFinallyContinuation( + d, (fixture["jump"], 2, "jmp", "eax"), SimpleNamespace(current_block=block) + ) + assert target == fixture["continuation"] diff --git a/tests/testDelphiReSymProvider.py b/tests/testDelphiReSymProvider.py new file mode 100644 index 00000000..e4800066 --- /dev/null +++ b/tests/testDelphiReSymProvider.py @@ -0,0 +1,554 @@ +import json +import struct +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import patch + +import pytest + +from smda.common.BinaryInfo import BinaryInfo +from smda.common.labelprovider.DelphiPythiaProvider import ( + MAX_INTERFACE_METHOD_SCAN, + DelphiClassInfo, + DelphiPythiaProvider, +) +from smda.common.labelprovider.DelphiReSymProvider import DelphiReSymProvider +from smda.common.LanguageAnalyzer import LanguageAnalyzer +from smda.SmdaConfig import SmdaConfig + +CORPUS = Path(__file__).parent / "delphi_corpus" + + +def load_fixture(bitness): + fixture = json.loads((CORPUS / f"inno-7.1.0-tobject-{bitness}.json").read_text()) + data = bytearray(fixture["size"]) + for offset, hex_data in fixture["segments"]: + segment = bytes.fromhex(hex_data) + data[offset : offset + len(segment)] = segment + info = BinaryInfo(bytes(data)) + info.base_addr = fixture["base_addr"] + info.bitness = fixture["bitness"] + info.code_areas = fixture["code_areas"] + return info, fixture + + +def image(bitness=32, vmt=0x201, metadata=0x500, method_target=0x1100): + size = bitness // 8 + base = 0x400000 if bitness == 32 else 0x140000000 + jump = 88 if bitness == 32 else 200 + data = bytearray(0x2000) + data[:2] = b"MZ" + data[0x80:0x87] = b"TObject" + + def ptr(offset, value): + struct.pack_into(" Date: Mon, 26 Oct 2026 13:32:58 +0000 Subject: [PATCH 2/5] style: ruff format --- profiling/delphi_audit/audit.py | 9 ++++++--- .../common/labelprovider/DelphiPythiaProvider.py | 16 +++++++--------- tests/testDelphiReSymProvider.py | 15 ++++++--------- 3 files changed, 19 insertions(+), 21 deletions(-) diff --git a/profiling/delphi_audit/audit.py b/profiling/delphi_audit/audit.py index 7c3e7f27..fe6c7c62 100644 --- a/profiling/delphi_audit/audit.py +++ b/profiling/delphi_audit/audit.py @@ -74,8 +74,7 @@ def provider_profile(path, provider_type): "seconds": entry.totaltime, } for entry in profiler.getstats() - if isinstance(entry.code, types.CodeType) - and "Delphi" in entry.code.co_filename + if isinstance(entry.code, types.CodeType) and "Delphi" in entry.code.co_filename ), key=lambda entry: entry["seconds"], reverse=True, @@ -153,7 +152,11 @@ def ghidra_result(path, install, disabled_analyzers=()): if not edge.getFlowType().isCall() ] blocks.append( - [int(block.getFirstStartAddress().getOffset()), int(block.getMaxAddress().getOffset()) + 1, edges] + [ + int(block.getFirstStartAddress().getOffset()), + int(block.getMaxAddress().getOffset()) + 1, + edges, + ] ) functions[int(function.getEntryPoint().getOffset())] = { "name": str(function.getName(True)), diff --git a/src/smda/common/labelprovider/DelphiPythiaProvider.py b/src/smda/common/labelprovider/DelphiPythiaProvider.py index 2d7bc22d..eb47999b 100644 --- a/src/smda/common/labelprovider/DelphiPythiaProvider.py +++ b/src/smda/common/labelprovider/DelphiPythiaProvider.py @@ -342,7 +342,11 @@ def _extract_dynamic_methods( function_offset += self._ptr_size def _extract_method_table_symbols( - self, class_info: DelphiClassInfo, function_offsets: set, name_mapping: Dict[int, str], seen_tables: Optional[set] = None + self, + class_info: DelphiClassInfo, + function_offsets: set, + name_mapping: Dict[int, str], + seen_tables: Optional[set] = None, ) -> None: """Parse the classic Delphi method table and recover symbol names.""" if not class_info.method_table_addr: @@ -359,16 +363,10 @@ def _extract_method_table_symbols( for _ in range(entry_count): entry_size = self._read_word(entry_offset) minimum_entry_size = 2 + self._ptr_size + 1 - if ( - entry_size is None - or entry_size < minimum_entry_size - or entry_offset + entry_size > len(self._binary) - ): + if entry_size is None or entry_size < minimum_entry_size or entry_offset + entry_size > len(self._binary): break method_addr = self._read_ptr(entry_offset + 2, self._ptr_size) - method_name = self._read_pascal_string( - entry_offset + 2 + self._ptr_size, entry_offset + entry_size - ) + method_name = self._read_pascal_string(entry_offset + 2 + self._ptr_size, entry_offset + entry_size) if method_addr and self._binary_info.isInCodeAreas(method_addr): function_offsets.add(method_addr) if method_name: diff --git a/tests/testDelphiReSymProvider.py b/tests/testDelphiReSymProvider.py index e4800066..2ac487f2 100644 --- a/tests/testDelphiReSymProvider.py +++ b/tests/testDelphiReSymProvider.py @@ -272,9 +272,10 @@ def test_pythia_parses_shared_method_table_once(): DelphiClassInfo(method_table_addr=base + table), DelphiClassInfo(method_table_addr=base + table), ] - with patch.object(provider, "_scan_code_range", return_value=classes), patch.object( - provider, "_read_word", wraps=provider._read_word - ) as read_word: + with ( + patch.object(provider, "_scan_code_range", return_value=classes), + patch.object(provider, "_read_word", wraps=provider._read_word) as read_word, + ): provider._parse_delphi_objects() assert read_word.call_count == 2 assert provider.getFunctionSymbols() == {target: "Run"} @@ -325,9 +326,7 @@ def test_pythia_method_name_cannot_read_past_entry(): provider._bitness = 32 offsets = set() names = {} - provider._extract_method_table_symbols( - DelphiClassInfo(method_table_addr=info.base_addr + table), offsets, names - ) + provider._extract_method_table_symbols(DelphiClassInfo(method_table_addr=info.base_addr + table), offsets, names) assert offsets == {target} assert names == {} @@ -353,9 +352,7 @@ def test_pythia_bounds_interface_method_walk(): provider._binary = bytes(data) offsets = set() with patch.object(provider, "_read_ptr", wraps=provider._read_ptr) as read_ptr: - provider._extract_interface_methods( - DelphiClassInfo(interface_table_addr=base + table), offsets - ) + provider._extract_interface_methods(DelphiClassInfo(interface_table_addr=base + table), offsets) assert offsets == {target} assert read_ptr.call_count == MAX_INTERFACE_METHOD_SCAN + 1 From 1ffc13f43ea41f2734c0e8688cce036719ca04b1 Mon Sep 17 00:00:00 2001 From: r0ny123 <49360849+r0ny123@users.noreply.github.com> Date: Thu, 29 Oct 2026 22:36:31 +0530 Subject: [PATCH 3/5] fix(intel): reject stack changes in finally continuations Reject flag-stack and push/pop-all instructions, and writes involving SP, before inferring a saved Delphi finally continuation. Add nine cases decoded by Capstone and correct the audit output link. Validation: 171 focused tests; eight control report hashes unchanged. --- profiling/delphi_audit/README.md | 2 +- src/smda/intel/X86Backend.py | 7 +++++- tests/testDelphiFinallyContinuation.py | 33 ++++++++++++++++++++++++++ 3 files changed, 40 insertions(+), 2 deletions(-) diff --git a/profiling/delphi_audit/README.md b/profiling/delphi_audit/README.md index 47cb7661..bbdc1b17 100644 --- a/profiling/delphi_audit/README.md +++ b/profiling/delphi_audit/README.md @@ -151,7 +151,7 @@ Ghidra's process exit status as proof that a script succeeded. The upstream READ asks users to run its mutating script twice; this audit's extraction comparison requires no second mutation pass. -Machine-readable counts and disagreement examples are in [results.json](results.json). +The audit CLI writes machine-readable measurements to the directory passed with `--output`. Function-count differences are **not** automatically false positives or negatives. For example, the supplied sample's compressed switch tables contain real cases missing from Ghidra's recovered bodies; conversely, Ghidra exposed the constructor diff --git a/src/smda/intel/X86Backend.py b/src/smda/intel/X86Backend.py index 1c1f32be..4716a3b0 100644 --- a/src/smda/intel/X86Backend.py +++ b/src/smda/intel/X86Backend.py @@ -586,7 +586,12 @@ def _resolveDelphiFinallyContinuation(d, instruction, state): for index in range(len(block) - 2, -1, -1): previous = block[index] if previous[0] >= cleanup: - if previous[2] in ("push", "pop", "enter", "leave") or "esp" in previous[3]: + mnemonic = previous[2].rpartition(" ")[2] + if ( + mnemonic.startswith(("push", "pop")) + or mnemonic in ("enter", "leave") + or re.search(r"\b(?:esp|sp)\b", previous[3]) + ): return None continue if previous[0] + previous[1] != cleanup: diff --git a/tests/testDelphiFinallyContinuation.py b/tests/testDelphiFinallyContinuation.py index 57ba7c8d..7dbcf54c 100644 --- a/tests/testDelphiFinallyContinuation.py +++ b/tests/testDelphiFinallyContinuation.py @@ -94,3 +94,36 @@ def test_real_inno_cleanup_dispatch(): d, (fixture["jump"], 2, "jmp", "eax"), SimpleNamespace(current_block=block) ) assert target == fixture["continuation"] + + +@pytest.mark.parametrize("mutation", ["9c", "669c", "9d", "669d", "60", "6660", "61", "6661", "6683c402"]) +def test_rejects_implicit_and_partial_stack_pointer_changes(mutation): + import struct + + body = bytes.fromhex(mutation) + dispatch = BASE + 5 + len(body) + continuation = dispatch + 10 + code = ( + b"\x68" + + struct.pack(" Date: Thu, 29 Oct 2026 19:03:30 +0000 Subject: [PATCH 4/5] fix(labels): require two validated VMTs before ReSym marks an image Delphi --- src/smda/common/LanguageAnalyzer.py | 8 +++++- .../labelprovider/DelphiReSymProvider.py | 7 ++++++ tests/testDelphiReSymProvider.py | 25 +++++++++++++++++-- 3 files changed, 37 insertions(+), 3 deletions(-) diff --git a/src/smda/common/LanguageAnalyzer.py b/src/smda/common/LanguageAnalyzer.py index 49304e35..f2fcce1c 100644 --- a/src/smda/common/LanguageAnalyzer.py +++ b/src/smda/common/LanguageAnalyzer.py @@ -18,6 +18,7 @@ _PRINTABLE_STRINGS_RE = re.compile(b"[ -~]{6,128}") _BORLAND_LOCALES = b"borland\\locales" +_MIN_RESYM_VMTS = 2 class LanguageAnalyzer: @@ -163,7 +164,12 @@ def _computeDelphiScore(self): delphi_score = max(delphi_score, 0.9) elif delphi_objects: delphi_score = max(delphi_score, 0.6) - if delphi_score < 0.6 and self.getDelphiReSymObjects(): + # One self-referencing VMT can occur by chance; the RTL alone puts dozens in a real image. + if ( + delphi_score < 0.6 + and self.getDelphiReSymObjects() + and self.delphi_resym_resolver.getVmtCount() >= _MIN_RESYM_VMTS + ): delphi_score = 0.6 return delphi_score diff --git a/src/smda/common/labelprovider/DelphiReSymProvider.py b/src/smda/common/labelprovider/DelphiReSymProvider.py index e31301f1..07f30a66 100644 --- a/src/smda/common/labelprovider/DelphiReSymProvider.py +++ b/src/smda/common/labelprovider/DelphiReSymProvider.py @@ -112,6 +112,7 @@ class DelphiReSymProvider(AbstractLabelProvider): def __init__(self, config): self._config = config self._func_symbols: dict[int, str] = {} + self._vmt_count = 0 self._binary: bytes = b"" self._base_addr: int = 0 self._settings = ArchitectureSettings(4, VMT_JUMP_DIST_32BIT) @@ -121,6 +122,7 @@ def __init__(self, config): def update(self, binary_info): self._func_symbols = {} + self._vmt_count = 0 self._binary = binary_info.binary self._base_addr = binary_info.base_addr self._settings = ArchitectureSettings(4, VMT_JUMP_DIST_32BIT) @@ -361,6 +363,7 @@ def _parse_delphi_symbols(self): for vmt in self._find_vmts(): if not self._take_entry(): break + self._vmt_count += 1 mdt_addr = self._read_ptr(vmt + self._settings.mdt_offset) if not mdt_addr: continue @@ -394,6 +397,10 @@ def getSymbol(self, address): def getFunctionSymbols(self): return self._func_symbols + def getVmtCount(self): + """Number of validated VMTs in the last update, with or without method tables.""" + return self._vmt_count + def getRelocations(self): return {} diff --git a/tests/testDelphiReSymProvider.py b/tests/testDelphiReSymProvider.py index 2ac487f2..73460f62 100644 --- a/tests/testDelphiReSymProvider.py +++ b/tests/testDelphiReSymProvider.py @@ -88,6 +88,17 @@ def mutate(info, layout): return info +def add_bare_vmt(info, layout, vmt=0xA00): + """Add a second validated VMT with no RTTI or method table, sharing the first one's name and code.""" + size = info.bitness // 8 + jump = 88 if size == 4 else 200 + layout.ptr(vmt, info.base_addr + vmt + jump) + layout.ptr(vmt + 8 * size, info.base_addr + 0x380) + for field in range(11, 22): + layout.ptr(vmt + field * size, 0 if field == 14 else info.base_addr + 0x1100) + return mutate(info, layout) + + @pytest.mark.parametrize("bitness", [32, 64]) def test_real_tobject_metadata(bitness): info, fixture = load_fixture(bitness) @@ -488,11 +499,21 @@ def test_named_tkunknown_type_is_preserved(): assert provider.getSymbol(info.base_addr + 0x1100).endswith(": TOther") -def test_structural_resym_metadata_is_delphi_language_evidence(): - info, _ = image() +@pytest.mark.parametrize("bitness", [32, 64]) +def test_structural_resym_metadata_is_delphi_language_evidence(bitness): + info = add_bare_vmt(*image(bitness)) analyzer = LanguageAnalyzer(SimpleNamespace(binary_info=info)) assert analyzer.getDelphiObjects() == {} assert analyzer.checkDelphi() + assert analyzer.delphi_resym_resolver.getVmtCount() == 2 + + +def test_single_resym_vmt_is_not_delphi_language_evidence(): + info, _ = image() + analyzer = LanguageAnalyzer(SimpleNamespace(binary_info=info)) + assert analyzer.getDelphiReSymObjects() + assert analyzer.delphi_resym_resolver.getVmtCount() == 1 + assert not analyzer.checkDelphi() def test_pdb_supplement_does_not_erase_delphi_symbols(): From 80cd527df9daac949b716e2e8d6b9213fbbb5100 Mon Sep 17 00:00:00 2001 From: Rony <49360849+r0ny123@users.noreply.github.com> Date: Thu, 29 Oct 2026 19:03:30 +0000 Subject: [PATCH 5/5] docs: changelog entries and RTL attribution for the Delphi audit --- CHANGELOG.md | 28 ++++++++++++++++++++++------ NOTICE | 11 +++++++++++ tests/delphi_corpus/README.md | 2 ++ 3 files changed, 35 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ba7ce455..0d520228 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,8 @@ past roughly six lines it belongs in the PR the entry links. ## [Unreleased] ### Added +- **(labels)** `MAX_DELPHI_METADATA_ENTRIES` (default 1,000,000) caps the metadata entries the ReSym provider + reads per image, so a corrupt or hostile image cannot drive VMT and method-table parsing without bound. - **(binja)** Export 32-bit ARM views (`armv7` and `thumb2`) as `arm` reports. Each function is decoded in its own instruction set and carries `thumb` metadata like the file-based backend. A T32 function's blocks in A32 (PLT veneers) are left out. Where an A32 and a T32 function claim the same bytes, only the one with more evidence @@ -80,6 +82,12 @@ past roughly six lines it belongs in the PR the entry links. set. Cost: four `udf #0xf9` in the ARMNT DLL that Capstone cannot decode. Big-endian ARM stays unsupported. ### Changed +- **(labels)** Delphi detection is looser. `_computeDelphiScore` now reaches the 0.6 that `checkDelphi()` needs + when ReSym validates at least two self-referencing VMTs (one can occur by chance; a real image has one per class, + dozens from the RTL alone), which turns on the Delphi candidates and finally recovery. Pythia no longer requires + `_validate_metadata_addresses`, and Pascal strings decode as cp1252, so 0x80-0x9F count as printable. Cost: a + non-Delphi image is somewhat more likely to be classed as Delphi. *Not measured on non-Delphi samples:* none of + the bundled non-Delphi fixtures contains `TObject`, so they never reach these checks. - **(aarch64)** Jump the gap scan over non-executable sections instead of stepping through them a word at a time. `nextGapCandidate` now bisects into the sorted, merged executable section ranges and resumes at the next one, as the ARM backend already does. *Measured on `04652e6`:* gap scan time on a 50 MB Go ELF (gh 2.62.0 linux/arm64) @@ -96,15 +104,23 @@ past roughly six lines it belongs in the PR the entry links. A vertex that only appears as a successor raised `KeyError`. Nothing in SMDA calls `closure()`, so no report output changes. (#414) - **(labels)** Repair Delphi class RTTI offsets, published/extended method-table traversal, record bounds, - Unicode names, executable-target validation and Pythia table bounds; preserve Delphi labels when - supplementing a PDB, and prefer explicit PDB names over generic PE entry-point labels. Cache shared - legacy tables, share provider results and use a dependency-free pointer-suffix scan. *Measured on the supplied - `1d4e5452…5681c` sample against `04652e6`:* 292 usable labels become 7,969; five-run median provider - time falls from 1.100s to 0.084s, with lower peak Python allocation. Real metadata excerpts and - malformed-input tests are bundled; the full binary is private. See `profiling/delphi_audit/`. + Unicode names, executable-target validation and Pythia table bounds, and prefer explicit PDB names over generic + PE entry-point labels. Cache shared legacy tables, share provider results and use a dependency-free + pointer-suffix scan. *Measured on the supplied `1d4e5452…5681c` sample against `04652e6`:* 292 usable labels + become 7,969; five-run median provider time falls from 1.100s to 0.084s, with lower peak Python allocation. + Real metadata excerpts and malformed-input tests are bundled; the full binary is private. See + `profiling/delphi_audit/`. +- **(labels)** PE export and entry-point labels (and their candidates) are no longer dropped when a PDB is + supplied. The PDB pass re-ran every label provider on the PDB's empty `BinaryInfo`, which cleared + `PeSymbolProvider`; it now updates only `PdbSymbolProvider`. With `USE_SYMBOLS_AS_CANDIDATES` on by default, + exports and the OEP are candidates again on PE+PDB runs, so the function set can grow where the PDB does not + already name an export. *Measured against `bfe39ec`:* `msvc_cxx_pdb_pe` with its PDB keeps 3 PE symbols instead + of 0; the function sets of `msvc_cxx_pdb_pe` and `rust_pe_msvc_i686` with their PDBs are unchanged. - **(intel)** Recover normal-flow continuations from the validated 32-bit Delphi finally-dispatch idiom. Constructor tails previously ended at `pop eax; jmp eax`; match the saved continuation and linked handler thunk before following the missing edge. Real and generated regression fixtures are bundled. + *Measured against `bfe39ec`:* complete reports bit-identical on 18 bundled non-Delphi x86/x64 PE, ELF and + PE+PDB fixtures. *Reproduced by reviewer on 16.* ### Security diff --git a/NOTICE b/NOTICE index 3218a15a..d9c9afe0 100644 --- a/NOTICE +++ b/NOTICE @@ -63,3 +63,14 @@ Lengauer-Tarjan dominator tree -------------------------------------------------------------------------------- src/smda/common/DominatorTree.py is based on the implementation by Armin Rigo (https://bitbucket.org/arigo/arigo/src/default/hack/pypy-hack/heapstats/dominator.py). + +================================================================================ +Delphi test fixtures +-------------------------------------------------------------------------------- +tests/delphi_corpus/ holds sparse, modified metadata and code excerpts from the +official Inno Setup 7.1.0 installers, used only to test the Delphi label +providers. Inno Setup is Copyright (C) 1997-2026 Jordan Russell, portions +Copyright (C) 2000-2026 Martijn Laan; its license is in +tests/delphi_corpus/INNO-LICENSE.txt. The TObject VMT, the System unit RTTI and +the _HandleFinally sequence in these excerpts are Delphi run-time library code +compiled into Inno Setup, Copyright (C) Embarcadero Technologies, Inc. diff --git a/tests/delphi_corpus/README.md b/tests/delphi_corpus/README.md index e70eaa7e..1acd8019 100644 --- a/tests/delphi_corpus/README.md +++ b/tests/delphi_corpus/README.md @@ -13,6 +13,8 @@ https://github.com/jrsoftware/issrc/blob/is-7_1_0/README.md Original software: Copyright (C) 1997–2026 Jordan Russell; portions Copyright (C) 2000–2026 Martijn Laan. https://jrsoftware.org/ . See [INNO-LICENSE.txt](INNO-LICENSE.txt). The SMDA project did not write the original software. The excerpts are modified for parser testing. +The TObject VMT, the System unit RTTI and the `_HandleFinally` sequence are Delphi run-time library code +compiled into Inno Setup rather than Inno Setup's own: Copyright (C) Embarcadero Technologies, Inc. `testDelphiReSymProvider.py` reconstructs these images and verifies the entire recovered symbol map. Additional generated layouts cover published entries preceding extended RTTI, metadata outside code,