Repository navigation
+ PartitionedDisk write support, 4Kn GPT, BSD/Amiga/Sun/SGI labels, resolver descent - #471
Merged
Merged
Conversation
Hawkynt
force-pushed
the
feat/partitioned-disk-write
branch
from
October 23, 2026 12:08
14cb422 to
bd8954a
Compare
Symptom: after any merge, every open pull request went DIRTY on screenshots/main-window.png (and docs/screenshots/*). The Generate and Branch screenshots workflows re-captured the window on every push to every branch and committed the result back, so branches that only touch a codec or a format each carried their own slightly different PNG, and the first merge turned all the others into binary conflicts. Each resolution re-ran the capture, restarted CI and needed a new approval. Both workflows now run only for pushes that touch the shell (Compression.NativeUI, Compression.Shell), the NativeForms pin, or the capture scripts and workflows themselves. workflow_dispatch still refreshes any branch on demand.
ZIP entries compressed with method 95 threw NotSupportedException, so every archive 7-Zip writes with -mm=xz (and WinZip .zipx XZ entries) was unreadable. APPNOTE 4.4.5 defines the payload as one complete .xz stream, so the entry is decoded and encoded with the existing XzStream (LZMA2, CRC-64 check); the writer falls back to Store when XZ would not shrink the data, like every other method. "xz" is exposed in the method lists (descriptor schema, CLI method spec, UI). Oracle: 7-Zip 26.03 tests and extracts our archives byte for byte and reports "Method = xz"; we decode archives 7-Zip writes with -mm=xz. A 7-Zip-written reference archive is committed under Compression.Tests/Zip/ReferenceVectors so the read direction is pinned without the tool. Sourcing: rung 3 (APPNOTE.TXT); the codec is our own XZ implementation.
# XAR: bzip2, lzma and xz members decoded and checked against the TOC Symptom: lzma and xz members came back still compressed, bzip2 members threw, and any other encoding style was returned raw. Root cause: Extract only knew zlib and "none"; every lzma/xz style was folded into one name and fell through to the raw-bytes default. Change: encoding styles map one-to-one (octet-stream, x-gzip, x-bzip2, x-lzma, x-xz). bzip2 and xz go through our own stream decoders; x-lzma is sniffed, because xar's own encoder writes .xz there while libarchive writes the legacy .lzma format. Unknown styles are refused by name. Every member is checked against its declared size and its SHA-1, MD5 or SHA-256 extracted-checksum. Evidence: none/gzip/bzip2/lzma/xz archives written by bsdtar 3.7.2 (libarchive 3.7.2), committed with provenance; every member passes its SHA-1, all five decode to identical bytes, inputs match the generator's SHA-256s. XarOracleTests re-runs bsdtar live. Sourcing: rung 3 (xar format documentation) for the encoding styles; codecs are our own. * regenerate package references
…refused # LHA: -lh2- and -lh3- decoded with their own decoders, writing them refused Symptom: -lh2- and -lh3- members were decoded with the -lh5- decoder, and the writer emitted -lh5- data labelled -lh2-/-lh3-, which no real LHA reads. Root cause: both are distinct LHarc 2.x methods. -lh2- is LZSS over an 8 KiB window with two adaptive Huffman trees (the position tree grows a leaf every 64 output bytes); -lh3- sends static trees per block, with a "ready made" position table when none is sent. Neither shares -lh5-'s bit stream. Change: Lh2Decoder and Lh3Decoder (with the LHarc fillbuf/getbits bit reader) decode them; LhaWriter refuses both methods and NativeUI no longer offers them. Evidence: lh2.lzh and lh3.lzh from lhasa's test suite (ISC), written by UNLHA32.DLL 2.67a, committed with provenance. Both decode to the CRC-16 UNLHA32 recorded, to 2,853 bytes of MIT licence text, and to identical bytes as each other. Sourcing: rung 1, converted from XADMaster's XADLZHOldHandles.m (LGPL-2.1-or-later; descended from LHa for UNIX via libxad), notice in THIRD-PARTY-NOTICE.XADMaster.txt. XADMaster sizes the -lh3- position alphabet as 1 << 10; the ready-made table only satisfies Kraft's equality with 128 symbols (8 KiB / 64), which is what the real files need. * regenerate package references
…ir archivers wrote them # LHA: level-1 extended headers, level-3 headers and directory names read Symptom: on lhasa's test suite, members behind level-1 headers with extended headers came out a few bytes off (CRC mismatch or end of stream), level-3 archives were refused, directory names were dropped, and the modifier and layout map walked level-2 chains with the level-0 size byte. Root cause: a level-1 member's compressed size includes its extended headers, which the reader did not subtract; level 3 (32-bit sizes) was not implemented; extended header 0x02 (directory, 0xFF-separated) was ignored; each of the three walkers had its own partial parser. Change: one LhaHeaderParser for levels 0-3 (header checksum, the level-1 size correction, the OS-9/68k level-2 quirk, 0x01 file and 0x02 directory headers) used by the reader, the modifier and the layout map. A truncated last member is listed and fails on extraction instead of failing the whole archive. Evidence: archives from lhasa's test suite (LHA for Amiga 1.22, LHA OS/2 2.08, ...) committed with provenance; every member passes the CRC-16 its archiver recorded. Sourcing: rung 1, converted from lhasa's lha_file_header.c and ext_header.c (ISC), notice in THIRD-PARTY-NOTICE.lhasa.txt. # LHA: -lh1-, -lzs-, -lz5-, -pm1-, -pm2- decoded as their archivers wrote them Symptom: on lhasa's test suite every -lh1-, -lzs-, -lz5-, -pm1- and -pm2- member failed (CRC mismatch, end of stream, divide by zero); our writer's output for these methods was not readable by lhasa; -lhx- was refused. Root cause: the decoders implemented other schemes: -lh1- rebuilt a static tree every 4 KiB instead of LZHUF's adaptive tree; -lzs- was read byte-aligned with relative offsets instead of an MSB-first bit stream of absolute ring positions; -lz5- lacked LArc's predefined initial ring and start position; -pm1-/-pm2- were a PPMd model with no relation to PMarc. The encoders mirrored the same mistakes. Change: decoders converted from lhasa for -lh1- (shared adaptive tree), -lzs-, -lz5-, -pm1- and -pm2- (move-to-front history, PMarc's static and rebuilt trees); new -lh1-, -lzs- and -lz5- encoders built on the same tree and ring rules; writing -pm1-/-pm2- is refused (no encoder) and NativeUI stops offering them; -lhx- reads through the lh5 decoder with a 1 MiB window. Evidence: LHarc 1.13, LArc 3.33, PMarc 1.24/2 and UNLHA32 archives from lhasa's suite pass their CRC-16s, and five methods decode the same GPL-2 text identically. LhaOracleTests: every method x header level (0-2) our writer produces extracts byte-identically in lhasa (lha t / lha x). Remaining: LHARK's lk7 variant of -lh7-. Sourcing: rung 1, decoders converted from lhasa (ISC), tables copied exactly, notice in THIRD-PARTY-NOTICE.lhasa.txt; encoders are our own. * regenerate package references
# PackIt: real on-disk layout, Huffman entries decoded, encrypted ones refused
Symptom: real PackIt archives did not read: "PMa4" Huffman members came
back as compressed bytes, and stored ones were cut at the wrong offsets.
Archives we wrote were not PackIt files at all.
Root cause: reader, writer and modifier shared an invented 87-byte
header. The real header is 94 bytes after the magic (name, type,
creator, Finder flags, fork sizes, dates, CRC-16), the forks are followed
by a CRC-16, the archive ends with "PEnd", and a "PMa4" entry Huffman-
codes the header, both forks and the CRC as one stream with no stored
length.
Change: reader, writer and modifier rewritten to the real layout. PMa4
entries are decoded (StuffIt method-3 style pre-order tree, MSB-first)
and walked to find the next entry; both CRC-16s are checked; PMa5/PMa6
(XOR/DES-encrypted) entries are listed and refused. The writer emits
stored entries with both CRCs and the end marker; defrag carries each
entry over byte for byte so resource forks and Huffman coding survive.
Evidence: nine real PackIt archives (dexvert sample set; seven Huffman-
coded) decode with every header and fork CRC matching and every data
fork identical to unar's; they are not committed (unknown redistribution
terms) and run from test-corpus/packit when present. unar extracts
archives we write, and decodes a test-encoded PMa4 entry to the same
bytes we do. The header CRC (unchecked by XADMaster) is CRC-16/XMODEM
over the 92 bytes before it, measured on every real header.
Sourcing: rung 1, layout and Huffman scheme converted from XADMaster's
XADPackItParser.m and XADStuffItHuffmanHandle.m (LGPL-2.1-or-later),
notice in THIRD-PARTY-NOTICE.XADMaster.txt.
# DiskDoubler: real header, methods 0/1/6/9 decoded, the rest refused
Symptom: compressed DiskDoubler forks were returned as their compressed
bytes, without an error, and real DiskDoubler files did not parse.
Root cause: the reader and writer used an invented 82-byte header with a
name field. A real file opens with the magic 0xABCD0054 and an 84-byte
header (sizes, per-fork method, version bytes, dates, type/creator,
per-fork checksums and delta types, CRC-16), often behind a MacBinary
wrapper; the extract path had no decoder for any method but stored.
Change: reader and writer rewritten to the real layout, with header CRC
check and MacBinary detection; magic-based detection added. Decoded:
0 (stored), 1 (Unix compress with its 0x5A-masked header and byte-sum
check), 6 and 9 ("AD" LZSS blocks with per-block header XOR). Methods
2-5, 7, 8, 10 and delta pre-processing throw NotSupportedException.
The writer emits a stored single file.
Evidence: on the dexvert DiskDoubler sample set (not committed: unknown
redistribution terms; runs from test-corpus/diskdoubler when present)
every method 6/9 fork is identical to unar's output; the one method-1
sample decodes 2 bytes short of its size in unar and in gzip alike and
we refuse it the same way; the Compact Pro (8) samples are refused.
unar extracts files we write.
Sourcing: rung 1, layout and the AD decoder converted from XADMaster's
XADDiskDoublerParser.m and XADDiskDoublerADnHandle.m (LGPL-2.1-or-later),
notice in THIRD-PARTY-NOTICE.XADMaster.txt.
* regenerate package references
* FreeArc: real FreeArc 0.67 archives, read and write
Symptom: FreeArcReader/FreeArcWriter used a private layout of our own
("ArC\1" + flags + typed blocks). Not one real FreeArc archive could be
read, and FreeArc rejected everything we wrote, but the descriptor still
claimed create and modify.
Root cause: the format was never taken from FreeArc. A real archive is
read from its end. The footer descriptor sits in the last 4 KiB and
lists the control blocks. Each directory block lists its solid blocks
with offsets relative to itself and the per-file columns. Every
descriptor and control block is CRC-32 checked. Integers use FreeArc's
own variable-length form, and the directory and footer are normally
lzma-compressed.
Change: the reader and writer are rewritten from docs/FREEARC-ON-DISK.md
and the private layout is removed with no backward compatibility.
- Reading: the full container (SFX prefix, comment, directories, UTF-8
names, several solid blocks). storing and lzma (raw LZMA1, dictionary
and lc/lp/pb from the method string) decode. tornado, ppmd, lzp, grzip,
rep, delta, exe, 4x4 and encryption are listed but refused with
NotSupportedException. Archives whose directory cannot be read
(encrypted headers) list as FULL.arc + metadata.ini
(parse_status=partial).
- Writing: FreeArc's layout with a storing or lzma solid block. For
storing input in FreeArc's order it is byte-identical to
"arc a -m0 -dmstoring".
Oracle: FreeArc 0.67 (2014-03-15) Arc.exe and unarc.exe, run as black
boxes. "unarc t/x" and "arc t/lb/x" extract our storing and lzma
archives byte-identically, and we extract arc's -m0/-mlzma/-mlzma:lc4
archives byte-identically (FreeArcOracleTests). Nine archives Arc.exe
wrote are checked in as fixtures, with provenance. Create/modify stay
because FreeArc itself accepts our output.
Sourcing: rung 2. FreeArc's sources are GPL-2.0 and the 0.67 binaries
freeware, so neither can be absorbed. The sources (svn2github/freearc:
Unarc/ArcStructure.h, ArhiveStructure.hs, ArhiveDirectory.hs,
ByteStream.hs) were read to write docs/FREEARC-ON-DISK.md, every field
was confirmed against hex dumps of Arc.exe's output, and the code was
written from that document. Rung 1 does not apply: there is no
permissively licensed FreeArc implementation.
* FreeArc: regenerate the Archives API reference
The reader and writer used an invented layout: compression taken from the first-header flags, the block compression bit inverted, no script header. Real installers never parsed; the "writer" produced files only our reader accepted. Both are replaced. The reader follows NSIS's fileform.h: first header at a 512-byte boundary, compressor recognised from the data (zlib, bzip2, LZMA, stored), solid streams split on the fly, file names recovered by replaying SetOutPath/File from the script header, NSIS 2 and 3 string escapes, Unicode and ANSI, 32- and 64-bit headers, file times. Deflate gains the NSIS 2 stored-block rule (LEN without NLEN); bzip2 gains NSIS's header-less framing. Detection scans aligned first headers instead of a tail search. Creation, defrag and the fake writer are removed: nothing but makensis can produce a runnable installer. Oracle: 11 installers compiled by makensis 2.51 and 3.10 (zlib/bzip2/LZMA, solid and not, Unicode and ANSI); names, sizes and SHA-256 of every file match what 7-Zip 26.03 extracts. Sourcing: rung 1. NSIS sources (zlib/libpng) for layout and constants; 7-Zip's NSIS handler (LGPL-2.1+) for the signature table and dialect checks.
…esolver descent
+ partitioned disks: 4Kn GPT, GPT backup recovery, BSD/Amiga RDB/Sun/SGI labels
Symptom: a GPT disk with 4096-byte logical blocks was not recognised (the
header was only looked for at byte 512), a damaged primary GPT lost the
whole table, and bare BSD disklabels, Amiga RDB, Sun and SGI disks were not
recognised at all. PartitionEditor's GPT LastUsableLBA overlapped the first
backup entry sector, partitions could be added inside the GPT's own
structures, and every rewrite replaced unique partition GUIDs, attribute
bits and the disk GUID.
Root cause: GptParser, MbrParser and PartitionEditor hard-coded 512-byte
LBAs; the detector knew GPT, MBR and APM only; the editor validated
partitions against the disk length, not the usable LBA range.
Fix: the block size is the one at which LBA 1 holds a CRC-valid header
(512 or 4096); a primary header or entry array failing its CRC falls back
to the backup at AlternateLBA / the last LBA. MBR/EBR parsing takes the
block size; the EBR walk stops at loops, at links leaving the extended
partition and at EBRs without 0x55AA; a lone protective 0xEE entry is no
partition. New readers for the stand-alone BSD disklabel, Amiga RDB
(checksummed RDSK/PART chain), Sun label (magic + XOR checksum) and SGI
volume header (magic + sum checksum), probed after GPT/MBR/APM. The editor
counts in the disk's block size, refuses ranges outside
[FirstUsableLBA, LastUsableLBA] (also before ConvertMbrToGpt writes), puts
LastUsableLBA before the backup array, keeps existing entries' unique
GUIDs and attributes and the disk GUID, and can write an empty GPT/MBR.
Sourcing: rung 3 — UEFI 2.10 §5.2/§5.3 for MBR/GPT; 4.4BSD disklabel.h,
AmigaOS NDK hardblocks.h/filehandler.h, Sun dklabel.h/vtoc.h and IRIX dvh.h
for the label layouts and checksum rules. util-linux sfdisk/fdisk and GNU
parted (GPL) were used only as black-box oracles: fdisk -b 4096 and
sfdisk-written tables are read at the offsets they report, and parted
mklabel amiga/sun/dvh/bsd labels at parted's offsets. parted's BSD labels
do not satisfy the dkcksum rule (measured), so a stand-alone BSD label is
accepted on its two magics and sane geometry instead.
+ PartitionedDisk: edit partition filesystems in place, add/remove partitions, create disks
Symptom: a raw partitioned disk was read-only; files inside a partition
could not be edited, partitions not added or removed, and no disk could be
created from partition images. The partition-aware TryAdd/TryRemove the
virtual-disk descriptors use wrote straight into the partition (a refusal
half-way left a half-edited volume), accepted the generic re-create
default as an "editor", deleted the wrong entry on MBR disks with an
extended partition (detector and editor index differently), and could
place a new partition over the backup GPT.
Root cause: no write path on the descriptor; inner edits had no staging;
whole-partition deletes used list indices instead of the entry itself.
Fix: StagedWindowStream holds a partition edit's writes in 4 KiB pages,
refuses writes or SetLength past the partition, and commits only after
every edit in the call succeeded. TryAdd/TryRemove route
PartitionN_Type/path through the partition filesystem's own Add/Remove
(refused when it only has the interface's re-create default), resolve BSD
slices, plan new partitions (1 MiB-aligned, inside the usable range,
table capacity checked) before writing anything, and delete entries by
their start and size. PartitionedDiskFormatDescriptor implements
IArchiveModifiable, IArchiveCreatable (partition images on a fresh GPT or
MBR, scheme/sectorSize options, loose files refused) and
IArchiveWriteConstraints; purge is not offered. Every refusal leaves the
image byte-identical. The shell evaluates drops against the folder being
browsed, so a file dropped into a partition folder is accepted.
Sourcing: rung 3 — UEFI 2.10 §5.2/§5.3 for the table layouts; the inner
edits are the existing filesystem editors. Verified against GPL tools as
oracles only: sgdisk -v ("No problems found"), sfdisk --verify, parted and
fdisk -b 4096 read the created disks at our offsets; fsck.vfat -n and
e2fsck -fn are clean after in-place adds/removes on an sfdisk-built GPT
disk, mcopy and debugfs read the edits back, and no byte outside the
partitions changes.
# NestedStreamResolver descends into raw partitioned disks
Symptom: ResolveDeepest on a raw MBR/GPT disk image (not wrapped in a
VHD/VMDK/...) found no filesystem, and filesystems whose superblock lies
past 4 KiB were not recognised at any level.
Root cause: partition tables were only tried below a virtual-disk
container, and format detection matched magic bytes in a fixed 4 KiB
header.
Fix: detection goes through FormatDetector.DetectByContent (the disk-image
probe plus signatures as deep as the deepest registered one), after the
fixed-VHD footer check; a PartitionedDisk result descends into its
partitions, BSD slices included.
Sourcing: no format work; reuses the content probe and partition readers.
Hawkynt
force-pushed
the
feat/partitioned-disk-write
branch
from
October 26, 2026 12:06
28746e8 to
fc8ce06
Compare
The required tier failed because the existing XADMaster conversions for LHA lh2/lh3 and the ARJ prerequisite were absent from the provenance ledger. Record the pinned XADMaster revision, LGPL-2.1-or-later grant and existing notice, and retain the current generated screenshot. Verified: Release solution build; complete Linux and Windows core runs had only the ledger failure; all 72 LHA and provenance tests now pass. Existing implementation sourcing is rung 1.
The legacy-LHA/header PR failed only the provenance check: its ISC-licensed lhasa conversions were missing from the ledger. Record the pinned upstream, all converted decoder/header files and both existing notices, and merge the validated prerequisites. Verified: Release solution build; complete Linux and Windows core runs had only the ledger failure; all 113 LHA and provenance tests pass. Existing implementation sourcing is rung 1.
Required CI failed only because the existing XADMaster conversion was absent from the provenance ledger. Record the pinned source, LGPL-2.1-or-later grant and existing notice, and merge the validated prerequisite updates. Verified: Release solution build; complete Linux and Windows core runs had only that ledger failure; 83 PackIt, DiskDoubler and provenance tests passed, 2 corpus-dependent cases skipped. Existing implementation sourcing is rung 1.
The FreeArc branch failed the required tier only because its prerequisites lacked provenance ledger entries. Merge the validated corrections and retain the current generated screenshots. Verified: Release solution build; complete Linux and Windows core runs had only the provenance failure; 67 FreeArc and provenance tests passed, 9 external-tool cases skipped.
The NSIS branch failed its required tier only because converted-code provenance was missing from its prerequisites. Merge the validated corrections and retain the current generated screenshot. Verified: Release solution build; complete Linux and Windows core runs had only the provenance failure; all 53 NSIS and provenance tests pass.
Required CI failed only because prerequisite conversions lacked provenance ledger entries. Merge the validated corrections and retain the current generated screenshots. Verified: Release solution build; complete Linux and Windows core runs had only the provenance failure; 252 partition and provenance tests passed, 10 tool-gated or format-limited cases skipped.
Preserve the accumulated history while restoring current main and applying only partition parsing, editing, nested filesystem access, UI integration, tests and documentation. Recount gap summary classes from the combined catalogue instead of retaining stale branch totals. Verified: Release solution build, 271 focused tests passed with 10 tool-dependent skips, documentation generation completed without errors.
Preserve partition editing while incorporating merged Btrfs, ISO, Android, NSIS and G722 changes. Recount catalogue gap totals from the combined CSV; the shared detector and generated API references merge without source conflicts. Verified: Release solution build, 278 focused filesystem/UI/architecture/documentation checks passed with 11 tool-dependent skips; all three generated API documentation checks matched. The prior exact branch head passed full required Linux and Windows CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Raw partitioned disks (
PartitionedDisk) become writable, read more schemes, andNestedStreamResolverreaches into them.What changed
PartitionN_Type/path) is added or removed by that partition's own filesystem editor, working in place on a staged window (StagedWindowStream). Nothing reaches the disk unless every edit in the call succeeds. Edits are refused when the filesystem's only editor is the generic re-create default, when the volume is full, or when the edit would grow past the partition.*.rawimages become new GPT/MBR partitions, 1 MiB-aligned, inside the usable LBA range.PartitionN_Typedeletes a GPT/MBR entry, logical partitions included.Createlays partition images out on a fresh GPT or MBR (scheme,sectorSize=512|4096).Verification
sgdisk -v("No problems found") andsfdisk --verify, andpartedandfdisk -b 4096report the same offsets we do.fsck.vfat -nande2fsck -fnare clean,mcopy/debugfsread the edits back, and no byte outside the partitions changes.fdisk -b 4096, sfdisk (extended + logicals) and parted (bsd,amiga,sun,dvh) are read at the tools' offsets.