Parallax Propeller2 USB driver - the MashUp

Filesystem regression report — 2026-08-27
Login

Filesystem regression report — 2026-08-27

Companion to characterization-report.md, which covers throughput and protocol limits. This one covers correctness coverage: what has been run against real media, what passed, what failed, and — the part that matters most — what is still untested.

Fossil is the record. Every number here is traceable to a ticket comment and a log under harness/logs/ or tests/regression/logs/. Where this document and a ticket disagree, the ticket is right.

1. The FAT32 geometry matrix — now complete for 16 of 17 suites

Three volumes, deliberately formatted to different geometries, each pinned by serial (never by first-found — auto-binding on this rig picks the SSD).

drive cluster size data-area alignment 16 non-defrag suites RT_defrag_tests
A 32 KB 0 332 pass, 0 fail, 0 recoveries 14 pass
B 4 KB 0 332 pass, 0 fail, 0 recoveries wedges the drive
C 4 KB 5 332 pass, 0 fail, 0 recoveries wedges the drive

All three geometries produce the identical 332, with zero failures and zero recovery events on each. Drive A's full 17-suite run is the standing 346/346 gate, re-verified three times on 2026-08-26.

Drive C had never had a single suite run against it before this date. Its non-zero cluster alignment is the only geometry on the bench that reaches commit a359d54149's entryOffsetInCluster sign fix, so that fix is now exercised on live media and passing rather than argued from code reading.

What the matrix establishes by elimination

Alignment is exonerated as a factor in the defrag wedge. Drives B (align 0) and C (align 5) behave identically in both directions: both pass all sixteen suites cleanly, and both wedge on RT_defrag_tests. The variable they share is the 4 KB cluster size, not the alignment — and drive A's 32 KB clusters pass the same suite 14/14.

This is recorded as a correlation between a suite, a cluster size, and a wedge. No mechanism is proposed. Ticket 74f78296de carries the detail.

A procedural finding worth as much as the data

run_gate.sh runs suites in alphabetical order, which puts RT_defrag_tests second. On drive C that wedged the drive and cost the remaining fifteen suites — which then reported pass=0 fail=1 apiece. Those were not test failures. Every one was the fixture reporting no FAT32 volume matched the selection among 3 device(s): the drive was gone from the roster and the suite never started. The gate's own total for that run, "16 passed, 21 failed", is misleading and must not be read as an alignment-5 result.

Re-running with an explicit sixteen-suite list, defrag excluded, produced the clean 332. On 4 KB media, run the explicit list and leave RT_defrag_tests out or last. run_gate.sh already accepts a suite list; this needs discipline, not code.

2. exFAT — from zero coverage to a working fixture and first tests

exFAT is the project's stated primary default and drove a 953 GB SSD for days with no regression coverage of any kind. That is no longer wholly true, but it is closer to "started" than "covered".

Landed

The facade ladder — 192 tests now pass on exFAT

Eleven suites have been ported from the FAT32 driver object to the fs_* facade, so one source runs on both filesystems:

suite FAT32 (drive A) exFAT (SSD) note
FS_seek_tests 38 / 38 38 / 38 past-EOF refuse enforced at the facade (6170)
FS_cogcwd_tests 5 / 5 5 / 5 facade per-cog exFAT cwd (6170)
FS_stress_tests 4 / 4 4 / 4 clean
FS_fatchain_tests 2 / 2 2 / 2 clean — see caveat
FS_delete_guard_tests 8 / 8 8 / 8 clean once its fixture artefact fell
FS_multihandle_tests 22 / 22 22 / 22 MAX_OPEN raised to 6 (6170)
FS_read_write_tests 49 / 49 49 / 49 was 40/49 before the open-path fix
FS_dirhandle_tests 25 / 25 25 / 25 −111/−114 error-code split landed (6170)
FS_subdir_ops_tests 18 / 18 18 / 18 exFAT rename implemented (6170)
FS_error_handling_tests 19 / 19 18 / 18 write-on-exit contract landed; MBR white-box test FAT32-only (totals 18)
FS_directory_tests 33 / 33 30 / 30 exFAT moveFile implemented; cluster-zero raw-patch group FAT32-only (totals 30)
total 223 / 223 219 / 219 FULL FACADE PARITY — every gap in the 27-item 08de1c7aaa ledger resolved (6170, 2026-08-28)

THE WEDGE IS ROOT-CAUSED AND FIXED — and it was ours, not the media's (ticket 1464ab6579). The USB host cog compared absolute getct() values against deadlines at 12 sites (transaction timeouts and the SOF scheduler's due-checks); every such comparison misfires when its window straddles the 32-bit CNT wrap — a 21.47 s hazard grid whose phase is fixed at chip reset. That grid produced the whole phenomenology: per-suite quasi-deterministic wedge points, "clustering in time", immunity of short suites, one gate pass clean while identical standalone runs wedged 3-for-3, and reseat/power-cycle immunity. What cracked it: four consecutive runs threw their first BOT event at mount + 10.66..10.70 s — 40 ms repeatability across a physical reseat, which no device property can do. Fixed with the wrap-safe subtract-then-sign-test form at all 12 sites; verified fail-first (the 4-for-4 reproducer runs clean through the hazard instant), TX canary green, and the full gate green on both filesystems with the fixed engine.

Every FAT32 column matches its RT_ original exactly. That is the control set: a port that changed behaviour would show a different number, one that dropped assertions a smaller one. All eleven match, so the exFAT column is a result rather than a port artefact.

Port discipline learned the hard way (see the corrected 6691a78673 finding below): suite fixtures are built with absolute paths, never chdir + bare names — fs_chdir is a documented no-op on exFAT, so a chdir-built fixture lands its files in root and the suite goes green-and-hollow, or reports phantom driver defects. Only tests whose subject is cwd semantics keep chdir. Subdirectory entry counts count non-dot entries, since FAT32 subdirs carry ./.. and exFAT stores none.

Caveat on the clean columns. A suite can be green on exFAT while asserting nothing about it. FS_cogcwd's Test #2 — "one cog's cd does not affect another cog's CWD" — passes on exFAT because there is no cwd at all, so no cog's cd affects anything. And FS_fatchain passing does not mean exFAT's allocation structures are tested: its tests work through the file API, while the bitmap, up-case table and NoFatChain files remain untested (ticket 1891ce6d6a).

Two parity divergences, both the same shape — RESOLVED (6170, 2026-08-28)

Both were silent success hiding a semantic difference — the most dangerous API failure mode, because the caller has no way to detect it. Both are gone: fs_seek past EOF is refused at the facade on both filesystems, and fs_chdir drives a real facade-level per-cog cwd on exFAT (joined onto every relative path), so both filesystems answer identically. The original records below stand as the finding's history.

call FAT32 exFAT (pre-6170)
fs_seek past EOF refuses with an error silently clamped to EOF, returned success
fs_chdir real directory change returned success having done nothing

An application seeking to a computed offset and writing gets an append at EOF on exFAT where FAT32 fails loudly. One that chdirs and creates by bare name gets the file in root on exFAT and in the cwd on FAT32. Both are on ticket 08de1c7aaa; neither is fixed, because each is a genuine design decision about what the facade should promise.

Neither could have been found by the driver-bound ladder — each RT_ suite only ever runs against its own driver.

The error codes now match — and the port found data loss

The two drivers' error codes did not merely differ, they collided: −93 is E_NOT_A_DIR_HANDLE on FAT32 and E_DISK_FULL on exFAT; E_DIR_NOT_EMPTY is −44 vs −96; E_NOT_MOUNTED −20 vs −80. An application testing a raw number got a different meaning per filesystem.

The facade has published an error table (README) in the −80..−96 range since exFAT landed — FAT32's codes simply passed through untranslated, so the facade was violating its own documented contract. usb_app now translates them, and adds codes at −110.. for concepts FAT32 can report that the exFAT-derived table had no word for. This changes what fs_* returns for FAT32 volumes — that is the point, but it is a behaviour change for anything matching raw FAT32 numbers through the facade. Code testing st < 0 is unaffected.

The translation is verified by FS_delete_guard_tests passing 8/8 on FAT32 while asserting facade codes (−96, −112) against a driver that natively returns −44 and −47.

Pointed at exFAT, that same unchanged suite scored 2/8 and found two defects (ticket 6691a78673, Critical):

The two real ones traced to one missing mechanism: exFAT kept handles (h_used, h_pos, h_size) but nothing mapping a path to them, which is what every one of these guards needs to ask its question.

Fixed. No new per-handle storage was required — a file is identified by where its directory entry set lives, and exFAT already recorded both halves per handle (h_pdir_f, h_dent) because the write path needs them to flush the entry set. A path_is_open(pdir_f, dsi, write_only) predicate now scans the handle table, and openFileWrite and deleteFile consult it. That is the same identity FAT32 keys on — (dir_sector, dir_offset) — so both drivers now answer the question the same way instead of one of them not answering at all.

Two of the three defects are closed. And the fix cleared nine failures in a different suite: FS_read_write on exFAT went 40/49 → 49/49. Those were the large-file and cluster-boundary tests, logged as an open question, with only 5 BOT lines in the run — not wedge fallout. They were state corruption left by an earlier test that deleted or re-opened a file while a handle was live. One missing guard, nine downstream failures, in a suite that looked like it had its own problem.

Resolved (2026-08-27): the populated-directory "defect" was the suite's own fixture. It built the directory with fs_chdir + bare names — and fs_chdir is a documented no-op returning success on exFAT, so the "child" was created in root and the directory under test was genuinely empty. deleteFile's success was correct. A discriminating probe (tests/regression/ZZ_dgprobe.spin2) that builds the same shape with absolute paths and enumerates the child inside the directory first gets −96 E_DIR_NOT_EMPTY on both filesystems — the guard fires. The suite now builds its fixture with absolute paths and carries that enumeration check; what stops it at 5/8 on exFAT today is a deterministic BOT wedge (ticket 1464ab6579), not a delete defect.

No new test was written for either. RT_delete_guard_tests has passed for as long as the estate has existed, but it binds the FAT32 driver, so it had only ever asked FAT32. Ported to the facade, it asked exFAT and both defects appeared on the first run.

exFAT is now gated — run_fs_gate.sh

run_gate.sh is the FAT32 driver gate: seventeen RT_ suites bound to usb_fat32_fs. It cannot reach exFAT, because those suites declare the FAT32 driver object directly. So exFAT had no gate at all — not a failing one, none — and a driver with no gate cannot be regression-tested.

run_fs_gate.sh runs the FS_ suites through the facade, so the same source runs on both filesystems, and it checks expected counts rather than zero failures. exFAT legitimately falls short on twenty-seven of these today and each is filed; a gate demanding zero would be red forever and get ignored, while a gate printing only a total would let a new failure hide behind a fixed one — the number stays 192 while the set changes underneath. Any deviation in either direction fails the run: a count going up means either a defect was fixed (update the table in the same commit) or a test stopped running. FS_delete_guard is the live example of both rules: its exFAT count read 5 first for the fixture artefact, then 5 again for the wedge — same number, entirely different failure set — and when a wedge-free pass finally scored the true 8, the gate flagged the improvement as a deviation too, which is exactly how the expected value got corrected instead of silently drifting.

The product build matrix

The gate opens by compiling the configurations an integrator would actually ship — a check, not an assumption:

build result size
USB_MSC USB_EXFAT builds 58,700 B
USB_MSC USB_FAT32 builds 71,869 B
USB_MSC USB_EXFAT USB_FAT32 builds 81,876 B

The three different footprints are the evidence the gating works — each build carries only what it asked for, which is Phase 0's requirement that a disabled class leave no remnant. The exFAT-only build is the one nothing used to verify: usb_app declares xfs : "usb_exfat_fs" inside #IFDEF USB_EXFAT, so every FAT32-only build skips the exFAT driver entirely and nothing proved the exFAT-only build still compiled. --build-only runs this with no hardware, so it belongs in a build pipeline.

The structural blocker, measured

All seventeen suites bind the FAT32 driver object directly — sd : "usb_fat32_fs", 258 sd. calls in one suite. They mount through the facade and then query a driver with no volume, so no build gate can retarget them.

Of 85 distinct symbols the suites use: 24 already exist in exFAT (the whole core file/directory API), 29 are constants — cheap to re-export except that the values differ, so re-exporting FAT32's numbers would make suites pass against wrong error codes — and 32 are methods exFAT lacks.

A test-only shim was designed for this and rejected: the common interface already exists as the usb_app fs_* facade, it is what applications use, and it already reconciles semantics (fs_chdir is a no-op on exFAT, which resolves full paths, and a real cwd on FAT32). Building a second one would duplicate that reconciliation somewhere it can drift, and would have the estate validating an interface no application uses.

The port target is therefore the facade, and every gap the port hits is a product gap — not test plumbing.

Application-API gaps the port exposed

Filled by this work: fs_tell, fs_eof, fs_free_space, fs_sec_per_cluster and fs_cluster_bytes. The last two are the clearest case of the pattern — both drivers had published the values through volumeGeometry() all along with an identical contract, but the facade offered only the raw three-LONG triple, so every caller wanting the commonest field had to know the layout. Three suites were blocked on exactly that.

Still absent: fs_rename, fs_move, fs_set_vol_label, per-handle sync (syncHandle), and the contiguous/defrag family.

A minority of suites are white-box by design and should stay driver-bound: RT_write_integrity, RT_file_ops and RT_volume use raw-sector and debugGet* access, and RT_defrag drives FAT chains. exFAT's structural equivalents belong in ticket 1891ce6d6a.

3. exFAT scan-storm limit — measured, not theorised

freeSpace() on exFAT population-counts the allocation bitmap, because the format stores only a rounded PercentInUse. On the bench's 953 GB volume that is ~1,900 sector reads and ~4.6 s per call.

Three full scans in one session wedged the SSD — twice out of two, the second ending in BOT recover GIVING UP … ST_BOT_WEDGED. This is ticket 3d99c25528's scan-storm hazard, previously seen only on the FAT32 thumbs, reproduced on exFAT and on a different vendor's controller.

Consequences, both recorded rather than worked around:

4. Defects found and fixed

6220 — untrusted FSInfo next-free hint escapes its bounds check (fixed)

allocateCluster() validated the media-supplied next-free hint with a signed compare and no upper bound. A hint in [$2000_0000, $7FFF_FFFF] passes, << 2 sets bit 31, and the signed >= fat_limit guard cannot catch a negative. Spin2's >> is logical, so the allocator pre-loads a sector far outside the FAT — measured at LBA 4,194,316 on a 160-sector synthetic volume. If that read errors, noise; if it lands on a real LBA, the loop scans arbitrary data as FAT entries.

Upstream (v1.8.0-dev) fixed this and carries a comment naming the trap; this fork was the pre-fix version, and it was not on the backport worklist.

Fix is two parts: upstream's unsigned guards, plus deriving the allocation front for any hint the allocator will not accept — widened from == $FFFF_FFFF, which caught "no hint" but not "garbage hint". Without that second half the bounds fix would have cost the one wear-policy axis where this fork beats upstream.

Proven fail-first with a no-hardware synthetic FAT32 volume (harness/top_6220_fsinfo.spin2): 4/6 before, 7/7 after. Gate re-run 346/346.

fs_eof return convention (fixed)

Both drivers return Spin2's TRUE (−1), and their error codes are negative too — so at driver level "at end of file" and "that handle does not exist" are both negative, and if eof(h) treats an error as EOF: stop reading, report success on a file never read. The facade now normalises to 1 / 0 / negative.

Two wrong versions preceded the right one, both documented in the method: the first told callers to test > 0 (wrong — −1 is not > 0), the second used (yn < 0) ? yn : …, which classifies TRUE as an error and reproduces the exact ambiguity. The discriminator cannot be the sign; it has to be the value.

Evidence integrity — the recorder was manufacturing evidence (fixed)

bulk_in assigns last_hs only on the success path, so the handshake byte printed on the failure exit is a leftover from an earlier packet. Ticket 293576b2fc spent its life describing "three devices stopping at the same byte count with a NAK ($5A)" — that $5A was stale, and a NAK exit was arithmetically impossible for the 11 ms recorded (budget 10000 with 1 ms per retry needs ~10 s).

usb_client now exposes last_hs_valid(), last_phy_status() and last_saw_sop(), captured at the failure exits; the ring gained FL_HS_STALE / FL_PHY_TIMEOUT / FL_PHY_ERR / FL_SAW_SOP. last_hs is deliberately not cleared — csw_patient() branches on it, so zeroing it would silently change a retry path while calling itself an evidence fix.

First use: 11 of 11 failures are phy_st=-2 sop=0 — PHY timeout with no start-of-packet at all. The device goes completely silent. That kills every candidate requiring the device to be transmitting.

5. Corrections — claims withdrawn

Recorded because a wrong claim left standing costs more than never having made it.

6. Open, and not being chased

Per operator direction, the wedge mechanism is not under active investigation. Failure data is being collected passively through test execution until the full estate has been run.

7. What is still untested

Stated plainly, because a coverage report that only lists successes is a marketing document.

The 332 and the 346 measure one drive at a time, one filesystem, well-formed media, no removals. They are a floor, not a finish line.