This document explains the data-movement architecture: where bytes rest, who moves them, and how flow control works at every layer. Companion references: docs/api/usb-app-api.md (API + timing ledger), docs/perf-1tb-ssd.md (measured numbers behind the claims here).

The design in one paragraph

This is not an interrupt-driven, ring-buffered USB stack. It is a synchronous, consumer-pull pipeline: except for the PHY cog's 1 ms SOF heartbeat — and the isochronous audio rings that ride on it (see the audio exception) — every byte moves because a consumer called a blocking method on its own cog. Data rests in exactly two places — the device's internal buffers or the caller's buffer — and the driver holds bytes only transiently, in small fixed scratch areas, for the microseconds it takes to move one packet. Blocking calls are the flow control; per-class NAK budgets are the shock absorbers; one packet is in flight at any moment. Since the whole-transfer engine (1170) the per-packet BULK loop runs inside the PHY cog (UH_CMD_INX/OUTX) instead of interpreted Spin - same one-packet-at-a- time wire discipline, ~14x the throughput. The trade is streaming overlap for determinism, tiny memory, and exactly one cog.

The pipeline

 TX (host -> device), one 64-byte chunk of e.g. a WRITE(10) data stage:

 caller buffer --copy--> pkt[72] --(pointer via mailbox)--> PHY cog --> smart pin --> wire
 (your cog)    +PID,CRC16  (usb_client, caller's cog)     rdbyte per byte,  NRZI + bit-
                                                          paced by the pin's stuffing in
                                                          buffer-empty event silicon

 RX (device -> host), one IN transaction:

 wire --> smart pin --> PHY cog --writes--> rxb[72] --CRC check, copy--> caller buffer
          destuff,      byte events        (usb_client)     bytemove       (your cog)
          NRZI decode

Stage by stage:

• Smart pin (silicon). The P_USB_PAIR pair does all serialization: NRZI, bit stuffing/destuffing, line states. It holds ~one byte; the cog feeds or drains it per buffer-empty / byte-received event.
• PHY cog (usb_host PASM). Stateless byte mover. It owns the pins and executes one bus primitive per mailbox command: "transmit this packet and collect the handshake", "issue an IN token and capture the response". It also emits SOF autonomously (see below). It buffers nothing. Since ticket 1150 its cold handlers (init, line state, SOF enable, shutdown, sampling) execute from LUT RAM — cycle-identical to cog exec — leaving ~60 longs of cog-RAM headroom that the dual-bus (1160) and isochronous (1180) engines have since consumed, while every transaction hot loop stays in cog RAM.
• The mailbox — 16 longs in hub RAM, guarded by a hardware lock — is the only channel between Spin and the PHY. UH_L_BUF carries a pointer, so TX bytes are read by the PHY straight out of usb_client's packet buffer and RX bytes are written straight into its bounce buffer: exactly one copy on each side of the wire.
• usb_client (Spin, caller's cog). The protocol/policy engine. Control transfers still move packet-by-packet through pkt[72]/rxb[72] (PID + payload staged in Spin; CRC16 is generated AND verified in-cog by the PHY, which reports its verdict in RES bit 8). Bulk transfers post ONE whole-transfer command: the PHY loops token+DATAx+handshake in-cog, reading/writing the caller's buffer directly, and returns with progress on every exception - NAK pacing, retry budgets, toggles-table bookkeeping and abort policy stay here. One packet in flight on the wire, always.
• Class layers add framing on the same loop: MSC wraps BOT around it (CBW → up to 4 KB of data packets per SCSI command → CSW); FTDI strips its 2-byte status prefix off every IN packet; CDC, CP210x and PL2303 pass raw; HID moves one 8-byte report per interrupt IN.

Buffer inventory (complete)

Layer	Buffer	Size	Role
smart pin	internal shifter	~1 byte	serialization only
PHY cog	—	0	stateless mover
usb_client (one per class)	pkt[] / rxb[]	72 B each	control transfers only - bulk data moves directly between the PHY and the caller's buffer (1170)
MSC class	CBW/CSW/sense	~80 B	protocol frames — data phases go directly to/from the caller's buffer
exFAT	3 sector caches + entry-set buffer	≈2.5 KB	metadata read-modify-write, partial-sector bounce; whole-sector data bypasses the caches into your buffer
FTDI class	hdrbuf[64]	64 B	status-strip bounce
CP210x / PL2303 class	setup + line-coding scratch	≈30 B	control requests only — data is raw bulk, no bounce
expect engine	chunk[64] → acc[120] → cur[120] → toks[120]	≈400 B	packet → line assembly → delivered line → token copy
HID class	last-report buffers	8 B each	latest state snapshot
audio (USB_AUDIO)	ISO OUT ring + IN slots	4 KB + 8×256 B	the exception: hub-RAM rings the PHY services every SOF (below)

Outside the audio rings, the largest data buffer the driver owns anywhere is one 512-byte sector. Big buffers live where they belong: in your application (you pick the chunk size handed to fs_read/fs_write) and inside the devices themselves. The audio rings are the deliberate exception, sized in usb_hub_ram.spin2 — a hard-real-time stream needs a jitter reservoir the pull model cannot provide.

Backpressure

Device → host (IN). Flow control is inherent in USB's token protocol: nothing moves until the host asks. The driver only asks when a consumer calls (ser_read, fs_read, kbd_poll), so backpressure toward the device is automatic and absolute — unconsumed data stays in the device (the FTDI chip's FIFO, a storage bridge's cache, a CDC gadget's endpoint buffer). When the host asks and the device isn't ready, the device NAKs; the per-class patience budgets define how long we keep asking: ~64 ms/packet for interactive classes, ~10–20 s for storage data stages and 15 s for a post-write CSW (flash/GC stalls of seconds were measured — ticket 2087). Those budgets are the shock absorbers.

Host → device (OUT). A device with a full buffer NAKs the data packet; the driver retries the same packet, same toggle (duplicate-safe per spec) within the class budget. A device that stalls mid-write holds the calling cog in that retry loop — the backpressure propagates up to your blocking fs_write, which is exactly where a pull model wants it.

Where backpressure leaks — the one honest gap. Serial RX with an unattended host: the FTDI is programmed with no hardware flow control (3-wire instrument cables), so if a device transmits while the application ignores the link, pressure lands on the FTDI's internal FIFO (hundreds of bytes — ~100 ms of slack at 38400). Request/response protocols never accumulate; a device that streams unsolicited could overflow it, and the loss is silent unless you check line_status() (overrun bit). If you ever need to service such a device, add a pump: one cog looping ser_read into a hub-RAM ring, with the expect engine's recv_mp re-pointed at the ring's reader — the method-pointer transport binding exists precisely so that swap touches nothing else.

Concurrency and the SOF exception

All classes funnel through one mailbox and one lock: transfers from different cogs serialize, so a cog's kbd_poll waits while another cog is inside a long storage write. Bus routing is per calling cog (ticket 1200, from a field report): each cog's transfers carry that cog's bus selection, so a multi-cog application driving both buses at once — serial instruments on one, storage on the other — interleaves at command granularity without one cog's routing ever re-aiming another's in-flight sequence. Control transfers through the shared host path additionally hold a dedicated lock for their multi-command duration. The critical exception is SOF: the PHY cog emits the 1 ms keepalive autonomously between commands. Devices therefore never see the 3 ms idle bus that would suspend them, no matter how long the application computes between calls — a stalled consumer delays data, never bus liveness.

The audio exception: SOF-slaved isochronous streams

Isochronous audio (USB_AUDIO, tickets 1180/2110) is the one plane that cannot be consumer-pull: USB delivers ISO exactly once per 1 ms frame, no handshake, no retry — a sample not ready at SOF is a sample lost. So the audio path inverts the model, and does it entirely inside machinery that already existed for SOF:

 playback:  your DSP cog --aud_write--> OUT ring (4 KB hub RAM) --PHY, every SOF--> ISO OUT packet
 capture:   ISO IN packet --PHY, every SOF--> IN slots (8 x 256 B) --aud_read--> your DSP cog

• The PHY services the rings itself, immediately after each SOF — never Spin-paced, never through the mailbox. Packet size per frame comes from a 16.16 samples-per-frame accumulator (44.1 kHz is exact: 44 samples most frames, 45 when the fraction carries).
• Ring ops take no locks. aud_write is single-producer, aud_read single-consumer, over free-running hub-RAM counts. A storage call deep in its error budget cannot block the stream — measured: 60 s of zero-underrun playback while the async job cog copied files at 251 KB/s (harness/top_audio_accept.spin2).
• Backpressure is reported, not imposed. aud_write never blocks; it returns how much it accepted, and you pace on that or aud_free(). An empty ring at SOF is an underrun — the PHY sends nothing that frame (adaptive sinks hold or interpolate) and counts it. A full IN ring makes the PHY skip the frame's token entirely — the device discards the packet — counted as an overrun. The counters — aud_underruns()/aud_overruns()/aud_frames()/aud_in_frames() — are the stream-health contract.
• Bulk defers around the audio slot. aud_open computes the ISO frame reservation (UH_L_AUD_RSV, wire math + stuffing margin) and the frame guards add it to the audio bus's SOF deadline, so a bulk transaction never starts where it would squat on the frame's audio time. (Only the audio bus carries the reservation — taxing both buses' staggered 1 ms grids left almost no bulk window, measured 27 KB/s disk.) MSC keeps ~600–700 KB/s alongside a 48 kHz stereo + mic pair by budget.
• aud_read consumes whole frames only — a frame that doesn't fit stays queued (size your buffer ≥ the mic max packet). Partial reads would punch phase discontinuities into a DSP consumer (found the hard way; see the README validation notes).

Throughput economics

The wire runs 12 Mbit/s — a 64-byte packet occupies ~44 µs of bus. Each packet also costs a full interpreted-Spin round trip (build + CRC, mailbox post, PHY execution, handshake, toggle bookkeeping): ~1.2 ms all-in. Bulk throughput is therefore Spin-bound, not wire-bound: ≈52 KB/s raw, ≈45–48 KB/s through the filesystem — identical for an SSD and an SD reader (measured tables in perf-1tb-ssd.md). Corollary from the same measurements: chunk size barely matters above one sector (a 512-byte fs_write costs one ~12 ms block command; 16 KB chunks already reach ~95 % of ceiling), so size your buffers for your convenience.

The async job service (the one optional second consumer)

usb_fs_job (ticket 2088) adds an opt-in worker cog that runs whole filesystem jobs through the same facade. Nothing in the pipeline above changes - the worker is just another caller-cog consumer, so its transfers interleave with other classes at packet granularity (measured: a 100 ms instrument cadence kept 30 ms round trips during a 192 KB copy). It owns one 4 KB bounce buffer for copy(); for buffer jobs the caller's buffer is the destination as usual (and belongs to the job until reap()). Backpressure story unchanged: the job's blocking calls absorb device stalls on the worker cog instead of yours, and a surprise removal aborts its in-flight I/O in milliseconds via the per-instance abort flag.

Consumer chains, per class

• Storage/FS: fs_read(h, buf, n) → exFAT resolves clusters (metadata through its sector caches) → whole-sector spans go straight into buf via disk_read → MSC BOT chunks of 4 KB → 64-byte packets. Strictly request/response; the device's own cache is the only queue.
• Serial/expect: instrument → FTDI FIFO (+4 ms latency-timer flush) → one IN packet per ser_read poll (max 62 data bytes) → expect's line accumulator → readline()'s delivered line → tokens. drain() is the deliberate discard-consumer; its quiet detection is sampled (two consecutive empty polls), not timed.
• HID: one interrupt IN per kbd_poll/mouse_poll; the report is a state snapshot in an 8-byte class buffer; NAK simply means "no change".
• Audio: the inverted chain — producer/consumer cogs exchange PCM with the hub-RAM rings at their leisure; the PHY moves one ISO packet per direction per SOF, unconditionally. The intended shape is a dedicated DSP cog owning aud_read/aud_write while the main cog keeps poll() (examples/top_audio_dsp_example.spin2).