Streams - Tapedrive

Large files split into segment tracks plus one manifest track that maps byte ranges to segments (tracks covers the model). On the read side, a range read resolves the manifest and fetches only the segments the range touches, so partial reads of huge files stay cheap (reading).

Stream writes

The streaming path splits the payload into segments. The SDK consumes input incrementally from any AsyncRead source. Each segment is written as a coded track, and the manifest is written last. The receipt carries the manifest address. The stream as a whole may carry a name and become an object. Segment tracks stay nameless and never appear in listings. Certification applies per coded track, and the engine handles it.

let file = tokio::fs::File::open("launch.mp4").await?;
let size = StorageUnits::from_bytes(file.metadata().await?.len());
let receipt = client.write_stream(&tape, size, file).await?;
println!("manifest: {}", receipt.manifest);

The manifest

For readers working below the one-call write. The manifest is an ordered list of entries, each mapping a byte range to a segment track address. It is itself a coded track, addressed like any other. Tooling authors get codec helpers for it on docs.rs (the wire type is ChunkManifest). Manifest-last ordering has one consequence: a stream is readable only once its manifest lands, so the manifest address is the stream’s identity.

A source split into segment tracks, a manifest mapping byte ranges to them, and a range read fetching only the two segments it touches. — A stream write: segments first, manifest last, and a range read touching only two segments.

Reading streams

Read by manifest address (or name), whole or by range. Reassembly is transparent: you receive bytes, never segments. For very large content, the streaming read consumes segments incrementally instead of buffering the whole object.

let whole = reader.read_bytes(&manifest_id).await?;

let mut sink = tokio::fs::File::create("launch.mp4").await?;
reader.read_into(&manifest_id, &mut sink).await?;

curl -H "Range: bytes=0-1048575" "$TAPE_GATEWAY/object/<manifest-address>" -o first-mb.bin

Interruption and resume

An interrupted stream write leaves completed segments behind as registered coded tracks. The manifest is absent, so nothing lists and nothing reads. Two ways forward:

Re-run the write. New segments are written; the orphans can be deleted to reclaim capacity.
Assemble the manifest yourself. Pipeline builders can write a manifest over already-written segments through the staged pipeline.

Re-running the one-call stream write does not adopt the orphaned segments; it rewrites every chunk from the start.

Sizing

The segment size is a network-sourced parameter (it is also the largest single track a gateway serves), and the boundaries between inline, coded, and stream dispatch are client-side defaults. None of them are protocol limits (tracks, tape replay).

​Stream writes

​The manifest

​Reading streams

​Interruption and resume

​Sizing

Stream writes

The manifest

Reading streams

Interruption and resume

Sizing