12 — Chunked storage
Strand-Blobs
Multi-megabyte payloads split into BLAKE3-addressed chunks (64 KiB default), compressed with zstd, lz4, or snappy, and streamed end-to-end. A metadata Strand records the manifest; a data Strand holds the chunks. Identical content stored once across every writer.
A closer lookBigger files. A precise address.
Content-addressed payloads sit beside signed records. A reference identifies the content an agent requests.
- 01Address the content
- 02Attach the reference
- 03Retrieve the payload
01 — Concepts
01
BLAKE3 chunking — 64 KiB default, tunable per writer.
02
Chunk-level deduplication across every writer on the node.
03
zstd, lz4, snappy compression negotiated per blob.
04
Streaming read / write with backpressure — multi-GB files stay flat in memory.
05
Metadata Strand records the manifest; Data Strand holds the chunks.
02 — Code
A handful of lines.
use strand_blobs::{Node, BlobConfig, ChunkConfig};
let node = Node::open(BlobConfig {
chunk: ChunkConfig::default().chunk_size(64 * 1024),
compression: strand_blobs::Compression::Zstd,
..Default::default()
}).await?;
// Stream a multi-GB file in.
let id = node.put_stream("video.mp4", tokio::fs::File::open(&path).await?).await?;
// Stream it back — identical chunks served from local store.
let mut r = node.get_stream(&id).await?;
tokio::io::copy(&mut r, &mut sink).await?;