12 · Chunked storage

Strand-Blobs

Multi-megabyte payloads split into BLAKE3-addressed chunks (64 KiB default), compressed with zstd, lz4, or snappy, and streamed end-to-end. A metadata Strand records the manifest; a data Strand holds the chunks. Identical content stored once across every writer.

Concepts

What you’re working with.

  • 01BLAKE3 chunking — 64 KiB default, tunable per writer.
  • 02Chunk-level deduplication across every writer on the node.
  • 03zstd, lz4, snappy compression negotiated per blob.
  • 04Streaming read / write with backpressure — multi-GB files stay flat in memory.
  • 05Metadata Strand records the manifest; Data Strand holds the chunks.
Code

A handful of lines.

use strand_blobs::{Node, BlobConfig, ChunkConfig};

let node = Node::open(BlobConfig {
    chunk: ChunkConfig::default().chunk_size(64 * 1024),
    compression: strand_blobs::Compression::Zstd,
    ..Default::default()
}).await?;

// Stream a multi-GB file in.
let id = node.put_stream("video.mp4", tokio::fs::File::open(&path).await?).await?;

// Stream it back — identical chunks served from local store.
let mut r = node.get_stream(&id).await?;
tokio::io::copy(&mut r, &mut sink).await?;

Ready to build?

Start with the SDK or load the agent context. Every library is independently documented and versioned.