Architecture

EdgeStore is built around three principles: append-only writes, SSD-aware deathtime-cohort compaction, and layered purity (vector/text on top of KV, never inside it).

Crate Dependency

EdgeStore ships as a Cargo workspace. Each crate has a distinct purpose and audience.

edgestore core

KV + vector + text engine. Sync core. No async runtime dependency.

Engine WAL SegmentStore Compactor

edgestore-tokio async

Async wrapper. Every operation runs in spawn_blocking. Thin layer.

AsyncEngine

edgestore-repl network

REPLication transport — HTTP + S3. Not a REPL shell. Adds network and remote durability to the core engine.

HttpReplicationServer HttpReplicationClient AntiEntropyLoop S3RemoteStore

edgestore-cli admin

Administrative CLI tool. Create, put, get, stats, compact, export/import.

clap

edgestore ← used by all downstream crates

High-Level Architecture

Application

KV API

Vector API

Text API

Replication

Engine

Single-writer + group commit + snapshot RAII

WAL

LZ4 + CRC32C, rotated 64 MB / 60s

Memtable

BTreeMap, swappable trait

Segment Store

.dat (ZSTD) + .idx (sparse) + .xf (xor filter) + .meta (JSON)

Storage Backend

NVMe / SSD / S3 / FDP / Memory

Write Path

put(key, value, ttl?)

↓

append WAL record (LZ4, CRC32C)

↓

insert into Memtable (BTreeMap)

↓ flush triggered by size threshold

write sorted run → segment-{id}.dat (ZSTD)

build sparse index → segment-{id}.idx

build xor filter → segment-{id}.xf

write metadata → segment-{id}.meta

append to manifest → manifest file

↓

remove flushed keys from memtable

Read Path

get(namespace, key)

↓

search Memtable first

hit → return value

miss → continue

for each live segment (newest → oldest):

xor filter check → if absent, skip

sparse index seek → locate block

decompress block → scan for key

hit → return value with highest LSN

↓

not found → return None

Compaction Path

compact_once(write_budget)

↓

identify cohorts where max(death_time) < now

↓ fully expired cohorts first

for each selected cohort within budget:

read all segments in cohort

merge-sort by key, keep highest-LSN

write new segments with updated cohort buckets

recompute Merkle roots + BLAKE3 hashes

update manifest (append new, mark old obsolete)

delete obsolete files (if not pinned by snapshot)

↓

return CompactionStats

File Formats

WAL Record

txid
u64

lsn
u64

timestamp
i64

ttl
u32

namespace
bytes

key
bytes

operation
u8

value_hash
[u8;32]

value_bytes
bytes

Segment Files

dat

ZSTD data blocks (4 KiB aligned)

idx

Sparse offset index (key → block)

Xor filter (~8 bits/key, 1% FPR)

Namespace Encoding

Single flat keyspace. No ColumnFamily complexity.

ns_len

2 bytes

namespace

variable

user_key

variable

Enables: prefix scans, range queries, time-series, secondary indexes — all within one sorted keyspace.

Concurrency Model

Single Writer

One active write tx at a time

Multiple Readers

Never block writers; snapshot manifest

Group Commit

Batch tx commits into one fsync

No MVCC in v1. WAL txid field reserved for future layering.

References

prod.md

Full technical specification (source of truth)

AGENTS.md

Developer quick-start, API surface, GSD workflow

CHANGELOG.md

Release history and fixes

Research Papers

VLDB 2026, SSD WAF, SlateDB, NVMe+S3