Features

EdgeStore bundles KV storage, vector search, full-text search, and replication into a single embedded library. Each layer sits cleanly on top of the core without polluting it.

Key-Value Store

Ordered byte keys with namespace isolation. Every key is prefix-encoded as {ns_len:u16}{ns}{key}, giving you a single flat sorted keyspace with logical separation.

put / get / delete — single-record operations
range — exclusive-end range scans
prefix — prefix-bounded iteration
namespaces — multi-tenant isolation without ColumnFamily complexity

// Basic operations
db.put(b"users", b"alice", b"{"age":30}")?;
let val = db.get(b"users", b"alice")?;

// Range scan within a namespace
for (k, v) in db.range(b"users", b"a", b"z")? {
    println!("{} = {}", k, v);
}

// Prefix scan
for (k, v) in db.prefix(b"events", b"2024-01-")? {
    // all keys starting with 2024-01-
}

Transactions

Atomic multi-record batches with single-writer semantics. A transaction accumulates puts and deletes in memory, then commits as a single WAL-append with group fsync.

begin() — start a transaction
commit_transaction() — atomic commit to WAL
rollback_transaction() — discard without writing
All-or-nothing: partial commits are impossible

let mut tx = db.begin();
tx.put(b"accounts", b"alice", b"900", 0, 0)?;
tx.put(b"accounts", b"bob", b"1100", 0, 0)?;

// Both writes land atomically, or neither does
db.commit_transaction(tx)?;

TTL & Lazy Expiry

Records with TTL are grouped by their predicted death time into cohorts. When a cohort fully expires, compaction removes it with zero live-data relocation — the key mechanism that drives write amplification toward 1.0.

put_with_ttl — write with seconds-to-live
Lazy expiry — expired records remain readable until compaction
Zero-cost collection — fully-expired cohorts need no live-record scans

// Session expires in 60 seconds
db.put_with_ttl(b"sessions", b"sess-42", b"active", 60)?;

// Still readable immediately after TTL expires
// (lazy expiry contract)
std::thread::sleep(Duration::from_secs(61));
let val = db.get(b"sessions", b"sess-42")?; // Some(...)

// After compaction, the cohort is gone
db.compact_once()?;
let val = db.get(b"sessions", b"sess-42")?; // None

Vector Search

Store and search dense vectors with SIMD-accelerated distance computation. Two search modes: flat brute-force scan (exact, fast for <500K vectors) and HNSW graph index (approximate, scales to millions).

Metrics — Cosine, L2 (Euclidean), Dot Product
Dtypes — f32, f16, i8 with automatic widening
SIMD — wide::f32x8 on x86_64/AVX2, scalar fallback everywhere
HNSW — probabilistic layer assignment, greedy beam search

// Insert vectors
db.vector_put(b"embeddings", b"doc1", 128, Dtype::F32, &vec_data)?;

// Build HNSW index for fast ANN search
db.build_vector_index(b"embeddings", 128, Dtype::F32, Metric::Cosine)?;

// Search top-10 nearest neighbors
let results = db.vector_search(
    b"embeddings", &query_vec, 10, Metric::Cosine
)?;
for r in results {
    println!("{} distance={:.4}", r.key, r.distance);
}

Full-Text Search

Embedded Algolia-like search with BM25 relevance scoring. Single merged inverted index per namespace, incrementally updated on write. Search reads the merged index directly — O(1) single-record deserialize, not O(N) per-document merging. No separate server process.

BM25 ranking — k1=1.2, b=0.75, configurable
Tokenization — English with stopwords and simple stemming
Faceting — filter results by facet values
Typo tolerance — Levenshtein distance ≤ 1 with 0.5 weight penalty

use edgestore_text::TextIndex;
use std::collections::HashMap;

let mut text = TextIndex::new();

// Index a document
text.index_document(&mut db, b"docs", b"doc1", "EdgeStore is fast and reliable", HashMap::new())?;
text.persist(&mut db)?;

// Search with BM25 ranking
let results = text.search(&db, b"docs", "fast database", 5)?;
for hit in results {
    println!("{} score={:.3}", hit.doc_id, hit.score);
}

Cost Accounting & Bounded Scans v1.5.0

Every read API has a _with_stats variant that returns a QueryStats struct alongside the result. Agent loops and quota-enforcement systems can measure exactly how many bytes and segments each query touched — no guesswork.

get_with_stats — point lookup + QueryStats{segments_scanned, bytes_scanned, items_examined}
range_with_stats / prefix_with_stats — scan variants that count bytes as they iterate
vector_search_with_stats — same for ANN paths
range_budgeted / prefix_budgeted — accept a ScanBudget{max_items, max_bytes} and stop early, returning a BudgetedScan{items, truncated}

// Cost-aware point lookup
let (val, stats) = db.get_with_stats(b"docs", b"doc1")?;
println!("bytes={} segs={}", stats.bytes_scanned, stats.segments_scanned);

// Bounded range scan — stop after 500 items or 1 MB
let budget = ScanBudget { max_items: 500, max_bytes: 1024 * 1024 };
let result = db.range_budgeted(b"logs", b"2026-01", b"2026-07", budget)?;
println!("{} items, truncated={}", result.items.len(), result.truncated);

// For cursor pagination use range_page instead (see below)

// ANN search with cost reporting
let (hits, stats) = db.vector_search_with_stats(
    b"vec", &query, 10, Metric::Cosine
)?;

Search Snippets v1.5.0

text.search_with_snippets returns SnippetResult values that bundle a document ID, BM25 score, and a list of Snippet objects — each with a context window around every matched term. The tokenizer records exact character offsets so context windows align precisely with the original text.

SnippetResult — {doc_id, score, snippets: Vec<Snippet>}
Snippet — {term, span, context_start, context_end} where span is the matched text
Character offsets — tokenizer v3 stores char-start positions, enabling accurate window extraction for any Unicode text
Ideal for RAG pipelines: pass span to an LLM as highlighted evidence without sending the full document

use edgestore_text::TextIndex;
use std::collections::HashMap;

let mut text = TextIndex::new();
text.index_document(&mut db, b"docs", b"doc1",
    "The quick brown fox jumps over the lazy dog", HashMap::new())?;
text.persist(&mut db)?;

let results = text.search_with_snippets(
    &db, b"docs", "fox", 5
)?;

for r in results {
    println!("doc={} score={:.3}", r.doc_id, r.score);
    for s in r.snippets {
        // span = "fox", context window around it
        println!("  term={} span={:?}", s.term, s.span);
    }
}
// Output:
//   doc=doc1  score=1.847
//   term=fox  span="fox"

Concurrent Vector Search v1.8.0

vector_search, vector_search_with_stats, and preload_vector_index now take &self — N concurrent searches no longer serialize against each other. The HNSW index cache uses an interior std::sync::RwLock with double-checked locking: a shared read lock is held for cache hits; a write lock is acquired only on the first load of a sidecar file, then released before returning results.

BM25 text search was already &self; vector search now matches
HNSW lazy load: write lock acquired once per namespace, then falls back to read-only access
build_vector_index still takes &mut self — it is a write operation
In AsyncEngine, HNSW and flat-scan paths both use blocking_read
Downstream wrappers using Arc<Mutex<Engine>> can switch to Arc<RwLock<Engine>> — all search methods are now compatible with shared read access

// vector_search now takes &self — concurrent reads compile
use std::thread;

let engine = Engine::open(config)?;
// ... put vectors, build_vector_index ...

// 8 threads, one shared &Engine — no Mutex needed
thread::scope(|s| {
    for _ in 0..8 {
        s.spawn(|| {
            let results = engine.vector_search(
                b"products", &query, 10, Metric::Cosine
            ).unwrap();
            process(results);
        });
    }
});

// AsyncEngine: HNSW + flat scan both use read lock
let results = async_engine
    .vector_search(b"products", &query, 10, Metric::Cosine)
    .await?;

Cursor Pagination v1.7.0

range_page and range_rev_page provide bounded page-by-page iteration without loading the full range into memory. Each call reads only the segment blocks that overlap the current page window; the scan stops at the I/O level once page_size live items are collected.

range_page — ascending iteration; cursor = last returned key
range_rev_page — descending iteration; cursor = smallest key returned. Efficient for latest-N queries when keys encode time.
RangePage.next_key — None when the range is exhausted, otherwise the cursor for the next call
Streaming block reads (one 4 KiB block per segment at a time) — I/O stops as soon as page_size items are collected across all segments
Works identically via AsyncEngine in edgestore-tokio

// Forward — walk all events in order
let mut cursor: Option<Vec<u8>> = None;
loop {
    let page = db.range_page(
        b"events", b"", b"\xff",
        cursor.as_deref(), 100
    )?;
    for (key, val) in &page.items { process(key, val); }
    cursor = page.next_key;
    if cursor.is_none() { break; }
}

// Reverse — newest-first, one page
let page = db.range_rev_page(
    b"logs", b"", b"\xff", None, 50
)?;
// page.items[0] is the largest key in the namespace

Replication

Merkle-based delta sync compares range-level Merkle trees between nodes to identify divergent segments. Only missing segments are transferred — not full copies. Lives in edgestore-repl, not the core crate.

Content-addressed — BLAKE3 hashes guarantee integrity
LWW conflict resolution — wall-clock timestamp wins
Resume-friendly — interrupted sync resumes from last Ack(lsn)
S3 backend — cold archive and replication mailbox (enable s3 feature)

// Requires edgestore-repl crate
use edgestore_repl::{AntiEntropyLoop, S3RemoteStore};

// Primary: export manifest
let segments = primary.export_manifest()?;

// Replica: compare and find missing
let missing = replica.missing_segments(&segments);

// Transfer each missing segment
for seg in missing {
    let data = primary.read_segment(&seg)?;
    replica.import_segment(&seg, &data)?;
}

// Verify Merkle roots match
let match_result = replica.compare_merkle(&primary_root)?;

Replication Safety v1.9.0

Three primitives for safe failover and crash-resilient archiving, inspired by the BtrLog paper (Kuschewski et al., VLDB 2026 — arXiv:2606.27051):

Commit watermark — engine.confirmed_lsn() returns the highest LSN whose segment is durably flushed. GET /watermark exposes it over HTTP alongside the write token. Replicas know exactly what is safe to serve.
Write-token fencing — engine.write_token() is a monotonic u64 persisted to WTOKEN. ReplicatedEngine::promote_to_primary(config, bind_addr, old_primary_url) fetches the old primary's token, advances it by one, and starts the HTTP server. Anti-entropy loops detect the jump and log a failover warning — no gossip required.
Idempotent uploads — RemoteStore::upload_if_absent returns true on first write and false when already present. Filesystem: O_CREAT|O_EXCL. S3: If-None-Match: * (HTTP 412 → false). Retry loops in TieredEngine and AntiEntropyLoop never double-upload.

// Failover: promote a replica to primary (v1.9.0)
use edgestore_repl::ReplicatedEngine;

// Old primary URL — fetch its write token before fencing.
// Pass None if the old primary is already down.
let new_primary = ReplicatedEngine::promote_to_primary(
    EdgestoreConfig::new("/var/db/replica"),
    "0.0.0.0:8900",
    Some("http://old-primary:8900"),
)?;

// Inspect the watermark on any engine
let eng = new_primary.engine().lock().unwrap();
println!("confirmed_lsn={} wtoken={}",
    eng.confirmed_lsn(), eng.write_token());

// Idempotent-safe archiving (called automatically by TieredEngine)
let wrote = remote_store.upload_if_absent(&hash, &data)?;
// wrote == false → already on S3, skip re-compress

Tiered Storage edgestore-tier

When local disk can't hold your full dataset, edgestore-tier wraps a local Engine with a transparent S3 read-through layer. Reads try local first; on a miss they fetch the matching archived segment from S3 ephemerally without permanently importing it.

Caller-driven archiving — policy (when to archive, when to prune) stays in your code
Read lock fast path — hot-window queries never block behind the writer (v1.4.0)
Flush-wake notify — flush_notify() on AsyncTieredEngine wakes a tiering worker the moment a segment lands
Text index stripping — with_text_stripping(true) auto-strips text index bytes from archived segments to reclaim local disk
Vector index stripping — with_vector_stripping(true) (v1.5.0) strips __vec__ records from archived segments. Paired with strip_vector_index(seg_id) on the core engine for manual use.

// Requires edgestore-tier crate
use edgestore_tier::TieredEngine;
use edgestore_repl::S3RemoteStore;

let local = Engine::open(config)?;
let remote = S3RemoteStore::new("bucket", None, None)?;
let mut tiered = TieredEngine::new(local, Box::new(remote));

// Writes stay local
tiered.put(b"logs", b"2026-07-12", data)?;

// Reads fall through to S3 on local miss
let val = tiered.get(b"logs", b"2025-01-01")?;

// Archive old segments to S3
let old = tiered.local().list_segment_metas();
tiered.archive_segments(&old)?;

Snapshots

RAII point-in-time reads that pin segments in the manifest. Compaction can proceed on live data while snapshots hold old segments for consistent reads.

snapshot() — capture current segment set
RAII release — Drop unpins segments automatically
SnapshotRegistry — tracks active snapshots, excludes pinned segments from compaction

// Capture point-in-time view
let snap = db.snapshot()?;

// Modify live data
db.put(b"data", b"key", b"new")?;

// Snapshot still sees old value
let old = snap.get(b"data", b"key")?;

// Release when done — pins are dropped automatically
drop(snap);

SSD Optimization

Every design decision prioritizes SSD health: append-only writes avoid in-place rewrites, deathtime-cohort compaction aligns invalidation with erase blocks, and FDP hints place related data together on NVMe 2.0 hardware.

Append-only — no page rewrites, minimal SSD GC pressure
ZSTD L1 compression — reduces bytes written without CPU cost
4 KiB block alignment — matches SSD page size
FDP hints — NVMe 2.0 placement guidance (Linux stub)

Write Amplification Comparison

Database	App WA	Device WA
EdgeStore	~1x	~1.2x
RocksDB	10-30x	~90x
LevelDB	10-30x	~90x
SQLite WAL	2-4x	~8x

For TTL workloads. See Papers for methodology.