cr
Data

When a local database needs to survive its host

A local database can become important long before anyone decides how to recover it. Start with the failure you need to survive, and let that decide what you run.

#databases#durability#backups#recovery#chdb

A local database can become important long before anyone decides how to recover it. A script starts caching results. A developer tool starts saving history. An agent starts remembering decisions. Eventually, losing that state would hurt, but the recovery plan is still “the file is on my laptop.”

The useful question is simple: if this machine disappeared now, where would the next machine get its state?

Start with the failure you need to survive

“Durable” needs a failure boundary. Surviving a process restart is different from surviving a lost disk, a deleted virtual machine, or a compromised storage account. A local database can provide transactional durability without providing recovery after the entire host disappears.

Write down the event you care about before picking infrastructure. For a disposable cache, rebuilding might be sufficient. For saved customer work, rebuilding may be impossible. Those two cases should not inherit the same storage design just because they started with the same embedded engine.

Separate the working copy from the recovery copy

A ClickHouse engineering post on 28 September 2026 describes this problem with agent memory stored in chDB. The team wanted local queries while making state recoverable in CI, temporary sandboxes, and other machines. Their Durable Layer keeps an embedded working copy and publishes recoverable state to object storage, with explicit flush/checkpoint operations and one writer per object enforced by a lease.

That is one concrete implementation of a broader pattern: the machine doing the work does not have to be the only place from which the work can be recovered.

It also makes the next questions more visible.

Decide what you can afford to lose

How much recent work could you repeat? Five minutes of cached analysis and five minutes of customer edits are very different losses.

Choose a recovery target, then check whether the storage system’s documented guarantees actually meet it. Do not assume that a local write is already recoverable elsewhere. Distinguish the operation that changes the working copy from the operation that confirms remote recovery is possible.

If that distinction is visible in your application, use precise status text. “Saved locally” and “Recovery copy updated” describe different observations. A single reassuring “Saved” can hide a gap the user needs to understand.

Decide who can write

A copy in a bucket does not, by itself, solve concurrent editing. Two machines can both have plausible local state and disagree about which should become authoritative.

Before adding synchronization, define ownership: one active writer, a coordinated handover, or genuine concurrent writers? Then choose a system whose guarantees fit that requirement. A single-writer design can be appropriate, but it should not silently become a multi-writer system when someone opens a second laptop.

Test recovery as a separate workflow

A successful upload is evidence of an upload. Recovery needs its own check.

A practical exercise is to restore into a fresh environment, open the database, and verify a few known records or queries. Include whatever the application needs around the data: compatible software, configuration, access permissions, and a way to obtain credentials if the original machine is gone.

Document the steps that were actually needed. If recovery depends on a file that exists only beside the database you lost, the exercise has found something useful.

A small next step

Pick one local database you would miss. Write down its failure boundary, acceptable data loss, writer ownership, and recovery procedure. Then attempt a restore away from the original working directory.

That exercise gives you a concrete requirement for the next infrastructure decision. It may lead to a backup, a replicated embedded store, or a server database. Start with what must survive, and let that determine what you run.

← All posts