After the incident: the health panel shows 7 of 7 checks failing and every price ten times too high

I have written two long posts about protecting SurrealDB on my Talos home-lab cluster with Kasten. This is the short version. If you only read one thing:

  • SurrealDB is one database with seven data models: relational, document, graph, full-text, vector, geospatial and time series.
  • It has no write pause, so a disk snapshot alone is only crash-consistent. But
    surreal export
    reads each database in one transaction, so the export is consistent while writes continue.
  • Kasten turns that export into real protection: a small keeper pod writes the export to its own volume before each backup, and Kasten snapshots the whole namespace and copies it off-cluster.
  • One backup covers every model, including the indexes, the functions and the database users, because the export writes definitions as well as data.
  • I proved it by breaking all seven models at once and restoring from object storage: all 7 health checks went red, and a restore took them back to green in 2 minutes 38 seconds.

Why bother?

Availability is not resilience. Kubernetes, or a managed service such as SurrealDB Cloud, keeps the database running, but it keeps serving a bad migration or a mistaken script just as faithfully. Getting the data back to a known good point, and being able to move it to another cluster, is on us.

How it works

  1. Before each backup, Kasten runs a blueprint that execs into a small keeper pod. The keeper runs
    surreal export
    for every database onto its own volume, then flushes it to disk with
    sync
    .
  2. Kasten snapshots the whole namespace: the app, SurrealDB, the keeper, the Secrets and both volumes. Then it exports the restore point to object storage.
  3. On restore, Kasten brings everything back, and the blueprint removes each database and re-imports it from the consistent dump.

Nothing pauses writes, and nothing in the restore is done by hand.

The backup flow: Kasten runs the prehook in the keeper, the keeper exports SurrealDB in one transaction to its own volume and syncs, Kasten snapshots both volumes and exports them to object storage
The backup flow: the keeper writes a consistent dump, Kasten snapshots and exports it.

Seven models, one backup

In SurrealDB these are not seven systems. They are tables, fields, edges and indexes in one database, so one export captures them all:

ModelUsed for in the demo shopWhat the export brings back
RelationalProduct catalogueTyped table, unique index, rows
DocumentCustomersSchemaless records as they are
GraphWho bought whatEvery edge, plus the stored functions
Full-textReview searchThe analyzer and the BM25 index
VectorSimilar productsThe HNSW index and the embeddings
GeospatialNearest storeThe location points
Time seriesLive order feedEvery time-ordered event

Break it, restore it

The demo app has a Simulate incident button that damages every model at once: prices times ten, a customer and their orders deleted, the review table and the vector index dropped, a store removed. Then I restored the namespace from the copy Kasten had exported to object storage.

StepResult
Incident0 of 7 checks pass
Restore from object storage2 minutes 38 seconds
After restore7 of 7 checks pass, the app reconnects by itself
After the incident: the health panel shows 7 of 7 checks failing and every price ten times too high
After the incident: 7 of 7 checks fail.
After the restore: all 7 checks pass again and prices are back to normal
After the restore from object storage: 7 of 7 pass.

One lesson from doing it for real: the restore re-creates the database, which silently drops SurrealDB live queries. The data was perfect, but the app’s live feed went quiet until I taught it to subscribe again. Test what your clients hold on the server, not just the rows.

Numbers worth knowing

MeasurementResult
Exports taken during continuous writes5 of 5 consistent
Restore from object storage to a healthy app2 min 38 s
Re-export with no changes0.7% of the first export
Re-export after 1% of records changed, consecutive keys6.4%
Re-export after 1% of records changed, scattered51%

The last two rows are the surprise. Kasten’s export deduplicates, but the dump is written in key order, so changes spread across a table touch almost every chunk. Time-ordered keys, such as SurrealDB’s

ulid()
, keep new data at the end and the export cheap.

Read more

Tested on Talos 1.12.6, Kubernetes 1.34.1, Kasten 9.0.8 and SurrealDB 3.3.5. It is a home-lab demo with a small dataset, so run the same test with your own data before you rely on it.

Leave a Reply

Your email address will not be published. Required fields are marked *