After the incident: the health panel shows 7 of 7 checks failing, prices multiplied by ten, and Bob missing from the graph

In my last post I protected SurrealDB on my Talos home-lab cluster with Kasten. The test data was deliberately tiny: five products, three customers and a handful of graph edges. That proved the method, but it left an obvious question. SurrealDB’s whole pitch is that it is one database for many data models. Does one backup really bring all of them back?

So I built a small demo app that uses every model SurrealDB offers: relational, document, graph, full-text search, vector search, geospatial and time series, with live queries on top. Then I broke all of it at once and restored it. This post walks through each model, what the app uses it for, how the protection covers it, and what it looks like when a bad thing happens and you get everything back to the last known good, consistent state.

Before getting into the models, one distinction that is easy to blur: availability is not resilience. My Kubernetes cluster keeps the database available: if a node or a pod dies, it is rescheduled and SurrealDB comes back. A managed service such as SurrealDB Cloud does the same for its platform. But availability faithfully keeps serving whatever is in the database, including a bad migration, a buggy deploy or a script run against the wrong environment. Keeping the data recoverable to a known good point is on us. So is being able to restore it somewhere else: to another Kubernetes cluster, and perhaps one day from my own platform to a managed one.

That is also why I protect the whole application namespace with Kasten, not just the database.

surreal export
on its own is a perfectly good way to get a consistent copy of the data, and if that is all you need, it may be enough. For resilience I want more than the data:

surreal export
on its own
Kasten with the keeper blueprint
Consistent copy of each databaseYesYes: it uses the same export
The app, its Deployments, Services, routes and SecretsNoYes, from the same restore point
Scheduling, retention and an off-cluster copyYou build itPolicies, retention and export to object storage
RestoreRe-create everything around the data, then importOne action: the namespace comes back with a consistent database
Move to another clusterCopy files and redeploy by handRestore the exported restore point on another cluster that runs Kasten

In this post every restore went back into the same cluster. Restoring into a second cluster is the natural next test, and the dump is plain SurrealQL, which is also what a move to a managed SurrealDB would start from.

The demo app at a glance

The app is a small shop: a catalogue, customers, orders, reviews and stores. It is one Go binary that serves a single web page, and it runs in the same namespace as SurrealDB and the keeper. That detail matters: one Kasten restore brings back the app, its configuration and its data together.

demo architecture · one namespace, two routes, one Kasten policy
demo architecture · one namespace, two routes, one Kasten policy

The app talks to SurrealDB over the internal Service, as a database user that can only work in

demo/shop
. Two separate routes reach the cluster:
shop.<ip>.nip.io
for the app, and
surrealdb.<ip>.nip.io
for the SurrealDB Studio desktop app, so I can browse the data during a demo. A Generate orders switch keeps writes flowing, which is the point: the backup has to be consistent while the database is busy.

Each panel on the page is one data model, and a health panel at the top checks all of them.

The shop with orders running: the health panel shows all 7 checks passing, above the catalogue and the who-bought-what graph
Healthy, with Generate orders on: 7 of 7 checks pass. The graph fills up as orders arrive.
Review search with highlighted matches, similar products by cosine distance, nearest stores on a map, and the live order feed
The other four models: full-text search with highlights, vector similarity, nearest store, and the live order feed with orders per minute.

One database, seven models

Everything below lives in one SurrealDB namespace and database,

demo/shop
, defined in about 25 lines of SurrealQL. There is no second database, no search engine and no vector store bolted on the side.

ModelWhat the shop uses it forStored as
RelationalThe product catalogue, with typed fields and a unique nameSchemafull table
product
+ unique index
DocumentCustomers, whose shape can varySchemaless table
customer
GraphWho bought what, and “customers who bought this also bought”Edge table
ordered
(
customer->ordered->product
)
Full-textSearching product reviewsTable
review
+ analyzer + BM25 index
Vector“Similar products”
embedding
field on
product
+ HNSW index
GeospatialNearest store to each customerPoint fields on
store
and
customer
Time seriesOrders per minute and a live order feedTable
order_event
with time-ordered ULID keys + a live query

Relational: the catalogue

Prices have to be numbers and product names have to be unique, so

product
is schemafull: SurrealDB rejects a record that does not match.


DEFINE TABLE product SCHEMAFULL;
DEFINE FIELD price ON product TYPE number;
DEFINE INDEX product_name ON product FIELDS name UNIQUE;

Document: customers

Customers are schemaless: a name, a city and a location today, anything else tomorrow, with no migration.


UPSERT customer:alice SET name = 'Alice', city = 'Paris', location = (2.3522, 48.8566);

Graph: who bought what

An order is an edge between a customer and a product, and the edge has its own fields. Recommendations are a two-hop traversal with no JOIN: from a product back to its buyers, then forward to everything else they bought. A stored function,

fn::customer_total
, walks the same edges to total a customer’s spend.


RELATE customer:alice->ordered->product:kettle SET qty = 2;
SELECT out.name AS name, count() AS count FROM ordered
  WHERE in IN $p<-ordered<-customer AND out != $p GROUP BY name;

Full-text: review search

Reviews are indexed with an English analyzer (lower-case, stemming) and ranked with BM25. The app asks for highlights, so the matched words come back marked.


DEFINE ANALYZER review_en TOKENIZERS blank, class FILTERS lowercase, snowball(english);
DEFINE INDEX review_text ON review FIELDS text FULLTEXT ANALYZER review_en BM25 HIGHLIGHTS;
SELECT text, search::score(1) AS score FROM review WHERE text @1@ 'quiet';

Vector: similar products

Each product has a six-number embedding: how much it is about the kitchen, electricity, heat, blending, coffee and a budget price. I made these by hand so that the results are easy to explain on stage; a real app would use a model. An HNSW index answers nearest-neighbour queries by cosine distance.


DEFINE INDEX product_vec ON product FIELDS embedding HNSW DIMENSION 6 DIST COSINE;
SELECT name, vector::distance::knn() AS distance FROM product WHERE embedding <|3,40|> $p.embedding;

Geospatial: nearest store

Stores and customers have longitude/latitude points.

geo::distance
returns metres, so the nearest store is a sort.


SELECT name, geo::distance(location, (-1.4701, 53.3811)) AS metres FROM store ORDER BY metres LIMIT 1;

Time series and live queries: orders

Every order is an event keyed by a ULID, an ID that sorts by creation time, so the latest orders are simply the highest keys. The chart groups events by minute, and the live feed is pushed to the browser by a SurrealDB live query: the app subscribes once, and SurrealDB sends each new order as it is written.


CREATE order_event:ulid() SET at = time::now(), customer = $c, product = $p, qty = $q;
SELECT time::group(at, 'minute') AS minute, count() AS orders FROM order_event GROUP BY minute;
LIVE SELECT * FROM order_event;

The ULID choice matters for backup cost too. In the first post, scattered changes made the export upload close to a full dump, while changes on consecutive keys cost a small fraction. Time-ordered keys put new orders at the end of the table.

How one backup covers every model

The protection is exactly the one from the first post, unchanged: the keeper-PVC blueprint, one Kasten policy, one backup and one export. I did not touch the blueprint to add the app. Its

backupPrehook
lists every SurrealDB namespace and database, so the new
demo/shop
database was picked up on the first run.

what one backup captures · 7 models, 1 file per database
what one backup captures · 7 models, 1 file per database

The reason one backup covers seven models is that, in SurrealDB, they are not seven systems. They are tables, fields, edges and indexes in one database, and

surreal export
writes all of it, in one read transaction, as SurrealQL:

ModelWhat the export writes
Relational
DEFINE TABLE product SCHEMAFULL
, every
DEFINE FIELD
, the
UNIQUE
index, then
INSERT
of the rows
Document
DEFINE TABLE customer
, then
INSERT
of each document as it is
Graph
DEFINE TABLE ordered TYPE RELATION
, then
INSERT RELATION
of every edge with its fields;
DEFINE FUNCTION
for the stored functions
Full-text
DEFINE ANALYZER
, the
FULLTEXT ... BM25
index definition, and the reviews
VectorThe
HNSW
index definition and every
embedding
array
GeospatialThe points, as part of each record
Time seriesEvery
order_event
, under its ULID key

Two things in that table matter more than they look. First, one read transaction: every model is captured at the same instant, so an order event can never be in the backup while its graph edge is not, even with orders being written during the backup. The app writes both in one transaction, and the export reads both in one transaction. Second, definitions, not just data: the dump carries the analyzer and the index definitions, so a restore rebuilds working search, not just the rows.

The dump also carries the app’s own database user (

DEFINE USER demo ON DATABASE
), so the app can sign in again after a restore without anyone touching it. For the demo database, the dump was about 40 KB on the first backup, and 106 KB after an hour of generated orders.

The bad day

A backup only counts once you have watched it bring something back, so the app has a Simulate incident button. It damages every model in one go, which is roughly what a bad migration, a buggy deploy or a mistaken script against production does:

ModelWhat the incident doesWhat you see in the app
RelationalMultiplies every price by 10Catalogue prices jump; the relational check fails
DocumentDeletes
customer:bob
Bob disappears from the customer list
GraphDeletes Bob’s order edgesFewer lines in the graph; seeded edge count and Alice’s total no longer match
Full-textDrops the
review
table, and its index with it
Review search finds nothing
VectorRemoves the HNSW indexSimilar products returns nothing
GeospatialDeletes the Manchester storeSheffield’s nearest store changes
Time seriesDeletes the seeded order eventsThe time-series check fails

The health panel at the top of the app is what makes this visible in a demo. It runs one check per model against the seeded data, so live traffic never turns a check red; only damage does. Before the incident it shows 7 of 7. After it, 0 of 7.

Two of these are worth pausing on. Dropping a table and removing an index are schema changes, not data changes. A backup that only copied rows would bring the reviews back without their full-text index, and the products back without their vector index, and both searches would still be broken.

After the incident: the health panel shows 7 of 7 checks failing, prices multiplied by ten, and Bob missing from the graph
Seconds after the incident: 7 of 7 checks fail, every price is ten times too high and Bob has gone from the graph.
After the incident: review search reports the review table does not exist, similar products has no index, and Dana's nearest store is now London
Search fails because the table has gone, the vector panel has no index, and Dana’s nearest store is now London, 262 km away. The live feed even shows a “?” where an order lost its customer.

Back to the last known good state

The restore is one Kasten action. I picked the restore point that Kasten had exported to object storage, not the local snapshot, to prove the off-cluster copy is enough on its own. Here is the timeline from one run, with orders being generated throughout:

Time (UTC)What happened
12:42:01Kasten backup: the prehook exports every database, including
demo/shop
(106 KB), in one read transaction each
12:43:30Simulate incident: the health panel drops to 0 of 7
12:43:58Restore starts from the exported restore point
12:46:16Kasten has restored the PVCs and started the pods again, including the app
12:46:36The
restorePosthook
finishes: every database removed and re-imported from the dump taken at 12:42:01
12:46:36Health panel: 7 of 7

From the restore button to a healthy app took 2 minutes 38 seconds, most of it pulling the volumes back from object storage. Every model came back to the same instant, 12:42:01: prices, Bob and his orders, the reviews and their full-text index, the vector index, the Manchester store and the seeded order events. Orders written between the backup and the incident are gone, which is exactly what “last known good state” means.

Nothing was done by hand. There is no step where someone re-creates an index, re-runs a migration or re-defines the app’s database user: they are all in the dump.

After the restore: all 7 checks pass again, prices are back to normal and Bob is back in the graph
After the restore from the exported copy: 7 of 7 again, prices back to normal, Bob back in the graph, and orders flowing.

One thing the demo caught

The first time I ran this with the app inside the cluster, the health panel went green but the live order feed stayed silent. The app had reconnected while the database was still being re-imported, and removing a database in SurrealDB also removes the live queries registered on it, without closing the connection. So the app was connected, healthy and subscribed to nothing.

The fix was small: every five seconds the app checks that its live query is still listed in

INFO FOR TABLE order_event
, and subscribes again if it is not. The lesson is bigger: a restore can be correct for the data and still break a client’s state. Live queries, cached prepared statements and anything else a client holds on the server side deserve a test after a restore, not just the rows.

I ran the whole cycle again with the fix deployed. Five seconds after it reconnected, the app logged

live query ... on order_event no longer exists (database replaced?)
, subscribed again three seconds later, and the live feed was back with all 7 checks green. That restore took 2 minutes 48 seconds.

What this does and does not prove

The demo is honest about its size, and so should this post be.

It provesIt does not prove
One export captures every model in a database: records, edges, schema, analyzers, vector and full-text indexes, functions and database usersThat it scales to your data: the shop’s dump is around 100 KB. Run the same test at real size, with your data types
The copy is consistent while writes continue: each database is read in one transactionThat separate databases are consistent with each other: each one is exported in its own transaction, so keep data that must agree in one database
A restore from the off-cluster copy brings every model back, and the app reconnects by itselfHow long index rebuilds take at scale: on import, SurrealDB rebuilds the HNSW and full-text indexes from the data, so restore time grows with the data, not just the bytes
The restore puts the database back to the backup: changes made after it are gone, which is what you want after a bad changeRecovery to a point between backups: there is no log replay, so the backup schedule is your recovery point

Root- and namespace-level users are not in a database export either. They come back with the SurrealDB data volume, which Kasten restores too, or from the Kubernetes Secret on an empty instance.

Try it yourself, and what’s next

The app is open source at MichaelCade/surrealdb-webapp: a single Go binary using the official SurrealDB Go SDK, plain HTML and JavaScript with no external libraries, and Kubernetes manifests. GitHub Actions publishes the image to

ghcr.io/michaelcade/surrealdb-webapp
. The README has a ten-minute demo script:

  1. Tour the panels with Generate orders on.
  2. Run the Kasten policy while the orders keep flowing.
  3. Click Simulate incident: 7 of 7 becomes 0 of 7.
  4. Restore from the exported restore point.
  5. Watch the health panel return to 7 of 7.

You need SurrealDB, the keeper and the blueprint from the blueprint-ai

surrealdb-example
runbook first; the app adds one setup script and one manifest. To browse the data while you demo, connect the SurrealDB Studio desktop app to the database through its own route.

ComponentVersion tested
Talos / Kubernetes1.12.6 / 1.34.1
Kasten9.0.8
SurrealDB3.3.5, SurrealKV
SurrealDB Go SDK2.0.1

Next up is SurrealDB 4.0, which drops RocksDB and TiKV and keeps SurrealKV as its only on-disk engine. The Go SDK already supports it, so the same app should make a good test of the upgrade.

Sources

Leave a Reply

Your email address will not be published. Required fields are marked *