Kasten – vZilla https://vzilla.co.uk One Step into Kubernetes and Cloud Native at a time, not forgetting the world before Thu, 19 Nov 2026 14:13:07 +0000 en-GB hourly 1 https://wordpress.org/?v=7.1.4 https://vzilla.co.uk/wp-content/uploads/2018/01/cropped-profile_picture_symbol-32x32.png Kasten – vZilla https://vzilla.co.uk 32 32 TL;DR: protecting SurrealDB on Kubernetes with Kasten https://vzilla.co.uk/vzilla-blog/tldr-protecting-surrealdb-on-kubernetes-with-kasten https://vzilla.co.uk/vzilla-blog/tldr-protecting-surrealdb-on-kubernetes-with-kasten#respond Thu, 19 Nov 2026 14:13:03 +0000 https://vzilla.co.uk/?p=3759 I have written two long posts about protecting SurrealDB on my Talos home-lab cluster with Kasten. This is the short version. If you only read one thing:

  • SurrealDB is one database with seven data models: relational, document, graph, full-text, vector, geospatial and time series.
  • It has no write pause, so a disk snapshot alone is only crash-consistent. But
    surreal export
    reads each database in one transaction, so the export is consistent while writes continue.
  • Kasten turns that export into real protection: a small keeper pod writes the export to its own volume before each backup, and Kasten snapshots the whole namespace and copies it off-cluster.
  • One backup covers every model, including the indexes, the functions and the database users, because the export writes definitions as well as data.
  • I proved it by breaking all seven models at once and restoring from object storage: all 7 health checks went red, and a restore took them back to green in 2 minutes 38 seconds.

Why bother?

Availability is not resilience. Kubernetes, or a managed service such as SurrealDB Cloud, keeps the database running, but it keeps serving a bad migration or a mistaken script just as faithfully. Getting the data back to a known good point, and being able to move it to another cluster, is on us.

How it works

  1. Before each backup, Kasten runs a blueprint that execs into a small keeper pod. The keeper runs
    surreal export
    for every database onto its own volume, then flushes it to disk with
    sync
    .
  2. Kasten snapshots the whole namespace: the app, SurrealDB, the keeper, the Secrets and both volumes. Then it exports the restore point to object storage.
  3. On restore, Kasten brings everything back, and the blueprint removes each database and re-imports it from the consistent dump.

Nothing pauses writes, and nothing in the restore is done by hand.

The backup flow: Kasten runs the prehook in the keeper, the keeper exports SurrealDB in one transaction to its own volume and syncs, Kasten snapshots both volumes and exports them to object storage
The backup flow: the keeper writes a consistent dump, Kasten snapshots and exports it.

Seven models, one backup

In SurrealDB these are not seven systems. They are tables, fields, edges and indexes in one database, so one export captures them all:

ModelUsed for in the demo shopWhat the export brings back
RelationalProduct catalogueTyped table, unique index, rows
DocumentCustomersSchemaless records as they are
GraphWho bought whatEvery edge, plus the stored functions
Full-textReview searchThe analyzer and the BM25 index
VectorSimilar productsThe HNSW index and the embeddings
GeospatialNearest storeThe location points
Time seriesLive order feedEvery time-ordered event

Break it, restore it

The demo app has a Simulate incident button that damages every model at once: prices times ten, a customer and their orders deleted, the review table and the vector index dropped, a store removed. Then I restored the namespace from the copy Kasten had exported to object storage.

StepResult
Incident0 of 7 checks pass
Restore from object storage2 minutes 38 seconds
After restore7 of 7 checks pass, the app reconnects by itself
After the incident: the health panel shows 7 of 7 checks failing and every price ten times too high
After the incident: 7 of 7 checks fail.
After the restore: all 7 checks pass again and prices are back to normal
After the restore from object storage: 7 of 7 pass.

One lesson from doing it for real: the restore re-creates the database, which silently drops SurrealDB live queries. The data was perfect, but the app’s live feed went quiet until I taught it to subscribe again. Test what your clients hold on the server, not just the rows.

Numbers worth knowing

MeasurementResult
Exports taken during continuous writes5 of 5 consistent
Restore from object storage to a healthy app2 min 38 s
Re-export with no changes0.7% of the first export
Re-export after 1% of records changed, consecutive keys6.4%
Re-export after 1% of records changed, scattered51%

The last two rows are the surprise. Kasten’s export deduplicates, but the dump is written in key order, so changes spread across a table touch almost every chunk. Time-ordered keys, such as SurrealDB’s

ulid()
, keep new data at the end and the export cheap.

Read more

Tested on Talos 1.12.6, Kubernetes 1.34.1, Kasten 9.0.8 and SurrealDB 3.3.5. It is a home-lab demo with a small dataset, so run the same test with your own data before you rely on it.

]]>
https://vzilla.co.uk/vzilla-blog/tldr-protecting-surrealdb-on-kubernetes-with-kasten/feed 0
One database, seven models: protecting all of SurrealDB with Kasten https://vzilla.co.uk/vzilla-blog/one-database-seven-models-protecting-all-of-surrealdb-with-kasten https://vzilla.co.uk/vzilla-blog/one-database-seven-models-protecting-all-of-surrealdb-with-kasten#respond Thu, 19 Nov 2026 13:39:46 +0000 https://vzilla.co.uk/?p=3752 In my last post I protected SurrealDB on my Talos home-lab cluster with Kasten. The test data was deliberately tiny: five products, three customers and a handful of graph edges. That proved the method, but it left an obvious question. SurrealDB’s whole pitch is that it is one database for many data models. Does one backup really bring all of them back?

So I built a small demo app that uses every model SurrealDB offers: relational, document, graph, full-text search, vector search, geospatial and time series, with live queries on top. Then I broke all of it at once and restored it. This post walks through each model, what the app uses it for, how the protection covers it, and what it looks like when a bad thing happens and you get everything back to the last known good, consistent state.

Before getting into the models, one distinction that is easy to blur: availability is not resilience. My Kubernetes cluster keeps the database available: if a node or a pod dies, it is rescheduled and SurrealDB comes back. A managed service such as SurrealDB Cloud does the same for its platform. But availability faithfully keeps serving whatever is in the database, including a bad migration, a buggy deploy or a script run against the wrong environment. Keeping the data recoverable to a known good point is on us. So is being able to restore it somewhere else: to another Kubernetes cluster, and perhaps one day from my own platform to a managed one.

That is also why I protect the whole application namespace with Kasten, not just the database.

surreal export
on its own is a perfectly good way to get a consistent copy of the data, and if that is all you need, it may be enough. For resilience I want more than the data:

surreal export
on its own
Kasten with the keeper blueprint
Consistent copy of each databaseYesYes: it uses the same export
The app, its Deployments, Services, routes and SecretsNoYes, from the same restore point
Scheduling, retention and an off-cluster copyYou build itPolicies, retention and export to object storage
RestoreRe-create everything around the data, then importOne action: the namespace comes back with a consistent database
Move to another clusterCopy files and redeploy by handRestore the exported restore point on another cluster that runs Kasten

In this post every restore went back into the same cluster. Restoring into a second cluster is the natural next test, and the dump is plain SurrealQL, which is also what a move to a managed SurrealDB would start from.

The demo app at a glance

The app is a small shop: a catalogue, customers, orders, reviews and stores. It is one Go binary that serves a single web page, and it runs in the same namespace as SurrealDB and the keeper. That detail matters: one Kasten restore brings back the app, its configuration and its data together.

demo architecture · one namespace, two routes, one Kasten policy
demo architecture · one namespace, two routes, one Kasten policy

The app talks to SurrealDB over the internal Service, as a database user that can only work in

demo/shop
. Two separate routes reach the cluster:
shop.<ip>.nip.io
for the app, and
surrealdb.<ip>.nip.io
for the SurrealDB Studio desktop app, so I can browse the data during a demo. A Generate orders switch keeps writes flowing, which is the point: the backup has to be consistent while the database is busy.

Each panel on the page is one data model, and a health panel at the top checks all of them.

The shop with orders running: the health panel shows all 7 checks passing, above the catalogue and the who-bought-what graph
Healthy, with Generate orders on: 7 of 7 checks pass. The graph fills up as orders arrive.
Review search with highlighted matches, similar products by cosine distance, nearest stores on a map, and the live order feed
The other four models: full-text search with highlights, vector similarity, nearest store, and the live order feed with orders per minute.

One database, seven models

Everything below lives in one SurrealDB namespace and database,

demo/shop
, defined in about 25 lines of SurrealQL. There is no second database, no search engine and no vector store bolted on the side.

ModelWhat the shop uses it forStored as
RelationalThe product catalogue, with typed fields and a unique nameSchemafull table
product
+ unique index
DocumentCustomers, whose shape can varySchemaless table
customer
GraphWho bought what, and “customers who bought this also bought”Edge table
ordered
(
customer->ordered->product
)
Full-textSearching product reviewsTable
review
+ analyzer + BM25 index
Vector“Similar products”
embedding
field on
product
+ HNSW index
GeospatialNearest store to each customerPoint fields on
store
and
customer
Time seriesOrders per minute and a live order feedTable
order_event
with time-ordered ULID keys + a live query

Relational: the catalogue

Prices have to be numbers and product names have to be unique, so

product
is schemafull: SurrealDB rejects a record that does not match.


DEFINE TABLE product SCHEMAFULL;
DEFINE FIELD price ON product TYPE number;
DEFINE INDEX product_name ON product FIELDS name UNIQUE;

Document: customers

Customers are schemaless: a name, a city and a location today, anything else tomorrow, with no migration.


UPSERT customer:alice SET name = 'Alice', city = 'Paris', location = (2.3522, 48.8566);

Graph: who bought what

An order is an edge between a customer and a product, and the edge has its own fields. Recommendations are a two-hop traversal with no JOIN: from a product back to its buyers, then forward to everything else they bought. A stored function,

fn::customer_total
, walks the same edges to total a customer’s spend.


RELATE customer:alice->ordered->product:kettle SET qty = 2;
SELECT out.name AS name, count() AS count FROM ordered
  WHERE in IN $p<-ordered<-customer AND out != $p GROUP BY name;

Full-text: review search

Reviews are indexed with an English analyzer (lower-case, stemming) and ranked with BM25. The app asks for highlights, so the matched words come back marked.


DEFINE ANALYZER review_en TOKENIZERS blank, class FILTERS lowercase, snowball(english);
DEFINE INDEX review_text ON review FIELDS text FULLTEXT ANALYZER review_en BM25 HIGHLIGHTS;
SELECT text, search::score(1) AS score FROM review WHERE text @1@ 'quiet';

Vector: similar products

Each product has a six-number embedding: how much it is about the kitchen, electricity, heat, blending, coffee and a budget price. I made these by hand so that the results are easy to explain on stage; a real app would use a model. An HNSW index answers nearest-neighbour queries by cosine distance.


DEFINE INDEX product_vec ON product FIELDS embedding HNSW DIMENSION 6 DIST COSINE;
SELECT name, vector::distance::knn() AS distance FROM product WHERE embedding <|3,40|> $p.embedding;

Geospatial: nearest store

Stores and customers have longitude/latitude points.

geo::distance
returns metres, so the nearest store is a sort.


SELECT name, geo::distance(location, (-1.4701, 53.3811)) AS metres FROM store ORDER BY metres LIMIT 1;

Time series and live queries: orders

Every order is an event keyed by a ULID, an ID that sorts by creation time, so the latest orders are simply the highest keys. The chart groups events by minute, and the live feed is pushed to the browser by a SurrealDB live query: the app subscribes once, and SurrealDB sends each new order as it is written.


CREATE order_event:ulid() SET at = time::now(), customer = $c, product = $p, qty = $q;
SELECT time::group(at, 'minute') AS minute, count() AS orders FROM order_event GROUP BY minute;
LIVE SELECT * FROM order_event;

The ULID choice matters for backup cost too. In the first post, scattered changes made the export upload close to a full dump, while changes on consecutive keys cost a small fraction. Time-ordered keys put new orders at the end of the table.

How one backup covers every model

The protection is exactly the one from the first post, unchanged: the keeper-PVC blueprint, one Kasten policy, one backup and one export. I did not touch the blueprint to add the app. Its

backupPrehook
lists every SurrealDB namespace and database, so the new
demo/shop
database was picked up on the first run.

what one backup captures · 7 models, 1 file per database
what one backup captures · 7 models, 1 file per database

The reason one backup covers seven models is that, in SurrealDB, they are not seven systems. They are tables, fields, edges and indexes in one database, and

surreal export
writes all of it, in one read transaction, as SurrealQL:

ModelWhat the export writes
Relational
DEFINE TABLE product SCHEMAFULL
, every
DEFINE FIELD
, the
UNIQUE
index, then
INSERT
of the rows
Document
DEFINE TABLE customer
, then
INSERT
of each document as it is
Graph
DEFINE TABLE ordered TYPE RELATION
, then
INSERT RELATION
of every edge with its fields;
DEFINE FUNCTION
for the stored functions
Full-text
DEFINE ANALYZER
, the
FULLTEXT ... BM25
index definition, and the reviews
VectorThe
HNSW
index definition and every
embedding
array
GeospatialThe points, as part of each record
Time seriesEvery
order_event
, under its ULID key

Two things in that table matter more than they look. First, one read transaction: every model is captured at the same instant, so an order event can never be in the backup while its graph edge is not, even with orders being written during the backup. The app writes both in one transaction, and the export reads both in one transaction. Second, definitions, not just data: the dump carries the analyzer and the index definitions, so a restore rebuilds working search, not just the rows.

The dump also carries the app’s own database user (

DEFINE USER demo ON DATABASE
), so the app can sign in again after a restore without anyone touching it. For the demo database, the dump was about 40 KB on the first backup, and 106 KB after an hour of generated orders.

The bad day

A backup only counts once you have watched it bring something back, so the app has a Simulate incident button. It damages every model in one go, which is roughly what a bad migration, a buggy deploy or a mistaken script against production does:

ModelWhat the incident doesWhat you see in the app
RelationalMultiplies every price by 10Catalogue prices jump; the relational check fails
DocumentDeletes
customer:bob
Bob disappears from the customer list
GraphDeletes Bob’s order edgesFewer lines in the graph; seeded edge count and Alice’s total no longer match
Full-textDrops the
review
table, and its index with it
Review search finds nothing
VectorRemoves the HNSW indexSimilar products returns nothing
GeospatialDeletes the Manchester storeSheffield’s nearest store changes
Time seriesDeletes the seeded order eventsThe time-series check fails

The health panel at the top of the app is what makes this visible in a demo. It runs one check per model against the seeded data, so live traffic never turns a check red; only damage does. Before the incident it shows 7 of 7. After it, 0 of 7.

Two of these are worth pausing on. Dropping a table and removing an index are schema changes, not data changes. A backup that only copied rows would bring the reviews back without their full-text index, and the products back without their vector index, and both searches would still be broken.

After the incident: the health panel shows 7 of 7 checks failing, prices multiplied by ten, and Bob missing from the graph
Seconds after the incident: 7 of 7 checks fail, every price is ten times too high and Bob has gone from the graph.
After the incident: review search reports the review table does not exist, similar products has no index, and Dana's nearest store is now London
Search fails because the table has gone, the vector panel has no index, and Dana’s nearest store is now London, 262 km away. The live feed even shows a “?” where an order lost its customer.

Back to the last known good state

The restore is one Kasten action. I picked the restore point that Kasten had exported to object storage, not the local snapshot, to prove the off-cluster copy is enough on its own. Here is the timeline from one run, with orders being generated throughout:

Time (UTC)What happened
12:42:01Kasten backup: the prehook exports every database, including
demo/shop
(106 KB), in one read transaction each
12:43:30Simulate incident: the health panel drops to 0 of 7
12:43:58Restore starts from the exported restore point
12:46:16Kasten has restored the PVCs and started the pods again, including the app
12:46:36The
restorePosthook
finishes: every database removed and re-imported from the dump taken at 12:42:01
12:46:36Health panel: 7 of 7

From the restore button to a healthy app took 2 minutes 38 seconds, most of it pulling the volumes back from object storage. Every model came back to the same instant, 12:42:01: prices, Bob and his orders, the reviews and their full-text index, the vector index, the Manchester store and the seeded order events. Orders written between the backup and the incident are gone, which is exactly what “last known good state” means.

Nothing was done by hand. There is no step where someone re-creates an index, re-runs a migration or re-defines the app’s database user: they are all in the dump.

After the restore: all 7 checks pass again, prices are back to normal and Bob is back in the graph
After the restore from the exported copy: 7 of 7 again, prices back to normal, Bob back in the graph, and orders flowing.

One thing the demo caught

The first time I ran this with the app inside the cluster, the health panel went green but the live order feed stayed silent. The app had reconnected while the database was still being re-imported, and removing a database in SurrealDB also removes the live queries registered on it, without closing the connection. So the app was connected, healthy and subscribed to nothing.

The fix was small: every five seconds the app checks that its live query is still listed in

INFO FOR TABLE order_event
, and subscribes again if it is not. The lesson is bigger: a restore can be correct for the data and still break a client’s state. Live queries, cached prepared statements and anything else a client holds on the server side deserve a test after a restore, not just the rows.

I ran the whole cycle again with the fix deployed. Five seconds after it reconnected, the app logged

live query ... on order_event no longer exists (database replaced?)
, subscribed again three seconds later, and the live feed was back with all 7 checks green. That restore took 2 minutes 48 seconds.

What this does and does not prove

The demo is honest about its size, and so should this post be.

It provesIt does not prove
One export captures every model in a database: records, edges, schema, analyzers, vector and full-text indexes, functions and database usersThat it scales to your data: the shop’s dump is around 100 KB. Run the same test at real size, with your data types
The copy is consistent while writes continue: each database is read in one transactionThat separate databases are consistent with each other: each one is exported in its own transaction, so keep data that must agree in one database
A restore from the off-cluster copy brings every model back, and the app reconnects by itselfHow long index rebuilds take at scale: on import, SurrealDB rebuilds the HNSW and full-text indexes from the data, so restore time grows with the data, not just the bytes
The restore puts the database back to the backup: changes made after it are gone, which is what you want after a bad changeRecovery to a point between backups: there is no log replay, so the backup schedule is your recovery point

Root- and namespace-level users are not in a database export either. They come back with the SurrealDB data volume, which Kasten restores too, or from the Kubernetes Secret on an empty instance.

Try it yourself, and what’s next

The app is open source at MichaelCade/surrealdb-webapp: a single Go binary using the official SurrealDB Go SDK, plain HTML and JavaScript with no external libraries, and Kubernetes manifests. GitHub Actions publishes the image to

ghcr.io/michaelcade/surrealdb-webapp
. The README has a ten-minute demo script:

  1. Tour the panels with Generate orders on.
  2. Run the Kasten policy while the orders keep flowing.
  3. Click Simulate incident: 7 of 7 becomes 0 of 7.
  4. Restore from the exported restore point.
  5. Watch the health panel return to 7 of 7.

You need SurrealDB, the keeper and the blueprint from the blueprint-ai

surrealdb-example
runbook first; the app adds one setup script and one manifest. To browse the data while you demo, connect the SurrealDB Studio desktop app to the database through its own route.

ComponentVersion tested
Talos / Kubernetes1.12.6 / 1.34.1
Kasten9.0.8
SurrealDB3.3.5, SurrealKV
SurrealDB Go SDK2.0.1

Next up is SurrealDB 4.0, which drops RocksDB and TiKV and keeps SurrealKV as its only on-disk engine. The Go SDK already supports it, so the same app should make a good test of the upgrade.

Sources

]]>
https://vzilla.co.uk/vzilla-blog/one-database-seven-models-protecting-all-of-surrealdb-with-kasten/feed 0
Protecting SurrealDB on Kubernetes with Kasten https://vzilla.co.uk/vzilla-blog/protecting-surrealdb-on-kubernetes-with-kasten https://vzilla.co.uk/vzilla-blog/protecting-surrealdb-on-kubernetes-with-kasten#respond Thu, 19 Nov 2026 13:37:34 +0000 https://vzilla.co.uk/?p=3744 I have been exploring databases a lot more in 2026, and the landscape is busy. There are “new” database vendors everywhere, on Kubernetes and as PaaS or DBaaS offerings. Some build on the shoulders of giants, mainly PostgreSQL and MySQL. Others start from a blank page.

One of the blank-page databases I keep coming back to is SurrealDB. In this post I deploy it on my home-lab Talos Kubernetes cluster, load a small sample dataset, and then protect it with Kasten. The aim is a copy that is consistent at the application level, sent off-cluster to object storage, and proven by restoring it.

Everything here is open source: the blueprint, the runbook and the test data are in a pull request to the blueprint-ai repository under

surrealdb-example
.

What is SurrealDB, and where could it be used?

SurrealDB is a multi-model database written in Rust: one engine for data you would otherwise spread across several databases. According to its README, it combines:

ModelWhat that means in practice
Document and relationalTables that can be schemaless or schemafull, with typed fields and unique indexes
GraphRecords joined by typed, directed edges (
RELATE a->ordered->b
) and queried without JOINs
Key-value, time-series, geospatialNative data types instead of extensions
RetrievalFull-text, vector and hybrid search

You query all of it with SurrealQL, a SQL-like language, over HTTP, WebSocket or GraphQL, with multi-table ACID transactions. It also has row-level permissions, live queries that push changes to the application, stored functions, and an MCP server for AI agents. That combination is what makes it more than “another database”: it can act as the API back end as well as the data store.

The README lists the use cases it is aimed at: applications that mix several data types, a data layer for AI agents, knowledge graphs, real-time apps such as recommendation engines and fraud detection, and embedded or edge workloads.

Where can I run SurrealDB?

Almost anywhere: the same single binary is the server, the CLI and the import/export tool.

WhereHow
ManagedSurrealDB Cloud, where SurrealDB runs the infrastructure, monitoring and backups
Laptop or VMOne binary:
brew install surrealdb/tap/surreal
on macOS, an install script on Linux and Windows
Container
docker run surrealdb/surrealdb:<tag> start
KubernetesThe official Helm chart (
surrealdb/surrealdb
), a single-replica Deployment with an optional PVC
EmbeddedAs a library inside an application, including in the browser

The choice that matters most for protection is the storage engine, set by the start path:

EngineSurrealDB 3.3SurrealDB 4.0
memory
(the Helm chart’s default)
Yes, nothing written to diskYes
surrealkv://
YesYes, the only on-disk engine
rocksdb://
YesNot in the build
tikv://
(distributed)
YesNot in the build

I used SurrealKV on 3.3.5, because it is the engine that carries forward to 4.0. One trap: the Helm chart still defaults to SurrealDB 2.6 and to

memory
, so pin
image.tag
and set
surrealdb.path
, or your “database” disappears with the pod.

The sample application

The sample is a tiny shop of about 1.2 KB, chosen so that a restore which only brought back rows would visibly fail. It lives in namespace

kasten
, database
test
:

ObjectContentWhat a restore must bring back
product
Schemafull table, 5 records,
UNIQUE
index on
name
Schema and index definitions, not just data
customer
Schemaless table, 3 recordsFree-form documents
ordered
5 graph edges:
RELATE customer:alice->ordered->product:1 SET qty = 2
Relations, with their own fields
fn::order_total
Stored function that walks a customer’s
ordered
edges and sums price × quantity
Logic stored in the database

One query checks everything at once, because the totals only come out right if the function, the edges and the products all survived:


RETURN { products: count(SELECT * FROM product), customers: count(SELECT * FROM customer), orders: count(SELECT * FROM ordered), alice_total: fn::order_total(customer:alice), bob_total: fn::order_total(customer:bob), chen_total: fn::order_total(customer:chen), has_index: (INFO FOR TABLE product).indexes.product_name != NONE };

A healthy database returns

alice_total 110, bob_total 160, chen_total 105
with 5 products, 3 customers, 5 orders and the index present. The full seed.surql and
verify.surql
are in the pull request.

To be clear, this is a very basic concept and dataset, built to show the protection workflow rather than a real application. A production database will hold far more data, and probably other data types too: vector embeddings, full-text indexes, time-series and geospatial records, live queries and more users and permissions. The export is a logical SurrealQL dump, so it should carry all of that, but I only tested tables, an index, a function and graph edges. Before you rely on this pattern, repeat the backup-and-restore test with data that looks like yours, at its real size.

Gotchas: running SurrealDB on Talos

None of these stopped the project, but each one cost time, so here they are in one place. The cluster is Talos 1.12.6 with Kubernetes 1.34.1 and Rook Ceph RBD (

ceph-block
) for storage.

GotchaWhat you seeFix
Talos warns on the
restricted
Pod Security profile by default
helm install
prints “would violate PodSecurity restricted:latest”
Add
runAsNonRoot
, a
RuntimeDefault
seccomp profile,
allowPrivilegeEscalation: false
and drop all capabilities in the chart values
The image names its user
nonroot
, not a number
Pod stuck in
CreateContainerConfigError
: “image has non-numeric user (nonroot), cannot verify user is non-root”
Set
runAsUser: 65532
and
runAsGroup: 65532
The production image has no shell
kubectl exec ... sh
fails: it is distroless (Chainguard
glibc-dynamic
)
Use the
-dev
tag of the same version for anything that needs a shell
surreal
is not on the
PATH
in the
-dev
image
sh: surreal: not found
Call
/surreal
The CLI reads piped input one line at a timeA statement written over several lines returns parse errors for each partial lineKeep each statement on one line in files you pipe to
surreal sql
surreal sql
exits 0 when a statement fails
Scripts carry on after an error; the error is a JSON string such as
["The table 'x' does not exist"]
Check the output, not the exit code
surreal --help
prints environment defaults
The root password from
SURREAL_PASS
appears in the help text
Never call
--help
in automation whose output is logged
SurrealDB 3 re-creates the default
main
database as soon as it is removed
REMOVE DATABASE main
then
DEFINE DATABASE main
fails: “already exists”
DEFINE DATABASE IF NOT EXISTS
The Helm chart defaults to 2.6 and the
memory
engine
Data is gone after a pod restartPin
image.tag
and set
surrealdb.path: surrealkv:///data/srdb

What about protecting this database?

SurrealDB gives you two building blocks, and neither is a backup strategy on its own.

OptionWhat you getWhat is missing
surreal export
on its own
A consistent SurrealQL file per databaseScheduling, retention, an off-cluster copy, immutability, and the rest of the application (Secrets, Services, the Deployment itself)
A CSI snapshot of the data PVC on its ownThe whole volume, fastOnly crash-consistent: SurrealDB has no command to pause writes or flush a clean state, so the copy must recover as if the power was cut

The interesting part is the combination, and one fact makes it work:

surreal export
reads a whole database inside one transaction. I checked this in the source, then tested it under load. A client committed transactions that each wrote a matching pair of records into two tables, about 50 per second on top of 20,000 pairs. Five exports ran during the writes, and all five had exactly equal counts in both tables. An export that was not a single snapshot would have more records in the table it exported second.

So the plan is: let SurrealDB produce the consistent copy, put that copy on a PVC, and let Kasten do what it is good at with PVCs. This is pattern 4 of the blueprint-ai method, a dump on a permanent keeper PVC:

  • A small keeper Deployment sits next to SurrealDB. It runs the
    -dev
    image of the same SurrealDB version, so it has the
    surreal
    CLI and a shell, and it permanently mounts its own PVC.
  • Before Kasten snapshots anything, Kasten execs into the keeper. The keeper exports every database over HTTP to its PVC, then runs
    sync
    .
  • Kasten snapshots both PVCs and exports the restore point to object storage.

Why a keeper and not a second volume on the SurrealDB pod? Because the production image has no shell, so nothing could run

sync
inside it, and a CSI snapshot copies the block device, not the page cache. Without
sync
, the dump can land in the snapshot with the right name and zero bytes.

How Kasten orchestrates the whole application

Kasten protects the whole namespace in one policy: the Deployments, Services, the Secret and both PVCs. A Kanister blueprint adds the SurrealDB-specific steps at the right moment.

backup flow · SurrealDB, keeper and Kasten, 5 steps
backup flow · SurrealDB, keeper and Kasten, 5 steps

The arrows run in the order of a backup. Only the keeper writes during the prehook, and its

sync
is the last thing that happens before Kasten snapshots anything.

The blueprint has two actions:

ActionWhen Kasten runs itWhat it does
backupPrehook
Before the PVC snapshotsExports every namespace and database into a staging folder with a SHA-256
MANIFEST
, swaps it in, runs
sync
last, then re-checks every file. An empty export or a bad checksum fails the backup.
restorePosthook
After a restore, once the pods are runningChecks the dump, then removes and re-imports each database (next section)

There is no

backupPosthook
, because nothing was paused, so there is nothing to undo.

A BlueprintBinding attaches the blueprint to any Deployment labelled

blueprint-ai/surrealdb-keeper: "true"
, in any namespace. The binding is on the keeper, not on SurrealDB, because the keeper is the pod the blueprint execs into. A second SurrealDB instance only needs its own keeper; nothing in
kasten-io
changes.

The policy itself is a plain Kasten policy: back up the namespace, then export the restore point to an S3 Location profile. In my lab, the whole backup and export of the small dataset took about a minute.

What does a restore workflow look like?

A restore is one Kasten action with no manual steps before it, whether it goes back in place or into a new namespace. Two copies come back, and only one of them is authoritative:

  1. Kasten scales the workloads down and restores both PVCs from the restore point, from the local snapshot or from object storage.
  2. The pods start again. SurrealDB starts on its crash-consistent data volume, and SurrealKV recovers from it.
  3. Kasten runs the blueprint’s
    restorePosthook
    in the keeper, which:
    1. checks every dump file against the SHA-256 manifest before touching anything, so a damaged dump never costs you the live data;
    2. waits for SurrealDB to answer;
    3. for each database in the dump, runs
      REMOVE DATABASE
      ,
      DEFINE DATABASE IF NOT EXISTS
      , then
      surreal import
      .
  4. Every database is now exactly what the consistent dump held. Records written after the backup do not survive, which is the point of a restore.

The data volume still matters for what a database export does not hold: root- and namespace-level users and access methods. On an empty instance, the root user comes back from the Kubernetes Secret.

To test it, I damaged the data after the backup and restored from the restore point that Kasten had exported to object storage, which proves the off-cluster copy works on its own:

CheckBefore restore (damaged)After restore
alice_total
2058 (kettle price changed to 999)110
bob_total
, orders
0, 3 (Bob deleted, with his 2 edges)160, 5
customer:mallory
Present (added after the backup)Gone
RestoreAction
Complete
; the hook logged
Restoring the dump taken at 14:44:32Z
then
Restore complete

I then deleted everything from the cluster and rebuilt it by running the runbook’s commands exactly as written, and got the same result.

Do we get any storage savings from Kopia?

Yes, but how much depends on where the data changes, not how much of it changes. Every backup writes a full dump, so the dump itself is not incremental. Kasten exports to object storage through Kopia, though, which splits files into content-defined chunks and uploads only chunks it has not seen before.

To measure it, I loaded 30,000 records of about 1 KB each with random content: a 30.8 MiB dump next to a 32.1 MiB SurrealDB data volume. Random content matters, because identical rows deduplicate far better than real data and make the result look much better than it is. I then exported four times and measured how much the repository in the bucket grew each time. Both PVCs are exported, so the figures include the data volume as well as the dump.

RunChange since the previous runRepository growthShare of the first export
130,000 records loaded56.9 MiB (first export)100%
21% of records rewritten, scattered at random29.1 MiB51%
3No change0.4 MiB0.7%
41% of records rewritten, consecutive keys3.6 MiB6.4%

What that means:

  • Unchanged data is almost free. An unchanged database produces a byte-identical dump, and SurrealKV did not rewrite its files while idle.
  • Scattered changes cost close to a full upload.
    surreal export
    writes records in key order, in
    INSERT
    statements of 1,000 records. 300 random updates put about 10 changes in every statement, so every Kopia chunk of the file was new.
  • The same 300 changes on consecutive keys touched one region of the file, and most chunks were reused.

For planning: a workload that mostly appends records with increasing keys (events, logs, time series) should export cheaply. A workload that updates rows across a large table should be budgeted as close to a full dump per backup. SurrealDB’s default record IDs are random, so new records behave like the scattered case unless the application uses ordered IDs such as

ulid()
. I worked that last point out from runs 2 and 4 rather than measuring it separately.

Wrapping up, and what’s next

SurrealDB has no write pause, but it does have something nearly as useful: an export that reads each database as one consistent snapshot. Put that export on a keeper PVC, flush it, and Kasten turns it into scheduled, off-cluster, restorable protection for the whole application, with no write pause and a restore that needs no manual steps.

ComponentVersion tested
Talos / Kubernetes1.12.6 / 1.34.1
Kasten9.0.8
SurrealDB (Helm chart 0.5.0)3.3.5, SurrealKV
StorageRook Ceph RBD

The blueprint is an

-example
on purpose: it encodes my versions, storage class and namespace layout. Copy it, adapt it, and run your own backup-and-restore test before you trust it. It is in review as kastenhq/blueprint-ai#21, with the runbook, the test data and a concept deck.

Next up:

  • SurrealDB 4.0, released in November 2026. It keeps the same export and import CLI but drops RocksDB and TiKV, leaving SurrealKV as the only on-disk engine. Because 3.3 already runs on SurrealKV, I expect mostly an image-tag change, but that is a test to run, not a promise.
  • Talos is still on my technology watch list, and this whole project ran on my Talos home-lab cluster. If you want to hear more about it, I have covered Talos in earlier posts on this blog.

Sources

]]>
https://vzilla.co.uk/vzilla-blog/protecting-surrealdb-on-kubernetes-with-kasten/feed 0