backup flow · SurrealDB, keeper and Kasten, 5 steps

I have been exploring databases a lot more in 2026, and the landscape is busy. There are “new” database vendors everywhere, on Kubernetes and as PaaS or DBaaS offerings. Some build on the shoulders of giants, mainly PostgreSQL and MySQL. Others start from a blank page.

One of the blank-page databases I keep coming back to is SurrealDB. In this post I deploy it on my home-lab Talos Kubernetes cluster, load a small sample dataset, and then protect it with Kasten. The aim is a copy that is consistent at the application level, sent off-cluster to object storage, and proven by restoring it.

Everything here is open source: the blueprint, the runbook and the test data are in a pull request to the blueprint-ai repository under

surrealdb-example
.

What is SurrealDB, and where could it be used?

SurrealDB is a multi-model database written in Rust: one engine for data you would otherwise spread across several databases. According to its README, it combines:

ModelWhat that means in practice
Document and relationalTables that can be schemaless or schemafull, with typed fields and unique indexes
GraphRecords joined by typed, directed edges (
RELATE a->ordered->b
) and queried without JOINs
Key-value, time-series, geospatialNative data types instead of extensions
RetrievalFull-text, vector and hybrid search

You query all of it with SurrealQL, a SQL-like language, over HTTP, WebSocket or GraphQL, with multi-table ACID transactions. It also has row-level permissions, live queries that push changes to the application, stored functions, and an MCP server for AI agents. That combination is what makes it more than “another database”: it can act as the API back end as well as the data store.

The README lists the use cases it is aimed at: applications that mix several data types, a data layer for AI agents, knowledge graphs, real-time apps such as recommendation engines and fraud detection, and embedded or edge workloads.

Where can I run SurrealDB?

Almost anywhere: the same single binary is the server, the CLI and the import/export tool.

WhereHow
ManagedSurrealDB Cloud, where SurrealDB runs the infrastructure, monitoring and backups
Laptop or VMOne binary:
brew install surrealdb/tap/surreal
on macOS, an install script on Linux and Windows
Container
docker run surrealdb/surrealdb:<tag> start
KubernetesThe official Helm chart (
surrealdb/surrealdb
), a single-replica Deployment with an optional PVC
EmbeddedAs a library inside an application, including in the browser

The choice that matters most for protection is the storage engine, set by the start path:

EngineSurrealDB 3.3SurrealDB 4.0
memory
(the Helm chart’s default)
Yes, nothing written to diskYes
surrealkv://
YesYes, the only on-disk engine
rocksdb://
YesNot in the build
tikv://
(distributed)
YesNot in the build

I used SurrealKV on 3.3.5, because it is the engine that carries forward to 4.0. One trap: the Helm chart still defaults to SurrealDB 2.6 and to

memory
, so pin
image.tag
and set
surrealdb.path
, or your “database” disappears with the pod.

The sample application

The sample is a tiny shop of about 1.2 KB, chosen so that a restore which only brought back rows would visibly fail. It lives in namespace

kasten
, database
test
:

ObjectContentWhat a restore must bring back
product
Schemafull table, 5 records,
UNIQUE
index on
name
Schema and index definitions, not just data
customer
Schemaless table, 3 recordsFree-form documents
ordered
5 graph edges:
RELATE customer:alice->ordered->product:1 SET qty = 2
Relations, with their own fields
fn::order_total
Stored function that walks a customer’s
ordered
edges and sums price × quantity
Logic stored in the database

One query checks everything at once, because the totals only come out right if the function, the edges and the products all survived:


RETURN { products: count(SELECT * FROM product), customers: count(SELECT * FROM customer), orders: count(SELECT * FROM ordered), alice_total: fn::order_total(customer:alice), bob_total: fn::order_total(customer:bob), chen_total: fn::order_total(customer:chen), has_index: (INFO FOR TABLE product).indexes.product_name != NONE };

A healthy database returns

alice_total 110, bob_total 160, chen_total 105
with 5 products, 3 customers, 5 orders and the index present. The full seed.surql and
verify.surql
are in the pull request.

To be clear, this is a very basic concept and dataset, built to show the protection workflow rather than a real application. A production database will hold far more data, and probably other data types too: vector embeddings, full-text indexes, time-series and geospatial records, live queries and more users and permissions. The export is a logical SurrealQL dump, so it should carry all of that, but I only tested tables, an index, a function and graph edges. Before you rely on this pattern, repeat the backup-and-restore test with data that looks like yours, at its real size.

Gotchas: running SurrealDB on Talos

None of these stopped the project, but each one cost time, so here they are in one place. The cluster is Talos 1.12.6 with Kubernetes 1.34.1 and Rook Ceph RBD (

ceph-block
) for storage.

GotchaWhat you seeFix
Talos warns on the
restricted
Pod Security profile by default
helm install
prints “would violate PodSecurity restricted:latest”
Add
runAsNonRoot
, a
RuntimeDefault
seccomp profile,
allowPrivilegeEscalation: false
and drop all capabilities in the chart values
The image names its user
nonroot
, not a number
Pod stuck in
CreateContainerConfigError
: “image has non-numeric user (nonroot), cannot verify user is non-root”
Set
runAsUser: 65532
and
runAsGroup: 65532
The production image has no shell
kubectl exec ... sh
fails: it is distroless (Chainguard
glibc-dynamic
)
Use the
-dev
tag of the same version for anything that needs a shell
surreal
is not on the
PATH
in the
-dev
image
sh: surreal: not found
Call
/surreal
The CLI reads piped input one line at a timeA statement written over several lines returns parse errors for each partial lineKeep each statement on one line in files you pipe to
surreal sql
surreal sql
exits 0 when a statement fails
Scripts carry on after an error; the error is a JSON string such as
["The table 'x' does not exist"]
Check the output, not the exit code
surreal --help
prints environment defaults
The root password from
SURREAL_PASS
appears in the help text
Never call
--help
in automation whose output is logged
SurrealDB 3 re-creates the default
main
database as soon as it is removed
REMOVE DATABASE main
then
DEFINE DATABASE main
fails: “already exists”
DEFINE DATABASE IF NOT EXISTS
The Helm chart defaults to 2.6 and the
memory
engine
Data is gone after a pod restartPin
image.tag
and set
surrealdb.path: surrealkv:///data/srdb

What about protecting this database?

SurrealDB gives you two building blocks, and neither is a backup strategy on its own.

OptionWhat you getWhat is missing
surreal export
on its own
A consistent SurrealQL file per databaseScheduling, retention, an off-cluster copy, immutability, and the rest of the application (Secrets, Services, the Deployment itself)
A CSI snapshot of the data PVC on its ownThe whole volume, fastOnly crash-consistent: SurrealDB has no command to pause writes or flush a clean state, so the copy must recover as if the power was cut

The interesting part is the combination, and one fact makes it work:

surreal export
reads a whole database inside one transaction. I checked this in the source, then tested it under load. A client committed transactions that each wrote a matching pair of records into two tables, about 50 per second on top of 20,000 pairs. Five exports ran during the writes, and all five had exactly equal counts in both tables. An export that was not a single snapshot would have more records in the table it exported second.

So the plan is: let SurrealDB produce the consistent copy, put that copy on a PVC, and let Kasten do what it is good at with PVCs. This is pattern 4 of the blueprint-ai method, a dump on a permanent keeper PVC:

  • A small keeper Deployment sits next to SurrealDB. It runs the
    -dev
    image of the same SurrealDB version, so it has the
    surreal
    CLI and a shell, and it permanently mounts its own PVC.
  • Before Kasten snapshots anything, Kasten execs into the keeper. The keeper exports every database over HTTP to its PVC, then runs
    sync
    .
  • Kasten snapshots both PVCs and exports the restore point to object storage.

Why a keeper and not a second volume on the SurrealDB pod? Because the production image has no shell, so nothing could run

sync
inside it, and a CSI snapshot copies the block device, not the page cache. Without
sync
, the dump can land in the snapshot with the right name and zero bytes.

How Kasten orchestrates the whole application

Kasten protects the whole namespace in one policy: the Deployments, Services, the Secret and both PVCs. A Kanister blueprint adds the SurrealDB-specific steps at the right moment.

backup flow · SurrealDB, keeper and Kasten, 5 steps
backup flow · SurrealDB, keeper and Kasten, 5 steps

The arrows run in the order of a backup. Only the keeper writes during the prehook, and its

sync
is the last thing that happens before Kasten snapshots anything.

The blueprint has two actions:

ActionWhen Kasten runs itWhat it does
backupPrehook
Before the PVC snapshotsExports every namespace and database into a staging folder with a SHA-256
MANIFEST
, swaps it in, runs
sync
last, then re-checks every file. An empty export or a bad checksum fails the backup.
restorePosthook
After a restore, once the pods are runningChecks the dump, then removes and re-imports each database (next section)

There is no

backupPosthook
, because nothing was paused, so there is nothing to undo.

A BlueprintBinding attaches the blueprint to any Deployment labelled

blueprint-ai/surrealdb-keeper: "true"
, in any namespace. The binding is on the keeper, not on SurrealDB, because the keeper is the pod the blueprint execs into. A second SurrealDB instance only needs its own keeper; nothing in
kasten-io
changes.

The policy itself is a plain Kasten policy: back up the namespace, then export the restore point to an S3 Location profile. In my lab, the whole backup and export of the small dataset took about a minute.

What does a restore workflow look like?

A restore is one Kasten action with no manual steps before it, whether it goes back in place or into a new namespace. Two copies come back, and only one of them is authoritative:

  1. Kasten scales the workloads down and restores both PVCs from the restore point, from the local snapshot or from object storage.
  2. The pods start again. SurrealDB starts on its crash-consistent data volume, and SurrealKV recovers from it.
  3. Kasten runs the blueprint’s
    restorePosthook
    in the keeper, which:
    1. checks every dump file against the SHA-256 manifest before touching anything, so a damaged dump never costs you the live data;
    2. waits for SurrealDB to answer;
    3. for each database in the dump, runs
      REMOVE DATABASE
      ,
      DEFINE DATABASE IF NOT EXISTS
      , then
      surreal import
      .
  4. Every database is now exactly what the consistent dump held. Records written after the backup do not survive, which is the point of a restore.

The data volume still matters for what a database export does not hold: root- and namespace-level users and access methods. On an empty instance, the root user comes back from the Kubernetes Secret.

To test it, I damaged the data after the backup and restored from the restore point that Kasten had exported to object storage, which proves the off-cluster copy works on its own:

CheckBefore restore (damaged)After restore
alice_total
2058 (kettle price changed to 999)110
bob_total
, orders
0, 3 (Bob deleted, with his 2 edges)160, 5
customer:mallory
Present (added after the backup)Gone
RestoreAction
Complete
; the hook logged
Restoring the dump taken at 14:44:32Z
then
Restore complete

I then deleted everything from the cluster and rebuilt it by running the runbook’s commands exactly as written, and got the same result.

Do we get any storage savings from Kopia?

Yes, but how much depends on where the data changes, not how much of it changes. Every backup writes a full dump, so the dump itself is not incremental. Kasten exports to object storage through Kopia, though, which splits files into content-defined chunks and uploads only chunks it has not seen before.

To measure it, I loaded 30,000 records of about 1 KB each with random content: a 30.8 MiB dump next to a 32.1 MiB SurrealDB data volume. Random content matters, because identical rows deduplicate far better than real data and make the result look much better than it is. I then exported four times and measured how much the repository in the bucket grew each time. Both PVCs are exported, so the figures include the data volume as well as the dump.

RunChange since the previous runRepository growthShare of the first export
130,000 records loaded56.9 MiB (first export)100%
21% of records rewritten, scattered at random29.1 MiB51%
3No change0.4 MiB0.7%
41% of records rewritten, consecutive keys3.6 MiB6.4%

What that means:

  • Unchanged data is almost free. An unchanged database produces a byte-identical dump, and SurrealKV did not rewrite its files while idle.
  • Scattered changes cost close to a full upload.
    surreal export
    writes records in key order, in
    INSERT
    statements of 1,000 records. 300 random updates put about 10 changes in every statement, so every Kopia chunk of the file was new.
  • The same 300 changes on consecutive keys touched one region of the file, and most chunks were reused.

For planning: a workload that mostly appends records with increasing keys (events, logs, time series) should export cheaply. A workload that updates rows across a large table should be budgeted as close to a full dump per backup. SurrealDB’s default record IDs are random, so new records behave like the scattered case unless the application uses ordered IDs such as

ulid()
. I worked that last point out from runs 2 and 4 rather than measuring it separately.

Wrapping up, and what’s next

SurrealDB has no write pause, but it does have something nearly as useful: an export that reads each database as one consistent snapshot. Put that export on a keeper PVC, flush it, and Kasten turns it into scheduled, off-cluster, restorable protection for the whole application, with no write pause and a restore that needs no manual steps.

ComponentVersion tested
Talos / Kubernetes1.12.6 / 1.34.1
Kasten9.0.8
SurrealDB (Helm chart 0.5.0)3.3.5, SurrealKV
StorageRook Ceph RBD

The blueprint is an

-example
on purpose: it encodes my versions, storage class and namespace layout. Copy it, adapt it, and run your own backup-and-restore test before you trust it. It is in review as kastenhq/blueprint-ai#21, with the runbook, the test data and a concept deck.

Next up:

  • SurrealDB 4.0, released in November 2026. It keeps the same export and import CLI but drops RocksDB and TiKV, leaving SurrealKV as the only on-disk engine. Because 3.3 already runs on SurrealKV, I expect mostly an image-tag change, but that is a test to run, not a promise.
  • Talos is still on my technology watch list, and this whole project ran on my Talos home-lab cluster. If you want to hear more about it, I have covered Talos in earlier posts on this blog.

Sources

Leave a Reply

Your email address will not be published. Required fields are marked *