A hurdle—though we don’t let hurdles get in the way of a good story and use case—is support. The Veeam Agent for Linux officially supports enterprise and LTS distributions like RHEL, Rocky, Alma, Ubuntu, Debian, SLES, and Fedora. Arch Linux isn’t on that official matrix, but spoiler alert: I have it protecting my rolling-release desktop just fine.

Gotchas

Support aside, a couple of things can trip up backup jobs on fast-moving distros. Whenever Manjaro or Omarchy pushes a major kernel update, Veeam requires the matching linuxXX-headers package installed concurrently so the veeamblksnap DKMS module can recompile successfully. Omarchy runs fast-moving Arch kernels, meaning if your headers fall behind the kernel version, the out-of-tree block snapshot driver will fail to build.

Package Management

Omarchy uses the standard Arch User Repository (AUR). While DHH and the Omarchy team have discussed building custom repository efforts, right now we rely on standard Arch tooling to pull down packages.

Getting Started

For testing purposes, I set this up inside a VM using VirtualBox on my Manjaro machine—though wiping the bare-metal desktop might be the next adventure. Running omarchy update from the foot terminal gives you a clean baseline to start from. From that terminal I ran

omarchy update

Step 1 – Check system and kernel headers are up to date

The goal here is to get Veeam Agent for Linux installed and configured to backup and recover any data that goes bad. Out of the box at least at the time of writing this some of the packages we need are either not included.

As you can see we are missing some of the packages we need here. We can get them installed with the following command.

sudo pacman -Syu --needed linux-headers dkms

Lets just check we get a better output from the first command

Step 2 – Install Veeam Agent for Linux

The foundation is set with the above and we have the required packages, now we are ready to get the Veeam Agent deployed so that we can start taking some of those sweet backups.

yay -S veeam

There are a couple of prompts throughout the installation that we have to hit y on. But also take note of what other packages are being installed here. I actually don’t know if those initial packages would have been deployed here, not had chance to just go straight in with a Veeam installation.

Continued, below you will see that the veeamblksnap package is being installed. You can read up on this here

And the final one in the sequence. We finish off with the agent installation

Step 3 – Veeam Agent TUI & Configure

Installed, Job Done. Nope a little bit of configuration needs to take place. There is a TUI and a CLI and will cover both as we move through the rest of this post. Running the following command the first time after install will open a configuration TUI.

sudo veeam

Have fun reading the EULA and accept the terms. The following screen is if you want to create a bare metal recovery ISO, I skipped this step for now.

Next up we have the licensing, “But you said this was free” it is select workstation and Finish. The community edition of Veeam enables you to run this with minimum support, and as Arch Linux is not documented in the support documentation we are good to go.

To wrap up the configuration, we get a nice message saying we are finished.

We can now move onto the Configure, selecting the c key will put us through another wizard. Here we are going to create a backup job. So give the job a name, I think I left this as the default. If you have multiple machines you might want to be a little more specific.

I have to say this is where that green colour, was really quite hard to read. Now we can choose what sort of a backup do we want to perform. The top option is “Entire Machine (Recommened)” But you can see if you know you only want to protect a specific volume, file and folder structure then you can choose those, I have not tested them at all.

Next up, where do we want to send these backups, we have a few options and the commercial product of the agent has a few more. But for my backups I want to send them to a simple NFS share (note on this later)

For the NFS or SMB option for a shared folder, provide your configuration, this is where you can also define your retention.

Our next mission is to define our backup schedule. This will be a full backup on day 1 and then incremental backups after this. (Only the changes since the last backup)

Finally, a summary of what we configured for our first backup job.

I selected “Start Job now” so as expected the job started.

You can monitor the backup job if you have nothing better to do at this point

And at some point the job will complete, hopefully successfully like mine did

Step 3a – Sorry if you went through the above and hit the same wall

As I mentioned we can also manage this backup via a CLI as well, I will touch on that shortly. First I need to add a little gotcha that I stumbled across purely because I was wanting to use NFS. Omarchy doesn’t come with an NFS client out of the box.

So I had to exit out of the TUI and back into the shell, hopefully you are smarter than me and are able to either just open another terminal and run the following to get that NFS client on the system.

sudo pacman -S nfs-utils

Then we go back into the TUI if you did have to leave and run through that Step 3 process again.

Step 4 – From the CLI

I kept teasing you with talk of a CLI, to access the CLI for the agent you can use the following command

veeamconfig

I won’t go through everything you can do but the usual help flag will get you started, but for example if you now wanted to trigger your next incremental backup or maybe you didn’t hit that cross on start backup now you could run.

veeamconfig job start --name "BackupJob1"

Pro-Tip for Rolling Distros: Because Arch environments change rapidly, subvolume block layouts can occasionally cause incremental change-tracking metadata (veeamblksnap) to throw an ENOSPC or block translation error if extents shift. If your incremental run ever trips up, forcing a single active full baseline (veeamconfig job start –name “BackupJob1” –activefull) resets the change-tracking index cleanly and keeps your subsequent incremental chain fast and reliable.

Step 5 – Testing some restore scenarios

Ok, great we have a backup but what you really want is the backup. Things will go wrong regardless of the system you are running on so we need to be able to do some sort of restore. I am going to show some very simple file level recovery options via the TUI and the CLI.

I have created a mission critical file as you can see,

Sneakily I created this before we even started the Veeam backup so I know that our backup has this file in. We need that back!

And surely its as simple as going into the TUI or CLI, mounting that backup and copying the file from one place to the other… Well not really.

But we need that file back so we had to do some hacking. I confirmed the version of FUSE that we get out of the box with Omarchy is newer than the one Veeam is looking for.

But that did not stop me! I was determined to get that file back not just for me, but for you all as well. You will see from the short video below I went on a wild ride of compiling the old FUSE version for us to use for our restores.

First we needed some more packages

Second we needed to clone the repo for fuse.

Then it was build and compile time.

There is a step in the above video where I have to edit a specific veeam.ini file

sudo nano /etc/veeam/veeam.ini

In this file we have to make a change to point to this specific fuse option.

And this libfuse3.so.3 also in the video above is what we built and then moved to that location.

# Path to fuse3 library (for file-level restores) 

fuse3Path = /opt/veeam/transport/libfuse3.so.3

I think that covers everything for now, way longer than I anticipated but wanted to share.

The Year of the Linux Desktop might still be a running joke in some circles, but when you can comfortably snapshot, protect, and recover a bleeding-edge Arch-based desktop environment using enterprise-grade tooling, it feels a lot closer than ever.

Today, this challenge is amplified even more by AI. We live in an era where an LLM can spin up a Docker compose file, write a complex SQL query, or draft an entire CI/CD pipeline in seconds. It’s tempting to use AI to bypass the friction entirely. But if the AI does all the thinking, you get the illusion of progress without the muscle memory. The trick isn’t using AI to avoid learning—it’s using AI as an interactive co-pilot to explain error logs, challenge your assumptions, and accelerate your hands-on loops. Be smart with AI not addicted! Use it as a tool.

Start with Clear Goals (Treat Tech as a Tool, Not a Collection)

When you first look at a massive landscape like DevOps, Data Engineering, or Cloud-Native architecture, the sheer volume of tools can cause instant paralysis.

• Define the Scope: Break the massive umbrella term down into discrete domains (e.g., version control, CI/CD, infrastructure as code, monitoring).
• Audit Your Baseline: Honestly map out what you already know. Don’t start from absolute zero if you already understand networking or basic scripting.
• Anchor Your “Why”: Are you trying to pivot careers, solve a specific bottleneck in your current role, or just scratch a technical itch? Your motivation dictates the depth required.
• Write the Roadmap: Plan your milestones before you touch a keyboard so you have a finish line in sight.

Theory vs. Application (The 80/20 Hands-On Rule)

We have all accumulated digital graveyards of half-finished Udemy courses and O’Reilly books. Passive reading feels like progress, but it’s an illusion.

• Resist Tutorial Hell: Limit the time spent watching videos or reading docs to the absolute minimum needed to understand why something works.
• Embrace the Friction: No cheat codes or blindly copy-pasting GitHub snippets. Type out the code, break the config files, and stare at the error logs—that is where the real learning happens.
• Find Your Medium: Know how you best absorb foundational concepts (whether it’s technical documentation, video walk-throughs, or audiobooks on a commute), but always pivot quickly back to the terminal.

Solve Real, Relatable Problems

Abstract tutorials are boring, and they don’t stick. The best way to learn a technology is to force it to solve a problem you actually care about.

• Build a Home Lab: Spin up a mini rack, deploy a Kubernetes cluster, or automate a painful manual chore around your house or workflow. I have written a lot about homelabs here, they don’t have to be expensive, especially today where it seems components are around 10x more than they were a year ago! A laptop or even a VM on your machine is something to play with.
• Scratch Your Own Itch: Build something useful—even if a commercial tool already exists. The act of engineering the solution teaches you the underlying mechanics. I have tons of ideas that pop into my head, they are not always niche ideas but they are great to just build a prototype, understand what is happening and adapt, improve or delete it and move on.
• Make It Interesting: If you are bored by the sample project (like yet another generic to-do app), change the domain to something you love, whether that’s tracking video game stats or analysing sports data. I am currently learning lots about data engineering and using my cricket team stats and data to help keep me interested.

Incremental Learning & Feedback Loops

Learning isn’t a linear climb; it’s a series of iterations. Once you get something working, the real question is: Could it be better?

• Iterative Refactoring: Get the MVP working first. Once it’s functional, go back and optimize it for security, efficiency, or scalability.
• Embrace the Error Log: Treat error messages not as failures, but as your primary feedback mechanism. Debugging teaches you more in ten minutes than a smooth tutorial does in ten hours. “Learn from your mistakes”, why did this fail?
• Build in Public: Share your code, document your journey, or write about what broke and how you fixed it. Explaining it to others forces absolute clarity. I appreciate not everyone is a blogger or youtuber but it’s a great way to build accountability, during 90daysofdevops I was tweeting updates every single day and I know people were getting annoyed “Just let us know if you complete it and will go back and read it then” but me putting 10/90 out there was my own accountability to finish that long list and learn.

Consistent and Correct Action

Occasional weekend binges won’t build durable neural pathways. Technology moves too fast for sporadic cramming. I can sit on YouTube all day watching videos about relatable technical videos that are associated with the plan, but sometimes that is just background noise and its not solid learning time. You have to have that right mindset to be in learning mode.

• Compound Your Efforts: Thirty minutes of focused, hands-on practice every single day beats a five-hour marathon session once a week.
• Practice the Right Habits: “Correct” action means not taking shortcuts early on. Learn the proper syntax, understand security best practices, and use version control from day one so you don’t have to unlearn bad habits later.

I am no learning oracle by any stretch, I was terrible in school. I have just been able to pick up topics I am interested in, I am equally in a privileged world where I have the time to spend time and learn new areas without the forcing function of finding a job or being desperate for that move into this new area.

As with all the posts, sometimes I write stuff down for me to go back to and remember the why. But as always if this helps one person get a plan together and execute on that then it is worth posting on this little corner of the internet for all the AI monsters to find.

You can find out what the Veeam Application Backup Repository is here below.

As I said I wanted a demo app that would enable a few things.

• Multi-Database Support
• Scheduling
• Web Based UI
• Some sort of connection testing
• Visual Monitoring and Observability
• Container based, run anywhere

I have seen a rise in the amount of people running databases and other data services in Docker Containers, not going all in on Kubernetes and simply using a Docker container to run their data services, I have also seen a strong growth area around DBaaS, including the public cloud but also the offerings from Supabase, PlanetScale, MongoDB Atlas to name a few.

I have said about these platforms and the potential for backups to be stored within the Veeam Application Backup Repository, so combining the two I went to work and created this demo tool.

Deployment

It is deployed as 2 Docker containers using Docker Compose, it has the following architecture.

• Backend: Golang with Chi router, GORM, and SQLite
• Frontend: Next.js 16 with React 19, Material UI, and TypeScript
• Database: SQLite for metadata storage
• Scheduler: Cron-based job scheduling

Configuration

When up and running you navigate a browser to http://localhost:3000

The first thing to do is to create your sources.

You then select your source type, provide a name for the connection and fill int he details for host, port, database name, username and password. You will also choose the NFS Backup Target (this would be one of those NFS Exports, backed by the Veeam Application Backup Repository)

Once added you can use the backup now button to take a backup out of schedule as a point in time. If you wanted to you could then select the Schedules tab and start creating the cronjob schedules for each source.

When you click the create schedule you will see the following window

From the source drop down you can select the source you just added, you can see I have been running a lot of tests here against many different databases.

On my machine you can see all the Application Backup Repositories I have added.

This is where for example your PGdump files etc will land and then be protected by the Veeam schedule that we setup during the ABR being added.

Restore

Generally speaking the PGDump files that live on that export location can be recovered natively within Postgres providing they are within the retention that is being stored there, if something happens to these or the retention has aged out then you might want to recover them back. This tool is focused on restoring the dump files back to the database.

You can select that restore button and perform the restore, you select the specific restore point (dump file) that you want to restore back to the database. Choose the source and target database, in the below image we took it from the same source and target. \

As supabase is our example here, we know that this is mostly postgres under the hood so I wanted a way to also be able to restore this to other locations where possible. So I made it possible to restore this backup file to say a postgres instance running in a local docker container.

You will hopefully see that many of the other options are greyed out as they are not compatible.

To note again this is a demo application I created to show how we could use the Veeam Application Backup Repository as a place to manage additional retention of those dump files rather than relying on them just landing on an NFS export or even worse a local disk of the database server.

I was not able to find a commercial equivalent or an open source project that provided the same breadth of database and data service support but open to any feedback on that. I have not released the code but you will have to let me know if this is something that people would want to see and use in their home labs.

For the record, it is not a Veeam tool I am just leveraging the Veeam ABR to store those exports from the database and then using that to capture snapshots and then backup copies of those exports. In theory though this will work with any NFS export location.

I posted about the Application Backup Repository

We start by creating a fresh new Application Backup Repository for our LXC backups within Veeam Backup & Replication.

Now we head over to our Proxmox environment to add the new NFS Export. Select the datacenter and then Storage

Click add and select NFS

Give it a name,

The server will be your ABR Server

The Export will be the path

Select VZdump backup file and unselect anything else,

Also make sure you select NFS Version

We will now configure Proxmox backup to send to our Application Backup Repository. Firstly select your LXC container and choose Backup.

Select the storage to be our newly added NFS Export

This can then be scheduled via Proxmox backup and you will see the backups that you have created in the following screen.

Amazingly, this dump will be found in our export location. This will then be captured by the snapshot schedule we created when adding the Application Backup Repository in Veeam Backup & Replication. You can then use the Backup Copy feature to send this to another location.

I have tested recovery as well; to recover you will need to shut down your container and then restore the contents.

Brad put together this post on the themes around 13.1 and targeted areas to continue to enable our customers to protect their ever changing estates.

https://www.veeam.com/blog/veeam-data-platform-v13-1-identity-cyber-ai.html

Anthony got into some of this but has a focus around the hypervisor support and continued coverage we continue to strive to help our customers protect their virtual machines.

I wanted to take one of those features that I actually believe could be a pretty big deal for those looking after the data across a business.. Anthony mentioned the Application Backup Repository in his post,

Application Backup Repository, which takes the Veeam Infrastructure Appliance running in Hardened Repository mode and turns local storage into a pool of virtual volumes, each one exposed as its own NFS target. Applications write to it directly with no awareness of Veeam underneath, while Veeam handles snapshots, immutability and access control behind the scenes. Oracle RMAN Incremental Merge is the flagship use case, but the door is open to anything from IoT devices to standalone databases that can export or dump data to NFS.

When we were first briefed on this feature, I was thinking of all the use cases that this could really help our customers. In this post I want to cover from the Veeam side, how do we get an Application Backup Repository setup and then touch on what I think could be a massive use case for this feature.

Getting Started with the Application Backup Repository (ABR)

To note, most of the screenshots I share are from a beta / technical preview and not the GA code so might differ slightly but the standard wizard driven approach won’t be far from the reality in the GA code. As Anthony states this is an extenstion and use case that leverages the Veeam Infrastructure Appliance, this is a packaged up ISO or OVA that enables you to simply deploy to your environment and then add to your overall Veeam Backup & Replication environment.

What is the Application Backup Repository?

• A new repository type in VBR – ’Smart’ NFS Share:
• If you export data to it, it is protected.
• Veeam Schedules Snapshots and Backup Copy
• Instant Recovery & Rollback: Full snapshot revert or Granular recovery

Why?

Veeam have been great with bringing out Enterprise Application plugins and providing application consistency and bridging the gap between application and database owners with making sure that backup admins have a grasp on what is being protected vs App owners going rogue and just storing large backup files where they wish, sometimes not storing in great places… we all saw the agentic situation that deleted the database and its backups… dont store backups on the same machine… bad idea!

This feature provides a catch all for those applications where there is not yet the enterprise application integration with Veeam. It was also the basis of large database incrementals for Oracle. But the oppurtunity here is the platforms that are not natively supported by Veeam yet today as well. Think about the databases running on a docker host in a container, Database as a Service offerings like Supabase, PlanetScale, MongoDB and this list is getting longer and longer as well as anything that exports a configuration such as a switch or other network devices.

How?

The quick how is, deployed as a Veeam Infrastructure Appliance that enables an NFS export. •Applications write native exports/dumps directly to the NFS share on their own schedule (cron, DBaaS export, Proxmox LXC backup, network device config export, etc.)​

Veeam then takes scheduled snapshots, which can then be backed up to other storage types for longer term retentions and offsite copies.

​Restore wise, this offers the ability to restore from snapshot, ‘Revert Snapshot’ (full rollback of the NFS export) or ‘Export Snapshot’ (mount as temp read-only path for granular restore)

​This VIA leverages the immutability flag meaning that we have an immutable copy stored as well as being able to backup copy to a secondary location.

We should consider one application per repository as the snapshot revert will apply to the entire dataset.

To get started we are going to deploy our new Veeam Infrastructure Appliance with our required disk layout to store our backups, in my case I am doing this on a VM but this could be a physical storage server as well. When we have that new Veeam server added to our managed servers we can go and create our new Application Backup Repository. Give it an appropriate name and click next.

Because we have added the VIA as a managed server we will see this in the drop down, select this but if you have not yet added the server you could select “Add New” and run through that process.

We then need to define our repository, we will grant permissions for the NFS share.

We can then define our snapshot schedule.

And now on your client, whatever it is you want to send to the repository you would mount this NFS export and start sending those exports or dumps to it. Within Veeam Backup & Replication you will see the snapshots looking something like this based on the schedule you set.

We should also talk about recovery options, this is where my images might differ from the GA but the same concept will be available. When you right click on your snapshots in the above screen you will have a recovery option.

When the “Instant Recovery” is clicked a  new window appears and you can choose your available restore point.

There are two restore modes available,

Revert the snapshot is going to rip and replace the contents of the NFS Export

Export snapshot via a temporary path is going to enable you to mount the snapshot as a new export so you can go and pick what you want and possibly copy and paste.

 If you chose the revert snapshot option then you will need to unmount and remount the nfs export.

We can then configure backup copy options to send our backups to additional storage types, offsite locations and different retentions. In a simple way you could give this export to any of your app owners and this is where they can start dumping their data.

“Anything” Backup Repository…

Now you have seen how simple it is to create, there are so many exciting and useful scenarios we could use this for, the first idea I had was a database that lives on a platform without Veeam native support, maybe this is a containerised version of MySQL.

The concept of this repository is that you can send anything that Veeam doesn’t support natively into this and be sure that backup admins have visibility of these backups.

This is focused on MySQL but in theory any database living anywhere could be an option here.

I have been working on a small demo tool that allows me to schedule these database dumps from many different database sources including many of the logos you see above, you could think about this as a centralised database export scheduler.

Maybe worth another post down the line on this as well as some of the other use cases I have been thinking about like Proxmox LXC containers exports to this NFS export.

Many ideas, the really exciting part of this feature for me is finding what the community will use this for.

Today we started the plan.

We started this course linked above, it had a bit of everything I was looking for the foundations. We are around 2 hours in and I decided that might be a good time to look back over the notes and look at the additional areas that have been mentioned already that we need to get into. I am not going to reinvent the wheel here, If you are interested then get into this first. I would say you can take the first hour and learn something.

• 00:00 Welcome
• 01:20 Introduction
• 13:14 Storing Data
• 24:17 Processing Data Sources
• 31:33 Orchestration
• 38:03 Stream Processing
• 45:22 AI and ML with Data Engineering
• 53:42 Serving Data
• 1:00:47 Could and Data Engineering

The next hour was more talking about technologies that would be needed with some overview, In the above hour you will hear many tooling options as well so I will start there. My plan here is to try and find some useful, focused content that I can watch through to learn more of the fundamentals of these tools.

PSA: The Apache Software Foundation (ASF)

Many of the tools listed below are fronted with “Apache”, I had obviously heard of many of these Apache titled bits of software but I was maybe a little naive to the background. The Apache Software Foundation (ASF) actually manages over 350 open-source projects spanning almost every corner of enterprise technology, including web servers, operating systems, development tools, and database management systems.

It All Started with a Web Server, I am pretty confident that anyone that has read this far has used the Apache Web Server before?

The Broader Apache Universe

To give you an idea of how diverse the community is, here are some major non-data engineering projects you might run into:

• Web & Application Servers: Along with the original HTTP Server, they manage Apache Tomcat, which is used globally to run Java-based web applications.
• Databases & Storage: They host highly popular NoSQL databases like Apache Cassandra (originally built by Facebook to power their inbox search) and Apache CouchDB.
• Build & Dev Tools: Tools that developers use every day to build their code, like Apache Maven and Apache Ant, are standard parts of the software development lifecycle.
• Operating Systems: They even house Apache CloudStack, which is open-source software designed to deploy and manage large networks of virtual machines as a cloud computing platform.

All of the above is out of scope for this learning journey, but in the reverse it reminded me of the Cloud Native Computing Foundation (CNCF) in a way.

Lets get into the Apache Data Engineering Stack and surrounding technologies.

Ingestion/Streaming (Apache Kafka)

The highway that captures live data and moves it into the architecture.

Storage (Apache Iceberg, Delta Lake inside Cloud Storage (S3))

The smart organising layer that formats the data safely on disk.

Compute/Processing (Apache Spark, Apache Flink)

The heavy-duty engines that reach into storage to transform and clean the data.

Architectural Platforms (Snowflake, DataBricks, Microsoft Fabric)

The overarching cloud environments where all of this compute and storage lives.

Orchestration (Apache Airflow)

The conductor that schedules when the ingestion, storage, and compute actions trigger.

Languages/Interfaces (Python, SQL, Jupyter).

The tools you use to write the logic and talk to every single layer above.

Other Tools (Containers, Virtualisation, Cloud)

Docker was mentioned but I have covered this in the #90DaysOfDevOps series, so going to drop that link below.

https://github.com/MichaelCade/90DaysOfDevOps/blob/main/2022/Days/day42.md

My goal now instead of using Gemini to craft these somewhat interesting images above, is to find where people much smarter than me have already digested a lot of these tools and areas so I can then be smarter myself when I go back to the crash course.
Also side note… that initial crash course is not the full course free on youtube, its an almost 4 hour teaser to a 17 hour paid for bootcamp.. Jury is out if thats the way to go.

For those that know me, I have been a constant and consistent learner. I started my IT career with hardware—building servers for Cambridge University, getting server operating systems up and running, and handling systems administration and operations. That was before the virtualization era. From there, I moved to the cloud, transitioned my learning into DevOps and Kubernetes, and that brings me to the here and now.

We are surrounded by AI, but “AI” means so many different things to different people, teams, and companies.

Lately, I have been intrigued by a specific challenge: How can we ensure the validity of our AI experience when it interacts with our own data? How can we ensure the data we hover above an LLM is actually going to provide us with the correct information?

From that perspective, I have found myself drawn into the world of Data Engineering and DataOps.

So, what’s the plan?

The first area of focus is watching foundational content designed to establish core concepts before diving directly into specific platforms. I’m currently working my way through this crash course: Data Engineering Foundations Crash Course!

I am specifically looking at recent content that discusses data engineering core principles and how they land with AI and machine learning in mind. This video is proving to be a great starting point because it details how data flows through real-world systems—exploring how data is stored, processed, orchestrated, and served. It bridges the gap between raw data and consumption, illustrating how foundational steps like proper data collection and pipeline stability directly impact the success of downstream machine learning models and AI dashboards.

Before I step into heavy-hitting enterprise tooling like Databricks, Snowflake, and Microsoft Fabric, this stage is entirely about cementing those basic Data Engineering Foundations through a structured, multi-phase approach:

Phase 1: The Storage Revolution (From Blocks to Formats)

Typically, my learning has gone straight into the platform layer, but we need to focus on the data layer for this new world. In Data Engineering, the bucket alone is just the graveyard; the magic is in the open table formats sitting inside them. To master this, you must understand Delta Lake, Apache Iceberg, and Apache Hudi. These formats add an ACID compliance and metadata layer to Parquet files, effectively making object storage act like a high-performance database.

Phase 2: The Data Pipeline Lifecycle (CI/CD for the DevOps Crew)

Data Engineering has its own version of a DevOps pipeline, and you need to map ETL/ELT (Extract, Load, Transform) to concepts you already know. Instead of application deployment, the focus here shifts to data ingestion, code-driven data transformation (using tools like dbt), and modern workflow orchestration (like Airflow or Dagster).

Phase 3: Data Preparation for LLMs (The RAG & Semantic Layer)

Before an LLM can interact with your data safely and accurately, that data has to be translated into a mathematical language the model understands. This is where traditional data engineering pipelines pivot into AI-native architectures. The focus here shifts from flat relational tables to managing unstructured data (PDFs, documentation, customer tickets).

I need to understand the lifecycle of chunking strategies (how you slice up long text), generating Vector Embeddings via embedding models, and optimizing Approximate Nearest Neighbor (ANN) indexing algorithms like HNSW (Hierarchical Navigable Small World) for sub-millisecond similarity search.

I’ll be exploring dedicated vector systems like Milvus and Qdrant (written in Rust for low latency), seeing how standard databases adapt via extensions like pgvector for PostgreSQL, and learning how orchestration frameworks like LangChain and LlamaIndex stitch pipelines and LLMs together.

Phase 4: Data Privacy, Masking & Governance for AI

When you “hover data above an LLM,” your security perimeter changes completely. Traditional Role-Based Access Control (RBAC) doesn’t gracefully translate when an LLM can access an entire data lakehouse and accidentally leak sensitive executive info or PII (Personally Identifiable Information) through a single user prompt.

This phase is about moving toward modern Data Security Posture Management (DSPM) and context-aware, adaptive policies. I need to understand and learn programmatic Dynamic Data Masking (redacting data at query time based on user role) and Row-Level Access Policies within the pipeline before the data ever reaches the LLM’s context window.

Phase 5: DataOps & Data Observability

Think of this as Prometheus and Grafana, but for data quality. If an infrastructure pipeline fails, a pod crashes. If a data pipeline fails silently, a null value enters a database, poisoning downstream AI models and corporate decision-making without anyone noticing until it’s too late.

Phase 6: Platform Distinctions & Architectures

Once the fundamentals are locked in, it’s time to get into the weeds of the platform layer. At the moment, I’m framing this exploration around three core pillars:

1. How data enters
2. How it is secured
3. How it is consumed (including by AI)

At this stage, I don’t know what I don’t know, so this list may evolve, but the dominant enterprise platforms and ecosystems I keep hearing about are Databricks, Snowflake, and Microsoft Fabric.

TLDR;

• Phase 1: Storage Formats (Iceberg, Delta Lake) — Where it lives.
• Phase 2: Pipelines & Orchestration (dbt, Airflow) — How it moves and transforms.
• Phase 3: Data Preparation for LLMs (Vectors & Embeddings) — How the AI reads it.
• Phase 4: Governance & Privacy (DSPM & Masking) — How we keep it secure.
• Phase 5: DataOps & Observability (Data Quality) — How we ensure it isn’t poison.
• Phase 6: Enterprise Platforms (Databricks, Snowflake, Fabric) — The big suites that tie it all together.

My other “Big Rocks”

As well as navigating the brave new world of data engineering, I also need to continue diving deeper into a few of my existing technology areas:

• DevOps Tooling & Source Code Repositories: Deepening expertise across GitHub, GitLab, and Azure DevOps.
• DevOps, Product Management & Development Services: Focusing on the Atlassian stack (Jira & Confluence).
• Platform Focus: Gaining an even stronger foothold in Red Hat OpenShift and KubeVirt.
• Databases: Expanding operational knowledge of modern database architectures.

I wrote about the Veeam Software Appliance and ran through the steps to get the system up and running as the brains of your backup environment along with some of the benefits it brings.

For those familiar with the process I took in the above link in vSphere, be warned defining virtual machines in Kubernetes is all about the YAML.

What you will need

A Kubernetes cluster, I will be using Talos Linux as my Kubernetes distribution in the home lab but any bare metal Kubernetes cluster should work providing you have the system requirements.

The system requirements for the nodes in your cluster should have enough to meet the requirements of the Veeam Software Appliance. 8 vCPU, 16+ GB RAM, and two disks (≈240 GB+ each)

You should have a StorageClass with enough capacity to store the above disk requirements.

KubeVirt + Containerized Data Importer (CDI) installed and working. CDI is used to create DataVolumes and import/upload images. (I am running version 1.6.0 of kubevirt)

Client side you will need kubectl and virtctl, kubectl is how we will interact with the cluster, virtctl helps us to upload images, start virtual machines and open a VNC/Console to the machine.

High Level Steps

• Upload the Veeam Software Appliance ISO into the cluster (DataVolume)
• Create Blank DataVolumes (PVCs) that will become the appliance target disks (this needs to be 2 x 250gb due to us needing 248gb)
• Create a Virtual Machine that attaches the ISO and the blank PVCs as disks (ensuring the correct boot order)
• A service so that the system can be accessed outside of the cluster.
• Start the VM and work through the initial configuration wizard of the appliance

Upload the ISO into the cluster

We now need to get the ISO into our cluster as a DataVolume to be used with the virtual machine creation in the next step. First we need to create a namespace where will create the virtual machine and everything associated with it.

kubectl create ns veeam

You could use port-forward but for a large ISO like this it might take a considerable amount of time. I opted to use a local web server on my system to share the ISO. I used a simple python web server with the following command.

python -m http.server 8080

We will then create a DataVolume for our ISO and we will specify the URL to get our ISO. Be sure to change your path and storageclass.

# veeam-iso-dv.yaml

apiVersion: cdi.kubevirt.io/v1beta1

kind: DataVolume

metadata:

name: veeam-iso-dv

namespace: veeam

spec:

pvc:

accessModes: ["ReadWriteOnce"]

resources:

requests:

storage: 13Gi

storageClassName: ceph-block

source:

http:

url: http://192.168.169.5:8080/VeeamSoftwareAppliance_13.0.0.4967_20250822.iso

kubectl apply -f veeam-iso-dv.yaml

The import process will then start and you can use the following command to see the import progress.

kubectl -n veeam get dv veeam-iso-dv -w

You can keep on eye on this, don’t panic if it gets to 99.99% and sits there for the same time again with no updates. You can come out of this watch command and check the pod logs, you will have an importer pod in your veeam namespace that you can run this command against.

kubectl logs importer-prime-a1ec931f-5335-4f59-aa97-6f165f1c38eb -n veeam

When complete you can run the following command and hopefully you will see something like the screenshot below

kubectl -n veeam get dv veeam-iso-dv

Note when I first tried to use port-forward with this iso it was suggesting 7+ hours so this way is much more efficient.

Create blank target disks (DataVolumes)

You might at this stage be realising the difference between Kubevirt and a hypervisor like vSphere…

We have the ISO uploaded to the cluster and we are now ready to create our Virtual Machine.

There is an option in the following YAML to use CloudInitNoCloud to inject an SSH user for access after installation, I have not added this as although you can SSH into the Veeam Software Appliance from a security standpoint it is not enabled and requires elevated security officer approval, maybe more on this in another post.

We are now going to define our Virtual Machine in code. Again if you are copying and pasting then please amend resources, storageclass and namespace if different to what I have used above.

The appliance requires UEFI vs BIOS so we have made that change to the VM configuration below.

You can find the VM code listed here in this GitHub repository
veeam-vm.yaml

https://github.com/MichaelCade/vsa_kubevirt

Now we can create our VM using the command below

kubectl apply -f veeam-vm.yaml

At this stage our virtual machine is powered off but to confirm the status we can check using

kubectl get virtualmachine -n veeam

Start the VM

Four YAML files later, we are now ready to start the machine.

When ready we can start things using the following virtctl command

virtctl start veeam-appliance -n veeam

lets then use that command again to check the status of our virtual machine

kubectl get virtualmachine -n veeam

Now we need a console to complete our initial configuration wizard within the TUI, I have installed tigervnc as my client

you can then run a virtctl command to connect

virtctl vnc veeam-appliance -n veeam

and with this command you will also see the following pop up,

Ok, I think that is enough for this post. I have been storing the code snippets in a repository on GitHub that can be found here. – https://github.com/MichaelCade/vsa_kubevirt

Once you get to this page of the Initial configuration, you can then follow this post to complete the steps. – https://vzilla.co.uk/vzilla-blog/the-veeam-software-appliance-the-linux-experience

I have also skipped over how “Virtualisation was just the start” and it was, now the Veeam Data Platform protects workloads and data across many different platforms, VMware vSphere, Microsoft Hyper-V, Proxmox, Oracle Linux Virtualisation and lots more hypervisors, as well as protecting public cloud workloads on AWS, Microsoft Azure and Google Cloud. The protection of Kubernetes came almost five years ago with the acquisition of Kasten now known as Veeam Kasten or Veeam Kasten for Kubernetes. We have M365, Salesforce backup, EntraID, Agents for Windows, Linux, Solaris, AIX and then we also protect unstructured data by way of NAS devices and Object Storage locations.

As of this week the management layer of Veeam is no longer just an option on Windows, Linux entered the room.

Home Lab Setup Plan

My morning of the 3^rd September was sat waiting for the downloadable, public release of the Veeam Software Appliance (VSA) We as Veeam have made big splashes about this at our event earlier in the year and showed off a lot of the features and functionality, I am not going to get into too much of that here.

When the downloads became available it was time to start the engines, but not without a solid plan. I have documented my home lab setup in previous posts. As much as this is a home lab I still want to treat things with some real life accountability. However, what I am about to show you is that my repository will be living on the same virtual environment I am likely going to be protecting… for now and this will be resolved down the line with the Veeam Data Cloud Vault option where I can store those backups offsite in the Veeam first party solution.

OK, back to the plan.

As you can see from the above image we had some virtual machines to create, 10 if we include that DNS server down there. We also have some futures that are documented from Veeam as coming in the not too distant future.

The VSA is available in an OVA and ISO format. There is then a smaller ISO for the Veeam Components such as the Hardened Repository and Proxies. You can see we have 5 proxies planned and a Hardened Repository (again bad practice to store a backup repository on the infrastructure you are going to protect, don’t be like me)

For the VBR (Veeam Backup & Replication) server we will use the OVA and import this into our vSphere environment and create a new VM.

For the EM (Enterprise Manager) server we will use the ISO and create a VM and run through the configuration steps to get this setup.

The proxies and hardened repository will use the smaller ISO

Veeam ONE is a powerhouse with this release, still Windows but massively important. We will provision a Windows 2025 Server for this.

I will mention for this plan, I also provisioned a Windows 2025 Server Core for the DNS server and created a new forest called “vzilla.local”

Hopefully that makes everything in the picture a little clearer, the web client can be any OS and any mainstream browser (I have tried with Chrome, Edge and Firefox) The thick client aspect is used to access some of the Veeam ONE and the VBR server. I have installed the VBR thick client on the Veeam ONE Windows Server.

VBR Deployment Process

As mentioned I am going to be using vSphere here to deploy my virtual machines for this plan, clearly anything can be used including physical systems. Anything that supports the OVA but equally this is why there is an ISO for those physical systems or where OVA is not supported.

Select the OVA

I am not going to go through step by step here, basically select the OVA, give the machine a relevant name for your environment choose the DC, Cluster or Host you wish to deploy to, select some shared storage and finish.

When that import is complete, you will have a powered off virtual machine ready to be powered on. (I created another import to capture the process here and it is named 2025-veeam-vbr1)

Powering on the VM

Next we will see a booting ‘Veeam Backup & Replication” followed by the initial configuration wizard, starting with the license agreements.

Accept this then we can give the box a name.

Next we can configure our network, I have used the static option to set a standard static ip4 but ipv6 is also available to set here.

Then we need to change the time, I am based in the UK so I selected change and searched for London to update the timezone but left the available NTP servers as is.

Then we are onto setting a host administrator, this process is the same for any managed Veeam machine, proxies, repositories and enterprise manager.

The next screen is configuring the multi factor authentication (MFA), I am using the Microsoft Authenticator for all of my MFA needs so I hit show QR code, scan that within the app and then type the number provided from the app to proceed to the next step.

Following that step you are then asked if you would like to create a Security Officer,

At the Security Officer step of the Initial Configuration wizard, configure the default security officer account to perform specific operations in the Host Management console — veeamso. This account type provides an additional security layer to protect your infrastructure against malicious system administration.

The above was taken from the documentation pages – https://helpcenter.veeam.com/docs/vbr/em/deployment_linux_iso_install_security_officer.html?ver=13

if you choose to skip this step, you are prompted with a very red warning. It will not be possible to enable this role later on after this step. For the purposes of this demo walk through I have said OK but in my lab environment I have created that veeamso account on all hosts, when you set a password in this wizard, you will be required to change that when your security officer first logs into the management console to approve tasks.

The final page is a summary of what you have configured

Select finish and then wait for the state configuration to be saved and then for the services to come up. When they are up you will be able to access.

Host Management Console

The Host Management Console allows administrators to perform configuration and maintenance tasks, including managing network settings, server time, host users and roles, backup infrastructure, OS and Enterprise Manager updates, maintenance operations, and security settings.

This can be accessed via https://192.168.169.122:10443/

obviously changing to your set IP address, you will then be able to login here with the veeamadmin account.

Web UI Preview

The Veeam Backup & Replication web UI is a browser-based interface that enables you to manage backup and recovery operations, monitor your backup infrastructure, and configure system settings from any supported device. The web UI provides a modern, streamlined experience designed to simplify daily administration and deliver at-a-glance visibility into your data protection environment.

Please note that this is a preview and not all capabilities within Veeam Data Platform are available here today in this release, you will be able to manage some of your environment from here but you will need the thick client access for all tasks.

This can be accessed via https://192.168.169.122/

again changing your IP address above.

Thick Client Access

For those of you familiar with the thick client approach, it has been enhanced and looks much better, its faster and is used and designed to let you quickly find commands that you need and perform data protection and disaster recovery tasks.

As stated in my diagram above, you will need a Windows based machine to install and use the thick client. Point the client to your address above.

Next Steps

To complete the home lab setup, I went and created 6 further virtual machines using the smaller JeOS ISO and this is what was used to create the hardened Linux repository (bad practice to run this on a VM, the security score in threat center will also warn you against this, don’t be like me) and then the five proxies, the process is very much the same as the initial configuration wizard we went through above and then you add them into your VBR thick client.

With the proxies, I created one proxy per vSphere host, when I added them in you have the choice of what proxies they are going to be.

I then repeated the steps for VMware and VMware CDP as I want these to be the data movers for all tasks.

I had also deployed a Windows Server, downloaded the new Veeam ONE ISO and got things up and running there, maybe another post on those steps to come. Adding Veeam ONE is important as this gives you some great insight into the security posture of your backup environment. The threat center element is pulling data from Veeam ONE to display within the thick client and the web UI.

Next, I wanted to get my Enterprise Manager VM up and running and to do this I used the Veeam Software Appliance ISO and this runs again through a similar configuration wizard as shown but on first boot you will get a choice of Enterprise Manager or Veeam Backup and Replication to install on this, process was to create a VM with the required CPU, Memory and disks and then run through that process.

The final steps was to add some infrastructure to protect and then create some backup jobs. If you are familiar with Veeam this should be the same process as before.

To caveat once again here, this is a lab environment where I can show demos of Veeam Software, I am using virtualization for components that in production should be on physical hardware and not virtual machines, but for home lab environments what I have built will cover elements of what I need to cover in demonstrations. I will also be adding cloud based protection workloads and Veeam Kasten instances later on down the line that I have running to extend the lab into different platforms.

I am able to get into the vCenter Server Management console at https://192.168.169.181:5480 a port etched into my brain for some reason!

But when we head to services we have many that are not running and they should be.

I check the access settings and ensure that SSH is enabled so we can dive deeper and try and get these services up and running.

I first try ssh root@192.168.169.181 but I am met with

So my alternate way is using

ssh -o IdentitiesOnly=yes -o PreferredAuthentications=password -o PubkeyAuthentication=no root@192.168.169.181

Which gets me in to

I then use the service-control –status command to see the same what we have in the management UI

Followed by service-control –start –all which seems to sit here for a very long time but noting this down at this stage to make sure future me doesn’t do anything silly.

While the above was going on I checked the time tab in the management UI and we were back in time, around 2023 and wondered if this could be why things were not quite right and we were also using host based NTP, I changed this setting and got things up to date

In changing the time we were now able to get into

We do have to wait a while for things to finish initializing here. But maybe it was all down to time! I actually got impatient and hit the reboot button from the management UI.

Things do not look all that pretty and I may update if relevant here with what is going on!