The Challenge

The challenge we have with Kubernetes storage is that it’s not all that easy and it’s very manual to achieve some of the tasks that Kubestr helps you with, for example, the adoption of CSI drivers and choice of storage available to us within our Kubernetes clusters is growing so fast. This tool is going to assist in validating that the CSI driver is configured correctly for snapshots for example this, in turn, means we can use data protection methods within our cluster. Another hard task is benchmarking storage, it can be done today or prior to Kubestr but it’s a potential pain to make this happen and it takes time. Kubestr allows us to hit the easy button to evaluate.

All of this whilst there are so many options out there for storage, we want to make sure we are using the right storage for the right task, at the end of the day you can go and pay for the most expensive disk especially in the public cloud but let’s make sure you need it and you don’t overspend and also instead of spending your time building benchmarking tools manually this will save you time to giving you a better understanding and visibility into your storage options.

You can find out more here on the Kasten by Veeam blog explaining in more detail the challenges and the reasons Kubestr was born.

Getting Started with Kubestr

We all use different operating systems to manage our Kubernetes clusters, first and foremost Kubestr is available across Windows, macOS and Linux you can find links to these releases as well as source code here.

Once you have this installed on your OS the first command, I suggest is (I am running windows) We can see then the simplicity of what can be used from a command point of view as well as additional available commands.

.\kubestr.exe --help

Identify your Kubernetes Storage options

The first step that this handy little tool can help you with is just giving you visibility into your Kubernetes storage options available to you. I am running this below against an Amazon EKS cluster using the Bottlerocket OS on the nodes. I have also installed the AWS EBS CSI drivers and snapshot capabilities that now is not deployed by default. Now my cluster is new and has been configured correctly but this tool is going to highlight when things are not configured maybe you have the storage class available but you do not have the Volume Snapshot class or maybe you have multiple storages available and some of that is not being used and this highlights that you have this storage attached and could highlight that you could save by removing it.

.\kubestr.exe

Validate your Storage

Now that we have our Storage classes and our volume snapshot class, we can now run a check against the CSI driver to confirm if this was configured correctly. If we run the same help command with the csicheck command, you get the following options.

If we run against our Kubernetes cluster, storage class and volume snapshot class we will see the process on the below image that runs through creating the application, taking a snapshot, restoring the snapshot and confirming that the configuration is complete.

.\kubestr.exe csicheck -s ebs-sc -v csi-aws-vsc

Evaluate your Storage

Obviously, most people will not just have access to one Kubernetes cluster, for us to run against additional clusters you simply change the kubectl config context to the cluster you would like to perform the tests against. In this section, we want to look into the options around evaluating your Kubernetes storage. This has a very similar walkthrough to the CSIcheck we mentioned and covered above apart from there is no restore but we are going to get the performance results from Flexible IO.

Let’s start with the help command to see our options.

.\kubestr.exe fio –help

Now we can run a test against our storage class with the following and default configurations as listed above.

.\kubestr.exe fio -s ebs-sc

Now we can get more catered to specific workloads with different file sizes for the tests.

.\kubestr.exe fio -s ebs-sc -z 400Gi

Then we can output this to JSON and this is where we see the community helping here to be able to extract that JSON and allow for a better reporting method on all of the results so that the community can understand storage options without having to run these tests manually on their own clusters.

.\kubestr.exe fio -s ebs-sc -z 400Gi -o json


.\kubestr.exe fio -s ebs-sc -z 400Gi -o json > results.json

I won’t post the whole JSON but you get the idea.

Finally, we also can bring your own FIO configurations, you can find these open source files here

#BYOFIO - # Demonstrates how to read backwards in a file.

.\kubestr.exe fio -s ebs-sc -f "D:\Personal OneDrive\OneDrive\Veeam Live Documentation\Blog\Kubestr\fio\examples\backwards-read.fio"


#BYOFIO - fio-seq-RW job - takes a long time!


.\kubestr.exe fio -s ebs-sc -f "D:\Personal OneDrive\OneDrive\Veeam Live Documentation\Blog\Kubestr\fio\examples\fio-seq-RW.fio"

I have just uploaded a quick lightning talk I gave at KubeCon 2021 EU on this handy little tool

My next ask is simple, please go and give it a go and then give us some feedback,

The image below shows pre Veeam Backup & Replication v10 when you go to configure your application aware processing.

Configuration

Now in Veeam Backup & Replication v10 and specifically those orchestrated snapshots only jobs we can be more granular on what we protect as per below.

This option enables you to take a periodic log backup to a Veeam Backup repository, whilst using the storage snapshot as the baseline. This is only applicable to orchestrated storage snapshots (this capability is already there for image based backups) to achieve this in your storage snapshot only job you will walk through the wizard until you get to the guest processing option page and then confirm that you require application-aware processing and then you can select Applications.

Next and this will depend on how many machines you have in your job and the placement of those VMs will depend on if you are wanting to achieve the VMware Snapshot-less approach to getting an application aware and consistent storage snapshot. Select your VM and edit to make the relevant application log file decisions.

For this VM which is a machine with SQL present I can now set the appropriate configuration that I need here.

Oracle would be the same here on the next tab over.

When this job is now scheduled to run if you have adhered to the configuration in the previous blog post around VM, datastore placement and backup job configuration then you can now achieve an application aware storage snapshot, without the requirement of a VMware snapshot whilst also being able to create a periodic log backup and use the storage snapshot as the baseline for any restore functionality.

Recovery

From a recovery point of view obviously this means we can have a much better RPO for our database recoveries, we have the speed from the storage snapshot baseline and we have the transaction log backups stored on a Veeam repository as a .VLB file or files.

A common question I have been asked about this feature is will this work with secondary snapshots, for example with NetApp ONTAP, HPE Nimble and HPE 3PAR we have the capability of being able to leverage their storage replication to create and send their snapshots to secondary systems. This feature will work in those scenarios also.

Another thing to note here is that with HPE Nimble those VM Disks for snapshot-less processing must reside on the same volume collection.

Ok, so how do we recover this data using our Veeam Explorers for SQL or Oracle.

Firstly, you should select the ribbon at the top and choose Restore, choose Restore from backup as our option.

Next, we want to choose the Application items restore option.

You will then see the list of Veeam Explorer options that we have, for this example we will choose SQL but this is also applicable to Oracle recovery options.

At this point we are then faced with the Veeam Explorer for SQL and you should choose the machine that is relevant for your restore operation, specifically here you will see a list of backup jobs and storage snapshots. We will choose our storage snapshot and our SQL server.

You will then be faced with the available restore points, or you can choose the latest available snapshot with the option at the top of the wizard.

Then choose a location for a temporary VM will be registered but remained powered off, this will be automatically removed after the restore operation is closed.

To finish the wizard, give a reason for the restore for audit purposes and then select finish on the summary screen this will then open the Veeam Explorer for SQL with the mounted databases from the restore point you chose.

Up until this point this is not different you could always do this before but you would have created the snapshot whilst also taking a VMware snapshot and you only have point in time copies of your databases on your storage snapshot. What this new feature in Veeam Backup & Replication v10 enables is the ability to restore to a specific point in time using those transaction log backups that we put on our Veeam repository as the .VLB files on a schedule.

Getting down to the specific transaction can also be achieved by selecting the “perform restore to the specific transaction” at the bottom of the wizard above. You then have the ability to fine tune which transaction you wish to restore back to, this is a lab environment so not much happening, but I created a transaction to show you an example.

The transaction log backup job runs permanently in the background, shipping transaction logs to the backup repository at a specific time interval (by default, every 15 minutes). This process was once only available for backup jobs but now you can achieve this with your storage snapshots also.

The transaction log snapshot /backup session starts and stops in the following way:

• The initial session starts when the parent backup job schedule is enabled. After that, the session starts with every new session of the parent backup job.
• The session ends before the next session of the parent backup job, and/or when this parent backup job is disabled.
• When the session ends, Veeam Backup & Replication stops the runtime process and uninstalls it from the VM guest OS. When a new session starts, the runtime process is deployed again.

Hope that was useful any questions let me know here below or get in touch on twitter @MichaelCade1

This was a nice feature that was sneaked into V10 without many people realising.

A common challenge with High IO workloads with VMware snapshots is that it would or could kill performance of likely this mission critical system when a VMware snapshot takes place, which either means you take that risk and manage to get a backup to happen during a relevant window, you leverage crash consistent storage snapshots which also do not require a VMware snapshot or GASP! You just do not do anything and hope that nothing happens to this MISSION CRITICAL system.

Ok so what can we do? – The theory

What if I told you, you could take an application aware and consistent storage snapshot without having to take a VMware snapshot?

If you are running one of the many storage integrations that are supported by Veeam then you are in luck. You can configure a Storage Snapshot only job with your application aware processing configured within the wizard and that is it. Ok there are some caveats. The VM’s virtual disks must be located on the same datastores and they must be unique within the backup job.

Here are some examples:

If you have one Veeam orchestrated snapshot job configured and you have 3 VMs from the same VM Datastore then the traditional method of creating a backup will take place which will involve a VMware snapshot.

Example 1

If you have one VM on a datastore that has high IO and you do not wish this to be affected with VMware snapshots then create one backup job that contains only this VM and you will achieve this application consistent storage snapshot.

Example 2

Finally lets take a Tier 1 application that is made up of 2 virtual machines, these machines reside on their own VM datastores and storage volumes, they are also both added to the same backup job this will also achieve that application consistent storage snapshot.

Example 3

Things to remember

• VM must be the only VM in storage volume / datastore within the backup job.
• If the VM is not the only VM in the storage volume / datastore and backup job, then VMware Snapshot will still take place.
• All VMs that do not qualify will be processed in parallel.
• All VMs that do qualify they will be processed sequentially.

Walkthrough

I am going to pick 3 virtual machines for this test

To match these jobs with the examples above:

• Snapshot-less Orchestrated Snapshot – Same – Example 1 (2 VMs on the same datastore and same storage volume)
• Snapshot-less Orchestrated Snapshot – Single – Example 2 (Single VM on its own unique datastore)
• Snapshot-less Orchestrated Snapshot – Multi – Example 3 (2 VMs on their own unique datastores within the job)

The key to note here is that the datastores mentioned above all contain other VMs in the environment but they are not included in the backup job.

Ok let us walk through creating the orchestrated snapshot job to make this happen. Ok so this is pretty simple but it is important to know where VMs are being stored to take advantage of this functionality, if that VM moves then it will revert back to a VMware snapshot unless it is a VM in it’s own Backup Job. First of all as with all Veeam jobs, give it a relevant name.

Click add and choose your virtual machines, this can be straight via the VM name and remember there is a search function there that will help with choosing out granular machines. You can also use vSphere tags but remember where the VMs are placed in order to use that option.

Next up, because we are running orchestrated snapshots for this role, at this point you will have already had to install or add your storage system within Veeam Backup & Replication, this way depending on the storage system you are using you can select the option for Primary Storage Snapshot Only for your specific storage integration.

The next screen is for application aware processing, this is the whole reason for the enhancement, we could already do crash consistent snapshot orchestration prior to v10 and this does not have the same limitations. For crash consistent you can have multiple VMs residing on the same storage volume. Add in your credentials and hit the test here to confirm all is good.

Most likely you are going to want to schedule this to happen to sort your RPO requirements.

Summary, this is what the job looks like, you can either save this and exit or you can say start the job now.

The Results

Now its worth pointing out that the systems I have chosen for this demo are lab machines, if you didn’t guess they are an Active Directory Domain Controller, Veeam ONE server running SQL Express and Veeam Backup & Replication also running SQL Express. (not the Veeam Backup & Replication server running the jobs) none of these systems are actually running high IO workloads this is to prove the concept so please ignore the durations.

This first one is in line with example 1, we have two VMs stored on the same datastore and storage volume and clearly both within the same backup job, you can see I have highlighted that in this instance we are going to take a VMware snapshot.

The second example is of a single VM in its own backup job. You can see that there is no “creating VM snapshot” or “Removing VM snapshot” you only see “Primary storage snapshot created successfully”

Finally, we have example 3, this shows 2 VMs in one backup job but both VMs are stored on two different storage volumes / datastores. Much the same as example 2 above now you can see that no VMware snapshot is taking place just the storage snapshot.

And just to confirm that we are application consistent here we have the logs being truncated in the summary below. There were some also enhancement in this area when it comes to SQL and Oracle that I will have to get to in another post.

So, you could do this kind of prior to Veeam Backup & Replication v10 but they would be crash consistent and that’s a huge risk to take on such important systems. Veeam also has this capability already with the storage integration with Cisco HyperFlex as this uses the native VM snapshot engine and API available on the HyperFlex system to achieve this.

I think it’s fair to say NetApp have really pulled things back in the industry. Maybe they did lose their way and maybe they grew too fast for a period. But the main man Kurian seems to have steered things in the right direction from a business perspective and the tech is I believe in front of the curve.

This week you are going to see announcements from NetApp this is going to range from some significant updates when it comes to ONTAP 9.4, Security, FabricPool and FlexGroups. Oh, and those platforms. Yep this software must run on some hardware, well if you wish too.

I want this post to be an overview post for the other announcements and personally I want to call out some of those milestone features that will arrive with ONTAP 9.4.

Snapshots

Was a limit of 255 snapshots ever an issue…? I only came up against a few enterprise clients where it did, but they were some of the biggest organisations in the world and their use case was some of the most frequent snapshots and retention for really fast recovery.

Now I don’t want to turn this post into a Snapshots vs backup, we all know my feelings on this and it’s a snapshots and backups argument. But these large enterprises were also Veeam customers and they would use Veeam to trigger those application or crash consistent snapshots on a retention as well as leverage those snapshots for a backup to a different media type. The limitation they faced was the 255 snapshots per volume as it meant they could not keep minute-based snapshots for a long period of time. Now with the latest release of ONTAP this is going to increase things by over four times to 1023 snapshots.

ONTAP image installation

Hands up if you are a field installation engineer, or you have faced the task of getting to site and you must download the update image but then you have to setup your own network including web server to get your storage updated, now I cannot complain this method out of hours and some others went toward paying for our wedding. However, …

It might only be a minor thing to the most, but this is massive for those engineers and admin. To be able to update your version of ONTAP via OnCommand System Manager from your laptop is a great thing.

Platform

I will probably come back and have a complete post touching on the platform, and by platform we are talking that physical hardware offering. I am sure someone still gets excited by the hardware side of things.

We will see a new high end as well as entry level all flash FAS and a new entry FAS, 100Gbe and 25GbE networking capabilities. More NVMe over fabric and SSD and 30TB SAS attached SSD.

FlexGroup

Lots of new features and enhancements around Quality of Service and extending current support. I touched on the benefits of FlexGroups when they first came out here.

https://vzilla.co.uk/vzilla-blog/im-bringing-flexy-back-netapp-flexgroup

FlashPool

FlashPool as a feature was released a few versions ago and this update of ONTAP sees some significant enhancements.

Again, like the other pieces I could go into more detail in a separate post. But for the record FlashPool excites me because it changes the barriers, no longer is about keeping data siloed in a datacenter, also it doesn’t have to start with the datacenter with this release support for Cloud ONTAP and ONTAP Select bringing a cloud based or software defined way of tiering into cloud-based storage.

Full disclosure, I am a member of the NetAppATeam and I have attended forum calls and will be attending this week future briefings.

Veeam backup and replication 9.5 update 3 was released just before Christmas, this updated included a major feature in the Universal Storage API, this would allow Veeam to support a much wider range of storage vendors for integration into the product.

A closed process led by Veeam and storage vendor the private SDK is provided to the storage vendor to develop against, at completion the storage vendor will then provide the Veeam QC team with the new plug-in where it can be tested in the system for any bugs or configuration issues. At this point it can then be released to the public.

This post will run through the simple process of installing the universal Storage API plug-in for Infinidat storage arrays.

Firstly, login to your Veeam account here.

If you then scroll down this page you will see an area named “Download additional storage plug-ins” in here today you will see the Infinidat plug-in for Veeam Backup & Replication, expect to see a few more land in here as we move through the year.

Hit the download button for the plug-in you require within your environment. Once downloaded navigate to the download location.

Next up you should extract that zip file, for ease I chose the same location.

Next step is to install the plug in, if you receive the following User Access Control message then select Yes to proceed.

If you have not closed all Veeam Backup & Replication console then you will receive the following message.

Once you have been through and closed all related windows we can then proceed with this update.

Click Next. Do you agree to the terms?

As you expected this plug-in enables Veeam Backup & Replication to perform backup and restore of VMware vSphere VMs directly from storage snapshots of Infinidat storage arrays. I would add that it also allows us to do some pretty cool things around virtual lab environments and the ability to explore into historic snapshots that were not even created by Veeam. Click Install to begin.

This process is going to stop the Veeam services.

And then once it has installed the components it’s going to start those services up again, depending on the environment this should be a fast process.

Now that we have the plug-in installed and the services are now back and running we can open a new console to our Veeam Backup & Replication and start adding our storage array.

Once logged into the console navigate down to the storage infrastructure tab in the bottom left corner. Select Add Storage.

You will then see the nice new Infinidat logo and option to add the array to the Veeam Backup & Replication management console.

If you are running the standalone console on your laptops and desktops then the same installation of the plug is required on those devices to see the storage options.

This will be the same for all new storage plug-ins stay tuned for more that are coming soon.