In a previous post, I posted about Pac-Man as a mission-critical application, I have decided that this is a great way to show off the stateful approach to data within Kubernetes, this is great as you have your stateful data residing in a MongoDB database, this consists of your high scores. I have been running ad hoc demos of Kasten K10 in various clusters and platforms, but something I found in AWS was worth sharing.

Without repeating the build-up of the Pac-man configuration mentioned in the blog just linked. We have a front end NodeJS web server (this is where we play Pac-man) and we have a MongoDB backend which is where we store the high scores. There is a service created for both pods that expose them out using the AWS load balancer.

In the deployment we leverage load balancers, if we apply this to an EKS cluster, we use the ELB by default, which gives us an AWS DNS name linked to the load balancer which forwards to our pods. As you can see in the below screenshot, the associated security group created for this load balancer is wide open to the world.

Obviously, there are some gaping holes here both in the security group configuration, and there is very limited access control for the application itself. But I wanted to highlight that bad things happen, or mistakes happen. Let’s get into this.

High level – Bad Practices

Basically, by configuring things in this way our services are very exposed, whilst our application works and takes advantage of all the good things with Kubernetes and AWS and the Public cloud in general (this is not limited to AWS) Obviously by setting up the above way is not going to be best practice especially when it is a little more critical than Pac-Man and the back end high scores.

Before we talk about the considerations about making these bad practices into best practices, let me talk about the honeypot and some of the reasons why I did this.

The Ransomware Attack

I have been involved in a lot, a lot of online video demos throughout the last 12 months and the creativeness must be on point to keep people interested but also to get the point across.

Given that the service created for mongo was as below when deployed it will take advantage of the LoadBalancer available within the Kubernetes cluster, when I wrote the original blog this was MetalLB and this was exposed over my internal home network. When you get to AWS or any of the public cloud offerings then this becomes a public-facing IP address which means you have to be more aware of this and more on this later.

It is very easy at that point with the default settings that are configured from a security group point of view within AWS to gain access from any internet-connected device to your Mongo configuration. I will highlight this process now. First of all, you will need MongoDB Compass you can find the download for your OS here.

Once downloaded you can run this and then it is time to test out your unsecured connectivity to your Mongo instance. From here you will need that forward-facing DNS from AWS or in our case we have access to our Kubernetes cluster so we can run the following command.

kubectl get svc -namespace pacman

then within MongoDB compass, you can add the following and connect from anywhere because everything is open. Notice as well that we are using the default port, this is the attack surface, how many Mongo deployments out there are using this same approach with access not secured?

Mongodb://External-IP:27017

Here is a good copy of our data, you can see our Pac-man database there gathering our high scores.

Now we can flip to what happens next, once this is exposed it was likely 12 hours max before the attack was made, sometime between 4 am and 5 am of a morning. Now remember there is no important data here and the experiment is to highlight 2 things, make sure you have thought about all-access security for your application and everything is not exposed to the world to access. But my main point and reason for the demo are making sure you have a backup! The first point is going to protect you in a prevention state the latter is going to be what you need when things go wrong. I cannot help you too much with the data that you are storing in your database but just make sure that you are regulating that data and know what that data is and why you are keeping it.

As you can see from the above we have a new database now with a readme entry that gives us the detail of the attack and also no Pac-Man database this has been removed and no longer available to our front end web server. Just like that because of an “accident” or misconfiguration, we have exposed our data and in fact, lost our data in return for ransom.

The Fix and Best Practices

I can only imagine what this feels like when this is real life and not a honeypot test for a demo! But that is why I wanted to share this. I have been mentioning throughout the requirement to check security and access on everything you do, least privilege, and then work from there. DO NOT OPEN EVERYTHING TO THE WORLD, that probably seems like simple advice but if you google MongoDB ransomware attacks you will be amazed at how many real companies get attacked and suffer from this same access issue.

The second bit after configuring your security correctly is making sure you have a solid backup, the failure scenarios that we have with our physical systems, virtualisation, cloud, and cloud-native are all the same. The attacker did not care that this was a mongo pod within a Kubernetes cluster, this could easily have been a mongo IaaS EC2 instance exposed to the public in the same way. Backups are what will help remediate the issue it will help you get back up and running.

I was of course using Kasten K10 to protect my workloads, so I was able to restore and get back up and running quickly, it’s all part of the demo.

and we are back in business with that restore

Any questions let me know, no data was harmed in the making of this blog and demo. I have also deleted everything that may have been exposed in the screenshots above. I would also note that if you are walking through my lab and you are running through the examples again be conscious of where you are running, at home in your own network using MetalLB you are going to be fine as it will only expose to your home network, in AWS or any of the other public cloud offerings then that will be public-facing and available for the internet to see and access.

DataLabs is a really strong and included feature set included for the most part in all versions of Veeam Backup and Replication and plays a huge part in the Veeam Availability Orchestrator product.

It serves so many purposes:

• Automated Backup and Replica verification using SureBackup & SureReplica
• OnDemand Sandbox from backups and replicas giving you the ability to offer out sandbox environments of that backup data to your security teams, data teams basically anyone that you want to give access to your data but you don’t want them impacting your production systems.
• OnDemand Sandbox from Storage Snapshots, this takes the above one step further, the ability to leverage data and workloads from an application consistent fast performant storage snapshot and in some cases on a secondary system in a secondary location gives that isolated sandbox a few more use cases.
• Ever wondered when restoring from a backup, if that backup could be infected with some malicious threat(s) or ransomware, there is Secure Restore to ensure that you are clean for the majority of the Veeam restore functions.
• The ability to inject a process into the restore process, for example if you need to remove someones data from a restore process because they have asked to be forgotten through GDPR or other regulation then you can ensure this happens via Staged Restore.

All of these are covered in much more detail in the three whitepapers listed below.

v10 also introduced further functionality within Veeam Backup & Replication to enable you to do more with your data. The above options for sandbox and automated testing require the full system to be powered up, we will use the backup storage or the storage snapshot to present the data into an isolated environment but in order for us to use this data we also need to run the operating system of the backup data which means we need compute resources in memory and CPU available to run.

in v10 the introduced functionality is called the “Data Integration API” this enables you to take all of your backed up disks only and present them to a location, the premise for this is that it removes the requirement of additional CPU and memory but it also enables for many different use cases on what you can do with this data.

• Data Classification
• Security / Scanning for vulnerabilities
• Analytics
• Understanding what data is being kept / duplication of data.

Niels has a great article here on the new Data Integration and a walkthrough on how to get going.

That’s just some of the examples I can think of at least.

Veeam DataLabs – VeeamHUB – PowerShell Scripts

What is Staged Restore and how you can use it?

Activate your data with Veeam DataLabs Part 1: Overview

Activate your data with Veeam DataLabs Part 2: Configuration

Activate your data with Veeam DataLabs Part 3: Alternate Use Cases

Activate your data with Veeam DataLabs – Webinar Session with Demos and Theory based on VeeamON 2019 session

H1N1 Was The Last Pandemic. Here’s Why COVID-19 Isn’t Yet In That Category

I travel a lot for my work, in a global position my role requires me to be on a plane a lot and travelling to each corner. There is a mixed bag of travel for me, some of my travel is for customer and partner meetings but for the most part my travel is based around large IT conferences around the world. So far this year I have been to Shanghai, China in the first week back in 2019 (don’t worry I have had no symptoms of any cold or flu) then I have also been to Prague, Czech Republic and then also Atlanta, USA. Then I have had a huge block of being at home and no time on a plane.

What strikes me is the amount of conferences being cancelled due to the outbreak, I am not sure what happened in the IT industry going back to H1N1 in 2009 but now I am in this space its more visible.

As of today 29^th February, Happy Leap Day people. The following events have been cancelled within our Industry.

• Mobile World Congress in Barcelona
• Facebook Global Marketing Summit in San Francisco
• Facebook F8 in San Jose, California
• EmTech, Asia in Singapore
• Google News Initiative Global Summit in Sunnyvale, California
• Shopify’s developer conference, Unite, in Toronto
• Game Developers Conference, in San Francisco on March 16-17
• F5 Agility – March 16-19, 2020 in Orlando
• Cisco Live Melbourne
• Cisco Marketing Velocity Summit in Austin
• CloudFest 2020
• T Systems International Portfolio & Partner Day on March 5th in Berlin
• Microsoft Ignite Zurich cancelled
• IDC Direction SANTA Clara March 3rd

Currently KubeCon which is due to be at the end of March in Amsterdam is still on with some good communication from the organisers, I know this because this is the next event I am supposed to be attending but as the numbers grow and grow this and many more events will likely be considered for cancellation.

Another swarm of events being cancelled are sales kick off events generally around the beginning of the year or at the end of the company financial year. These are generally not published online but you just have to look at your twitter feed and speak to peers within in the industry.

It’s not just the IT industry there are lots of other events that were scheduled that have also been cancelled.

This is also affecting where people work, and really opens the door for companies to have to keep their employees at home to function away from large shared workspaces to prevent and additional risk of exposure, companies like Zoom and other collaboration and remote calling software is going to be some of the vendors that will benefit from this outbreak. I think one thing that will come from this is how companies deal with this in the future, I can see more and more people being allowed to work from home.

Lots more events are also listed here – https://www.cnbc.com/2020/02/06/reuters-america-update-2-dozens-of-asia-trade-fairs-conferences-postponed-amid-coronavirus-fears.html obviously as you can expect the Asian and in general APJ/APAC events are going to be the first ones to be cancelled.

Update: I am in the market for a Nintendo Switch and those supplies already seem to be affected by the virus and supply chain. Just one article outlining some of the details that we could be coming up against in that supply chain across the globe.

I am going to open up the comments here and would love to keep the events list growing with any that you know of so we can keep people informed.

Keep safe people,

That’s some of the things from a security perspective that should be in place but what if something does get breached and data could have got out of the business. Then you must report any data and security breaches to the information commissioner. This is new as this was a choice from the company if they felt it required by themselves to report. It is not mandatory.

Now we have the overview, I think you will agree it’s going to mean for a huge review for a lot of businesses to understand the way their systems work as well as their employees.

Data Minimisation

Another area that I feel this is going to highlight is the security of systems. Often overlooked or its been the cutting corners for getting things done quicker. GDPR will mean a much stronger focus on technical security, this is going to introduce the use of encryption and having to make sure those security patches are installed on those workloads.

As well as Encryption, pseudonymisation will also be another technique where businesses will have to explore. Pseudonymisation is a technique that allows to replace some of the identifiers with fictitious entries to protect people’s data privacy.

Quote from WikiPedia:

“Pseudonymisation enhances privacy by replacing most identifying fields within a data record by one or more artificial identifiers, or pseudonyms. There can be a single pseudonym for a collection of replaced fields or a pseudonym per replaced field.”

Road Warriors

Many of us carry laptops and other mobile devices around with us on a daily basis, the majority of these devices actually contain some work content. How is this affected? Well it comes down to that review of process and directive from the data protection officer. There will be a requirement as there is today but with more of a highlight on making sure staff members are reliable when taking personal data and business data offsite on these devices. Device encryption is one of the areas I can quickly see being the easy tick here but I am sure there are more options around this like remote workers with thin clients on the road. Because this could also put data at risk of exposure and failure to ensure these points could expose businesses to a fine.

As we move close to that GDPR-Day of the 25th May 2018,I wanted to collate all of the related blogs, white papers and other media together so that they could be seen and read. The need to become compliant should have already started for most but for some it’s been put in the corner. It’s happening people.

I have to share this link as well as I thought the URL was great.

https://howmanydaystill.com/its/gdpr

Veeam were very cautious about just releasing the next GDPR compliant message and marketing which generally speaking is completely false, there is not one backup vendor that can make any one company GDPR compliant. The process that Veeam has taken to release this information is basically on how Veeam itself has become GDPR compliant already and some of the features within the Veeam platform that assist you as a customer to becoming GDPR compliant state and then to retain that compliance.

I wanted to put together a list of resources that consolidates all of those links into one place. obviously this list I expect to continue growing but the bulk of the content is listed below.

Blogs

Fast approaching a year ago the first GDPR communications from Veeam were released via Danny Allan this blog really touches on the high level why and what this GDPR is all about, it touches on that this is a necessary change and how the world of data is very different now to what it was pre smart phones and all of this other connected technology. It also starts to touch on the journey that Veeam had to take to reach that compliant state and then to sharing that story.

One Year out – Considerations for the next 12 months

At the beginning of the year @DannyAllan5 began the blog series on GDPR and this really comes from the battle scars that were gleaned from the findings from Veeam becoming or starting that compliance journey. This 5 part series that is linked below touches on some of the key principles required and things that you need to know prior to getting to that compliant stage but then also how do you maintain that, it’s not a finish line that you as a business need to get to. This is a new way of life and managing data into the future.

GDPR: Lesson 1, Know Your Data

GDPR:Lesson 2, Manage Your Data

GDPR:Lesson 3, Protect Your Data

GDPR: Lessons 4 & 5, Document-Comply-Improve

White Papers

These 5 lessons can also be found in more detail in a couple of white papers released in January of this year. Mark Wong who holds the position within Veeam as the General Counsel has written these white papers. The first one is for IT staff this one covers the 5 steps but in more detail to assist in what the process should look like to become compliant from that IT manager perspective within a business. The second is aimed toward the CIO how Veeam plays a critical role within data management and protection strategies to ensure Veeam remain compliant while delivering Availability and how this information can help your business to achieve the same.

GDPR: 5 lessons learned Veeam compliance Experience Shared – A Step by Step guide for IT professionals

GDPR: 5 Lessons Learned, Veeam Compliance Experience Shared. CIO Summary.

Webinars

A recorded webinar from February this year.

• Our insights and five key lessons that we learned through our own compliance to help you on the path to thinking about GDPR compliance

• How to accelerate your GDPR efforts today

• Existing data management strategies and tactics for efficient IT assessment

Avoiding GDPR Penalties – 5 Key Principles. Veeam compliance Experience Shared

Other Resources

Finally we have the Veeam landing page with further resources on how Veeam can assist in that journey to being GDPR compliant. Some two pagers, recorded VeeamLive sessions and a Data privacy impact survey.

https://go.veeam.com/gdpr-compliance

At the recent Cisco Live I spoke on the Cube of the Veeam GDPR messaging.

A year ago I had my good friend Paul Stringfellow on the Veeam communities Podcast where we were talking about security but there were many mentions of GDPR but other security considerations that everyone should consider – Episode 123 – A Security 1,2,3 chat with Paul Stringfellow

Finally I have also been adding some of the technical features that will really help when it comes to this journey here.