Security and Virtualization
Thursday, 1. November 2012
Another long gap between since my last post, what else is new. I have no idea where the time has gone between working on development projects and testing too many things to list. So to kick off this post here is some interesting personal perspective on security in virtualization.
Now I am more than aware that 99% of the virtualization technology in production today is VMware and that Hyper-V is not at the top of most organizations’ consideration. I hope to change that perspective through this small review of some approaches that NVINT is taking as a small company. I am not here to try and sell Microsoft products (I am more of a Google Fan than anything) but when something is done right for more than one reason I always will like that product.
This will be a multi-part blog post that covers a few topics across Server 2012’s Hyper-V 3.0 to the 2012 System Center product family as they relate to the DFIR world.
Hyper-V
I will start with Hyper-V 3.0 inside server 2012 because there are a lot of great new features that really put it above other hypervisors. The first thing that really stands out as a major feature in 3.0 is that you can fully encrypt your storage clusters and allow access to that Shared SMB cluster over the network. This allows you to separate the Hyper-V boxes hosting the Virtual Machines from the machines hosting the SMB Storage cluster. This allows you to lock down access from only the Hyper-V servers to the storage cluster and monitor what accesses that data directly.
At NVINT I currently work with a Hyper-V 2008 R2 cluster that hosts over one-hundred virtual machines across six HP blade servers that make up the Hyper-V host machines. This Hyper-V cluster hosts the production virtual servers for our clients. I have been testing Hyper-V 3.0 for the past ten months, and I still feel like I have only opened one of the presents Santa left me under the tree. In my experience the stability and redundancy is really now on par with other hypervisors as where our current production 2008 R2 Hyper-V cluster lacks.
Windows Server 2012 features a lot of new capabilities, such as the new cloud networking features that scale a higher level network across WAN connections. It also contains the same powerful security features from the past like ASLR and DEP that are critical for security. I am not going to go into the infrastructure aspects because that has been done by the Microsoft engineers and I have provided the links below.
Check out these links if you want to know more about Server 2012 Hyper-V features.
Windows Server 2012 Hyper-V & Network Card (NIC) Teaming – http://www.aidanfinn.com/?p=12924
Hyper-V Network Virtualization Gateway Architectural Guide – http://technet.microsoft.com/en-us/library/jj618319.aspx
Hyper-V Network Virtualization Overview – http://technet.microsoft.com/library/jj134230.aspx
Network Virtualization technical details – http://technet.microsoft.com/en-US/library/jj134174.aspx
Windows Server 2012 Hyper-V Replica http://www.aidanfinn.com/?p=12147
The major upside to Hyper-V for NVINT is the fact that the Datacenter edition of windows allows you to host unlimited Standard editions of Windows in virtual machines without any extra license fees. (Note: this has limitations but that’s outside the scope of this post)
There is one really big feature of Hyper-V that I want to point out, as I feel there is a huge benefit from an Incident Response/Forensics point of view in terms of what you can do with the system that other hypervisors lack. For starters there are a large number of tools that you can use to analyze the Windows OS in many different ways. One of the best features is the ability to take a memory dump of the host os itself.
I have taken memory captures up to 80GB in the system and used Volatility to analyze those dumps successfully. This allows me to peer into the virtualization platform with more in depth detail that can be critical to understanding what is going on in the environment. Most people use VMware which makes it the most targeted Platform. Advanced techniques can be used to detect if the system is a virtual machine and even exploit out of the system to the host causing problems they were intended to prevent. This may change in the future, but currently Hyper-V is not the largest target on the market in its virtual machines. I am well aware the Windows OS is targeted by everything because it’s used by almost any originations today. I also feel that VMware currently has risks with USB support, currently however that isn’t a drastic problem yet. Someday however I would guess that it will become a more targeted exploit.
Alongside Server 2012 and the features it brings to the table the system center products allow a great deal of management and monitoring across all typed of devices outside the Microsoft Family.
DPM 2012
I currently work with Data Protection Manager almost every single day, and I have come to enjoy this product for a few good reasons. A major benefit for DPM is that it allows you to actively back up the memory as a scheduled process that Microsoft calls a “System State” backup. It also has standard features with Full-Disk and Incremental backups based on a scheduled format. It does anything from attached SAN storage to Tape backups which allows you to scale it out fairly easily.
At NVINT we maintain 15 days of full disk backups and the “System State” included. This allows me to restore a Virtual Machine to a saved state inside Hyper-V or to a network location much like the features inside VMware consolidated backup. The huge benefit from an analysis standpoint is that Hyper-V stores the system RAM for a virtual machine in two separate files the .bin and the .vsv which can be used with vm2dpm http://archive.msdn.microsoft.com/vm2dmp to convert them into a crash dump format that is compatible with Volatility or Microsoft Debugger for analysis. In an incident I can go into my Hyper-V server and use FTK Imager to create a copy of the live system’s disc and memory without installing anything in the Virtual Machine.
A key benefit of this process with FTK is to gather evidence without direct system interaction thus reducing the footprint on the compromised system. As a hosting provider this is critical because we can’t actively monitor with tools inside the OS itself. The tool Vm2dmp can also convert the DPM backup files because the .bin and .vsv file format is used to store the “System State” for protected systems. I feel that if something happens, that this type of insight can be critical with huge amounts of detail in what happened in the past for a better investigation. These backup files could reduce the data lost by an attacker using anti-forensics techniques. In most cases today the bad guys are getting really good at this process of covering their tracks so the DFIR community needs to get ahead of the problem before it occurs. Outside of virtual machines DPM can still perform “System State” backups of physical machines from Windows 7, Windows 8, Server 2008 R2 and Server 2012.
Here is some quick information on how to use vm2dmp
Create a dump file using virtual machine state files:
vm2dmp.exe -bin C:\VM\example.bin -vsv C:\VM\example.vsv -dmp C:\VM\memory.dmp
Create a dump file from virtual machine and snapshot name:
vm2dmp.exe –vm SERVER2008R2SP1-ENT-64 -dmp C:\VM\memory.dmp
vm2dmp.exe –vm vm SERVER2008R2SP1-ENT-64 –snap “vm SERVER2008R2-ENT-64 -snapshot-SP1” -dmp C:\VM\memory.dmp
Create a dump file using local symbols:
vm2dmp.exe -bin C:\VM\example.bin -vsv C:\VM\example.vsv -dmp C:\VM\memory.dmp –sym C:\symbols
For more information download the vm2dmp guide here: http://archive.msdn.microsoft.com/vm2dmp/Release/ProjectReleases.aspx?ReleaseId=3866
SCOM 2012
System Center Operations manager allows you to manage your network and give some detailed information into your environment in a central location. It supports Windows/Linux and SNMP devices alike. It allows options from an installed agent or agentless monitoring for information.
It has plenty of powerful rules that integrate into active directory and auto discovery rules to find new devices in the environment. This management server allows a centralized event log collection instead of taking up critical space on production systems. It can alert you via text or email if an event occurs based on custom rules. It even provides infrastructure information for System Administrators on descriptions, possible resolutions, and common causes.
I feel like it’s a more advanced Splunk that allows you target information with custom dashboards. I will not go into all the details because it can almost do anything and I provided some links to check out if you are interested.
For more information on SCOM check out these websites
http://www.scom2k7.com/installing-the-scom-2012-web-console-prerequisites-the-easy-way/
http://www.scom2k7.com/installing-the-scom-2012-web-console-prerequisites-the-easy-way/
http://technet.microsoft.com/library/hh278852.aspx
http://blogs.technet.com/b/momteam/
http://technet.microsoft.com/library/hh278852.aspx
http://blogs.technet.com/b/momteam/
VMM 2012
Virtual Machine Manger simply allows you to deploy fully updated virtual machines to your Infrastructure that follow a “Best Practice” standard with repeatable processes to avoid human mistakes. Even the simplest detail can be catastrophic for security protections in a complex environment. This also allows you to save time for the infrastructure guys allowing them to focus on much more important tasks.
I would love to go into more detail but I feel this post has covered enough for the day. Please feel free to leave any questions below or email me and I will respond as quickly as possible. Constructive criticism is always desired so please don’t hesitate.
I would to thank Andrew Case and Michael Hale Ligh for some proof reading and great suggestions. They did more than I expected as my first version was a little rough.
