Download Hyper-v Health Check -Real World Scenario

SKV PROPOSAL
TO OTC FOR HYPER-V HEALTH CHECKS
DATE:
June 17, 2013
Prepared by:
Sainath K.E.V
Microsoft Most Valuable Professional
Directory Services
Introduction:
SKV Consulting is a Premier Consulting providing Enterprise solutions on designing Microsoft
Technologies. SKV follows Microsoft standard frameworks and proven methodologies in designing
and implementing the Infrastructure solutions.
SKV has successfully performed Enterprise Infrastructure transformations including both Desktop
transformations and Server transformations. SKV has proven track record of quality and delivery
methodologies and provide value to its customers by reducing the Operations costs and increase the
revenue.
1
SKV Solution for OTC
Solution Description:
OTC has their infrastructure hosted on Datacenters which are hosted in Sydney and Brisbane. OTC
hosts their Datacenter on Cisco blade servers with Datacenter Interconnect capability and hosts
Microsoft Operating System on their Physical and Virtual machines.
Datacenter runs on CISCO Catalyst 6500 series switches over IP and uses IEEE 802.1ae encryption
with the help of SUP2T.
The virtual infrastructure servers are configured with Microsoft Active Directory, Microsoft Exchange
Server, Microsoft System Center Operations Manager, File Server, OTC Application Servers, and
Microsoft SQL Servers.
OTC has 2 Production VLANs and 1 Client VLAN configured on Cisco Catalyst Switches, each VLAN
is configured on Cisco switches 6500X series with IP networking configured on a dedicated switch.
Each of the VLAN has mix of IBM Physical servers and Microsoft Virtualized Servers. VLAN tags
configured for communication between servers and Storage arrays.
OTC is engaging SKV, a Microsoft Premier Consulting firm to validate their existing Hyper-v
infrastructure. Existing configuration is implemented on Clustered Hyper-v roles and Microsoft
System Center Virtual Machine Manager to manage the Storage, Network and Virtualization tiers.
SKV will perform best practice analysis and identify any risks associated with the infrastructure and
provide Mitigation plan to increase the ROI.
OTC Data Center:
OTC Datacenters are hosted in Sydney and Brisbane and is managed by IBM. OTC has both Inhouse users and mobile users and connects to both Datacenters for Applications, Files or resource
access.
OTC follows industry standard to manage its Datacenter. It has Core, Aggregation and Access
layers to provide different functionality at each layer. The Core is built on CISCO 6513E switches
which provides high speed packet switching which manages in-flow and out-flow data across
Datacenters and Internet. The Aggregation is built on CISCO 6500 series catalyst switches which
has the support for NAT, VPN services. The Access layer is where all the servers connect to the
networking, the SAN storage connects to the virtual infrastructure and there is a dedicated client
VLAN which is used for the clients to connect to corporate network.
On the Virtual Infrastructure, Microsoft services such as Active Directory infrastructure is hosted
with the Forest Functional Level of Windows Server 2008 and Domain Functional Level configured
with Windows Server 2008. OTC have single Forest namespace with Additional Domain Controller
configured and most of the delegation is done using Organizational Units. Below table shows the
existing servers and Network infrastructure of Sydney Datacenter on which the Active Directory
Restructuring activity occurs.
OTC Network Infrastructure
Cisco Router 6500 x 8
Cisco NETRANGE x 2
Cisco VPN concentrator x 2
DELL Equilogic SAN x 1
IBM Servers x 4
Description
Deployed at Core, Aggregation and Access layers
Security intrusion detection
Secure communication
SAN Storage
Application servers and Hyper-v
Server VLAN
2Server VLANs
1 Client VLAN
Microsoft Infrastructure
Components
Primary Domain Controller
VLAN
Description
VLAN 1
Forest Root Domain
Microsoft Hyper-v
Microsoft Exchange Server
VLAN 1
VLAN 1
Virtualization Stack
Exchange Server 2010
Microsoft SharePoint Server 2010
Additional Domain Controller
VLAN 2
VLAN 2
Microsoft System Center Operations
Manager
Microsoft System Center
Configuration Manager
VLAN 2
Hosts SharePoint Web Portal
Secondary Domain Controller with
DNS
Manages Server Incidents
VLAN 2
Operating System Deployment, Patch
Management and Software
Distribution
DNS Namespace
Local
Description
OTC.LOCAL
Global
OTC.com
Domain Controllers
Prime-SYD.OTC. LOCAL
Sec-SYD.OTC.LOCAL
Hosted by ISP
Solution Diagram:
Internet
CORE
Cisco 6500
Cisco 6500
Cisco 6500
Cisco 6500
AGGREGATION
Cisco 6500
Cisco 6500
VPN Concentrator
VPN Concentrator
NetRanger
NetRanger
ACCESS
Cisco 6500
Hybrid Cloud
Hybrid Cloud
Application Server
Hyper-v
Hyper-v
DELL EQUALLOGIC
SAN
SYDNEY DATACENTER
Cisco 6500
Requirement Understanding:
SKV consulting to perform complete analysis of OTC Virtualization infrastructure which includes
a)
b)
c)
d)
Microsoft Hyper-v environment
General Active Directory configuration
General Domain Name Service configuration
Networking validation.
SKV will run few of many commands on the servers to validate the output and recommend OTC the
best practices going further. SKV will raise any errors or configuration issues to OTC. OTC has to
provide required privileges to SKV consulting to perform checks on the physical and virtual
infrastructures.
OTC Tasks:
1.
2.
3.
4.
5.
6.
Data center hosting is performed by OTC Employees
Configuration of CISCO Switches, VLAN configuration is performed by OTC
Provision of Internet Protocol Addresses are provided to SKV Consultants by OTC
Firewall exception rules are performed by OTC
Server Maintenance is performed by OTC which includes Server Patch Management
Storage provisioning is performed by OTC which includes provision of LUNs and
Configuration of ISCSI on Windows Servers.
7. Communications between VLANs is provisioned by OTC
8. DR procedures are managed by 3rd party vendor
9. Private Namespace is hosted by OTC
ASSUMPTIONS:
a) The health checks on the Microsoft suite of products are based on Microsoft standards
b) SKV to run scripts where necessary
c) Any server failure during the validation is not owned by SKV and the issue needs to be fixed
by OCT consultants.
d) SKV will not do any modification to existing infrastructure but provide valuable guidance and
report to OTC Technical Director
PRE-CHECKS
Below sections would cover Pre-checks list for different Microsoft technologies which are in scope
DNS Health Checks:
Below are the DNS checks that will be performed by SKV consultant to understand the Name
Resolution in OTC environment.
a) Understand the DNS infrastructure by validating the Namespace information
b) Note the DNS servers zone information and the IPaddress information.
c) Understand how the client requests are getting routed covering both intranet namespace and
internet namespace.
d) Validate the response time for Recursive DNS requests either with the help of Powershell
script or using Microsoft System Center Operations Manager.
e) Check the average amount of queries that DNS server receives each second, this would assist
in validating the response times between server and client. Powershell script needs to be run
to validate the response time.
f) Validate whether ever Active directory integrated zones are loaded within specified time, this
would help virtualized client / servers to minimize the response time for DFS / Applications or
logon operations.
g) Verify if the DNS Server service is performing slow and connected to appropriate adapter.
Active Directory Health Checks:
Below are the Active Directory checks that will be performed by SKV consultant to understand the
Active Directory Infrastructure in OTC environment and report back any of the configuration issues
or the errors related to Active Directory infrastructure.
a) Collect the Active directory information such as Number of Forests, Number of Domains, and
number of child domains and Organizational Units respectively.
b) Register the namespace being used within Active Directory forest and between Forests.
c) Check for the Forest Functional Level and Domain Functional Level configured for OTC
d) Verify the Time service configuration.
e) Verify the accounts needs to perform the health checks activities have the required
permissions
f) Register the amount of RODC vs Writable Domain controllers.
g) Validate the Group policy infrastructure and ensure the policies are not misconfigured. Run
GPResult on clients ranging from different domains and validate from the GPResult for NO
Overide is disabled for all Active directory Node
h) Verify for Blocked GPO inheritance on the OU nodes. If found, register in the log which will be
submitted to OTC
i) Verify all Domain Controllers are listed under Domain Controller Organizational Unit.
j)
Test the availability of each domain controller , the SYSVOL share should be accessible to
every domain controller.
Network Inventory:
Networking plays key role in designing and implementing successful virtualization solution, hence it
is desired to collect the following information from OCT.
a) Detailed Network architectural diagram of OCT which should show the Network path
information from source to destination.
b) IP space information which includes IP ranges and subnet configuration
c) ACL information which helps understanding the rules defined on the network.
d) VLAN, VLAN Trunk, VLAN Prune, VLAN Port Assignment and VLAN Tags inventory
e) Hyper-v External , Internal and Private Networks information
f) Firewall Ports information
Microsoft Hyper-v Health Checks:
Below are the Microsoft Hyper-V checks that will be performed by SKV consultant to understand the
Hyper-v configuration and supportability to manage Virtual Infrastructure in OTC environment. The
issues will be reported back to OTC respectively.
a) Verify if the Microsoft Hyper-V installation is stand-alone or clustered.
b) Note the storage configuration for Microsoft Hyper-V virtual machines.
c) Verify the number of Internal , External and Private Virtual Networks configured. This would
include validating and documenting Networking interfaces along with IPAddress information
for all the Interfaces.
d) Validate the adapter bindings for appropriate virtual networks and validate the VLAN
configuration along with VLAN Tags.
e) Understand the Hyper-v installation method. Validate how the Hyper-v role was installed and
joined to the Active Directory Domain.
Note: If customer is running Domain Controller on the VM, then the ideal installation
should be
1) Install Hyper-v role on the Physical Machine
2) Create a VM and install Windows Server Operating System, promote the Server to
Domain controller
3) Join the Hyper-v server to the domain
4) (for cluster ) Install the Second Hyper-v server on second Physical server and join to the
AD domain
5) Install the secondary domain controller on Node 2
6) Form the Hyper-v Cluster
f)
g)
h)
i)
j)
k)
l)
m)
n)
o)
p)
q)
r)
Ensure the following services are present in the Service Control Manager / Services.msc
1) Hyper-v Data Exchange Service
2) Hyper-v Guest Shutdown Service
3) Hyper-v Heartbeat Service
4) Hyper-v Remote Desktop Virtualization service
5) Hyper-v Time Synchronization service
6) Hyper-v Virtual Machine Management
7) Hyper-v Volume Shadow Copy Requester
Ensure the Hyper-v Administrators local group is created ( check using lusrmgr.msc 
groups )
Verify the Integration services being used on the virtual machines and upgrade to the latest
Integration services on the virtual machines.
Check for the Hyper-v Virtual Machine Management service account which should be running
with Local system account. If customer using different account, it should be having local
administrator rights. Most of the other Hyper-v services are Manual – Trigger start )
Verify the memory assignment for the Virtual Machines and ensure Dynamic memory is set
for low productivity servers / client Virtual Machines.
Validate the Memory consumption on the Hyper-v host and on the Virtual Machines. Divide
the Virtual Machines into Heavy Usage vs Light usage and run the Performance monitor
diagnostics on the virtual machine and on servers.
Verify the storage capacity configured on the virtual machines and verify the IOPS between
the virtual machines and SAN or NAS storage and identify the bottlenecks. There should be
sufficient diskspace to start the virtual machines. If virtual machines are starting from SAN (
BOOT From SAN ) verify the delay and check for the Jumbo packet support.
For each Virtual Machine verify the processor type and its performance on selected virtual
machines.
Verify the Network performance on the virtual machines for each subnet. Configure Resource
Monitor to capture the performance statistics. Run Tracert to determine number of hops on
every subnet.
Validate the amount of snapshots being taken and the reason for the snapshot. Disable the
feature on Client operating systems and enable the snapshot only for those servers which are
high performance centric.
Check for unused virtual machines that are not in use and decommission them. This can be
done through System Center Virtual Machine management.
Validate the permission on the Hyper-v folders which contain Virtual Hard disks ( VHDs).
Perform strict access check validation to ensure only desired active directory group gets
access to the folder, when remove the permissions , document the groups and perform
testing.
If the guest Virtual Machines are using Teamed NIC ensure
1) Team network should not have any protocols bound to the network
2) Ensure the Microsoft virtual Network switch protocol is left blank
3) Connect the virtual machines to virtual switch which is connected to teamed NIC
s) Verify the Hyper-v Backup strategies and the backup solution for Snapshot management and
verify the snapshot chain length to determine only the valid snapshots are backed-up
t) Run a sanity check on the Audit logs to determine the errors and unauthorized connection to
Hyper-v server. Restrict the users / access management to the Hyper-v server.
u) To have isolated access to Hyper-v server, remove any instance of remote utilities configured
on client operating system to manage Hyper-v.
Conclusion: This document produces best practices to validate Hyper-v installation and
configuration in an Enterprise implementations. This document provide intermediate level check list
to validate Hyper-v configurations.