August 25, 2011 VMware vSphere Performance Monitoring and Troubleshooting

Thank you to everyone who came out today to join us at Sports Authority Field.  As a reminder, please let me/us know what we can do to help make you and your company successful.

  • VMware vSphere 5 architecture and design session, we’ve been part of the beta program now for many months and we’re ready to help you successfully upgrade or implement vSphere 5
  • vSphere Health Check, configuration drift happens, best practices changes, lets get your environment back on track
  • Onsite vSphere Performance Analysis session
  • POC or implementation of vCenter Operations, we’ve got it setup in our offices, you are all welcome to come poke around and see what it can do.
  • “Enlightened Enterprise”, not ready for cloud IAAS yet but interested in using some of the features being developed in public cloud infrastructures?
Thanks again for coming, follow me on twitter @dbrinkmann – see you all again soon!

Safari Based Malware (even on a Mac)

Saw this today where an end-user clicked on a safe looking site for a background picture and got infected with “browser malware”, which completely hi-jacks the browser where you cannot do anything until “accepting” their “survey”. They were using Safari v4 on a Mac.
The message stated to go take a survey from “” and locked the browser.

I found this very helpful discussion on the Apple forums, which had the resolution:

Steps to Solve:
1) Force Quit Safari
2) Disconnect from network / disable WiFi
3) Start Safari
4) “Accept” prompt that comes up (ok since you are not connected to internet)
5) Page will time-out, browser should be usable again
6) Click on Safari in the upper toolbar and Select Reset Safari.
7) Click at least (I’d recommend all checkmarks) Empty Cache and Remove All Cookies.
8) Click the Reset button.
9) Quit and re-launch the browser.

The above worked and resolved the issue in my situation. Hope it helps!

Maybe it’s a good time to go download FireFox and start using that browser ūüėČ

Cisco’s Latest UC 8.6 Solution opens up infrastructure options

Processor Requirements

Servers must have Intel Xeon 5600 or 7500 series of processors. No other processor vendors or models are supported.

Total physical core count required is based on the sum of UC virtual machine core requirements and the co-residency support policy).

Minimum physical core speed required is based on what UC virtual machines will be used, and at what intended load per VM. Processors of Tested Reference Configurations are sized for full-load virtual machines. It is recommended to use processors with same or higher speeds, as Cisco UC does not test or document lower performance points.

Recall that physical CPU cores may not be over-subscribed for UC VMs at this time (one physical CPU core = one vCPU core).

Cisco TAC will not troubleshoot performance problems in deployments with insufficient physical cores.



The only supported server vendors are:

Cisco Unified Computing System



Cisco UCS Express, Dell and all other server vendors are not supported at this time.

All servers used must be on the VMware Hardware Compatibility List for the version of ESXi you will be running, and must meet all other policy requirements such as required CPU.

Otherwise, any server model/generation from the above vendors that satisfies all other criteria of this policy is supported for UC.



Minimum physical RAM required is 2GB for ESXi plus the sum of UC virtual machines’ vRAM.

Recall that physical memory may not be over-subscribed for UC VMs.

Aside from total physical RAM, UC does not mandate memory module size, density, speed or quantity – follow server vendor requirements for memory hardware configuration.

Cisco TAC will not troubleshoot performance problems in deployments with insufficient physical RAM.


IO Devices

All I/O controllers and adapters used must be on the VMware Hardware Compatibility List for the version of ESXi you will be running.

Only the following I/O Devices are supported:


FC ‚Äď 2Gbps or faster



Ethernet ‚Äď 1Gbps or faster

NFS and iSCSI are supported, but require minimum 10Gbps and dedicated NIC for network storage access

Converged Network Adapter or Cisco VIC

FCoE + Ethernet ‚Äď 10Gbps or faster

RAID Controllers for DAS






Note that diskless servers for “boot from SAN” (FC, iSCSI, or FCoE) are only supported for UC if the UC app supports both ESXi 4.1 and the “boot from SAN” feature on the VMware Requirements page.

The customer is responsible for configuring an adequate number of I/O devices to handle the aggregate load that the virtual machines running on the server will generate.

Storage access I/O requirements for UC VMs are described in the IO Operations Per Second (IOPS) page..

LAN access I/O requirements for UC VMs are described in the UC application design guides. See also network link sizing and QoS considerations here.

The customer is also responsible for configuring redundant interfaces on the server to handle component failures (e.g. redundant NIC, CNA, HBA or VIC adapters.)

There are no UC restrictions on hardware vendors for I/O Devices other than that VMware and the server vendor/model must both support them.

Cisco TAC will not troubleshoot performance problems in a deployment designed with insufficient I/O devices or overloaded I/O devices. For example, a single 100Mbps NIC servicing eight “CUCM 7500 user OVAs” would be both insufficent and overloaded.



Each OVA provided by Cisco for running a UC application has a published IOPS and disk space requirement. It is the responsibility of the customer to provide a storage system that exceeds the disk space (see Unified Communications Virtualization Downloads (including OVA/OVF Templates) and average IOPS requirements (see IO Operations Per Second (IOPS)) of the UC virtual machines they will be running on that storage system.

If you are using NFS, iSCSI, or FCoE for storage connectivity, the networking configuration must provide Cisco Platinum Class QOS (Fiber Channel Equivalent):

See also Shared Storage Considerations here.

It is not necessary to configure the storage to handle the simultaneous maximum IOPS load of every virtual machine on the storage system, but the customer must be aware of the excess capacity of the storage system and not, for example, run multiple software upgrades on the virtual machines such that the storage system is over extended.

The kernel disk command latency must not be greater than 2-3 ms and the physical device command latency must not be greater than 15-20 ms. When either of these metrics is not met, Cisco considers the storage system inadequate to serve the UC virtual machines. Cisco will not troubleshoot performance problems in an environment where either metric is not being met.

As a guideline, Cisco has found the use of 15K rpm SAS or FC drives in a RAID 5 configuration to work well. The number of drives used in the array is 5. The recommended size of the hard drives is 300 to 450GB. Recommended LUN size is 500GB to 1.5TB, so that not more than 10 virtual machines reside on a LUN – preferably 8 or less.

This is only a guideline, it is left to the customer to configure their storage for adequate performance and for the redundancy level desired.



NetBackup 7.1 NDMP Gotcha

A new feature introduced in NetBackup 7.1 can cause NDMP backup failures. The Append-Only Mode described in this Symantec Tech Note causes the drive to prevent the overwrite of data on a cartridge unless the writing application performs a pre-authorization for the overwrite.  This feature is supported on IBM LTO-5 drives  with LTO-4 or LTO-5 media.  The feature is enabled by default and does not normally cause any issues if all systems accessing the tape device are NetBackup 7.1 Media Servers.

In an NDMP environment, however, the storage device (filer, data mover) writes directly to the tape drives and may not be aware of the pre-authorization requirement.  If these drives are shared between the NDMP device and the Netbackup Media Server, write errors occur, represented by a write protect message issued by the NDMP device.  This has been observed with an EMC VNX array and EMC support was previously unaware of the Append-Mode feature.

The solution to the problem is to disable Append-Only mode by touching the <InstallPath>NetbackupdbconfigDISABLE_APPEND_MODE on the Netbackup Master Server and restarting the NetBackup Services.