Disaster Recovery Event was a Great Success

Thank you to all who attended the event last week at Fortrust where we discussed how we help clients with the DR planning, implementation ongoing management.  We discussed premise solutions and DRaaS (DR to our cloud infrastructure) solutions as well. Thanks also to out partners: Fortrust, Faction and Zerto for participating.  Many of you have asked for the presentation and I’ve posted it here for your convenience.

DR Preso Setup (customer preso)

Screen Shot 2015-05-11 at 11.20.15 AM

Screen Shot 2015-05-11 at 11.17.21 AM

Do you have systems running Microsoft Windows 2003 that you don’t even know about?


Not only do folks need to worry about the Windows 2003 systems that support the business applications they use, but also be on the lookout for “embedded systems” that may be lurking within your environment that you don’t really consider application server, e.g., phone systems, alarm systems, management systems, etc…

If you’d like some help in discovering your Windows 2003 exposure please reach out to us sooner rather than later.  July will be here before you know it and those who plan early will get the resources that are available to assist.  Those who don’t….won’t.

A problem we’re going to face here shortly in the Cisco UC world. Time to prepare…The Coming Certificate SAN Nightmare – How it Affects Jabber and Cisco UC

A good blog post on a problem we’re going to face here shortly in the Cisco UC world.  Time to prepare…  If you need help, give us a ring.


Cisco Collab Engineering Tips

Mike White

The Coming Certificate SAN Nightmare – How it Affects Jabber and Cisco UC

The coming storm is here:

The public CAs are no longer signing certificates with subject alternative names (SAN) for internal server names — (https://www.digicert.com/internal-names.htm).

An excerpt:

An internal name is a domain or IP address that is part of a private network. Common examples of internal names are:

Any server name with a non-public domain name suffix. For example, http://www.contoso.local or server1.contoso.internal.

NetBIOS names or short hostnames, anything without a public domain. For example, Web1, ExchCAS1, or Frodo.

Any IPv4 address in the RFC 1918 range.

Any IPv6 address in the RFC 4193 range.

Why do we care? 

Jabber authenticates TLS encryption using certificates for services from CUCM, CUC, IM&P, etc.   Historically these have typically been deployed with IP addresses only, or internal domains (e.g. domain.local, etc.).  Because of this you can no longer get a certificate for the Expressway-C box that has SANs with IPs or internal names.  Jabber requires valid certificates for login now.

See the Expressway Certificate guide p.7 for Expressway-C here – http://www.cisco.com/c/dam/en/us/td/docs/voice_ip_comm/expressway/config_guide/X8-2/Cisco-Expressway-Certificate-Creation-and-Use-Deployment-Guide-X8-2.pdf

Without a certificate with proper SANs, Jabber will either throw an invalid cert error, or will completely deny login to UC services.

Using Collab Edge MRA,  Jabber authenticates to the Expressway-E server and uses it’s certifciate.  Internally Jabber communicates directly with each component.

Dealing with the issue for Collab Edge MRA

Basically we have two options to work around the SAN issue:

1) Change the domain name of the UC components to a valid public domain name that the public CA will sign for.  This doesn’t mean the server has to be accessible from the internet by any means or that it is an existing domain name your company is using.

Option 1a:  Deploy a new public domain name for UC services internally.  For example if your domain name was domain.com you might see if domain.info or domain.net or something similar is available to register and use as the internal UC domain name.  The domain wouldn’t need to resolve externally at all.

If you do this, then you need to take in to consideration that the MRA deployment becomes a multi-domain (or split-domain) deployment which requires some special treatment like the VoiceServicesDomain option.  (See my previous post about multi-domain deployments.)

Configuration example here – http://www.cisco.com/c/en/us/support/docs/unified-communications/expressway-series/117811-configure-vcs-00.html

Options 1b:  The seemingly easier deployment would be to just match your public domain name that you use for email (e.g. domain.com) for your UC components (not suggesting all internal servers — file, print or otherwise, need to be in this domain).  This makes services discovery nice and clean.

The challenge to this method is usually the need to deploy a split DNS for internal and external name resolution.  (The internal DNS server also serving the domain.com zone and having the A records for internal services, where the external DNS server have A records for external services.)

2) Create certs using your own internal CA, like Microsoft AD Certificate Services, or OpenSSL, etc.  There are no restrictions on SANs with your own certificate server.  I detail how to use OpenSSL to sign certs in an earlier post.

The major constraint to this deployment option is the need to get the trusted cert from your CA server on to all devices that will use MRA.  AD does it for your Windows machines automatically, but mobile devices will need to have this certificate installed.  Using an MDM like Meraki MDM (freemium service) or others to push the certificates would be the way I’d attempt to deploy the certificates

The Implications of changing the domain name of CUCM/CUC/IM&P

Anyone who’s attempted to change the hostname of a CallManager knows the trainwreck and ensuing TAC calls that will ensue.

I’ve personally not tried to change the domain name of a CallManager or CUC in recent memory, but doing so for IM&P/CUP is relatively straightforward.

The hostname/domain name change procedure is here for CUCM/IM&P – http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/install/10_0_1/ipchange/CUCM_BK_C3782AAB_00_change-ipaddress-hostname-100.html

The name change procedure is here for CUC – http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/connection/9x/upgrade/guide/9xcucrugx/9xcucrug060.html

I’d do this with a healthy amount of trepidation.  🙂

VMTurbo, a unique way to better control your virtualized environments

Been trying out this VMTurbo stuff and it’s definitely a unique approach versus traditional systems management. It’s more actionable and automated versus telling me a bunch of information that I would have to decipher and deal with.  Both approaches have merit but it depends on how you’d like to control your virtualization environment that makes the difference.  Give it a try and let me know your thoughts.
What Is Control Watch Video
Why You Need Control Watch Video
How We Give Control Watch Video

Give it a try: VMTurbo Download

the disk is full trying to write to macintosh hd – Microsoft Word 2011 for Mac

From our internal blog.


Experienced this issue today. Apparently updating a word document with “track changes” can cause Word to error saying that the Macintosh HD is full. (Which it is NOT). http://answers.microsoft.com/en-us/mac/forum/macoffice2011-macword/the-disk-is-full-trying-to-write-to-macintosh-hd/8284db3c-bfa1-4aec-ad51-a97f5c134e48

the disk is full trying to write to macintosh hd

Word for Mac 2011 keeps giving me the following message: “the disk is full trying to write to macintosh hd”, followed by a message saying that autorecover cannot save the file to the chosen location.

Problem using the SD cards for ESXi boot on UCS B200 M3 blades

I found this conversation on an internal blog our SE’s use and thought the outside world might find it helpful.

I have not been able to find any formal Cisco documentation on this issue, but I encountered a problem using the SD cards for ESXi boot on B200 M3 blades.  After enabling FlexFlash and adding the SD card to the boot order per the Cisco documentation, and using a ‘RAID1’ Local Disk Configuration Policy to enable the use of local hard drives in that configuration, the resulting service profile deployment had an error that FlexFlash RAID was in a degraded state.  By changing the local disk config policy to ‘any configuration’ and setting a scrub policy on the SD cards, and re-acknowledging the blade a couple times (which unfortunately wiped out our original ESXi installation), we were able to overcome this error: http://quraishi.wordpress.com/2014/08/21/cisco-ucs-flexflash-configuration-of-secure-digital-sd-card/

Ibrahim Quraishi

Cisco UCS FlexFlash /SDCard Configuration for Booting Server Operating System.
As most of the Cisco UCS users might be aware that Cisco has now add a lot of improvements and new features on the latest upgrade 2.2(1e).
One of the feature which we have started using is Cisco FlexFlash which is an SD Card. Its fully support if you buy the SD  Card from Cisco and we can even do mirroring on the Flash cards for redundancy, we will be installing ESXi Server 5.5 on these blades after mirroring them.
So where and How to install the Secure Digital (SD) Card  ?
If the CISCO UCS B200 Blade is in the Chassis you will need to power it off and slide it out just half way through and you should find the SD Card slot on the left

for more information on how about this can be found on the following link http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/blade-servers/B200M3.html
As you might be aware that CISCO UCS blade is just as equal to a brick and all the configuration is done on the logical server called Service profile. you will need to create a new  / clone an existing Service profile template and modify with the below changes which I have shown below on this post.
You will have to create a boot policy with CD Drive (legacy ) and SD Card and Some time you may end up getting this error ‘ advanced boot or secure boot configuration cannot be applied to the specified server There are not enough resources overall ‘ if you see it follow my blog post.
I have to make you aware that if you want to use SD card with Mirroring or RAID 1 you have to create the following new policies and create a Boot policy as mentioned above.
1. Disk policy with Mirror and any configuration
2.   We have to create two Scrub policy under policy ( you can call it any thing you want which makes you understand what configuration you have selected)
Flex Flash Scrub
No Scrub
So where will you find the Flex Flash Controller?
Click on the Left pan on Equipment and Select the Chassis then the Server, on the right you will find the below details
click on Inventory and select the last tab which is Storage on the Storage tab you will find FlexFlash Controller
Note: this FlexFlash is only support if you are running 2.2 (d1) firmware.
Once you assign the new service profile policy, we will see the below error on the FlexFlaash Cards
Go to the service Profile and select the Policies and change the Scrub policy
in this create 2 policy the default is no scrub but you can create one called FlexFlash Scrub and the  other one No Scrub
To create the scrub policy click on Service profile of the server and click Policy > Scrub and click Create Scrub Policy

You will have to use the first policy for Flex flash Scrub which should have the below settings.
FlexFlash Scrub: should set to Yes
You then need to create a new policy for No-Scrub as shown below :

After creating two policy this is the tricky bit to have the SD card as  Mirror you will need to do two reboots
first set the scrub policy to FlexFlash-Scrub ( note this will scrub the SD cards)  and click Save

Do the following steps:

Go to the server and Click server Maintenance this will scrub the flash
Reboot the server  in the following manor, go the the Equipment tab select the Chassis and the server Click on ‘Server Maintenance’ and click on ‘Re-acknowledge’ server
Acknowledge for Reboot
You will find the rediscovery
Once its configured go back to service profile
Go to > Polices
Select Scrub Policy and change it to No-Scrub from the drop down menu
5. Go to the server and Click server Maintenance this will Mirror and pair the SD card to Raid1/ Mirroring
6. Reboot the server  in the following manor, go the the Equipment tab select the Chassis and the server Click on ‘Server Maintenance’ and click on ‘Re-acknowledge’ server
Acknowledge for Reboot

After the server does the  rediscovery and boots successfully, You will see that this is successfully paired with is RAID 1 Mirroring.
Please note the Raid Status : Enabled Paired

Error message if the Raid is not working, this means you have to go through the process again. always make sure you only change the scrub policy on the specific server profile and not the service profile template as this will create problems on the working profiles.
Another example of error is Raid Status Disabled

How to test ?
The only confirmation we can get is on the below image where it says Raid Status Enabled Paired, however I have tested swapping the SD cards out one by one and the ESXi  Server boots successfully you will immediately see the error message that the pair is disabled. Unfortunately you will need to pull the server out to change swap the SD cards but hay at least you will find out if the mirroring is working.

Also you will need to remember that after installing ESXi on SD card will give you an error message on the ESXi host that  ‘The ESXi host does not have persistent storage’  you will need to add the storage and  a scrach partition.
To create a scrach partition follow the below VMware KB.
or I might create a blog on how to do it later…
My friend Vikram has just posted the process on his blog http://www.viktec.com System logs on host are stored on non-persistent storage
Please find the new updates the latest firmware is bringing.  Release 2.2(1b) adds support for the following:
IPv6 Management Support
Cisco Integrated Management Controller (CIMC) In-band Management
Fabric scaling: VLAN, VIFs, IGMP, Network Adapter Endpoints
Uni-Directional Link Detection (UDLD) Support
User Space NIC (usNIC) for Low Latency
Support for Virtual Machine Queue (VMQ)
C-Series Servers Direct Connect to FI without FEX
Two-factor Authentication for UCS Manager Logins
VM-FEX for Hyper-V Management with Microsoft SCVMM
Direct KVM Access
Server Firmware Auto Sync
Enhanced Local Storage Management
Flash Adapters and HDD Firmware Management
Precision Boot Order Control
Secure Boot
UEFI Boot Support
FlexFlash (Local SD card) Support
Trusted Platform Module (TPM) Inventory
DIMM Blacklisting and Correctable Error Reporting
C-Series Board Controller Firmware Management

Additional resources related to this post:

If you want to learn more about the process of doing a Firmware update please checkout my link CISCO UCS Firmware Update Process
If you want to learn more about the process of doing a Firmware update please checkout my link advanced boot or secure boot configuration cannot be applied to the specified server There are not enough resources overall

Unfortunately for me, this meant reinstalling and reconfiguring (4) ESXi hosts all over again… hopefully it will save someone else a little pain.