Benefits of Cisco ACI (SDN) architecture

Cisco ACI, Cisco’s software-defined networking (SDN) architecture, enhances business agility, reduces TCO, automates IT tasks, and accelerates data center application deployments.

Why Today’s Solutions Are Insufficient:

Today’s solutions lack an application-centric approach. The use of virtual overlays on top of physical layers has increased complexity by adding policies, services, and devices.

Traditional SDN solutions are network centric and based on constructs that replicate networking functions that already exist.

ACI Key Benefits:

Centralized Policy-Defined Automation Management

• Holistic application-based solution that delivers flexibility and automation for agile IT
• Automatic fabric deployment and configuration with single point of management
• Automation of repetitive tasks, reducing configuration errors

Real-Time Visibility and Application Health Score

• Centralized real-time health monitoring of physical and virtual networks
• Instant visibility into application performance combined with intelligent placement decisions
• Faster troubleshooting for day-2 operation

Open and Comprehensive End-to-End Security

• Open APIs, open standards, and open source elements that enable software flexibility for DevOps teams, and firewall and application delivery controller (ADC) ecosystem partner integration
• Automatic capture of all configuration changes integrated with existing audit and compliance tracking solutions
• Detailed role-based access control (RBAC) with fine-grained fabric segmentation

Application Agility

•  Management of application lifecycle from development, to deployment, to decommissioning—in minutes
• Automatic application deployment and faster provisioning based on predefined profiles
• Continuous and rapid delivery of virtualized and distributed applications

ACI Technology Benefits

The main purpose of a datacenter fabric is to move traffic from physical and virtualized servers, bring it in the best possible way to its destination and while doing so apply meaningful services such as:

• Traffic optimization that improves application performance
• Telemetry services that go beyond classic port counters
• Overall health monitoring for what constitutes an application
• Applying security rules embedded with forwarding

The main benefits of using a Cisco ACI fabric are the following:

•  Single point of provisioning either via GUI or via REST API
• Connectivity for physical and virtual workloads with complete visibility on virtual machine traffic
• Hypervisors compatibility and integration without the need to add software to the hypervisor
• Ease (and speed) of deployment
• Simplicity of automation
• Multitenancy (network slicing)
• Capability to create portable configuration templates
• Hardware-based security
• Elimination of flooding from the fabric
• Ease of mapping application architectures into the networking configuration
• Capability to insert and automate firewall, load balancers and other L4-7 services
• Intuitive and easy configuration process

More information can be found at www.cisco.com/go/aci

Disk Physics

Today’s topic – Disk Performance.  A warning to the squeamish ..Math ahead.

Throughput refers to the amount of data  read or written per unit of time.  Generally measured in units like Megabytes per second (MB/s), or Gigabytes per Hours (GB/h).  Often when dealing with networks we see Kilobits per second (Kb/s) or Megabits per second (Mb/s).   Note that the abbreviations of some of those units look similar, pay attention to the capitalization because the differences are a factor of at least 8x.

It’s easy to talk about a hard drive, or a backup job, or even a network interface providing 250MB/s throughput and understand that if I have 500GB of data that it’s going to take a little over a half hour to transfer the data. (500GB * 1024MB/GB / 250MB/s / 3600s/h = 0.56h)

Throughput is by far the most talked about and in general most understood measure of performance.   By the same token when people ask about network performance they often go to speedtest.com and tell me that “my network is fast, because I get 15Mb/s download.”  I agree that’s a pretty decent throughput, but that’s not the only measure of  performance that’s important.

A second measure is Response Time (or latency).   This is a measure of how long it takes a request to complete.   In the network world we think about this being how long it takes a message to arrive at it’s destination after being sent.   In the disk world we think about how long from when we request an IO operation happen until the system completes it.  Disk latency (and network latency) are often measured in direct units of time – milliseconds (ms) or microseconds (us), and occasionally in seconds (s).   Hopefully you never see IT technology latency measured in hours or days unless you’re using an RFC1149 circuit.

The combination of a request response time and throughput, combined with the size of the request (amount of data moved at a time) yields a metric which amounts to how many requests can be completed per unit of time.  We see this most often in the disk world as I/O operations per second or IOPS.   We talk about IOPS a way of thinking about how “fast” a given disk system is, but it’s arguably more of a measure of workload capability than either latency or throughput; however both latency and throughput contribute to the maximum IO operations per second a given disk can mange.

For example – if we have a hard disk with an maximum physical throughput of 125MB/sec, which is capable of processing requests at a rate of 80 requests per second, what is the throughput of the drive if my workload consists of 4KB reads and writes?  Well in theory at 125MB/sec throughput the drive could process 125MB/s * 1024KB/MB / 4KB/IO = 32,000 IO/s.   Hold on, the drive is only capable of 80 IOPS so the maximum physical throughput won’t be achieved.  80IO/s * 4KB/IO = 320KB/s.   If we wanted to maximize this drive’s throughput we need to increase the size (or payload) of the IO requests.  Ideally we’d perform reads an writes in blocks equal to the maximum throughput divided by the maximum IO rate (125MB/s / 80IO/s = 1.562MB).

This last trick by the way is what many vendors use to improve the performance of relatively “slow” hard disks; referred to as IO coalescing they take many small IO operations and buffer them until they can perform one large physical IO.

What governs the drive’s maximum IOPS is actually a function of multiple factors.

Physical media throughput is one of them – which is governed by the physical block size (often 512bytes), the number of blocks per track on the platter, the number of platters, and the rotational velocity of the drive (typically measured in revolutions per minute – RPM).   The idea here being that the drive can only transfer data to or from the platters at the rate at which the data is moving under the heads.   If we have a drive spinning at 7200RPM, with say 100 blocks per track, and 512bytes/block and a single platter/head we have a drive with a maximum physical media transfer rate of 512B/block * 100blocks/track * 7200 tracks/minute / 60seconds/minute / 1024bytes/KB / 1024KB/MB = 5.85MB/s.  Under no circumstances can the physical throughput of the drive exceed this value, because the data simply isn’t passing under the head any faster.

To improve this value you can add more heads (with two platters and 4 heads this drive could potentially move 23.4MB/s). You can increase the number of blocks per track (with 200 blocks per track and one head the drive would have a throughput of 11.7MB/s). Or you can increase the velocity at which the drive spins (at 36,000 RPM this drive would move 29.25MB/sec).  As you can see though this maximum throughput is governed by the physical characteristics of the drive.

A second factor impacting the IOPS is the question of how long it takes to position the head to read or write a particular block from the disk.  IO operations start at a given block and then proceed to read or write subsequent sequential blocks until the size of the IO request has been fulfilled.  So on our sample drive above a 4KB request is going to read or write 8 adjacent (sequential) blocks.  We know what the physical transfer rate for the drive is, but how long does it take to physically move the mechanism so that the 8 blocks we care about will pass under the head?   Two things have to happen, first we have to position the head over the right track, and then we have to wait for the right block to pass under the head.  This is the combination of “seek time” and “rotational latency”.   Our 7200RPM drive completes one revolution every (7200RPM / 60 seconds/minute = 120 revolutions/second or every 120th of a second or 0.00833 seconds or 8.33 milliseconds).  On average then every IO operation will take 4.16ms to start performing IO after the head are aligned.  Again we can reduce the rotational latency by spinning the drive faster.  Seek time (how long it takes to align the heads) varies by drive, but if it takes 6ms then the average physical access time for the drive would be 10.15ms.   Drives which are physically smaller will have to move the heads shorter distances and will have lower seek times, and therefore lower access times.  Larger drives, or drives with heavier head assemblies (more heads) will have higher seek times. For a given drive you can typically look up the manufacture’s specs to see what the average seek time is.  So, let’s say that it takes 10ms to typically position a head and read a block, then our drive could potentially position the head 100 times per second.  That means the maximum IOPS for this drive is 100 per second.

So, IOPS is governed by physics, and throughput (from media) is governed by physics.   What else is a factor?   There are additional latencies introduced by controllers, interfaces, and other electronics.  Generally these are fairly small, measured in micro-seconds (us) relative to the latencies we’ve talked about generally become negligible.   The other side is physical interface throughput   ATA-133 for instance had a maximum throughput on the channel of 133MB/s; where Ultra-160 SCSI was limited to 160MB/s.  The maximum throughput of a given drive will be limited to the throughput of the channel to which it’s attached.  The old ATA and SCSI interfaces noted earlier also attached multiple devices to a channel which limited the sum of all devices to the bandwidth of the channel.  Newer SAS and SATA architectures generally dedicated a channel per device, however the use of expanders serves to connect multiple devices to the same channel.  The net of this being that if you have 10 devices at 25MB/sec throughput each connected to a channel with a maximum throughput of 125MB/sec then the maximum throughput you’ll see is 125MB/sec.

So that covers what governs the speed of a hard disk.   Some may ask “what makes SSD’s so fast?”  The short answer is that it’s because they aren’t spinning magnetic devices, and therefore don’t have the same physical limits.   The long answer is a topic for another blog.

Why is a smaller number of virtual CPUs better?

Note: This article is designed to serve as a high level introduction to the topic and as such uses a very basic explanation. Papers for those that wish to dive into more technical details of the topic are available elsewhere.

In a virtual environment such as VMware or Hyper-V, multiple virtual machines (VMs) operate on the same physical hardware. In order to make this function, a small piece of software, called a hypervisor operates to schedule the virtual resources with the physical hardware. As a virtual machine enters a state where CPU resources are required the VM is placed into a CPU ready state until enough physical CPUs are available to match the number of virtual CPUs.

The hypervisor will schedule VMs to available physical resources until all resources that can be scheduled are used.

Each VM will run on the physical CPUs until either it needs to wait for an I/O operation or the VM uses up its time slice. At that point the VM will either be placed into the I/O wait state until the I/O completes or be placed back in the ready queue, waiting for available physical resources.

As physical resources become available, they hypervisor will schedule VMs to run on those resources. In some cases, not all physical resources will be in use, due to the number of virtual CPUs required by the VMs in the ready state.

The process continues as VMs either wait for I/O or use their time slice on the physical CPUs.

In some cases there are no VMs in the ready state, at which point the scheduled VM will not time out until another VM requires the resources

Often a VM with fewer virtual CPUs will be able to be scheduled before one with more virtual CPUs due to resource availability.

In some cases a VM will complete an I/O operation and immediately be scheduled on available physical resources.

Algorithms are in place to ensure that no VM completely starves for CPU resources but the VMs with more virtual CPUs will be scheduled less frequently and will also impact the amount of time the smaller VMs can utilize the physical resources.

A VM with high CPU utilization and little I/O will move between the ready queue and running on the CPUs more frequently. In this case, the operating system will report high CPU utilization, even though the VM may not be running for a majority of the real time involved.

In these situations, operating system tools that run within the VM may indicate that more CPUs are required when, in reality, the opposite is actually the case. A combination of metrics at the hypervisor and at the operating system level is usually required to truly understand the underlying issues.

A brief history of storage

I’ve told several groups I’ve spoken to recently that “disk storage hasn’t gotten faster in in 15 years.” Often that statement is met with some disbelief. I thought I’d take a few paragraphs and explain my reasoning.

First – Lets cover some timeline about the evolution of spinning disk storage.

• 7200 RPM HDD introduced by Seagate in 1992
• 10,000 RPM HDD introduced by Seagate in 1996
• 15,000 RPM HDD introduced by Seagate in 2000
• Serial ATA introduced in 2002
• Serial Attached SCSI introduced 2004
• 15,000 RPM SAS HDD ships in 2005

So, my argument starts with the idea that this is 2015, and the “fastest” hard disk I can buy today is still only 15,000 RPM, and those have been shipping since 2000.  Yes, capacities have gotten larger, data densities greater, but they have not increased in rotational speed, and hence have not significantly increased in terms of IOPS.

To be fair, the performance of a drive is a function of several variables, rotational latency (the time for a platter complete one revolution) is just one measure.  Head seek time is another measure.  As is the number of bits which pass under the head(s) in a straight line per second.

Greater data densities will increase the amount of data on a given cylinder for a drive, and thus increase the amount of data that can be read or written per revolution – So you could argue that throughput may have increased as a function of greater density.   But only if you don’t have to re-position the head, and only if you are reading most of a full cylinder.  I also submit that the greater densities have lead to drives having fewer platters and thus fewer heads.  This leads to my conclusion that the reduction in drive size mostly offsets any significant increased throughput due to the greater densities.

Today we’re seeing a tendency towards 2.5″ and sometimes even 1.8″ drives.   These form factors have a potential to increase IO potential by decreasing seek times for the heads.   Basically the smaller drive has a shorter head stroke distance and thus potentially will take less time to move the head between tracks.   The theory is sound, but unfortunately the seek latency is typically much lower than the rotational latency; so the head gets there faster, but is still waiting for the proper sector to arrive as the disk spins.

Interestingly some manufacturers used to take advantage of a variable number of sectors per track and recognized that the outer tracks held more sectors.  For this reason they would use the outer 1/3 of the platter for “fast track” operations looking to minimize the head seek time and maximize the sequential throughput.   Again a sound theory, but the move from 3.5″ to 2.5″ drives eliminates this faster 1/3 of the platter.  Again, negating any gains we may have made.

Another interesting trend in disk storage is a movement to phase out 15,000RPM drives.  These disks are much more power hungry, and thus produce more heat than their slower (10,000RPM and 7,200RPM) counterparts.   Heat eventually equates to failure.  Likewise the rest of the tolerances in the faster drives are much tighter.   This results in faster drives having shorter service lives and being more expensive.   For those reasons (and the availability of flash memory) many storage vendors are looking to discontinue shipping of 15,000RPM disks. A 10K drive has only 66% of the IOP potential of a 15K drive.

So I submit that any gains we’ve had in the last 15 years in spinning disk performance have largely be offset by the changes in form factor.   Spinning disk hasn’t gotten faster in 15 years.  The moves towards 2.5″ and 10K drives could arguably suggest that disks are actually getting slower.

So IO demands for performance are getting greater.   VDI, Big Data Analytics, Consolidation and other trends demand more data and faster response times.  How do we address this?  Many would say the answer is flash memory, often in the form of Solid State Disk (SSD).

SSD storage is not exactly new

• 1991 SanDisk sold a 20MB SSD for $1000
• 1995 M-Systems introduced Flash based SSD
• 1999 BiTMICRO announced a 18GB SSD
• 2007 Fusion IO PCIe @ 320GB and 100,000 IOPS
• 2008 EMC offers SSD in Symmetrix DMX
• 2008 SUN Storage 7000 offers SSD storage
• 2009 OCZ demonstrates a 1TB flash SSD
• 2010 Seagate offers Hybrid SSD/7.2K HDD
• 2014 IBM announces X6 with Flash on DIMM

But Flash memory isn’t without it’s flaws.

We know that a given flash device has a finite lifespan measured in write-cycles.  This means that every time you write to a flash device you’re wearing it out.  Much like turning on a light bulb, each time you change the state of a bit you’ve consumed a cycle. Do it enough and you’ll eventually consume them all.

Worse is that the smaller the storage cells used for flash are (and thus the greater the memory density) the shorter the lifespan.   This means that the highest capacity flash drives will sustain the fewest number of writes per cell.   Of course they have more cells so there is an argument that the drive may actually sustain a larger total number of writes before all the cells are burned out.

But… Flash gives us fantastic performance.   And in terms of dollars per IOP flash has a much lower cost than spinning disk.

DRAM memory (volatile) hasn’t gone anywhere either – in fact it keeps increasing in it’s own densities and reduced cost per GB.  DRAM doesn’t have the wear limit issue of Flash, nor the latencies associated with Disk. However it suffers from it’s inability to store data without power. If DRAM doesn’t have it’s charges refreshed periodically (every few milliseconds) it will loose whatever it’s storing.

Spinning disk capacities keep growing, and getting cheaper.  In December of 2014 Engadget announced that Seagate was now shipping 8TB hard disks for $260.

So the ultimate answer (for today) is that we need to use Flash or DRAM for performance and spinning disk (which doesn’t wear out from being written to, or forget everything when the lights go out) for capacity and data integrity.  Thus the best overall value comes from solutions which combine technologies to their best use.  The best options don’t ask you to create pools of storage of each type, but allow you to create unified storage pools which automatically store data optimally based on how it’s being used.

This is the future of storage.