Indianapolis was a fantastic place for the big event, handled extremely well (and mucho friendly, which cannot be easy to do to a bunch of hated Pats fans).  My only complaint is that there are only 437 cabs in Indianapolis, and there were 165,000 people.  Oh yeah, and the bus system was fictitious.  I did not see one in 3 days.  Otherwise, it was superbly done.  Hats off.

At an average face value of $1,000 per seat, the NFL alone generated $68,000,000 just from ticket sales.  Since no one gets to buy those, the real ticket income was about $250,000,000 more than that.  That’s a lot of money.  They were processing “big data” transactions all over the place.  They never seemed to run out of whatever was a hot seller, anywhere.

This woman didn’t know who Aaron Rodgers is, clearly, as when I told her I was he, she said, “well, welcome to Indianapolis, Mr. Rodgers!”

I got Katy Perry to say hi to me.  She was really quite attractive.

I got David Arquette to shake my hand.  He was dressed like an insane asylum patient.

If you pause Eli’s Disney commercial when Manningham makes that catch and look to the left (goal line), you’ll fleetingly see me.  Mostly you’ll see my wife, but I’m the little guy next to her.  Trust me.

Enough of that.

EMC announced project “Lightning” yesterday, and LOTS of companies are scrambling to explain why no one should care.  You should care.  EMC could put a wad of gum in a box and sell $300M worth, so don’t try to tell me anything they attempt to do that is disruptive doesn’t matter.

There has been analysis ad nauseum to this announcement, but here’s what I’m not hearing yet–while this isn’t the perfect end all, (EMC isn’t claiming it to be), it is a kick in the head to Fusion-io to start.  It will kick others in other parts later on when they do all the things they talked about – namely:

• There is an excellent reason to have really fast flash stuff at every  level of the infrastructure – the server, the network, and the storage.
• The chips on the boards are far less relevant than the orchestration/control elements that are smart enough to put the right data on the right hunk of flash at the right time.  This is what they expect FAST to be.
• In a perfect world, all the data you ever need to access would be at the server.  Since life isn’t perfect, you need tiers.  Deal with it.  Tiers aren’t the problem.  Managing optimized data placement is the issue.  People can’t put their entire data set in each and every server, no matter what anyone says.  You need bulk in back, and then super fast, deterministic placement of those bulk bits onto the fastest chip you can put it on – as close to the processor as you can put it.

I have to say I told you so–at least I have to agree with this strategy, since I wrote it last April in this blog.

To think that this (albeit beginning) entry is even remotely good for Fusion-io is nuts.

It won’t bode well for any of the new generation of Flash in the Pans (sorry, it was too easy. And name their big one hit wonder – Walking In The Rain.  Thank you.  Goodnight.).  Will EMC kill the market?  No. Will they kill 85% of the nouveau flashers?  Yep.

I think their concept of how it needs to play out is accurate.  You simply can’t ONLY be a point in the server, any more than you can ONLY be at the back of the wire.  You need to orchestrate and control all points of performance.

Not sure if they spoke about this, but you can stick the EMC VFCache in front of ANY storage – which is also super interesting.  IBM, NTAP, Dell, etc., won’t be none too thrilled to have the evil E running around telling those customers that “we can make your NTAP box go fast.”  It also means that it won’t be long before all the big guns come out with their own stuff – which will further marginalize some of the little guys.  It also may mean some bigger fish will buy up some of the little guys earlier than normal.

No matter what, it’s interesting.

Speaking of EMC, I ran into some old friends at the game.  The kind of friends that were so dumb that they didn’t know EMC could never make it in 1989 and elected to stay.  The kind of friends that are silly rich and successful.

One of them is Billy Scannell.  Billy was a sales guy in 1986.  He brought me on my first ever sales call (required for a class I was taking in college) when at EMC.  He convinced me to leave my fantastic college job, take a pay cut, and work for him as his telemarketer.  Billy may be the greatest salesperson/executive ever.  He has made more money than god, held every major sales position at EMC, and been one of the biggest success stories in a company loaded with them.  I’m much smarter than Billy.  I find myself far more attractive than Billy.  I wish I were Billy.

It was nice to run into past and present EMC 1986ers, Bob Scordino (EMC), Larry Murray(EMC), Peter Bell (Storage Networks, now Highland Capital), Randy Seidl (Runs America’s for HP ESSN under another alumni, Dave Donatelli), and John McCarthy (far as I can tell he sits on any board that asks and plays golf now).  Pretty sure I’m smarter than all of them, richer than none of them.  It’s ok, all that money would have turned me into a real A*hole.  (sarcasm.  clearly I already am an a*hole.  I just am a poor one).

I cannot express enough happiness that Madonna did not attempt to pull off “Like a Virgin” at age 76.  There is a god.

Speaking of God – there were a ton of bible thumping street preachers trying to outdo each other.  It was just like that scene in Life of Brian.  I couldn’t help but say “blessed are the cheese makers” every time I was within earshot.  I’m fairly sure if I wasn’t previously damned, I am now.

Oh well.

You can read Steve’s other blog entries at The Bigger Truth.

For starters, let’s cover the basics very briefly. VFCache is server-based flash cache (at least mostly and usually it is – users also have the option of splitting the card to use some as cache and some as persistent storage). It integrates with EMC’s ‘FAST’ (Fully Automated Storage Tiering) and thereby extends FAST up into the server. The impact for users is either more transactions and/or less wait time, depending on what floats your storage and application boats. As such, it is aimed squarely at databases, OLTP, e-mail, analytics and the like…places that need storage ‘ooomph,’ are mostly read activity, and where a relatively small percentage of the data drives a very large percentage of the I/O activity. This is simply because the PCIe based flash used in VFCache remains (like all flash) relatively expensive and so it needs to be used -

• For prime caching opportunities – to spread the cost impact and have it make better IT-economic-sense.
• In mission-critical applications - to make spending even the ameliorated-$$ worthwhile in a business-economic-sense).

That’s it in a nutshell – like all good things it’s pretty easy to ‘get’ the concept.

So, what about the impact? I suspect many competitors will miss the point and talk functional intricacies rather than impact….specsmanship rather than market importance. Indeed it is true that EMC is not the first to use server-based flash/cache. In fact, EMC often isn’t first into things for the enterprise market, although – somewhat ironically given our topic here – it was EMC that kicked off the current ‘solid-state era’ when it introduced Enterprise Flash Drives in early 2008!  But competitive rebukes of the ‘yes but we were first/faster/bigger/wash whiter’ variety, are frankly missing the point. I was reminded of a great quote I saw when reading on a plane last week: ”Facts and truth really don’t have much to do with each other” (William Faulkner, quoted in the Associated Press). Why do I say that here? Well the facts may well be – or might not, but that’s not the point here – that someone else can go faster in some manner, or has some better management in some way. But the truth is that this is EMC making the introduction – this is a market leader that has shipped over 24 PB of flash to date with way over an Exabyte of storage under the management of FAST. VFCache being the absolute best in any one area doesn’t matter (and for all I know EMC is best in all of the technical aspects…). What matters is that it is more than good enough for most users and applications. It offers tremendous throughput and response time improvements, and does so within a management construct that myriad users already employ, and furthermore it has the extensive underlying security (HS, data protection, shareability) that users expect from EMC.

So, yes, this is a significant product announcement…but is far more of a market announcment. EMC is giving its formal blessing to server-based flash, and indeed to flash throughout the storage ecosystem. The next iteration that EMC has indicated having sometime next quarter is ‘Thunder,’ a network flash appliance. Combined, these products show a clear direction to users, and maybe offer some solace and opportunity to all the start-ups entering the solid-state storage market. Clearly it ain’t all about spinning rust any more….and if you didn’t buy into that before today, maybe now it is time to believe.

Bottom line? This particular lightning is way more than flashy - it packs a market punch.

You can read Mark’s blog entries at The Business of Storage.

via Nutanix aims to ban the san from virtualized datacenters – Application Delivery and Virtualization News: Citrix, Microsoft Virtualization Resources, XenApp, XenDesktop, Remote Desktop Services and VMware News.

It was a privilege to work with two ESG Lab Engineers, Tony Palmer and Ajen Johan, to do some hands-on with the new Riverbed technologies.

To me, the really cool part of Granite is how it changes some presumptions for branch offices.  Traditionally, I still see large customers doing things in two very different ways:

• For their large data center facilities, SAN-based manageable and consolidated storage.
• For their remote offices, standalone servers with direct-attached disk.

The reason for this is that most presume that managing a SAN at each branch office is non-viable.  So, even though managed storage may be more effective for many scenarios, it only seems to happen in the data centers of many companies.   If only there was a way to do it for the branches?

The Granite technology effectively extends an iSCSI scenario from your data center SAN to a branch office server.   Technically, there are two iSCSI scenarios in play:

• Within the data center (diagram-left), the Granite “core” device has an iSCSI initiator to mount LUNs from your existing iSCSI SAN provider.
• At the branch office (diagram-right), the Granite “edge” device caches a copy of that LUN and becomes an iSCSI target, whereby any machine in your branch with an iSCSI initiator can mount it.

The result is that managed storage blocks that appear local are in fact replicated from a centralized data center copy.  This leads to some really interesting (and presumption-twisting) changes from a Data Protection perspective.  Maybe you still run your backups at the branch?  Maybe you want to finally adopt NDMP and back up directly from the SAN at the data center?  There are some options worth looking at, without ripping out whatever backup solution is currently in play.

There is more to the Steelhead EX + Granite solution than just the iSCSI capabilities, including a built-in VMware hypervisor inside the new Riverbed device – enabling further server consolidation to VMs within the edge device, using iSCSI LUNs that are actually from the corporate data center.

Check out the ESG Lab Validation on Riverbed Steelhead EX + Granite for more.

As always, thanks for reading.

Background

A growing number of organizations are using solid-state storage solutions in the data center. As shown in Figure 1, 34% of respondents to a recent ESG survey are currently using solid-state storage technology in either servers or external storage systems, and another 35% are currently evaluating it or have plans to do so in the next 12 months.[1] While early adopters report that improved performance was their primary reason for deploying a solid-state storage solution, they’ve achieved a number of additional benefits, including improved power and cooling efficiency, increased environmental tolerance, enhanced longevity, and improved reliability.

The first wave of widespread solid-state storage adoption began about four years ago, when flash memory became available as a solid-state disk drive tier in enterprise-class disk arrays. More recently, a growing number of organizations have installed PCIe-attached flash storage in servers to create a low-latency pool of primary disk or an extended disk cache. As matter of fact, 21% of respondents to a recent ESG survey indicate that they are currently using flash storage solutions in servers, and 15% plan on doing so in the next 12 months.

Early adopters report that an increasing number of performance-critical applications are accelerated with solid-state storage, including OLTP database, ERP financial, OLAP business intelligence, supply chain management, and high-performance computing (HPC) applications. Solid-state storage is also accelerating the performance of consolidated virtual server and virtual desktop infrastructures.

Virident FlashMAX

FlashMAX is a small form factor PCIe drive that delivers high-speed, flash-based storage in capacities ranging from 300 GB to 1.4 TB. FlashMAX is designed to bridge the ever-growing performance gap between server CPU cores and traditional storage solutions. Typical performance issues common to real-world workloads are eliminated as FlashMAX delivers extremely high levels of predictably fast and sustained performance for mixed-application workloads.

Virident uses unique software and hardware to leverage the benefits of flash technology inside the FlashMAX SCM drive. The SCM architecture provides storage-like capacity and resilience while delivering memory-like performance in a small, universal form factor. The software layer, called vFAS (Virident Flash management with Adaptive Scheduling), serves as a gatekeeper, granting access to the flash media as efficiently as possible at all times. Without the need for slower, legacy storage protocols or interconnects, major improvements in application performance occur. These improvements occur due to vFAS’s virtualization of the primary flash media, which is accessible to applications via a standard block device interface. vFAS also intelligently and efficiently manages the asymmetric read/write/erase latencies of flash media to deliver consistent, predictable performance to the applications.

vFAS maximizes flash lifetime with global wear-leveling techniques. When necessary, data is relocated to less-used parts of the flash media to prevent hot-spots and overuse. Concerns regarding reliability and data availability are put to rest by the support of built-in flash-aware RAID. Data is spread across a RAID group that spans multiple flash chips and is protected by a RAID-5-like scheme, which prevents disruption from media failures while maintaining application data access and operational continuity. The RAID implementation is flash-aware and is tied tightly into the garbage collection and wear-leveling mechanisms.

FlashMAX provides a high level of consistency across all application workloads, whether the drive is brand new or fully utilized. The challenges associated with many first-generation PCIe flash adapters have been addressed with the multi-dimensional performance capabilities of FlashMAX, which offers:

• High throughput for small, medium, and large IO block sizes.
• Similar levels of performance for random and sequential access patterns.
• High levels of performance for reads, writes, and a mix of reads and writes.
• Sustained consistent performance over time.
• Extreme performance scalability with multiple FlashMAX adapters in a single server.
• Exceptionally low latencies and fast response times for real-world application workloads.

ESG Lab Validation

ESG Lab performed hands-on evaluation and testing of the FlashMAX at Virident corporate headquarters in Milpitas, California. Testing was designed to demonstrate the multi-dimensional performance capabilities using the industry-standard FIO utility.[2]

First Dimension: Small, Medium, and Large Block Sizes

Performance-sensitive applications that benefit from solid-state storage often have high throughput requirements (e.g., an HPC application processing a large machine-generated data set with 512 KB IOs). Others require high performance for relatively small IO requests (e.g., an OLTP database application with 4 KB IOs or a financial application writing logs with 1 KB block size). A third class of applications requires high performance for a mix of block sizes, with large IOs being used for data requests and small IOs used for metadata requests. The FlashMAX optimizes performance for each of these workloads, delivering predictably fast performance for a mix of IO block sizes.

ESG Lab Testing

ESG Lab used the FIO utility to test the sequential read throughput capabilities of a single FlashMAX adapter as it processed IO requests with block sizes ranging from 512 bytes to 512 kilobytes. The results are shown in Figure 3.

What the Numbers Mean

• Much like the horsepower rating of a car, the aggregate throughput of a storage solution is a good indicator of the underlying power of that storage solution’s engine.
• Storage throughput is a measure of the bandwidth available to the system. Throughput can be measured on a stream or aggregate basis. A stream is represented by one application or user communicating through one IO interface to one device. Aggregate throughput is a measure of how much data the storage solution can move, as a whole, for all applications and users.
• Aggregate FlashMAX throughput scaled in a near-linear fashion for the smaller block sizes, shown toward the left side of Figure 3.
• Sequential read throughput reached a peak of 1,394 MB/sec at a 4 KB block size.
• A peak aggregate throughput of 1.394 GB/sec (1,394 MB/sec) is an excellent result for a single PCI flash drive.
• Performance remained predictably high as IO block sizes increased from 512 bytes up to 512 KB, shown toward the right side of Figure 3.

Second Dimension: Sequential and Random Access Patterns

HPC applications can have a variety of workload access patterns including random, sequential, and a mix of random and sequential access. Regardless of the IO access pattern, one key advantage of PCIe flash storage solutions is the ability to perform thousands of times more IOs per second (IOPS) than traditional spinning disk drives or ten times more IOPS than drive form-factor (SATA or SAS) SSDs.

While early adopters of flash storage solutions in the HPC market were initially focused on throughput-intensive sequential workloads, broader adoption in the wider (more horizontal) database, server virtualization, and desktop virtualization markets has begun to take off. Multi-user database and virtualization applications tend to have more random IO access patterns. First-generation flash solutions tended to have different performance characteristics for random and sequential access patterns. FlashMAX SCM provides similar levels of high performance for random and sequential workloads.

ESG Lab Testing

As shown in Figure 4, ESG Lab ran both random and sequential 8 KB reads at four different queue depth sizes to show not only how the access patterns perform, but also how they scale.

What the Numbers Mean

• Online database applications, including those that rely on the latest version of Microsoft SQL Server, are typically composed of a mix of random and sequential IO access patterns, with a block size of 8 KB as tested during this phase of the ESG Lab validation.
• Extremely fast sub-millisecond response times of 38 and 64 microseconds were recorded for 4 KB random write and read workloads respectively.
• The extremely fast response times recorded during ESG Lab testing are significantly faster than a drive form factor SSD. SSDs are slower due to the additional overhead of an IO protocol (e.g., SAS) vs. the low latency of a PCIe bus and the vFAS software advantage with the FlashMAX.
• The extremely fast response times recorded during ESG Lab testing are 25 to 200 times faster than a traditional disk drive.
• The total number of IOPS processed at a single queue depth was slightly higher for the sequential access pattern (14,286 IOPS) compared with the random access pattern (13,716 IOPS).
• At a queue depth size of 16, the random access slightly outperformed the sequential access.
• As queue depth continued to increase, performance eventually leveled out at a little more than 170,000 IOPS for random 8 KB reads and 165,000 IOPS for sequential 8 KB reads. It would take more than 1,000 power-hungry disk drives to deliver 170,000 random 8 KB IOPS.

Third Dimension: Sustained Performance over Time

Solid-state storage solutions have historically had problems maintaining performance over time. Because space is needed to service incoming write requests, a “garbage collection” process needs to run in the background. The goal of the garbage collection process is to free-up necessary space by consolidating written data that was fragmented due to the erase-before-write nature of flash media. Because this process happens along with all of the other user requests, enterprise application performance can be greatly affected. This can be seen by a severe drop in throughput and large response-time spikes. The performance drop is even more severe when flash devices are filled to capacity. The phenomenon is often referred to as a “write cliff,” which aptly describes how write performance seems to fall off a cliff over time. FlashMAX SCM was designed with a goal of providing sustained performance over time and avoiding the write cliff problem.

ESG Lab Testing

ESG Lab tested the ability of a FlashMAX to sustain performance over time. An online database workload was emulated using a mixed 8 KB random workload with a mix of 70% reads and 30% writes. The database was sized to utilize the full capacity of the drive. The duration of the test was set for more than three hours (one hour warm-up followed by two hours of recorded runtime) to allow ample time for the flash device to reach full capacity and potentially be affected by the garbage collection process. The results are shown in Figure 5.

Performance varied minimally over the entire test, and the sustainability is clear. The FlashMAX was able to deliver consistent performance of just under 100,000 IOPS throughout the full duration of the test.

The Fourth Dimension: Scaling Performance with Additional FlashMAX

Scalability, put very simply, is the ability to elegantly handle more work or to physically grow to accommodate that additional work. In this case, scalability applies specifically to the near-linear performance increase that can be achieved with more than one FlashMAX drive installed in a single server. Illustrating this concept, Figure 6 shows how performance and capacity increase in a near-linear fashion with each additional FlashMAX drive that is installed in a server.

ESG Lab Testing

Eight FlashMAX SCM drives were installed in a powerful NEC GX server with fourteen PCI slots during this phase of testing. The Lab used the FIO utility to test 4 KB random read and write workloads. Results are shown in Figure 7.

What the Numbers Mean

• Performance scaled in a near-linear fashion for random read and write workloads as up to eight FlashMAX drives were installed in a single server.
• Performance peaked at an extremely high level of 2.2 million IOPS and 10.6 GB/sec of throughput on a single server.

Fifth Dimension: Real-World, Mixed-Application Workloads

Having looked at the throughput, IOPS, and response-time ratings of the turbo-charged FlashMAX engine, here’s where ESG Lab found “the rubber meets the road” when examining FlashMAX performance with real-world application workloads.

ESG Lab Testing

ESG Lab used the FIO utility to measure the performance of a single FlashMAX card for three application workloads:

• Virtual Desktop Infrastructure (VDI): Designed to emulate a virtual desktop environment composed of heavy knowledge-worker users sharing a common gold image (a.k.a., a linked clone). This workload is composed of 80% 16 KB random writes and 20% 16 KB random reads.
• Online Transaction Processing (OLTP): Order entry and reservation systems are two examples of OLTP applications. Oracle and Microsoft SQL Server are two examples of database applications used to create such OLTP applications. OLTP applications are characterized by a number of users accessing a shared system in parallel. This workload was composed of mostly random reads (70%) with relatively fewer writes (30%).
• Decision Support System (DSS): This workload, also referred to as data mining, emulates a database application that is doing a large-scale random query with a block size of 4 KB. An end-of-month analysis of the effect of a coupon-redemption program on same-store sales is an example of a decision support application.

In order to test the worst-case scenario, the drive was filled to capacity in all three application workload scenarios. Figure 8 shows the throughput scalability of these applications as queue depth increased.

What the Numbers Mean

• Having one outstanding request at a time, the VDI workload was able to achieve 162 MB/sec throughput, and it eventually scaled up to more than 400 MB/sec, with as little as 16 outstanding requests.
• The OLTP simulation reached a maximum of 794 MB/sec throughput at a high queue depth of 256.
• Showing the largest scaling factor, the DSS workload scaled from 117 to 1,352 MB/sec as queue depth increased.
• The performance of a single FlashMAX SCM drive that was recorded during simulated VDI workload testing can be used to support more than 1,000 heavy desktop users.[3]
• A traditional disk array with more than 1,000 disk drives would be needed to deliver the performance recorded during the OLTP testing with a single FlashMAX drive.

ESG Lab Validation Highlights

• Predictable performance scalability with a variety of IO block sizes that peaked at 1.39 GB/sec from a single FlashMAX drive (4 KB sequential reads).
• Predictably fast performance for reads, writes, and a mix of reads and writes for workloads simulating real-world OLTP, VDI, and DSS applications.
• Nearly identical levels of high performance for sequential and random read workloads.
• Sustained performance of 96,825 IOPS over two hours for an 8 KB OLTP workload and drive filled to capacity.
• Up to 345,046 IOPS from a single FlashMAX drive for a 4 KB random read workload.
• Extremely fast sub-millisecond response times (38 and 64 microseconds for 4 KB random write and read workloads, respectively).
• Near-linear performance scalability as up to eight FlashMAX cards delivered up to 2.2 million IOPS and 10.6 GB/sec of throughput on a single server.

Issues to Consider

• The high cost of solid-state capacity compared with traditional hard drive capacity has focused early adoption mainly among businesses whose revenue depends strictly on application performance (e.g., trading applications within the financial industry). As a matter of fact, ESG research indicates that the high cost of solid-state capacity is the number-one objection by organizations that have not yet deployed a flash-based storage solution. As the cost of flash capacity decreases over the next three to five years and performance needs increase, ESG expects that the adoption of PCIe flash drives will grow in the broader horizontal enterprise IT market. This is especially true within server virtualization and desktop virtualization environments with high performance needs. In this case, PCIe flash drives have an economic advantage ($/IOP) compared with traditional hard drives.
• Early adopters considering using a flash-based PCI drive in a server to solve a performance problem with a business-critical, high-performance application should consider the extra costs of installing FlashMAX drives in multiple clustered servers for high availability and failover. Solid-state disk drives installed at the other end of the wire (in a SAN-attached storage array) are a viable alternative for more cost-effective sharing and failover. However, they are typically much slower than a PCI flash drive. A FlashMAX drive that’s installed inside of a SAN-attached disk array could be used to create a simply elegant, highly available alternative that cost-effectively accelerates the performance of tier-0 applications and consolidated virtual server environments.
• ESG Lab ran performance scalability tests on a high-end server that could support multiple FlashMAX drives in a single server. The server being tested needed to have enough PCIe slots to support eight FlashMAX drives, and enough processing power to support the necessary requests to drive each FlashMAX to its limit. To achieve the performance scalability results documented in this report, a high-end server with similar specifications is recommended.[4]

The Bigger Truth

The growing gap between the speed of servers and traditional disk-based storage solutions is causing a number of problems in the data center. Though flash memory solutions have filled this void by serving as an answer for some of the most performance-critical application workloads in recent years, issues still exist. First-generation flash-based storage solutions often have challenges maintaining predictably high levels of performance, over time, for real-world applications with mixed-IO patterns. Sustainable performance over the life of a drive is a particularly vexing challenge due to the impact of background garbage collection processes.

ESG Lab has confirmed that Virident FlashMAX is a next-generation PCIe flash drive that leverages intelligent algorithms to provide high levels of sustained, multi-dimensional performance. Extremely low latencies and high levels of performance were recorded with a variety of workloads. ESG Lab was most impressed with the paucity of “saw-tooth” and “drop-off” performance patterns associated with first-generation PCI flash drives. Scalability testing with up to eight FlashMAX drives delivered extremely high levels of near-linear performance scalability that sustained more than 10 GB/sec of throughput and two million IOPS with a single server.

With a proven ability to deliver predictably fast real-world application performance over the life of the drive, FlashMAX is well suited for the growing number of performance-sensitive OLTP, OLAP, DSS, HPC, and VDI workloads that are migrating from traditional disk drives to high-speed flash memory. ESG Lab believes that Virident, with the FlashMAX family of storage-class memory solutions, has unlocked the potential for affordable, large-scale deployment of flash technology in the modern data center.

Appendix

[2] http://freecode.com/projects/fio

[3] Using a conservatively high estimate of 20 IOPS per user

[4] The specifications for the server used during ESG Lab testing are listed in the Appendix.

This announcement coincides with the publishing of ESG’s whitepaper on “DR in the Cloud” using AWS.

All things being considered, I am very excited about the AWS Storage gateway (AWS SG), mostly because it reminds me in some ways of Microsoft’s for-sale backup product, System Center Data Protection Manager that I used to manage.  DPM wasn’t the most full-featured backup software on the market, but it did at least two very good things:

The AWS Storage Gateway is a credible offering for what its initial release is designed to do.  And like most cloud-based offerings, one can expect it to be enhanced in months, not years, as customers give feedback and operational lessons are learned.  As Terri points out, the AWS Storage Gateway may not be taking over the world of cloud-based storage enablement quite yet.  But the AWS Storage Gateway, when seen alongside all of the other AWS offerings, shows how Amazon is continuing to evolve its Platform-as-a-Service (PaaS) and infrastructure-as-a-Service (IaaS) offerings.   And those evolutions are good not only to Amazon and its ever-growing AWS direct customer base, but also to the partners that will develop even more solutions based on them for the rest of us.

ESG recently authored a white paper on “DR in the Cloud“, based on where we see companies struggling with home-grown DR solutions — and how the AWS offerings can help.

To read the ESG Whitepaper on “DR in the Cloud” using AWS, click here.

Thanks for reading.

But an AWS new gateway won’t kill the independent gateway market near term – this beta version is really basic and does not have much in terms of bells and whistles – it is much less feature-rich than the initial implementations from the existing independent gateway vendors (CTERA, Nasuni, Panzura, Riverbed, StorSimple, Twinstrata, and I probably missed some), and those vendors have not stood still. There is no deduplication (but there is compression) – depending on how compressible the data set is, without deduplication the AWS capacity and data transfer service charges could lead to some pretty big fees on the storage services side depending on the nature and use of the data. Uploading data (data transfer IN) is free, but retrieving data (data transfer out) costs $$, so a restore of any size or frequent restores could add up. This is a pure backup and DR play for now rather than a local cache for primary data in the cloud, but AWS is expected to expand functionality and use cases over time.

It is interesting to note that there is also no hardware node–not surprising, but interesting, and a possible challenge. The AWS gateway runs in a VMware virtual machine and uses local disk capacity. While some may consider this a good thing, experience has taught gateway providers like TwinStrata that sometimes you need to sell an appliance that can be dropped into the data center and dedicated to being the gateway – you often get better performance and have no worries about resource contention or workload management. Plus – there is the mentality that storage buyers want to touch and feel something – pure cloud is still a tough sell in the enterprise.

And I hate to use the term “vendor lock in” because at some level everyone is locked into someone, but the independent gateway providers do provide users an option of transparently migrating data to a different service provider if their current provider does something like change the pricing model or SLA. Just think about what happened when Iron Mountain decided to get out of the storage services business (not that I think this will happen at AWS, but pricing model tweaks are certainly a possibility). Nasuni went out and proactively migrated their customers under the covers to their new service provider of choice. By the time the IM news was public, Nasuni had done a ton of migration work. While not an “apples-to-apples” comparison, it is illustrative of what is possible when using an independent.

The initial target use cases for AWS are dead on though – of the current cloud storage users we surveyed in our spending intentions research, the biggest use cases for them are backup (a whopping 67% are using cloud storage for backup), followed by DR (58%) and Archive (58%). And that certainly reflects where some of the independent gateway vendors like Riverbed are finding a niche, so this will increase pressure on them.

Looking a little deeper – this is really a brilliant move by AWS, not because they have a gateway for cloud storage use, though. It is because these snapshots are stored as Elastic Block Storage (EBS) snapshots. This means they can be used to create EBS volumes and run against applications in EC2. Test and dev operations can now be run against production snapshots in EC2. And over time, users can just migrate applications into the cloud – after all, the data migration will already be done for them. Once you do test and dev and get comfortable, the leaps to running tier 2 applications, then tier 1, become smaller – just ask VMware. At the end of the day I don’t think Amazon is really interested in being a gateway so some of your data can be stored in the cloud – I believe that they want all of your data, and compute, in the cloud. And it may take years, but this is the gateway to that.

You can read Terri’s other blog entries at IT Depends.

ESG recently published our market landscape report on OLFS at www.enterprisestrategygroup.com/2011/12/online-file-sharing-and-collaboration-in-the-enterprise/

Last year, ESG shared its perspectives on BaaS at www.enterprisestrategygroup.com/2011/02/data-protection-backup-as-a-service/

BaaS and OLFS have a lot in common, so their convergence is natural, and frankly, almost inevitable.

• They work by installing an agent on the range of consumer devices that you carry. After installation and a usually user-friendly (wizard or push-button) experience, they routinely if not near-continuously transmit changed data to the cloud.
• They rely on a massive cloud-based storage architecture, whether it is self-maintained by the original vendor or leveraging a public cloud’s storage platform, e.g. Amazon.
• They’re sold usually on a subscription basis, almost always with tiered offerings, based on how much storage you plan to consume

So, what is different between OLFS and BaaS?

• BaaS is focused on multiple recovery points as a key design criteria, often including a definable retention period for past versions
• OLFS is focused on sharing – whether that means across your varied devices, or between you and your friends/coworkers, will vary based on the OLFS offering

Some OLFS offerings do support previous versions, though its usually within the context of restoring that Word document that you just accidently overwrote — and not preserving your data for a year. And while whole-machine recovery may not be a primary design function of most OLFS, the reality is that if your machine is re-image-able from either it’s factory DVDs, a monthly backup to a USB drive, or perhaps your corporate backup solution … and your data is regularly uploaded to some OLFS cloud — then whole machine recovery really can be a fairly trivial event.

BaaS-only solutions know that that they are “backup” solutions, so sharing options aren’t typically part of the model — which makes sense.

So, how does Mozy’s Stash solution stack up?

I took the opportunity to try out both the Mozy Backup and Stash offerings on one of my home machines. The setup for backup was relatively straight forward and I found it interesting how they pre-define data types and then simply prompt you to either back them up or not. I will look closer at its BaaS capabilities in a later blog post (or ESG Lab validation) but for now, I just wanted to get it installed so that I could be one of the millions of Mozy Backup customers that could now try the Stash feature.

Stash functionality enabled pretty easily, with the standard OLFS concept of defining a root-level folder for data storage. And as exciting as it sounds, I dropped some files into it and watched my drive light and network lights start blinking away. Sure enough, by installing the same Mozy client software on my work laptop, the files were there. Yay! But let’s be clear — it is a beta of a first release in the space. I am actually an avid user of another OLFS service which has a key feature that Mozy doesn’t yet offer — sharing between users. For that reason alone, I can’t use it yet. If you don’t share data with others, is Mozy viable for you? Maybe.

Although it is ‘beta’, it isn’t fair to call their offering a ‘1.0’ — because they aren’t standing it up from scratch. Mozy has oodles of experience with what it takes to create a lightweight agent technology across a variety of consumer devices. They understand how to build and operate a cloud-based storage platform at scale. They have millions of subscribers. Some of them may be using another OLFS, and if they aren’t sharing with others, may be happy to run one less agent and pay one less monthly bill. Other Mozy subscribers may have been thinking about OLFS, and the Stash offering will be what gets them started. And don’t forget, it is still only in beta.

So, more functionality will eventually come, and like most cloud-services, incremental features will come months, not years, later. Some of Mozy’s backup users will jump on this (likely increasing their storage consumption subscription in the process) — and Mozy will invariably hear the feedback of what their install base wants vs. needs. And with Mozy’s agility, as well as their commitment to cloud-enabled storage, things can only go up.

What excites me the most is seeing examples of the convergence between BaaS and OLFS. And if Stash helps more folks to get their data into the cloud, that is goodness. Beyond the convergence, I’m also looking forward to seeing what happens with Mozy Stash 1.1 … 1.5 … 2.0.

via IceWeb, nDataStor Partner on Unified Data Storage Platform – Storage news from Channel Insider.

Well, so what, you might say….most people don’t give much thought to how many atoms it takes to store data. After all, a bit isn’t much is it? I need a few gigs for my latest Hollywood blockbuster! Well, think on this – a single bit can usually take around one million atoms to store! So suddenly 12 is looking awfully good. I’m optimistic in so many areas of life (from power to food to transport) that mankind will remain continually ingenious and figure ways around seemingly intractable problems. For decades you may have heard about the superparamagnetic effect…basically it’s what prevents us packing data ever-more-densely because the magnetic field of each bit starts interacting and affecting the others around it. We’ve found innovative ways around it – vertical recording technology, for instance – but the essence of the problem has remained. As the absolute demand for storage has grown faster than the relative price decline of storage, so we’ve also concentrated on ways to make better use of the space we actually have: hence the popularity of things like thin provisioning, deduplication, etc., etc. But in historical terms we’re only putting fingers in the dam, fighting a losing battle as valiantly as we can.

The news from IBM provides hope. Yes, it’s ‘only’ research, but this early stage storage capability offers the potential to be literally orders of magnitude more dense than anything we have today. As to the cold I mentioned? Well, right now the antiferromagnetism used by the IBM boffins is being deployed at 1 degree Kelvin (minus 458 Fahrenheit, which is distinctly chilly!) and the data is only retained for hours. BUT, that’s not the point….early cars only went at a few miles an hour and had someone with a red flag walk in front of them, early telephones were all hard-wired, and – hey – downloading a single photo over the early WWW just a few years ago was something you only did if you had plenty of time to spare. We’ll figure this out, too.

You can read Mark’s other blog entries at The Business of Storage.