VMware promotes big data apps on VMs

 

New frontier on virtual machines.

VMware has established a crack team of engineers to virtualise applications usually considered too technically challenging to operate in a virtual container, including big data, high-frequency trading, and high-performance computing platforms.

Successful big data deployments at Google, Yahoo, Facebook and Target have prompted organisations such as banks to attempt to apply the technology to their businesses.

But most organisations experimenting with Hadoop do so in small iterations, often on standalone, physical servers and storage – undermining the use of VMware’s vSphere as an enterprise standard operating environment.

At a technical level, big data applications are I/O and compute intensive, and are designed to take advantage of very different storage architecture to most corporate apps.

Hadoop users have also opted for cheaper storage options than VMware-supported SAN and NAS, by using clustering and replication software to create SAN-like redundancy with cheap disks ($0.10 per GB vs up to $5 per GB on a SAN).

“We asked ourselves the question: is virtualisation ready for I/O and CPU intensive applications?” VMware's Australian-born CTO of application infrastructure Richard McDougall told iTnews at the VMworld conference last week.

“We sat down with our partners for some co-engineering tests. The good news is we have hit some big milestones – we’re now able to prove that when systems are set up properly, there is no performance overhead to running virtualisation underneath Hadoop.

“That will be a big eye opener for a lot of people.”

McDougall took to the stage at VMworld to demonstrate ‘Project Serengeti’, an open source effort aiming to run Apache Hadoop on virtual machines.

Using the high availability features of vSphere, McDougall demonstrated how a single system could be multi-purposed to focus nearly all of its processing power on production web servers during business hours, with only one or two nodes running a Hadoop instance collecting click stream data in the background.

Trying to analyse that data in real-time could have a negative I/O impact on live transactions. But McDougall demonstrated how the system could be set up to share resources between the web server and Hadoop application in a flexible fashion.

As traffic to the web server dropped away into the evening, the system was set up to gradually dedicate more of the available nodes to the Hadoop cluster to run analytics against the stored click-stream data.

The exercise might have seemed trivial – but for the fact that McDougall was demonstrating the collection and processing of terabytes of data per node, without having to “shift any data around”.

McDougall told iTnews the “obvious value prop” for running Hadoop on virtual machines was utilisation.

“It’s reported that Yahoo’s big Hadoop cluster has CPU utilization of only 11 percent or thereabouts. And you see that trend across other environments, the majority of customers aren’t making use of their computing capacity when systems are set up to tackle single purpose problems. Being able to use that platform for more than one thing would save huge amounts of money.”

But like all other apps before it, perhaps the bigger driver is the ability to clone or provision Hadoop environments in a rapid fashion. “We want to make it dead simple to create a new virtual cluster on existing infrastructure,” McDougall said.

VMware now just needs a big name reference customer – someone of the size of Facebook, Google, Yahoo or eBay – to drive the point home.

Trading systems and HPC

McDougall’s team is also tackling the virtualisation of low-latency applications for financial trading systems.

Brokers have a fairly unique set of requirements when connecting to stock exchange systems, and there is massive competitive advantage to be gained from making any given trading decision faster.

Applications require extremely fast response times and networks as little latency as possible – to the point in which many stock exchanges offer co-location to bring customers closer to the action.

Whilst a subset of these applications aren’t likely to live on virtualised infrastructure any time soon, McDougall says there is a larger share of apps in the market that don’t require such extreme performance.

“We have spent time with most of the software vendors in that space – and most talk of low-latency and real-time trading,” McDougall.

“But for the majority of apps - when they say real time what they mean is that it is sensitive to milliseconds of latency.

“It is only the pointy end that needs to buy the data centre space at the speed of light distance away. The great majority of apps in the middle are low hanging fruit for virtualisation.”

The same goes for high performance computing – increasingly used in mining and manufacturing, insurance, biotechnology and other research fields. These systems tend to run simulations on terabytes of data, with thousands of CPU’s in a cluster.

Like big data apps, these high performance computing clusters tend to be run in silos from corporate IT.

McDougall’s team wants to abstract away that hardware infrastructure so that whether you are running a simulation or checking your email, you are still accessing the same shared pool of hardware resources.

“If there are adjacencies where virtualisation hasn’t been used, we are making sure we have everything we need to prove to customers it can be done.”

Copyright © iTnews.com.au . All rights reserved.


VMware promotes big data apps on VMs
 
 
 
Top Stories
Frugality as a service: the Amazon story
Behind the scenes, Amazon Web Services is one lean machine.
 
Negotiating with the cloud email megavendors
[Blog post] Lessons from Woolworths’ mammoth migration.
 
Qld govt to move up to 149k staff onto Office 365
Australia's largest deployment, outside of the universities.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

The great data centre opportunity on Australia's doorstep
The great data centre opportunity on Australia's doorstep
Scott Noteboom, CEO of LitBit speaking at The Australian Data Centre Strategy Summit 2014 in the Gold Coast, Queensland, Australia. http://bit.ly/1qpxVfV Scott Noteboom is a data centre engineer who led builds for Apple and Yahoo in the earliest days of the cloud, and who now eyes Asia as the next big opportunity. Read more: http://www.itnews.com.au/News/372482,how-do-we-serve-three-billion-new-internet-users.aspx#ixzz2yNLmMG5C
Interview: Karl Maftoum, CIO, ACMA
Interview: Karl Maftoum, CIO, ACMA
To COTS or not to COTS? iTnews asks Karl Maftoum, CIO of the ACMA, at the CIO Strategy Summit.
Susan Sly: What is the Role of the CIO?
Susan Sly: What is the Role of the CIO?
AEMO chief information officer Susan Sly calls for more collaboration among Australia's technology leaders at the CIO Strategy Summit.
Meet the 2014 Finance CIO of the Year
Meet the 2014 Finance CIO of the Year
Credit Union Australia's David Gee awarded Finance CIO of the Year at the iTnews Benchmark Awards.
Meet the 2014 Retail CIO of the Year
Meet the 2014 Retail CIO of the Year
Damon Rees named Retail CIO of the Year at the iTnews Benchmark Awards for his work at Woolworths.
Robyn Elliott named the 2014 Utilities CIO of the Year
Robyn Elliott named the 2014 Utilities CIO of the Year
Acting Foxtel CIO David Marks accepts an iTnews Benchmark Award on behalf of Robyn Elliott.
Meet the 2014 Industrial CIO of the Year
Meet the 2014 Industrial CIO of the Year
Sanjay Mehta named Industrial CIO of the Year at the iTnews Benchmark Awards for his work at ConocoPhillips.
Meet the 2014 Healthcare CIO of the Year
Meet the 2014 Healthcare CIO of the Year
Greg Wells named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at NSW Health.
Meet the 2014 Education CIO of the Year
Meet the 2014 Education CIO of the Year
William Confalonieri named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at Deakin University.
Meet the 2014 Government CIO of the Year
Meet the 2014 Government CIO of the Year
David Johnson named Government CIO of the Year at the iTnews Benchmark Awards for his work at the Queensland Police Service.
Q and A: Coalition Broadband Policy
Q and A: Coalition Broadband Policy
Malcolm Turnbull and Tony Abbott discuss the Coalition's broadband policy with the press.
AFP scalps hacker 'leader' inside Australia's IT ranks.
AFP scalps hacker 'leader' inside Australia's IT ranks.
The Australian Federal Police have arrested a Sydney-based IT security professional for hacking a government website.
NBN Petition Delivered To Turnbull's Office
NBN Petition Delivered To Turnbull's Office
UTS CIO: IT teams of the future
UTS CIO: IT teams of the future
UTS CIO Chrissy Burns talks data.
New UTS Building: the IT within
New UTS Building: the IT within
The IT behind tomorrow's universities.
iTnews' NBN Panel
iTnews' NBN Panel
Is your enterprise NBN-ready?
Introducing iTnews Labs
Introducing iTnews Labs
See a timelapse of the iTnews labs being unboxed, set up and switched on! iTnews will produce independent testing of the latest enterprise software to hit the market after installing a purpose-built test lab in Sydney. Watch the installation of two DL380p servers, two HP StoreVirtual 4330 storage arrays and two HP ProCurve 2920 switches.
The True Cost of BYOD
The True Cost of BYOD
iTnews' Brett Winterford gives attendees of the first 'Touch Tomorrow' event in Brisbane a brief look at his research into enterprise mobility. What are the use cases and how can they be quantified? What price should you expect to pay for securing mobile access to corporate applications? What's coming around the corner?
Ghost clouds
Ghost clouds
ACMA chair Chris Chapman says there is uncertainty over whether certain classes of cloud service providers are caught by regulations.
Was the Snowden leak inevitable?
Was the Snowden leak inevitable?
Privacy experts David Vaile (UNSW Cyberspace Law and Policy Centre) and Craig Scroggie (CEO, NextDC) claim they were not surprised by the Snowden leaks about the NSA's PRISM program.
Latest Comments
Polls
Which bank is most likely to suffer an RBS-style meltdown?





   |   View results
ANZ
  21%
 
Bankwest
  9%
 
CommBank
  11%
 
National Australia Bank
  17%
 
Suncorp
  24%
 
Westpac
  19%
TOTAL VOTES: 1456

Vote