Hadoop 2 released with next-gen resource management

By

A big data YARN.

The Apache Software Foundation overnight launched the general availability version of Hadoop 2, its open source framework for scalable distributed computing.

Hadoop 2 released with next-gen resource management

Hadoop 2 brings several new features not available from Hadoop 1, including high availability, data snapshots and federation for the Hadoop Distributed File System (HDFS), as well as support for Microsoft Windows.

NFS or network file system version 3 support is also built into Hadoop 2, with 64-bit sizes and offsets for very large files, and performance improvements.

The MapReduce layer has been superceded by YARN cluster resource management, which sits on top of HDFS to run data processing applications and services. MapReduce no longer handles the resource management and is becoming a user-land library instead.

With the help of YARN, applications can run inside Hadoop, providing multiple ways to process enterprise data.

 
Compatiblity with MapReduce applications from Hadoop 1 is maintained in YARN, which promises better scalability and improved cluster utilisation. Workloads other than MapReduce are supported with YARN and Hadoop 2, such as graph processing and iterative modelling.
 
YARN follows the eccentric naming tradition of the Hadoop project, which itself is named after a toy elephant owned by the son of creator Doug Cutting.
 
Full-time Hadoop contributor Arun Murthy told Application Development Trends in August last year that YARN means Yet Another Resource Negotiator.
 
"We were sitting around at lunch one day, trying to come up with the most inane names for our product. The result was 'Yet Another Resource Negotiator—YARN. I know: it's a really bad name," Murthy told ADN.
 
Got a news tip for our journalists? Share it with us anonymously here.
Copyright © iTnews.com.au . All rights reserved.
Tags:
big datahadoophortonworkssoftware

Sponsored Whitepapers

Gaining a Competitive Advantage with Communication APIs
Gaining a Competitive Advantage with Communication APIs
Leverage Technologies: Industry-Tailored ERP Implementation for Growth and Compliance
Leverage Technologies: Industry-Tailored ERP Implementation for Growth and Compliance
Service Over Signatures: The Truth About No Lock-In IT
Service Over Signatures: The Truth About No Lock-In IT
Wasabi Reveals Hidden Costs and Cloud Storage Shifts in ANZ for 2025
Wasabi Reveals Hidden Costs and Cloud Storage Shifts in ANZ for 2025
Datacom + Microsoft Azure: Turn Ideas Into Impact in Just 4 Weeks
Datacom + Microsoft Azure: Turn Ideas Into Impact in Just 4 Weeks

Events

Most Read Articles

AGL taps AI agents in retail transformation

AGL taps AI agents in retail transformation
Vic Police uses AI to reformat community-submitted crime reports

Vic Police uses AI to reformat community-submitted crime reports
Australia takes another step toward a central bank digital currency

Australia takes another step toward a central bank digital currency
Nvidia-backed Perplexity launches AI-powered browser

Nvidia-backed Perplexity launches AI-powered browser
techpartner.news logo
Dave Stevens on Brennan's evolution and the need for Aussie tech unity
Dave Stevens on Brennan's evolution and the need for Aussie tech unity
Sydney's ITKnocks on contact centre AI and the slow death of the IVR
Sydney's ITKnocks on contact centre AI and the slow death of the IVR
"It's an exciting time to be part of the health and aged care sector"
"It's an exciting time to be part of the health and aged care sector"
Insicon founder Matt Miller on the coming 'tsunami' of compliance and educating boards about cyber security
Insicon founder Matt Miller on the coming 'tsunami' of compliance and educating boards about cyber security
Orro claims Australia first with managed digital asset discovery service
Orro claims Australia first with managed digital asset discovery service

Log In

  |  Forgot your password?