Hadoop 2 released with next-gen resource management

By Juha Saarinen on Oct 17, 2013 10:39AM

A big data YARN.

The Apache Software Foundation overnight launched the general availability version of Hadoop 2, its open source framework for scalable distributed computing.

Hadoop 2 brings several new features not available from Hadoop 1, including high availability, data snapshots and federation for the Hadoop Distributed File System (HDFS), as well as support for Microsoft Windows.

NFS or network file system version 3 support is also built into Hadoop 2, with 64-bit sizes and offsets for very large files, and performance improvements.

The MapReduce layer has been superceded by YARN cluster resource management, which sits on top of HDFS to run data processing applications and services. MapReduce no longer handles the resource management and is becoming a user-land library instead.

With the help of YARN, applications can run inside Hadoop, providing multiple ways to process enterprise data.

Compatiblity with MapReduce applications from Hadoop 1 is maintained in YARN, which promises better scalability and improved cluster utilisation. Workloads other than MapReduce are supported with YARN and Hadoop 2, such as graph processing and iterative modelling.

YARN follows the eccentric naming tradition of the Hadoop project, which itself is named after a toy elephant owned by the son of creator Doug Cutting.

Full-time Hadoop contributor Arun Murthy told Application Development Trends in August last year that YARN means Yet Another Resource Negotiator.

"We were sitting around at lunch one day, trying to come up with the most inane names for our product. The result was 'Yet Another Resource Negotiator—YARN. I know: it's a really bad name," Murthy told ADN.

Got a news tip for our journalists? Share it with us anonymously here.