1 / 12

YARN Code Overview

YARN Code Overview. Ocular bleeding is no reason to stop programing!. Quick Bio – Joseph Niemiec. Hadoop user for 2+ years 1 of 5 Author’s for Apache Hadoop YARN (March 2014) Originally used Hadoop for location based services Destination Prediction Traffic Analysis

kana
Download Presentation

YARN Code Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. YARN Code Overview Ocular bleeding is no reason to stop programing!

  2. Quick Bio – Joseph Niemiec • Hadoop user for 2+ years • 1 of 5 Author’s for Apache Hadoop YARN (March 2014) • Originally used Hadoop for location based services • Destination Prediction • Traffic Analysis • Effects of weather at client locations on call center call types • Pending Patent in Automotive/Telematics domain • Defensive Paper on M2M Validation • Started on analytics to be better at an MMORPG

  3. Agenda • What Is YARN • YARN Concepts & Architecture • Code and more Code • Q&A

  4. From Batch To Anything Single Use System Batch Apps Multi Purpose Platform Batch, Interactive, Online, Streaming, … HADOOP 1.0 HADOOP 2.0 MapReduce (data processing) Others (data processing) MapReduce (cluster resource management & data processing) YARN (cluster resource management) HDFS (redundant, reliable storage) HDFS2 (redundant, reliable storage)

  5. Concepts • Application • Application is a job submitted to the framework • Examples • Map Reduce Job • MoYaCluster • Container • Basic unit of allocation • Fine-grained resource allocation across multiple resource types (memory, cpu, disk, network, gpu etc.) • container_0 = 2GB, 1CPU • container_1 = 1GB, 6 CPU • Replaces the fixed map/reduce slots

  6. Architecture • Resource Manager • Global resource scheduler • Hierarchical queues • Node Manager • Per-machine agent • Manages the life-cycle of container • Container resource monitoring • Application Master • Per-application • Manages application scheduling and task execution • E.g. MapReduce Application Master

  7. To the code!

  8. Q&A

  9. YARN - ApplicationMaster • ApplicationMaster • ApplicationSubmissionContextis the complete specification of the ApplicationMaster, provided by Client • ResourceManager responsible for allocating and launching ApplicationMaster container

  10. YARN – Resource Allocation & Usage • ContainerLaunchContext • The context provided by ApplicationMaster to NodeManager to launch the Container • Complete specification for a process • LocalResource used to specify container binary and dependencies • NodeManager responsible for downloading from shared namespace (typically HDFS)

  11. YARN – Resource Allocation & Usage • ResourceRequest

  12. YARN – Resource Allocation & Usage • Container • The basic unit of allocation in YARN • The result of the ResourceRequestprovided by ResourceManager to the ApplicationMaster • A specific amount of resources (cpu, memory etc.) on a specific machine

More Related