1 / 31

The Hadoop Fair Scheduler

The Hadoop Fair Scheduler. Matei Zaharia Cloudera / Facebook / UC Berkeley. UC Berkeley. Outline. Motivation / Hadoop usage at Facebook Fair scheduler basics Configuring the fair scheduler Future plans. Motivation. Provide short response times to small jobs in a shared Hadoop cluster

jana
Download Presentation

The Hadoop Fair Scheduler

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Hadoop Fair Scheduler Matei Zaharia Cloudera / Facebook / UC Berkeley UC Berkeley

  2. Outline • Motivation / Hadoop usage at Facebook • Fair scheduler basics • Configuring the fair scheduler • Future plans

  3. Motivation • Provide short response times to small jobs in a shared Hadoop cluster • Improve utilization over private clusters / HOD

  4. Hadoop Usage at Facebook • Data warehouse running Hive • 600 machines, 4800 cores, 2.4 PB disk • 3200 jobs per day • 50+ engineers have used Hadoop

  5. Facebook Data Pipeline Web Servers Scribe Servers Network Storage Hive Queries Analysts Summaries Hadoop Cluster MySQL Oracle RAC

  6. Facebook Job Types • Production jobs: load data, compute statistics, detect spam, etc • Long experiments: machine learning, etc • Small ad-hoc queries: Hive jobs, sampling GOAL: Provide fast response times for small jobs and guaranteed service levels for production jobs

  7. Outline • Motivation / Hadoop usage at Facebook • Fair scheduler basics • Configuring the fair scheduler • Future plans

  8. Fair Scheduler Basics • Group jobs into “pools” • Assign each pool a guaranteed minimum share (split up among its jobs) • Split excess capacity evenly between jobs

  9. Pools • Determined from a configurable job property • Default (before 0.20): mapred.queue.name • At Facebook: user.name (one pool per user) • Unmarked jobs go into a “default pool” • Pools have properties: • Minimum map slots • Minimum reduce slots • Limit on # of running jobs

  10. Scheduling Algorithm • Divide each pool’s min share among its jobs • Divide excess capacity among all jobs* • When a slot needs to be assigned: • If there is any job below its min share, schedule it • Else schedule the job that we’ve been most unfair to (based on “deficit”) * Fair schedulers from Hadoop 0.20 on will share equally between pools, not jobs; patch available at https://issues.apache.org/jira/browse/HADOOP-4789

  11. Scheduler Dashboard

  12. Scheduler Dashboard

  13. Scheduler Dashboard Reassign pool

  14. Scheduler Dashboard Change priority

  15. Scheduler Dashboard FIFO mode (for testing)

  16. Additional Features • Job weights for unequal sharing: • Based on priority (each level is 2x more) • Based on size (mapred.fairscheduler.sizebasedweight) • Limits on # of running jobs: • Per user • Per pool

  17. Outline • Motivation / Hadoop usage at Facebook • Fair scheduler basics • Configuring the fair scheduler • Future plans

  18. Installing the Scheduler • Compile it: • ant package • Place it on the classpath: • cp build/contrib/fairscheduler/*.jar lib • Alternatively, add the JAR to HADOOP_CLASSPATH in conf/hadoop-env.sh

  19. Configuration Files • Hadoop config (conf/hadoop-site.xml) • Contains scheduler options, pointer to pools file • Pools file (pools.xml) • Contains min share allocations and limits on pools • Reloaded every 15 seconds to allow reconfiguring pools at runtime

  20. Minimal hadoop-site.xml <property> <name>mapred.jobtracker.taskScheduler</name> <value>org.apache.hadoop.mapred.FairScheduler</value> </property> <property> <name>mapred.fairscheduler.allocation.file</name> <value>/path/to/pools.xml</value> </property>

  21. Minimal pools.xml <?xml version="1.0"?> <allocations> </allocations>

  22. Configuring a Pool <?xml version="1.0"?> <allocations> <pool name="ads"> <minMaps>10</minMaps> <minReduces>5</minReduces> </pool> </allocations> • Any pools not configured in pools.xml will have minMaps=0 and minReduces=0

  23. Setting Running Job Limits <?xml version="1.0"?> <allocations> <pool name="ads"> <minMaps>10</minMaps> <minReduces>5</minReduces> <maxRunningJobs>3</maxRunningJobs> </pool> <user name="matei"> <maxRunningJobs>1</maxRunningJobs> </user> </allocations>

  24. Default Jobs Limit for Users <?xml version="1.0"?> <allocations> <pool name="ads"> <minMaps>10</minMaps> <minReduces>5</minReduces> <maxRunningJobs>3</maxRunningJobs> </pool> <user name="matei"> <maxRunningJobs>1</maxRunningJobs> </user> <userMaxJobsDefault>10</userMaxJobsDefault> </allocations>

  25. Other hadoop-site.xml Properties mapred.fairscheduler.assignmultiple: • Assign a map and reduce on each heartbeat; improves ramp-up speed and throughput; recommendation: set to true

  26. Other hadoop-site.xml Properties mapred.fairscheduler.poolnameproperty: • Which jobconf property to use to determine what pool a job is in • Default: mapred.queue.name (queue name) • Another useful option: user.name • Can also make up your own, e.g. “project”

  27. Other hadoop-site.xml Properties mapred.fairscheduler.weightadjuster: • Allows modifying job weights through a plugin class; one useful example is provided – a new job booster to let short jobs finish faster: Please see README for details <property> <name>mapred.fairscheduler.weightadjuster</name> <value>org.apache.hadoop.mapred.NewJobWeightBooster</value> </property>

  28. Outline • Motivation / Hadoop usage at Facebook • Fair scheduler basics • Configuring the fair scheduler • Future plans

  29. Future Plans • Share equally between pools, not jobs (Hadoop 0.20 release, HADOOP-4789) • Preemption if a job is starved of its min or fair share for some timeout (HADOOP-4665) • Locality wait optimization (HADOOP-4667)

  30. Future Plans • Simpler scheduling model (HADOOP-4803) • FIFO pools (HADOOP-4803, HADOOP-5186) • Delayed job initialization (HADOOP-5186) • Scalability and operational improvements

  31. Thanks! • The Fair Scheduler is available in Hadoop 0.19; docs in src/contrib/fairscheduler/README • Hadoop 0.17 and 0.18 versions at http://issues.apache.org/jira/browse/HADOOP-3746 matei@cloudera.com

More Related