The Pig Experience: Building High-Level Data flows on top of Map-Reduce

1. DISTRIBUTED INFORMATION SYSTEMS The Pig Experience: Building High-Level Data flows on top of Map-Reduce Presenter: Javeria Iqbal Tutor: Dr.Martin Theobald

2. Outline • Map-Reduce and the need for Pig Latin • Pig Latin • Compilation into Map-Reduce • Optimization • Future Work

3. Data Processing Renaissance • Internet companies swimming in data • TBs/day for Yahoo! Or Google! • PBs/day for FaceBook! • Data analysis is “inner loop” of product innovation

4. Data Warehousing …? • High level declarative approach • Little control over execution method Scale • Prohibitively expensive at web scale • Up to $200K/TB Price SQL Often not scalable enough

5. Map-Reduce Map : Performs filtering Reduce : Performs the aggregation These are two high level declarative primitives to enable parallel processing BUT no complex Database Operations e.g. Joins

6. Execution Overview of Map-Reduce Buffered pairs are written to local disk partitions, Location of buffered pairs are sent to reduce workers Worker reads, parses key/value pairs and passes pairs to user-defined Map function Split the Program Master and Worker Threads

7. Execution Overview of Map-Reduce Unique keys, values are passed to user’s Reduce function. Output is appended to the output file for this reduce partition. Reduce worker sorts data by the intermediate keys.

8. The Map-Reduce Appeal • Scalable due to simpler design • Explicit programming model • Only parallelizable operations Scale Runs on cheap commodity hardware Less Administration Price SQL Procedural Control- a processing “pipe”

9. Disadvantages M R 1. Extremely rigid data flow Other flows hacked in M M M R Chains Join, Union Split • 2. Common operations must be coded by hand • Join, filter, projection, aggregates, sorting, distinct • 3. Semantics hidden inside map-reduce functions • Difficult to maintain, extend, and optimize • 3. No combined processing of multiple Datasets • Joins and other data processing operations

10. Motivation Need a high-level, general data flow language

11. Enter Pig Latin Need a high-level, general data flow language Pig Latin

12. Outline • Map-Reduce and the need for Pig Latin • Pig Latin • Compilation into Map-Reduce • Optimization • Future Work

13. Pig Latin: Data Types Rich and Simple Data Model Simple Types: int, long, double, chararray, bytearray Complex Types: Atom: String or Number e.g. (‘apple’) Tuple: Collection of fields e.g. (áppe’, ‘mango’) Bag: Collection of tuples { (‘apple’ , ‘mango’) (ápple’, (‘red’ , ‘yellow’)) } Map: Key, Value Pair

14. Example: Data Model Atom: contains Single atomic value • Tuple: sequence of fields • Bag: collection of tuple with possible duplicates ‘lanker’ ‘ipod’ ‘alice’ Atom Tuple

15. Pig Latin: Input/Output Data Input: queries = LOAD `query_log.txt' USING myLoad() AS (userId, queryString, timestamp); Output: STORE query_revenues INTO `myoutput' USING myStore();

16. Pig Latin: General Syntax Discarding Unwanted Data: FILTER Comparison operators such as ==, eq, !=, neq Logical connectors AND, OR, NOT

17. Pig Latin: Expression Table

18. Pig Latin: FOREACH with Flatten expanded_queries = FOREACH queries GENERATE userId, expandQuery(queryString); ----------------- expanded_queries = FOREACH queries GENERATE userId, FLATTEN(expandQuery(queryString));

19. Pig Latin: COGROUP Getting Related Data Together: COGROUPSuppose we have two data setsresult: (queryString, url, position)revenue: (queryString, adSlot, amount)grouped_data = COGROUP result BY queryString, revenue BY queryString;

20. Pig Latin: COGROUP vs. JOIN

21. Pig Latin: Map-Reduce Map-Reduce in Pig Latinmap_result = FOREACH input GENERATE FLATTEN(map(*));key_group = GROUP map_result BY $0;output = FOREACH key_group GENERATE reduce(*);

22. Pig Latin: Other Commands UNION : Returns the union of two or more bags CROSS: Returns the cross product ORDER: Orders a bag by the specified field(s) DISTINCT: Eliminates duplicate tuple in a bag

23. Pig Latin: Nested Operations grouped_revenue = GROUP revenue BY queryString; query_revenues = FOREACH grouped_revenue { top_slot = FILTER revenue BY adSlot eq `top'; GENERATE queryString, SUM(top_slot.amount), SUM(revenue.amount); };

24. Pig Pen: Screen Shot

25. Pig Latin: Example 1 Suppose we have a table urls: (url, category, pagerank) Simple SQL query that finds, For each sufficiently large category, the average pagerank of high-pagerank urls in that category SELECT category, Avg(pagetank) FROM urls WHERE pagerank > 0.2 GROUP BY category HAVING COUNT(*) > 106

26. Data Flow Filter good_urlsby pagerank > 0.2 Group by category Filter categoryby count > 106 Foreach category generate avg. pagerank

27. Equivalent Pig Latin good_urls = FILTER urls BY pagerank > 0.2; groups = GROUP good_urls BY category; big_groups = FILTER groups BY COUNT(good_urls) > 106 ; output = FOREACH big_groups GENERATE category, AVG(good_urls.pagerank);

28. Example 2: Data Analysis Task Find the top 10 most visited pages in each category Visits Url Info

29. Data Flow Load Visits Group by url Foreach url generate count Load Url Info Join on url Group by category Foreach category generate top10 urls

30. Equivalent Pig Latin visits = load‘/data/visits’ as (user, url, time); gVisits = group visits by url; visitCounts = foreach gVisits generate url, count(visits); urlInfo = load‘/data/urlInfo’ as (url, category, pRank); visitCounts = join visitCounts by url, urlInfo by url; gCategories = group visitCounts by category; topUrls = foreach gCategories generate top(visitCounts,10); store topUrls into ‘/data/topUrls’;

31. Quick Start and Interoperability Operates directly over files visits = load‘/data/visits’ as (user, url, time); gVisits = group visits by url; visitCounts = foreach gVisits generate url, count(urlVisits); urlInfo = load‘/data/urlInfo’ as (url, category, pRank); visitCounts = join visitCounts by url, urlInfo by url; gCategories = group visitCounts by category; topUrls = foreach gCategories generate top(visitCounts,10); store topUrls into ‘/data/topUrls’;

32. Quick Start and Interoperability Schemas optional; Can be assigned dynamically visits = load‘/data/visits’ as (user, url, time); gVisits = group visits by url; visitCounts = foreach gVisits generate url, count(urlVisits); urlInfo = load‘/data/urlInfo’ as (url, category, pRank); visitCounts = join visitCounts by url, urlInfo by url; gCategories = group visitCounts by category; topUrls = foreach gCategories generate top(visitCounts,10); store topUrls into ‘/data/topUrls’;

33. User-Code as a First-Class Citizen • User-defined functions (UDFs) can be used in every construct • Load, Store • Group, Filter, Foreach visits = load‘/data/visits’ as (user, url, time); gVisits = group visits by url; visitCounts = foreach gVisits generate url, count(urlVisits); urlInfo = load‘/data/urlInfo’ as (url, category, pRank); visitCounts = join visitCounts by url, urlInfo by url; gCategories = group visitCounts by category; topUrls = foreach gCategories generate top(visitCounts,10); store topUrls into ‘/data/topUrls’;

34. Nested Data Model finance yahoo , email news • Pig Latin has a fully nested data model with: • Atomic values, tuples, bags (lists), and maps • Avoids expensive joins

35. Nested Data Model group by url • Common case: aggregation on these nested sets • Power users: sophisticated UDFs, e.g., sequence analysis • Efficient Implementation (see paper) I frankly like pig much better than SQL in some respects (group + optional flatten), I love nested data structures).” Ted Dunning Chief Scientist, Veoh Decouples grouping as an independent operation 35

36. CoGroup results revenue Cross-product of the 2 bags would give natural join

37. Pig Features Explicit Data Flow Language unlike SQL Low Level Procedural Language unlike Map-Reduce Quick Start & Interoperability Mode (Interactive Mode, Batch, Embedded) User Defined Functions Nested Data Model

38. Outline Map-Reduce and the need for Pig Latin Pig Latin Compilation into Map-Reduce Optimization Future Work

39. Pig Process Life Cycle Parser PigLatinto Logical Plan Logical Optimizer Logical Plan to Physical Plan Map-Reduce Compiler Map-Reduce Optimizer Hadoop Job Manager

40. Pig Latin to Physical Plan LOAD LOAD FILTER JOIN GROUP FOREACH STORE A = LOAD ‘file1’ AS (x,y,z); B = LOAD ‘file2’ AS (t,u,v); C = FILTER A by y > 0; D = JOIN C by x,B by u; E = GROUP D by z; F = FOREACH E generate group, COUNT(D); STORE F into ‘output’; x,y,z,t,u,v x,y,z group , count

41. Logical Plan to Physical Plan 3 LOAD 1 1 LOAD 3 LOAD FILTER LOAD 2 2 4 GLOABL REARRANGE LOCAL REARRANGE FILTER JOIN 4 4 PACKAGE 5 4 FOREACH GROUP 4 6 LOCAL REARRANGE GLOBAL REARRANGE FOREACH 5 5 PACKAGE 7 5 FOREACH 6 STORE STORE 7

42. Physical Plan to Map-Reduce Plan LOAD 1 LOAD 3 FILTER 2 GLOABL REARRANGE LOCAL REARRANGE 4 4 PACKAGE 4 FOREACH 4 LOCAL REARRANGE GLOBAL REARRANGE 5 5 PACKAGE 5 FOREACH 6 STORE 7 Filter Local Rearrange Package Foreach LocalRearrange Package Foreach

43. Implementation SQL user automatic rewrite + optimize Pig Pig is open-source. http://hadoop.apache.org/pig or or Hadoop Map-Reduce cluster • ~50% of Hadoop jobs at • Yahoo! are Pig • 1000s of jobs per day

44. Compilation into Map-Reduce Every group or join operation forms a map-reduce boundary Map1 Load Visits Group by url Reduce1 Map2 Foreach url generate count Load Url Info Join on url Reduce2 Map3 Group by category Other operations pipelined into map and reduce phases Reduce3 Foreach category generate top10(urls)

45. Nested Sub Plans FILTER SPLIT LOCAL REARRANGE FILTER FILTER GLOBALREARRANGE FOEACH FOREACH MULTIPLEX PACKAGE PACKAGE FOREACH FOREACH SPLIT

46. Outline Map-Reduce and the need for Pig Latin Pig Latin Compilation into Map-Reduce Optimization Future Work

47. Using the Combiner Input records Output records map reduce map reduce • Can pre-process data on the map-side to reduce data shipped • Algebraic Aggregation Functions • Distinct processing

48. Skew Join • Default join method is symmetric hash join. cross product carried out on 1 reducer • Problem if too many values with same key • Skew join samples data to find frequent values • Further splits them among reducers

49. Fragment-Replicate Join • Symmetric-hash join repartitions both inputs • If size(data set 1) >> size(data set 2) • Just replicate data set 2 to all partitions of data set 1 • Translates to map-only job • Open data set 2 as “side file”

50. Merge Join • Exploit data sets are already sorted. • Again, a map-only job • Open other data set as “side file”