1 / 25

Communication Steps For Parallel Query Processing

Communication Steps For Parallel Query Processing. Paraschos Koutris Paul Beame Dan Suciu. University of Washington PODS 2013. Motivation. Understand the complexity of parallel query processing on big data Focus on shared-nothing architectures MapReduce is such an example

carlos-armstrong
Download Presentation

Communication Steps For Parallel Query Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Communication Steps For Parallel Query Processing ParaschosKoutris Paul Beame Dan Suciu University of Washington PODS 2013

  2. Motivation • Understand the complexity of parallel query processing on big data • Focus on shared-nothingarchitectures • MapReduceis such an example • Dominating parameters of computation: • Communication cost • Number of communication rounds

  3. Computation Models • The MapReducemodel • [Afratiet al., 2012] tradeoff between reducer size (input size of a reducer) and replication rate (in how many reducers a tuple is sent) • The MUD model [Feldman et al., 2010] • (Massive, Unordered, Distributed) model • The MRC model [Karloff et al., 2010] • MapReduce computation + load balancing

  4. The MPC Model • N: total input size (in bits) • p:number of servers • Servers have unlimited computational power • Computation proceeds in synchronous rounds: • Local computation • Global communication Server 1 Input N . . . . . . . . . . . . Server p Round 1 Round 2

  5. MPC Parameters • Each server receives in total a bounded number of bits: O(N/p× pε) 0 ≤ ε < 1 • Complexity parameters: • Number of computation roundsr • Space exponent ε(governs data replication) What are the space exponent/round tradeoffs for query processing in the MPC model ?

  6. Our Results • ONE ROUND: • Lower bounds on the space exponent for any (randomized) algorithm that computes a Conjunctive Query • The lower bound holds for a class of inputs (matching databases), for which we show tight upper bounds • MULTIPLE ROUNDS: • Almost tight space exponent/round tradeoffs for tree-like Conjunctive Queries under a weaker communication model

  7. Outline • Warm-up: The Triangle Query • One Communication Round • Multiple Communication Rounds

  8. Conjunctive Queries • We mainly study full Conjuctive Queries w/o self-joins: Q(x, y, z, w, v) = R(x,y,z), S(x,w,v), T(v,z) • The hypergraphof the query Q: • Variables as vertices • Atoms as hyperedges S x w R y v z T

  9. The Triangle Query (1) • Find all trianglesQ(x,y,z) = R(x,y), S(y,z),T(z,x) • 2-round Algorithm: • ROUND 1: [R hash-join S] • R(a, b)  h(b) • S(b, c)  h(b) • JoinlocallyU(a, b, c) = {R(a, b), S(b, c)} • ROUND 2: [T’ hash-join T] • U(a, b, c)  h(c) • T(c, a)  h(c) • JoinlocallyQ(a,b,c) = {U(a, b ,c), T(c, a)} • Replicationε = 0

  10. The Triangle Query (2) • 1-round Algorithm:[Ganguly ’92, Afrati ’10, Suri’11] • The p servers form a cube: [p1/3] ×[p1/3] × [p1/3] • Sendeachtupleto servers: • R(a, b)  (h1(a), h2(b), - ) • S(b, c)  (-, h2(b), h3(c) ) eachtuplereplicatedp1/3times • T(c, a) (h1(a), -, h3(c) ) • Replicationε = 1/3 • (h1(a), h2(b), h3(c))

  11. Lower Bound For Triangles (1) • Theorem: No (randomized) algorithm can compute triangles in one round with space exponent ε < 1/3 • Say that R, S,T are random permutations over [n]2 • Expected #triangles = 1 Lemma: For any deterministic algorithm and ε=0, the p servers report in expectation O(1/p1/2) tuples • Each relation contains N = (n logn) bits of information • Any server knows a 1/pfraction of input: N/pbits

  12. Lower Bound For Triangles (2) • axy = Pr[server knows tuple R(x,y)] • axy ≤ 1/n • ∑x,yaxy = O(n/p) • Similarly for S(y,z), T(z,x): byz, czx • Friedgut’s inequality: ∑x,y,zaxybyzczx ≤ (∑x,yaxy2)1/2 (∑y,zbyz2)1/2 (∑z,xczx2)½ • #know-triangles = O(1/p3/2) • Summing over all servers, O(1/p1/2) known output tuples

  13. Outline • Warm-up: The Triangle Query • One Communication Round • Multiple Communication Rounds

  14. Matching Databases • Every relation R(A1, …, Ak) contains exactly n tuples • Every attribute Aicontains each value in {1, …, n} only once • A matching database has no skew Relation R(X, Y, Z) 1 1 2 2 3 … … … n-1 n n

  15. Fractional Vertex Cover • Vertex cover number τ:minimum number of variables that cover every hyperedge • Fractional vertex cover number τ*: minimum weight of variables such that each hyperedge has weight at least 1 Vertex Cover τ = 2 1/2 Fractional Vertex Cover τ* = 3/2 x 0 w 1/2 y 0 v 1/2 z Q(x, y, z, w, v) = R(x,y,z), S(x,w,v), T(v,z)

  16. Lower Bounds • Theorem: Any randomized algorithm in the MPC model will fail to compute a Conjunctive Query Q with: • 1 round • ε < 1 – 1/ τ*(Q) • Input a matching database

  17. Upper Bounds • Theorem: The HYPERCUBE (randomized) algorithm can compute any Conjunctive QueryQwith: • 1 round • ε≥ 1 – 1/ τ*(Q) • Input a matching database (no skew) • Exponentially small probability of failure (on input N)

  18. HyperCube Algorithm • Q(x1,…, xk) = S1(…), …, Sl(…) • Compute τ* and minimum cover: v1,v2, …, vk • Assign to each variable xi a share exponent e(i) = vi / τ* • Assigneachof the p servers topoints on a k-dimensional hypercube: [p] = [pe(1)] × … × [pe(k)] • Hasheachtupleto the appropriatesubcube Q(x,y,z,w,v)=R(x,y,z),S(x,w,v),T(v,z) τ* = 3/2 : vx = vv = vz = ½ vy = vw = 0 e(x) = e(v) = e(z) = 1/3 e(y) = e(w) = 0 [p] =[p1/3]×[p0]×[p1/3]×[p0]×[p1/3] e.g. S(a,b,c)  (hx(a), 1, -, 1, hv(c))

  19. Examples • Cycle query: Ck(x1,…, xk) = S1(x1, x2), …, Sk(xk, x1) • τ*= k/2 • ε= 1 - 2/k • Star query: Tk(z, x1,…, xk) = S1(z, x1), …, Sk(z, xk) • τ*= 1 • ε= 0 • Line query: Lk(x0, x1,…, xk) = S1(x0, x1), …, Sk(xk-1, xk) • τ*=k/2 • ε= 1 - 1/ k/2

  20. Outline • Warmup: The Triangle Query • One Communication Round • Multiple Communication Rounds

  21. Multiple Rounds • Ourresultsapplyto a weakermodel (tuple-based MPC): • onlyjointuplescan be sent in rounds > 1 e.g.{R(a,b), S(b,c)} • routingofeachtuple tdependsonly on t • Theorem: For every tree-like query Q, any tuple-based MPC algorithm requires at least • log(diam(Q)) / log2/(1-ε) rounds • This bound agrees with the upper bound within 1 round • diam(Q): largestdistancebetweentwovertices in the hypergraph • tree-like queries: #variables + #atoms - Σ(arities) = 1

  22. Example • Line query: Lk(x0,x1,…, xk) = S1(x0,x1), …, Sk(xk-1,xk) • tree-like: • #variables = k+1 • #atoms = k • Σ(arities) = 2k • diam(Lk) = k • For space exponent ε = 0, we need at least log(k)/log(2/(1-0)) = log(k) rounds diameter = k xk x1 x2 x3 x4 …

  23. Connected Components • As a corollaryofourresults for multiplerounds, weobtainlowerboundsbeyondconjunctivequeries: Theorem: Any tuple-based MPC algorithm that computes the Connected Components in any undirected graph with space exponent any ε<1 requires requires Ω(logp)communicationrounds

  24. Conclusions • Tightlower and upper boundsfor one communication round in the MPC model • The first lower bounds for multiple communication rounds • Connected components cannot be computed in a constant number of rounds • Open Problems: • Lower and upper bounds for skewed data • Lower bounds for > 1 rounds in the general model

  25. Thank you !

More Related