1 / 34

Efficient Skyline Computation on Vertically Partitioned Datasets

Efficient Skyline Computation on Vertically Partitioned Datasets. Dimitris Papadias , David Yang, Georgios Trimponias CSE Department, HKUST, Hong Kong. Outline. Problem Statement Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm

sitara
Download Presentation

Efficient Skyline Computation on Vertically Partitioned Datasets

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Efficient Skyline Computation on Vertically Partitioned Datasets DimitrisPapadias, David Yang, GeorgiosTrimponiasCSE Department, HKUST, Hong Kong

  2. Outline • Problem Statement • Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm • Algorithms for Top-k Query Processing • FM Sketches • Putting Everything Together

  3. Outline • Problem Statement • Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm • Algorithms for Top-k Query Processing • FM Sketches • Putting Everything Together

  4. A Motivating Example • Consider a database containing information about hotels. The y-dimension represents the price of the room, whereas the x-dimension captures the distance of the room from the beach. Dominance Region of p Price Skyline objects p Hotel rooms Borders of p’s Dominance Region Distance

  5. Skyline Preliminaries[ICDE, 2001] • Skylines constitute a very useful tool in numerous disciplines, such as for multidimensional decision making and data mining. • Given a set of d-dimensional objects p1, …, pN, the skyline operator retrieves all these objects that are nor dominated by any other object in the set. • An object pi dominates another point pj, if it is not worse than pj in all dimensions and better than it in at least one dimension. • Properties: • The top-1 tuple according to any preference function that assigns scores to tuples is in the skyline tuple. • Conversely, for any skyline tuple, there exists a preference function according to which it is the top-1.

  6. 4 Common Data Distributions

  7. Problem Definition • Compute the skyline when the dataset is vertically decomposed among a set of N servers. • Goal: minimize the data that must be retrieved from each server. • We assume wireless environments, where communication overhead constitutes the dominant factor in battery consumption. • Consider mobile phone applications as real world examples.

  8. First Observations • The global skyline may contain points that do not appear in the local skylines. • Instead of transmitting all records over the network, avoid sending out points that are guaranteed to be dominated globally by an anchor point.

  9. Outline • Problem Statement • Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm • Algorithms for Top-k Query Processing • FM Sketches • Putting Everything Together

  10. Balke’s Algorithm[EDBT, 2004] • Assume that the d-dimensional database is vertically partitioned into d lists, one for each dimension, assigned to different servers. The lists contain values in ascending order. • Idea: perform sorted accesses on the d lists in a round-robin manner, until a point p (anchor), is reached in every list. • Points that have not showed up at this moment in any list can be safely pruned, since they are dominated by the anchor.

  11. Example • Let a 2-dimensional database with the following two lists: • L1 • L2 … …

  12. Example (cont.) • Let a 2-dimensional database with the following two lists: • L1 • L2 … … The first point to be retrieved from both lists.

  13. Example (cont.) • Let a 2-dimensional database with the following two lists: • L1 • L2 … … The first point to be retrieved from both lists. These points cannot be part of the skyline.

  14. Further Improvement • Efficiency can be improved, if instead of visiting the lists in a round-robin manner, we access the most promising list with random accesses. • As a result, only the least expansion is performed on each list. avoid visiting these points ∙ ∙ ∙ P ∙ ∙ ∙ L1 ∙ ∙ ∙ P ∙ ∙ ∙ L2

  15. Outline • Problem Statement • Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm • Algorithms for Top-k Query Processing • FM Sketches • Putting Everything Together

  16. Setting • Let N1, .., Nm be m servers storing the same dataset DB. • For each record PDB every server Ni maintains a local scoresi(P), and sorts all records in decreasing order of their local scores. • A client wishes to obtain the k records of DB with the maximum global score s. • The score is computed using a monotonic function f on the local scores, i.e., s(P) = f(s1(P), .., sm(P)). • Goal: minimize the required number of accesses.

  17. Fagin’s Algorithm[PODS, 2001] • Each server Ni performs sorted round-robin accesses and sends to the client the next record and its local score. • When the first common record Panc is encountered by all servers, the client terminates the sorted accesses. • Then, it obtains the missing local scores of the other encountered points through random accesses. • The candidate with the highest global score is the top-1 result.

  18. Threshold Algorithm[PODS, 2001] • It utilizes an upper bound TA on the global score to terminate earlier than FA. • The client retrieves the local scores of newly encountered points with random accesses at the remaining servers and computes their global scores, and picks the best score sbest. • The threshold TA is equal to the sum of the local thresholds at each server. • As long TA > sbest, TA continues the sorted accesses, while it keeps updating TA. • Eventually, the top-1 point will be returned.

  19. Example for FA and TA

  20. Best Position Algorithm[VLDB, 2007] • It further improves TA by utilizing a tighter threshold. • Let bpi be the position at server Ni such that all points up to bpi have been encountered through sorted or random accesses. • The global threshold BP is equal to the sum of the local thresholds at bpi.

  21. Outline • Problem Statement • Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm • Algorithms for Top-k Query Processing • FM Sketches • Putting Everything Together

  22. Flajolet / Martin sketches[JCSS ’85] • Goal: Estimate the distinct number of objects from a small-space representation of a set. • Sketch of a union of items is the OR of their sketches • Insertion order and duplicates don’t matter! Prerequisite:Let h be a random, binary hash function. Sketch of an item For each unique item with ID x, For each integer 1 ≤ i ≤ k in turn, Compute h (x, i). Stop when h (x, i) = 1, and set bit i. ∩

  23. Flajolet / Martin sketches (cont.) Estimating COUNT Take the sketch of a set of N items. Let j be the position of the leftmost zero in the sketch. j is an estimator of log2 (0.77 N) S 1 1 1 0 1 j = 3 Best guess: COUNT ~ 11 • Fixable drawbacks: • Estimate has faint bias • Variance in the estimate is large.

  24. Flajolet / Martin sketches (cont.) • Standard variance reduction methods apply. • Compute m independent sketches in parallel. • Compute m independent estimates of N. • Take the mean of the estimates. • Provable tradeoffs between m and variance of the estimator.

  25. S1∪S2∪S3 S4 S4 S3 S2 S1 S1 S2 Application to COUNT in Sensor Databases • Each sensor computes k independent sketches of itself (using unique ID x) • sensor computes a sketch of its value. • Use a robust routing algorithm to route sketches up to the sink. • Aggregate the k sketches via union en-route. (OR) • The sink then estimates the count. S1∪S2∪S3∪S3 sink

  26. Outline • Problem Statement • Skyline Computation on Vertically Partitioned Datasets using Balke’s Algorithm • Algorithms for Top-k Query Processing • FM Sketches • Putting Everything Together

  27. Problem Characteristics • Each vertical decomposition has arbitrary dimensionality, contrary to Balke’s setting. • Anchor selection substantially determines the total number of transmitted data. • VPS adopts sorting on the local dominance. In particular, the local dominance count domi(P)of a point P with respect to subspace Di is the number of points dominated by P in Di. • Balke selects as the anchor, the data point P with the maximal domSUM(P). • We utilize a tighter upper bound for dom(P) is the minimum domMIN among all local dominance counts.

  28. Anchor Selection C: optimal anchor point A: has maximal domMIN B: has maximal domSUM

  29. Our algorithm on the previous example

  30. 1st Optimization: Multiple Anchor Points • The previous algorithm performs pruning with a single anchor Panc. Specifically, a point P that is locally dominated by Panc in all subspaces is not sent to the client. • On the other hand, if P is incomparable with Panc even in a single subspace Di, it will be transmitted by the corresponding server Ni. • We suggest that multiple points can often achieve more effective pruning.

  31. Pruning with 2 points

  32. 2nd Optimization: Integration of Sketches • So far, we have estimated the (expected) global dominance dom(P) of a point P using domMIN(P). • This approach is biased towards points that have high local dominance counts in all subspaces, but dominate few records globally (A). • Thus, we propose an unbiased approach that directly estimates the global dominance counts using sketches that count the number of distinct objects approximately. • We assume that each Ni server has a local dominance sketchski(P) for every point P, which aggregates all points that P dominates locally in Di.

  33. Experiments

  34. Thank you!

More Related