Pipelining and load-balancing in parallel joins on distributed machines

Size: px

Start display at page:

Download "Pipelining and load-balancing in parallel joins on distributed machines"

Alisha Newman
8 years ago
Views:

1 NP-PAR 05 p. / Pipelining and load-balancing in parallel joins on distributed machines M. Bamha bamha@lifo.univ-orleans.fr Laboratoire d Informatique Fondamentale d Orléans (France)

2 NP-PAR 05 p. / Plan Introduction Shared Nothing Machines Join operations in RDBMS Mono-processor join and its complexity Parallel Join and its complexity Parallelization of join operation using hash functions Problem of load imbalance due to data skew Attribute Value Skew (AVS) Join Product Skew (JPS) Osfa-join : a skew-insensitive parallel join algorithm Parallel execution strategies for multi-join queries Sequential Parallel vs Pipelined Osfa_join execution Conclusion

Join and its complexity Parallelization of join operation using hash functions Problem of load imbalance due to data

3 NP-PAR 05 p. / Shared Nothing machine (SN) Interconnection Network P P Pn Memory Memory Memory Disk(s) Disk(s) Disk(s)

4 NP-PAR 05 p. 4/ Join of two relations The join of two relations R and S on attribute A of R and attribute B of S is the relation T, written R S, obtained by concatenating the pairs of tuples from R and S for which R.A = S.B.

5 NP-PAR 05 p. 5/ Example -- Relation R Relation S R S Product Item Product Item prod prod prod prod4 prod5 prod6 6 tuples item item item item4 item5 item6 item7 7 tuples 4 5 prod prod prod prod prod prod prod prod4 prod4 prod4 prod5 prod5 prod5 item4 item5 item4 item5 item item item6 item item item6 item item item6 tuples

prod4 prod5 prod6 6 tuples item item item item4 item5 item6 item7 7 tuples 4 5

6 NP-PAR 05 p. 6/ Join s complexity The sequential join processing of two relations R and S requires at least the time for input and output: bound inf = Ω ( R + S + R S ). Parallel processing using p processors requires therefore bound infp = p Ω ( R + S + R S )

7 NP-PAR 05 p. 7/ Parallel join Parallel join usually proceeds in two phases:. a redistribution phase where the relations to join are partitioned into distinct buckets. These buckets are generally generated using a hash function of the join attribute and sent to distinct processors. a join phase where each processor computes the join of its local buckets.

These buckets are generally generated using a hash function of the join attribute and

8 NP-PAR 05 p. 8/ Example -.- Number of processors = Hashing function : ( mod ) + Processor Processor Processor Relation R Relation R Relation R Produit Produit Produit prod5 prod prod6 tuples () () () prod tuples () prod prod4 tuples () () Relation S Relation S Relation S Item Item Item item item7 item6 tuples 4 5 () () () item item5 tuples () () item4 item tuples () ()

R Relation R Relation R Produit Produit Produit prod5 prod prod6 tuples () () () prod tuples

9 NP-PAR 05 p. 9/ Example -.- Processor Processor Processor Relation R Product prod prod4 prod5 tuples Relation R Product prod6 tuples Relation R Product prod prod tuples Relation S Relation S Relation S Item item item item6 tuples Item item 4 tuples Item item4 item5 item7 5 tuples

tuples Relation R Product prod6 tuples Relation R Product prod prod

10 NP-PAR 05 p. 0/ Example -.- Processor Processor Processor R S R S R S Product Item Product Item Product Item prod prod prod prod4 prod4 prod4 prod5 prod5 prod5 item item item6 item item item6 item item item6 0 tuples prod prod prod prod item4 item5 item4 item5 4 tuples 9 tuples

Product Item prod prod prod prod4 prod4 prod4 prod5 prod5 prod5 item

11 NP-PAR 05 p. / Problem of data skew Attribute Value Skew (AVS): It arises due to non-uniform distributions in the join attribute in the input relation. It may result a biased distribution of data among the processors whenever values are mapped to processors without taking their frequencies into account. Join Product Skew (JPS): it arises due to differences in the join selectivity of data processed by different nodes. JPS can occur even if the input relations have no AVS and is one of the subtler kinds of skew to detect.

12 NP-PAR 05 p. / Example -. Relation R Relation S R S Frequency 000 tuples Frequency tuples Frequency tuples

13 NP-PAR 05 p. / Example -. R R R Frequency tuples Frequency tuples Frequency tuples S S S Frequency Frequency Frequency tuples tuples 00 tuples R S R S R S Frequency Frequency Frequency tuples 0 tuples 000 tuples

000 tuples S S S Frequency Frequency Frequency 000 4 000 6 0 000

14 NP-PAR 05 p. 4/ Osfa_join algorithm Osfa join a skew insensitive parallel join, it proceeds as follow: Computes Histograms of bases relations, Creates communication templates according to the frequencies of join attribute Exchange data Computes local join avoids the slowdown due to AVS and JPS has an optimal complexity guarantee a linear speedup

follow: Computes Histograms of bases relations, Creates communication templates

15 NP-PAR 05 p. 5/ Example -. Relation R Relation S R S Frequency Frequency tuples tuples Frequency tuples

000 000 000 4 5 4 5 5 5 ++++++++++++++++

16 NP-PAR 05 p. 6/ Example -. Processor Processor Processor R R R Frequency Frequency Frequency tuples 46 tuples 4 tuples S S S Frequency Frequency tuples tuples Frequency tuples

Frequency 4 8 8 8 000 000 000 4 5 5 4 tuples 46 tuples 4

17 NP-PAR 05 p. 7/ Example -. Processor Processor Processor R S R S R S Frequency Frequency Frequency tuples 6700 tuples tuples

Frequency Frequency Frequency 4 664 664 67

18 NP-PAR 05 p. 8/ Parallel strategies for multi-join queries Optimization of the query execution, Allow efficient resource allocation.

19 NP-PAR 05 p. 9/ Sequential parallel execution (SP) Joins 4 Processors Time 0 0

20 NP-PAR 05 p. 0/ Parallel synchronous execution (PS) Joins 4 Processors Time 4 50

21 NP-PAR 05 p. / Segmented right-deep execution (SD) Joins 4 Processors Time 0 70

22 NP-PAR 05 p. / Full parallel execution (FP) Processors Time

23 NP-PAR 05 p. / equential parallel vs Pipelined Osfa_join a R 5. Create Hist(R) on a a a R. Create Hist(R) on a. Create Hist(S). 4. Compute comm. templates, Exchange data, Compute S= R join S and save it to disk, R. Create Hist(R) on a 5. Create Hist(S). 6. Create comm. templates, Exchange data, Compute S4=R join S and save it to disk. a S. Create Hist(S) on a. Compute comm. templates, Exchange data, Compute S=R join S and save it to disk, Sequential parallel execution R. Create Hist(R) on a R. Create Hist(R) on a a R. Create Hist(R) on a 4. Create comm. templates, Exchange data, Compute S4=R join S and save it to. Compute comm. templates, Exchange data, Create Hist(S) on a, 4. Compute S= R join S. a S. Create Hist(S) on a. Compute comm. templa Exchange data, Create Hist(S) on a,. Compute S= R join S Pipelined execution

24 NP-PAR 05 p. 4/ Pipelined Osfa_join Reduces disk input/output for intermediate join results, Allows flexible resource allocation, Avoids the slowdown due to AVS and JPS.

25 NP-PAR 05 p. 5/ Conclusion Pipelined parallelism has been induced successfully in Osfa join algorithm: Avoids the slowdown due to AVS and JPS, Guarantee a perfect balancing properties during all the stages of join computation, Reduces disk input/output for intermediate results, Allows flexible resource allocation, Guarantee a linear speed-up.

26 NP-PAR 05 p. 6/ Future work: Improving the pipelined algorithm to generate only relevant tuples for intermediate join results, Extend the algorithm to handle long chains of pipeline, Extend the algorithm for GRID computing.

Parallel Databases. Parallel Architectures. Parallelism Terminology 1/4/2015. Increase performance by performing operations in parallel

Parallel Databases. Parallel Architectures. Parallelism Terminology 1/4/2015. Increase performance by performing operations in parallel Parallel Databases Increase performance by performing operations in parallel Parallel Architectures Shared memory Shared disk Shared nothing closely coupled loosely coupled Parallelism Terminology Speedup: