1 / 19

Analytical Modeling of Parallel Programs

Analytical Modeling of Parallel Programs. Sources of Overhead in Parallel Programs Performance Metrics for Parallel Systems Formulating Maximum Speedup: Amdahl’s Law Scalability of Parallel Systems Review of Amdahl’s Law: Gustafson-Barsis’ Law. Analytical Modeling - Basics.

evelia
Download Presentation

Analytical Modeling of Parallel Programs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Analytical Modeling of Parallel Programs • Sources of Overhead in Parallel Programs • Performance Metrics for Parallel Systems • Formulating Maximum Speedup: Amdahl’s Law • Scalability of Parallel Systems • Review of Amdahl’s Law: Gustafson-Barsis’ Law

  2. Analytical Modeling - Basics • A sequential algorithm is evaluated by its runtime (in general, asymptotic runtime as a function of input size). • The asymptotic runtime of a sequential program is identical on any serial platform. • On the other hand, the parallel runtime of a program depends on • the input size, • the number of processors, and • the communication parameters of the machine. • An algorithm must therefore be analyzed in the context of the underlying platform. • A parallel system is a combination of a parallel algorithm and an underlying platform.

  3. Sources of Overhead in Parallel Programs • If I use n processors to run my program, would it run n times faster? • Overheads! • Interprocessor Communication & Interactions • Usually the most significant source of overhead • Idling • Load imbalance, Synchronization, Serial components • Excess Computation • Sub-optimal serial algorithm • More aggregate computations • Goal is to minimize these overheads!

  4. Performance Metrics for Parallel Programs • Why analyze the performance of parallel programs? • Determine the best algorithm • Examine the benefit of parallelism • A number of metrics have been used based on the desired outcome of performance analysis: • Execution time • Total parallel overhead • Speedup • Efficiency • Cost

  5. Performance Metrics for Parallel Programs • Parallel Execution Time • Time spent to solve a problem on p processors. • Tp • Total Overhead Function • To = pTp – Ts • Speedup • S = Ts/Tp • Can we have superlinear speedup? • exploratory computations, hardware features • Efficiency • E = S/p • Cost • pTp(processor-time product)

  6. Performance Metrics: Working Example

  7. Performance Metrics: Example on Speedup • What is the benefit from parallelism? • Consider the problem of adding n numbers by using n processing elements. • If n is a power of two, we can perform this operation in log n steps by propagating partial sums up a logical binary tree of processors. • If an addition takes constant time, say, tcand communication of a single word takes time ts + tw, the parallel time is TP = Θ (log n) • We know that TS = Θ (n) • Speedup S is given by S = Θ (n / log n)

  8. Performance Metrics: Speedup Bounds • For computing speedup, the best sequential program is taken as the baseline. • There may be different sequential algorithms with different asymptotic runtimes for a given problem • Speedup can be as low as 0 (the parallel program never terminates). • Speedup, in theory, should be upper bounded by p. • In practice, a speedup greater than p is possible. • This is known as superlinear speedup • Superlinear speedup can result • when a serial algorithm does more computations than its parallel formulation • due to hardware features that put the serial implementation at a disadvantage • Note that superlinear speed happens only if each processing element spends less than time TS/p solving the problem.

  9. Performance Metrics: Superlinear Speedups • Superlinearity effect due to exploratory decomposition

  10. Cost of a Parallel System • As shown earlier, Cost is the product of parallel runtime and the number of processing elements used (pTP). • Cost reflects the sum of the time that each processing element spends solving the problem. • A parallel system is said to be cost-optimal if the cost of solving a problem on a parallel computer is asymptotically identical to serial cost. • Since E = TS / p TP, for cost optimal systems, E = O(1). • Cost is sometimes referred to as work or processor-time product. • The problem of adding n numbers on n processors is not cost-optimal.

  11. Formulating Maximum Speedup • Assume an algorithm has some sequential parts that are only executed on one processor. • Assume the fraction of the computation that cannot be divided into concurrent tasks is f. • Assume no overhead incurs when the computation is divided into concurrent parts. • The time to perform the computation with pprocessors is: • Hence, the speedup factor is (Amdahl’s Law):

  12. Visualizing Amdahl’s law

  13. Speedup Against Number of Processors

  14. Speedup against number of processors • From the preceding formulation, f has to be a small fraction of the overall computation if significant increase in speedup is to occur • Even with infinite number of processors, maximum speedup limited to 1/f: • Example: With only 5% of computation being serial, maximum speedup is 20, irrespective of number of processors. • Amdahl used this argument to promote single processor machines

  15. Scalability • Speedup and efficiency are relative terms. They depend on • Number of processors • Problem size • The algorithm used • For example, efficiency of a parallel program often decreases as the number of processors increases • Similarly, a parallel program may be quite efficient for solving large problems, but not for solving small problems • A parallel program is said to scale if its efficiency is constant for a broad range of number of processors and problem sizes • Finally, speedup and efficiency depend on the algorithm used. • A parallel program might be efficient relative to one sequential algorithm but not relative to a different sequential algorithm

  16. Gustafson’s Law • Presented an argument based upon scalability concepts. • To show that Amdahl’s law was not as significant as first supposed in limiting the potential speedup. • Observation: In practice a larger multiprocessor usually allows a larger size of the problem to be undertaken in a reasonable execution time. • Hence, the problem size is not independent of the number of processors. • Rather than assume the problem size is fixed, we should assume that the parallel execution time is fixed. • Using the parallel constant execution time constraint, the resulting speedup factor will be numerically different from Amdahl’s speedup factor and is called a scaled speedup factor

  17. Speedup vs Number of Processors

  18. Speedup vs Number of Processors

  19. Formulating Gustafson’s Law • Assuming the parallel execution time, Tp, is normalized to unity: • Assuming that in the serial execution time, Ts, below, fTs is a constant, • Then the scaled speedup factor (Gustafson’s Law) is:

More Related