To GPU Synchronize or Not GPU Synchronize?

1. To GPU Synchronize or Not GPU Synchronize? Wu-chun Feng and Shucai Xiao Department of Computer Science, Department of Electrical and Computer Engineering, Virginia Tech ISCAS 2010

2. Outline • Introduction • Preliminaries • Related Works • Proposed GPU-based Synchronization • Problems, Experiments, and Analysis • Conclusions

3. Introduction • Multi(many)-core era has come. • General purpose GPU (GPGPU) allows massively parallel computation with low cost. • GPUs typically map well only to data-parallel or task-parallel applications • Due to the lack of support for communication between streaming multiprocessors (SMs).

4. Introduction (cont.) • Communication could be done via global memory. • Need barrier synchronization. • CPU barrier synchronization • By (inefficiently) implementing the barrier synchronization via the host CPU. • Slow.

5. Introduction (cont.) • GPU barrier synchronization • Improve performance by 10~40%. • Theoretically run the risk that barrier may release earlier. • CUDA 2.2 support new function _threadfence() to solve this problem. • The correctness could be guarantee.

6. Introduction (cont.) • Unfortunately, _threadfence() incurs so much overhead in the proposed GPU barrier synchronization. • That is, CPU barrier synchronization performs as well as or better than the GPU barrier synchronizations in many cases. • “Whether to GPU synchronize or not GPU synchronize?”

7. Preliminaries: CUDA • Compute Unified Device Architecture • Developed by nVIDIA. • The CPU code does the sequential part. • Highly parallelized part usually implement in the GPU code, called kernel. • Calling GPU function in CPU code is called kernel launch. • In a kernel, threads are grouped as a grid of thread blocks, and each thread block contains a number of threads. • Multiple blocks can be executed on the same SM, but one block cannot be executed across different SMs.

8. Preliminaries: GPU architecture

9. Synchronization in parallel programming. Making sure that each thread get the right data for computation. CUDA provides a data communication mechanism for threads within a single block via the barrier function syncthreads(). Intra-SM communication. However, there is no explicit software or hardware support for data communication of threads across different blocks. Inter-SM communication. Preliminaries: synchronization

10. When multiple GPU thread blocks are scheduled to execute on a single SM simultaneously, deadlock might occurs. In multi-core environment, a process can yield its execution to other processes, but CUDA blocks do not. [17] assign only one block per SM to address this problem. Related Works

11. Related Works (cont.) • When a barrier synchronization is needed across different blocks, programmers traditionally use a kernel launch as a way to implicitly barrier synchronize [4], [7]. • [14] propose a protocol for data communication across multiple GPUs. • Data needs to be transferred to the host memory first and then copied back to the device memory, and hence poor performance in different SMs on a single GPU.

12. GPU synchronization Lock-based synchronization. Single mutex variable for all thread blocks. Once a block finishes its computation on an SM, it atomically increments the mutex variable. Lock-free synchronization. One distinct variable to control each block, thus eliminating the need for different blocks to contend for the single mutex variable. The need for atomic addition is removed. Proposed GPU-based Synchronization

13. Experiments • Environment • GeForce GTX 280: 30SMs, 8 cores each, running at 1.3GHz (shader clock). • CUDA 2.2 SDK. • Details are omitted. • Two experiments. • Dynamic programming (DP) for genomic sequence alignment (specifically the Smith-Waterman algorithm). • Bitonic sort (BS).

14. Performance comparisons

15. In order to eliminate the infinitesimal risk that barrier may release earlier when the proposed synchronization run, _threadfence() is used and hence incurred the overhead. The same experiments with modified barrier using _threadfence(). Problems, Experiments, and Analysis

16. Performance comparisons

17. Problems, Experiments, and Analysis (cont.) • Analyze GPU lock-based synchronization as an example, whoseoperation set is a superset of thatare used in the other one. • Synchronization overhead • ta is the overhead of atomic add • tc is the mutex variable checking time • ts is the time consumed by syncthreads() • tf is the threadfence() execution time.

18. Problems, Experiments, and Analysis (cont.) • Unfortunately, the execution times for these component operations cannot be measured directly on the GPU. • An indirect approach. • A kernel’s execution time can be expressed as • Measuring the kernel execution time both with and without a specific operation and taking the time difference as the overhead of that operation.

19. Execution time profiling • Use micro benchmark to test. • Calculates the average of two floats over 10,000 iterations. • CPU synchronization • Each kernel calculates the average once, and the kernel is launched 10,000 times. • GPU synchronization • The kernel is launched only once, and there is a 10,000-iteration for loop used in the kernel with the GPU barrier function called in each loop.

20. Results t5 t4 t2 t1 • For 10,000 times of execution, ts = 0.541, ta = 2.300 × n, tc = 5.564, and tf = 0.333 × n + 7.267, where n is the number of blocks in the kernel, and the units are in milliseconds. t3 ts=t3-t1 ta=t2 tc=t4-t3-t2 tf=t5-t4

21. Demonstrate the efficiency of inter-SM communication using GPU-based barrier synchronization. To eliminate the risk of asynchronous, threadfence() is used though it incurs high overhead. Grudgingly conclude that one should GPU synchronize (with or without threadfence()) on the current generation, but more definitive ’yes’ for the next generation GPUs. Conclusions