HIGH PERFORMANCE FOURIER VOLUME RENDERING ON GRAPHICS PROCESSING UNITS (GPUS)

Transcription

1 HIGH PERFORMANCE FOURIER VOLUME RENDERING ON GRAPHICS PROCESSING UNITS (GPUS) By Marwan Mohamed Ahmed Abdellah Systems & Biomedical Engineering Department Faculty of Engineering, Cairo University A thesis submitted to the Faculty of Engineering, Cairo University in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE in SYSTEMS & BIOMEDICAL ENGINEERING FACULTY OF ENGINEERING CAIRO UNIVERSITY GIZA, EGYPT 2012

2 HIGH PERFORMANCE FOURIER VOLUME RENDERING ON GRAPHICS PROCESSING UNITS (GPUS) By Marwan Mohamed Ahmed Abdellah Systems & Biomedical Engineering Department Faculty of Engineering, Cairo University A thesis submitted to the Faculty of Engineering, Cairo University in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE in SYSTEMS & BIOMEDICAL ENGINEERING Under the supervision of Assoc. Prof. Ayman El-Dieb Assoc. Prof. Amr Shaarawi Systems & Biomedical Engineering Department Faculty of Engineering, Cairo University FACULTY OF ENGINEERING CAIRO UNIVERSITY GIZA, EGYPT 2012

3 HIGH PERFORMANCE FOURIER VOLUME RENDERING ON GRAPHICS PROCESSING UNITS (GPUS) By Marwan Mohamed Ahmed Abdellah Systems & Biomedical Engineering Department Faculty of Engineering, Cairo University A thesis submitted to the Faculty of Engineering, Cairo University in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE in SYSTEMS & BIOMEDICAL ENGINEERING Approved by the Examining Committee Prof. Dr. Yasser Kadah, Member Prof. Dr. Mohamed El-Adawy, Member Assoc. Prof. Dr. Ayman El-Dieb, Main Advisor Assoc. Prof. Dr. Amr Shaarawi, Thesis Advisor FACULTY OF ENGINEERING CAIRO UNIVERSITY GIZA, EGYPT 2012

4 Engineer : Marwan Mohamed Ahmed Abdellah Date of Birth : 5 / 7 / 1987 Nationality : Egyptian [email protected] Phone : Address : No. 4, Muhammad Kasem St., Maadi, Cairo, Egypt. Registration Date : 1 / 10 / 2009 Awarding Date : / / Degree : Master s of Science (M.Sc.) Department : Systems & Biomedical Engineering Department Supervisors : Assoc. Prof. Dr. Ayman M. Eldieb Assoc. Prof. Dr. Amr A. Shaarawi Examiners : Prof. Dr. Yasser Mostafa Kadah Prof. Dr. Mohamed Ibrahim. Eladawy Assoc. Prof. Dr. Ayman M. Eldieb Assoc. Prof. Dr. Amr A. Shaarawi (Faculty of Engineering Helwan University) Title of Thesis : High Performance Fourier Volume Rendering on Graphics Processing Units (GPUs) Key Words : Fourier Volume Rendering, Medical Image Reconstruction, Projection-slice Theory, GPU Computing, CUDA. Summary : The past years have seen tremendous advances in volume visualization techniques that have been used broadly in medical imaging. In particular, volume rendering techniques have received a considerable attention in this area. However, spatial domain volume rendering has achieved a wide acceptance from scientists and physicians, but this category of rendering techniques was associated with several constrains due to their O(N 3 ) time-complexity, which limited their usability in several aspects. Fourier Volume Rendering (FVR) is an alternative technique that operates on the frequency spectrum of the volume with lower time complexity of order O(N 2 logn) relying on the projection-slice theory. This technique allows the generation of attenuation- only renderings or projections of volumetric data that look like x-ray radio- graphs. It has been used extensively in digital radiography. In this work, a high performance pure GPU-accelerated implementation for the Fourier volume rendering pipeline is proposed to achieve 30X of speed up over a naive implementation by mapping the entire pipeline to be executed on the GPU.

5 DECLARATION I, Marwan Abd Ellah, hereby declare that all information in this document has been obtained and presented in accordance with academic rules and ethical conduct. I also declare that, as required by these rules and conduct, Ihavefullycitedandreferencedallmaterialandresultsthatarenotoriginal to this work. Marwan Abd Ellah Date i

6 ACKNOWLEDGEMENTS This work would not have been possible without the invaluable support, advice and encouragement of my dear supervisors. I am honored to present my special thanks and deepest gratitude to Dr. Ayman El Deib & Dr. Amr Sharawi for their guidance and insightful feedback during the duration of this project. Also, I would like to thank my professor Dr. Yasser Kadah for his outstanding Medical Image Reconstruction course, which has formed the fundamentals of image reconstruction in general and the Fourier volume rendering in particular, and also have fueled me to investigate deeper to end up with this work. As well as, I would like to thank Dr. Stefano Cozzini for accepting me to attend his advanced school in High Performance Computing that was held in the Abdulsalam International Center of Theoretical Physics (ICTP) in Italy. It was really a nice, valuable, and unforgettable experience. ii

7 ABSTRACT The past several years have seen tremendous advances in volume visualization techniques that have been used broadly in medical imaging. In particular, volume rendering has received a considerable attention in this area. However, spatial domain volume rendering has achieved a wide acceptance from scientists and physicians, but this category of rendering techniques was associated with diverse constrains due to their O(N 3 ) time-complexity, which limited their usability in several aspects. Fourier Volume Rendering (FVR) is an alternative technique that operates on the frequency spectrum of the volume with lower time complexity of order O(N 2 logn) relying on the projectionslice theory. This technique allows the generation of attenuation-only renderings or projections of volumetric data that look like x-ray radiographs. It has been used extensively in digital radiography. In this work, a high performance pure GPU-accelerated implementation for the Fourier volume rendering pipeline is proposed to achieve 30X of speed up over a hybrid implementation by mapping the entire pipeline to be executed on the GPU. Keywords: Fourier Volume Rendering, Medical Image Reconstruction, Projection- Slice Theory, GPU Computing, CUDA.

8 PREFACE In this work, an in-depth investigation has been carried out to achieve a high performance implementation of the Fourier volume rendering pipeline on the GPU. It considered in particular CUDA-enabled GPUs to be used as a high performance computing architectures that can leverage the performance of data-parallel algorithm, which completely suits our problem. In advance, in Chapter 1, Introduction, volume visualization techniques that have been used widely in the medical arena are presented. It concentrated mainly on volume rendering as a scientific tool to explore the internal structures of volumetric objects. Then, it focused on Frequency domain volume rendering as an alternative technique to spatial domain algorithms at which it reduces the rendering time-complexity to order of O(N 2 logn). Afterwards, we summarize the previous work in this area and our contribution. Chapter 2, Theory Behind Frequency Domain Volume Rendering, aims at providing a gentle introduction to the theories relevant to frequency domain volume rendering. Sampling theory, Fourier transform, Hartley transform, and projection-slice theory are briefly discussed to set the stages to chapters to come by. Basically, High Performance Computing as we understand deals with the implementations of some algorithm and the hardware it run on, but as a iv

9 research tool, it demands at least a basic understanding of several disciplines, concepts, and methodologies that range from algorithms, computer programming, software and hardware architectures. In Chapter 3, High Performance Computing on Graphics Processing Units, weexplainhowtheevolutionof GPUs has turned them to be high performance platforms relying on their massively parallel architecture. A special treatment for the CUDA architecture is considered. Although we tried to keep this chapter comprehensive and concise, but the temptation to cover everything is overwhelming and the reader is assumed to have some familiarity with programming and high-level computer architecture. In Chapter 4, Algorithm & Implementation, the Fourier volume rendering algorithm is presented and demystified to the reader. This chapter is intended as an attempt to summarize the Fourier volume rendering pipeline. It started with a general description on a level independent of specific architecture and then it moves towards a certain strategy that will be adopted to leverage the performance of the GPU-accelerated implementation. It is the author s persuasion that a good understanding of the implementation aspects of this algorithm will reflect the significance of the achieved results. In Chapter 5, Results, we discuss reconstruction and performance benchmarking results of both the naive implementation and our proposed one that is executed entirely on the GPU. In Chapter 6, Conclusion & Future Work, we wrap up and conclude what have been presented in this sequel followed by some future work that might be undertaken either by us or by future researchers working in the same area. v

10 ACRONYMS 1D One-Dimensional 2D Two-Dimensional 3D Three-Dimensional ALU Arithmetic Logic Unit APIs Application Programming Interfaces BO Buffer Object Cg C for Graphics CPU Central Processing Unit CT Computed Tomography CUDA Computer Unified Device Architecture CUFFT CUDA FFT DFT Discrete Fourier Transform vi

11 DHT Discrete Hartley Transform DRR Digital Reconstructed Radiograph ECC Error-Correcting Code FBO Frame Buffer Object FFT Fast Fourier Transform FFTW FFT in the West FHT Fast Hartley Transform FVR Fourier Volume Rendering GLSL OpenGL Shading Language GPGPU General Purpose Graphics Processing Unit GPU Graphics Processing Unit HPC High Performance Computing MRI Magnetic Resonance Imaging OpenCL Open Computing Library OpenGL Open Graphics Library PBO Pixel Buffer Object SIMT Single Instruction Multiple Thread SM Stream Multiprocessor SP Streaming Processor TP Thread Processor VBO Vertex Buffer Object vii

12 CONTENTS 1 INTRODUCTION Medical Visualization Volume Rendering Frequency Domain Volume Rendering Previous Work Contribution & Thesis Objectives THEORY BEHIND FREQUENCY DOMAIN VOLUME REN- DERING Notation Special Functions Delta Dirac Shah Function Sinc Function Rect Function Sampling Theory Nyquist Shannon Sampling Theorem Aliasing Windowing viii

13 2.4 Fourier Transform Transform Pair Properties of Fourier Transform Multi-Dimensional Fourier Transform D Fourier Transform D Fourier Transform Separability Theorem Convolution Theorem Discrete Fourier Transform Fast Fourier Transform Hartley Transform Definition Discrete Hartley Transform Pros & Cons Projection-Slice Theory Definition Proof HIGH PERFORMANCE COMPUTING ON GRAPHICS PROCESSING UNITS (GPUS) High Performance Computing The Era of GPU Computing GPGPU & GPU Computing GPU Architecture Evolution CPU & GPU In Comparison Heterogeneous Computing Model Compute Unified Device Architecture Understanding CUDA Architecture CUDA Programming Model Threading Hierarchy Memory Model Global Memory Shared Memory Register Memory Local Memory Constant Memory Texture Memory ix

14 3.5.5 Execution Model CUDA Software Programming Environment CUDA Computing Architecture Limitations of CUDA GPU Contexts FFT on GPU ALGORITHM & IMPLEMENTATION Objective & Flow Algorithm Implementation Strategy The Naive Hybrid Approach Analyzing the Naive Algorithm Naive Algorithm Bottlenecks Suppressing Multidimensional Arrays Algorithm Mapping to the GPU CUDA Kernels FVR Pipeline on GPU Mapping Analysis RESULTS Volume Reconstruction Results Benchmarking Results Eliminating Multi-Dimensional Arrays Mapping Computational Context to GPU CONCLUSION & FUTURE WORK Conclusion Future Work BIBLIOGRAPHY 124 x

15 LIST OF FIGURES 1.1 Computer-generated Rendering for a Skull Dataset, reference: Wikipedia Surface Rendering of a Head Dataset, reference: Wikipedia The Process of Volume Rendering a Tooth, reference : GPU Gems High Definition Volume Rendering for a Skull with Volume Ray-Casting, reference: Wikipedia Mouse Skull (CT) Rendering using the Shear Warp Algorithm, reference: Wikipedia Example of Rendering CT Data (Visible Male Dataset) using the Fourier Volume Rendering Algorithm A Projection of the Foot Dataset Reconstructed using the Fourier Volume Rendering Algorithm Continuous Dirac Delta Function δ(t) Kronecker Delta Function δ[n] The Shah Function X(t) The Sinc Function sinc(t) The Rect or Box Function Π(t) xi

16 2.6 Sampling Process - Time Domain is on Left, and Frequency Domain is on Right Aliasing Hamming Window & its Frequency Response Projecting 3D Volume to a 2D X-ray like Image Graphical illustration of the projection-slice theory in twodimensions. f(x, y) and F (k x,k y )are two-dimensional Fourier transform pairs, p(x) is the projection of f(x, y) on the x axis, and s(k x ) is the projection slice of p(x) in the frequency domain High Performance Computing Interdisciplinarity Memory Bandwidth Improvements for CPU & GPU [72] Single & Double Precision Floating-Point Operations Per Second for CPU & GPU [72] The GeForce 7800 architecture with 3 kinds of Programmable Engines (Courtesy of NVIDIA) The G80 GPU with Unified Shader Architecture CPU & GPU Computing Architectures in Comparison, GPU Devotes More Transistors to Data Processing Heterogenous Computing Model with CPU & GPU Problem Decomposition for Serial Parts to be executed on CPU & Parallel Parts to be executed on the GPU Block Diagram for CUDA Stream Multiprocessor (SM) Three-Dimensional Blocks of Two-Dimensional Grids CUDA Memory Model Executing a Kernel Grid on two different GPUs Executing Two Different Kernel Grids on the GPU, (Courtesy of NVIDIA) Thread Index Calculations with 1D Grid & 1D Blocks NVIDIA Compilation Process CUDA Framework Architecture GT200 GPU Architecture Fermi Architecture Block Diagram Fermi SM Architecture CUDA Interoperability with OpenGL Block Diagram for the FVR Algorithm xii

17 4.2 FVR Pipeline FVR Pipeline is Divided into Preprocessing Stage & Rendering Loop. The Rendering Loop is Executed 3 Times to Generate 3 Different Projections for the Same Volume Naive Hybrid Implementation for the FVR Pipeline D Wrapping-Around with 3D Arrays for Real Data D Wrapping-Around with 3D Arrays for Complex Data Repacking the Complex Spectrum from FFTW Array into 1D Array Compatible with OpenGL 3D Texture A Block Diagram Illustrating the Execution of the OpenGL Off-Screen Context D Wrapping-Around Involving 2D Arrays Rendering the Projection Image Eliminating 2D Arrays from the FVR Pipeline Eliminating 3D Arrays from the FVR Pipeline FVR Pipeline on GPU Linking OpenGL Off-Screen Rendering Context with CUDA Context Linking OpenGL Off-Screen Rendering Context with CUDA Context Linking OpenGL CUDA Context with OpenGL On-Screen Context Sagittal View for Visible Male Dataset (256 x 256 x 256) The Central Part of the Visible Male Dataset (128 x 128 x 128) Axial View for Visible Male Dataset (256 x 256 x 256) Sagittal View for the Skull Dataset (256 x 256 x 256) Coronal View for the Skull Dataset (256 x 256 x 256) Foot Dataset (256 x 256 x 256) Engine Dataset (256 x 256 x 256) Bonsai Tree Dataset (256 x 256 x 256) Teapot Dataset (128 x 128 x 128) Hydrogen Atom Dataset (128 x 128 x 128) Nieg Dataset (64 x 64 x 64) Tri-Linear Interpolation Scheme Nearest-Neighbor Interpolation Orthogonal Projection for the Visible Male Dataset xiii

18 5.15 Oblique Projections for the Visible Male Dataset without & with high order reconstruction filter in A & B respectively xiv

19 LIST OF TABLES 3.1 CUDA Memory Types supported by its Memory Model for NVIDIA Quadro FX A Table Summarizing the Features of the Three Main CUDA GPU Architectures Benchmarking for the 3D Wrapping-Around Operation on CPU with 3D & 1D Arrays Benchmarking for the 3D Wrapping-Around Operation on CPU with 3D & 1D Arrays including the Time Consumed During the Replacement of Arrays D Wrapping-Around of Real Data on CPU & GPU D Wrapping-Around Operation of Real Data on CPU & GPU D Wrapping-Around of Complex Data on CPU & GPU D FFT with FFTW & CUFFT Libraries D FFT with FFTW & CUFFT Libraries Comparing Performance for a volume of xv

20 This page is left intentionally blank