Download Monica Borra 2

Parallel Programming
Models
Monica Borra
Outline
 Shared Memory Models – Revision
 Comparison of shared memory models
 Distributed Memory Models
 Parallelism in Big Data Technologies
 Conclusion
Shared Memory Models
 Multi-threaded – Posix Threads(PThreads), TBB, OpenMP
 Multi-processors – Cilk, ArBB, CUDA, Microsoft Parallel Patterns
 Compiler Directives and Library functions(PRAM-like)
Comparative Study I
Most commercially available general purpose computers include hardware
features to increase the parallelism.
hyperthreading, multi-core, ccNUMA architecture
General-purpose threading
CPU vector instructions and GPUs
SIMD
Compared models
 Four Parallel programming models have been selected.
 Each of these models exploits different hardware parallel features
mentioned earlier.
 Also, they require different levels of programming skills
 OPenMP , Intel TBB – parallel threads on multicore systems
 Intel ArBB – threads + multicore SIMD features
 CUDA – SIMD GPU features.
CUDA
•
CUDA (Compute Unified Device Architecture) is a C/C++ programming
model and API (Application Programming Interface) introduced by NVIDIA
to enable software developers to code general purpose apps that run on
the massively parallel hardware on GPUs.
•
GPUs are optimal for data parallel apps aka SIMD (Single Instruction
Multiple Data).
•
Threads running in parallel use extremely fast shared memory for
communication.
Evaluations:
4 benchmarks:
Matrix multiplication, Simple 2D convolution, Histogram computation and
Mandelbrot.
Different underlying computer architectures.
Comparison between OpenMP – TBB and ArBB –
CUDA for simple 2D Convolution
Comparison Summary:
 OpenMP and TBB show a very low performance compared to ArBB and
CUDA.
 TBB seems to have a lower performance than OpenMP for single socket
architectures, the situation seems to reverse when running on ccNUMA
architectures, where TBB shows a significant improvement.
 ArBB and CUDA. But also that ArBB performance tends to be comparable
with CUDA performance in most cases (although it is normally lower).
 Hence, there are evidences that a carefully designed top range multicore
and multisocket architecture( advantage of the TLP and SIMD features) like
ArBB applications may approach the performance of top range CUDA
GPGPU.
Comparative Study II
 OpenMP, Pthread, Microsoft Parallel Patterns APIs
 Computation of matrix multiplication
 Performed on an Intel i5 processor
 Execution time and speed up
Experimental Results:
Distributed Parallel Computing
 Cluster based
 Message Passing Interface(MPI) – de-facto standard
 More advantageous when communication between the nodes is high
 Originally designed for HPC
 Apache Hadoop
 Parallel processing for Big Data
 Implementation of a programming Model, “Map Reduce”
Why is parallelism in
Big Data important?
 Innumerable sources – RFID, Sensors,
Social Networking
 Volume, Velocity and Variety
Apache Hadoop
 Framework that allows for the distributed parallel processing of large data
sets.
 Batch processes raw unstructured data
 Highly reliable and scalable
 Consists of 4 modules: common utilities, storage, resource management
and processing
Parallel
Case Study: Can we take advantage
of MPI to overcome communication
overhead in Big Data Technologies?
 Challenges:
1. Is it worth to speed-up communication?
a. Percentage of time taken for communications alone
b. Comparisons of achievable latency and peak bandwidth for
point to point communications through MPI against Hadoop.
2. How difficult it is to adapt MPI to Hadoop and what are the minimal
extensions to the MPI standard?
A pair of new MPI calls supporting Hadoop data communication specified
via key-value pairs.
Contributions of the case study:
 Abstracting the requirements of the communication model
 Dichotomic, dynamic, data-centric bipartite model.
 Key-Value pair based
 Novel design of DataMPI – High Performance Communication
Library
 Various benchmarks to prove efficiency and ease of use.
Contributions:
Comparision: DataMPI vs Hadoop
 Several big data representative benchmarks
WordCount, Terasort, K-means, Top K, PageRank
 Compared for various parameters
Efficiency, fault tolerance, easy of use
Comparisons for
Terasort
Both Hadoop and DataMPI exhibit
similar trends
DataMPI shows better results in all
cases.
Results:
 Efficiency: DataMPI speeds up varied Big Data workloads and improves job
execution time by 31%-41%.
 Fault Tolerance: DataMPI supports fault tolerance. Evaluations show that
DataMPI-FT can attain 21% improvement over Hadoop.
 Scalability: DataMPI achieves high scalability as Hadoop and 40%
performance improvement.
 Flexibile and the coding complexity of using DataMPI is on par with that of
using traditional Hadoop
Conclusion:
 The efficiency of a model in shared memory parallel computing depends
on the type of the program and best use of underlying hardware parallel
processing features.
 Extending MPI for high computational problems like big data mining is
much more efficient than the traditional frameworks.
 Shared memory models are easy to implement but MPI gives best optimal
results for more complex problems.
References
 L. SanChez, J. Fernandez, R. Sotomayor, J. D. Garcia, “A Comparative
Evaluation of Parallel Programming Models for Shared-Memory
Architectures”, IEEE 10th International Symposium on Parallel and
Distributed Processing with Applications, 2012, pp 363 - 374
 M. Sharma, P. Soni, “Comparative Study of Parallel Programming Models to
Compute Complex Algorithm”, IEEE International Journal of Computer
Applications, 2014, pp 174 - 180
 Apache Hadoop, hadoop.apache.org
 Xiaoyi Lu, Fan Liang, Bing Wang, Li Zha, Zhiwei Xu, “DataMPI: Extending MPI
to Hadoop-like Big Data Computing”, IEEE 28th Internation Parallel and
Distributed Processing Symposium, 2014, pp 829 - 838
 Lorin Hochstein, Victor R. Basili, Uzi Vishkin, John Gilbert, “A pilot study to
compare programming effort for two parallel programming models”, The
Journal of Systems and Software, 2008
Questions?
Thank you!!