Downloads & Free Reading Options - Results

We present a simple way to add an arbitrary equation of state to a automaton gas modelled in the lattice Boltzmann limit. As a way of interpreting the lattice Boltzmann equation we present a new derivation based on an automaton Hamiltonian and the Liouville equation. A convective-gradient term added to the LBE is shown to be a sufficient route for modeling hydrodynamic flow with a general equation of state. The method generalizes to multi-speed gases in three-dimensions.

The supervised relaxation operator combines the information from multiple ancillary data sources with the information from multispectral remote sensing image data and spatial context. Iterative calculation integrate information from the various sources, reaching a balance in consistency between these sources of information. The supervised relaxation operator is shown to produce substantial improvements in classification accuracy compared to the accuracy produced by the conventional maximum likelihood classifier using spectral data only. The convergence property of the supervised relaxation algorithm is also described. Improvement in classification accuracy by means of supervised relaxation comes at a high price in terms of computation. In order to overcome the computation-intensive problem, a distributed/parallel implementation is adopted to take advantage of a high degree of inherent parallelism in the algorithm.

Over the past decade there have been significant advances in bringing parallel computing and new database management systems to a wider audience. Through a number of efforts such as the National Strategic Computing Initiative (NSCI), there has been a push to merge these Big Data and Scientific Computing communities to a single computational platform. At the Massachusetts Institute of Technology, Lincoln Laboratory, we have been developing HPC and database technologies to address a number of scientific problems including biomedical processing. In this article, we briefly describe these technologies and how we have used them in the past. We are interested in learning more about the needs of clinical pathologists as we continue to develop these technologies.

Over the past decade there have been significant advances in bringing parallel computing and new database management systems to a wider audience. Through a number of efforts such as the National Strategic Computing Initiative (NSCI), there has been a push to merge these Big Data and Scientific Computing communities to a single computational platform. At the Massachusetts Institute of Technology, Lincoln Laboratory, we have been developing HPC and database technologies to address a number of scientific problems including biomedical processing. In this article, we briefly describe these technologies and how we have used them in the past. We are interested in learning more about the needs of clinical pathologists as we continue to develop these technologies.

Over the past decade there have been significant advances in bringing parallel computing and new database management systems to a wider audience. Through a number of efforts such as the National Strategic Computing Initiative (NSCI), there has been a push to merge these Big Data and Scientific Computing communities to a single computational platform. At the Massachusetts Institute of Technology, Lincoln Laboratory, we have been developing HPC and database technologies to address a number of scientific problems including biomedical processing. In this article, we briefly describe these technologies and how we have used them in the past. We are interested in learning more about the needs of clinical pathologists as we continue to develop these technologies.

Over the past decade there have been significant advances in bringing parallel computing and new database management systems to a wider audience. Through a number of efforts such as the National Strategic Computing Initiative (NSCI), there has been a push to merge these Big Data and Scientific Computing communities to a single computational platform. At the Massachusetts Institute of Technology, Lincoln Laboratory, we have been developing HPC and database technologies to address a number of scientific problems including biomedical processing. In this article, we briefly describe these technologies and how we have used them in the past. We are interested in learning more about the needs of clinical pathologists as we continue to develop these technologies.

Over the past decade there have been significant advances in bringing parallel computing and new database management systems to a wider audience. Through a number of efforts such as the National Strategic Computing Initiative (NSCI), there has been a push to merge these Big Data and Scientific Computing communities to a single computational platform. At the Massachusetts Institute of Technology, Lincoln Laboratory, we have been developing HPC and database technologies to address a number of scientific problems including biomedical processing. In this article, we briefly describe these technologies and how we have used them in the past. We are interested in learning more about the needs of clinical pathologists as we continue to develop these technologies.

The matrix element method utilizes ab initio calculations of probability densities as powerful discriminants for processes of interest in experimental particle physics. The method has already been used successfully at previous and current collider experiments. However, the computational complexity of this method for final states with many particles and degrees of freedom sets it at a disadvantage compared to supervised classification methods such as decision trees, k nearest-neighbour, or neural networks. This note presents a concrete implementation of the matrix element technique using graphics processing units. Due to the intrinsic parallelizability of multidimensional integration, dramatic speedups can be readily achieved, which makes the matrix element technique viable for general usage at collider experiments.

The matrix element method utilizes ab initio calculations of probability densities as powerful discriminants for processes of interest in experimental particle physics. The method has already been used successfully at previous and current collider experiments. However, the computational complexity of this method for final states with many particles and degrees of freedom sets it at a disadvantage compared to supervised classification methods such as decision trees, k nearest-neighbour, or neural networks. This note presents a concrete implementation of the matrix element technique using graphics processing units. Due to the intrinsic parallelizability of multidimensional integration, dramatic speedups can be readily achieved, which makes the matrix element technique viable for general usage at collider experiments.

The matrix element method utilizes ab initio calculations of probability densities as powerful discriminants for processes of interest in experimental particle physics. The method has already been used successfully at previous and current collider experiments. However, the computational complexity of this method for final states with many particles and degrees of freedom sets it at a disadvantage compared to supervised classification methods such as decision trees, k nearest-neighbour, or neural networks. This note presents a concrete implementation of the matrix element technique using graphics processing units. Due to the intrinsic parallelizability of multidimensional integration, dramatic speedups can be readily achieved, which makes the matrix element technique viable for general usage at collider experiments.

The idea of upgrading performance and utility of computer systems by incorporating parallel processing has been around since at least the 1940s. A significant investment in parallel processing in the 1980s and 1990s has led to an abundance of parallel architectures that due to technical constraints at the time had to rely on multi-chip multi-processing. Unfortunately, their impact on mainstream computing was quite limited. 'Tribal lore' suggests the following reason: while programming for parallelism tends to be easy, turning this parallelism into the 'coarse-grained' type, needed to optimize performance for multi-chip multi-processing (with their high coordination overhead), has been quite hard. The mainstream computing paradigm has always relied on serial code. However, commercially successful processors are entering a second decade of near stagnation in the maximum number of instructions they can issue towards a single computational task in one clock. Alas, the multi-decade reliance on advancing the clock is also coming to an end. The PRAM-On-Chip research project at UMD is reviewed. The project is guided by a fact, an old insight, a recent one and a premise. The fact: Billion transistor chips are here, up from less than 30,000 circa 1980. The older insight: Using a very simple parallel computation model, the parallel random access model (PRAM), the parallel algorithms research community succeeded in developing a general-purpose theory of parallel algorithms that was much richer than any competing approach and is second in magnitude only to serial algorithmics. However, since it did not offer an effective abstraction for multi-chip multi-processors this elegant algorithmic theory remained in the ivory towers of theorists. The PRAM-On-Chip insight: The Billion transistor chip era allows for the first time low-overhead on-chip multi-processors so that the PRAM abstraction becomes effective. The premise: Were the architecture component of PRAM-On-Chip feasible in the 1990s, its parallel programming component would become the mainstream standard. In 1988-90 standard algorithms textbooks chose to include significant PRAM chapters (some still have them). Arguably nothing could stand in the way of teaching them to every student at every computer science program. Programming for concurrency/parallelism is quickly becoming an integral part of mainstream computing. Yet, industry and academia leaders in system software and general-purpose application software have maintained a passive posture: their attention tends to focus too much on getting the best performance out of architectures that originated from a hardware centric approach, or very specific applications, and too little on trying to impact the evolving generation of multi-core and/or multi-threaded general-purpose architectures. We argue, perhaps provocatively, that: (i) limiting programming for parallelism to fit hardware-centric architectures imports the epidemic of programmability problems that has plagued parallel computing into mainstream computing, (ii) it is only a matter of time until the industry will seek convergence based on parallel programmability: the difference in the bottom line between a successful and a less successful strategy on parallel programmability will be too big to ignore, (iii) a more assertive position by such leaders is necessary, and (iv) the PRAM-On-Chip approach offers a more balanced way that avoids these problems. URL: http://www.umiacs.umd.edu/~vishkin/XMT ©2005 Microsoft Corporation. All rights reserved.

The research of this project concerned the investigation of the suitability of the SOR iteration as a preconditioner for the GMRES method for solving large sparse nonsymmetric systems of linear equations. Preliminary results on a serial computer (RS 6000) showed that SOR was indeed a good preconditioner, at least for the convection-diffusion equations used in this study. This was in contradiction to various statements that had appeared in the literature, questioning the suitability of SOR as a preconditioner. These experiments using a serial computer were described more completely in previous semi-annual reports as well as in. The second phase of this project was to develop parallel codes for the Intel Paragon at NASA-Langley and the Center for Advanced Computing Research at Caltech, and the IBM SP-2 at NASA-Langley and NASA-Ames. Highly parallel codes were developed. Indeed, an unexpected result was the superlinear speedup in some cases due to better cache utilization as the problem sizes and number of processors increased. Preliminary results on these parallel experiments were given.

The research of this project concerned the investigation of the suitability of the SOR iteration as a preconditioner for the GMRES method for solving large sparse nonsymmetric systems of linear equations. Preliminary results on a serial computer (RS 6000) showed that SOR was indeed a good preconditioner, at least for the convection-diffusion equations used in this study. This was in contradiction to various statements that had appeared in the literature, questioning the suitability of SOR as a preconditioner. These experiments using a serial computer were described more completely in previous semi-annual reports as well as in. The second phase of this project was to develop parallel codes for the Intel Paragon at NASA-Langley and the Center for Advanced Computing Research at Caltech, and the IBM SP-2 at NASA-Langley and NASA-Ames. Highly parallel codes were developed. Indeed, an unexpected result was the superlinear speedup in some cases due to better cache utilization as the problem sizes and number of processors increased. Preliminary results on these parallel experiments were given.

The research of this project concerned the investigation of the suitability of the SOR iteration as a preconditioner for the GMRES method for solving large sparse nonsymmetric systems of linear equations. Preliminary results on a serial computer (RS 6000) showed that SOR was indeed a good preconditioner, at least for the convection-diffusion equations used in this study. This was in contradiction to various statements that had appeared in the literature, questioning the suitability of SOR as a preconditioner. These experiments using a serial computer were described more completely in previous semi-annual reports as well as in. The second phase of this project was to develop parallel codes for the Intel Paragon at NASA-Langley and the Center for Advanced Computing Research at Caltech, and the IBM SP-2 at NASA-Langley and NASA-Ames. Highly parallel codes were developed. Indeed, an unexpected result was the superlinear speedup in some cases due to better cache utilization as the problem sizes and number of processors increased. Preliminary results on these parallel experiments were given.

The research of this project concerned the investigation of the suitability of the SOR iteration as a preconditioner for the GMRES method for solving large sparse nonsymmetric systems of linear equations. Preliminary results on a serial computer (RS 6000) showed that SOR was indeed a good preconditioner, at least for the convection-diffusion equations used in this study. This was in contradiction to various statements that had appeared in the literature, questioning the suitability of SOR as a preconditioner. These experiments using a serial computer were described more completely in previous semi-annual reports as well as in. The second phase of this project was to develop parallel codes for the Intel Paragon at NASA-Langley and the Center for Advanced Computing Research at Caltech, and the IBM SP-2 at NASA-Langley and NASA-Ames. Highly parallel codes were developed. Indeed, an unexpected result was the superlinear speedup in some cases due to better cache utilization as the problem sizes and number of processors increased. Preliminary results on these parallel experiments were given.

The research of this project concerned the investigation of the suitability of the SOR iteration as a preconditioner for the GMRES method for solving large sparse nonsymmetric systems of linear equations. Preliminary results on a serial computer (RS 6000) showed that SOR was indeed a good preconditioner, at least for the convection-diffusion equations used in this study. This was in contradiction to various statements that had appeared in the literature, questioning the suitability of SOR as a preconditioner. These experiments using a serial computer were described more completely in previous semi-annual reports as well as in. The second phase of this project was to develop parallel codes for the Intel Paragon at NASA-Langley and the Center for Advanced Computing Research at Caltech, and the IBM SP-2 at NASA-Langley and NASA-Ames. Highly parallel codes were developed. Indeed, an unexpected result was the superlinear speedup in some cases due to better cache utilization as the problem sizes and number of processors increased. Preliminary results on these parallel experiments were given.

The research of this project concerned the investigation of the suitability of the SOR iteration as a preconditioner for the GMRES method for solving large sparse nonsymmetric systems of linear equations. Preliminary results on a serial computer (RS 6000) showed that SOR was indeed a good preconditioner, at least for the convection-diffusion equations used in this study. This was in contradiction to various statements that had appeared in the literature, questioning the suitability of SOR as a preconditioner. These experiments using a serial computer were described more completely in previous semi-annual reports as well as in. The second phase of this project was to develop parallel codes for the Intel Paragon at NASA-Langley and the Center for Advanced Computing Research at Caltech, and the IBM SP-2 at NASA-Langley and NASA-Ames. Highly parallel codes were developed. Indeed, an unexpected result was the superlinear speedup in some cases due to better cache utilization as the problem sizes and number of processors increased. Preliminary results on these parallel experiments were given.

Wide Area Information Servers Project Documentation, Scanned and uploaded in 2013.

Wide Area Information Servers Project Documentation, Scanned and uploaded in 2013.

Computation of optimal cycle mean in a directed weighted graph has many applications in program analysis, performance verification in particular. In this paper we propose a data-parallel algorithmic solution to the problem and show how the computation of optimal cycle mean can be efficiently accelerated by means of CUDA technology. We show how the problem of computation of optimal cycle mean is decomposed into a sequence of data-parallel graph computation primitives and show how these primitives can be implemented and optimized for CUDA computation. Finally, we report a fivefold experimental speed up on graphs representing models of distributed systems when compared to best sequential algorithms.

Parallel Computing 1990: Volume 14 , Issue Index. Digitized from IA1652917-03 . Previous issue: sim_parallel-computing_1990-03_13_3 . Next issue: sim_parallel-computing_1990-05_14_1 .

Establishing a test-bed for clustered parallel computing is reported along with performance evaluation of various clusters for a number of applications/parallel algorithms.

Establishing a test-bed for clustered parallel computing is reported along with performance evaluation of various clusters for a number of applications/parallel algorithms.

Establishing a test-bed for clustered parallel computing is reported along with performance evaluation of various clusters for a number of applications/parallel algorithms.

Establishing a test-bed for clustered parallel computing is reported along with performance evaluation of various clusters for a number of applications/parallel algorithms.

Establishing a test-bed for clustered parallel computing is reported along with performance evaluation of various clusters for a number of applications/parallel algorithms.

Establishing a test-bed for clustered parallel computing is reported along with performance evaluation of various clusters for a number of applications/parallel algorithms.

First Class

First Class

First Class

polski test turinga gnu marek nowicki

polski test turinga gnu marek nowicki

polski test turinga gnu marek nowicki

Cluster-based data-parallel frameworks such as MapReduce, Hadoop, and Dryad are increasingly popular for a large class of compute-intensive tasks. Although such systems are designed for large-scale clusters, they also offer a convenient and accessible route to data-parallel programming for small-scale clusters. This potentially allows applications traditionally targeted at supercomputers or remote server farms, such as sophisticated video processing, to be deployed in a small-scale ad-hoc fashion by aggregating the servers and workstations in the home or office network. The default scheduling algorithms of these frameworks perform well at scale, but are significantly less optimal in a small (3-10 machine) cluster environment where nodes have widely differing performance characteristics. To make effective use of an ad-hoc cluster, we require a 'planner' rather than a scheduler that takes account of the predicted resource consumption by each vertex in the dataflow graph and the heterogeneity of the available hardware. In this talk I will describe our enhancements to DryadLINQ and Dryad for ad-hoc clusters. We have integrated a constraint-based planner that maps the dataflow graph generated by the DryadLINQ compiler onto the cluster. The planner makes use of DryadLINQ operator performance models that are constructed from low-level traces of vertex executions. The performance models abstract the behaviour of each vertex in sufficient detail to predict the bottleneck resource, which can change during vertex execution, on different hardware and with different sizes of input. Experimental evaluation shows reasonable predictive accuracy and good performance gains for parallel jobs on ad-hoc clusters. ©2009 Microsoft Corporation. All rights reserved.

The Microsoft .NET Framework 4 and Visual Studio 2010 include new technologies for expressing, debugging, and tuning parallelism in managed applications. Dive into key areas of support, including Parallel Language Integrated Query (PLINQ), cutting-edge concurrency views in the Visual Studio profiler, and debugger tool windows for analyzing the state of concurrent code. In addition to exploring such features, we will examine some common parallel patterns prevalent in technical computing and how these features can be used to best implement such patterns. ©2010 Microsoft Corporation. All rights reserved.

viii, 203 p. : 24 cm

Cluster-based data-parallel frameworks such as MapReduce, Hadoop, and Dryad are increasingly popular for a large class of compute-intensive tasks. Although such systems are designed for large-scale clusters, they also offer a convenient and accessible route to data-parallel programming for small-scale clusters. This potentially allows applications traditionally targeted at supercomputers or remote server farms, such as sophisticated video processing, to be deployed in a small-scale ad-hoc fashion by aggregating the servers and workstations in the home or office network. The default scheduling algorithms of these frameworks perform well at scale, but are significantly less optimal in a small (3-10 machine) cluster environment where nodes have widely differing performance characteristics. To make effective use of an ad-hoc cluster, we require a 'planner' rather than a scheduler that takes account of the predicted resource consumption by each vertex in the dataflow graph and the heterogeneity of the available hardware. In this talk I will describe our enhancements to DryadLINQ and Dryad for ad-hoc clusters. We have integrated a constraint-based planner that maps the dataflow graph generated by the DryadLINQ compiler onto the cluster. The planner makes use of DryadLINQ operator performance models that are constructed from low-level traces of vertex executions. The performance models abstract the behaviour of each vertex in sufficient detail to predict the bottleneck resource, which can change during vertex execution, on different hardware and with different sizes of input. Experimental evaluation shows reasonable predictive accuracy and good performance gains for parallel jobs on ad-hoc clusters. ©2009 Microsoft Corporation. All rights reserved.

Scalable Parallel Computing on Many/Multicore Systems This set of lectures will review the application and programming model issues that will one must address when one gets chips with 32-1024 cores and “scalable” approaches will be needed to make good use of such systems. We will not discuss bit-level and instruction-level parallelism i.e. what happens on a possibly special purpose core, even though this is clearly important. We will use science and engineering applications to drive the discussion as we have substantial experience in these cases but we are interested in the lessons for commodity client and server applications that will be broadly used 5-10 years from now. We start with simple applications and algorithms from a variety of fields and identify features that makes it “obvious” that “all” science and engineering run well and scalably in parallel. We explain why unfortunately it is equally obvious that there is no straightforward way of expressing this parallelism. Parallel hardware architectures are described in enough detail to understand performance and algorithm issues and need for cross-architecture compatibility; however in these lectures we will just be users of hardware. We must understand what features of multicore chips we can and should exploit. We can explicitly use the shared memory between cores or just enjoy its implications for very fast inter-core control and communication linkage. We note that parallel algorithm research is hugely successful although this success has reduced activity in an area that deserves new attention for the next generation of architectures. The parallel software environment is discussed at several levels including programming paradigm, runtime and operating system. The importance of libraries, templates, kernels (dwarfs) and benchmarks is stressed. The programming environment has various tools including compilers with possible parallelism hints like OpenMP; tuners like Atlas; messaging models; parallelism and distribution support as in HPF, HPCS Languages, co-array Fortran, Global arrays and UPC. We also discuss the relevance of important general ideas like object-oriented paradigms (as in for example Charm++), functional languages and Software Transactional Memories. Streaming, pipelining, co-ordination, services and workflow are placed in context. Examples discussed in the last category include CCR/DSS from Microsoft and the Common Component Architecture CCA from DoE. Domain Specific environments like Matlab and Mathematica are important as there is no universal silver programming bullet; one will need interoperable focused environments. We discuss performance analysis including speedup, efficiency, scaled speedup and Amdahl's law. We show how to relate performance to algorithm/application structure and the hardware characteristics. Applications will not get scalable parallelism accidentally but only if there is an understandable performance model. We review some of the many pitfalls for both performance and correctness; these include deadlocks, race conditions, nodes that are busy doing something else and the difficulty of second-guessing automatic parallelization methods. We describe the formal sources of overhead; load imbalance and the communication/control overhead and the ways to reduce them. We relate the blocking used in caching to that used in parallel computation. We note that load balancing was the one part of parallel computing that was easier than expected. We will mix both simple idealized applications with “real problems” noting that usually it is simple problems that are the hardest as they have poor computation to control/communication ratios. We will explain the parallelism in several application classes including for science and engineering: Finite Difference, Finite Elements, FFT/Spectral, Meshes of all sorts, Particle Dynamics, Particle-Mesh, and Monte Carlo methods. Some applications like Image Processing, Graphics, Cryptography and Media coding/decoding have features similar to well understood science and engineering problems. We emphasize that nearly all applications are built hierarchically from more “basic applications” with a variety of different structures and natural programming models. Such application composition (co-ordination, workflow) must be supported with a common run-time. We contrast the difference between decomposition needed in most “basic parallel applications” to the composition supported in workflow and Web 2.0 mashups. Looking at broader application classes we should cover; Internet applications and services; artificial intelligence, optimization, machine learning; divide and conquer algorithms; tree structured searches like computer chess; applications generated by the data deluge including access, search, and the assimilation of data into simulations. There has been a growing interest in Complex Systems whether it be for critical infrastructure like energy and transportation, the Internet itself, commodity gaming, computational epidemiology or the original war games. We expect Discrete Event Simulations (such as DoD’s High Level Architecture HLA) to grow in importance as they naturally describe complex systems and because they can clearly benefit from multicore architectures. We will discuss the innovative Sequential Dynamical Systems approach used in the EpiSims, TransSims and other critical infrastructure simulation environments. In all applications we need to identify the intrinsic parallelism and the degrees of freedom that can be parallelized and distinguish small parallelism (local to core) compared to large parallelism (scalable across many cores) We will not forget many critical non-technical Issues including “Who programs – everybody or just the marine corps?”; “The market for science and engineering was small but it will be large for general multicore”; “The program exists and can’t be changed”; “What features will next hardware/software release support and how should I protect myself from change?” We will summarize lessons and relate them to application and programming model categories. In last lecture or at end of all lectures we encourage the audience to bring their own multicore application or programming model so we can discuss examples that interest you. ©2007 Microsoft Corporation. All rights reserved.

Scalable Parallel Computing on Many/Multicore Systems This set of lectures will review the application and programming model issues that will one must address when one gets chips with 32-1024 cores and “scalable” approaches will be needed to make good use of such systems. We will not discuss bit-level and instruction-level parallelism i.e. what happens on a possibly special purpose core, even though this is clearly important. We will use science and engineering applications to drive the discussion as we have substantial experience in these cases but we are interested in the lessons for commodity client and server applications that will be broadly used 5-10 years from now. We start with simple applications and algorithms from a variety of fields and identify features that makes it “obvious” that “all” science and engineering run well and scalably in parallel. We explain why unfortunately it is equally obvious that there is no straightforward way of expressing this parallelism. Parallel hardware architectures are described in enough detail to understand performance and algorithm issues and need for cross-architecture compatibility; however in these lectures we will just be users of hardware. We must understand what features of multicore chips we can and should exploit. We can explicitly use the shared memory between cores or just enjoy its implications for very fast inter-core control and communication linkage. We note that parallel algorithm research is hugely successful although this success has reduced activity in an area that deserves new attention for the next generation of architectures. The parallel software environment is discussed at several levels including programming paradigm, runtime and operating system. The importance of libraries, templates, kernels (dwarfs) and benchmarks is stressed. The programming environment has various tools including compilers with possible parallelism hints like OpenMP; tuners like Atlas; messaging models; parallelism and distribution support as in HPF, HPCS Languages, co-array Fortran, Global arrays and UPC. We also discuss the relevance of important general ideas like object-oriented paradigms (as in for example Charm++), functional languages and Software Transactional Memories. Streaming, pipelining, co-ordination, services and workflow are placed in context. Examples discussed in the last category include CCR/DSS from Microsoft and the Common Component Architecture CCA from DoE. Domain Specific environments like Matlab and Mathematica are important as there is no universal silver programming bullet; one will need interoperable focused environments. We discuss performance analysis including speedup, efficiency, scaled speedup and Amdahl's law. We show how to relate performance to algorithm/application structure and the hardware characteristics. Applications will not get scalable parallelism accidentally but only if there is an understandable performance model. We review some of the many pitfalls for both performance and correctness; these include deadlocks, race conditions, nodes that are busy doing something else and the difficulty of second-guessing automatic parallelization methods. We describe the formal sources of overhead; load imbalance and the communication/control overhead and the ways to reduce them. We relate the blocking used in caching to that used in parallel computation. We note that load balancing was the one part of parallel computing that was easier than expected. We will mix both simple idealized applications with “real problems” noting that usually it is simple problems that are the hardest as they have poor computation to control/communication ratios. We will explain the parallelism in several application classes including for science and engineering: Finite Difference, Finite Elements, FFT/Spectral, Meshes of all sorts, Particle Dynamics, Particle-Mesh, and Monte Carlo methods. Some applications like Image Processing, Graphics, Cryptography and Media coding/decoding have features similar to well understood science and engineering problems. We emphasize that nearly all applications are built hierarchically from more “basic applications” with a variety of different structures and natural programming models. Such application composition (co-ordination, workflow) must be supported with a common run-time. We contrast the difference between decomposition needed in most “basic parallel applications” to the composition supported in workflow and Web 2.0 mashups. Looking at broader application classes we should cover; Internet applications and services; artificial intelligence, optimization, machine learning; divide and conquer algorithms; tree structured searches like computer chess; applications generated by the data deluge including access, search, and the assimilation of data into simulations. There has been a growing interest in Complex Systems whether it be for critical infrastructure like energy and transportation, the Internet itself, commodity gaming, computational epidemiology or the original war games. We expect Discrete Event Simulations (such as DoD’s High Level Architecture HLA) to grow in importance as they naturally describe complex systems and because they can clearly benefit from multicore architectures. We will discuss the innovative Sequential Dynamical Systems approach used in the EpiSims, TransSims and other critical infrastructure simulation environments. In all applications we need to identify the intrinsic parallelism and the degrees of freedom that can be parallelized and distinguish small parallelism (local to core) compared to large parallelism (scalable across many cores) We will not forget many critical non-technical Issues including “Who programs – everybody or just the marine corps?”; “The market for science and engineering was small but it will be large for general multicore”; “The program exists and can’t be changed”; “What features will next hardware/software release support and how should I protect myself from change?” We will summarize lessons and relate them to application and programming model categories. In last lecture or at end of all lectures we encourage the audience to bring their own multicore application or programming model so we can discuss examples that interest you. ©2007 Microsoft Corporation. All rights reserved.

Scalable Parallel Computing on Many/Multicore Systems This set of lectures will review the application and programming model issues that will one must address when one gets chips with 32-1024 cores and “scalable” approaches will be needed to make good use of such systems. We will not discuss bit-level and instruction-level parallelism i.e. what happens on a possibly special purpose core, even though this is clearly important. We will use science and engineering applications to drive the discussion as we have substantial experience in these cases but we are interested in the lessons for commodity client and server applications that will be broadly used 5-10 years from now. We start with simple applications and algorithms from a variety of fields and identify features that makes it “obvious” that “all” science and engineering run well and scalably in parallel. We explain why unfortunately it is equally obvious that there is no straightforward way of expressing this parallelism. Parallel hardware architectures are described in enough detail to understand performance and algorithm issues and need for cross-architecture compatibility; however in these lectures we will just be users of hardware. We must understand what features of multicore chips we can and should exploit. We can explicitly use the shared memory between cores or just enjoy its implications for very fast inter-core control and communication linkage. We note that parallel algorithm research is hugely successful although this success has reduced activity in an area that deserves new attention for the next generation of architectures. The parallel software environment is discussed at several levels including programming paradigm, runtime and operating system. The importance of libraries, templates, kernels (dwarfs) and benchmarks is stressed. The programming environment has various tools including compilers with possible parallelism hints like OpenMP; tuners like Atlas; messaging models; parallelism and distribution support as in HPF, HPCS Languages, co-array Fortran, Global arrays and UPC. We also discuss the relevance of important general ideas like object-oriented paradigms (as in for example Charm++), functional languages and Software Transactional Memories. Streaming, pipelining, co-ordination, services and workflow are placed in context. Examples discussed in the last category include CCR/DSS from Microsoft and the Common Component Architecture CCA from DoE. Domain Specific environments like Matlab and Mathematica are important as there is no universal silver programming bullet; one will need interoperable focused environments. We discuss performance analysis including speedup, efficiency, scaled speedup and Amdahl's law. We show how to relate performance to algorithm/application structure and the hardware characteristics. Applications will not get scalable parallelism accidentally but only if there is an understandable performance model. We review some of the many pitfalls for both performance and correctness; these include deadlocks, race conditions, nodes that are busy doing something else and the difficulty of second-guessing automatic parallelization methods. We describe the formal sources of overhead; load imbalance and the communication/control overhead and the ways to reduce them. We relate the blocking used in caching to that used in parallel computation. We note that load balancing was the one part of parallel computing that was easier than expected. We will mix both simple idealized applications with “real problems” noting that usually it is simple problems that are the hardest as they have poor computation to control/communication ratios. We will explain the parallelism in several application classes including for science and engineering: Finite Difference, Finite Elements, FFT/Spectral, Meshes of all sorts, Particle Dynamics, Particle-Mesh, and Monte Carlo methods. Some applications like Image Processing, Graphics, Cryptography and Media coding/decoding have features similar to well understood science and engineering problems. We emphasize that nearly all applications are built hierarchically from more “basic applications” with a variety of different structures and natural programming models. Such application composition (co-ordination, workflow) must be supported with a common run-time. We contrast the difference between decomposition needed in most “basic parallel applications” to the composition supported in workflow and Web 2.0 mashups. Looking at broader application classes we should cover; Internet applications and services; artificial intelligence, optimization, machine learning; divide and conquer algorithms; tree structured searches like computer chess; applications generated by the data deluge including access, search, and the assimilation of data into simulations. There has been a growing interest in Complex Systems whether it be for critical infrastructure like energy and transportation, the Internet itself, commodity gaming, computational epidemiology or the original war games. We expect Discrete Event Simulations (such as DoD’s High Level Architecture HLA) to grow in importance as they naturally describe complex systems and because they can clearly benefit from multicore architectures. We will discuss the innovative Sequential Dynamical Systems approach used in the EpiSims, TransSims and other critical infrastructure simulation environments. In all applications we need to identify the intrinsic parallelism and the degrees of freedom that can be parallelized and distinguish small parallelism (local to core) compared to large parallelism (scalable across many cores) We will not forget many critical non-technical Issues including “Who programs – everybody or just the marine corps?”; “The market for science and engineering was small but it will be large for general multicore”; “The program exists and can’t be changed”; “What features will next hardware/software release support and how should I protect myself from change?” We will summarize lessons and relate them to application and programming model categories. In last lecture or at end of all lectures we encourage the audience to bring their own multicore application or programming model so we can discuss examples that interest you. ©2007 Microsoft Corporation. All rights reserved.

Scalable Parallel Computing on Many/Multicore Systems This set of lectures will review the application and programming model issues that will one must address when one gets chips with 32-1024 cores and “scalable” approaches will be needed to make good use of such systems. We will not discuss bit-level and instruction-level parallelism i.e. what happens on a possibly special purpose core, even though this is clearly important. We will use science and engineering applications to drive the discussion as we have substantial experience in these cases but we are interested in the lessons for commodity client and server applications that will be broadly used 5-10 years from now. We start with simple applications and algorithms from a variety of fields and identify features that makes it “obvious” that “all” science and engineering run well and scalably in parallel. We explain why unfortunately it is equally obvious that there is no straightforward way of expressing this parallelism. Parallel hardware architectures are described in enough detail to understand performance and algorithm issues and need for cross-architecture compatibility; however in these lectures we will just be users of hardware. We must understand what features of multicore chips we can and should exploit. We can explicitly use the shared memory between cores or just enjoy its implications for very fast inter-core control and communication linkage. We note that parallel algorithm research is hugely successful although this success has reduced activity in an area that deserves new attention for the next generation of architectures. The parallel software environment is discussed at several levels including programming paradigm, runtime and operating system. The importance of libraries, templates, kernels (dwarfs) and benchmarks is stressed. The programming environment has various tools including compilers with possible parallelism hints like OpenMP; tuners like Atlas; messaging models; parallelism and distribution support as in HPF, HPCS Languages, co-array Fortran, Global arrays and UPC. We also discuss the relevance of important general ideas like object-oriented paradigms (as in for example Charm++), functional languages and Software Transactional Memories. Streaming, pipelining, co-ordination, services and workflow are placed in context. Examples discussed in the last category include CCR/DSS from Microsoft and the Common Component Architecture CCA from DoE. Domain Specific environments like Matlab and Mathematica are important as there is no universal silver programming bullet; one will need interoperable focused environments. We discuss performance analysis including speedup, efficiency, scaled speedup and Amdahl's law. We show how to relate performance to algorithm/application structure and the hardware characteristics. Applications will not get scalable parallelism accidentally but only if there is an understandable performance model. We review some of the many pitfalls for both performance and correctness; these include deadlocks, race conditions, nodes that are busy doing something else and the difficulty of second-guessing automatic parallelization methods. We describe the formal sources of overhead; load imbalance and the communication/control overhead and the ways to reduce them. We relate the blocking used in caching to that used in parallel computation. We note that load balancing was the one part of parallel computing that was easier than expected. We will mix both simple idealized applications with “real problems” noting that usually it is simple problems that are the hardest as they have poor computation to control/communication ratios. We will explain the parallelism in several application classes including for science and engineering: Finite Difference, Finite Elements, FFT/Spectral, Meshes of all sorts, Particle Dynamics, Particle-Mesh, and Monte Carlo methods. Some applications like Image Processing, Graphics, Cryptography and Media coding/decoding have features similar to well understood science and engineering problems. We emphasize that nearly all applications are built hierarchically from more “basic applications” with a variety of different structures and natural programming models. Such application composition (co-ordination, workflow) must be supported with a common run-time. We contrast the difference between decomposition needed in most “basic parallel applications” to the composition supported in workflow and Web 2.0 mashups. Looking at broader application classes we should cover; Internet applications and services; artificial intelligence, optimization, machine learning; divide and conquer algorithms; tree structured searches like computer chess; applications generated by the data deluge including access, search, and the assimilation of data into simulations. There has been a growing interest in Complex Systems whether it be for critical infrastructure like energy and transportation, the Internet itself, commodity gaming, computational epidemiology or the original war games. We expect Discrete Event Simulations (such as DoD’s High Level Architecture HLA) to grow in importance as they naturally describe complex systems and because they can clearly benefit from multicore architectures. We will discuss the innovative Sequential Dynamical Systems approach used in the EpiSims, TransSims and other critical infrastructure simulation environments. In all applications we need to identify the intrinsic parallelism and the degrees of freedom that can be parallelized and distinguish small parallelism (local to core) compared to large parallelism (scalable across many cores) We will not forget many critical non-technical Issues including “Who programs – everybody or just the marine corps?”; “The market for science and engineering was small but it will be large for general multicore”; “The program exists and can’t be changed”; “What features will next hardware/software release support and how should I protect myself from change?” We will summarize lessons and relate them to application and programming model categories. In last lecture or at end of all lectures we encourage the audience to bring their own multicore application or programming model so we can discuss examples that interest you. ©2007 Microsoft Corporation. All rights reserved.

Scalable Parallel Computing on Many/Multicore Systems This set of lectures will review the application and programming model issues that will one must address when one gets chips with 32-1024 cores and “scalable” approaches will be needed to make good use of such systems. We will not discuss bit-level and instruction-level parallelism i.e. what happens on a possibly special purpose core, even though this is clearly important. We will use science and engineering applications to drive the discussion as we have substantial experience in these cases but we are interested in the lessons for commodity client and server applications that will be broadly used 5-10 years from now. We start with simple applications and algorithms from a variety of fields and identify features that makes it “obvious” that “all” science and engineering run well and scalably in parallel. We explain why unfortunately it is equally obvious that there is no straightforward way of expressing this parallelism. Parallel hardware architectures are described in enough detail to understand performance and algorithm issues and need for cross-architecture compatibility; however in these lectures we will just be users of hardware. We must understand what features of multicore chips we can and should exploit. We can explicitly use the shared memory between cores or just enjoy its implications for very fast inter-core control and communication linkage. We note that parallel algorithm research is hugely successful although this success has reduced activity in an area that deserves new attention for the next generation of architectures. The parallel software environment is discussed at several levels including programming paradigm, runtime and operating system. The importance of libraries, templates, kernels (dwarfs) and benchmarks is stressed. The programming environment has various tools including compilers with possible parallelism hints like OpenMP; tuners like Atlas; messaging models; parallelism and distribution support as in HPF, HPCS Languages, co-array Fortran, Global arrays and UPC. We also discuss the relevance of important general ideas like object-oriented paradigms (as in for example Charm++), functional languages and Software Transactional Memories. Streaming, pipelining, co-ordination, services and workflow are placed in context. Examples discussed in the last category include CCR/DSS from Microsoft and the Common Component Architecture CCA from DoE. Domain Specific environments like Matlab and Mathematica are important as there is no universal silver programming bullet; one will need interoperable focused environments. We discuss performance analysis including speedup, efficiency, scaled speedup and Amdahl's law. We show how to relate performance to algorithm/application structure and the hardware characteristics. Applications will not get scalable parallelism accidentally but only if there is an understandable performance model. We review some of the many pitfalls for both performance and correctness; these include deadlocks, race conditions, nodes that are busy doing something else and the difficulty of second-guessing automatic parallelization methods. We describe the formal sources of overhead; load imbalance and the communication/control overhead and the ways to reduce them. We relate the blocking used in caching to that used in parallel computation. We note that load balancing was the one part of parallel computing that was easier than expected. We will mix both simple idealized applications with “real problems” noting that usually it is simple problems that are the hardest as they have poor computation to control/communication ratios. We will explain the parallelism in several application classes including for science and engineering: Finite Difference, Finite Elements, FFT/Spectral, Meshes of all sorts, Particle Dynamics, Particle-Mesh, and Monte Carlo methods. Some applications like Image Processing, Graphics, Cryptography and Media coding/decoding have features similar to well understood science and engineering problems. We emphasize that nearly all applications are built hierarchically from more “basic applications” with a variety of different structures and natural programming models. Such application composition (co-ordination, workflow) must be supported with a common run-time. We contrast the difference between decomposition needed in most “basic parallel applications” to the composition supported in workflow and Web 2.0 mashups. Looking at broader application classes we should cover; Internet applications and services; artificial intelligence, optimization, machine learning; divide and conquer algorithms; tree structured searches like computer chess; applications generated by the data deluge including access, search, and the assimilation of data into simulations. There has been a growing interest in Complex Systems whether it be for critical infrastructure like energy and transportation, the Internet itself, commodity gaming, computational epidemiology or the original war games. We expect Discrete Event Simulations (such as DoD’s High Level Architecture HLA) to grow in importance as they naturally describe complex systems and because they can clearly benefit from multicore architectures. We will discuss the innovative Sequential Dynamical Systems approach used in the EpiSims, TransSims and other critical infrastructure simulation environments. In all applications we need to identify the intrinsic parallelism and the degrees of freedom that can be parallelized and distinguish small parallelism (local to core) compared to large parallelism (scalable across many cores) We will not forget many critical non-technical Issues including “Who programs – everybody or just the marine corps?”; “The market for science and engineering was small but it will be large for general multicore”; “The program exists and can’t be changed”; “What features will next hardware/software release support and how should I protect myself from change?” We will summarize lessons and relate them to application and programming model categories. In last lecture or at end of all lectures we encourage the audience to bring their own multicore application or programming model so we can discuss examples that interest you. ©2007 Microsoft Corporation. All rights reserved.

Scalable Parallel Computing on Many/Multicore Systems This set of lectures will review the application and programming model issues that will one must address when one gets chips with 32-1024 cores and “scalable” approaches will be needed to make good use of such systems. We will not discuss bit-level and instruction-level parallelism i.e. what happens on a possibly special purpose core, even though this is clearly important. We will use science and engineering applications to drive the discussion as we have substantial experience in these cases but we are interested in the lessons for commodity client and server applications that will be broadly used 5-10 years from now. We start with simple applications and algorithms from a variety of fields and identify features that makes it “obvious” that “all” science and engineering run well and scalably in parallel. We explain why unfortunately it is equally obvious that there is no straightforward way of expressing this parallelism. Parallel hardware architectures are described in enough detail to understand performance and algorithm issues and need for cross-architecture compatibility; however in these lectures we will just be users of hardware. We must understand what features of multicore chips we can and should exploit. We can explicitly use the shared memory between cores or just enjoy its implications for very fast inter-core control and communication linkage. We note that parallel algorithm research is hugely successful although this success has reduced activity in an area that deserves new attention for the next generation of architectures. The parallel software environment is discussed at several levels including programming paradigm, runtime and operating system. The importance of libraries, templates, kernels (dwarfs) and benchmarks is stressed. The programming environment has various tools including compilers with possible parallelism hints like OpenMP; tuners like Atlas; messaging models; parallelism and distribution support as in HPF, HPCS Languages, co-array Fortran, Global arrays and UPC. We also discuss the relevance of important general ideas like object-oriented paradigms (as in for example Charm++), functional languages and Software Transactional Memories. Streaming, pipelining, co-ordination, services and workflow are placed in context. Examples discussed in the last category include CCR/DSS from Microsoft and the Common Component Architecture CCA from DoE. Domain Specific environments like Matlab and Mathematica are important as there is no universal silver programming bullet; one will need interoperable focused environments. We discuss performance analysis including speedup, efficiency, scaled speedup and Amdahl's law. We show how to relate performance to algorithm/application structure and the hardware characteristics. Applications will not get scalable parallelism accidentally but only if there is an understandable performance model. We review some of the many pitfalls for both performance and correctness; these include deadlocks, race conditions, nodes that are busy doing something else and the difficulty of second-guessing automatic parallelization methods. We describe the formal sources of overhead; load imbalance and the communication/control overhead and the ways to reduce them. We relate the blocking used in caching to that used in parallel computation. We note that load balancing was the one part of parallel computing that was easier than expected. We will mix both simple idealized applications with “real problems” noting that usually it is simple problems that are the hardest as they have poor computation to control/communication ratios. We will explain the parallelism in several application classes including for science and engineering: Finite Difference, Finite Elements, FFT/Spectral, Meshes of all sorts, Particle Dynamics, Particle-Mesh, and Monte Carlo methods. Some applications like Image Processing, Graphics, Cryptography and Media coding/decoding have features similar to well understood science and engineering problems. We emphasize that nearly all applications are built hierarchically from more “basic applications” with a variety of different structures and natural programming models. Such application composition (co-ordination, workflow) must be supported with a common run-time. We contrast the difference between decomposition needed in most “basic parallel applications” to the composition supported in workflow and Web 2.0 mashups. Looking at broader application classes we should cover; Internet applications and services; artificial intelligence, optimization, machine learning; divide and conquer algorithms; tree structured searches like computer chess; applications generated by the data deluge including access, search, and the assimilation of data into simulations. There has been a growing interest in Complex Systems whether it be for critical infrastructure like energy and transportation, the Internet itself, commodity gaming, computational epidemiology or the original war games. We expect Discrete Event Simulations (such as DoD’s High Level Architecture HLA) to grow in importance as they naturally describe complex systems and because they can clearly benefit from multicore architectures. We will discuss the innovative Sequential Dynamical Systems approach used in the EpiSims, TransSims and other critical infrastructure simulation environments. In all applications we need to identify the intrinsic parallelism and the degrees of freedom that can be parallelized and distinguish small parallelism (local to core) compared to large parallelism (scalable across many cores) We will not forget many critical non-technical Issues including “Who programs – everybody or just the marine corps?”; “The market for science and engineering was small but it will be large for general multicore”; “The program exists and can’t be changed”; “What features will next hardware/software release support and how should I protect myself from change?” We will summarize lessons and relate them to application and programming model categories. In last lecture or at end of all lectures we encourage the audience to bring their own multicore application or programming model so we can discuss examples that interest you. ©2007 Microsoft Corporation. All rights reserved.

Scalable Parallel Computing on Many/Multicore Systems This set of lectures will review the application and programming model issues that will one must address when one gets chips with 32-1024 cores and “scalable” approaches will be needed to make good use of such systems. We will not discuss bit-level and instruction-level parallelism i.e. what happens on a possibly special purpose core, even though this is clearly important. We will use science and engineering applications to drive the discussion as we have substantial experience in these cases but we are interested in the lessons for commodity client and server applications that will be broadly used 5-10 years from now. We start with simple applications and algorithms from a variety of fields and identify features that makes it “obvious” that “all” science and engineering run well and scalably in parallel. We explain why unfortunately it is equally obvious that there is no straightforward way of expressing this parallelism. Parallel hardware architectures are described in enough detail to understand performance and algorithm issues and need for cross-architecture compatibility; however in these lectures we will just be users of hardware. We must understand what features of multicore chips we can and should exploit. We can explicitly use the shared memory between cores or just enjoy its implications for very fast inter-core control and communication linkage. We note that parallel algorithm research is hugely successful although this success has reduced activity in an area that deserves new attention for the next generation of architectures. The parallel software environment is discussed at several levels including programming paradigm, runtime and operating system. The importance of libraries, templates, kernels (dwarfs) and benchmarks is stressed. The programming environment has various tools including compilers with possible parallelism hints like OpenMP; tuners like Atlas; messaging models; parallelism and distribution support as in HPF, HPCS Languages, co-array Fortran, Global arrays and UPC. We also discuss the relevance of important general ideas like object-oriented paradigms (as in for example Charm++), functional languages and Software Transactional Memories. Streaming, pipelining, co-ordination, services and workflow are placed in context. Examples discussed in the last category include CCR/DSS from Microsoft and the Common Component Architecture CCA from DoE. Domain Specific environments like Matlab and Mathematica are important as there is no universal silver programming bullet; one will need interoperable focused environments. We discuss performance analysis including speedup, efficiency, scaled speedup and Amdahl's law. We show how to relate performance to algorithm/application structure and the hardware characteristics. Applications will not get scalable parallelism accidentally but only if there is an understandable performance model. We review some of the many pitfalls for both performance and correctness; these include deadlocks, race conditions, nodes that are busy doing something else and the difficulty of second-guessing automatic parallelization methods. We describe the formal sources of overhead; load imbalance and the communication/control overhead and the ways to reduce them. We relate the blocking used in caching to that used in parallel computation. We note that load balancing was the one part of parallel computing that was easier than expected. We will mix both simple idealized applications with “real problems” noting that usually it is simple problems that are the hardest as they have poor computation to control/communication ratios. We will explain the parallelism in several application classes including for science and engineering: Finite Difference, Finite Elements, FFT/Spectral, Meshes of all sorts, Particle Dynamics, Particle-Mesh, and Monte Carlo methods. Some applications like Image Processing, Graphics, Cryptography and Media coding/decoding have features similar to well understood science and engineering problems. We emphasize that nearly all applications are built hierarchically from more “basic applications” with a variety of different structures and natural programming models. Such application composition (co-ordination, workflow) must be supported with a common run-time. We contrast the difference between decomposition needed in most “basic parallel applications” to the composition supported in workflow and Web 2.0 mashups. Looking at broader application classes we should cover; Internet applications and services; artificial intelligence, optimization, machine learning; divide and conquer algorithms; tree structured searches like computer chess; applications generated by the data deluge including access, search, and the assimilation of data into simulations. There has been a growing interest in Complex Systems whether it be for critical infrastructure like energy and transportation, the Internet itself, commodity gaming, computational epidemiology or the original war games. We expect Discrete Event Simulations (such as DoD’s High Level Architecture HLA) to grow in importance as they naturally describe complex systems and because they can clearly benefit from multicore architectures. We will discuss the innovative Sequential Dynamical Systems approach used in the EpiSims, TransSims and other critical infrastructure simulation environments. In all applications we need to identify the intrinsic parallelism and the degrees of freedom that can be parallelized and distinguish small parallelism (local to core) compared to large parallelism (scalable across many cores) We will not forget many critical non-technical Issues including “Who programs – everybody or just the marine corps?”; “The market for science and engineering was small but it will be large for general multicore”; “The program exists and can’t be changed”; “What features will next hardware/software release support and how should I protect myself from change?” We will summarize lessons and relate them to application and programming model categories. In last lecture or at end of all lectures we encourage the audience to bring their own multicore application or programming model so we can discuss examples that interest you. ©2007 Microsoft Corporation. All rights reserved.

Scalable Parallel Computing on Many/Multicore Systems This set of lectures will review the application and programming model issues that will one must address when one gets chips with 32-1024 cores and “scalable” approaches will be needed to make good use of such systems. We will not discuss bit-level and instruction-level parallelism i.e. what happens on a possibly special purpose core, even though this is clearly important. We will use science and engineering applications to drive the discussion as we have substantial experience in these cases but we are interested in the lessons for commodity client and server applications that will be broadly used 5-10 years from now. We start with simple applications and algorithms from a variety of fields and identify features that makes it “obvious” that “all” science and engineering run well and scalably in parallel. We explain why unfortunately it is equally obvious that there is no straightforward way of expressing this parallelism. Parallel hardware architectures are described in enough detail to understand performance and algorithm issues and need for cross-architecture compatibility; however in these lectures we will just be users of hardware. We must understand what features of multicore chips we can and should exploit. We can explicitly use the shared memory between cores or just enjoy its implications for very fast inter-core control and communication linkage. We note that parallel algorithm research is hugely successful although this success has reduced activity in an area that deserves new attention for the next generation of architectures. The parallel software environment is discussed at several levels including programming paradigm, runtime and operating system. The importance of libraries, templates, kernels (dwarfs) and benchmarks is stressed. The programming environment has various tools including compilers with possible parallelism hints like OpenMP; tuners like Atlas; messaging models; parallelism and distribution support as in HPF, HPCS Languages, co-array Fortran, Global arrays and UPC. We also discuss the relevance of important general ideas like object-oriented paradigms (as in for example Charm++), functional languages and Software Transactional Memories. Streaming, pipelining, co-ordination, services and workflow are placed in context. Examples discussed in the last category include CCR/DSS from Microsoft and the Common Component Architecture CCA from DoE. Domain Specific environments like Matlab and Mathematica are important as there is no universal silver programming bullet; one will need interoperable focused environments. We discuss performance analysis including speedup, efficiency, scaled speedup and Amdahl's law. We show how to relate performance to algorithm/application structure and the hardware characteristics. Applications will not get scalable parallelism accidentally but only if there is an understandable performance model. We review some of the many pitfalls for both performance and correctness; these include deadlocks, race conditions, nodes that are busy doing something else and the difficulty of second-guessing automatic parallelization methods. We describe the formal sources of overhead; load imbalance and the communication/control overhead and the ways to reduce them. We relate the blocking used in caching to that used in parallel computation. We note that load balancing was the one part of parallel computing that was easier than expected. We will mix both simple idealized applications with “real problems” noting that usually it is simple problems that are the hardest as they have poor computation to control/communication ratios. We will explain the parallelism in several application classes including for science and engineering: Finite Difference, Finite Elements, FFT/Spectral, Meshes of all sorts, Particle Dynamics, Particle-Mesh, and Monte Carlo methods. Some applications like Image Processing, Graphics, Cryptography and Media coding/decoding have features similar to well understood science and engineering problems. We emphasize that nearly all applications are built hierarchically from more “basic applications” with a variety of different structures and natural programming models. Such application composition (co-ordination, workflow) must be supported with a common run-time. We contrast the difference between decomposition needed in most “basic parallel applications” to the composition supported in workflow and Web 2.0 mashups. Looking at broader application classes we should cover; Internet applications and services; artificial intelligence, optimization, machine learning; divide and conquer algorithms; tree structured searches like computer chess; applications generated by the data deluge including access, search, and the assimilation of data into simulations. There has been a growing interest in Complex Systems whether it be for critical infrastructure like energy and transportation, the Internet itself, commodity gaming, computational epidemiology or the original war games. We expect Discrete Event Simulations (such as DoD’s High Level Architecture HLA) to grow in importance as they naturally describe complex systems and because they can clearly benefit from multicore architectures. We will discuss the innovative Sequential Dynamical Systems approach used in the EpiSims, TransSims and other critical infrastructure simulation environments. In all applications we need to identify the intrinsic parallelism and the degrees of freedom that can be parallelized and distinguish small parallelism (local to core) compared to large parallelism (scalable across many cores) We will not forget many critical non-technical Issues including “Who programs – everybody or just the marine corps?”; “The market for science and engineering was small but it will be large for general multicore”; “The program exists and can’t be changed”; “What features will next hardware/software release support and how should I protect myself from change?” We will summarize lessons and relate them to application and programming model categories. In last lecture or at end of all lectures we encourage the audience to bring their own multicore application or programming model so we can discuss examples that interest you. ©2007 Microsoft Corporation. All rights reserved.

Scalable Parallel Computing on Many/Multicore Systems This set of lectures will review the application and programming model issues that will one must address when one gets chips with 32-1024 cores and “scalable” approaches will be needed to make good use of such systems. We will not discuss bit-level and instruction-level parallelism i.e. what happens on a possibly special purpose core, even though this is clearly important. We will use science and engineering applications to drive the discussion as we have substantial experience in these cases but we are interested in the lessons for commodity client and server applications that will be broadly used 5-10 years from now. We start with simple applications and algorithms from a variety of fields and identify features that makes it “obvious” that “all” science and engineering run well and scalably in parallel. We explain why unfortunately it is equally obvious that there is no straightforward way of expressing this parallelism. Parallel hardware architectures are described in enough detail to understand performance and algorithm issues and need for cross-architecture compatibility; however in these lectures we will just be users of hardware. We must understand what features of multicore chips we can and should exploit. We can explicitly use the shared memory between cores or just enjoy its implications for very fast inter-core control and communication linkage. We note that parallel algorithm research is hugely successful although this success has reduced activity in an area that deserves new attention for the next generation of architectures. The parallel software environment is discussed at several levels including programming paradigm, runtime and operating system. The importance of libraries, templates, kernels (dwarfs) and benchmarks is stressed. The programming environment has various tools including compilers with possible parallelism hints like OpenMP; tuners like Atlas; messaging models; parallelism and distribution support as in HPF, HPCS Languages, co-array Fortran, Global arrays and UPC. We also discuss the relevance of important general ideas like object-oriented paradigms (as in for example Charm++), functional languages and Software Transactional Memories. Streaming, pipelining, co-ordination, services and workflow are placed in context. Examples discussed in the last category include CCR/DSS from Microsoft and the Common Component Architecture CCA from DoE. Domain Specific environments like Matlab and Mathematica are important as there is no universal silver programming bullet; one will need interoperable focused environments. We discuss performance analysis including speedup, efficiency, scaled speedup and Amdahl's law. We show how to relate performance to algorithm/application structure and the hardware characteristics. Applications will not get scalable parallelism accidentally but only if there is an understandable performance model. We review some of the many pitfalls for both performance and correctness; these include deadlocks, race conditions, nodes that are busy doing something else and the difficulty of second-guessing automatic parallelization methods. We describe the formal sources of overhead; load imbalance and the communication/control overhead and the ways to reduce them. We relate the blocking used in caching to that used in parallel computation. We note that load balancing was the one part of parallel computing that was easier than expected. We will mix both simple idealized applications with “real problems” noting that usually it is simple problems that are the hardest as they have poor computation to control/communication ratios. We will explain the parallelism in several application classes including for science and engineering: Finite Difference, Finite Elements, FFT/Spectral, Meshes of all sorts, Particle Dynamics, Particle-Mesh, and Monte Carlo methods. Some applications like Image Processing, Graphics, Cryptography and Media coding/decoding have features similar to well understood science and engineering problems. We emphasize that nearly all applications are built hierarchically from more “basic applications” with a variety of different structures and natural programming models. Such application composition (co-ordination, workflow) must be supported with a common run-time. We contrast the difference between decomposition needed in most “basic parallel applications” to the composition supported in workflow and Web 2.0 mashups. Looking at broader application classes we should cover; Internet applications and services; artificial intelligence, optimization, machine learning; divide and conquer algorithms; tree structured searches like computer chess; applications generated by the data deluge including access, search, and the assimilation of data into simulations. There has been a growing interest in Complex Systems whether it be for critical infrastructure like energy and transportation, the Internet itself, commodity gaming, computational epidemiology or the original war games. We expect Discrete Event Simulations (such as DoD’s High Level Architecture HLA) to grow in importance as they naturally describe complex systems and because they can clearly benefit from multicore architectures. We will discuss the innovative Sequential Dynamical Systems approach used in the EpiSims, TransSims and other critical infrastructure simulation environments. In all applications we need to identify the intrinsic parallelism and the degrees of freedom that can be parallelized and distinguish small parallelism (local to core) compared to large parallelism (scalable across many cores) We will not forget many critical non-technical Issues including “Who programs – everybody or just the marine corps?”; “The market for science and engineering was small but it will be large for general multicore”; “The program exists and can’t be changed”; “What features will next hardware/software release support and how should I protect myself from change?” We will summarize lessons and relate them to application and programming model categories. In last lecture or at end of all lectures we encourage the audience to bring their own multicore application or programming model so we can discuss examples that interest you. ©2007 Microsoft Corporation. All rights reserved.

pages cm

We summarize the results of a survey on reproducibility in parallel computing, which was conducted during the Euro-Par conference in August 2015. The survey form was handed out to all participants of the conference and the workshops. The questionnaire, which specifically targeted the parallel computing community, contained questions in four different categories: general questions on reproducibility, the current state of reproducibility, the reproducibility of the participants' own papers, and questions about the participants' familiarity with tools, software, or open-source software licenses used for reproducible research.

The possibility of development of crucially new calculating structures and algorithms is described in the article. For solving this task the theory of parallel-hierarchical transformation is offered which is aimed into achievement the maximum possible algorithmic and schematic speed

Scalable Parallel Computing on Many/Multicore Systems This set of lectures will review the application and programming model issues that will one must address when one gets chips with 32-1024 cores and “scalable” approaches will be needed to make good use of such systems. We will not discuss bit-level and instruction-level parallelism i.e. what happens on a possibly special purpose core, even though this is clearly important. We will use science and engineering applications to drive the discussion as we have substantial experience in these cases but we are interested in the lessons for commodity client and server applications that will be broadly used 5-10 years from now. We start with simple applications and algorithms from a variety of fields and identify features that makes it “obvious” that “all” science and engineering run well and scalably in parallel. We explain why unfortunately it is equally obvious that there is no straightforward way of expressing this parallelism. Parallel hardware architectures are described in enough detail to understand performance and algorithm issues and need for cross-architecture compatibility; however in these lectures we will just be users of hardware. We must understand what features of multicore chips we can and should exploit. We can explicitly use the shared memory between cores or just enjoy its implications for very fast inter-core control and communication linkage. We note that parallel algorithm research is hugely successful although this success has reduced activity in an area that deserves new attention for the next generation of architectures. The parallel software environment is discussed at several levels including programming paradigm, runtime and operating system. The importance of libraries, templates, kernels (dwarfs) and benchmarks is stressed. The programming environment has various tools including compilers with possible parallelism hints like OpenMP; tuners like Atlas; messaging models; parallelism and distribution support as in HPF, HPCS Languages, co-array Fortran, Global arrays and UPC. We also discuss the relevance of important general ideas like object-oriented paradigms (as in for example Charm++), functional languages and Software Transactional Memories. Streaming, pipelining, co-ordination, services and workflow are placed in context. Examples discussed in the last category include CCR/DSS from Microsoft and the Common Component Architecture CCA from DoE. Domain Specific environments like Matlab and Mathematica are important as there is no universal silver programming bullet; one will need interoperable focused environments. We discuss performance analysis including speedup, efficiency, scaled speedup and Amdahl's law. We show how to relate performance to algorithm/application structure and the hardware characteristics. Applications will not get scalable parallelism accidentally but only if there is an understandable performance model. We review some of the many pitfalls for both performance and correctness; these include deadlocks, race conditions, nodes that are busy doing something else and the difficulty of second-guessing automatic parallelization methods. We describe the formal sources of overhead; load imbalance and the communication/control overhead and the ways to reduce them. We relate the blocking used in caching to that used in parallel computation. We note that load balancing was the one part of parallel computing that was easier than expected. We will mix both simple idealized applications with “real problems” noting that usually it is simple problems that are the hardest as they have poor computation to control/communication ratios. We will explain the parallelism in several application classes including for science and engineering: Finite Difference, Finite Elements, FFT/Spectral, Meshes of all sorts, Particle Dynamics, Particle-Mesh, and Monte Carlo methods. Some applications like Image Processing, Graphics, Cryptography and Media coding/decoding have features similar to well understood science and engineering problems. We emphasize that nearly all applications are built hierarchically from more “basic applications” with a variety of different structures and natural programming models. Such application composition (co-ordination, workflow) must be supported with a common run-time. We contrast the difference between decomposition needed in most “basic parallel applications” to the composition supported in workflow and Web 2.0 mashups. Looking at broader application classes we should cover; Internet applications and services; artificial intelligence, optimization, machine learning; divide and conquer algorithms; tree structured searches like computer chess; applications generated by the data deluge including access, search, and the assimilation of data into simulations. There has been a growing interest in Complex Systems whether it be for critical infrastructure like energy and transportation, the Internet itself, commodity gaming, computational epidemiology or the original war games. We expect Discrete Event Simulations (such as DoD’s High Level Architecture HLA) to grow in importance as they naturally describe complex systems and because they can clearly benefit from multicore architectures. We will discuss the innovative Sequential Dynamical Systems approach used in the EpiSims, TransSims and other critical infrastructure simulation environments. In all applications we need to identify the intrinsic parallelism and the degrees of freedom that can be parallelized and distinguish small parallelism (local to core) compared to large parallelism (scalable across many cores) We will not forget many critical non-technical Issues including “Who programs – everybody or just the marine corps?”; “The market for science and engineering was small but it will be large for general multicore”; “The program exists and can’t be changed”; “What features will next hardware/software release support and how should I protect myself from change?” We will summarize lessons and relate them to application and programming model categories. In last lecture or at end of all lectures we encourage the audience to bring their own multicore application or programming model so we can discuss examples that interest you. ©2007 Microsoft Corporation. All rights reserved.