Downloads & Free Reading Options - Results

One important feature of a parallel programming model is deterministic semantics: that is, a program produces the same output on every execution with a given input, regardless of the parallel schedule chosen. Determinism makes parallel programs much easier to write, understand, debug, and maintain. Further, many (though not all) parallel programs are, in fact, intended to have deterministic semantics. However, general-purpose languages, particularly those that allow arbitrary updates to shared data, typically do not guarantee determinism, leaving the burden on the programmer to ensure that the program is correctly synchronized. In this talk, I will present Deterministic Parallel Java (DPJ), an extension of the Java programming language that guarantees deterministic semantics in the presence of aliasing and updates to shared mutable objects. An important consequence of deterministic semantics is that a working sequential program needs no further testing after parallelization with DPJ. In DPJ, the programmer partitions the heap into regions and annotates method definitions with read and write effects on those regions. The programmer also explicitly marks parallel tasks and synchronization points. The compiler checks that the method effects are correctly summarized, and that the program is correctly synchronized to ensure determinism. I will describe the core language features, illustrate them with examples, and describe the results of an initial evaluation of the language. I will also describe ongoing work, including features we plan to add to the language. ©2009 Microsoft Corporation. All rights reserved.

One important feature of a parallel programming model is deterministic semantics: that is, a program produces the same output on every execution with a given input, regardless of the parallel schedule chosen. Determinism makes parallel programs much easier to write, understand, debug, and maintain. Further, many (though not all) parallel programs are, in fact, intended to have deterministic semantics. However, general-purpose languages, particularly those that allow arbitrary updates to shared data, typically do not guarantee determinism, leaving the burden on the programmer to ensure that the program is correctly synchronized. In this talk, I will present Deterministic Parallel Java (DPJ), an extension of the Java programming language that guarantees deterministic semantics in the presence of aliasing and updates to shared mutable objects. An important consequence of deterministic semantics is that a working sequential program needs no further testing after parallelization with DPJ. In DPJ, the programmer partitions the heap into regions and annotates method definitions with read and write effects on those regions. The programmer also explicitly marks parallel tasks and synchronization points. The compiler checks that the method effects are correctly summarized, and that the program is correctly synchronized to ensure determinism. I will describe the core language features, illustrate them with examples, and describe the results of an initial evaluation of the language. I will also describe ongoing work, including features we plan to add to the language. ©2009 Microsoft Corporation. All rights reserved.

A common approach in designing parallel languages is to provide some high level handles to manipulate the use of the parallel platform. This exposes some aspects of the target platform, for example, shared vs. distributed memory or task parallelism vs. data parallelism. Depending on the circumstances, these handles might be too powerful, not powerful enough or impose unnecessary constraints. Many potential programmers avoid these languages because they require too much understanding of parallelism. Many applications written this way don't come close to achieving their potential performance. Applications written in one such language often need to be rewritten for a parallel platform that is sufficiently different. Our goal is to significantly widen the set of programmers who can program for parallel targets by providing an approach that doesn't require any understanding of parallelism, while maximizing both the flexibility for optimization and the potential for reuse of the same program. Instead of viewing the language design problem as one of providing the programmer with high level handles, we view the problem as one of designing an interface. On one side of this interface is the programmer (domain expert) who knows the application but needs no knowledge of any aspects of the platform. On the other side of the interface is the performance expert (programmer or program) who demands maximal flexibility for optimizing the mapping to a wide range of target platforms (parallel / serial, shared / distributed, homogeneous / heterogeneous, etc.) but needs no knowledge of the domain. The separation of the description of the algorithm from the description of the mapping has several benefits. It allows the description of the algorithm to be stable across a much wider range of platforms. It allows for development of the algorithm by a domain expert who has no understanding or interest in parallelism. It allows for tuning by a performance expert who has little or no understanding of the domain. It facilitates independent development of the algorithm and the tuning even if they are performed by the same individual. ©2008 Microsoft Corporation. All rights reserved.

How do we get programmers to routinely write parallel software? We have been working in earnest on this problem for over 25 years; actually much longer if you consider that the first multi-threaded machine appeared in 1958 (the Gamma 60 by Bull). But at this point, I’m not sure we’re really getting any closer to solving it. If we study the history of parallel programming, it is clear that in order to solve the parallel programming problem we need to: (1) understand how people write parallel software (mine the key design patterns), (2) agree on how to discuss programmability, and (3) stop scaring away our software developers. In this talk, I will describe these issues and how my (our?) collaborations at the UC Berkeley’s ParLab are addressing them. ©2009 Microsoft Corporation. All rights reserved.

A common approach in designing parallel languages is to provide some high level handles to manipulate the use of the parallel platform. This exposes some aspects of the target platform, for example, shared vs. distributed memory or task parallelism vs. data parallelism. Depending on the circumstances, these handles might be too powerful, not powerful enough or impose unnecessary constraints. Many potential programmers avoid these languages because they require too much understanding of parallelism. Many applications written this way don't come close to achieving their potential performance. Applications written in one such language often need to be rewritten for a parallel platform that is sufficiently different. Our goal is to significantly widen the set of programmers who can program for parallel targets by providing an approach that doesn't require any understanding of parallelism, while maximizing both the flexibility for optimization and the potential for reuse of the same program. Instead of viewing the language design problem as one of providing the programmer with high level handles, we view the problem as one of designing an interface. On one side of this interface is the programmer (domain expert) who knows the application but needs no knowledge of any aspects of the platform. On the other side of the interface is the performance expert (programmer or program) who demands maximal flexibility for optimizing the mapping to a wide range of target platforms (parallel / serial, shared / distributed, homogeneous / heterogeneous, etc.) but needs no knowledge of the domain. The separation of the description of the algorithm from the description of the mapping has several benefits. It allows the description of the algorithm to be stable across a much wider range of platforms. It allows for development of the algorithm by a domain expert who has no understanding or interest in parallelism. It allows for tuning by a performance expert who has little or no understanding of the domain. It facilitates independent development of the algorithm and the tuning even if they are performed by the same individual. ©2008 Microsoft Corporation. All rights reserved.

Local search is a successful approach for solving combinatorial optimization and constraint satisfaction problems. With the progressing move toward multi and many-core systems, GPUs and the quest for Exascale systems, parallelism has become mainstream as the number of cores continues to increase. New programming models are required and need to be better understood as well as data structures and algorithms. Such is the case for local search algorithms when run on hundreds or thousands of processing units. In this paper, we discuss some experiments we have been doing with Adaptive Search and present a new parallel version of it based on GPI, a recent API and programming model for the development of scalable parallel applications. Our experiments on different problems show interesting speedups and, more importantly, a deeper interpretation of the parallelization of Local Search methods.

Local search is a successful approach for solving combinatorial optimization and constraint satisfaction problems. With the progressing move toward multi and many-core systems, GPUs and the quest for Exascale systems, parallelism has become mainstream as the number of cores continues to increase. New programming models are required and need to be better understood as well as data structures and algorithms. Such is the case for local search algorithms when run on hundreds or thousands of processing units. In this paper, we discuss some experiments we have been doing with Adaptive Search and present a new parallel version of it based on GPI, a recent API and programming model for the development of scalable parallel applications. Our experiments on different problems show interesting speedups and, more importantly, a deeper interpretation of the parallelization of Local Search methods.

The ability to dynamically adapt an unstructured -rid (or mesh) is a powerful tool for solving computational problems with evolving physical features; however, an efficient parallel implementation is rather difficult, particularly from the view point of portability on various multiprocessor platforms We address this problem by developing PLUM, tin automatic anti architecture-independent framework for adaptive numerical computations in a message-passing environment. Portability is demonstrated by comparing performance on an SP2, an Origin2000, and a T3E, without any code modifications. We also present a general-purpose load balancer that utilizes symmetric broadcast networks (SBN) as the underlying communication pattern, with a goal to providing a global view of system loads across processors. Experiments on, an SP2 and an Origin2000 demonstrate the portability of our approach which achieves superb load balance at the cost of minimal extra overhead.

by Pascal Costanza At: FOSDEM 2019 https://video.fosdem.org/2019/UD2.120/pargo.webm Go is an excellent tool for concurrent programming. Goroutines and channels make it easy to coordinate multiple interacting processes. With careful design, good concurrent Go programs run efficiently on multi-core CPUs because multiple goroutines can run in parallel on multiple CPUs. What is less known is that Go also works very well for expressing parallel algorithms that do not communicate via channels, but access shared memory directly. This goes against Go’s famous slogan: “Do not communicate by sharing memory; instead, share memory by communicating.” However, sometimes performance matters so much that you have to make an exception. Fortunately, Go is a very powerful language and makes it easy to make such exceptions. In this talk, I am going to walk you through Pargo, an open-source Go library explicitly designed for parallel programming rather than concurrent programming, which is available at https://github.com/exascience/pargo. This library is based on several years of experience with parallel programming at the ExaScience Lab at imec in Belgium, with many programming languages (C, C++, Java), many parallel programming libraries (Cilk, OpenMP, Intel Threading Building Blocks), and many applications (physics simulations, DNA sequencing, statistics). Go already provides primitives that help with parallel programming in the sync and sync/atomic packages of its standard library. Pargo builds on top of these primitives and offers high-level functions and data structures for expressing parallel algorithms. This covers simple convenience functions on top of goroutines, efficient parallel maps, parallel sort functions, and a very powerful pipeline concept for efficiently processing large data sets. This talk will cover the following topics: How Go schedules goroutines, and why they can therefore be used for task-based parallel programming. How Pargo provides a number of parallel programming primitives to make task-based parallel programming easier. How parallel Quicksort and Mergesort can be implemented with Pargo. How Pargo allows you to write parallel pipelines for efficiently streaming, filtering, and transforming large data sets in parallel. Pascal Costanza has a Ph.D. degree from the University of Bonn, Germany, and works as a senior researcher at imec, Leuven, Belgium. He was previously employed by Intel Corporation, and Vrije Universiteit Brussel, Belgium. He is an expert on high-performance and data-intensive computing with applications in bioinformatics, sequencing pipelines, and artificial intelligence. Room: UD2.120 (Chavanne) Scheduled start: 2019-02-02 16:00:00+01 Source: https://www.youtube.com/watch?v=3fem24-ZQKE Uploader: FOSDEM

by Pascal Costanza At: FOSDEM 2019 https://video.fosdem.org/2019/UD2.120/pargo.webm Go is an excellent tool for concurrent programming. Goroutines and channels make it easy to coordinate multiple interacting processes. With careful design, good concurrent Go programs run efficiently on multi-core CPUs because multiple goroutines can run in parallel on multiple CPUs. What is less known is that Go also works very well for expressing parallel algorithms that do not communicate via channels, but access shared memory directly. This goes against Go’s famous slogan: “Do not communicate by sharing memory; instead, share memory by communicating.” However, sometimes performance matters so much that you have to make an exception. Fortunately, Go is a very powerful language and makes it easy to make such exceptions. In this talk, I am going to walk you through Pargo, an open-source Go library explicitly designed for parallel programming rather than concurrent programming, which is available at https://github.com/exascience/pargo. This library is based on several years of experience with parallel programming at the ExaScience Lab at imec in Belgium, with many programming languages (C, C++, Java), many parallel programming libraries (Cilk, OpenMP, Intel Threading Building Blocks), and many applications (physics simulations, DNA sequencing, statistics). Go already provides primitives that help with parallel programming in the sync and sync/atomic packages of its standard library. Pargo builds on top of these primitives and offers high-level functions and data structures for expressing parallel algorithms. This covers simple convenience functions on top of goroutines, efficient parallel maps, parallel sort functions, and a very powerful pipeline concept for efficiently processing large data sets. This talk will cover the following topics: How Go schedules goroutines, and why they can therefore be used for task-based parallel programming. How Pargo provides a number of parallel programming primitives to make task-based parallel programming easier. How parallel Quicksort and Mergesort can be implemented with Pargo. How Pargo allows you to write parallel pipelines for efficiently streaming, filtering, and transforming large data sets in parallel. Pascal Costanza has a Ph.D. degree from the University of Bonn, Germany, and works as a senior researcher at imec, Leuven, Belgium. He was previously employed by Intel Corporation, and Vrije Universiteit Brussel, Belgium. He is an expert on high-performance and data-intensive computing with applications in bioinformatics, sequencing pipelines, and artificial intelligence. Room: UD2.120 (Chavanne) Scheduled start: 2019-02-02 16:00:00+01 Source: https://www.youtube.com/watch?v=3fem24-ZQKE Uploader: FOSDEM

Parallel programming techniques have been used for years to develop processing software that does the same work in only a fraction of the time. The research in this paper focuses on the I/O problem associated with a parallel application writing to a single physical disk. Included in our research are the original ideas that led to the first version of the parallel software, subsequent versions of the software derived from lessons learned from benchmark results, and speedup results of each version. The platform used was a custom built Linux Beowulf cluster running a standard Linux kernel and the MPICH parallel message-passing library. The underlying purpose of this software is to process hydrographic data having a complicated, multi-tiered format. The data processing involves reading tens to hundreds of files containing raw data, filtering out extraneous data values, and writing the filtered data to a single file used in additional processing. The problem is not computationally intensive, but bound by the system's file writing capability. Subsequent versions of the parallel software developed exploit the strengths of the system's hardware to write the output file in the most time efficient manner. Each software version uses advanced software architecture schemes to achieve better results. Results show that the more responsible the software was for organizing the data before writing, the better the speedup. The critical factor for writing data efficiently involved the limitation of writing data over a single I/O controller. Our parallel software has fantastic utility where system specifications do not allow for the use of parallel file systems, or writing data over multiple I/O controllers.

Under the current grant parallel hardware and systems software implemented on ZMOB in the previous year underwent extensive testing. A parallel problem solving system, PRISM (Parallel Inference System) implemented on the VAX/11-780 in the previous year was implemented on the PYRAMID and SUN machines. The initial version of PRISM uses a simulation of the ZMOB hardware, and has been fully tested and debugged. Experimental testing of PRISM on the simulated system was undertaken in the current year. In addition, several enhancements were made to PRISM to permit experimental analyses to be made, and to incorporate additional features to take full advantage of parallelism in a problem solving environment. The tracing and statistical gathering packages were extended. An AND-parallelism capability was added to achieve a second version of the PRISM system, and other features were added to the system to more fully exploit parallelism. A constraint solving machine was integrated with PRISM. In addition to the above, a general method to permit informative answers to be presented to a user has been developed. Theoretical results were obtained for circumscription and a method for computing in protected circumscription, using Horn clauses was developed. In the area of systems hardware and software, the ZMOB processor is now fully functional and in everyday use with 128 processors.

Object-oriented programming techniques have become a vital part of modern software engineering. Most large new commercial software products are designed today using objects oriented principles along with supporting development environments and tools. Such methodology is particularly appealing for parallel computing, where there is great need for approaches capable of simplifying the programming task and producing high quality software more quickly. In this project we have carried out research that will lead to a new parallel object-oriented language based on C++. The new language, C++-Linda, is a novel combination of the most widely used object-oriented language and a successful environment for parallel computing that supports a virtual shared memory model for process interaction. This combination is an especially appropriate one because Linda's memory model is naturally object-oriented, enabling a seamless design that is completely consistent with object-oriented methodology. In this Phase I project, we have carried out research to understand the basic issues underlying parallel object-oriented languages, and we have created a preliminary design for C++-Linda. In a future Phase II project, we plan to develop a prototype implementation that will serve as a guide for eventual commercialization. Parallel computing, Programming languages, Object-oriented languages, Linda, C++-Linda, (C++).

Automatic generation of sequential and parallel programs can bring the power of modern computers to more engineers and scientists. The project investigated parallel code generation and automatic derivation of parallel finite element routines. Software packages GENCRAY, a code translator producing Cray Fortran, and PIER, a finite element code derivation system, have been constructed as research tools.

There is a lot more to parallel programming in Python than multiprocessing.Pool().map. In this talk I will share some hard-learned knowledge gained in several years of parallel programming. Covered topics will include performance, ways to measure the performance, memory occupation, data transfer and ways to reduce the data transfer, how to debug parallel programs and useful libraries. I will give some practical examples, both in enterprise programming (importing CSV files in a database) and in scientific programming (numerical simulations). The initial part of the talk will be pedagogical, advocating the convenience of parallel programming in the small (i.e. in single machine environment); the second part will be more advanced and will touch a few things to know when writing parallel programs for medium-sized clusters. I will also briefly discuss the compatibility layer that we have developed at GEM to be independent from the underlying parallelization technology (multiprocessing, concurrent.futures, celery, ipyparallel, grid engine...). Please see our speaker release agreement for details: https://ep2017.europython.eu/en/speaker-release-agreement/

This progress report summarizes work performed under AFOSR-88-0152 on parallel logic programming, problem solving, and deductive databases. A parallel problem solving system, PRISM (Parallel Inference System), that was implemented on McMOB was ported to the BBN Butterfly machine. Two versions of PRISM were developed and are operational on the Butterfly: a message passing ring structure system and a shared-memory system. Experimental testing of PRISM on McMOB continued, while experiments were also conducted on the Butterfly systems. Three enhancements were made and completed during the grant period. These are: a capability to handle negated queries and a capability to assert and retract statements. In addition to the above, work continued in the area of informative answers to queries in deductive data bases. A thesis was completed on the subject. An interpreter was developed and is running, that can take restricted natural language as input and can respond with a cooperative natural language output. In the area of parallel software development, the following were accomplished. Theoretical work on slicing/splicing was completed. Tools were provided for software development using artificial intelligence techniques. AI software for massively parallel architectures was started.

Challenges addressed under the grant include: (i) improving our understanding of the many-electron quantum mechanisms by which nature uses strong electron correlation for efficient energy transfer, particularly in photosynthesis and bioluminescence, (ii) providing an innovative paradigm for energy transfer in photovoltaic materials by which new levels of solar efficiency are achieved through the use of strong electron correlation and entanglement, (iii) enhancing two-electron reduced-density-matrix (2-RDM)-based electronic-structure methods that significantly expand the range of strongly correlated molecular systems that can be studied with applications throughout science and engineering, and (iv) developing a new generation of large-scale, parallel algorithms for performing semidefinite programming with applications to problems in engineering, computer science, statistics, finance and economics. Research led to important technology transitions including the formation of RDMCHEM LLC, a software company that is developing the next generation of computational software for chemistry with applications to engineering, molecular biology, and physics.

The objective of this effort was to investigate methods for distributing a knowledge base over multiple processors using the logic programming paradigm. A three-fold approach was employed and is discussed in this report. The first phase involved the testing of the Parallel Knowledge- based Execution System (PKES). Secondly, memory allocation techniques were investigated and evaluated. Lastly, a study of the implementation of data-typing in a logic programming paradigm is discussed. (CP)

CONTEXT: Writing software for the current generation of parallel systems requires significant programmer effort, and the community is seeking alternatives that reduce effort while still achieving good performance. OBJECTIVE: Measure the effect of parallel programming models (message-passing vs. PRAM-like) on programmer effort. DESIGN, SETTING, and SUBJECTS: One group of subjects implemented sparse-matrix dense-vector multiplication using message-passing (MPI), and a second group solved the same problem using a PRAM-like model (XMTC). The subjects were students in two graduate-level classes: one class was taught MPI and the other was taught XMTC. MAIN OUTCOME MEASURES: Development time, program correctness. RESULTS: Mean XMTC development time was 4.8 hours less than mean MPI development time (95% confidence interval, 2.0-7.7), a 46% reduction. XMTC programs were more likely to be correct, but the difference in correctness rates was not statistically significant (p=.16). CONCLUSIONS: XMTC solutions for this particular problem required less effort than MPI equivalents, but further studies are necessary which examine different types of problems and different levels of programmer experience.

As the complexity of machines and architectures has increased, performance tuning has become more challenging, leading to the failure of general compilers to generate the best possible optimized code. Expert performance programmers can often hand-write code that outperforms compiler-optimized low-level code by an order of magnitude. At the same time, the complexity of programs has also increased, with modern programs built on a variety of abstraction layers to manage complexity, yet these layers hinder efforts at optimization. In fact, it is common to lose one or two additional orders of magnitude in performance when going from a low-level language such as Fortran or C to a high-level language like Python, Ruby, or Matlab.

We present a library of generic software components for parallel and distributed computations on graphs, based on the Boost Graph Library (BGL). The BGL consists of a rich set of generic graph algorithms and supporting data structures, but it was not originally designed with parallelism in mind. In this talk, we revisit the abstractions comprising the BGL in the context of distributed-memory parallelism, lifting away the implicit requirements of sequential execution and a single shared address space. We illustrate our approach by describing the process as applied to one of the core algorithms in the BGL, breadth-first search. The result is a generic algorithm that is unchanged from the sequential algorithm, requiring only the introduction of external (distributed) data structures for parallel execution. More importantly, the generic implementation retains its interface and semantics, such that other distributed algorithms can be built upon it, just as algorithms are layered in the sequential case. By characterizing these extensions as well as the extension process, we develop general principles and patterns for using (and reusing) generic parallel software libraries. We demonstrate that the resulting algorithm implementations are both efficient and scalable with performance results for several algorithms implemented in the open-source Parallel Boost Graph Library. ©2007 Microsoft Corporation. All rights reserved.

Highly parallel architectures will be useful in meeting the demands of computationally intensive tasks only to the extent that it is possible to write efficient parallel software. The problems are enormous. The parallel programmer must simultaneously code for multiple processes, orchestrating their communication and synchronization; he must efficiently map logical processes onto disparate hardware configurations and schedule their execution. Further, he must debug-both for correctness and performance - in spite of a potentially overwhelming amount of relevant information and in the absence of reproducibility or consistent global state. If it is not possible to provide sophisticated programming support for these activities, it is unlikely that highly parallel computation will be generally available to either the scientific or the commercial communities. In our research, we have investigated aspects of parallel computation that are specific to massive parallelism.

At CodeCamp NYC Aaron Armstrong details Parallel Programming in Visual Studio with .Net 4.0 See Part 2

Experimenting with Chapel Programming Language - a parallel processing programming language by Cray the Supercomputer people. It uses as much parallelism as you have available. This is a dual core laptop, so it splits up hello, world across the 2 cores efficiently. If I had 2 million? Same.

International Journal of Parallel Programming 1990: Volume 19 , Issue CONTENTS. Digitized from IA1631414-07 . Previous issue: sim_international-journal-of-parallel-programming_1989-12_18_6 . Next issue: sim_international-journal-of-parallel-programming_1990-02_19_1 .

The objectives of this investigation are to develop a new programming model, called lattice parallelism, that will significantly reduce the amount of effort required to parallelize non-uniform structured applications. We implemented the model as two C++ class libraries, called KeThP and LPARX. We used XCeLP and LPARX to develop a real-space adaptive mesh solver for a significant grand challenge application---first principles simulations of real materials. Our adaptive solver reduces %emory and tire consumption by two orders of magnitude over a uniform mesh based method. The code is fully portable and runs on the Intel Paragon, Cray C90 and IBM SP2. Performance on the MPP systems is competitive with the Cray C90.

International Journal of Parallel Programming 2002: Volume 30 , Issue CONTENTS. Digitized from IA1631414-07 . Previous issue: sim_international-journal-of-parallel-programming_2001-12_29_6 . Next issue: sim_international-journal-of-parallel-programming_2002-02_30_1 .

This paper addresses the problem of universal synchronization primitives that can support scalable thread synchronization for large-scale many-core architectures. The universal synchronization primitives that have been deployed widely in conventional architectures like CAS and LL/SC are expected to reach their scalability limits in the evolution to many-core architectures with thousands of cores. We introduce a non-blocking full/empty bit primitive, or NB-FEB for short, as a promising synchronization primitive for parallel programming on may-core architectures. We show that the NB-FEB primitive is universal, scalable, feasible and convenient to use. NB-FEB, together with registers, can solve the consensus problem for an arbitrary number of processes (universality). NB-FEB is combinable, namely its memory requests to the same memory location can be combined into only one memory request, which consequently mitigates performance degradation due to synchronization "hot spots" (scalability). Since NB-FEB is a variant of the original full/empty bit that always returns a value instead of waiting for a conditional flag, it is as feasible as the original full/empty bit, which has been implemented in many computer systems (feasibility). The original full/empty bit is well-known as a special-purpose primitive for fast producer-consumer synchronization and has been used extensively in the specific domain of applications. In this paper, we show that NB-FEB can be deployed easily as a general-purpose primitive. Using NB-FEB, we construct a non-blocking software transactional memory system called NBFEB-STM, which can be used to handle concurrent threads conveniently. NBFEB-STM is space efficient: the space complexity of each object updated by $N$ concurrent threads/transactions is $\Theta(N)$, the optimal.

This paper addresses the problem of universal synchronization primitives that can support scalable thread synchronization for large-scale many-core architectures. The universal synchronization primitives that have been deployed widely in conventional architectures like CAS and LL/SC are expected to reach their scalability limits in the evolution to many-core architectures with thousands of cores. We introduce a non-blocking full/empty bit primitive, or NB-FEB for short, as a promising synchronization primitive for parallel programming on may-core architectures. We show that the NB-FEB primitive is universal, scalable, feasible and convenient to use. NB-FEB, together with registers, can solve the consensus problem for an arbitrary number of processes (universality). NB-FEB is combinable, namely its memory requests to the same memory location can be combined into only one memory request, which consequently mitigates performance degradation due to synchronization "hot spots" (scalability). Since NB-FEB is a variant of the original full/empty bit that always returns a value instead of waiting for a conditional flag, it is as feasible as the original full/empty bit, which has been implemented in many computer systems (feasibility). The original full/empty bit is well-known as a special-purpose primitive for fast producer-consumer synchronization and has been used extensively in the specific domain of applications. In this paper, we show that NB-FEB can be deployed easily as a general-purpose primitive. Using NB-FEB, we construct a non-blocking software transactional memory system called NBFEB-STM, which can be used to handle concurrent threads conveniently. NBFEB-STM is space efficient: the space complexity of each object updated by $N$ concurrent threads/transactions is $\Theta(N)$, the optimal.

Equational Programming Language, abbreviated EPL, has been designed and implemented at Rensselaer Polytechnic Institute to increase productivity of programmers in the area of real-time and parallel programming. EPL is a simple non-strict functional language with type inference. The language is defined in terms of just a few constructs: generalized arrays and subscripts for the data value definitions, ports for process interactions and virtual processors for execution directives. Yet, its powerful complier can generate object code for a variety of parallel and distributed architectures. The compilation is based on conditional data dependence analysis and data attribute propagation. The interplay between the user supplied annotations and complier transformation has also been investigated. The software tools for integration of EPL programs into a parallel computation have been developed.

Equational Programming Language, abbreviated EPL, has been designed and implemented at Rensselaer Polytechnic Institute to increase productivity of programmers in the area of real-time and parallel programming. EPL is a simple non-strict functional language with type inference. The language is defined in terms of just a few constructs: generalized arrays and subscripts for the data value definitions, ports for process interactions and virtual processors for execution directives. Yet, its powerful complier can generate object code for a variety of parallel and distributed architectures. The compilation is based on conditional data dependence analysis and data attribute propagation. The interplay between the user supplied annotations and complier transformation has also been investigated. The software tools for integration of EPL programs into a parallel computation have been developed.

For better utilization of computing resources, it is important to consider parallel programming environments in which the number of available processors varies at runtime. In this paper, we discuss runtime support for data parallel programming in such an adaptive environment. Executing programs in an adaptive environment requires redistributing data when the number of processors changes, and also requires determining new loop bounds and communication patterns for the new set of processors. We have developed a runtime library to provide this support. We discuss how the runtime library can be used by compilers of HPF-like languages to generate code for an adaptive environment. We present performance results for Navier-Stokes solver on a multigrid template run on a network of workstations and an IBM SP-2. Our experiments show that if the number of processors is not varies frequently, the cost of data distribution is not significant when compared to the time required for the actual computation. Overall, our work establishes the feasibility of compiling HPF for a network on non-dedicated workstations, which are likely to be an important resource for parallel programming in the future.

In order to be truly portable, a program must be tolerant of a wide range of development and execution environments, and a parallel program is just one which must be tolerant of a very wide range. This paper first defines the term "tolerant programming", then describes many layers of tools to accomplish it. The primary focus is on F-Nets, a formal model for expressing computation as a folded partial-ordering of operations, thereby providing an architecture-independent expression of tolerant parallel algorithms. For implementing F-Nets, Cooperative Data Sharing (CDS) is a subroutine package for implementing communication efficiently in a large number of environments (e.g. shared memory and message passing). Software Cabling (SC), a very-high-level graphical programming language for building large F-Nets, possesses many of the features normally expected from today's computer languages (e.g. data abstraction, array operations). Finally, L2(sup 3) is a CASE tool which facilitates the construction, compilation, execution, and debugging of SC programs.

When algorithmic skeletons were first introduced by Cole in late 1980 the idea had an almost immediate success. The skeletal approach has been proved to be effective when application algorithms can be expressed in terms of skeletons composition. However, despite both their effectiveness and the progress made in skeletal systems design and implementation, algorithmic skeletons remain absent from mainstream practice. Cole and other researchers, focused the problem. They recognized the issues affecting skeletal systems and stated a set of principles that have to be tackled in order to make them more effective and to take skeletal programming into the parallel mainstream. In this thesis we propose tools and models for addressing some among the skeletal programming environments issues. We describe three novel approaches aimed at enhancing skeletons based systems from different angles. First, we present a model we conceived that allows algorithmic skeletons customization exploiting the macro data-flow abstraction. Then we present two results about the exploitation of meta-programming techniques for the run-time generation and optimization of macro data-flow graphs. In particular, we show how to generate and how to optimize macro data-flow graphs accordingly both to programmers provided non-functional requirements and to execution platform features. The last result we present are the Behavioural Skeletons, an approach aimed at addressing the limitations of skeletal programming environments when used for the development of component-based Grid applications. We validated all the approaches conducting several test, performed exploiting a set of tools we developed.

When algorithmic skeletons were first introduced by Cole in late 1980 the idea had an almost immediate success. The skeletal approach has been proved to be effective when application algorithms can be expressed in terms of skeletons composition. However, despite both their effectiveness and the progress made in skeletal systems design and implementation, algorithmic skeletons remain absent from mainstream practice. Cole and other researchers, focused the problem. They recognized the issues affecting skeletal systems and stated a set of principles that have to be tackled in order to make them more effective and to take skeletal programming into the parallel mainstream. In this thesis we propose tools and models for addressing some among the skeletal programming environments issues. We describe three novel approaches aimed at enhancing skeletons based systems from different angles. First, we present a model we conceived that allows algorithmic skeletons customization exploiting the macro data-flow abstraction. Then we present two results about the exploitation of meta-programming techniques for the run-time generation and optimization of macro data-flow graphs. In particular, we show how to generate and how to optimize macro data-flow graphs accordingly both to programmers provided non-functional requirements and to execution platform features. The last result we present are the Behavioural Skeletons, an approach aimed at addressing the limitations of skeletal programming environments when used for the development of component-based Grid applications. We validated all the approaches conducting several test, performed exploiting a set of tools we developed.

International Journal of Parallel Programming 2004: Volume 32 , Issue CONTENTS. Digitized from IA1631414-07 . Previous issue: sim_international-journal-of-parallel-programming_2003-12_31_6 . Next issue: sim_international-journal-of-parallel-programming_2004-02_32_1 .

International Journal of Parallel Programming 2006: Volume 34 , Issue CONTENTS. Digitized from IA1631414-07 . Previous issue: sim_international-journal-of-parallel-programming_2005-12_33_6 . Next issue: sim_international-journal-of-parallel-programming_2006-02_34_1 .

International Journal of Parallel Programming 2006: Volume 34 , Issue CONTENTS. Digitized from IA1631414-07 . Previous issue: sim_international-journal-of-parallel-programming_2005-12_33_6 . Next issue: sim_international-journal-of-parallel-programming_2006-02_34_1 .

International Journal of Parallel Programming 2006: Volume 34 , Issue CONTENTS. Digitized from IA1631414-07 . Previous issue: sim_international-journal-of-parallel-programming_2005-12_33_6 . Next issue: sim_international-journal-of-parallel-programming_2006-02_34_1 .

Includes bibliographical references (p. [709]-736) and indexes

How do we get programmers to routinely write parallel software? We have been working in earnest on this problem for over 25 years; actually much longer if you consider that the first multi-threaded machine appeared in 1958 (the Gamma 60 by Bull). But at this point, I’m not sure we’re really getting any closer to solving it. If we study the history of parallel programming, it is clear that in order to solve the parallel programming problem we need to: (1) understand how people write parallel software (mine the key design patterns), (2) agree on how to discuss programmability, and (3) stop scaring away our software developers. In this talk, I will describe these issues and how my (our?) collaborations at the UC Berkeley’s ParLab are addressing them. ©2009 Microsoft Corporation. All rights reserved.

DokuWiki: Programming Massively Parallel Multiprocessors with CUDA Dumped with DokuWiki-Dumper v0.1.44, and uploaded with dokuWikiUploader v0.1.44.

This research is an attempt to reason about the control of parallel computation in the world of applicative programming languages. Applicative languages, in which computation is performed through function application and in which functions are treated as first-class objects, have the benefits of elegance expressiveness and having clean semantics. Parallel computation and real-world concurrent activities are much harder to reason about than the sequential counterparts. Many parallel applicative languages have thus hidden most control details with their declarative programming styles, but they are not expressive enough to characterize many real world concurrent activities that can be easily explained with concepts such as message passing pipelining and so on. Ease of programming should not come at the expense of expressiveness. Therefore we design a parallel applicative language Pscheme such that programmers can express explicitly the control of parallel computation while maintaining the clean semantics and the ease of programming of applicative languages. In Pscheme, we propose the concept of ports to model the general control in parallel computation. Through program examples we show how Pscheme and ports support various parallel programming paradigms. We have also built libraries for higher level control facilities with ports so that programming in Pscheme becomes easier. We provide an operational semantics for Pscheme, and develop a compiler and a run time system on NYU's Ultracomputer. Our experiments with parallel programs have shown satisfactory speedup. We claim that ports are the natural parallel extensions of continuations in sequential computation, and thus conclude that representing general control in parallel applicative programming is feasible.

Parallel programming is more difficult than sequential programming in part because of the complexity of reasoning, testing, and debugging in the context of concurrency. In the thesis, we present and investigate a parallel programming model that provides direct control of parallelism in a notation with sequential semantics. Our model consists of a standard sequential imperative programming notation extended with the following three pragmas.

We have implemented in simulation a prototype language for the Connection Machine called CLI. CLi is an extrapolation of serial machine programming language technology: in CLl one programs the individual processors to perform local computations and talk to the communications network. We present details of the largest of our experiments with CLl, an interpreter for Scheme (a dialect of Lisp) that allows a large number of different Scheme programs to be run in parallel on the otherwise SIMD Connection Machine. Our aim was not to propose Scheme as a language for Connection Machine programming, but to gain experience using CLl to implement an interesting and familiar algorithm. Consideration of the difficulties we encountered led us to the conclusion that CLl programs do not capture enough of the casual structure of the processes they describe. Starting from this observation, we have designed a successor language called CGL (for Connection Graph Language).

This document proposes an interface to parallel file systems intended for use with a variety of parallel computers. This proposal is based on the separation of programmer convenience functions from high performance enabling functions. We propose that the former be supported above this interface, possibly in client libraries. The latter, functions that enable high performance, are defined by this proposed API under the assumption that these functions are more likely to need system and vendor specific support. Specifically, this proposal includes functions which support reading and writing with scatter-gather addressing for memory and file ranges, and asynchronous operations. It also includes mechanisms that permit client control over client caching, and file access and layout hints. Finally, it includes a mechanism by which this API can be extended and extensions for fast file copy and batching collective I/O operations.

This project supports the conversion of codes in Computational Structural Mechanics (CSM) to a parallel form which will efficiently exploit the computational power available from multiprocessors. The work is a part of a comprehensive, FORTRAN-based system to form a basis for a parallel version of the NICE/SPAR combination which will form the CSM Testbed. The software is macro-based and rests on the force methodology developed by the principal investigator in connection with an early scientific multiprocessor. Machine independence is an important characteristic of the system so that retargeting it to the Flex/32, or any other multiprocessor on which NICE/SPAR might be imnplemented, is well supported. The principal investigator has experience in producing parallel software for both full and sparse systems of linear equations using the force macros. Other researchers have used the Force in finite element programs. It has been possible to rapidly develop software which performs at maximum efficiency on a multiprocessor. The inherent machine independence of the system also means that the parallelization will not be limited to a specific multiprocessor.

Fortran and C++ are the dominant programming languages used in scientific computation. Consequently, extensions to these languages are the most popular for programming massively parallel computers. We discuss two such approaches to parallel Fortran and one approach to C++. The High Performance Fortran Forum has designed HPF with the intent of supporting data parallelism on Fortran 90 applications. HPF works by asking the user to help the compiler distribute and align the data structures with the distributed memory modules in the system. Fortran-S takes a different approach in which the data distribution is managed by the operating system and the user provides annotations to indicate parallel control regions. In the case of C++, we look at pC++ which is based on a concurrent aggregate parallel model.

In creating a heterogeneous parallel processing capability we are really trying to approach three basic problems with current systems: (1) Supercomputer and parallel computer hardware architectures vary widely but need to support one or two fairly standard programming languages and programming models. A particularly important issue concerns the short life cycle of individual hardware designs; (2) Many algorithms require capabilities beyond the reach of single superconducters but could be approached by several machines working together; and (3) Performing a given task requires integration of a 'system' that may contain many components in addition to the super or parallel computer itself. Peripherals from many different manufacturers must be incorporated. (KR)

Parallel programming involves finding the potential parallelism in an application, choosing an algorithm, and mapping it to the architecture at hand. Since a typical algorithm has much more potential parallelism than any single architecture can effectively exploit, we usually program the parallelism that the available control constructs easily express and that the given architecture efficiently exploits. This approach produces programs that exhibit much less parallelism than the original algorithm and whose performance depends entirely on the underlying architecture. To port such a program to a new architecture, we must rewrite the program to remove any ineffective parallelism and to recover any lost parallelism appropriate for the new machine. In this paper we show how to adapt a parallel program to different architectures using control abstraction. With control abstraction we can define and use a rich variety of control constructs to represent an algorithm's potential parallelism. Since control abstraction separates the definition of a construct from its implementation, a construct may have several different implementations, each exploiting a different subset of the parallelism admitted by the construct. By selecting an implementation for each control construct using annotations, we can vary the parallelism we choose to exploit without otherwise changing the source code. This approach produces programs that exhibit most of, if not all, the potential parallelism in an algorithm, and whose performance can be tuned for a specific architecture simply by choosing among the various implementations for the control constructs in use. Parallel programming, Architectural adaptability, Control abstraction, Architectural independence, Potential parallelism, Exploited parallelism, Closures, Annotations.