"...while HPC has been primarily limited to large enterprises, R&D firms, and academic institutions in the past, there is now also a broad swath of mid-market companies adopting HPC due to the availability of affordable and open solutions, supplanting the costly and proprietary solutions of the past. All of these factors are contributing to a logical progression: HPC is maturing from high performance to high productivity."
Users of high performance computing (HPC) have long enjoyed the availability of both commercial and open source software (OSS). But they typically resided in separate worlds. Open source was favored by the academic and government research communities while commercial independent software vendors (ISVs) served the HPC needs of specific industries, such as aerospace or energy. This article examines how the wall between these two software worlds is crumbling as industry increasingly adopts more open source tools.
HPC Enters the Mainstream
The ubiquity of Linux for HPC servers is the obvious beach head for the trend in increased usage of OSS in HPC environments. But many more open source tools have become de rigueur in commercial HPC labs, such as MPICH2, an open source implementation of the Message-Passing Interface (MPI) standard for distributed-memory applications used in parallel computing. The software stacks from the OpenFabrics Alliance are designed to support server and storage clustering and grid connectivity using InfiniBand and iWARP fabrics.
At the same time, businesses increasingly need to model larger, more complex problems through simulation to stay competitive, while the cost of HPC hardware continues to plummet thanks to the availability of commodity multi-core chips. The result: demand for HPC is exploding across all industries.
And with this change comes a massive influx of new users who lack the computer science and programming expertise required for parallel computing. For example, few engineers or scientists are skilled in the low-level programming methods of MPI that are needed to achieve top computational performance. Consequently, the future of HPC applications will neither be the sole domain of OSS nor commercial applications, but rather a graceful integration of both, according to Nancy Wilkins-Diehr of the San Diego Supercomputing Center (SDSC).
She should know. Wilkins-Diehr helps thousands of users from U.S. business, government and academia take advantage of SDSC's vast HPC resources every year. As one of a small number of National Science Foundation (NSF)-funded supercomputing centers, the facility provides HPC systems, tools and support at no cost to academic users. "We're seeing a lot of blending of open source and commercial software across the scientific and engineering community," she said. "Many new users coming to us worked on PCs using popular commercial tools; others preferred open source tools. But the common problem across all users is that they've run out of computational steam on their desktops. Our goal is to let them tap into our HPC resources while continuing to work in their familiar development environments, extending their codes' reach to parallel systems using a variety of open source and commercial tools." [Editor's note: The quote is from a conversation with the author and is used with permission.]
Hybrid HPC Software
To this end, SDSC and many other research centers have deployed the Star-P interactive supercomputing platform. Star-P is an open commercial software package designed from the ground up for easy integration with a wide range of open source mathematical libraries, solvers, schedulers and other tools favoured by technical computing users. As seen in Figure 1, Star-P is suitable for use in many diverse types of industries and provides the interface to large sets of data, parallel libraries, and the management of the underlying hardware.
Figure 1: Star-P Architecture 
While the notoriously complex programming requirements of parallel systems are beyond the capabilities of the average scientist, engineer and analyst, Star-P enables them to transparently work with parallel HPC clusters and servers using familiar desktop tools. Users can code models and algorithms using either popular commercial technical computing applications or open source programming tools. Either way, users can then run their applications instantly and interactively on HPCs. It eliminates the need to re-program the code to run on parallel systems, which typically takes months to complete for large, data-intensive problems.
Star-P is client/server software that connects a Windows or Linux desktop client to a Linux-based Star-P server running on either a locally networked HPC resource, or in cases like SDSC's lab, remotely over the Internet. Linux as Star-P's server operating system is a natural fit since most of today's HPC clusters and multi-core servers are based on Linux. Much of this migration to open source HPC has been driven by the ability of standards-based commodity multi-core processors from Intel and AMD to deliver significant performance at low cost. An HPC Linux cluster can provide better performance than a traditional proprietary supercomputer at a fraction of the cost. Linux-based HPC offers an unbeatable mix of scalability, flexibility and an ever growing ecosystem of HPC hardware and software vendors to provide support.
On the front end, Star-P gives the user a choice of proprietary MATLAB or open source Python development environments. When using MATLAB, Star-P provides users of the popular very high level language (VHLL) with easy access to many open source libraries and supporting tools. Conversely, Python users can take advantage of commercial third-party add-on modules for Star-P. The workflow between proprietary and open source tools is seamless.
Support for the Python language is apropos given Star-P's goal of making users more productive. Python is an object-oriented, high-level programming language with a simple, easy to learn syntax. It requires no compilation step, and debugging Python applications is fast and easy. Consequently, programmers appreciate the increased productivity it provides.
A key benefit of using Python in HPC environments is the wide array of Python modules for technical computing. For example, users can leverage any of Python's hundreds of functions in a task parallel computation. They can also take advantage of Python-specific numerical libraries and functions, including NumPy and SciPy, the Python programming extensions that add support for large, multi-dimensional arrays and matrices. These, as well as many other modules from the Python open source community, can be run as parallel tasks to speed up tasks that can be executed independently. The Star-P Python client works in a way that feels comfortable to users of NumPy, the de facto Python package for numeric and scientific computing, as the syntax for creating and operating on distributed arrays in the Star-P Python client resembles NumPY. In fact, users can run their code on a parallel HPC with only a couple of trivial syntax changes.
This integration of open source and commercial software enables users to write a Python program handling very large matrix and array objects on their desktop PCs. During the coding they can take advantage of Python's intrinsic interactivity, as well as any of the convenient Integrated Development Environments (IDEs) available for working in Python, such as Komodo, BlackAdder or PyScripter. When processing extremely large datasets, they can export their calculation to an HPC running the Star-P server while still controlling their computation from the interactive session running on the desktop PC.
Python support is particularly useful for image processing applications in life sciences, defense and other industries thanks to the wide variety of open source modules available for them.
The combination of Python's programming speed and ease of use combined with Star-P's interactive parallelization capabilities means that Python modules can now be automatically parallelized, yielding significant productivity gains for users.
OpenConnect
Python-specific libraries and modules are just one of many kinds of open source tools that can interoperate seamlessly with Star-P. Scores of open source libraries, solvers and algorithms are available to the technical computing community. To facilitate integration, Star-P includes an enhanced application programming interface (API) called OpenConnect that enables users to easily plug in open source tools to boost productivity. Since most of these open source tools were written in low level languages like C or FORTRAN, users can plug them in into Star-P, work on them with their preferred desktop tool, and execute the code in task- or data-parallel modes with little reprogramming effort. This easy integration also enables users to utilize some powerful open source numerical libraries which may have previously been beyond their technological reach.
For example, using Star-P OpenConnect, users can plug in a variety of solvers from Sandia National Labs' Trilinos library. Trilinos uses application development building blocks called packages for computing tasks such as creating dense matrices, graphs and vectors. In a real world example, NASA scientists are combining Star-P with OSS they developed called Optical Systems Characterization and Analysis Research (OSCAR) to improve next generation space telescopes and other stellar imaging applications. OSCAR is modeling software used to design and analyze large space-based imaging systems. The software was instrumental in solving the highly publicized optical flaws of the Hubble Space Telescope. Because systems of this type require large, high-fidelity optical modeling, NASA runs OSCAR on open source-based Beowulf parallel clusters to handle the huge datasets and meet the large memory requirements.
The research project is enabling scientists in NASA's Stellar Vision Mission to boost productivity by reusing existing serial and parallel OSCAR code, without having to master OSCAR's complex programming requirements. To facilitate parallel computing, OSCAR is written entirely in C with MPI handling the computations across many processor nodes. With Star-P, scientists can re-use and extend the functionality of existing code by linking it to easy-to-use desktop tools such as Python or MATLAB. Star-P then automatically transforms the programs to run interactively on the Beowulf clusters.
Star-P is also aiding in the development of new OSS. Biomedical engineers at the University of Virginia School of Engineering and Applied Science have developed a new imaging tool that hopes to dramatically improve medical ultrasounds, potentially leading to more accurate and timely diagnoses of breast cancer and other life threatening conditions. Using Star-P, the University's biomedical engineering research team created an open source beamforming algorithm, called the Time-domain Optimized Near-field Estimator (TONE), which significantly improves the contrast and resolution of ultrasound images.
A small sampling of other open source tools that work with Star-P include:
- ScaLAPACK, a linear algebra library for parallel computers
- FFTW, a C subroutine library for computing Fast Fourier Transforms
- MUltifrontal Massively Parallel Solver (MUMPS), a tool for solving systems of linear equations
- Torque, an open source resource manager providing control over batch jobs and distributed compute nodes
- HDF5, a suite of database tools for managing extremely large and complex data collections
Conclusion
In HPC, a blending of commercial and OSS can be mutually beneficial, enhancing the strengths and functionality of each. HPC is quickly going mainstream and is no longer a rare and exotic resource limited to computer science labs. With this shift comes a huge influx of new users unfamiliar with the complexities of parallel computing. The ease of use provided by commercial software combined with the vast and rich offering of open source tools can work together to make HPC more accessible and users more productive.
Recommended Resources