A few months ago, I was chatting with a friend on the phone about HPC when he made the statement which is the title of this post. I jotted it down, and it’s been rattling around in my brain ever since. At first it seemed very odd, HPC is a neiche market, a small proportion of the overall IT market. 
The more research I did the more it made sense, especially after I read a book called “The Fourth Paradigm: Data-Intensive Scientific Discovery”, free for download from Microsoft Research. It’s not the typical marketing piece, but an excellent read if you enjoy Science and IT. It speaks about four paradigms of Scientific enquiry and then goes on to expand on the new discipline of data-intensive computing. The four paradigms of Scientific persuit are:
- empirical – describing natural phenomena, performing experiments
- theoretical – models, generalizations, developing ‘laws’ of science
- computational – simulation of complex phenomenon
- data exploration – eScience, discoveries are to be found in searching huge volumes of captured data from devices recording natural phenomena, experiments generating new results and simulations modelling problems in levels of detail beyond that acheiveable in paradigm 3.
In last year’s blog idol contest, I put some focus on the technology adoption curve and where I thought virtualization technology fell for 5 different classes of computing; Mainframes, Servers, Desktop, Embedded and High Performance Computing . In the final category, HPC, I concluded that virtualization technology was in the earliest stages of adoption. Compute virtualization is a pre-requisite technology to cloud computing (some might consider shared HPC resources using a batch scheduler without virtualization as cloud computing, but I think that’s stretching the definition of cloud too far). But my experience in HPC to that point had only been up to paradigm 3.
Paradigm 4 is a poster child for cloud computing, and makes the case of HPC dominating cloud compute cycles an obvious conclusion the more you think about it. HPC users are a small community, traditionally in the scientific and engineering disciplines. With the advent of data-intensive science, this small community will consume a disproportionate amount of the computing capacity available in the world. Not only that, but the HPC community is growing beyond science and engineering. Business analytics are making their way into HPC to accelerate time to solution, exploring mountains of structured and unstructured data for patterns that provide intelligence towards competitive advantage. I’ve also read about the potential for HPC in the Humanities.
We’ve got big data from science, engineering and now business too; open data from governments world wide as well as organizations like the World Bank. So much data and so little time - HPC enables us to look at the data in different ways with thousands of scenarios, perhaps millions of scenarios being tested concurrently – this reminds me of Hugh’s midnight musings on programming and large numbers but in this case sorting real data for usefulness not binaries of potential programs.
Yes, cloud computing changes everything, and HPC applications in business, engineering and science will dominate.
No, cloud computing changes nothing, enquiring minds will always explore the frontiers of our universe and we’ll use what ever tools they have invented along the way; and enterprising individuals will use these tools to competitive advantage to create wealth.
Ron is the founder of High Performance Computing Insights Canada and Director, Research Development, HIgh Performance Computing at Ontario Centres of Excellence. He is an electrical engineer with nearly three decades of experience in the development and management of telecommunications and High Performance Computing (HPC) products and services. His passion is to see HPC technology employed to make Canadian enterprises more globally competitive.
