The Power and Legacy of Virginia Tech’s Mac-Based Supercomputers

When we think about data centers today, we often picture rows of GPU-packed racks run by companies like Microsoft, Google, Amazon, and Meta, and used by AI labs such as OpenAI and Anthropic. Much of that hardware does one of two jobs: training new models or running finished ones to answer requests from millions of people around the world.

A supercomputer is built from many of the same parts but for a different purpose. It is typically owned by a government lab or university and designed for scientific simulation, such as modeling climate, nuclear physics, or new materials. That work demands high-precision math and very fast communication between processors. Rather than serving the masses, a supercomputer is shared among researchers, whose large jobs wait in a queue and run according to a schedule.

Early examples like UNIVAC and the IBM 7030 were used by scientists working on atomic research. The Cray-1 came along in the mid-70s and accelerated such work to new levels. Each of these machines brought with it new technologies that made those breakthroughs possible. They were also fairly bespoke, built of components regular nerds could only dream about.

Over time, that changed. By the time the 1990s rolled around, companies like Cray saw their hardware being outpaced by newcomers who were packing far more processors into their products. The kicker? Those processors were much more pedestrian than the custom silicon older supercomputers were built around.

In 1998, David Bader built what he calls “the first bona fide Linux supercomputer” with off-the-shelf parts, including the Intel Pentium II.¹

That brings us to 2003, and the subject of today’s article: Virginia Tech’s Mac-based supercomputers.

Bang for the Buck

As you can see from these photos, System X — pronounced “System Ten”² — was built with 1,100 Power Mac G5s:

Photos from Virginia Tech’s Advanced Research Computing website, via the Wayback Machine.

Honestly, I could end this article right here. Look how badass this all is! I really do love the look of the Power Mac G5.

Virginia Tech built this supercomputer in the summer of 2003, with the goal of hitting 10 teraflops. Once it was up and running that fall, it ranked as the world’s third-fastest computer, and the fastest at an academic institution. You can see from this video that the G5 was chosen to hit these numbers without breaking the bank:

(This video played as part of Apple’s product introduction for the Power Mac G5 itself.)

The total cost was $5.2 million,³ a far cry from the $400 million spent to build the fastest supercomputer at the time, the Earth Simulator.

(System X also looked much cooler.)

Both Virginia Tech faculty members and students could get time on the machine free of charge, and thanks to the dual 2.0 GHz processors inside each G5, their projects could tap into vast resources easily.

On the software front, every node ran Mac OS X. The machines arrived running Mac OS X 10.2.7, with plans to move to Panther shortly after.

Each G5 came equipped with 4 GB of DDR400 RAM, a 160 GB Serial ATA hard drive, and a Mellanox InfiniBand PCI-X card, which allowed the machines to be linked to each other and to the network over high-speed cables. Notably, the Power Mac G5 didn’t support ECC memory. That would matter later.

The G5s were unboxed and installed by students and faculty:

The whole thing came together pretty quickly, according to this FAQ:

Planning began in January of 2003. Facilities planning began in April of 2003. Renovations began the first week of August and were completed on September 23, 2003. Computers, network equipment, and cables began arriving the first week of September 2003. The computers arrived in six shipments during a seven day period and were processed into the racks in less than 2 hours per shipment with no more than 30 volunteers working a shift.

Ethernet cabling (roughly 1200 connections) was done by a four man team from our internal communications and network services group and was completed in less than 14 days.

InfiniBand cabling (roughly 1700 connections) began September 13, 2003 and was completed by a team of five people in 10 days.

I was lucky to get an email about this from Sam Davies, who was a student and an Apple Campus Representative at Virginia Tech at the time. He told me:

In the summer of 2003, I was fortunate to be a part of the team working to put System X together (that was the official name, unofficially the Mac community and everyone else called it “Big Mac”).

The first 1,100 G5s off of Apple’s assembly line were unloaded from trucks, unboxed, and carried into server racks by a huge team of faculty, staff, Apple project folks, and student volunteers like me. Only one G5 tower in the batch of 1100 couldn’t be used, and that was because one of the Apple sales reps dropped it. (Those handles are sharp!)

Of course, having a bunch of tower machines stuffed into racks isn’t very space-efficient. The next year, Apple had a solution.

The Xserve G5

Here’s Apple PR, in January 2004:

Apple today introduced Xserve G5, the most powerful Xserve yet, delivering over 30 gigaflops of processing power per system, about 60 percent more than the PowerPC G4-based Xserve. Xserve G5 features the same revolutionary PowerPC G5, 64-bit processor used in Virginia Tech’s cluster of Power Mac G5s—the world’s third fastest supercomputer. With unparalleled performance and manageability in a 1U rack-mount enclosure, Xserve G5 is the perfect server for today’s UNIX-based applications.

Xserve G5 includes a new system controller with up to 8GB of PC3200 error correcting code (ECC) memory; three hot-plug Serial ATA drive modules that deliver up to 750GB of storage; optional internal hardware RAID; dual PCI-X slots, supporting 133 MHz PCI-X cards with over 1 GBps of throughput; and dual on-board Gigabit Ethernet for high-performance networking.

The press release continues:

“Xserve G5 delivers just what our customers have been asking for,” said Philip Schiller, Apple’s senior vice president of Worldwide Product Marketing. “With dual G5 processing power and up to 8GB of ECC memory, Xserve G5 is affordable, easy to manage and easy to cluster. Starting as low as $2,999, no one can offer better price performance in a 1U rack-mount design.”

Available with either single or dual 2.0 GHz PowerPC G5 processors, the Xserve G5 architecture is based on an execution core that features massively parallel computation for an unprecedented 215 in-flight instructions, full symmetric multi-processing, two double precision floating-point units and an optimized Velocity Engine. Xserve G5 has the industry’s fastest front side processor bus—running at up to 1.0 GHz and delivering 8 GBps per processor.

Building on the engineering feats first introduced in the Power Mac G5, Xserve G5 squeezes the incredible performance of dual PowerPC G5 processors into an ultra-thin 1U enclosure. An innovative copper heat sink design effectively removes heat from the processors while eight high-performance fans provide airflow for cooling the system. The fans are individually managed and monitored by a dedicated fan control processor; if a single fan fails the others speed up to compensate. The 1U enclosure has an array of over 30 sensors, including eight dedicated to temperature control, that continually monitor all critical system functions throughout the server.

Tech tore down the Power Mac version of System X in January 2004 and rebuilt it with Xserve G5s that summer. ECC memory was a big part of the reason. As the school put it: “We needed to upgrade the system to something with error correcting code (ECC) RAM. The Power Macs did not support it and the XServes were coming.”

The new array was made up of 1,100 dual 2.3 GHz Xserve G5 servers and six Xserve RAIDs. Those 2.3 GHz machines were custom-built by Apple for Virginia Tech; the public wouldn’t be able to buy one until January 2005. Each one had 4 GB of that sweet, sweet ECC RAM and an 80 GB Serial ATA drive.

The installation photos really show the difference between the G5 towers and G5 servers when it comes to density:

The aforementioned FAQ sheds some light on this version of System X:

The initial machine was torn down by eight people between January 14 and January 19, 2004.

The Xserve upgrade was completed by a team of Virginia Tech and Apple volunteers over a much more staggered period of time. Rack preparation and pre-cabling of Ethernet began in February and was completed by less than ten people.

Machines arrived in odd batches beginning in the middle of June 2004 and were processed into the racks by no more than 16 people working for no more than 9 hours a shift (first shipment; 304 Xserves racked in 9 hours by a 16 person team).

Final cabling was completed over a two-week period by no more than ten people working a shift. The machine was back online for benchmarking by August 1, 2004.

The new hardware pushed System X to 12.25 teraflops. The school boasted that “the upgraded version was #7 in the 2004 TOP500 list, and cost one-fifth as much as the second least-expensive system in the top 10.”

Apple boasted about it as well, profiling the machine on its website:

Virginia Tech gained international honors for building the fastest supercomputer at any academic institution in the world. And they did it using Power Mac G5s. From that milestone, Virginia Tech has moved ahead again with System X, a new cluster using 1,100 Xserve G5s.

[…]

“When we built the first supercomputer from Power Mac G5s, we proved that a radically-different communications technology could be used to create a large-scale computing platform,” says Dr. Srinidhi Varadarajan, director of the university’s Terascale Computing Facility and the system’s lead designer.

“We’re recognized for knowing how to build these terascale computing machines — and with very good price performance.

“But the measure of the facility is the kind of science it produces.”

Varadarajan later became CTO of AppAssure, a backup software company that Dell acquired in 2012. May we all be as cool as him one day:

Tech Goes Intel

By the middle of 2007, System X had slipped to 71st place on the TOP500 list, but was still serving the community in and around Tech.

In November 2008, its younger sibling System G was brought online. Firmly in the Intel era, System G was made up of 324 Mac Pros,⁴ each with two 4-core 2.8 GHz Intel Xeon processors (for a total of 2,592 processor cores) and 8 GB of RAM.

It debuted at #280 on the November 2008 TOP500 list with a Linpack score of 16.87 teraflops, though Virginia Tech claimed 22.8 teraflops of sustained performance.

Chris Foresman wrote about the supercomputer for Ars Technica:

Virginia Tech’s Center for High-End Computing System [sic] (CHECS) is famous for building a 1100-node, G5-based supercomputing cluster in 2003 called System X. At the time it was built, it ranked as the third fastest supercomputer in the world with a sustained performance of just over 12 teraflops. The CHECS is building a new cluster, this time using 324 Mac Pro towers, that will have a theoretical computing capacity of 29 teraflops.

The new system will not replace System X, according to CHECS Director Dr. Srinidhi Varadarajan. “Unlike System X, which is a production machine for computational science and engineering researchers, the new system is intended to be pure computer science research system,” he told Ars. The system will be used to study two main areas: power-aware software systems that can adjust performance automatically to maximize efficiency, and distributed shared memory systems that can run existing threaded code on high-performance clusters “as if they were scaled up versions of multi-core desktops.”

The article continues and explains why the Intel Xserve wasn’t chosen for the new array:

Varadarajan and his team chose Mac Pros for a few reasons. For one, Xserves are power dense, which requires a lot of cooling to prevent thermal hot spots. “System X has a hybrid liquid-air cooling system in a dedicated facility, which enables significantly better cooling without hot spots. This machine is however, housed in a conventional air cooled room,” Varadarajan told us. Also, the Mac Pro has a large number of power and temperature sensors built in, which is necessary for the power/performance research and tuning, as well as an additional full-height PCI Express 2.0 slot for use in other research projects. And, despite common wisdom to the contrary, the Mac Pros beat other systems on price. “The Mac Pros are highly competitive even against building a white box off the cheapest prices,” he added.

System G was designed to be as power-efficient as possible. Here’s a bit from an old CHECS website:

The system is the first supercomputer running over quad data rate (QDR) InfiniBand (40Gbs) interconnect technology. System G (for “green”) also has unique power-aware capabilities, with thousands of power and thermal sensors allowing CHECS researchers to design and develop algorithms and systems software that achieve high-performance with modest power requirements, and to test such systems at unprecedented scale. With a sustained (Linpack) performance of 22.8 TFlops, System G is the largest power-aware research system and one of the largest computer science systems research clusters in the world.

This VT Magazine article has more:

“It’s twice as powerful, a third the size, and uses a third of the power,” says Kirk W. Cameron, associate professor of computer science in the College of Engineering. Cameron helped build G with Srinidhi Varadarajan, associate professor of computer science and director of the Center for High-End Computing Systems.

Unlike System X, System G was created to improve system design first and foremost and to be used as a production computer second. The $1.2 million machine was built to develop new, high-performance software tools and applications with paramount efficiency.

When Cameron and Varadarajan, assisted by a team of fellow faculty members and scores of graduate students, set out to build G, they had an end-goal in mind. They wanted G to not only be fast but also environmentally friendly. Hence the “G” for green. When Cameron arrived at Virginia Tech in 2004–on the heels of X’s opening, and attracted to Tech because of X–“green computing” wasn’t even an industry catchphrase. But Cameron already was looking ahead at the need to conserve power.

He previously worked for Los Alamos National Laboratory, handling performance duties, “squeezing as much power as possible” out of the facilities’ supercomputers, he says. This was in 2002, well before the recent call to arms in green energy. G went on-line in 2008. “Now everybody says they are doing it, but we were lucky enough to have done it first,” Cameron says.

The system is stocked with advanced power-management capabilities, such as power-aware central processing units (CPUs), disks, and memory. A network of advanced power and thermal sensors runs throughout the entire system, with Cameron and Varadarajan able to track how much power each CPU unit in each Apple tower is using.

Using power caps, the system is able to run on lower steam when not in use or in partial use, as opposed to older supercomputer systems that run at full blast, needlessly using power, on a continuous basis. Temperature control is carefully monitored as to not overuse the massive air-conditioning units that keep the stacked, side-by-side computers from overheating.

“We wanted to try and find the most efficient way to use computers,” says Cameron.

While System X served researchers across the university and was eventually run by Virginia Tech’s Advanced Research Computing division, System G was a computer science research machine. It belonged to CHECS, was sponsored in part by the National Science Foundation (NSF), and was housed on the campus of VT’s Corporate Research Center.

That makes finding additional details about System G tricky. Even locating photos of the array proved futile; the ones Virginia Tech published in 2008 have vanished from the web.

That’s when Dr. Cameron answered an email I had sent him, asking about System G and, if possible, for any photos he had of the Mac Pros. He graciously replied with first-hand information about the project, including a couple of incredibly rare photos:

These are the only images of System G I have seen, and I cannot imagine how amazing this must have been in person.

They also prove yet again that guys with racks and racks of Mac hardware are just much cooler than the rest of us:

He also provided insight into how the project was designed to make it easier to capture and analyze huge amounts of data. The team captured data from the Mac Pros’ built-in temperature sensors, recorded the voltage and current going into the racks, and logged environmental data from the room itself.

All of this data was synthesized into human-readable information that helped inform their research and keep System G as efficient as possible:

The Legacies of System X & System G

Both systems are now gone, but their legacy remains. Choosing Mac hardware to build a supercomputer isn’t the status quo, but the work of Varadarajan, Cameron, and others proved it could be done, and on a budget.

System X may be the most famous Mac-based supercomputer, but it wasn’t the only one. In January 2005, Bowie State University in Maryland unveiled XSEED, a $1 million array of 224 Xserve G5s. In the same timeframe, the University of Illinois at Urbana-Champaign launched its Turing Xserve Cluster, made up of 640 dual-processor 2.0 GHz Xserve G5s. The University of California, Los Angeles had Dawson, which started as an array of 128 Xserve G5s.

System G’s legacy includes its focus on power usage, under the direction of Dr. Cameron. His work has centered on the power consumption of high-performance computing systems since the early 2000s. He is a founding member of SPECpower, helped create the Green500 list, and has worked to steer the supercomputer industry toward treating performance per watt as an important metric when architecting systems.

He is still a faculty member at Virginia Tech today.

---

My research into System X started in July, and what was going to be a simple blog post turned into something much bigger and much more special. I had no idea System G existed when I started this, and I am thrilled I could help tell its story. My thanks to everyone who helped make this article possible, including Dr. Cameron, Sam Davies, Casey Liss, Jason Snell, and others.