Hybrid Scaling: Scale-Up, Scale-Out, and Scale-Across Gain New Meaning

Getting your Trinity Audio player ready...

Confused? Let’s start at the beginning….

What was the original idea behind scale-up vs. scale-out?

Originating in the IT world, scale-up and scale-out are two means to tackle a simple problem: the (imminent) lack of resources available to accommodate a specific workload requirement.

Whether this lack is in compute power, memory, network bandwidth or other, you can choose one of two options:

1. Scale-up: add resources to the current machine serving the workload (either by upgrading the machine or switching the workload to a stronger one)
2. Scale-out: add machine/s and distribute the workloads among the group of machines (with some kind of parallelism)

While scale-up is usually the preferred option, because it yields better performance and reduces complexity, it is limited in, well, scale.

The scale-up vs. scale-out tradeoff hits the AI wall

Scale-up keeps the workload on a single machine, leveraging the unique benefits of intra-machine communications, like very low latency and lossless connectivity.

Once you go out of the box, you are dependent on a network. With a network (any type of network, but Ethernet, in particular) comes complexity and performance degradation (e.g., packet loss, jitter etc.) – not to mention that introducing workload parallelism (of any kind) makes thing even more complex.

On the other hand, you can scale up only so much. There is a limit to the scale of a server, or a chassis, if only because of physical limitations.

When it comes to AI (and HPC, in general), parallelism is inherent, so you would expect scale-out to be the default choice. But AI also requires very high performance, which means that the performance compromise of scaling out is not acceptable.

So how do we make scale-out more like scale-up, and vice versa?

We can do two things – enhance the scale of scale-up systems and/or enhance the performance of scale-out systems.

What is the new scale-up?

In order to improve the scale of a scale-up solution, we need to get rid of the main limiting factor – the chassis. Once that’s done, we can make a multi-box solution a scale-up system and manage it as if it were one happy and very large box.

For that, there are two approaches in the industry:

1. The Nvidia way: This approach takes a scale-up communications technology out of the box; here, we take the very robust, low-latency technology used for intra-server communications (in this case – NVLink) and stretch it out of the box. Nvidia did it with their NVSwitch, reaching a 72xGPU scale-up system in the current generation (NVL72-Oberon), and promises to do it for a 576xGPU scale-up system next year (Rubin-based NVL576-Kyber).
   While still limited in scale, it is much less so than the classic scale-up technologies.
2. The Broadcom way: This approach takes a scale-out communications technology, adopts it to the requirements of scale-up ,and brings it into the box. This is what Broadcom did with Tomahawk Ultra, adding very low latency, lossless communications capabilities and a range of in-network collectives to allow the Ethernet-based Tomahawk chipset to perform as well as the NVLink.

The end result is, in any case, a new definition of scale-up – no longer a single-box upgrade, but a rack and even data center-wide upgrade.

What is the new scale-out?

While the new scale-up is an exciting development, you still need, in many cases, a complementary scale-out solution. 72, and even 576, GPUs are far from enough for many modern AI use cases.

In short, there needs to be a way to make a scale-out network perform as well as a scale-up network.

And there is. The solution is a scale-up networking architecture that acts as a scale-out network.

Confused? Don’t be…

When we use Ethernet as a scale-out network, our main issue is performance. When we are connected to a single Ethernet box (say, a chassis), this is not an issue because the internal fabric of a chassis is non-blocking and lossless. But when we need to go to a multiple-switch solution (using Clos architecture), we encounter packet loss, jitter, and high tail latency – so we miss our performance target.

The solution here is very similar to the one used by Nvidia to improve scale-up scale – take the internal, robust, communications protocol and use it outside the box. In this case, this is the cell-based fabric used in a backplane of a chassis. If taken and used outside the chassis, it can enhance the scale-out network performance to the required level.

This is exactly what is done in technologies like DDC (Distributed Disaggregated Chassis), FSE (Fabric-Scheduled Ethernet), and DSF (Distributed Switching Fabric). Any of these technologies (that are basically the same) can perform at the level required for AI workloads, with a scale-out network that employs a scale-up networking architecture.

The new frontier – scale-across

But what happens if even scale-out is not enough? A high-performance scale-out network is usually confined within the data center. But workloads sometimes need to span across multiple data centers (usually because one data center does not have enough power to support all of the needed compute resources).

Here, a scale-across network is required, enabling an intra-data center performance-level across multiple data centers. Once again, the fact that we can scale up the networking infrastructure across multiple boxes provides clear benefits; for instance, we can mix the shallow-buffer ASICs required for intra-data center comms with deep buffers required for data center interconnect (DCI), all within the same scale-up infrastructure. This makes the DCI robust enough to handle scale-across needs (read more about it here).

Curious to dive deeper into this topic? You can hear more about it at the AI Infra Summit panel, “Transformative Advancements in Scale-Up, Scale-Out, Scale-In, and Scale-Across Networks,” which will take place in Santa Clara on September 10th at 2:40 pm.

Frequently Asked Questions

Key Takeaways

• Scale-up vs. scale-out tradeoff: AI workloads need both parallelism (scale-out) and ultra-low latency (scale-up), making traditional approaches insufficient.
• Distributed fabrics (DDC, FSE, DSF): Bring chassis-level performance to scale-out Ethernet, solving packet loss and jitter at scale.
• Scale-across networks: Allow workloads to span multiple data centers while maintaining near–intra-data center performance.
• Redefinition of scaling: “Scale-up,” “scale-out,” and “scale-across” now blend together, creating flexible architectures designed for modern AI demands.

Related content for AI networking infrastructure

DriveNets AI Networking Solution

Latest Resources on AI Networking: Videos, White Papers, etc

Recent AI Networking blog posts from DriveNets AI networking infrastructure experts