The most expensive idle asset is a GPU waiting on the network

Getting your Trinity Audio player ready...

But here’s the paradox: even with the most powerful chips ever built, real-world AI infrastructure efficiency – measured by GPU utilization, job completion time, and cost per million tokens – is increasingly dictated (or limited) by the network. As AI clusters scale across tens-of-thousands of GPUs and specialized processors, performance depends less on the peak capability of any single accelerator and more on whether the cluster can keep every resource fully utilized.
This is why, in our latest funding round, I said that the most expensive idle asset in the world right now is a GPU waiting on the network. In large clusters, collective communications depend on fast, predictable data movement. Any network weakness reduces GPU utilization and extends job completion time. This is the true cost of AI infrastructure: not just the cluster you’re buying, but …the performance you lose when GPUs sit idle.

Download

DriveNets AMD System Reference Architecture

Problem #1: The Networking Bottleneck

Compute is scaling faster than communication.

Every additional GPU adds processing power…but also increases the volume of data moving across the cluster. AI workloads depend on collective communication and many-to-many traffic. As a result, the network is no longer just connectivity infrastructure. It directly determines how fast the AI workload can run.

In large AI clusters, the network is the computer: it directly determines how fast the AI workload can run. Even small delays can leave expensive GPUs idle.

During gradient sync, for example, the next training step cannot begin until all required updates have been exchanged. A delayed packet can hold back the collective operation – leaving a large cluster waiting.

This is why tail latency matters.

Average performance is not enough. The slowest flows often determine job completion time and GPU utilization. Reducing tail latency is not just a networking improvement, but directly lowers infrastructure cost.

Problem #2: Complexity and reliability challenges impact deployment time

All enterprises and AI developers are in a frantic sprint to deploy AI capabilities before their rivals. The rapid deployment of AI GPU clusters is a critical demand. And while getting the hardware in place is one thing, it’s the network that often causes delays.

Every idle moment during provisioning, validation, integration, or troubleshooting burns capital. In large AI clusters, even small delays in making resources available can translate into lost productivity and slower time-to-market.

High-performance GPUs only create value when they actively process workloads. If the network is not properly coordinated, GPUs wait for data, utilization drops, job completion time (JCT) increases, and cost per million tokens (CPMT) rises. That’s a key element of the Token Economy.

This is why CPMT became the new TCO for AI infrastructure.

While traditional TCO measures the costs to buy and operate clusters. CPMT measures how efficiently that cluster converts infrastructure spend into AI output. Two clusters with similar GPU counts and theoretical FLOPS can deliver very different token throughput due to …networking performance.

Heterogeneous AI Raises the Stakes

In a heterogeneous AI environment, optimization must span not only each accelerator type, but also the way different accelerators work together. GPUs, CPUs, XPUs, custom AI accelerators, and specialized silicon may each handle different stages the AI workload, based on its computational characteristics – sequential decision making, memory-bound, compute-bound, etc. By assigning each stage to the most efficient processing resource, you can improve CPMT.

Turning this collection of different accelerators into a single coordinated AI system depends heavily…on the network.

Data must move efficiently between different compute domains, or the gains from specialization are lost to latency, congestion, and idle (expensive) GPUs.

The tradeoff is complexity.

Running a single application across heterogeneous infrastructure requires abstraction, scheduling, and orchestration layers that can map each workload stage to the right resource and keep accelerators utilized.

What turns specialized compute islands into one functioning AI system?

The network.

The DriveNets Solution: Open, Full-Stack AI Networking

Networking is as critical to AI performance as GPUs. Jensen Huang was the first to say it, and this year Nvidia became the largest networking company in the world.

DriveNets addresses this with a Scheduled Ethernet-based AI fabric. We deliver high-performance, low-latency, reliable connectivity across large GPU clusters, supporting scale-up, scale-out, and scale-across architectures, along with front-end and storage connectivity. We also optimize the AI stack end-to-end, from collective communication libraries and transport protocols to NICs, the network fabric, and system-level orchestration. Maximizing GPU utilization.

AMD didn’t just partner with us as a networking vendor. AMD collaborated with an open alternative to single-vendor AI infrastructure. Together, DriveNets and AMD optimized AMD-based AI clusters across ROCm, RCCL, scale-up and scale-out networking, as well as the integration between the infrastructure stack and the AI model.

We’ve already published a reference architecture for AMD-based AI clusters, demonstrating how full-stack optimization can improve token economics in open, multi-vendor environments.
DriveNets is now extending this to additional AI accelerator partners, as the market moves to open, heterogeneous AI infrastructure.

Why DriveNets AI networking?

With more than $1B in secured business, DriveNets is looking to support its growing AI fabric pipeline and expand our Heterogeneous AI solution.

Initially building large-scale networking solutions for the world’s leading service providers, we took the cloud-scale principles to high-performance networks. That same foundation now comes to AI infrastructure, where GPU clusters require massive scale, resilience, and operational efficiency. And we work with many AI accelerator vendors to maximize token economics.

It is not a networking game, but an AI infrastructure utilization one – every percentage point of utilization translates into hundreds of millions of dollars.

The networking choice for Heterogeneous AI

It’s the end of the “one-vendor” era.

Shifting from training to inference requires multi-vendor (Heterogeneous) clusters. Each compute element (GPU, CPU, XPU) is optimized for a different stage in the process.

DriveNets AI fabric is uniquely positioned to support heterogeneous multi-vendor AI environments. Our ability to perform full-stack optimization for different AI accelerators in the cluster, maximizes the performance and utilization of the entire cluster.

DriveNets’ high-performance AI Fabric eliminates networking bottlenecks through end-to-end networking optimization across the entire AI stack, including collective communication libraries, transport protocols, NICs, the network fabric, and system-level orchestration.

Meeting the rising demand for open, multi-vendor, and Heterogeneous AI infrastructure

Our latest financing round marks a pivotal step in scaling our company to meet the surging demand for large-scale AI infrastructure. The most expensive idle asset in the world right now is a GPU waiting on the network. Solving that, is what we’re applying a decade of high-performance networking expertise to, so our customers can achieve higher AI compute utilization and lower their cost per workload – on any AI accelerator they choose.

Key Takeaways

Networking is as critical to AI performance as GPUs
While immense capital is poured into buying the world’s best hardware, any weakness in the network directly reduces GPU utilization and extends job completion times. Therefore, the true cost of AI infrastructure is not the hardware itself, but the massive performance lost when the network leaves these elite chips sitting completely idle.
The Scaling Bottleneck
Compute capability is scaling faster than communication infrastructure. While adding more GPUs increases raw processing power, it also increases the volume of many-to-many traffic, making the network the definitive factor in how fast an AI workload can run.
The Role of Tail Latency
In large AI clusters, even tiny network delays can stall the entire system. During collective operations like gradient synchronization, the next training step cannot begin until all updates are exchanged, meaning a single delayed packet can leave an entire cluster of expensive GPUs completely idle.
Network Dependency Across Heterogeneous AI Architectures
In an environment featuring a mix of different GPUs and specialized processors, overall cluster performance depends less on the standalone peak capability of any single accelerator. Instead, the ultimate success of this heterogeneous, mixed-silicon infrastructure relies entirely on whether the network can efficiently move data to keep every diverse resource fully utilized.

Frequently Asked Questions

Why is a GPU waiting on the network considered the most expensive idle asset?

As AI clusters scale across tens-of-thousands of GPUs, infrastructure efficiency depends heavily on predictable data movement. Network weaknesses directly reduce GPU utilization and extend job completion times. When powerful accelerators sit idle waiting for data, it dramatically inflates the cost per million tokens and degrades collective cluster performance.

What causes the networking bottleneck in large-scale AI infrastructure?

The networking bottleneck occurs because compute capabilities are scaling faster than communication infrastructure. Every additional GPU increases the volume of many-to-many traffic moving across the cluster. Consequently, the network ceases to be mere connectivity, directly determining how fast collective AI workloads can run and causing expensive idle periods.

How does tail latency affect AI training during gradient synchronization?

Tail latency impacts AI training because the next step cannot begin until all required updates are completely exchanged. During gradient synchronization, a single delayed packet holds back the entire collective operation. This leaves large clusters of expensive GPUs completely idle, proving that network latency dictates overall workload speed.

The most expensive idle asset in the world right now is a GPU waiting on the network