IR4 Leaders

NVIDIA technical article 003

ConnectX-9 SuperNIC

How an 800 Gb/s network endpoint moves, schedules and protects AI traffic between a GPU server and an Ethernet or InfiniBand fabric.

01

What it is

ConnectX-9 is a network interface processor for accelerated servers. It sits between the host’s PCI Express bus and an external Ethernet or InfiniBand network, moving data at up to 800 Gb/s per port while applying transport, scheduling, congestion-control, isolation and security functions.

A conventional network card mainly transfers packets. A SuperNIC also understands communication patterns used by distributed AI systems. It can move data directly between network and accelerator memory, execute RDMA operations and coordinate traffic with the wider fabric. It does not replace a switch: ConnectX-9 is the server endpoint; Spectrum-X or Quantum-X switches form the network between endpoints.

Host interfacePCIe Gen 6 x16CPU and accelerator memory
Network protocolsEthernet / InfiniBandOne endpoint family, configurable fabric
Maximum port rate800 Gb/sSupported single-port configurations
System roleScale outServer, rack and POD connectivity
ConnectX-9 position in an AI serverGPU and CPU memory connect over PCI Express to ConnectX-9, which connects to an Ethernet or InfiniBand fabric. SERVER MEMORYGPU HBMCPU system memory PCIe Gen 6 CONNECTX-9Transport + RDMAScheduling, security 800 Gb/s SCALE-OUT FABRICEthernet or InfiniBandExternal switched network
ConnectX-9 is the controlled boundary between server memory and the scale-out network.
VideoThe AI Entrepreneurs · Why NVIDIA built AI EthernetYouTube ↗
02

Why a SuperNIC exists

Distributed training and inference generate synchronised, high-volume transfers. Many GPUs can start a collective operation together, creating short bursts that ordinary host networking may process too slowly or unevenly.

Workload problemWhy it mattersConnectX-9 response
CPU-mediated copiesExtra memory copies add latency and consume CPU cycles.RDMA and GPUDirect can transfer data without routing the payload through normal CPU copy paths.
Synchronised burstsCollective traffic can fill switch queues and delay every participating GPU.Telemetry, programmable congestion control and traffic shaping regulate injection into the fabric.
Shared infrastructureOne tenant or flow can interfere with another.Scheduling, rate controls and isolation policies divide network resources.
Protocol processingTransport and security work can become a host bottleneck.Hardware engines offload transport, encryption and packet processing.
03

The data path

ConnectX-9 transmit and receive data pathA work request identifies memory, ConnectX-9 fetches data over PCI Express, processes and schedules it, transmits it to the fabric, then reverses the path for received data. 1 · WORK REQUESTCPU or GPUMemory address,length, destination 2 · MEMORYPCIe Gen 6DMA reads or writeshost / accelerator 3 · SUPERNICTransport enginePacket processingSecuritySchedulingCongestion feedback 4 · PORTOSFP / QSFP112Electrical oroptical link 5 · FABRICEthernet orInfiniBandRemote endpoint
The receive path reverses the sequence: the SuperNIC validates, places and reports received data to the destination queue.
04

How a transfer works

  1. Post a request.Software identifies a registered memory region, transfer type and remote destination.
  2. Fetch the payload.The adapter uses DMA over PCIe to read CPU or accelerator memory.
  3. Build the transport.Hardware creates Ethernet or InfiniBand packets and maintains the required transport state.
  4. Apply policy.Security, isolation, rate control, prioritisation and congestion rules are applied before transmission.
  5. Cross the fabric.The port serialises the traffic onto an optical or electrical link towards a switch.
  6. Complete the operation.The remote endpoint places data into registered memory and reports completion to software.
05

Hardware variants

AdapterNetwork portsHost interfaceTypical fit
C91801 × OSFP; Ethernet or InfiniBand; up to 800 Gb/s.PCIe Gen 6 x16; optional auxiliary x16 in supported configurations.Single-port HHHL servers requiring the full port rate.
C92402 × QSFP112; up to 400 Gb/s per port in the documented configuration.PCIe Gen 6 x16 with Socket Direct support.Dual-port HHHL servers, redundancy or split-fabric designs.
OCP 3.0 variantsPort configuration depends on the ordering part number.OCP 3.0 host interface.Servers designed around the OCP network-card mechanical standard.
Ordering matters. Port count, protocol, connector, auxiliary PCIe interface and cryptographic capabilities vary by ordering part number. “ConnectX-9” alone is not a complete hardware specification.
06

Principal specifications

AreaPublished specificationInterpretation
Network protocolsEthernet and InfiniBand.One processor family supports either scale-out fabric; the enabled mode depends on product and configuration.
Maximum network rateUp to 800 Gb/s per port.This is a line-rate ceiling, not guaranteed application throughput.
Host interfacePCIe Gen 6.0, 64 GT/s, up to x16.The host link must sustain memory traffic in both directions.
ConnectorsOSFP on C9180; dual QSFP112 on C9240.The connector determines compatible modules and cable assemblies.
Form factorHHHL add-in card; OCP 3.0 variants also documented.Mechanical and cooling compatibility must be checked against the server.
Memory accessRDMA, GPUDirect RDMA and DMA engines.Payloads can bypass conventional CPU copy paths when the platform supports it.
SecuritySecure boot, attestation and hardware cryptographic offloads; feature set varies.Protects adapter firmware and selected traffic without executing every operation on the host CPU.
NVL72 integrationEight ConnectX-9 boards per compute tray; NVIDIA states up to 1.6 Tb/s throughput per GPU.The per-GPU figure is aggregated system connectivity, not one 1.6 Tb/s physical port.
07

Mechanisms that affect real performance

MechanismFunctionWhy it matters
RDMA and GPUDirectMove registered data directly between local and remote memory.Reduces host copies and CPU intervention.
Programmable congestion controlUses network feedback to adjust transmission.Limits queue growth and synchronisation delays during collective bursts.
Traffic shaping and schedulingControls rates, priorities and transmission order.Prevents competing flows from consuming an uncontrolled share of the link.
Isolation and telemetrySeparates workloads and reports network state.Supports multi-tenant operation and closed-loop fabric control.
IPsec and PSP offloadEncrypts and authenticates supported traffic in hardware.Reduces the CPU cost of protected transport.
SHARP for InfiniBandMoves selected collective operations into the network.Reduces repeated endpoint processing and network transfers.
Socket DirectAdds a second PCIe connection to improve locality in multi-socket systems.Can reduce traffic crossing CPU socket interconnects.
08

Terms used in this article

NIC
A server’s network interface controller.
SuperNIC
A NIC with transport, scheduling and control functions designed for accelerated systems.
RDMA
Direct memory transfer over a network with limited CPU handling.
DMA
Hardware transfer between an I/O device and system memory.
HHHL
Half-height, half-length PCIe add-in-card format.
OSFP
A high-density pluggable interface used for high-rate network links.
QSFP112
A four-lane pluggable interface using 112 Gb/s-class signalling per lane.
Collective
A coordinated operation involving a group of processors.
09

Limits of the evidence

  • Feature availability depends on adapter part number, firmware, transceiver, cable and server integration.
  • “Up to 800 Gb/s” is a physical port rate; application throughput is lower and workload-dependent.
  • The NVL72 figure of 1.6 Tb/s per GPU is aggregate system connectivity, not a single-port specification.
  • Published capabilities and performance data currently come primarily from NVIDIA.
  • The embedded video is third-party commentary, not a product specification source.
  • Price and promotional performance comparisons are excluded.
10

Sources

  1. NVIDIA ConnectX-9 SuperNICOfficial product scope, protocols and headline interfaces.
  2. ConnectX-9 SuperNIC datasheetFunctions, offloads, security and form-factor summary.
  3. ConnectX-9 hardware manual: introductionAdapter variants, ports, protocols and host interfaces.
  4. ConnectX-9 hardware specificationsMechanical, electrical and environmental specifications.
  5. ConnectX-9 firmware release notes, v82.48.1000Current firmware capabilities, changes and known limitations.
  6. Inside the NVIDIA Vera Rubin PlatformConnectX-9 integration and scale-out role inside NVL72.
  7. Why NVIDIA built AI EthernetThe AI Entrepreneurs video supplied for this article.