PCIe is just one of the many technologies that continue to evolve to handle large amounts of data. GPUs, storage devices, and networking equipment are the components that utilize PCIe slots with the most number of lanes for faster data transmission and higher bandwidth. These devices are usually inserted to x16 PCIe or x8 PCIe slots which have a direct connection to the PCIe lanes of the CPU or SoC.
However, even the PCIe lanes of such host devices are limited and are not flexible enough to handle multiple devices. Insert a PCIe switch and data processing becomes more dynamic.
PCIe Switches
PCIe switches are devices that expand the number of PCIe lanes than what is available from a host device so that more devices can be supported by the host. For example, a PCIe switch can double the PCIe lanes of a x16 CPU to support more GPUs than what the CPU alone can handle. Some switches can even expand the lanes to hundreds of ports. This will allow more devices to be connected, overcoming the CPUs limited number of PCIe lanes.
PCIe switches have one upstream port that connects to the host, several downstream ports that connect to the devices, and a switching logic that routes the data packets between the ports. Each downstream port is assigned a set of PCIe lanes and each port is connected to a device so the bandwidth is not shared among the devices; instead, data packets will only pass through the lanes of the device that requested it.
PCIe switches don’t require any special software or driver to work. The switches can be configured via a management software that manufacturers develop for easier deployment, configuration, and monitoring of the devices. There is also no need to install drivers for the devices connected to the PCIe switch unless otherwise required by the operating system. In Windows for example, the device added to the switch will be automatically detected and the driver will be automatically installed. In fact, you will instantly find the device under the Device Manager.
Other functions of a PCIe switch include interrupt handling, configuration access, power management and error reporting all of which can be monitored from the management software. The PCIe switch’s performance depends on the PCIe version it is built with. Newer PCIe versions mean half the number of lanes than the previous version for the same bandwidth but at a faster transmission rate. Just like with any PCIe device, it is paramount for PCIe switches to keep abreast with PCIe iterations to achieve peak performance.
Types of PCIe Switches
There are two types of PCIe switch – fan-out switch and fabric switch. Each switch handles data traffics differently. Both types support port bifurcation to efficiently utilize the bandwidth. With port bifurcation, the upstream port is split into smaller ports so the other ports can be used to connect to other peripherals to fully optimize the PCIe lanes and the bandwidth. Since most GPUs can work with x8 lanes without compromising their performance, the x16 lanes provided by the CPU are divided into smaller ports like two x8 ports or four x4 ports so that the other ports can be utilized by other devices.
Modern switches offer flexibility in handling port bifurcation. Instead of a fixed number of lanes for each port, the lanes can be expanded or reduced according to what a device needs for a specific workload. This will activate idle ports while there are still no requests from other devices and provide the active device with more PCIe lanes and thus higher bandwidth.
Fan-out switches have been widely used because of their simple architecture but the fabric type is gaining popularity as well because of its versatility in handling multiple hosts.
Fanout PCIe Switch
The fanout PCIe switch topology is much simpler than the Fabric switch’s topology. The PCIe lanes are multiplied and then divided into separate sets of lanes for different devices. There’s only one upstream port for the host device. Because data packets follow a straightforward path from the host to the destination, fanout switches can maintain signal integrity. In addition, fanout switches are generally easier to deploy than their fabric counterparts since they do not require a lot of settings. Fanout switches can however, only support one host device.
Fabric PCIe Switch
The fabric PCIe switch is more complex than the fanout switch but it is more versatile and flexible. Not only can it support multiple devices, it can also accept multiple hosts. The hosts are bridged to the same devices so whichever host is ‘not busy’ at the moment, can accept requests from the devices so the devices need not wait for the other host to be available before their requests can be processed. If multiple hosts are fabricated, a multitude of devices and systems can be interconnected helping organizations save cost by eliminating the need to buy more computer systems or go through costly upgrades to support numerous devices.
Conclusion
A PCIe switch basically functions as an I/O controller and magnifies the host devices’ capabilities to support more devices. Because of their high-performance, low-latency, and low power consumption, PCIe switches have become a core component of computer systems across industries. PCIe switches are typically used in defense, finance, healthcare, industrial, and enterprise servers and workstations, as well as in various test equipment, video production equipment, data center equipment, telecom infrastructures, networking, and other connectivity applications. PCIe switches are the go-to components of system designers for interconnecting multiple systems, numerous devices, and other peripherals. They can choose between the two types of PCIe switches to effectively implement their designs while efficiently saving cost.
