Overview of PCI in Linux

The above figure shows the PCI system, which has 3 PCI buses. Bus no 0 is the primary bus of the System as the CPU is connected to that bus; also, it is the bus where the root port bridge or the root complex is present.

Other buses, i.e., bus no 1 and 2, are connected to the primary bus with the help of PCI bridges. Bus no 1 is connected to bus no 0 with bridge 1. Bus no 2 is connected to bus no 1 with Bridge no 2. Overall, all the devices are connected, and several devices D1, D2, D3, etc., are present on different PCI buses. On any PCI system, 3 types of devices are present. Root Port or Complex device, Bridge device, and Endpoint devices. Comparing types of devices with our example diagram, CPU is the root port or complex device. Bridge 1, Bridge 2 are the PCI bridge devices. D1, D2, D3, etc., are the PCI endpoint devices of the system. D3 is present on bus no 2 and bus no 3, the same device on different buses.

PCI Config Space or Header:

All the PCI devices have the config space or header. This is the standard memory area present in all the devices. There are two types of PCI config header, based on the two types (Bridge and Endpoint) of PCI devices. Config space is known as Type 0 for Endpoint device and Type 1 for PCI Bridges. Fields of the config header are PCI specifications defined.

Type 0 config header:

Type 1 config header:

PCI Bus Enumeration:

During the Boot-up of the system, recognizing all the PCI devices in the system is done and is known as the PCI Bus enumeration. BIOS generally enumerates all the PCI devices present on all the buses and populates them to the sysfs. Users can access the details of PCI devices present with the help of the lspci utility. Another way is to browse through the sysfs files inside the /sys/bus/pci/devices directory. This directory will have all the devices present and known to the Linux kernel.

After the PCI Bus enumeration, all the devices get the number, number, and function number. These three components are enough to locate any device.

PCI Bus enumeration is performed by BIOS(Basic Input Output System). BIOS is a firmware software specific to the Machine/platform and provided by the manufacturer itself.

Linux Kernel Root Complex driver:

On any x86 based Linux platform, there is a root complex PCI driver or Linux PCI subsystem which reads the information populated by BIOS and exports the information to the sysfs filesystem. All the PCI devices present in the system can be found inside the /sys/bus/pci/devices directory. Root complex driver also provides the flexibility to rescan or reset the devices on any PCI Bus. Even full rescanning of all the PCI Buses can be done through /sys/bus/pci/rescan.

Command to rescan all the devices:

Users should have superuser rights for issuing this command.

For any device in the sysfs directory we can find below details/information:

Above are the files present for every device at a different path.

We will read the content of few file to verify the info :

Similarly, other files provide some other information.

Some files are write-only files : remove and rescan

remove files can be used to remove the device. Echo 1 to the file, and you will see lspci will not show this device.

echo 1 > /sys/bus/pci/devices/0000\:00\:00.0/remove

Recovery of the device from the previous step can be done through rescanning the device.

Echo 1 to the rescan file with below command :

echo 1 > /sys/bus/pci/devices/0000\:00\:00.0/rescan

Reading and writing to the config space:

lspci and setpci commands are available, which can be used to read and write the config space of any PCI device. lspci has very rich options to customize the output as per the user’s needs. setpci is another utility that can also be used to access the config space of the pci device.

We will not discuss these in detail here as there is a separate article to cover both of these utilities in detail. We will just have one example of both the commands :

lspci :

setpci:

Reading and writing of BAR Space:

There can be a total of 6 32-bit BAR or 3 64-Bit bar. Type 0 config space can be referred to to get the offset details of the BAR.

Let us take an example of a device with the below output:

Additional files with names resource[0-3] are present; these are the files that can be used to access the memory mapped to these regions. For example, to access the 4K space mapped to region 0, the resource0 file can be mapped to the user space with the mmap() function. After mapping the region0 to the user space, 4K space can be accessed as per the need/requirement.

Conclusion:

Linux PCI subsystem enumerates and populates the PCI devices. lspci and setpci devices can be used to get the information of the devices. Root complex driver of Linux also provides all the pci devices info in the sysfs files. There is a provision to reset, rescan and remove the devices from the sysfs files. BIOS performs the enumeration process, and the Linux driver parses the information and populates all the device’s info accordingly. With this much discussion, let us conclude this topic.