The authors are engineers for Intel and authors of Weaving High Performance Multi-Core Processor Fabric on which this article is based. Copyright (c) 2009 Intel Corporation. All rights reserved.
The architecture of the Intel QuickPath Interconnect (Intel QPI), like many components in a computer system, is comprised of a number layers. Each of these layers performs a distinct function in controlling the logical operation of the Intel QPI. This article introduces the four architectural layers of the Intel QPI -- the physical, link, routing, protocol layers -- and introduces other aspects of the architecture including performance and reliability, as well as some initial design considerations for a system using Intel QPI.
Intel QuickPath Interconnect System Terminology
As we dive into the details of systems, it is important to establish a consistent terminology. Let's take a look at the types of devices that make up such systems.
There is the processing unit that is usually assumed to be connected to the system interconnect through a high performance cache, and appears as a caching agent on Intel QPI. A typical system may have one or more caching agents.
The memory controller is responsible for providing access to the random access memory (RAM) in the system used to store programs and data. A system can have one or more memory controllers where each one covers a unique portion of the total system memory address range. For example, if there are four memory controllers in the system, then each could control one fourth of the entire addressable system memory. The addresses controlled by each controller are unique and are not duplicated on another controller. In Intel QPI terminology, a portion of the memory controller performs the home agent function for that particular range of memory addresses. A system would have at least one home agent per memory controller, possibly more than one.
Some devices in the system are responsible for connecting to the input/output subsystem. They are referred to as I/O agents.
One or more devices are responsible for providing access to the code required for booting up the system These are called firmware agents.
Depending upon the function that a given device is intended to perform, it may contain caching agents, home agents, and/or I/O agents into a single component. In fact multiple agents of each type may exist within a single device. We have been using the term processor (or sometimes socket) to refer to the main processing elements in the system. Processor devices typically will contain at least one home agent, one caching agent, as well as their primary cores and cache structures that perform the actual instruction set execution.
The connection between two Intel QPI devices is termed a link and is composed of a set of unidirectional signals transmitted by one device and received by the other. The individual signals within a link are called lanes. Multiple lanes operating in parallel form the link. A full bidirectional communication pathway between two devices uses two links working in concert with each other. This is sometimes referred to as a link pair.
Layers of the Architecture
Many contemporary communication architectures have been influenced by the Open Systems Interconnection (OSI) abstract seven-layer model of networking. Intel QPI is no exception and has its own layered model. The functions performed by Intel QPI are logically grouped into four different layers, as illustrated in Figure 1. These layers encapsulate similar functions together.
- The Physical Layer is responsible for dealing with details of the operation of the signals on a particular link between two agents. This layer manages data transfer on the signal wires, including the electrical levels, timing aspects, and logical issues involved in sending and receiving each bit of information across the parallel lanes.
- The Link Layer is responsible for handling flits of information, as given to and from the link layer and the physical layer, into complete messages. The link layer also manages the flow of these messages and handles errors that may occur during their transfer on that link.
- The Routing Layer is responsible for ensuring that messages are sent to their proper destinations. If a message handed up from the link layer is destined for an agent in another device, this layer forwards it to the proper link to send it on. All messages destined for agents on the local device are passed up to the protocol layer. The implementation details of this layer will vary from one type of device to another. For example, processors that are not required to route traffic from one Intel QPI link on to another may not have a full routing layer.
- The Protocol Layer has multiple functions. The protocol layer manages cache coherence for the interface using the write-back protocol. It also has a set of rules for managing non-coherent messaging. The protocol layer typically connects to the cache coherence state machine in caching agents, and to the home agent logic in memory controllers. The protocol layer also is responsible for system level functions such as interrupts, memory mapped I/O, and locks. One major characteristic of the protocol layer is that it deals with messages across multiple links, involving multiple agents in multiple devices. Lower layers typically deal only with two directly connected devices.
Figure 1 also introduces three terms that identify the granularity of the information being exchanged between the layers -- phits, flits, and packets. These terms, and each layer's functions, will be examined in a little more detail in the following sections.
Note that this layered architecture allows for a great deal of implementation flexibility and future growth, all within the scope of Intel QPI. The modular approach allows for extensions to be accommodated in an incremental manner. Therefore, in later sections of this text when we talk of certain characteristics of Intel QPI, we are pointing out things that usually will reflect current implementations. Those should not be construed as limitations on the overall scope of the Intel QuickPath Architecture. One prime example is the physical layer descriptions. Today a certain set of operational speed or interconnect channel characteristics may be supported. That does not prohibit Intel QPI from evolving into different speeds, communication channels, or even in the types of messages being used.