AdvancedTCA meets 10GBASE-KR

In March of 2007, IEEE approved 802.3ap, standardizing the physical layers for Ethernet over electrical backplanes. The specification included a 1 Gb Ethernet interface and two 10 Gb Ethernet interfaces: 10GBASE-KX4 and 10GBASE-KR. PICMG is currently at work incorporating these new standards into AdvancedTCA.

Doug surveys the new IEEE backplane Ethernet standards and ongoing PICMG activities and discusses the possible future of Ethernet's AdvancedTCA prospects.

The new IEEE backplane Ethernet standards

Table 1 summarizes the characteristics of the new IEEE standards. One key distinction between the 802.3ap interfaces is the number of lanes required to transmit a signal. 10GBASE-KX4 requires four lanes and leverages XAUI, making it similar to PICMG's 10GBASE-BX4 solution. 10GBASE-KR, on the other hand, uses a 10.3125 gigabaud (Gbd) signaling rate in order to transmit 10 Gb of data over a single lane. Backplane channels are limited in AdvancedTCA, so the more efficient transmission of 10GBASE-KR gives it some advantages over 10GBASE-KX4.

21
Table 1

Auto-negotiation is another key element of the new standards. Links are established using the fastest common connection available to both endpoints. 10GBASE-KX4 and 10GBASE-KR must both be able to auto-negotiate a connection with a single lane of 1000BASE-KX. Auto-negotiation is a key enabler of backward compatibility in bladed systems such as AdvancedTCA.

The best way to refer to these interfaces is to use their full names; however, for the sake of brevity, their names will be abbreviated by dropping all but the last few letters. Therefore, 10GBASE-KR is shortened to "KR". "KX4" refers to 10GBASE-KX4.

The eyes have it - KR details

In order to provide 10 Gb on a single lane, KR departs radically from other Ethernet options available on AdvancedTCA. This can best be seen by looking at its signal at the receiver.

The "eye-diagram", known for its eye-like shape, is a common method for evaluating signal integrity. The received signal is displayed on an oscilloscope, allowing multiple transitions to overlap on top of one another. The waveform on the left side of Figure 1 is typical of a clean signal. A wide open hole shows good signal-to-noise ratio and a signal that is easy to recover without errors.

21
Figure 1

The signal on the right side of Figure 1 shows an eye diagram that might be expected from KR. The eye is completely closed. By traditional measures, the signal is lost in the noise.

In order to overcome this, KR transceivers must work together, employing a com-bination of equalization, forward error correction, and other advanced techniques to recover the signal. These advances, along with improvements in backplane and connector technologies, are what make 10 Gb a viable option today.

PICMG activities

In January, PICMG kicked off a new subcommittee aimed at incorporating KR, KX4, and KX fabrics into the exist-ing AdvancedTCA Ethernet standard (PICMG 3.1). The subcommittee still has a lot of work ahead of it; however, it is making steady progress toward a release in the first part of next year. Two of the main areas of focus have been backward compatibility and channel characterization.

Backward compatibility

One of the key goals of the PICMG 3.1 R2.0 subcommittee is to ensure backward com-patibility with existing AdvancedTCA platforms. This means that Ethernet boards designed to work with 1000BASE-BX and BX4 PICMG interfaces should plug-and-play in KX4 and KR platforms. Likewise, blades with KR or KX interfaces should work in BX-enabled platforms.

Although this sounds simple in concept, there are a very large number of combinations that must be accounted for to make this work. To understand just how much complexity is involved, see Table 2, which shows a few of many backward-compatibility scenarios.

22
Table 2

In the first scenario, a 10GBASE-KX4 blade is plugged into an existing system, connecting it with a BX4-enabled endpoint. The most reasonable operation is to attempt connection using 10GBASE-BX4, given that the KX4 PHY may be capable of supporting BX4 operation as well. This connection needs to occur without requiring e-keying or management sub-system changes to the existing system.

In the next case, a KX4 endpoint is con-nected to a single lane of 1000BASE-BX. KX4 is required to auto-negotiate down to one lane of 1000BASE-KX, and 1000BASE-KX and 1000BASE-BX are similar, so the most reasonable operation is to attempt connection using 1000BASE-BX. Again, no changes to the existing system are allowed.

The last scenario is the most challenging of them all. In this case, two KR-enabled endpoints are placed in an existing system. We know the backplane will likely not support the 10.3125 Gbd symbol rate required, and the desired operation is for these blades to link up at 1.25 Gbd (1 Gb data rate). If, however, these same blades are placed in a backplane that supports higher speed signaling, a 10.3125 Gbd should be allowed. Backplane-specific e-keying behavior was not anticipated in the original AdvancedTCA specification.

The PICMG 3.1 R2.0 subcommittee has spent a great deal of time focusing on these and other scenarios to make sure that backward compatibility with existing AdvancedTCA platforms is maintained in a straight-forward and intuitive manner while still allowing higher speed operation in newer, 10.3125 Gbd-enabled platforms.

Communication channel characterization

The second major issue that the PICMG 3.1 subcommittee faces has to do with inter-operability of KR components: the speci-fication, measurement, and testing of a 10.3125 Gbd communication channel.

In communications theory, a channel consists of everything along the path that connects a transmitter to a receiver. In AdvancedTCA, this means the circuit traces on the blades and backplane, the passive components, and the connectors.

Accumulation of electrical losses or noise within the channel can contribute to bit errors at the receiver. To control these bit errors, communications channels must be built according to parameters required by the signaling technology. Typical parameters include characteristic impedance, insertion loss, return loss, and cross talk. In general, the higher the speed of the transmitted signal, the more important and tightly controlled these parameters become.

In order for the AdvancedTCA 10GBASE-KR ecosystem to flourish, the channel needs to be specified so that solutions from different suppliers can be developed independently yet still have a reasonable chance of working together. At the same time, the channel requirements can't be so overdesigned that they unnecessarily increase cost due to esoteric materials requirements or experimental fabrication techniques.

Although IEEE 802.3ap provides some guidelines for 10GBASE-KR channel charac-teristics, they can't be directly applied to AdvancedTCA. At the time of the writing of this article, the PICMG 3.1 subcommittee was working on strategies for characterizing the channel that meets all the requirements for KR interoperability. In addition, the PICMG Interconnect Characterization subcommittee is looking at channel specification, testing, and measurement as applied generally to all PICMG standards.

What KR Ethernet means

With all this extra work to incorporate 10GBASE-KR into AdvancedTCA, you might be wondering "why bother?" After all, AdvancedTCA already has a 10 Gb backplane standard, 10GBASE-BX4. Does it really need another? The answer is yes, and perhaps some of the best reasons to pay attention to 10GBASE-KR are bandwidth scalability, flexibility, and future-proofing.

AdvancedTCA defines a blade's fabric channel as a collection of four "ports". For 10GBASE-BX4 operation, all of these ports are used to make a connection. KR operation, on the other hand, requires only one "port", leaving the other three free for additional signaling (see Figure 2). System designers are now free to provide 20, 30, or even 40 Gb of fabric channel connectivity to any blade that requires it - something that is impossible to do with BX4.

22
Figure 2

A second advantage of KR over BX4 and KX4 is that it is more flexible. Although some older BX4 transceivers can support connection at 1 Gb rates, they are not required to support this connection rate. And while KX4 will auto-negotiate down to a KX link, it is limited to one lane (1 Gbps) of connection. This means that boards with multiple ports of BX connectivity, such as options 1, 2, or 3, will only be able to connect the first port. On the other hand, every KR lane is capable of connecting at 1 Gbps (KX) rates. All of the previous options could be supported.

A third reason to consider a KR-enabled system is future-proofing. IEEE 802.3ba is now working on the specification of a true 40 Gb backplane standard: 40GBASE-KR4. Due to the way in which 10GBASE-KR is being leveraged to create 40GBASE-KR4, a reasonably constructed 10GBASE-KR backplane should be able to support 40GBASE-KR4 when the time comes.

The future of Ethernet on AdvancedTCA

As stated earlier, PICMG 3.1 work is ongoing with hopes of completion in the first part of 2009 (see Figure 3). In 2009 and 2010 we should see KR components (PHYs and switches) and platforms with increasing port densities and sophistication. In 2010 IEEE 802.3ba is targeting completion. This introduces 40GBASE-KR4 true 40 Gb backplane Ethernet to the market. Soon after, it should be incorporated into a future PICMG 3.1 specification with platforms available as early as 2011.

23
Figure 3

What is in the future for AdvancedTCA after 40 Gb? Is 100 Gb possible without major overhauls? Even using four lanes, signaling rates in excess of 25 Gbd would be required. At one time the probable answer would have been that this is completely out of the question. However, with continued advances in PHY, connector, and backplane technologies, who knows what the future might hold? After all, even a few years ago 10GBASE-KR would have seemed out of reach.

Doug Sandy is the Senior Staff Technologist of the Technology and Strategy Department for the Embedded Computing business of Emerson Network Power. Doug is responsible for evaluating the performance of computer systems, constructing models for the systems, and predicting the systems behavior. He also chairs PICMG 3.1 R2.0, the subcommittee working on bringing 802.3ap Ethernet into AdvancedTCA. Doug holds a Bachelor's and Master's degree in Electrical Engineering from California Polytechnic State University, San Luis Obispo.

Emerson Network Power Embedded Computing
www.emersonnetworkpower.com/embeddedcomputing
Doug.Sandy@emerson.com