A 40G AdvancedTCA Chairperson's perspective

3If you have been following AdvancedTCA (ATCA) developments lately, you have probably already heard of the release of PICMG 3.1 R2.0. This specification incorporates 40 Gigabit Ethernet (GbE) into the ATCA platform, quadrupling throughput over what was previously possible and opening the pathway for the next generation of data-hungry applications.

As former chairperson of the PICMG 3.1 R2.0 technical subcommittee and Chief Technology Officer of Emerson Network Power’s Embedded Computing business, I would like to offer my perspective on the 40 Gb ATCA specification. PICMG 3.1 R2.0 includes much more than requirements for how to implement 40 Gb AdvancedTCA hardware. The specification, which is perhaps the most ambitious addition to ATCA since its inception, defines a total of four new backplane Ethernet standards, a new connector definition, detailed channel definitions, and testing methodologies. All of these items, and what they mean for 40 Gb on ATCA, can be best understood by starting with a brief history lesson.

The road to 40 Gigabit Ethernet

The journey toward 40 GbE in ATCA began when IEEE defined true standards for backplane Ethernet in May 2007. IEEE 802.3ap introduced 1000BASE-KX (similar to PICMG 1000BASE-BX) and 10GBASE-KX4 (similar to PICMG 10GBASE-BX4). More significantly, the new IEEE spec also introduced another 10 Gb backplane Ethernet standard: 10GBASE-KR. 10GBASE-KR uses a different encoding technique and advanced signal recovery techniques to transmit 10 Gb of data over a single differential pair running at just over 10 Gigabaud (GBd).
Four ports of 10GBASE-KR can fit in a single ATCA channel, paving the way toward 40 Gb transfer rates.

PICMG work to incorporate the IEEE backplane standards into ATCA began late in 2007. During the development period, IEEE 802.3ba defined true 40 GbE for backplanes using four synchronized 10GBASE-KR lanes (40GBASE-KR4). This interface was also included in PICMG 3.1 R2.0 (Table 1).

Table 1: There are four backplane Ethernet standards incorporated into PICMG 3.1 R2.0.

Underlying principles of the specification

PICMG 3.1 R2.0 is based upon the following fundamental principles:

  1. Backward compatibility: 40 Gb systems must be able to support boards operating at 10 Gb and 1 Gb data rates. Likewise, 40 Gb boards must be able to work in older shelves but at slower link speeds.
  2. Interoperability: Boards from multiple vendors must work together. The ATCA ecosystem is dependent upon vendor interoperability.
  3. Independence: Board vendors should be able to develop boards independently of backplane vendors and vice-versa.
  4. Differentiation: The specification should include enough requirements to guarantee the above items, but leave room for vendors to find alternate implementation solutions.

Backward compatibility issues were addressed in PICMG 3.1 R2.0 by minor additions to the AdvancedTCA Electronic Keying (E-Keying) mechanism. These extensions allow the shelf manager to determine the signaling capabilities of the boards and backplane so that 10 GBd signaling will only be established if all elements support the higher speeds.

Achieving the goals of interoperability, independence, and differentiation was relatively easy for 1000BASE-KX and 10GBASE-KX4. Since the existing ATCA channel already supported their signaling rates, no modifications were required. 10GBASE-KR and 40GBASE-KR4 posed a greater challenge – a new system approach would be required to support their higher signaling rates.

Defining the high-speed channel

In communications theory, a channel consists of everything along the path that connects a transmitter to a receiver. In ATCA, this means that the channel is composed of the circuit traces on the boards and backplane, the passive components, and the connectors.

Accumulation of electrical losses or noise within the channel can contribute to bit errors at the receiver. To control these bit errors, communications channels must be built according to parameters required by the signaling technology. Typical parameters include characteristic impedance, insertion loss, return loss, and crosstalk. In general, the higher the speed of the transmitted signal, the more important and tightly controlled these parameters become. In order to support the 10.3125 GBd signaling required for 10GBASE-KR and 40GBASE-KR4, the channel bandwidth of ATCA needed to be improved by a factor of more than three.

Although IEEE provided good guidelines for a 10GBASE-KR/
40GBASE-KR4 channel, they could not be directly applied to PICMG. IEEE specified channel characteristics from transmitter chip to receiver chip. In order to be useful, PICMG 3.1 needed to distill the IEEE guidelines into two different parts. Board designers need requirements for the portion of the channel from the transmitter or receiver to the board’s Zone 2 connector. Backplane designers, on the other hand, need specific channel requirements for the portion of the channel from one backplane connector to another (Figure 1).

Figure 1: PICMG 3.1 R2.0 had to define channel requirements for board and backplane connections.

In order to solve this problem, the PICMG 3.1 R2.0 technical subcommittee identified 22 main corner cases of interoperability that needed to be addressed. These included variations from normal ATCA use cases (which slots boards are plugged into), as well as manufacturing and design variability. Limits for board and backplane parameters were then tuned until all corner cases met or exceeded the IEEE guidelines. This effort represented untold hours of simulation and data analysis.

One significant difference between the new PICMG 3.1 channel and the original one defined in the ATCA base specification is the change of the fabric connector on 10GBASE-KR and 40GBASE-KR4 enabled boards. The new connector, referred to in the specification as “ADFplus,” provides a significant crosstalk performance improvement over the original “ADF” connector found in the AdvancedTCA specification. Since the ADFplus connector inter-mates with the existing ADF backplane connector, there is no issue with backward compatibility.

In order to facilitate vendors independently validating their products, PICMG 3.1 R2.0 defines four different test cards for testing KR-enabled boards. Due to the high-speed nature of the signals involved, the parameters for these cards are tightly controlled. Table 2 lists the four cards and how they are used to test PICMG 3.1 R2.0-compliant components.

Table 2: PICMG 3.1 R2.0 defines four different test cards to enable independent validation of KR-enabled boards.

What to do with all the bandwidth

Now that PICMG-compliant 40 Gb ATCA solutions are a reality, a logical question to ask is, “What can be done with all the bandwidth?”

40 Gb ATCA is especially suited to address the growing demand for packet processing, inspection, and filtering bump-in-the-wire applications. Its bladed architecture allows processing capability to scale over time based on network demand and growth. In order to facilitate this, however, the basic infrastructure (shelf and backplane) need to be purchased up front because they are the two elements of the system that cannot be easily upgraded later. For this reason, you may wish to consider a 40 Gb-ready shelf, even if your application demands don’t require it today.

For those who do need the processing power today, there are options available from the merchant market based on the x86 architecture or manycore processors. For example, Emerson Network Power offers the ATCA-7470
and ATCA-9405 blades (Figure 2). A combination of these boards, coupled with 40 Gb switching and stateful load balancing, should be enough to get any 40 Gb application off the ground.

Figure 2: The Emerson Network Power ATCA-7470 (left) and ATCA-9405 (right) blades are examples of currently available 40G-ready solutions.

Concluding thoughts

While the Embedded Computing business of Emerson Network Power invested heavily in the success of this specification, it would be unfair to imply that we were the only ones to do so. By the end of its 4.5 year development, PICMG 3.1 R2.0 boasted perhaps the largest roster of participating individuals of any PICMG specification. The commitment and collaboration of so many talented individuals makes the specification what it is today. I’m thankful for the opportunity to have worked with each one of them, and look forward to seeing how PICMG 3.1 R2.0-compliant platforms drive the next generation of AdvancedTCA applications.

Doug Sandy is Vice President and Chief Technology Officer at PICMG and Chief Technology Officer of the Embedded Computing business of Emerson Network Power. He blogs and moderates an online technology forum at MyEmbeddedComputing.com/Forum.