Ethernet - 40G to 400G: Interview with John D'Ambrosia
With Ethernet speeds advancing past 40G to 100G and higher, designers of systems such as AdvancedTCA (ATCA) meticulously follow activities in the IEEE 802.3 Ethernet subcommittee for any indicators on the future of backplane-based systems. In this interview with John D’Ambrosia, IEEE P802.3bs Chair, Chairman of the Ethernet Alliance, and Dell’s Chief Ethernet Evangelist, he discusses the development and deployment of the recently released 100GBASE-KR4 and -KP4 specifications, as well as the beginnings of IEEE work on 400G Ethernet and beyond.
Give a little background on the 100GBASE-KR4 spec, it’s development, and finalization.
D’AMBROSIA: The 100GBASE-KR4 spec was developed as part of what was referred to as the IEEE 802.3 100 GbE Backplane and Copper Cable project. That project was being considered as the development of 40 GbE and 100 GbE was coming to a close. In that case, the 40 GbE and 100 GbE project hadn’t developed any new electrical signaling on a per lane basis, meaning that all of the electrical interfaces developed as part of that project were based on 10 GbE signaling.
100 GbE optics actually was developed with a 4-lambda approach based on 25 GbE per lambda. When you look at the optical specs you see 100GBASE-LR4 and 100GBASE-ER4; that was really the initial 100 GbE project. So right off the bat there was this understanding that we were going to go to 25 GbE technology just because we had looked at it from an optics perspective, which really became a systems perspective.
As I said, the electrical interfaces that were developed for the 100 GbE project, which did not include a backplane solution by the way, were based on 10 lanes of 10 GbE. As an old backplane designer I can tell you that there’s no way in hell I want to have to deal with 10 lanes of 10 GbE across the backplane. So as the group was winding down on 100G development, which was really an optics focus, we realized that we needed to start introducing the next members of the 100G family. And following those discussions we had around developing the optics, we started looking at developing 4 x 25 Gbps Ethernet. That’s why it was called “Backplane and Copper Cable.”
As that project emerged, it became 100GBASE-KR4 (which is a backplane solution based on NRZ signaling), there’s 100GBASE-KP4 (which is based on PAM4 signaling), and then there’s 100GBASE-CR4 (which is across a twinaxial cable for 5 m based on four differential pairs of 25 Gbps Ethernet in each direction). The actual project started with a call for interest (CFI) that happened in November of 2010, and that spec was ratified in the middle of last year. That’s been finalized, so that spec is out there now.
What were some of the challenges with 100G spec development, and how do the KR4 and KP4 specs differ?
D’AMBROSIA: In hindsight, some of the FEC encoding became more involved, and the bigger point in terms of AdvancedTCA (ATCA) is you had new channel specs. This is where -KR4 and -KP4 came from.
As a kind of rule of thumb, you’re probably looking at a 25 dB to 35 dB solution at the Nyquist rate. So, if I’m running at 10 Gbps, Nyquist is roughly half of that – 5 Gbps – for NRZ signaling. That’s kind of constant, so when we jumped to 25 Gbps the channel was in the 25 dB to 35 dB range, but now at 12.5 Gbps. So as you move to higher speeds, the channel itself becomes an issue. That’s always one of the challenges you’ll face, and you can see the same kind of thing with the 10 GbE and 40 GbE specs.
NRZ signaling stands for non-return-to-zero, and it’s a form of pulse-amplitude modulation (PAM). So it’s your classic zeros and ones. In the case of PAM4, the way that I like to describe it is if you sit in a room and you ask people to raise their hands, those are ones; if they don’t put their hands up, those are zeros. So it’s usually very easy to tell the difference between zeros and ones. Normally it doesn’t really matter how quickly they put their hands up, but as you start moving faster and faster, you start to run into some issues. One of those is the challenges of the channel, and one of those challenges is cost.
If you were to look at PAM4, going back on my analogy, there would be more places for people to raise their hands. So zero would be their hands down, the next level might be at their waist, the next level would be at their head, and the last level would be above their head. Pulse amplitude modulation at four levels, or PAM4. Inherently, that reduces your signal-to-noise ratio right off the bat.
In the case of NRZ, you send a symbol that represents a 0 or a 1. In the case of PAM4, you send a symbol (the baud rate), but in this case it is enabled by two bits. So when you’re at level 0 it represents something, when you’re at level 1 it represents two 2 bits, when you’re at level 2 it represents two bits, when you’re at the highest level it represents two bits – also, depending on the different transitions, you get into representing what those two bits are. So you’re sending more information in a single symbol than you do with NRZ, so your baud rate is half now. I gave you that rule of thumb about 25 dB to 35 dB at Nyquist, but in the case of PAM4 it’s not really 25 dB to 35 dB at 12.5 Gbps, it’s really 25 dB to 35 dB 6.25 Gbps to 7 Gbps because of some overspeed. That becomes a way to deal with channels that may involve lower cost materials. That’s really the benefit here – being able to send more information over a lower cost channel is essentially the benefit of PAM4 over NRZ.
What kind of adoption are you seeing for each specification, and what type of interoperability challenges, if any, do the two types of signaling pose for backplane designers?
D’AMBROSIA: One of the things that we are seeing is a lot more work being devoted to 100GBASE-KR4, the NRZ specification. You see more products being introduced supporting 100GBASE-KR4 than 100GBASE-KP4 today. You have to realize that there’s an ecosystem out there that supports NRZ signaling. It goes from the test equipment to just the basic engineering understanding.
But the debate between NRZ and PAM4 has been going on for a long time, and there are more and more people interested in PAM4. Going back 13 years, I helped form a group called the High Speed Backplane Initiative, and we were looking at using PAM4 for backplanes back then. At that time there were a few parties interested in it, but ultimately the industry decided to not specify PAM4 and specify NRZ for 10 GbE signaling. When it came time to do 4 x 25 GbE signaling, more companies stepped up saying that they think we should do PAM4. Now as we look to going to 400 GbE and trying to go faster, more and more people are looking at using PAM4 signaling to support 50 Gbps electrical, especially over the longer channels just because of the inherent channel challenges.
It’s an interesting question as we go forward because a lot of the 25 GbE NRZ that has been developed and is now out in the market provides similar circuitry to what would be needed to support 50 GbE PAM4. There are some people who are pointing to that saying it’s a good thing because we could support backwards compatibility, and as we look at having to go backwards it would be to our benefit to consider doing 50 GbE PAM4 because of the popularity of 25 GbE NRZ, as opposed to having to go to newer processes to support 50 GbE NRZ. Those arguments are already happening now as we look at supporting 50 Gbps electrical signaling as part of the 400 GbE project.
The IEEE hasn’t announced any intentions of taking copper backplanes further than 100 GbE. From your perspective, what are the most viable options moving forward?
D’AMBROSIA: We’re developing 400G now. The 802.3bs Ethernet Task Force has decided that it will develop electrical interfaces for chip-to-chip (by chip-to-chip I mean on a card, not across cards) and chip-to-module. We will develop 25 GbE and 50 GbE electrical interfaces – 25 GbE will leverage NRZ signaling, 50 Gbps is still a topic of debate over whether it will be NRZ or whether it will be PAM4.
The interesting thing here is that, when I said from a backplane designer’s perspective that doing 10 x 10 GbE interfaces across a backplane didn’t sound appealing, for 400G it would be 16 x 25 GbE and 8 x 50 GbE. Even at 50 GbE that sounds really ugly to me, but it is pretty much recognized that 50 Gbps signaling is going to be the next development point.
The introduction of 50 Gbps though, if you look at what’s going on inside the IEEE right now, could raise some interesting questions. When you look at what’s happening with how we’re developing 25 Gbps signaling technology as part of 4 x 25 GbE for 100G, and then you look at 40G, which is leveraging 4 x 10 GbE today, what you see is that you can’t get the same port density into an ASIC at 40G as you can at 25G because the basic building blocks don’t line up. In the case of 25 GbE, they do – 25 GbE I/O on the chip to 25 GbE I/O on the boxes. So you can get to the maximum density this way and the maximum throughput on the chips, which helps to optimize your switches, which helps to optimize and reduce cabling costs, power costs, and so on. So there is this inherent niceness to 1x or 4x architectures.
Now we’re getting to the question, and this is still a question, “Will we see 50 GbE and 200 GbE emerge as speeds for backplanes?”
There’s been a lot of discussion and debate, and the logic of this is very relevant. But we’re still in the early stages of developing 50 GbE electrical signaling. I can tell you as Task Force Chair that this comes up and I have to keep people within the scope of our project par. It’s not a foregone conclusion yet by any means in IEEE 802 because there haven’t been any votes yet taken on any of these discussions. These are really industry debates around 50 GbE and 200 GbE.
So, will we see 50 GbE and 200 GbE emerge? That is the question. I don’t have the answer for you today, but I get the logic. The role of the IEEE 802 specs is to produce high-quality, market-relevant standards. The IEEE 802 does not decree, “Thou shalt develop.” That’s not the way the process works. By market-relevant standards, what we’re saying is that the market decides what it wants to develop a standard for next. So the market market comes forward and says, “We want to do this.”
Editor’s Note: The Ethernet Alliance recently announced its 2015 Ethernet Roadmap at the Optical Fiber Communication Conference and Exposition (OFC 2015) in Los Angeles. The Roadmap outlines the ongoing development of Ethernet through 2020, and includes projections for electrical/optical speeds possibly as high as 10 Tbps. A portion of the roadmap is located on page 17, with the entire Roadmap available for download at in its entirety at.
IEEE P802.3bs 400 Gb/s Ethernet Task Force
Dell Networking Solutions