Marching toward 40 Gb Ethernet testing

We've all been inundated with statistics on subscriber bandwidth demands tripling over the next years with the primary applications being video based. Today's network infrastructures need an answer to move up the bandwidth ladder. 40 and 100 Gigabit Ethernet (GbE) provide the rungs to that ladder. In addition, PICMG has released the 40G (ATCA) standard and has been doing 40 Gb interoperability testing on ATCA systems as part of their Interoperability Workshop program for over two years. These factors would indicate that 40 Gb is poised for a large growth in adoption.

Deploying 40 Gb in the or network infrastructure isn't a matter of simply replacing line cards and flipping a switch. There is a lot of innovation in the Ethernet specifications that enable this 4x jump in speed. And the test and validation of the physical layer interfaces, cabling, optical transceivers, and even higher layer protocols can be very tricky without the right understanding.

I had the opportunity to interview Charles Seifert, Senior Product Manager of the Ixia Layer 2/3 hardware team, and Michael Haugh, Ixia's Senior Manager of Product Marketing. Ixia is a market leader in test equipment solutions and with a focus on Ethernet, it's not surprising they have been at the forefront of the development of test solutions at 40 and 100 Gb. This column explores the technology behind 40 Gb, test considerations, and market applications. We'll look at the complexities, challenges, and opportunities involved with broad-based adoption and rollout for 40 Gb.

What's different about 40 Gb?

In order to understand test complexities at 40 Gb, we need to start at the bottom. The 40 GbE physical layer looks quite different from its 10 GbE predecessor. Seifert explained the key difference is that 40 GbE and 100 GbE maintain the link as a group of "Physical Coding Sublayer (PCS) lanes." The driving force behind 40 and 100 Gb is the PCS and the way the MACs define control and data blocks to multiplex encoded data using a concept called "lane markers" that ensures data sent and received over multiple lanes is properly formatted (Figure 1).

Figure 1: The physical/link layers in a 40 Gb system architecture are depicted here.

Unlike 10 Gb, where the transmission is serial, 40 and 100 Gb use new control codes and data blocks. A new data scheme has been added with some intelligence that allows the link partners to talk with each other to send synchronization bits and lane transmission information. The data scheme involves two processes -- sending frames and an average frame gap. Underneath the link itself are encoded data blocks in 64/66-bit (64/66b) encoding, and there are 8-byte alignment lane markers used for identifying where the data came from in the four-lane system. Then, at the receiving side, there is a system that synchronizes the encoded blocks of data and identifies the alignment lane marker where all the information is decoded, de-skewed, and reassembled into frames.

Link management and health are also more sophisticated because there is no pre-determined transmit pattern for sending data over the lanes with their alignment lane markers -- this makes more work for the receiver. Data striping over the lanes, reassembly, and handling lane skew all create the need for more comprehensive testing at the physical layer. Haugh mentioned that these complexities and the need to be able to flexibly test many forms of lane data distribution, different skews, and control/link health monitoring have made for a long learning curve for many of their customers.

One other thing worth noting is that all the lanes must be up and operational -- there are no provisions in the 40 and 100 GbE standards that dictate changing transmission schemes if any lanes fail. If one or more lanes go out of synchronization, the link is down. Fortunately, 40 Gb leverages the Link Fault Signaling State Machine (from the 10 Gb standard) to manage the state of the link as with or without fault. When there is a fault, no data is transmitted until the fault is cleared.

How 40 Gb affects testing and test equipment

Now that we understand the physical layer transmission characteristics of 40 Gb Ethernet, it's easier to understand the requirements and features needed for testing.

Haugh said that Ixia thought long and hard about how to handle these additional complexities and came to the conclusion that they needed to expose these new characteristics to give test control to the tester. For example, Ixia's equipment has the ability to inject a "lane skew" (in other words, a time delay between PCS lanes) to test different skewing of all the lanes so that the receiver can be rated to handle the skew. There are also knobs that can test de-skew -- how the receiver handles the time delay so it can reassemble and regain the right synchronization for frame data.

There is also no standard for the lane marker patterns for data transmission across the lanes that is sent from the transmitter to the receiver. Therefore, Ixia's equipment has the ability to change the data transmission lane marker pattern characteristics across the lanes using one or more of the lanes so that the flexibility and robustness of the receiver can be validated and even stressed.

So, Ixia starts at Layers 1 and 2, exposing all the lane alignment marker assignments, transmission patterns, and skews at Layer 1 to be able to robustly test that layer before moving up the stack. This kind of test capability can be invaluable for anyone building ASICs or , SERDES/PHY chips, optical transceivers, and cabling to validate their designs.

Another problem for 40 gig -- Layer 1 Bit Error Rate Testing (BERT). Seifert explained that Ixia's equipment sends prescribed bit patterns based on industry standard polynomials that create large bit patterns to stress the link partner's signal handling of the link at the receiver. This generates a Layer 1 Pseudo-Random Bit Stream (PRBS) transmitted at the lane's line rate (10.3125 Gbps) over any number of lanes. This stress testing helps greatly in qualifying cables that are very difficult to test and rate for reliable 40/100 GbE, transceivers and Active Optical Cables (AOCs), and active and passive copper that can be very problematic to test, qualify, and support.

How does 40 Gb affect test & validation at the layers above?

Ixia has been involved with 40 GbE since the development of the IEEE Std 802.3ba-2010 in 2007. In the beginning, port density and cost isn't as important, and their first 40 Gb test module was a two-slot, single-port blade. Haugh mentioned that 40 GbE has seen much more traction in the data center than it has with Network Service Providers (NSPs), and that while data centers can take advantage of 40 GbE, NSPs can, in most cases, achieve similar bandwidth gains using other port aggregation strategies. So, while there are pockets within the infrastructure that may utilize 40 GbE, service providers are already deploying 100 GbE solutions.

Due to traction in the data center market, Ixia needed more than just a data plane line card, and the next-step solution for this market involved running protocols as well. Adding this capability enabled data centers to test 40 GbE loading within the data center and perform a larger range of stress tests using various traffic mixes. The two-slot, single-port 40 GbE solution therefore evolved into both a single-slot, four-port 40 GbE and single-slot two-port 100 GbE test blade, enabling data centers to fully test the device beyond basic system-level testing.

Another unique characteristic of the data center involves the use of Fibre Channel to provide critically important low-latency connections between the compute and storage systems. For this reason, there are purpose-built switches in the data center with very-low-latency characteristics that support Fibre Channel over Ethernet (FCoE). Ixia's equipment supports wirespeed testing for all packet sizes at the link layer, then moves up the stack to test Operation and Maintenance (O&M) capabilities, lag/jitter characteristics, and event elements like spanning tree protocols. Haugh commented that 2012 was a boom for 40 GbE in the data center and products are coming to market to allow data centers to reduce their cable plants and use 40 GbE as an aggregation mechanism.

Fibre Channel won't be going away anytime soon, so 40 GbE and associated test systems need to accommodate interoperability and measurement of networks that utilize both.

Test equipment strategy: An example

Creating effective test equipment for what is essentially a new technology whose market extends beyond network equipment manufacturers and service providers to data center applications required a different approach to developing a solution. Mike summarized the strategy as a ground-up approach that includes:

  1. Giving control of new features of 40 GbE transmission technology to the testers by exposing adjustable "knobs" that allow transceivers and cabling to be thoroughly tested. This entails exposing things like PCS lane transmission and skew characteristics.
  2. Accommodating wirespeed traffic packet generation at any packet size in conjunction with precise latency measurements taken at the physical layer of the test equipment.
  3. Incorporating link aggregation, Fibre Channel, O&M, and routing/forwarding services like spanning tree.
  4. Adding VLAN tagging and MPLS for additional flexibility with test scenarios.
  5. Providing a full suite of products that handle items 1-4. This goes beyond simply validating a line card and system-level functional testing to encompass a comprehensive test perspective where there are tools available for scaling performance and reliability for NSP, data center, and network equipment applications.

Ixia's product line starts with the chassis -- a 2 or 12-slot chassis provides the right fit for low- to high-port count test scenarios. Next, the chassis is populated with "load modules" -- these are the test cards that provide the link connectivity to the test environment. The load modules provide the right port count with the right physical and link-layer connectivity for the network topology. At 40 GbE and 100 GbE, these load modules also provide the fine-grain timestamp, latency, and skew information needed at these higher data rates.

Beyond the chassis and load modules are the software packages. These provide full Layer 2 and 3 protocol processing test capability, as well as configurable PCS lane testing. This provides the foundation of functional testing for interface hardware development teams.

Emulating and running various protocols and traffic mixes are implemented within the test system with the Ixia IxNetwork software framework. This framework allows testers to pick and choose protocols needed for both functional and stress testing of the environment. The framework accommodates a number of protocols, including O&M, Ethernet Layer 2, and others needed for network operation emulation and testing within service provider and data center environments.

A key aspect for at 40 and 100 GbE is latency, especially when it comes to video traffic. At these data rates, the latency measurements must be built into the hardware and in turn be exposed to the for viewing and manipulation of the latency to test the resiliency of the network and server systems. Haugh mentioned that Ixia reduced latency resolution from 20 ns in 10 GbE systems to 2.5 ns at 40 GbE. So timing resolution and making time-based packet measurements has been significantly enhanced at higher data rates.

Haugh believes that the 40 GbE trend will continue to be very strong within the data center environment. The 40 GbE data rate appears to be a sweet spot for equipment in the data center and the bandwidth requirements needed therein. Because of this, it's critical for equipment manufacturers to differentiate at this data rate. Important characteristics include port scalability (price per port is becoming important at 40 Gb), finer latency measurements and control, and more protocols and traffic mixes included. Even your green footprint matters -- efficient, low-power systems are attractive since power is a big expense within the data center.

The use of purpose-built switches for 40 GbE looks to be very strong over the next couple years. And port densities will continue to increase -- more ports in a smaller form factor. Haugh says Ixia isn't stopping at the current port count -- they will continue to increase port density to make the price per port more attractive throughout 2013 and beyond.

In terms of the NSPs, 40 GbE doesn't seem it will have a dramatic impact. Network operators are already deploying and testing 100 Gb as they are pressured by the phenomenal growth in video streaming in conjunction with subscriber growth and mobile device technology advancements. Network operators will use 40 GbE in select places, but at most points within the network where 10 GbE environments are being upgraded aggregation technologies are being applied. And where larger bandwidth is needed, 100 GbE appears to be the solution of choice.

What's next for 40G test?

The 40 GbE solution is here, and test environments are available to ensure that reliable performance, connectivity, topology operation, interoperability, and quality can be maintained and advanced. PICMG's 40G ATCA standard will play a big role in enabling test equipment to drive 40 GbE solution deployment.

If you'd like more information on 40 and 100 GbE testing, Ixia provides "black book" test applications for a number of key IP and test issues and methodologies: