Trends in network testing: Test solutions for real-world testing

Understanding and evaluating much higher level metrics is crucial for multi-service networks.

Hearing the term network test equipment may cause visions of boxes that test the correctness of protocols to spring to mind. While it’s true that testing correctness of network protocols is an important part of lab validation, test equipment vendors are striving to extend beyond simple test capabilities to model very complex subscriber services and their interactions.


Anyone who has developed a product based on Internet Protocol (IP) is probably familiar with Ixia ( Ixia is well known in the industry as one of the leading IP-based test equipment companies, having started in Ethernet and IP protocol testers early on. When Ixia began, IP was primarily used for Local Area Network (LAN) connectivity within the enterprise. Since those humble beginnings, IP has grown to be the de facto standard for multi-service network transport. Initiatives like the Metro Ethernet Forum (MEF), 3G and 4G wireless, and WiMAX promise to make IP the ubiquitous network layer for the global converged network.

As IP-based services have spread, so too has the complexity of validating IP-based devices. Ixia has adapted to meet these grow-ing test solution needs. One of the key success factors has been the company's ability to partner with its customers to accelerate the development and deployment of converged IP networks and services.

The transition from simple protocol testing to a flexible configura-tion for multi-service network subscriber modeling is filled with challenges and complexity. I recently spoke with Sunil Kalidindi, Director of Product Management at Ixia, about how these new challenges are being met.

Test requirements and design strategy

One of the first requirements comes from the ability to provide a solution that allows the network operators to effectively utilize their CAPital EXpense (CAPEX) and OPerational EXpense (OPEX) dollars. From the design and product architecture perspective, preserving a single platform for doing all the required testing is important. This tends to be more complicated than it sounds. For example, Cisco makes a myriad of network infrastructure products that implement a number of switching and routing functions within the network. Ixia must supply a single form factor product with the ability to test and/or emulate all these functions. In order to achieve this, a load module approach that can be scaled within a multi-slot chassis has been used. Each load module is a plug-in card that runs a set of applications for testing Layer 2-7 functions. Currently Ethernet and Packet over SONET/ATM load modules are available that go up to 10 Gigabit and OC-192 respectively. This allows the test system to scale with the requirements of the testing to be performed.

The design of the load modules themselves is challenging. Each load module is a purpose-built hardware system tuned for high performance and designed to be scalable within a multi-slot chassis. These load modules are designed to allow hundreds of thousands of MAC addresses and millions of IP addresses for large-scale network testing. The underlying hardware components, involving high-speed CPUs and FPGAs containing optimized algorithms, enable the test environment to reach the performance targets necessary to test high-speed multi-service network testing.

A highly customized version of Linux is used to enable easy test application development, but the network stack within the Linux operating system must be bypassed to a large degree to enable the high-performance environment required by the load module applications. Sunil mentioned Ixia is spending more and more effort utilizing the new multi-core CPUs in the hope of increasing performance further.

The complete real-world test pyramid

Ixia has an interesting perspective on test product strategy, as illustrated in Figure 1. A comprehensive test environment is modeled as a pyramid.

Figure 1

The base of the pyramid is the traditional Layer 2-3 protocols that make up the foundation of traffic transport across the network. Protocols can be validated and performance measured at this layer of the pyramid.

The Layer 4-7 content switching adds another level of complexity. For example, testing the Session Initiation Protocol (SIP) can be problematic. While the individual elements of the messaging are well defined, how they are used to initiate a voice or data call can differ, depending on the equipment manufacturers of SIP servers and Voice over IP (VoIP) equipment. Testing must include validation of the individual message elements (that is, each message is constructed in a SIP-conformant way). But that's not enough - validation of proper call control signaling to create voice, data, and video connections is also required to validate real-world interoperability.

Real-world subscriber modeling is a new and critical test component for multi-service networks. It's important to be able to model hundreds of thousands or millions of subscribers using a mix of services. For example, it's extremely valuable that a service provider be able to understand the ramifications of a hugely popular pay-per-view event. Will the increased load on the network during the event slow down other network services? Could it bring down the network entirely? Using subscriber modeling can enable the service provider to model, measure, and eliminate these kinds of problems before they happen.

At the top of the pyramid is the term Quality of Experience (QoE). This level of testing helps service providers determine whether or not the subscriber has a good experience. This sounds subjective, but there are objective metrics that can be used to evaluate voice, data, and video quality. For example, what if a service provider would like to measure a rather subjective statement such as: "Here's the point where the subscriber begins to have a poor experience watching the event." Maybe this statement can be refined to say: "Poor experience is more than three human-viewable macrocell errors on the screen in 10 minutes and no more than one audio problem lasting longer than .5 seconds for the entire event". QoE functions can be used to set up a subscriber model and measure the number of dropped or errored packets that occur under the load conditions that match the requirements above. Using these objective metrics to measure and eliminate subjective problem statements can greatly reduce operational expense by avoiding a bad subscriber experience with a new service or preventing service interruptions before they happen.

Quantifying Quality of Experience

Identifying objective metrics by which QoE can be measured and communicated presents one of the biggest challenges. Ixia breaks the problem down into various dimensions, each of which can be tuned to model service variations:

  • Subscriber base - This variable scales from tens to a million subscribers.
  • Service mix - This variable defines the percentage of services being used at various times within the network.

The subscriber base and service mix knobs can be changed to measure the following things that make up QoE:

  • Full reference quality - This is a video quality metric that compares the video sent with the video received using a full MPEG layer analysis.
  • Traditional network metrics like dropped packets, load, and level of congestion - These metrics lead to poor service quality or even what can effectively be defined as service interruption. As far as the threshold definitions for these metrics are concerned, it's up to the service provider to define quality levels based on these metrics.

From requirements to product architecture

Figure 2 summarizes the architecture that enables Ixia to provide one unified platform that can integrate testing from the lowest to the highest layer of the real-world test pyramid.

Figure 2

The foundation of the Ixia product architecture is the hardware layer. This layer implements the interface blades, chassis, and Layer 2-3 protocols required to perform testing at the IP lab testing level of the pyramid. This test solution includes full conformance and performance tests to validate Carrier Ethernet. MEF test specifications MEF 9 and MEF 14 are included, as are Ethernet fault management Operation and Administrative Management (OAM) and Connectivity Fault Management (CFM) packets. There are pre-built libraries to save time in the test generation stage.

The IxNetwork block in the architecture performs the LAN switching and broadband delivery functions for the test environ-ment. In addition, IxNetwork is also capable of testing the L2-3 security aspects within the network. Comprehensive core and edge routing and switching support for IPv4 and IPv6 are included as well as MPLS. This block includes Internet Engineering Task Force (IETF) RFC-based tests and conformance suites for routing and switching standards. One truly impressive thing about this block is its ability to scale: A single Ixia box can scale to millions of routes involving thousands of routers using mixed routing protocol topologies. The broadband delivery test capabilities include the ability to test DSLAMs, remote access services, and Virtual Private Network (VPN) connections. Broadband access protocols such as PPP over anything, L2TP, and DHCP are included. Access security and encryption protocols can also be tested. One key element required to effectively test multi-service networks is stateful traffic over broadband control protocol, which adds a level of complexity within the test device. At a basic level, test equipment could simply generate the connection, creation, and termination messages to validate correct system operation. But for multi-play traffic and QoE testing, this isn't enough. The test device(s) must keep track of the connections being made, perform content delivery and analysis, and keep track of traffic loads versus connection frequencies and data delivery during system testing.

The IxLoad block implements the traffic loading environment for various services within the multi-service network. These services include video delivery, voice calls, Layer 4-7 security applications, content-aware capability for deep packet inspection, and other applications such as data delivery, gaming, and remote access. One dimension of the IxLoad block is data center and application delivery. This includes testing data center performance of infra-structure content switches, servers, load balancing algorithms, and similar components. The Ixia data center test solution can supply a 10 gigabit line rate of stateful traffic to test user access to data center content and applications and includes comprehensive SSL, IP security, and firewall testing for attack recognition and mitigation tests.

VoIP test solutions

Ixia's solution includes SIP testing in terms of protocol conformance and loading tests. The user can set up real-world subscribers and usage models on top of the IxNetwork capabilities and test true Quality of Experience.

Each of the functional blocks in the Ixia architecture may be tied together with Ixia's Test Conductor. This component provides an interface between the tester and the functional blocks. The tester uses this interface to define the level of the test and configure the blocks for traffic mix, subscriber model, and other parameters. Test Conductor enables regression tests ranging from low-layer interface and protocol testing to subscriber profiling and traffic load testing at the system level.

Figure 3 shows how the tester can create automated regression tests with these high-level device and system testing capabilities.

Figure 3

First, the tester defines the Device/System Under Test (DUT/SUT). The next step is to configure the device/system for the specific test. The Ixia test solution is used to configure the test equipment once the test environment is set up. From there, the test is executed, results analyzed, and updates or verification is performed.

The Ixia test equipment helps at each stage of this life cycle. Use of the graphi-cal interface to perform automated DUT configuration provides easy test definitions. Employing the equipment also makes it possible to determine the traffic configuration to use, to test the flow control for mix and frequency of traffic, and to display results and generate reports for test analysis. Over time, the test reports can be used to see device performance trends as device capabilities increase. Ixia includes an extensive set of pre-built libraries to facilitate creation of the device and system tests. Ixia also supports Tcl, which is commonly used for test scripting, so legacy or third-party tests can be integrated into the test system.


As we continue to drive toward a global IP multi-service network, it's no longer enough to validate equipment at the physical inter-face and connection protocol level. Multi-service networks demand the ability to understand and evaluate much higher level metrics in order to deliver reliable, enjoyable services from voice to video, data to gaming. The Ixia suite of test solutions provides an impressive mix of capabilities at all levels of the real-world test pyramid.

For more information, contact Curt at