Multicore-based AdvancedTCA architectures: High Availability key in networking layer

Meet the trio at the heart of any software IP layer that embraces High Availability: Multicore, AdvancedTCA, and Linux.

The telecommunications industry has adopted a flat IP-based architecture for its demanding wired and wireless services. This has led to true IP convergence: IP is also the de facto standard for enterprise information systems. All equipment including security gateways, VoIP and ToIP systems, Web 2.0 applications, storage systems, and the like rely on a sophisticated IP communications layer. The communications layer for multimedia and data-centric Next Generation Networks will need tremendous performance and sophistication. At the same time, infrastructure deployment schedules must be shortened for faster revenue generation. And customers will expect significant service reliability improvements. As a consequence, the software IP layer that is implemented in the equipment handling this bandwidth has become business critical.

The three musketeers at the heart of an IP layer that is High Availability-ready are multicore, AdvancedTCA, and Linux. Multicore technology makes the needed processing capabilities possible, offering a high level of integration with lower power consumption. AdvancedTCA standards enable the building of carrier-grade systems that include both computing and communications elements, meeting industry scalability, modularity, and High Availability demands. Linux is becoming a de facto standard as a flexible, open, and efficient operating system with carrier-grade capabilities to expedite developments for stringent applications.

High-performance multicore packet processing software architecture

Several years ago, efficient network equipment architectures were defined for high-speed routers as a way to ensure performance for a new explosion of Internet traffic. Today, this IP architecture has been extended to new services such as L2 protocols, security, mobility, and multicast. Figure 1 shows the familiar data, control, and management planes into which packet processing is partitioned.

Figure 1: Network Equipment Architecture

In equipment requiring high-bandwidth processing and deep packet inspection, one type of processor manages the data plane and another type of processors manages the control plane. Functions implemented at the data plane level are performance-oriented with simplified and efficient algorithms. Splitting the data plane into two parts, Fast Path and Slow Path, optimizes performance.

The Fast Path handles all mechanisms that require per-packet processing such as determining which port a packet should be forwarded to or encrypting and decrypting packets containing associated IPSec security.

However, some packets are too complex to be processed with Fast Path, so the Slow Path handles these. Generally, the Slow Path implements a complete IP stack and handles most exceptions not ideal in the Fast Path. The Slow Path corresponds to the networking stack of the operating system. The Slow Path and Control Plane can be co-located on the same processor.

The framework just discussed makes a great number of things possible in a multicore-based architecture. For example, many cores can be allocated to the Fast Path to process heavy traffic at the highest level of performance. A low-level executive is recommended for use outside the operating system environment to implement lock-free packet processing and optimize memory bandwidth contention. Use of an executive leads to far greater performance compared to that achieved using a standard operating system alone. The remaining cores are used by the Slow Path and the Control Plane under a standard operating system, such as Linux.

High Availability system requirements

Typical networking equipment designed to provide High Availability capabilities is based on N+M architecture; it brings a certain level of redundancy to the system, as illustrated in Figure 2.

Figure 2: N+M High Availability Architecture


The architecture shown in Figure 2 is based on active elements providing the expected service. There are some inactive elements that are not in operational use. The goal of the system is to replace a failing active element by an inactive one to restore the expected level of service within the shortest period of time.

Several strategies can be implemented according to the requirements for service interruption. Once a failure has been detected, an inactive element is configured to replace the failing one. This approach also means the whole configuration has to be restored. And the new element shoulders the task of learning complete information from the system. With routing for example, the configuration of the routing protocols has to be done on the new element, and the routing protocols have to complete the route learning process to provide the service. This could take a long time.

To avoid such long interruptions in service, a more sophisticated architecture, based on 1+1 architecture, comes into play. A pair of elements (one active, one inactive) is used. A process to maintain a coherent view of the system in both elements is defined. Synchronizing the required information between both elements provides the expected level of service within a very short time in case the active element fails.

Once the equipment architecture has been defined, the system-level architecture may benefit from the available hardware resources (interfaces) in a 1+1 High Availability architecture to enhance system availability. Connecting the equipment to the network architecture creates several physical paths. Standard protocols such as Link Aggregation or ECMP can be deployed on at least two interfaces for redundant connectivity.

Ensuring packet processing software is High-Availability-ready

Providing High Availability means the entire system must be monitored. This includes chassis as well as the different hardware and software elements. Packet processing software has to be interfaced with the High Availability framework to make possible such relevant decisions as restarting a specific daemon or rebooting the Control Plane and/or the Fast Path.

The extensions to a High Availability-ready IP layer fall into four categories:

·        Synchronization software has to be added to manage redundant architecture and synchronize information between peer Control Plane protocols (these include ARP-NDP, routing, IPSec, firewall, etc.). Each Control Plane protocol has specific synchronization software.

·        A Daemon Monitoring System has to be added to periodically check the health of networking software components and inform the High Availability framework about it. The Daemon Monitoring System receives commands from the HA framework to handle the networking software components (for example, start, stop, and graceful restart).

·              Designers must make graceful restart extensions for key pieces of packet processing software High Availability aware. Each one should be start and stop capable and provide its own status.

·        Export interface visibility is essential to make all interfaces of the system visible from the Control Plane, enabling traffic engineering.

Using AdvancedTCA multicore boards to make packet processing software High Availability-ready

Figure 3 describes the different pieces that must come together to make packet processing software High Availability-ready using AdvancedTCA multicore boards. First, the HA framework monitors all hardware elements of the system using protocols such as HPI or IPMI. Packet processing software is also monitored and should provide a library to mask implementation details to the framework so that the network monitoring system can relay high-level messages (stop, start, graceful restart).

Figure 3: High Availability-Aware Routing Architecture

In the Figure 3 example, a routing protocol is running in the Control Plane on both boards. To minimize service interruption a dedicated daemon synchronizes the routing tables to assure the inactive element will have all the information to restart within the shortest period of time. The daemon does not need to synchronize the forwarding tables of the Fast Path – they can be quickly updated by their Control Plane, and Fast Path by its construction provides nonstop forwarding.

All network software components need graceful restart designed in. With this, they can be restarted on request by the monitored daemon or by the monitoring system that has received a restart command from the High Availability framework. The management system plays a key role in the process because it configures some daemons, such as the routing daemons.

Each Control Plane should be able to send and receive packets on both local and remote Fast Paths to make it possible to implement system-level traffic engineering. All interfaces have to be visible from each Control Plane. Exceptions have to go through the active Control Plane and forwarding is enabled between Fast Paths. Once done, a protocol, such as Link Aggregation or ECMP, can be deployed to optimize all available interfaces.


Networking software architectures can be designed to fully benefit from improvements in multicore technology, so performance for AdvancedTCA and Linux can be fully maximized. The best architecture design options for high-performance packet processing is based on a flexible distribution of cores between a Data Plane specifically designed for multicore and the Control Plane. System-level High Availability requirements based on redundant hardware AdvancedTCA architectures have significant impact on the networking layer that is critical to provide services without any interruption. Synchronization mechanisms, monitoring systems, and graceful restart capabilities have to be implemented as well as extensions to reuse all existing interfaces to improve system availability.

Eric Carmès is Founder and CEO of 6WIND. He has more than 15 years' experience in IP standards and architectures and is an expert in current and next generation IP technologies, protocols, and multicore. Previously, he was Head of the Network Department at Thales (previously Thomson-CSF), managing large teams of advanced R&D engineers designing and developing IP network architectures and COTS technologies for stringent military programs. Carmès holds a Master of Science degree from both INSA (French University for Applied Sciences) and ESE (French Electrical Engineering University). He can be contacted at


Multicore Packet Processing Forum