COTS management building blocks for AdvancedTCA: Proven over the first decade

Over the past decade, the platform management layer of xTCA systems has proven itself a challenging necessity. Mark Overgaard of Pigeon Point Systems reviews various aspects and implementations of COTS managers, and offers a forecast for the ecosystem's hardware platform management layer.

4Ten years ago, in 2002, PICMG was working hard to complete the AdvancedTCA specification that was formally adopted on the last day of the year. The hardware platform management section accounted for 30 percent of the 430 specification pages. Late in the year, prototype ATCA shelves and boards were tested at PICMG’s first interoperability workshop, and the first prototypes of COTS management building blocks were delivered to early customers. It was already clear, however, as product development efforts ramped up, that the hardware platform management subsystem of ATCA represented a significant implementation challenge.

The first COTS management building blocks for ATCA were announced three weeks after specification adoption. Soon thereafter, in 2003, those building blocks were delivered and described in Systems , “Management Building Blocks Speed Product Development” in the March issue, and “Off-the-Shelf Management Solutions for AdvancedTCA Boards” in the May/June issue. Also that year, the PICMG interoperability workshops started including explicit testing of the ATCA management layer.

How has ATCA and its management layer evolved in the last decade?

Fast forward to 2012, and the market for just commercial ATCA CPU boards and (AMC) CPU modules is projected by VDC Research to exceed $550 million (the overall ATCA market includes major segments not covered by this forecast). Another study, done by Heavy Reading in 2011, forecasts a 31 percent annual growth rate for the overall market (which is dominated by ATCA and AMC modules used in ATCA systems), culminating in a market worth $4.3 billion by 2015. Every ATCA- or AMC-compliant board, module, and shelf must have a corresponding specification-defined management controller.

Tens of thousands of ATCA shelves, each potentially with a dozen or even dozens of management controllers, have now been delivered with COTS management technology. Major ATCA/AMC customers, such as Tier 1 TEMs, are identifying management-level interoperability of ATCA and AMC products as very important to their success and confirming the benefits of using COTS management controllers.

Meanwhile, the specification content relevant to the xTCA management layer has grown substantially, and includes:

  • 435 pages of management sections for the base specifications for ATCA, AMC, and
  • 66 pages for PICMG HPM.1, which defines a firmware upgrade architecture for xTCA management controllers
  • 130 pages or so for the PICMG HPM.2 and HPM.3 specifications, now nearing completion within PICMG (These specifications address LAN-attached xTCA management controllers, which have a LAN connection to supplement the mandatory Intelligent Platform Management Bus (IPMB) link to their supervising layers)
  • 568 combined pages for the Service Availability Forum (SAF) Hardware Platform Interface (HPI) specification and the corresponding mapping specification from HPI to the xTCA management layer

PICMG has now held more than 30 interoperability workshops and there are now commercially offered formal compliance test suites for xTCA management and HPI offered by Polaris Networks.

Considering the mounting body of content pertinent to the management layer, Nokia Siemens Networks and other xTCA users see particular benefits of COTS building blocks developed by a COTS management controller specialist who is immersed in the multitude of requirements embodied in these wide-ranging specifications. In addition to participating in PICMG interoperability events and applying commercial test suites, these specialists need to demonstrate a deep commitment to specification compliance and interoperability in management functionality.

What functions does an ATCA shelf-level management controller provide?

Figure 1 shows the context and major subsystems of a typical Manager, including the following:

  • Shelf Manager physical instances
  • Shelf Manager interface options
  • Shelf Manager subsystems

The logical Shelf Manager includes two physical instances, active and standby. Via a Software Redundancy Interface (SRI) and Hardware Redundancy Interface (HRI), the standby instance keeps tabs on what is happening in the shelf so it can take over quickly if necessary. Both physical Shelf Manager instances are connected to both ATCA hub boards in the shelf so that a higher level system manager has continual access to the logical Shelf Manager in case of a malfunctioning hub board, Shelf Manager instance, or both.

Figure 1: Dual redundant shelf manager instances offer system manager interfaces and implement key functional blocks to manage shelf operation.

Several interface options are provided by the logical Shelf Manager for use by the system manager, though the only specification-mandated interface is an IPMI-oriented Remote Management Control Protocol (RMCP). Others, including a Command Line Interface (CLI), Simple Network Management Protocol (SNMP), or a web interface (HTTP) are typically available as well. One popular interface is a set of Application Programming Interfaces (APIs) called the Hardware Platform Interface (HPI), which is maintained as an open standard by the SAF.

The logical Shelf Manager has a variety of subsystems, responsible for cooling, fabric connectivity, power, and other management areas. There is typically also a Shelf Adaption Layer to adapt the Shelf Manager for operation in a particular shelf, with its complement of auxiliary resources, such as power entry modules, fan trays, and the supported population of boards, each with its local management controller.

What is HPI’s role in the management of ATCA systems?

HPI is widely used in ATCA systems as the bottom layer in system manager applications to isolate those applications from the implementation details of the hardware platforms they supervise. In addition to the implementation-independent APIs for HPI, there is an open source implementation of these APIs called OpenHPI, which is widely used as well.

One approach for implementing HPI shown in Figure 2 uses OpenHPI, which binds an OpenHPI library into each system manager application. The library receives API calls from the application and uses a Remote Procedure Call (RPC) protocol to pass them to a server, which acts on the API calls and returns the results to the library. The server is an independent application connected to the library via IP link.

Figure 2: HPI can be implemented via OpenHPI by interfacing to a Shelf Manager via RMCP, or built into a Shelf Manager as a native interface.

The OpenHPI server implements a plugin architecture, which allows installation of a software plugin that realizes the implementation specifics of each API call in a manner suitable for a particular type of underlying managed system. For instance, when the managed system is an ATCA system, one popular approach is for the plugin to use RMCP to communicate with the Shelf Manager, since RMCP support is required in every compliant ATCA Shelf Manager.

A second approach for implementing HPI, also in Figure 2, is to include it as a native interface (a peer interface to the mandatory RMCP) to the Shelf Manager. This can be very convenient if the built-in HPI supports the same RPC protocol used by the OpenHPI library to interact with an OpenHPI server. In such a configuration a single system manager application can be connected with either or both types of HPI implementation, and the application need not care which. Making HPI a built-in Shelf Manager interface can enable substantial implementation efficiency optimization due to substantial overlap between the responsibilities of an ATCA Shelf Manager and an OpenHPI server with an ATCA-focused plugin.

There is one more crucial consideration, however. The HPI APIs are intentionally abstract (so that they can encompass a wide range of underlying managed systems). There are many ways to map those abstract APIs to the specifics of an ATCA shelf, as represented by an ATCA Shelf Manager. In order to achieve independence of a system manager application from the specifics of the ATCA platforms that it manages, there needs to be common mapping. Fortunately, the SAF has adopted an HPI-to-xTCA mapping specification. An OpenHPI plugin or built-in HPI implementation must comply with this mapping to enable the ATCA hardware independence promised by HPI.

What functions does an ATCA board management controller provide?

The IPMC on an ATCA board provides local management of the board (such as monitoring temperature, power, fabric connectivity) and represents the board to the Shelf Manager. The Carrier IPMC variant additionally handles overall management of AMC modules on an ATCA carrier board. Each of the AMC modules also has a local Module Management Controller (MMC) to monitor the management functions of the module and represent the module to the Carrier IPMC.

Figure 3 shows key functional blocks of an IPMC, along with a photo of one possible IPMC implementation based on the Microsemi SmartFusion intelligent . The small image is to scale with the ATCA board to demonstrate the tiny fraction of the ATCA board area taken by a typical IPMC.

Figure 3: Functional blocks and interfaces of a typical ATCA IPMC, including a photo of a physical IPMC implementation with a variant photo to scale with the board.

Additional IPMC functions

Many of the functional blocks and interfaces in Figure 3 are fairly obvious, but we will elaborate briefly on a couple of them.

A LAN-attached interface provides a connection to one or more of the shelf’s built-in communication fabric(s). Providing such a connection for an IPMC supplements the mandatory Intelligent Management Platform Bus (IPMB) connection with a much higher throughput (typically Ethernet) path for management traffic.

The payload portion of an ATCA board implements the main functionality of the board (such as a fabric switch or a farm). The functions of an IPMC are critical to board operation but must be implemented as compactly as possible to leave maximum room for the payload. The payload (perhaps one or more powerful Intel processors) can communicate with the IPMC via its payload interface. The IPMC also manages power for the payload, which is not powered until the IPMC has successfully negotiated a power budget with the Shelf Manager.

Example set of COTS ATCA/AMC management building blocks

One set of widely used and mature ATCA/AMC management building blocks is from . All the management controller types defined in the ATCA/AMC architecture are covered in this set, which is incorporated in tens of thousands of ATCA shelves deployed around the world.

The newest (fourth) generation of Pigeon Point’s Shelf Manager solutions is the ShMM-700R, a SODIMM module that is smaller and less expensive than its predecessors, but still provides the hardware and firmware core of an ATCA Shelf Manager suitable for use in a wide range of ATCA shelves. The ShMM-700R supports all the functional blocks and interfaces shown in Figure 1, including a built-in implementation of HPI called IntegralHPI, shown in Figure 2. Furthermore, Pigeon Point Systems also supports an implementation of OpenHPI for customers who prefer that approach. Both IntegralHPI and Pigeon Point OpenHPI fully comply with the SAF’s HPI-to-xTCA mapping specification so that system manager applications can operate equivalently in both contexts.

Because it is integrated with the Pigeon Point Shelf Manager, IntegralHPI can be dramatically more efficient than OpenHPI, especially during shelf startup time on a large and fully populated shelf. This “shelf discovery” time can happen in seconds with IntegralHPI, versus minutes with OpenHPI, making a newly powered shelf available for HPI-based management much sooner.

The current generation of Pigeon Point board and module level management controller solutions are based on Microsemi’s SmartFusion intelligent mixed signal FPGA. These BMR-A2F (Board Management Reference for SmartFusion) solutions pack extensive functionality and provisions for customer extensions into compact and cost-effective management controllers for customer boards or modules.

What does the future hold for xTCA hardware platform management?

One immediate driver for the continued evolution of the ATCA layer is revision and extension of the key base specifications, such as PICMG 3.1 R2, covering 40G Ethernet fabrics, and PICMG 3.7, the ATCA Extensions initiative, both nearing completion and both of which include management layer extensions. A second driver is specifically in the management area: augmenting the set of HPM.x specifications. Next up are the HPM.2 and HPM.3 specifications for LAN-attached IPMCs previously mentioned.

Considering a longer horizon, one perspective comes from the SCOPE Alliance, which brings together the world’s leading TEMs and their supporting ecosystem suppliers. SCOPE has profiled a successor to the ATCA architecture, targeted for use across the 2015-2025 timeframe, and recommended that the ATCA hardware platform management layer, with extensions, be reused. So this layer is likely to evolve with us for a long time to come!

Mark Overgaard is the Founder and CTO of Pigeon Point Systems. Within PICMG, he chairs the HPM.x subcommittee.

Pigeon Point Systems