Interest in xTCA rising at CERN as experiments plan for DAQ upgrade
At the European Organization for Nuclear Research, or CERN, interest in AdvancedTCA (ATCA) and MicroTCA (mTCA) as platforms for future Data Acquisition (DAQ) and control systems is rising rapidly. As several independent teams in collaborating institutes are already involved in the development of AdvancedMCs (AMCs) and ATCA blades, CERN's Electronics Support Group is trying to prepare the ground for a smooth introduction of this technology into experiments.
Large physics experiments like the ones done at CERN's Large Hadron Collider (LHC) are not possible without highly sophisticated DAQ and control systems. Although very different at the level of their physical implementation, the DAQ systems of the four main LHC experimental detectors (more on the ATLAS, CMS, ALICE and LHCb experiments can be found at http://opsy.st/CERNexperiments) have a common high-level architecture.
Three classes of electronic systems can be identified in every experiment. Directly attached to the detectors, one can find the first class: highly customized front-end systems that have to satisfy very demanding requirements with respect to aspects such as tolerance to radiation and magnetic fields, high integration density, and low material budget. At the other end of the DAQ chain, one finds the third class of electronics: large farms of commercial computer systems used for online analysis of the data. The bridge between these two ends forms the second class of systems, which interests us today: standards-based back-end electronics that receive raw data from the detectors, processes it, and then forward it to the computing farms (See Figure 1, page 14). In most cases, the processing of the data is done on FPGAs deployed in custom-designed modules housed in commercial subracks and controlled by commercial processor modules.
CERN's DAQ electronics history
As CERN already had a long tradition of using VMEbus, when the middle layer electronics for the LHC experiments were designed about 10 years ago, it was natural to implement most of the custom electronics in 6U and 9U VME64x format. Since the start-up of the LHC and its experiments in 2008, several hundred VMEbus systems have been in permanent operation and have, so far, shown an impressive reliability. Due to the low data transfer bandwidth of the backplane and, by today's standards, the inefficient data transfer protocol, VMEbus was never used to transfer the physics data into and out of the custom modules, but rather played an important role as a bus for the control and monitoring of the custom electronics. The transfer of the physics data into and out of the cards is performed by fiber optic or, less frequently, electrical links connecting directly to the front panels of the custom modules or their Rear Transition Modules (RTMs).
Traditionally, electronics for the CERN experiments are developed in a highly independent manner at High Energy Physics (HEP) institutes all over the world. The often cutting-edge requirements in terms of integration density, data throughput, and latency necessarily lead to a large number of highly customized designs, which sometimes even deviate intentionally from standards in order to optimally reach the required performance. Eventually, all of these external developments have to be integrated at CERN into one single system. This is only possible if, in the interest of interoperability and maintainability, the diversity of the components is limited from the start.
Upgrade cycles for the LHC
Even though some important discoveries have already been made by the ATLAS and CMS experiments, the LHC experiments are still very young. During the next 10-15 years they will continuously evolve in order to keep track of the increasing luminosity (the rate at which particles collide) of the LHC machine, and to match evolving physics requirements. It is predicted that over this period, the data volumes that the DAQ systems will be required to handle will grow by a factor of about 10.
Every three years, the LHC research programs will be stopped for about one or two years to allow for major upgrades of both the accelerator and the experiments. Such "Long Shutdowns" as they are known, are scheduled for 2013-2014, 2018, and 2022-2023, and are the only time windows that allow for the deployment of new electronic systems on a large scale both inside and outside the experiments.
During the last years, it has become obvious for many reasons (such as data transfer performance, power density and cooling limitations, and the unclear availability of COTS components for another 10-15 years) that VMEbus is not acceptable for these upgrade programs, and groups of experts have independently looked for alternatives. ATCA and mTCA are emerging as the technologies of choice for the next one to two decades. Some of the properties of the xTCA standards that contributed to their rising popularity at CERN are:
- A choice of form factors (single- and double-width AMCs in mTCA shelves as well as ATCA blades)
- A protocol-agnostic backplane allowing for the choice of data transfer protocol (PCIe, Ethernet, or custom for communication between FPGAs) based on high-bandwidth point-to-point links
- Redundancy at the infrastructure level (PSU, cooling, shelf management)
- A large base of suppliers for COTS components
- Positive feedback from early adopters such as the German Electron Synchrotron (DESY)
- Long expected lifetime of the standard
Already in 2013-14, a small number of pilot ATCA and mTCA systems will be installed in the experiments. The main aim, however, is to replace most of the VMEbus based systems in the maintenance periods of 2018 and 2022-2023.
DAQ applications for xTCA in the LHC
There are two main areas of application for xTCA in CERN's back-end electronic systems: Trigger and Read-Out.
Trigger systems serve to rapidly discriminate physics events that are likely to contain signatures of particles of interest from the many background events. A trigger system receives small packets of data at very high rates (currently 20-40 MHz), and has a very tight time budget on the order of a few microseconds for the computation of a decision. In such a system, each custom module receives a fraction of the data and has to communicate with other modules in the same shelf with very low latency and high speed. This necessarily asks for a very high integration density, as well as a large amount of I/O connectivity. The distribution of high frequency and low jitter clock signals in the shelf, required for the synchronization of the modules, is another challenge of a trigger system. Depending on the requirements of the specific application, both ATCA and mTCA will be used as platforms for trigger systems.
The main task of the Read-Out systems is to receive data from as many channels of the detector as physically possible and prepare that data for processing by the downstream elements of the DAQ chain. Data from the detectors (which typically have of the order of 100 million channels) usually arrives via optical fibers. On the signal processing boards, FPGAs receive the data, apply calibration algorithms, perform zero suppression or other data compression algorithms, and finally output the data toward the next layer, once more using optical interfaces. Because of its increased space on the front panel and RTM, as well as the larger PCB area of its blades that makes it easier to place and cool large numbers of FPGAs, the preferred format for Read-Out systems might well be ATCA.
CERN setbacks and solutions with xTCA
For the last two to three years, independent teams of CERN electronics designers have gathered experience in xTCA through the development of prototypes and the evaluation of commercial components. When complementing the experience gained from these activities with interactions from industry and early adopters of xTCA in other research institutes, however, it has become evident that there is a price to be paid for the new features.
Despite the fact that the large amount of options in xTCA is a blessing for the design of specialized electronics, so it can also be a curse from the perspective of system integration and interoperability. The complex control structures of xTCA are equally double edged. On one hand they provide an unprecedented amount of control over the deployed electronics through features such as Electronic Keying (E-Keying), plug and play based on FRU information, self-regulation of the cooling system, and large numbers of embedded sensors. On the other hand, the complex IPMI-based controls software that has to be provided for every custom module is non-negligible overhead in the development and system integration process. Finally, the challenges of PCB design with the large number of signals in the 5-10 GHz range must not be underestimated.
The Electronics Support Group for experiments at CERN has been involved in the development of the custom electronics of all four LHC experiments, and provides central services such as evaluation and maintenance of common elements like crates, power supplies, and embedded processors. In order to provide these services for future xTCA systems, the group began several related activities in 2010. Via its PICMG membership, the group has participated as an observer on behalf of CERN in the standardization processes of mTCA.4 and PICMG 3.8, and is also closely following the evolution of the standards in order to identify new trends (See Sidebar 1). Other members of the group are working on the evaluation of commercial components for both ATCA and mTCA systems with the aim of identifying elements of common interest, such as shelves and power supplies, which could eventually be centrally supported. In parallel, a number of components such as a Module Management Controller (MMC) mezzanine for AMCs, a set of AMC load boards and RTMs, and a complex FPGA and FMC-based AMC for the test and development of Gbps optical interfaces have been designed. The experience gained in these activities will also enable the group to provide advice to designers of new custom modules.
xTCA prospects at CERN
The CERN group, in cooperation with the experiments, is aiming at converging on the formulation of common requirements for xTCA systems, and the development of comprehensive support for the respective system elements. Another area of activity will be the formulation of acceptance and interoperability tests for xTCA components for use at CERN.
This is only the start of something that is likely to become a long-term commitment to xTCA by CERN. The groups responsible for the accelerator control systems are independently evaluating xTCA for their purposes, which can be quite different from those of DAQ systems in experiments.
It is the diversity of requirements, the need for performance, and the limitations set by boundary conditions that drive the choice of technologies. As a result of this, different teams developing electronics for CERN experiments will come to different conclusions and implement their systems in different ways. In order to manage this complexity, the use of common elements will be encouraged wherever possible.
Another key factor will be a good relationship with industry, which can be involved in many ways such as the provision of COTS components or the production of modules designed by CERN.
Overall, the direction modular electronics will take at CERN appears to be set. Over the next years, xTCA will gradually take over from VMEbus, yet many detailed technical decisions remain to be made in order to achieve a smooth introduction of this new technology.
CERN | www.cern.ch