xTCA lends benefits to high-energy physics

Smile, atoms and molecules, you are on high-speed camera, thanks to work being done by scientists with a new interest in xTCA.

1For about two years the High Energy Physics community has been developing industry specifications for experiment control and instrumentation systems based on the PICMG AdvancedTCA and MicroTCA (xTCA for short) specifications. The hardware specifications are nearing completion, and should soon be widely available. Here the authors share the rationale behind these specs and preview their content.

So why are we adapting these PICMG technologies? The physics community has long relied on standardized hardware for instrumentation, like NIM, CAMAC, Fastbus, and VME. A proven platform with an existing ecosystem of suppliers makes for lower costs and shorter lead-times, thereby accelerating the implementation of experiments.

In the last few years the importance of machine availability has come front and center. New and proposed large research machines and the associated detectors have gone through detailed analysis of the “real” costs of such installations. It has become painfully evident that downtime is costly, in some instances hundreds of thousands of dollars per hour. This led the Physics community to look at architectures with management features that support high availability, a path that brought them to PICMG and the xTCA specifications.

Target applications

We’ve been targeting three classes of physics machines and associated experiments:

·        Colliding Beams (electron, proton, heavy ion) probe subatomic – 3 km linac, 27 km ring

·        Light Sources (high energy X-Ray) probe molecular, atomic – 1 km linac, 2 km rings

·        Plasma Fusion Reactors – Vessel in high H field

Within these experiments we are interested in three classes of equipment that are candidates for xTCA implementation.

·        Machine Controls & Instruments

·        DC and Pulsed Power

·        Small to Very Large Detector Systems

 

Let’s look at the machines that are under construction or being upgraded. The German Physics Research Laboratory Deutsches Elektronen-Synchrotron (DESY) is constructing a new light source for studying chemical and molecular processes at the atomic scale. The XFEL is 3 km long and will produce extremely short and powerful X-ray flashes. Much of the control for this machine will be built with the new proposed PICMG MTCA.4 specification. This lab has done much of the work of prototyping and testing the ideas used to write this document.

Shedding some coherency on the subject

The SLAC National Accelerator Laboratory in the United States (Figure 1) has adapted its old two-mile long linear accelerator from colliding electrons (among other uses) into driving the Linac Coherent Light Source (LCLS) with the electron beam to make intense bursts of high-energy X-rays in a magnetic device called an undulator. A third of the accelerator is already in use producing pulses 100 femtoseconds long with 1012 X-ray photons. Consecutive short intense pulses act like a high-speed camera taking stop motion pictures of atoms and molecules. To take full advantage of this tool the linac RF electronics will be upgraded with the goal of producing pulses as short as 10 femtoseconds. At the same time all controls will migrate from the obsolete CAMAC to the MTCA.4 platform.

21
Figure 1: Hoisting a 30 m section of superconducting linac into position for XFEL test beam machine. Photo courtesy SLAC National Accelerator Laboratory

 

The ITER fusion tokomak machine is being built in southern France. Countries around the world are taking part in this project. Availability, high-speed data acquisition, and maintainability are paramount to its operation. The goal for this fusion machine is to generate 500 MW with 50 MW input power to demonstrate viability as the energy source for the future.

The largest high-energy physics project in the planning stage is the International Linear Collider (ILC). (See Figure 2.) In its full configuration it comprises two linear accelerators, each of which is about 20 km long. The purpose is to collide electrons and positrons. This project if successfully funded will be an international cooperative venture. In addition to the magnets, RF cavities, and a host of other systems, the amount of electronics is enormous. The number of machine sensors is in the millions and for the physics detectors, tens of millions. Sensors range from DC for voltage, temperature, vacuum monitoring, and the like to fast RF feeedback systems for beam control to multi-gigahertz speed data handling for the millions of detector channels.

22
Figure 2: Two-tunnel model of the International Linear Collider. (Detector area not shown.) Courtesy J. Liebfritz, FNAL

 

All these large science projects depend on the rapid acquisition and processing of terabytes of data. Along with that the usual myriad sensors need to be monitored. These control vacuum systems power supplies, temperatures, etc. The number of channels in some of the large detectors such as those at CERN can be as large as the number in the control of the accelerators. Many times the electronics is in areas not readily accessible because of radiation or location, so being able to remotely diagnose is critical.

Availability is an issue of prime importance at facilities costing as much as $10 billion. The cost of being down at the ILC has been estimated at $150,000 per hour. Having a system such as xTCA gives the system designer the ability to provide redundancy where required. The monitoring and control features not only can identify failures, but can also be used to predict problems before they occur. This all greatly reduces MTTR and improves uptime.

xTCA extensions for physics

The xTCA architectures fulfill almost all of the requirements for High Energy Physics: availability; monitoring; control; and commercial support. The only issue was the lack of rear I/O in MicroTCA and the unspecified Rear Transition Module (RTM) in AdvancedTCA. The cabling problems with the heavy I/O means that the cable plant needs to get off the front panel and into the rear. Many high cables are sensitive to motion at the sensitivity we require. Also, when a module does need changing not having to unplug and replug a “rats nest” of cables (probably ruining them or misplugging them in the process), the MTTR goes down and the reliability of the change goes up.

From the beginning the AdvancedTCA specification anticipated the need for rear I/O, reserving a portion of the AdvancedTCA blade edge for that purpose. Initially a form factor was specified for the AdvancedTCA Rear Transmission Module, but the spec did not define connectors or management features. Recently, however, that management of RTMs has been specified, leaving the problems of front to rear connectivity and power distribution to be resolved by user communities.

The right side of Figure 3 shows a typical AdvancedTCA front board with the associated RTM on the left. Note the direct connection between them using the same connector family that AdvancedTCA front board to back/midplane connections employ, which is generically identified as the ADF connector. Figure 3 (bottom) also shows the presence of a physical 64-position keying pin and block assembly, which has been specified by AdvancedTCA since its initial release.

23
Figure 3: Massively parallel processor on AdvancedTCA with RTM for terabit I/O per AdvancedTCA shelf. Photo courtesy SLAC.

 

Figure 4 shows the RTM concept for physics applications. It uses the same data connectors as AdvancedTCA in three positions, and adds a specially designed connector for power and management connections between front and back.

24
Figure 4: One of the issues addressed by the xTCA for Physics PICMG subcommittee was the lack of rear I/O in MicroTCA.

 

From its beginnings MicroTCA has been exclusively a front I/O architecture. The High Energy Physics community found the double-height variant of this form factor attractive but needed a rear I/O solution. The scope of this effort was much greater, involving the definition of a MicroRTM form factor, and a subrack to accommodate it, as well as connectivity and management solutions.

A rear I/O solution for MicroTCA

Figure 5 shows the concept for a double height MicroTCA front module and its mating MicroRTM. The MicroTCA front module uses the currently specified connectors for back/midplane connection with the subrack but specifies a variant of the ADF connector for front board to RTM connections. This block of connectors includes power and management as well as data pins.

25
Figure 5: A double-height MicroTCA module and its mating MicroRTM.

 

The MicroTCA vendor community has been very supportive of this effort, especially so in their willingness to develop prototype hardware for this new packaging concept.

A six-slot shelf developed cooperatively by DESY and Schroff can be seen in Figure 6.

26
Figure 6: DESY and Schroff GmbH teamed to create a six-slot MicroTCA shelf. Courtesy K. Rehlich DESY and Schroff GmbH.

 

Elma has developed a 12-slot shelf concept, as seen in Figure 7.

27
Figure 7: A 12-slot MicroTCA concept from Elma. Courtesy Elma GmbH.


The development of the PICMG specifications defining the Physics extensions is nearing completion, and development of experiments based on them is underway at a number of laboratories worldwide. A new generation of standards-based platforms for these applications is emerging with an unprecedented level of performance and maintainability.

Organization of the high-energy physics effort within PICMG

The PICMG for Physics Committee was initiated in late 2008 at the Dresden xTCA workshop held in conjunction with the annual IEEE Nuclear Science Symposium. In May 2009 three new PICMG Technical Committees were formed: (1) Coordinating Committee (CC); (2) Hardware Working Group (WG1); and (3) Software Working Group (WG2). The CC oversees the application to PICMG for formation of new working groups and has a long-term role as a liaison between PICMG and Physics interests. WG1 develops hardware specification extensions to accommodate physics I/O and timing needs in both AdvancedTCA (extensions to manage the RTM and definition of backplane extended options for precision timing and triggering protocols) and MicroTCA platforms (specification of new managed MicroRTM as well as backplane extended options to accommodate precision timing and triggering). Interoperability with existing AMC products is a requirement of the new specifications. Similarly WG3 develops generic software solutions to enhance interoperability of physics modules developed or specified to industry by different laboratories. Its road map includes four general areas: Routing and Protocols; System/Rack/Module Management; Operating System and Infrastructure; and Processing and Operations Libraries.

Specific WG1 tasks include development of three specifications each with a number of subtasks. These include:

·        Specification of interface connectors and standard pin assignments

·        Shelf management of RTMs via AdvancedTCA or AdvancedMC modules (in MicroTCA)

·        New mechanics for shelves and modules

·        Precision timing and triggering protocols

·        Keying and E-keying

 

WG2 tasks include:

·        Protocols for data transport, synchronization and command/control

·        Component management/failover/update

·        Common models for hardware API, process/thread, I/O devices and communications

 

WG1 is close to releasing two specifications for PICMG approval, while WG2 is moving toward developing guidelines as determined by user interest and demand.

All committees report directly to the PICMG Technical Officer while the CC has an oversight function for present priorities and future WG tasks. By PICMG rule, once the WG SOW is complete the Technical Subcommittee disbands. New TS’s are formed when new tasks arise, and a proposal to form a new group is approved by PICMG.

In addition to the standards work the community has held a total of four major workshops. The program and presentations from the 2010 workshop held in conjunction with the Lisbon IEEE Real Time Conference in May 2010 can be found at:

http://portal.ipfn.ist.utl.pt/rt2010/ATCA%20Workshop%20Presentations/Lisbon%20xTCA%20Workshop%20Presentation-Program-Abstracts_052710/

Robert Downing has his own consulting company. He has worked on high-speed data acquisition systems for Fermi National Accelerator Laboratory. Currently he is consulting with the Stanford Linear Accelerator Center. Robert has worked on standards committees for electronics over the last 30 years beginning with the National Bureau of Standards NIM Committee. He is the past chair of the VME Standards Organization and is a member of the PICMG xTCA for Physics Coordinating Committee and chair of the xTCA for Physics Hardware committee.

Ray Larsen is currently Deputy Director of the Accelerator Engineering Division at SLAC. He has spent his career in government and university research laboratories except for seven years founding a startup instrumentation company. He worked on four modular instrument standards for physics and is currently chair of the PICMG xTCA for Physics coordinating committee. His work is supported by the DOE International Linear Collider research program. He is a Life Fellow of IEEE recognized for work in high-speed sampled analog data systems and instrument systems, and co-holder of three patents in switched- capacitor ASICs. He holds BASc and MASc degrees from UBC and the Degree of Engineer from Stanford, all in Electrical Engineering.

Dick Somes served as the Vice President/Technical Officer of PICMG from 1997 through 2008, and continues to participate in specification development as an Associate Member. He received a BS from Tufts University in 1966 and an MS from the University of Pennsylvania in 1971, both in Electrical Engineering.

PICMG

www.picmg.org