High-energy physics: Controllers for X-ray laser accelerators
DESY (Deutsches Elektronen-Synchrotron) - one of the world's leading particle-accelerator centers, conducting top-level international research into the fundamental relationships of matter - was tasked with finding a successor technology for its control and data-acquisition systems, which were based on VME and proprietary hardware, some of which was more than 30 years old. The main requirements: better signal quality, higher bandwidth, and support for several decades of operation, allowing for a phased migration in current installations.
The chosen technology also had to act as the platform for the new X-Ray Free-Electron Laser (XFEL) accelerator (www.xfel.eu); DESY is the main shareholder in the European XFEL, which will generate ultrashort X-ray flashes – 27,000 times per second – with a brilliance that is a billion times higher than that of the best conventional X-ray radiation sources. The facility, the only one of its kind in the world, opens completely new research opportunities for scientists and industrial users. As the main shareholder, DESY plays a major part in the construction and operation of the facility.
From the beginning, N.A.T. worked with DESY’s engineering team and the ecosystem of MicroTCA companies to enhance the MTCA.0 standard for the physics community’s additional requirement. The MTCA.4 standard allows users to build solutions – scaling from small to large high-performance systems – ranging from single-unit installations to large installations of more than 200 crates with a timing synchronization in the picosecond range.
Engineers conducted rigorous benchmarking of various technologies – such as AdvancedTCA, VXI, VPX, and MicroTCA. MicroTCA.4 technology, with up to 20 systems per project, was successfully tested in existing installations and a newer FLASH-II accelerator. The experiences and results exceeded DESY’s expectations, thereby paving the way for the installation at the European XFEL. Assembly of this facility was completed at the end of 2016. Starting in the summer of 2017, scientists from around the world were able to use two out of the six initial planned scientific instruments for the intermediate term. The accelerator is now operating at a temperature of -271 °C and generates extremely bright and short X-ray flashes with laser-like properties in a self-amplifying process.
Why build it?
The technology DESY has used for high-speed, high-performance control of the beams in its accelerators, including PETRA and FLASH, were no longer state-of-the-art and suitable for the next generation of accelerators. In the year 2005, several task forces for different experimental projects had to define the requirements and find a new technology as a successor for the VMEbus and as a replacement for some of DESY’s proprietary hardware.
In the years after 2005, there were two major trends in embedded systems: One was to integrate high-speed serial interfaces with existing parallel bus architectures, offering backwards compatibility and forming new standards, such as VXS, CPSB, and CPCI-Serial. Another trend was to cut ties with existing architectures and ignore backwards compatibility. Instead, some companies adopted the latest high-speed serial interfaces (VPX technology is an example), while some standards organizations combined the latest switching interfaces with additional features, such as standardized management hardware and software. The standards emerging from these efforts are AdvancedTCA (ATCA) and MicroTCA (MTCA).
These trends made it very difficult to select the right technology, as no single standard could fulfill all the requirements of high-energy physics users. Moreover, in the early days it was unclear which new standard would be accepted in which market and have the longevity that research scientists needed.
The new technology needed to have the same advantages as the VMEbus, including robustness, modularity, scalability, suitability for 19-inch rack mount-ing, and long life cycle support; it also had to overcome the limitations of using a technology that is more than 30 years old. In addition, the functionality of DESY’s own proprietary hardware had to be considered.
Among the requirements:
- Open standard
- 19-inch-rack suitable
- Modular and scalable
- Very small individual systems, very large system installations
- Multiple sources for all components
- A viable merchant market – ecosystem of industrial partners
- Acceptance of the technology outside of the physics market
- Availability and long-term support of operating systems
- Bus bandwidth should be improved by a factor of 10
- An evolution path to increase the bandwidth further in the future
- Use the latest high-speed interconnect technology
- Improvement of signal quality in the system to increase the resolution of the ADC/DAC conversion
- Higher-precision timing system to fulfill the high-precision synchronization demands of a picosecond range
- Availability of different footprints
- Better remote control and monitoring, as the installations are in tunnels with no direct local access during operation
- Trigger and interlock bus integrated in the backplane
- Integrated high-precision (picosecond range) clock distribution
- Better cabling management; rear cabling via rear transition modules
- Reduction in plugging and unplugging cables (a major source of system failures)
- Faster and more reliable board replacement
- Some cables are sensitive to motion, such as RF cables
- Hot-plug support
- Redundant power and cooling
- Support for redundancy of all components, but without adding costs if not implemented
Finding a solution
Between 2009 and 2011, DESY and N.A.T. worked with the MicroTCA ecosystem through the PICMG standards organization to extend the trigger and clock signals in the Zone 1 connector of the MTCA standard. In 2011 the MTCA.4 standard, an evolution of the MTCA.0 standard, was ratified by PICMG to address the requirements of the physics community. The MTCA.4 standard adds rear transition module support and special trigger and low-latency clock distribution buses to the MTCA standard and increases the number of power modules slots from two to four.
During this time, the embedded computing ecosystem developed ATCA boards, AMC modules and RTMs, and built systems based on ATCA and MTCA for meticulous benchmarking. Comparing the signal quality, flexibility, system price, and the availability of commercial off-the-shelf (COTS) products, many physics researchers made the MTCA.4 platform their selected platform technology. It was chosen for accelerators and experiments inside DESY, as well as other Helmholtz institutes in Germany and worldwide in nearly all new low-level RF applications.
Following are several examples of where the MTCA.4 platform was tested intensively and successfully:
Example 1: FLASH – FLASH is a free-electron laser at DESY, which was commissioned in 2004 and has been used for research with shortwave ultraviolet and soft X-ray radiation since 2005. The facility is 260 meters long and generates soft X-ray radiation down to a wavelength of four nanometers (billionths of a meter). Until 2009, FLASH was the only free-electron laser in the world to produce radiation in the soft X-ray region.
FLASH I uses MTCA.0 platforms as a replacement for some of its VMEbus systems to compare both technologies. FLASH I was subsequently extended with FLASH II, which uses mainly MTCA.4 systems with the new timing capabilities. Major parts of the FLASH installation are based now on MTCA.4 platforms and run without disruption, proving the reliability and signal quality and paving the way to next big installation: XFEL.
Example 2: European XFEL – FLASH is a small version of the European XFEL, which is already producing laser flashes for science. The facilities differ mainly in the wavelengths of the light flashes generated. The new European XFEL light source was commissioned and built exclusively using MTCA.4 technology. (Figure 1.)
MTCA.4 systems are used as coupler-interlock master and slave, low-level RF master and slave, diagnostics, experiment readout, vacuum, and magnets.
The tunnel, with a length of 3.4 kilometers, is equipped with more than 200 MTCA.4 systems, most of which are 9U high, 19 inches, wide and contain 12 slots (similar to NATIVE-R9); some are 2U high with 6 slots. The MicroTCA controller hubs (MCHs) are a combination of NAT-MCH-PHYS and NAT-MCH-PHYS80 models with their respective rear transition module (RTM) NAT-MCH-RTM, providing remote and local system control and management, GbE and PCI Express non-blocking switches, and low-latency clock distribution.
Each system contains one or two redundant 1,000-watt power supplies with the potential to upgrade to four of N.A.T’s new NAT-PM-AC1000 units to provide double the power; an Intel processor AMC module with two local SSDs, with an upgrade path to the NAT-MCH-COMex-i7, potentially releasing an AMC slot. The system also contains an NAMC-psTimer, with picosecond timing accuracy; a machine-protection AMC (DAMC-02); a data intercommunication AMC (DAMC-TCK7, with the NAMC-TCK7 as second source); and data-acquisition ADC and DACs, Ethernet switches, digital IO, and other modules from the MicroTCA ecosystem.
XFEL and beyond
Besides being used for research in its own right, FLASH also serves as a pilot facility for the European XFEL. Its operation provides major insights that benefit the European XFEL, which will generate even shorter wavelengths down to one-tenth of a nanometer. At the same time, scientists can use the FLASH facility to continue development for the planned International Linear Collider (ILC) for particle physics. (Table 1.)
MTCA.4 is the ideal standard for a project of the size of XFEL but also for smaller installations. Reusing the same components, even in small systems with only two boards, can be implemented cost-effectively reducing maintenance costs by having a single technology for multiple applications.
DESY is drawing up plans for PETRA IV, upgrading PETRA III to become an extremely narrowly focused, high-resolution 3D X-ray microscope, offering outstanding research prospects for cutting-edge nanoscience and materials science. When completed, it will allow scientists to examine the physical and chemical processes taking place inside materials on all scales – from millimeters through to atomic dimensions.
The PETRA III X-ray source already produces highly focused, brilliant X-ray beams, which are extremely good at penetrating matter. A key parameter for this is the so-called emittance, a measure of the cross-section and concentration of a particle beam inside an accelerator. The smaller the emittance, the better. The planned modifications associated with PETRA IV will drastically reduce the emittance even further, by up to two orders of magnitude. MicroTCA.4 will be part of that success.