Shelf cooling for 400 W boards in AdvancedTCA (and other platforms)

The shelf's task when it comes to cooling is more complex than at first meets the eye.

In this third article in our thermal management series, David focuses on shelf cooling, examining conformance testing, fan swirl, and other issues. The first two articles in the series, Cooling 400 W boards in AdvancedTCA (and other platforms) and Fluid dynamics can be found at advancedtca-systems.com.

Ensuring effective heat transfer from the active devices through heatsinks (where needed) all the way through to the cooling medium is a board thermal design task demanding skilled effort. An appreciation of velocity (LFM), vector management, mass flow, and volumetric flow comes in handy.

At first glance the shelf’s task is simple. It must move the aggregate cooling medium away from the boards in the shelf to the outside environment, that is, either the shelf or room heat exchangers. However, to design a shelf capable of handling a range of boards – from 400 W boards packed with heatsinks with a high airflow impedance to equally demanding boards with respect to temperature sensitivity but with a geometry resulting in a comparatively low airflow impedance – is a significant challenge.

Much good material has been written on the subject of shelf cooling and the airflow geometry, but four points need to be stressed. Some of these challenges were covered in Cooling management: from hardware to middleware, available at compactpci-systems.com/articles/id/?2140.

In the early AdvancedTCA development days a move arose to make reduced shelf height a design goal. A good deal of effort was invested into qualifying a 10U height for shelves holding 200 W boards. The excessive airflow restriction made this ineffective. Regular shelves are available with shelf heights of 12U and 13U. However significantly more airflow is needed as board power increases, so a 14U shelf, allowing three shelves in a 42U rack, should be considered.

Over the past two to five years fan performance has increased considerably. One example of increased performance can be seen by comparing the fans we chose for cooling boards of 200 W+ with the fans we are fitting today to our 400 W+ compliant shelf (Table 1).

21
Table 1: Fan performance comparison. Note: Both flow and pressure are given for the operating points, not the end points.

The noise level increase can seem to be a design problem, but this is not the case. For a high availability/high reliability platform it’s necessary to design for performance exceptions such as fan failures and external cooling failures resulting in high ambient temperatures. The fan operating speed needed to maintain system operation under failure conditions may well have an unacceptable noise level, but with proper shelf design and successful shelf management algorithms, the fans can run quietly at low speeds. A well-designed fan employing effective shelf management algorithms running at 30 ºC ambient may prove quieter than another fan for the same shelf performance [i].

One problem that remains, and in fact has worsened as fan performance has increased, is that of swirl. This is not so critical where the fans operate to extract air at the exhaust of the shelf air path [ii]. It starts to be an issue when fans are positioned at the inlet of the shelf [iii]. It becomes critical when fans are mounted close to the boards, whether in a push, pull, or push-pull configuration [iv]. Typical fan swirl is illustrated in Figure 1[v]. Simulating tangential flow using CFD tools does raise issues of computational constraint. Turning swirl on in a CFD package is not without cost. It slows the solution (a little), and (maybe) takes more grid to resolve the complex flow near the fan. Also, the user has to specify the amount of swirl, plus which way the fan rotates. (Warning: Your idea of clockwise may not be the same as the CFD package clockwise.) A typical flow pattern with and without swirl selected is shown in Figure 2.

21
Figure 1: Typical fan swirl

22
Figure 2: Board slot airflow pattern solved with fan swirl activated takes on the opposite character of airflow pattern solved without fan swirl activated.

Note that Figure 2 shows the flow going directly into the board. The EMC mesh typically fitted to an AdvancedTCA shelf does appear to modify the swirl by introducing a level of flow straightening. However, simulating shelf behavior realistically does require swirl to be activated for every fan. This can mean activation of between 6 and 24 fans depending on the shelf design. It is a burden that can make the simulation computationally infeasible, particularly when simulating fan failures and where different fan speed settings are necessary [1-7]. Solving this problem is one of the many reasons justifying a high-performance shelf characterization tool.

In the previous article in this series, Fluid dynamics, we briefly mentioned board functional testing alongside the airflow impedance measurement of the board. Similarly, the shelf cooling tests should include functional and characterizational testing.

Shelf thermal functional testing is possible by using load boards with variable airflow impedance values and variable thermal dissipation levels tested in a thermal test chamber. The load board will have variable power dissipation levels independent of the supply voltage using set-power-level commands. SDR temperature sensors will enable slot comparisons to be carried out on the shelf.

Air movement expertise

The Air Movement and Control Association International, Inc. (AMCA), backed by more than 80 years of standards development, is the world’s leading scientific and engineering authority on air movement and air control devices. ANSI/AMCA Standard 210-07 is recognized as the authoritative standard for airflow measurement. For this reason, it is argued that a pressure-based shelf characterization tool based on the AMCA standard is the optimum approach.

An analog for airflow, pressure, and airflow impedance is I = V/R [vi]. Measuring pressure is relatively straightforward. Pressure can be read using manometer (tube), magnahelix dial, or silicon trans-ducers, all with high accuracy and repeatability. The airflow impedance variable ‘R’ needs to be repeatable and accurate. Using laser punched perforated steel plates achieves the impedance function. Accuracy is obtained by independent calibration on an ANSI/AMCA Standard 210-07 test unit. The result is an accurate and repeatable measurement of airflow. To meet the PICMG 3.0 requirement for qualification with Standard Air, the tool software needs to know the local atmospheric pressure, the temperature, and the humidity. The test tool needs to be provided with this data either manually or determined by specific sensors.

Airflow testing requires a tool set (Figure 3) that is fully compliant with PICMG board and shelf geometry. This is called out in REQ #11 and #20 of the specification. The model should include the specification compliant bypass paths such as the front board side 1 to side 2 clearances and the large lateral airflow path between the slots in the lower part of the RTM space. The tool should, however, be designed to ensure that there are no additional front (and RTM) panel leakage paths and that the Zone 3 area is blocked between the front board and the RTM. This will enable accurate and repeatable measurement of the airflow that a typical board would experience.

23
Figure 3: A drawing of a tool set designed for shelf airflow characterization. Note: Some information has been omitted from this drawing to protect our IP.

Testing should be fully automated, preferably managed through the tester IPMI and the shelf manager Ethernet interface. The only manual intervention required should be to change the impedance panels. To avoid the risk of incorrect fitting, the impedance panels should be electronically keyed.

A shelf characterization tool should meet the following minimum requirements:

1. Shall measure the volumetric and mass airflow of each front slot.

2. Shall measure the volumetric and mass airflow of each RTM positional.

3. Shall measure the airflow distribution along the depth of the front slot at both the airflow entry and exit.

4. Be able to identify reverse airflow.

5. Shall not exhibit sensitivity to airflow pattern.

6. Front slot pressure range shall be up to 250 Pa (1" H2O).

7. Front slot flow range shall be up to 150 m3/h (88 CFM).

8. RTM pressure range up to 100 Pa (0.4" H2O).

9. RTM flow range to 50 m3/h (30 CFM).

10. Provide an accuracy of better than 2.5 percent, 10 to 55 CFM.

The tool should run automatically with manual intervention only to change im-pedance plates and to immobilize fans for a fan fail measurement. Results are stored in Excel format with all the shelf and tool FRU data appended. Results shall be available to meet all the requirements of PICMG 3.0 section 5.

Next

The next article in this series will discuss the design of the rack and the placement of high-power racks in office layouts. We shall also look at some alternative strategies to obtain effective and low-noise cooling of boards of 400 W or greater without changes to the existing specification.

David Wright is the AdvancedTCA, ATCA300, and MicroTCA Systems Architect at Advanced Platforms Ltd.

Advanced Platforms Ltd. www.advanced-platforms.com david@advanced-platforms.com

References

  1. Kordyban Tellabs, Fan Swirl and Planar Resistance Don’t Mix
  2. Tom C. Currie, Nortel Networks, Modeling Fan-Induced Flow Distortion in a Push-Type Telecom Shelf
  3. J. Hennissen, W. Temmerman, J. Berghmans, K. Allaert, Flomerics, Modelling of Axial Fans for Electronic Equipment
  4. George L. Stefko, Lawrence J. Bober, and Harvey E. Neumannn, Lewis Research Center, Cleveland, Ohio, New Test Techniques and Analytical Procedures for Understanding the Behavior of Advanced Propellers
  5. Kulvir K. Dhinsa, Chris J. Bailey, Koulis A. Pericleous, Centre for Numerical Modelling and Process Analysis University of Greenwich, Turbulence Modelling and Its Impact on CFD Predictions for Cooling of Electronic Components
  6. Emerson Network Power, Shenzhen, China, The Effect of Fan Swirl on PSU Cooling
  7. Gerald Recktenwald, A Flow Bench for Measuring Fan Curves and System Curves for Air-Cooled Electronic Equipment

[i] To be measured during shelf characterization testing

[ii] Figure 3 Cooling management: from hardware to middleware compactpci-systems.com/articles/id/?2140

[iii] Figure 2 Cooling management from hardware to middleware compactpci-systems.com/articles/id/?2140

[iv] Figure 1 and 4 Cooling management from hardware to middleware compactpci-systems.com/articles/id/?2140

[v] See Kordyban Tellabs, Fan Swirl and Planar Resistance Don’t Mix

[vi] Where I = airflow; V = pressure and R = airflow impedance