Server Power Breakdown by Components
Figure 4 highlights how power is consumed on average within an individual server. Processors and memory consume the most amount of power, followed by the power supply efficiency loss. Disk drive power only becomes significant in servers with several disk drives.
Processor power consumption varies greatly by the type of server processor used. Power consumption can vary from 45W to 200W per multi-core CPU. Newer Intel processors include power saving technologies, such as Demand Based Switching and Enhanced Speed Step technology. These newer processors also support power saving states such as C1E and CC3. Multi-core processors are much more power efficient than previous generations. Servers using the recent Quadcore Intel(r) Xeon(tm) processors can deliver 1.8 teraflops at peak performance using less than 10,000 watts of power. Pentium(r) processors in 1998 would have consumed about 800,000 watts to achieve the same performance. Server processor power consumption will also vary depending on the server workload.
Figure 5 illustrates how processor energy efficiency (e.g., performance per watt) increases as server utilization increases for a typical workload. Tuning workloads with optimized processor utilization can greatly affect power consumption and energy efficiency.
By taking average processor utilization over a defined period of time, it is possible to calculate an estimate of the power consumed for that period. Many server workloads scale linearly from idle to maximum power. When you know the power consumption of a server at peak usage and at idle it becomes a simple arithmetic operation to estimate power usage at any utilization rate.
Estimating Power Consumption
An estimate of power consumption (P) at any specific processor utilization (n%) can be calculated if power consumption at maximum performance (Pmax) and at idle (Pidle) are known. Use the following formula:
For example, if a server has a maximum power draw of 400 watts (W) and an idle power draw of 200W, then at 25 percent utilization the power draw would approximate to:
In this example, if the server was running at that average utilization for a 24 hour period, then the energy usage would equate to the following:
Through empirical measurement of various servers using a power meter this approximation has proven to be accurate to within ±5 percent across all processor utilization rates.
The next largest consumer of power in a server is memory. Intel processor power levels are being well controlled and capped with the latest generations. However, power consumption by memory chips is growing and shows no signs of slowing down in the future. Furthermore, applications continually seek more memory. Here are some of the reasons why demand for memory is growing in servers:
- Increases in processor core counts in latest servers; the more cores, the more memory can be utilized in a server
- Increasing use of virtualization; data centers are adopting virtualization in increasing rates
- In new usages by Internet Protocol Data Centers, such as Google and Facebook, with memory intensive search applications
Memory is packaged in dual in-line memory modules (DIMMs) and these modules can vary in power from 5W up to 21W per DIMM, for DDR3 and FB-DIMM (Fully Buffered DIMM) memory technologies. The memory in a server with eight 1GB DIMMs can easily consume 80W. Many large servers now use 32 and 64 DIMMs, resulting in more power consumption by memory than processors.
For each generation of memory technology, there are key physical and electrical attributes of the DIMM that contribute to its power consumption and bandwidth. The Dynamic Random Access Memory (DRAM) packaging type and die count, number DRAM ranks on a DIMM, data transfer speed and data width define the DIMM capacity and power requirements. DIMMs can have registers, known as RDIMMs or be without registers, known as UDIMMs (Unregistered DIMMs). RDIMMs consume slightly more power than UDIMMs.
Figure 6 shows how power consumption differs among RDIMMs using DDR2 and DDR3 technology. For power consumption of the latest DIMMs, check the websites of the leading memory manufacturers.
Power consumed by DIMMs is typically measured in Active and Idle Standby states. Active Power is defined as: L0 state, 50 percent DRAM bandwidth, 67 percent read, 33 percent write, with primary and secondary channels are enabled. The DRAM clock is active, and CKE is high. Idle Power is defined as: L0 state; Idle (0 percent bandwidth), Primary channel enabled, Secondary channel disabled, CKE high; Command and address lines stable; and the SDRAM clock active.
On average, DDR3 DIMMs use 5W-12W when active. DIMMs from different vendors will vary based on their manufacturing process for the DRAM components and the components/configuration they use to make the memory module. Additionally, memory power consumed will vary depending upon the application and workload running.
Table 2 shows sample RDIMM power consumptions in 2008, for DDR2 technology running at speeds of 667MHz. Table 2 highlights that power consumption by memory products varies widely among suppliers and configurations. For power consumption of the latest UDIMMs or RDIMMs, check the websites of these vendors.
As DIMMs increase in capacity, going from 4GB in 2008 to 16GB or 32GB in the near future, their power consumption will increase. DIMMs will also increase in speed over time, which increases the power consumption of the DIMM. Table 3 shows DDR3 RDIMM raw cards, DRAM density, capacity and the forecasted power use based on different speed targets of 1066MHz, 1333MHz and 1600MHz. Power is forecasted to trend higher with the memory speeds of 1866MHz and 2133 MHz. As Table 3 shows, memory power can vary significantly depending upon the memory technology used, the memory configuration, and the vendor.
Reducing Energy Consumption by Memory Subsystems
Cooling of memory is increasingly challenging and requires additional power in most server systems. In the past, memory bandwidth requirements were sufficiently low that memory was relatively simple to cool and required no thermal enhancements on the DIMM, no thermal sensors, nor throttling. The opposite is now true. The thermal analysis of the memory module includes the power of each component, the spacing between the memory modules, air flow velocity and temperature, as well as the presence of any thermal solution (e.g. heat spreader).
DIMM memory is typically downstream from the processor, hard disks and fans, and therefore has a local higher ambient temperature. In typical server system layouts, cool air will flow from one end of the DIMM to the other, with the hottest DRAM component usually being the last one on the same side as the Register. However this conclusion is not consistent across all DIMM formats. For example, the fully buffered DIMM's hottest DRAM is near the center of the DIMM card, next to the Advanced Memory Buffer.
Memory thermals are important because good memory thermals improve system performance in addition to requiring less power. The thermal characteristics of memory subsystems are important because when memory subsystems operate at lower temperature, system performance improves and overall system power consumption is less. The memory thermals are characterized as a function of fan speed and preheat to the DIMMs. The required cooling capability in watts per DIMM varies depending upon whether the DIMM has a Full DIMM Heat Spreader (FDHS) or not and whether the DIMM is in double refresh or not.
A DIMM under double refresh has a case temperature specification of 95°C rather than 85°C, thereby enabling a higher overall safe system temperature at the expense of a small power loss and slightly increased power consumption. The impact of double refresh (85°C versus 95°C) is substantial improving cooling capability by approximately two to three watts resulting in a significant improvement in memory bandwidth capability.
