Skip navigation

Standard Performance Evaluation Corporation

Facebook logo LinkedIn logo Twitter logo Google+ logo
 
 

SPECpower Committee

The increasing demand for energy-efficient IT equipment has resulted in the need for power and performance benchmarks. In response, the SPEC community established the SPECpower Committee, an initiative to augment existing or create new industry-standard benchmarks and tools with a power/energy measurement. Leading engineers and scientists in the fields of benchmark development and energy efficiency, representing a diverse and competitive set of companies, made a commitment to tackle this task. The development of the first industry-standard benchmark that measures the power and performance characteristics of server-class compute equipment began on January 26 2006.

The SPECpower committee includes representatives from AMD, Dell, HPE, IBM, Intel, Microsoft, and the University of Würzburg. Veteran SPEC benchmark developers Hansfried Block, Greg Darnell, and Karin Wulf are supporting contributors.

SPECpower Committee timeline, starting with foundation in 2006 and including benchmark/tool releases.

The SPEC PTDaemon

The power temperature daemon (also known as PTDaemon) is used to offload the work of controlling a power analyzer or temperature sensor during measurement intervals to a system other than the SUT. It hides the details of different power analyzer interface protocols and behaviors from the benchmark software, presenting a common TCP/IP-based interface that can be readily integrated into different benchmark harnesses. Benchmarks already using PTDaemon include SPECpower_ssj®2008, SPECweb®2009, and SPECvirt_sc®2013, the SERT™ suite, and the Chauffeur™ WDK.

SPECpower_ssj2008

In December of 2007, the SPECpower_ssj2008 benchmark was released, which exercises the CPUs, caches, memory hierarchy, and the scalability of shared memory processors on multiple load-levels. The benchmark runs on a wide variety of operating systems and hardware architectures. In version 1.10, which was released on April 15 2009, SPEC augmented SPECpower_ssj2008 with multi-node support (e.g., blade-support). Several enhancements and code changes to all benchmark components, documentation updates, and run and reporting rules enhancements were included in version 1.11, released September 13, 2011 and version 1.12, released March 30, 2012.

The Committee is working on the next SPECpower benchmark, designed for the next generation of servers. This is based on the Chauffeur test harness, supported by the latest PTDaemon. The major effort is currently focused on designing a new transactional workload supporting multiple load levels. Under consideration is native code support, GPGPU enablement, extended storage support and targeting a run time of not more than two hours.

The SERT Suite

The work on the SERT suite started around 2009 with the ground-breaking design of the Chauffeur framework, followed by the design and implementation of the workload, automated hardware and software discovery, and GUI. After a series of successful Alpha and Beta test phases, the SERT suite was released on February 26, 2013. On March 15, 2013, the United States Department of Energy's (DOE) Environmental Protection Agency (EPA) mandated the reporting of SERT results in their ENERGY STAR for Server Certification program. The collection of these data sets was the basis for formulation of thresholds for future energy programs.

The SPECpower Committee released four upgrades over the years to further enhance the usability of the SERT suite and to increase its platform support.

The primary focus of the SERT suite 2.0 was the reduction of its run time and improvements on the automated hardware and software discovery to reduce test time and cost. In addition, the support of additional power analyzers and interfaces was adopted in order to provide wider choices for users. Also, for more accurate characterization of memory, some major improvements of the memory subsystem worklets, were implemented. Simultaneously, the SPECpower Committee, in collaboration with the more academic-focused SPEC Power Research Group, conducted large-scale experiments in order to determine the server efficiency metric (add link to metric document) for SERT Suite 2.0 which enabled its release on March 28, 2017.

A significant milestone was achieved on September 17, 2018 as the U.S. EPA released the final version of the ENERGY STAR Version 3.0 Computer Servers Program Requirements. It defines Active State Efficiency Thresholds based on the SERT 2.0.1 metric, replacing the idle power thresholds and will determine ENERGY STAR eligibility effective June 17, 2019.

The Chauffeur Worklet Development Kit (WDK)

The Chauffeur framework from SERT is made available as the Chauffeur Worklet Development Kit (WDK). This kit can be used to develop new workloads (or "worklets" in Chauffeur terminology). Researchers can also use the WDK to configure worklets to run in different ways, in order to mimic the behavior of different types of applications. These features can be used in the development and assessment of new technologies such as power management capabilities. The first version of the Chauffeur WDK was released on May 18, 2014, and succeeded by the 1.1.0 release on May 26, 2015 and version 2.0 on April 18, 2017.

The Chauffeur WDK 2.0 further simplifies workload development for researchers analyzing computer performance and power consumption. Targeting both industry and academia, the WDK provides the infrastructure for SERT 2.0, providing run-time & platform configuration support to simplify worklet development. This major release includes significant enhancements to the hardware detection, customization options of generating HTML reports, and developer documentation. It has now reduced memory requirements for the Director when signing results files and reduced the size of the result output for large systems or clusters. Chauffeur WDK 2.0 added worklet-specific normalization of results and an updated list of supported operating systems and the latest PTDaemon integration for power analyzers and temperature sensors, along with data collection, validation and reporting.

The Power and Performance Benchmark Methodology

The Power and Performance Benchmark Methodology document is intended for performance benchmark designers and implementers who want to integrate a power component into their benchmark. The document may also serve as an introduction to those who need to understand the relationship between power and performance metrics in computer systems benchmarks. The assumption is that the business model and benchmark application are already selected and may already be implemented. Guidance is provided for including power metrics in existing benchmarks, as well as altering existing benchmarks and designing new ones to provide a more complete view of energy consumption. Benchmarks and tools already using this methodology document include SPECpower_ssj2008, SPECweb2009, SPECvirt_sc2013, the SERT suite, and the Chauffeur WDK.

What's next for SPECpower Committee?

The SPECpower committee is working on a successor to the popular (and still the only) power and performance benchmark, the SPECpower_ssj2008 benchmark. The new benchmark is based on the SPECpower Committee's expertise in the field of energy assessment across a wide spectrum of server configurations. It will have extensive Full Disclosure Report (FDR) description, include automatic discovery of hardware / software described in FDR, with automatic update support mechanism in place. It will also include tests to verify compliance with run and reporting rules and detect result tampering. It will be economical, easy to use - requiring no specific skills or equipment.

The key characteristics under consideration for the new benchmark include assessment of new server components (e.g., GPGPU), and inclusion of emergent cypher support with stronger key length (AES-256, SHA2-512, RSA 4096 bit, ECDSA P-384 etc.). The benchmark is being developed in collaboration with academia, and will stress multiple subsystems in parallel (multi-dimensional), and emergent technologies like AI, ML, and NLP are being explored.


SPEC and the names SERT, SPECpower, Chauffeur, and PTDaemon are trademarks of the Standard Performance Evaluation Corporation. Additional product and service names mentioned herein may be the trademarks of their respective owners. Copyright © 1988-2017 Standard Performance Evaluation Corporation (SPEC). All rights reserved.