Skip navigation

Standard Performance Evaluation Corporation

Facebook logo LinkedIn logo Twitter logo
 
 

SPEC's Benchmarks and Tools

Current Benchmarks

Cloud

  • SPEC Cloud IaaS 2018
    [benchmark info] [published results] [order benchmark]
    SPEC Cloud IaaS 2018 builds on the original 2016 release, updates metrics, and workloads and adds easier setup. The benchmark stresses the provisioning, compute, storage, and network resources of infrastructure-as-a-service (IaaS) public and private cloud platforms with multiple multi-instance workloads. SPEC selected the social media NoSQL database transaction and K-Means clustering using Cassandra and Hadoop as two significant and representative workload types within cloud computing. For use by cloud providers, cloud consumers, hardware vendors, virtualization software vendors, application software vendors, and academic researchers.

CPU

  • SPEC CPU 2017
    [benchmark info] [published results] [support] [order benchmark]
    Designed to provide performance measurements that can be used to compare compute-intensive workloads on different computer systems, SPEC CPU 2017 contains 43 benchmarks organized into four suites: SPECspeed 2017 Integer, SPECspeed 2017 Floating Point, SPECrate 2017 Integer, and SPECrate 2017 Floating Point. SPEC CPU 2017 also includes an optional metric for measuring energy consumption.

Graphics and Workstation Performance

High Performance Computing: OpenMP, MPI, OpenACC, OpenCL

  • SPECaccel 2023
    [benchmark info] [published results] [support] [order benchmark]
    The SPECaccel 2023 benchmark suite tests performance with computationally intensive parallel applications running under the OpenACC and OpenMP target offloading APIs. The suite exercises the performance of the accelerator, host CPU, memory transfer between host and accelerator, support libraries and drivers, and compilers.

  • SPEC ACCEL
    [benchmark info] [published results] [support] [order benchmark]
    SPEC ACCEL tests performance with a suite of computationally intensive parallel applications running under the OpenCL 1.1, OpenACC 1.0, and OpenMP 4.5 APIs. The suite exercises the performance of the accelerator, host CPU, memory transfer between host and accelerator, support libraries and drivers, and compilers.

  • SPEChpc 2021
    [benchmark info][published results] [support][order benchmark]
    SPEChpc contains four suites, Tiny, Small, Medium, and Large, including groups of full applications or mini-apps covering a wide range of scientific domains and Fortran/C/C++ programming languages. Each suite uses increasingly larger workloads to allow for appropriate evaluation of HPC systems at different sizes, ranging from a single node to hundreds of nodes. All benchmarks are ported to use either pure-MPI or hybrid MPI+OpenACC, MPI+OpenMP (task/thread based), or MPI+OpenMP using "Target", thus allowing measurement on heterogenous system.

  • SPEC MPI 2007
    [benchmark info] [published results] [support] [order benchmark]
    SPEC MPI 2007 is SPEC's benchmark suite for evaluating MPI-parallel, floating point, compute intensive performance across a wide range of cluster and SMP hardware. The suite consists of the intial MPIM2007 suite and MPIL2007, which contains larger working sets and longer run times than MPIM2007. All benchmarks in the suite are developed in compliance with MPI 2.1 standard.

  • SPEC OMP 2012
    [benchmark info] [published results] [support] [order benchmark]
    The successor to the SPEC OMP 2001 suite, designed for measuring performance using applications based on the OpenMP 3.1 standard for shared-memory parallel processing. SPEC OMP 2012 also includes an optional metric for measuring energy consumption.

Java Client/Server

  • SPECjbb 2015
    [benchmark info] [published results] [support] [order benchmark]
    The SPECjbb 2015 benchmark has been developed from the ground up to measure performance based on the latest Java application features. It is relevant to all audiences who are interested in Java server performance, including JVM vendors, hardware developers, Java application developers, researchers and members of the academic community.

  • SPECjEnterprise 2018 Web Profile
    [benchmark info] [published results] [support] [order benchmark]
    SPECjEnterprise 2018 Web Profile measures full-system performance for Java Enterprise Edition (Java EE) Web Profile Version 7 or later application servers, databases and supporting infrastructure.

  • SPECjEnterprise 2010
    [benchmark info] [published results] [support] [order benchmark]
    SPECjEnterprise 2010 measures full-system performance for Java Enterprise Edition (Java EE) 5 or later application servers, databases and supporting infrastructure and expands the scope of the SPECjAppServer 2004 benchmark.

  • SPECjvm 2008
    [benchmark info] [published results] [support] [download benchmark]
    SPECjvm 2008 is a benchmark suite for measuring the performance of a Java Runtime Environment (JRE), containing several real life applications and benchmarks focusing on core java functionality. The SPECjvm 2008 workload mimics a variety of common general purpose application computations.

Storage

Power

  • SPECpower_ssj 2008
    [benchmark info] [published results] [support] [order benchmark]
    SPECpower_ssj 2008 is the first industry-standard SPEC benchmark that evaluates the power and performance characteristics of volume server class computers. The initial benchmark addresses the performance of server-side Java, and additional workloads are planned.

Virtualization

  • SPECvirt Datacenter 2021
    [benchmark info] [published results] [support] [order benchmark]
    The SPECvirt Datacenter 2021 benchmark is the third generation SPEC benchmark for evaluating the performance of virtualized environments and the first for measuring the performance of multiple hosts in virtualized datacenters.

  • SPEC VIRT_SC 2013
    [benchmark info] [published results] [support] [order benchmark]
    SPEC's updated benchmark addressing performance evaluation of datacenter servers used in virtualized server consolidation. SPEC VIRT_SC 2013 measures the end-to-end performance of all system components including the hardware, virtualization platform, and the virtualized guest operating system and application software. The benchmark supports hardware virtualization, operating system virtualization, and hardware partitioning schemes.

SPEC Tools

  • SPEC SERT Suite 2.0
    [benchmark info] [support] [order software]
    The SERT suite was created by Standard Performance Evaluation Corporation (SPEC) at the request of the US Environmental Protection Agency. The SERT suite 2.0 adds a single-value metric, reduces runtime, improves automation and testing, and broadens device and platform support. Designed to be simple to configure and use via a comprehensive graphical user interface, the SERT suite uses a set of synthetic worklets to test discrete system components such as processors, memory and storage, providing detailed power consumption data at different load levels.

    The SERT suite metric, created with the support of the RG Power Working Group, rates the server efficiency of single- and multi-node servers across a broad span of configurations.

  • SPEC SERT Suite 1.1.1
    [benchmark info] [support] [order software]
    The SERT suite was created by Standard Performance Evaluation Corporation (SPEC) at the request of the US Environmental Protection Agency. The SERT suite 1.1.1 is the most current SERT version supported by the U.S. EPA Energy Star v2.0 program. Designed to be simple to configure and use via a comprehensive graphical user interface, the SERT suite uses a set of synthetic worklets to test discrete system components such as processors, memory and storage, providing detailed power consumption data at different load levels.

  • SPEC Chauffeur WDK Tool
    [kit info] [forum] [order software]
    The Chauffeur WDK (Worklet Development Kit) Tool was designed to simplify the development of workloads for measuring both performance and energy efficiency. Because the Chauffeur WDK tool contains functions that are common to most workloads, developers of new workloads can focus on the actual business logic of the application, and take advantage of the Chauffeur WDK tool's capabilities for configuration, run-time, data collection, validation, and reporting.

    The Chauffeur WDK tool was initially designed to meet the requirements of the SERT. However, SPEC recognized that the framework would also be useful for research and development purposes. The Chauffeur framework is now being made available as the Chauffeur WDK (Worklet Development Kit). This kit can be used to develop new workloads (or "worklets" in Chauffeur terminology). Researchers can also use the Chauffeur WDK to configure worklets to run in different ways, in order to mimic the behavior of different types of applications. These features can be used in the development and assessment of new technologies such as power management capabilities.

    Version 2.0 is based on the SERT suite 2.0 infrastructure and includes significant enhancements to the hardware detection, customization options of generating HTML reports, and developer documentation. It has now reduced memory requirements for the Director when signing results files and reduced the size of the result output for large systems or clusters.

    The Chauffeur WDK tool 2.0 added worklet-specific normalization of results and an updated list of supported operating systems including Ubuntu (14.04 LTS and 16.04 LTS) as well as current versions of Windows Server, RHEL, SLES, AIX, and Solaris. The Chauffeur WDK tool also includes the latest PTDaemon integration for power analyzers and temperature sensors, along with data collection, validation and reporting.

  • PTDaemon
    [info]
    The power temperature daemon (also known as PTDaemon) is used to offload the work of controlling a power analyzer or temperature sensor during measurement intervals to a system other than the SUT. It hides the details of different power analyzer interface protocols and behaviors from the benchmark software, presenting a common TCP/IP-based interface that can be readily integrated into different benchmark harnesses. Benchmarks already using PTDaemon include SPECpower_ssj 2008, SPEC CPU 2017, SPEC VIRT_SC 2013, the SERT suite, and the Chauffeur WDK.

Retired Benchmarks