SPEC/GWPG
Benchmarks
Download/Order
SPEC
Mirror Sites
Resources
|
The Workstation Performance Characterization Project Committee Rules
Version 1.2
Last Updated: 05/29/2015
- Overview
- General Philosophy
- All rules declared in "The
Graphics and Workstation Performance Group (SPEC/GWPG): Rules For
Project Groups" document (known hereafter as the GWPG
Project Groups Ruleset) apply, unless specifically overruled by
a rule in this document.
- General Philosophy
- Within SPEC's Graphics and Workstation Performance Group (SPEC/GWPG)
there was a strong belief that it is important to benchmark system
performance running algorithms used in popular workstation applications,
but without requiring the full application and associated licensing
to be installed on the system under test. Thus, the Workstation Performance
Characterization (SPECwpcSM) Project was created
within the GWPG to create a broad-ranging set of standardized benchmarks
for workstation applications.
- The SPECwpc seeks to develop benchmarks for generating accurate
workstation performance measures in an open, accessible and well-publicized
manner.
- The SPECwpc wishes to contribute to the coherence of the field
of application performance measurement and evaluation so that vendors
will be better able to present well-defined performance measures
and customers will be better able to compare and evaluate vendors’products
and environments.
- The SPECwpc will provide formal beta benchmarks to members and
final benchmark releases to the public in a timely fashion.
- Hardware and software used to run the SPECwpc benchmarks must
provide a suitable environment for running typical (not just benchmark)
workloads for the applications in question.
- SPECwpc reserves the right to adapt its benchmarks as it deems
necessary to preserve its goal of fair and useful benchmarking (e.g.
remove benchmark, modify benchmark code or data, etc). If a change
is made to the suite, SPECwpc will notify the appropriate parties
(i.e. SPECwpc members and users of the benchmark) and SPECwpc will
re-designate the benchmark by changing its name and/or version. In
the case that a benchmark is removed in whole or in part, SPECwpc
reserves the right to republish in summary form "adapted" results
for previously published systems, converted to the new metric. In
the case of other changes, such a republication may necessitate re-testing
and may require support from the original test sponsor.
- Overview of Optimizations
- SPECwpc is aware of the importance of optimizations in producing
the best system performance. SPECwpc is also aware that it is sometimes
hard to draw an exact line between legitimate optimizations that
happen to benefit SPECwpc benchmarks and optimizations that specifically
target SPECwpc benchmarks. However, with the list below, SPECwpc
wants to increase awareness of implementers and end-users to issues
of unwanted benchmark-specific optimizations that would be incompatible
with SPECwpc's goal of fair benchmarking.
- To ensure that results are relevant to end-users, SPECwpc expects
that the hardware and software implementations used for running SPECwpc
benchmarks adhere to a set of general rules for optimizations.
- Optimization Rules for SPECwpc v1.2
- Optimizations must improve performance for a class of workloads where
the class of workloads must be larger than a single SPECwpc benchmark or
SPECwpc benchmark suite.
- For any given optimization a system should generate correct results
with and without said optimization. An optimization should not reduce
system stability.
- The vendor encourages the implementation for general use (not
just for running a single SPECwpc benchmark or SPECwpc benchmark
suite).
- The optimization implementation is generally available, documented
and supported by the providing vendor.
- In the case where it appears that the above guidelines have not
been followed, SPECwpc may investigate such a claim and request that
the optimization in question (e.g. one using SPECwpc benchmark-specific
pattern matching) be removed and the results resubmitted. Or, SPECwpc
may request that the vendor correct the deficiency (e.g. make the
optimization more general purpose or correct problems with image
generation) before submitting results based on the optimization.
- It is expected that system vendors would endorse the general use
of these optimizations by customers who seek to achieve good application
performance.
- No pre-computed intermediate or final results may be substituted
within a SPECwpc benchmark on the basis of detecting that said benchmark
is running (e.g. pattern matching of command stream or recognition
of benchmark's name).
- Benchmarks
- Benchmark Acceptance
- Benchmark components are defined as
- specific revision of an application,
- run rules, scripts and associated data sets.
- New or modified benchmark components require a 2/3-majority vote
to be accepted for publication. Selection of datecode versions of
a specific revision of an application is by majority vote.
- A minimum 3-week review period is required for new or significantly
modified benchmark components.
- At the end of the review period a vote will be called to approve
the proposed changes.
- An amendment to a benchmark component during the review period
must be unanimously accepted. If not, the review period shall be
restarted.
- Benchmark Code Versioning
- Benchmarks use the following version coding: M.m (e.g. SPECwpcSM v1.1)
M is the major release number and m is the minor release number.
- The minor release number is incremented when performance-neutral
modifications are made to the benchmark.
- The major release number is incremented when changes to the benchmark
cause results to no longer be comparable with previous results.
- When there is a new major release of a benchmark, submissions using
the previous release will be accepted for at least one submission cycle.
When a superseded benchmark is retired, the associated run-rules will
be archived in an "archived benchmark run-rules document" posted
on the Previous SPECwpc Benchmarks web-page.
- Submission, Review and Publication
- General Benchmark Run Rules
- The system under test must correctly perform all of the operations
being requested by the application during the benchmark.
- No changes to any files associated with the benchmark are permitted
excepted as noted in the benchmark-specific rules (section 4 of this
document).
- The entire display raster must be available for use by the application
being benchmarked, and the display must be powered on.
- It is not permissible to override the intended behavior of the
tests through any means including, but not limited to, registry settings
or environment variables.
- No interaction is allowed with the system under test during the
benchmark, unless required by the benchmark. It is not permissible
to interact with the system under test to influence the progress
of the benchmark during execution.
- The system under test cannot skip steps during the benchmark run.
- It is not permissible to change the system configuration during
the running of a given benchmark. That is, one can't power off the
system, make some changes, then power back on and run the rest of
the benchmark.
- Results submitted must be obtained using the scripts, models,
and application revisions which are specified for that submission
cycle by the SPECwpc.
- The system under test must conform to the following in minimum
system configuration:
- The color depth used must be at least 24 bits (true color),
with at least 8 bits of red, 8 bits of green and 8 bits of blue,
unless otherwise specified.
- The display used in the benchmark must support the stated
resolution and refresh rate.
- The benchmark must successfully
obtain all requested window sizes, with no reduction or clipping
of any benchmark-related windows. Windows created by the
benchmark must not be obscured on the screen by anything other
than other elements created by the benchmark.
- The system under test must be running Microsoft Windows 7
64bit or Microsoft Windows 8.1 64bit
- The system under test must have at least 8GB of system RAM
- General Submission Content Rules
- The submission upload file structures are defined in the benchmark-specific
section below.
- Submission Process Rules
- The submission file names are detailed below under the benchmark-specific
rules.
- Review Period Rules
- Reviewers will decide if the image quality and results of the
submission are sufficiently correct with respect to the intent of
the ISV to satisfy the intended end-users' expectations.
- SPECwpc v1.2 Benchmark Specific Rules and Procedures
- Optimizations for viewsets used in this benchmark must adhere to the
same standards as outlined in the SPECgpc Graphics Performance Characterization
Project Group Rules, Version 2.13, Section 3.
- Optimizations for the workloads used in this benchmark cannot modify
the executable or the dataset used in the test.
- Caching of files in a high performance storage device is allowed as
long as the mechanism for populating the cache is generally applicable
to a wide range of applications run on the device under test.
- SPECwpc is only supported on systems running Microsoft Windows 7 64bit
and Microsoft Windows 8.1 64 bit.
- PowerShell scripting must be enabled on the system where the scoring
utility is run.
- The display resolution must be at least 1920x1080 pixels unless the
system has an integrated display which cannot achieve this resolution
(example: a notebook), in which case the system's maximum possible resolution
must be used.
- The benchmark must be run with Windows 7 User Account Control (UAC)
and firewall disabled, or with administrator privileges and the appropriate
firewall settings to allow the benchmark to complete without interruption.
- The benchmark can be installed and run on any storage device connected
to the system.
- The scoring utility must be used to generate valid results.
- Virtualized configurations, defined as any operating system configuration
running on a Hyper Visor or virtualization layer of any kind, must include the
word “virtualized” in the comment field of the config.txt file. This information
must be populated before the execution of the benchmark to ensure the results reflect
this attribute.
- Virtualized configurations as defined in rule 11. must also include a declaration
of the transport layer name and version used by the virtualization software in
parenthesis after the graphics accelerator listed in the Graphics Accelerator field
of the Graphics Hardware Configuration section of the result file.
- Virtualized configurations as defined above must also include a declaration of
the Hyper Visor name and version used by the virtualization software in parenthesis
after the model listed in the Model field of the System Hardware Configuration section
of the result file.
- The submission must contain the SPECwpc/Results_xxxx directory structure
for the run.
- The directory structure of the submission must be as follows:
- .../company-name/system_1/SPECwpc/Results_xxxx directory structure
- .../company-name/system_1/SPECwpc/Results_xxxx/SPECwpcResults.xls
- The submission file must be named company_wpc_vN.zip where company
is the member company or organization name in lower case and vN is the
file version (e.g. hp_wpc_v0.zip.) The initial submission is v0. Resubmitted
files must have the version number incremented.
- Partial submissions can be made for individual sub-sections of the
benchmark. Subsections are defined as Media and Entertainment, Product
Development, Life Sciences, Financial Services, Energy, and General Operations.
|