Results
SPECapc
Download/Order
Resources
|
The
Application Performance Characterization Project Committee Rules
Version 2.01
Last Updated: 01/26/2006
-
Overview
-
General Philosophy
-
All rules declared in "The
Graphics Performance Characterization Group (SPEC/GPC): Rules
For Project Groups" document (known hereafter as the
GPC Project Groups Ruleset) apply, unless specifically overruled
by a rule in this document.
-
General Philosophy
-
Within SPEC's Graphics Performance
Characterization (GPC) Group there was a strong belief that it
is important to benchmark graphics performance based on actual
applications. Application-level benchmarks exist, but they are
not standardized and they do not cover a wide range of application
areas. Thus, the Application Performance Characterization (SPECapcSM) Project
was created within the GPC to create a broad-ranging set of standardized
benchmarks for graphics-intensive applications.
-
The SPECapc seeks to develop
benchmarks for generating accurate application-level graphics
performance measures in an open, accessible and well-publicized
manner.
-
The SPECapc wishes to contribute
to the coherence of the field of application performance measurement
and evaluation so that vendors will be better able to present
well-defined performance measures and customers will be better
able to compare and evaluate vendors products and environments.
-
The SPECapc will provide formal
beta benchmarks to members and final benchmark releases to the
public in a timely fashion.
-
Hardware and software used
to run the SPECapc benchmarks must provide a suitable environment
for running typical (not just benchmark) workloads for the applications
in question.
-
SPECapc reserves the right
to adapt its benchmarks as it deems necessary to preserve its
goal of fair and useful benchmarking (e.g. remove benchmark, modify
benchmark code or data, etc). If a change is made to the suite,
SPECapc will notify the appropriate parties (i.e. SPECapc members
and users of the benchmark) and SPECapc will re-designate the
benchmark by changing its name and/or version. In the case that
a benchmark is removed in whole or in part, SPECapc reserves the
right to republish in summary form "adapted" results
for previously published systems, converted to the new metric.
In the case of other changes, such a republication may necessitate
re-testing and may require support from the original test sponsor.
-
Overview of Optimizations
-
SPECapc is aware of the importance
of optimizations in producing the best system performance. SPECapc
is also aware that it is sometimes hard to draw an exact line
between legitimate optimizations that happen to benefit SPECapc
benchmarks and optimizations that specifically target SPECapc
benchmarks. However, with the list below, SPECapc wants to increase
awareness of implementers and end-users to issues of unwanted
benchmark-specific optimizations that would be incompatible with
SPECapc's goal of fair benchmarking.
-
To ensure that results are
relevant to end-users, SPECapc expects that the hardware and software
implementations used for running SPECapc benchmarks adhere to
a set of general rules for optimizations.
-
General Rules for Optimization
-
Optimizations must generate
correct images and results for the application under test, for
both the benchmark case and similar cases. Correct images and
results are those deemed by the majority of the SPECapc electorate,
potentially with input from the associated independent software
vendor (ISV) and/or end-users, to be sufficiently adherent to
the intent behind the application.
-
Optimizations must improve
performance for a class of workloads where the class of workloads
must be larger than a single SPECapc benchmark or SPECapc benchmark
suite.
-
For any given optimization
a system should generate correct images with and without said
optimization. An optimization should not reduce system stability.
-
The vendor encourages the implementation
for general use (not just for running a single SPECapc benchmark
or SPECapc benchmark suite).
-
The implementation is generally
available, documented and supported by the providing vendor.
-
In the case where it appears
that the above guidelines have not been followed, SPECapc may
investigate such a claim and request that the optimization in
question (e.g. one using SPECapc benchmark-specific pattern matching)
be removed and the results resubmitted. Or, SPECapc may request
that the vendor correct the deficiency (e.g. make the optimization
more general purpose or correct problems with image generation)
before submitting results based on the optimization.
-
It is expected that system
vendors would endorse the general use of these optimizations by
customers who seek to achieve good application performance.
-
No pre-computed (e.g. driver-cached)
images, geometric data, or state may be substituted within an
SPECapc benchmark on the basis of detecting that said benchmark
is running (e.g. pattern matching of command stream or recognition
of benchmark's name).
-
Benchmarks
-
Benchmark Acceptance
-
Benchmark components are defined
as
-
specific revision of an application,
-
run rules, scripts
and associated data sets.
-
New or modified benchmark components
require a 2/3-majority vote to be accepted for publication. Selection
of datecode versions of a specific revision of an application
is by majority vote.
-
A minimum 3-week review period
is required for new or significantly modified benchmark components.
-
At the end of the review period
a vote will be called to approve the proposed changes.
-
An amendment to a benchmark
component during the review period must be unanimously accepted.
If not, the review period shall be restarted.
-
Benchmark Code Versioning
-
Benchmarks use the following
version coding: M.m (e.g. SPECapcSM for Pro/ENGINEER
20.0 v1.1) M is the major release number and m is the minor release
number.
-
The major release number is
only incremented when large amounts of code are changed and the
scripting language is dramatically changed as a result -- backward
compatibility is highly unlikely when moving scripts or data sets
between major releases (e.g. running v2 scripts on a v3 executable
would almost certainly fail).
-
The minor release number is
bumped if some small set of code is replaced or removed - but
the standard, unchanged scripts and data sets, as a whole, must
run on the new version (but perhaps with different performance).
-
When there is a new major release
of a benchmark, submissions using the previous release will be
accepted for at least one submission cycle.
-
Submission, Review and Publication
-
General Benchmark Run Rules
-
The system under test must
correctly perform all of the operations being requested by the
application during the benchmark.
-
No changes to any files associated
with the benchmark are permitted excepted as noted in the benchmark-specific
rules (section 5 of this document).
-
The entire display raster must
be available for use by the application being benchmarked.
-
It is not permissible to override
the intended behavior of the application through any means including,
but not limited to, registry settings or environment variables.
-
No interaction is allowed with
the system under test during the benchmark, unless required by
the benchmark.
-
The system under test can not
skip frames during the benchmark run.
-
It is not permissible to change
the system configuration during the running of a given benchmark.
That is, one can't power off the system, make some changes, then
power back on and run the rest of the benchmark.
-
Results submitted must be obtained
using the scripts, models, and application revisions which are
specified for that submission cycle by the SPECapc.
-
The color depth
used must be at least 24 bits (true color), with at least 8 bits
of red, 8 bits of green and 8 bits of blue, unless otherwise specified.
-
The display raster
resolution must be at least 1280 pixels by 1024 pixels, unless
otherwise specified.
-
The monitor refresh
rate must be at least 75Hz. This requirement does not apply to
digital flat panel displays, unless otherwise specified.
-
The border width of the windows
created during the benchmark shall not exceed 10 pixels, unless
otherwise specified.
-
The monitor used in the benchmark
must support the stated resolution and refresh rate.
-
The benchmark must successfully
obtain all requested window sizes, with no reduction or clipping
of any benchmark-related windows. Windows created by the
benchmark must not be obscured on the screen by anything other
than other elements created by the benchmark.
-
Tests may be run with or without
a desktop/window manager if the application allows this, but must
be run on some native windowing system.
-
General Submission Content
Rules
-
The submission upload file
structures are defined in the benchmark-specific section below.
-
Submission Process Rules
-
The submission file names are
detailed below under the benchmark-specific rules.
-
Review Period Rules
-
Reviewers will decide if the
image quality and results of the submission are sufficiently correct
with respect to the intent of the ISV to satisfy the intended
end-users' expectations.
-
SPECapc Benchmark Specific Rules
and Procedures
-
Pro/Engineer 2001
-
The benchmark must be run using
the datecode version of Pro/ENGINEER 2001 specified on the SPECapc
website two weeks prior to the submission deadline.
-
The config.pro file must be
used as-is and may not be modified or overridden.
-
The script files utilities\runbench.bat
or utilities/runbench.csh may be modified as necessary to enable
execution of the benchmark on the system being tested. If
modified, the modified version must be included in the benchmark
submission.
-
The submission must contain
the proe_result.txt file (as generated by the proescore program)
as well as the corresponding trail.txt file generated from running
the benchmark. Both of these files may be found in the "results" directory
after a successful run of the benchmark. The first
section of the proe_result.txt file may be edited to reflect the
system configuration.
-
The directory structure of
the submission must be as follows:
.../<Company-name/<system_1/proe2001/proe_result.txt
.../<Company-name/<system_1/proe2001/trail.txt (may be compressed)
.../<Company-name/<system_1/proe2001/runbench.bat (or runbench.csh,
if modified, as required by Rule 5.15.c)
.../<Company-name/<system_2/proe2001/proe_result.txt
.../<Company-name/<system_2/proe2001/trail.txt (may be compressed)
.../<Company-name/<system_2/proe2001/runbench.bat (or runbench.csh,
if modified, as required by Rule 5.15.c)
etc...
-
Compression may be accomplished
using UN*X compress(1), tar -Z or zip. The reviewer may ask the
submitter to supply an uncompressed version of the trail file(s).
-
The submission file must be
named company_apc_proe2001_vN.zip or company_apc_proe2001_vN.tar.z
where company is the member company or organization name
in lower case and vN is the file version (e.g. sgi_apc_proe2001_v0.tar.z
and intel_apc_proe2001_v0.zip.) The initial submission is v0.
Resubmitted files must have the version number incremented.
-
Solid Edge V14
-
The benchmark must be run using
Solid Edge V14.00.00.70
-
The application can be run
with Tools-Options-View Graphics Display set to Graphics Card
Driven, Software Driven or Backing Store. The choice must be documented
in the notes portion of the results.
-
Application settings must not
be changed from the defaults set by the benchmark installation. Settings
that must not be changed include, but are not limited to:
-
Culling
-
Wireframe display in Move
Part command
-
Arc smoothness (3)
-
The submission must contain
the file result.txt that is generated during the benchmark run.
-
The directory structure of
the submission must be as follows:
.../company-name/system_1/SEV14/result.txt
.../company-name/system_2/SEV14/result.txt
etc...
-
The submission file must be
named company_apc_SolidEdgeV14_vN.zip where company is
the member company or organization name in lower case and vN is
the file version (e.g. hp_apc_SolidEdgeV14_v0.zip.) The initial
submission is v0. Resubmitted files must have the version number
incremented.
-
Solidworks 2005
-
The benchmark must be run using
Solidworks2005 service pack 0.
-
The application window size
must not be changed from its initial size.
-
The submission must contain
a results.txt file generated by running the benchmark. Its contents
are extracted from the file apcresultsN.txt, generated
by benchmark GUI , where N is the number of the
benchmark test run.
-
The submission will be derived
from the best composite generated by the default 5 benchmark runs,
as controlled and reported by the benchmark GUI.
-
The appearance of the the application's
Quick Tips / Dynamic Help box, or other pop-ups that do not dismiss
themselves, will cause the benchmark run to be invalid. The
benchmark FAQ has information on how to prevent this.
-
The directory structure of
the submission must be as follows:
.../company-name/system_1/sw2005/results.txt
.../company-name/system_2/sw2005/results.txt
etc...
-
The submission file must be
named company_apc_sw2005_vN.zip where company is
the member company or organization name in lower case and
vN is the file version (e.g. ibm_apc_sw2005_v0.zip.) The
initial submission is v0. Resubmitted files must have the version
number incremented.
-
3dsmax7
-
The benchmark must be run
using 3ds max version 7, service pack 0 (i.e. unpatched).
-
The application windows will
be visible as specified:
-
The "3ds max 7" window
is maximized and fills the screen (The window is not occluded
by any other windows.) Note: Task-bar auto-hide should be enabled.
-
Drivers and custom drivers
must be configured (explicitly or implicitly) to use:
-
GL_TEXTURE_MIN_FILTER =
GL_LINEAR_MIPMAP_LINEAR
-
GL_TEXTURE_MAG_FILTER = GL_LINEAR
-
Textures cannot be modified,
i.e. reduced in size or depth.
-
Wireframe objects must NOT
be drawn using triangle strips.
-
The following Viewport parameters
must be checked (i.e. enabled) in the Viewports Tab found in the
3ds max 7 Customize->Preferences menu (Viewports tab):
-
Backface cull on object
creation
-
Mask viewport to safe region
-
Update background while playing
-
Display world axis
-
The following Viewport parameters
must NOT be checked in the 3ds max 7 Customize->Preferences
menu (Viewports tab):
-
Attenuate lights
-
Filter environment backgrounds
-
The benchmark can be run using
either OpenGL or Direct 3D.
-
For OpenGL submissions, a custom
driver may be used, but the custom driver’s images must
not differ from the supplied 3ds max OpenGL driver by more than
5%. If the custom driver’s images do not match, the submitter
may apply for a waiver based on documented errors in the 3ds max
OpenGL driver.
-
The submission must contain
the files result.txt and result.xls that are generated during
the benchmark run.
-
The directory structure of
the submission must be as follows:
.../company-name/system_1/3dsmax7/result.txt
.../company-name/system_1/3dsmax7/result.xls
.../company-name/system_2/3dsmax7/result.txt
etc...
-
The submission file must be
named company_apc_3dsmax7_vN.zip where company is
the member company or organization name in lower case and vN is
the file version (e.g. hp_apc_3dsmax7_v0.zip.) The initial submission
is v0. Resubmitted files must have the version number incremented.
-
Submitters are not required
to include pixel-comparison images.
-
A committee member raising
a challenge to a submission can employ pixel-comparison images
to support the challenge. However, reference and grabbed images
must be generated on a system with identical GPU, graphics and
custom drivers and settings to the submitted configuration being
challenged.
-
Maya 6.5
-
The benchmark must be run
using Maya 6.5
-
The benchmark script must be
run with the command 'mayaTest(3)' where '3' is the number of
runs.
-
The submission must contain
the results.txt submission file as well as the scoring spreadsheet
used to calculate the result.
-
The directory structure of
the submission must be as follows:
.../company-name/system_1/maya65/results.txt
.../company-name/system_1/maya65/MayaResults.xls
etc...
-
The submission file must be
named company_apc_maya65_vN.zip where company is the member company or
organization name in lower case and vN is the file version
(e.g. ibm_apc_maya65_v0.zip.) The initial submission is v0. Resubmitted
files must have the version number incremented.
-
Pro/ENGINEER
Wildfire 2.0
-
The benchmark
must be run using build “M160” of Pro/ENGINEER Wildfire
2.0. Submissions may be made with 32-bit or 64-bit versions of
M160. However, submissions made with 32-bit Pro/ENGINEER Wildfire
2.0 on Windows XP 64-bit Edition are prohibited.
-
The config.pro
file must be used as-is and may not be modified or overridden.
-
The script
files “utilities\runbench.bat” or “utilities/runbench.csh” may
be modified as necessary to enable execution of the benchmark
on the system being tested. If modified, the modified version
must be included in the benchmark submission.
-
The submission
must contain the “proe_result.txt” file (as generated
by the “proescore” program) as well as the corresponding “trail.txt” file
generated from running the benchmark. Both of these files may
be found in the "results" directory after a successful
run of the benchmark. The first section of the “proe_result.txt” file
must be edited to reflect the system configuration.
-
There
must be no license-related warnings in the “trail.txt” file.
-
The directory
structure of the submission must be as follows:
.../company-name/system_1/proewildfire2/proe_result.txt
.../company-name/system_1/proewildfire2/trail.txt
(may be compressed)
.../company-name/system_1/proewildfire2/runbench.bat
(or runbench.csh, if modified, as required by rule c above)
.../company-name/system_2/proewildfire2/proe_result.txt
.../company-name/system_2/proewildfire2/trail.txt
(may be compressed)
.../company-name/system_2/proewildfire2/runbench.bat
(or runbench.csh, if modified, as required by rule c above)
etc...
-
The submission
file must be named company_apc_proewildfire2_vN.zip
or company_apc_proewildfire2_vN.tar.z where company is
the member company or organization name in lower case and vN is
the file version (e.g. “sgi_apc_proewildfire2_v0.tar.z” and “intel_apc_proewildfire2_v0.zip”.)
The initial submission is v0. Resubmitted files must have the
version number incremented.
-
UGS
NX3
- The benchmark must be run using build “3.0.3.2” of Unigraphics
NX3. Submissions are only accepted on Windows XP Professional SP2
and later.
- The script file “%ProgramFiles%\SPECapc\nx3bench\nx3runbench.bat” may
be modified to change the directory where the benchmark files are
located on the system being tested. If modified, the modified version
must be included in the benchmark submission.
- The application window must be maximized and all toolbars and
the resource bar must be turned off.
- NX3's Fixed Frame Rate feature must be disabled.
- The system's booted configuration as specified in the "boot.ini" file
must contain the "/3GB" flag.
- The chosen setting for View Frustum Culling must be documented
in the Comments field of the "results.txt" file.
- The "Disable Line Smooth" option under Preferences>Visualization
Performance>General Settings must be unchecked.
- All other application settings should be default.
- The submission must contain the “nx3result.txt” file as well
as the corresponding “nx3result.xls” file generated from running
the benchmark. Both of these files may be found in the "results" directory
after a successful run of the benchmark. The first section of the “nx3bench.xls” file
must be edited to reflect the system configuration.
- The directory structure of the submission must be as follows:
.../company-name/system_1/nx3bench/nx3result.txt
.../company-name/system_1/nx3bench/nx3bench.xls
.../company-name/system_1/nx3bench/result.txt
.../company-name/system_1/nx3bench/nx3bench.log
.../company-name/system_1/nx3bench/nx3runbench.bat,if
modified, as required by rule c above)
.../company-name/system_2/nx3bench/nx3result.txt
.../company-name/system_2/nx3bench/nx3bench.xls
.../company-name/system_2/nx3bench/result.txt
.../company-name/system_2/nx3bench/nx3bench.log
.../company-name/system_2/nx3bench/nx3runbench.bat,
if modified, as required by rule c above)
etc...
- The submission file must be named company_apc_nx3bench_vN.zip
or company_apc_nx3bench_vN.tar.z where company is the member company
or organization name in lower case and vN is the file version (e.g. “sgi_apc_nx3bench_v0.tar.z” and “intel_apc_nx3bench_v0.zip”.)
The initial submission is v0. Resubmitted files must have the version
number incremented.
|
|