SPECopc Project Group Rules

The OpenGL Performance Characterization Project Group Rules

Version 2.00

Last Updated: 01/25/2006

Overview
1. Rules Inheritance
  1. All rules declared in "The Graphics Performance Characterization Group (SPEC/GPC): Rules For Project Groups" document (known hereinafter as the GPC Project Groups Ruleset) shall apply, unless specifically overruled by a rule in this document.
  2. Rules declared in this document shall apply in addition to the rules declared in the GPC Project Groups Ruleset.
2. General Philosophy
  1. The OpenGL Performance Characterization Project of SPEC/GPC (henceforth abbreviated as SPECopc^SM) believes the user community will benefit from an objective series of tests, which can serve as common reference and be considered as part of an evaluation process.
  2. The SPECopc seeks to develop benchmarks for generating accurate OpenGL performance measures in an open, accessible and well-publicized manner.
  3. The SPECopc wishes to contribute to the coherence of the field of OpenGL performance measurement and evaluation so that vendors will be better able to present well-defined performance measures; and customers will be better able to compare and evaluate vendors' products and environments.
  4. The SPECopc will provide formal beta software to members and final software releases to the public in a timely fashion.
  5. Hardware and software used to run the SPECopc benchmarks must provide a suitable environment for running typical OpenGL programs.
  6. SPECopc reserves the right to adapt its benchmarks as it deems necessary to preserve its goal of fair and useful benchmarking (e.g. remove benchmark, modify benchmark code or data, etc). If a change is made to the suite, SPECopc will notify the appropriate parties (i.e. SPECopc members and users of the benchmark) and SPECopc will re-designate the metrics (e.g. changing the metric from DRV-04 composite to DRV-05 composite). In the case that a benchmark is removed in whole or in part, SPECopc reserves the right to republish in summary form "adapted" results for previously published systems, converted to the new metric. In the case of other changes, such a republication may necessitate re-testing and may require support from the original test sponsor.
3. Overview of Optimizations
  1. SPECopc is aware of the importance of optimizations in producing the best system performance. SPECopc is also aware that it is sometimes hard to draw an exact line between legitimate optimizations that happen to benefit SPECopc benchmarks and optimizations that specifically target SPECopc benchmarks. However, with the list below, SPECopc wants to increase awareness of implementers and end-users to issues of unwanted benchmark-specific optimizations that would be incompatible with OPC's goal of fair benchmarking.
  2. To ensure that results are relevant to end-users, SPECopc expects that the hardware and software implementations used for running SPECopc benchmarks adhere to a set of general rules for optimizations.
4. General Rules for Optimization
  1. Optimizations must generate correct images for a class of programs, where the class of programs must be larger than a single SPECopc benchmark or SPECopc benchmark suite. Correct images are those deemed by the majority of the SPECopc electorate to be sufficiently adherent to the OpenGL specification for the targeted end-user community (e.g. users of OpenGL on PDAs would have lower quality expectations than those using high-end workstations).
  2. Optimizations must improve performance for a class of programs where the class of programs must be larger than a single SPECopc benchmark or SPECopc benchmark suite and applicable to at least one end user application. For any given optimization a system must generate correct images with and without said optimization. An optimization must not reduce system stability.
  3. The vendor encourages the implementation for general use (not just for running a single SPECopc benchmark or SPECopc benchmark suite). As an indicator that the implementation is suitable for general use, graphics configurations submitted for the SPECopc benchmark suite must be able to run the corresponding SPECapc application benchmarks if applicable.
  4. The implementation is generally available, documented and supported by the providing vendor.
  5. It is expected that vendors would endorse the general use of these optimizations by customers who seek to achieve good application performance.
  6. No pre-computed (e.g. driver cached) images, geometric data, or OpenGL state may be substituted within an SPECopc benchmark on the basis of detecting that said benchmark is running (e.g. pattern matching of command stream or recognition of benchmark's name).
  7. Every OpenGL implementation in both immediate and display list mode must fully process every GL element presented to it that will impact the frame buffer and GL state.
  8. Differences to the frame buffer between immediate and display list modes must not exceed 0.01% of the number of pixels in the window.
  9. In the case where it appears the guidelines in this document have not been followed, SPECopc may investigate such a claim and request that the optimization in question (e.g. one using SPECopc benchmark-specific pattern matching) be removed and the results resubmitted. Or, SPECopc may request that the vendor correct the deficiency (e.g. make the optimization more general purpose or correct problems with image generation) before submitting results based on the optimization.

Benchmarks
1. Benchmark Acceptance
  1. Benchmark components are defined as
    1. code sets (e.g. SPECviewperf®),
    2. run rules, scripts and associated data sets (e.g. viewsets).
  2. New or modified benchmark components require a 2/3-majority vote of the SPECopc electorate to be accepted for publication.
  3. A minimum 3-week review period is required for new or significantly modified benchmark components.
  4. At the end of the review period a vote will be called to approve the proposed changes.
  5. An amendment to a benchmark component during the review period must be unanimously accepted. If not, the review period shall be restarted.
  6. It is the option of any future SPECviewperf Viewset author(s) to require passing of selected conformance tests prior to submission of results for that viewset.
2. Benchmark Code Versioning
  1. Benchmark code is defined as the set of source code required to build and run a benchmark executable (e.g. SPECviewperf).
  2. SPECviewperf Benchmark code uses the following version coding: M.m.p (e.g. 8.0.1) M is the major release number, m is the minor release number and p is the patch level.
    1. The major release number is only incremented when large amounts of code are changed and the scripting language is dramatically changed as a result -- backward compatibility is highly unlikely when moving scripts or data sets between major releases (e.g. running v2 scripts on a v3 executable would almost certainly fail).
    2. The minor release number is bumped if some small set of code is replaced or removed - but the standard, unchanged scripts and data sets, as a whole, must run on the new version (but perhaps with different performance).
    3. Patch releases can contain additions of new properties and additions of new attributes to existing properties, but cannot change or remove any existing properties, attributes or functionality. These are typically used for bug fixes, small enhancements and so forth.
3. SPECviewperf Viewset Versioning
  1. The version of a SPECviewperf viewset should be incremented if:
    1. changes to SPECviewperf affect the performance of the viewset,
    2. or changes to the Viewset script affect performance,
    3. or if the viewset data changes,
    4. or if rule changes affect the acceptance criteria.
  2. New results for the previous version of a Viewset will no longer be published.

Benchmark Run Rules
1. Benchmark Run Rules
  1. The system under test must perform all of the OpenGL functionality requested by the benchmark with the exception that the system does not have to support dithering.
  2. The systems under test must be OpenGL Conformant for the pixel format or visual used by the benchmark.
  3. Settings for environment variables, registry variables and hints must not disable compliant behavior.
  4. No interaction is allowed with the system under test during the benchmark, unless required by the benchmark.
  5. The system under test can not skip frames during the benchmark run.
  6. It is not permissible to change the system configuration during the running of a given benchmark. For example, one can not power off the system, make some changes, then power back on and run the rest of the benchmark.
  7. Screen grabs for SPECviewperf will be full window size.
  8. The color depth used must be at least 24 bits (true color), with at least 8 bits of red, 8 bits of green and 8 bits of blue.
  9. If a depth buffer is requested, it must have at least 16 bits of resolution.
  10. The display raster resolution must be at least 1280 pixels by 1024 pixels.
  11. The monitor refresh rate must be at least 75Hz. This requirement does not apply to digital flat panel displays.
  12. The monitor must support the stated resolution and refresh rate and must fully display all of the benchmark tests being submitted.
  13. Results to be made public must be run by official scripts that may not be changed, with the following exceptions (which must be documented if not the default):
    1. In SPECviewperf:
      1. specific selection of visual/pixel format on a per-test basis
      2. the multithreading flag (-th) on approved multi-threading viewsets
    2. Visual/pixel format required:
      1. May be selected on a per-test basis by submission of the viewset script.
      2. If RGB visual/pixel format is requested, it must have at least eight bits of red, eight bits of green and eight bits of blue.
      3. If destination alpha is requested, it must have at least 1 bit.
      4. If depth buffer is requested, it must have at least 16 bits of resolution.
  14. Screen resolution must be large enough to run the individual tests at their requested window size, with no reduction or clipping of test window.
  15. Tests may be run with or without a desktop/window manager, but must be run on some native windowing system.

Submission and Review Rules
1. Submission Content Rules
  1. These rules are specific to SPECopc and shall apply in addition to the Submission Content Rules in the GPC Project Groups Ruleset.
  2. A SPECviewperf submission can be for one or more viewsets per configuration.
  3. The SPECviewperf binary must be submitted if it is not one of the standard binaries.
  4. The SPECviewperf submission upload file must have the structure defined in Figure 1:
    
    Figure 1
  5. Submitters are not required to submit depth images with a submission. Submitters must retain depth images to be available upon request by any committee member during the review period. After the review period, submitters are not required to retain depth images.
2. Submission Process Rules
  1. These rules are specific to SPECopc and shall apply in addition to the Submission Process Rules in the GPC Project Groups Ruleset.
  2. The submission file names must contain opc_v for SPECviewperf, contain all lower case letters and not contain '.' except prior to the zip or tar file extension (e.g. intel_opc_v_jun99_v0.zip). The file version is denoted prior to the file extension. The initial file version is v0. Resubmitted files must increment the version number.
3. Review Period Rules
  1. These rules are specific to SPECopc and shall apply in addition to the Review Period Rules in the GPC Project Groups Ruleset.
  2. Reviewers will decide if the image quality of the submission is sufficiently adherent to the OpenGL specification to satisfy the intended end user's expectations. If a reviewer rejects the quality of an image for a stated reason, the submitter can ask for a vote of the full SPECopc electorate. In case of a tie the submission is rejected.
  3. System configurations submitted for the SPECopc benchmark suite must be able to run the corresponding SPECapc application benchmarks if applicable. If this criterion is not met the submission will be rejected.

Adoption

Changes for version 1.1 adopted June 10, 1999
Changes for version 1.2 adopted January 12, 2000
V1.4 changes -- 5.02 (d) added
V1.5 changes -- 4.01.i.2(2), 4.01.i.2(4), 5.02.w
V1.6 changes -- 1.03.c, 5.04.o - Adopted by the SPECopc on January 23, 2003
V1.17 Adopted by the SPECopc on August 13, 2004
V1.18 Adopted by the SPECopc on October 21, 2004
V1.19 Adopted by the SPECopc on April 19, 2005
V1.20 Adopted by the SPECopc on October 20, 2005
V2.00 Adopted by the SPECopc on January 26, 2006

Standard Performance Evaluation Corporation