The OpenGL Performance Characterization
Last Updated: 6/5/2001
- General Philosophy
- The OpenGL Performance Characterization Project of SPEC/GPC
(henceforth abbreviated as SPECopcSM)
believes the user community will benefit from an objective series of tests,
which can serve as common reference and be considered as part of an evaluation
- The SPECopc seeks to develop benchmarks for generating
accurate OpenGL performance measures in an open, accessible and well-publicized
- The SPECopc wishes to contribute to the coherence of
the field of OpenGL performance measurement and evaluation so that vendors
will be better able to present well-defined performance measures; and
customers will be better able to compare and evaluate vendors' products
- The SPECopc will provide formal beta software to members
and final software releases to the public in a timely fashion.
- Hardware and software used to run the SPECopc benchmarks
must provide a suitable environment for running typical OpenGL programs.
- SPECopc reserves the right to adapt its benchmarks as
it deems necessary to preserve its goal of fair and useful benchmarking
(e.g. remove benchmark, modify benchmark code or data, etc). If a change
is made to the suite, SPECopc will notify the appropriate parties (i.e.
SPECopc members and users of the benchmark) and SPECopc will re-designate
the metrics (e.g. changing the metric from DRV-04 composite to DRV-05
composite). In the case that a benchmark is removed in whole or in part,
SPECopc reserves the right to republish in summary form "adapted" results
for previously published systems, converted to the new metric. In the
case of other changes, such a republication may necessitate re-testing
and may require support from the original test sponsor.
- Overview of Optimizations
- SPECopc is aware of the importance of optimizations in
producing the best system performance. SPECopc is also aware that it is
sometimes hard to draw an exact line between legitimate optimizations
that happen to benefit SPECopc benchmarks and optimizations that specifically
target SPECopc benchmarks. However, with the list below, SPECopc wants
to increase awareness of implementers and end-users to issues of unwanted
benchmark-specific optimizations that would be incompatible with OPC's
goal of fair benchmarking.
- To ensure that results are relevant to end-users, SPECopc
expects that the hardware and software implementations used for running
SPECopc benchmarks adhere to a set of general rules for optimizations.
- General Rules for Optimization
- Optimizations must generate correct images for a class
of programs, where the class of programs must be larger than a single
SPECopc benchmark or SPECopc benchmark suite. Correct images are those
deemed by the majority of the SPECopc electorate to be sufficiently adherent
to the OpenGL specification for the targeted end-user community (e.g.
users of OpenGL on PDAs would have lower quality expectations than those
using high-end workstations).
- Optimizations must improve performance for a class of
programs where the class of programs must be larger than a single SPECopc
benchmark or SPECopc benchmark suite and applicable to at least one end
- For any given optimization a system must generate correct
images with and without said optimization. An optimization must not reduce
- The vendor encourages the implementation for general
use (not just for running a single SPECopc benchmark or SPECopc benchmark
- The implementation is generally available, documented
and supported by the providing vendor.
- It is expected that vendors would endorse the general
use of these optimizations by customers who seek to achieve good application
- No pre-computed (e.g. driver cached) images, geometric
data, or OpenGL state may be substituted within an SPECopc benchmark on
the basis of detecting that said benchmark is running (e.g. pattern matching
of command stream or recognition of benchmark's name).
- Every OpenGL implementation in both immediate and display
list mode must fully process every GL element presented
to it that will impact the frame buffer and GL state.
- Differences to the frame buffer between immediate and
display list modes must not exceed 0.01% of the number of pixels in the
- In the case where it appears the guidelines in this document
have not been followed, SPECopc may investigate such a claim and request
that the optimization in question (e.g. one using SPECopc benchmark-specific
pattern matching) be removed and the results resubmitted. Or, SPECopc
may request that the vendor correct the deficiency (e.g. make the optimization
more general purpose or correct problems with image generation) before
submitting results based on the optimization.
- Membership in the SPECopc is open to any organization
that has a direct and/or material interest in OpenGL graphics performance
- Members are expected but not required to be active participants
developing and improving SPECopc benchmarks.
- Members are entitled to secure access to development
- Members are entitled to unlimited publication rights.
- New members become eligible for voting on the 2nd
consecutive qualified meeting. The first qualified meeting may have been
attended prior to becoming a member. Qualified meetings are defined in
- A member maintains voting rights by attending 1 out of
the last 3 qualified meetings. A member loses their voting rights upon
missing 3 consecutive qualified meetings.
- A member regains voting rights on attending a second
consecutive qualified meeting.
- Associate Status
- Associate status is available to non-profit organizations.
- All SPECopc, GPC and SPEC Rules and Rights apply to Associates
unless specifically stated otherwise.
- Associates are entitled to secure access to development
- Associates do not have voting rights.
- Officers and Elections
- On an annual basis the SPECopc will elect from its membership
the following officers:
- Vice Chairperson
- The Chairperson's responsibilities are to
- conduct meetings,
- send out the agenda on time,
- conduct votes on time,
- deal with outside organizations such as the press,
- police the submission, review and appeal processes.
- The Vice-Chairperson's responsibility is to do the chairman's
job when the chairman is not available.
- The Secretary-Treasurer responsibilities are to:
- record minutes,
- maintain the rules document,
- keeps a history of email,
- track finances and interact with the GPC and SPEC Board
in that regard.
- The SPECopc has three types of meetings (not including
- Regular quarterly face to face meetings
- Special SPECopc face to face meetings for the full
- Conference Call meetings
- SPECopc meetings which qualify
for attendance are limited to:
- face to face meetings scheduled one month in advance
- conference calls scheduled two weeks in advance.
- Membership Dues and Billing
- Dues for the SPECopc will be set annually by the SPEC
Board of Directors with input from the SPECopc. Once set, the dues amount
will be recorded in the SPEC minutes and communicated to the SPECopc by
the SPEC office.
- Dues payment, purchase order, or letter of intent must
be received at the SPEC office in time for the January annual meeting.
Dues must be paid by the end of February. Failure to meet these deadlines
will result in loss of membership and voting rights which will be reinstated
when full payment is received at the SPEC office.
- Non-Member Publication
- The SPECopc will accept submissions from non-members
for review and publication on the SPEC public website.
- Non-member submissions must follow the same procedures
as member submissions.
- Non-members are not eligible to participate in reviewing
- Non-members will be charged per system configuration
for their submissions. Any change in hardware or software constitutes
a new configuration.
- On an annual basis the SPECopc will establish the pricing
for non-member publication. The amounts will be recorded in the SPECopc
- A configuration will be published on-line for one year,
unless the submitter notifies the publisher that it should be removed.
- After one year, the configuration will be removed automatically,
unless the submitter notifies the publisher that it should remain on-line.
- There are no additional non-member fees for extending
on-line publication beyond one year.
- The SPECopc project group may remove published results
due to benchmark revision. In this case, the submitter will be given notice
by the project group and may, at no charge, resubmit the identical configuration
for the revised benchmark.
- Benchmark Acceptance
- Benchmark components are defined as
- code sets (e.g. SPECviewperf®, SPECglperf®),
- run rules, scripts and associated data sets (e.g. viewsets
or SPECglperf script).
- New or modified benchmark components require a 2/3-majority
vote of the SPECopc electorate to be accepted for publication.
- A minimum 3-week review period is required for new or
significantly modified benchmark components.
- At the end of the review period a vote will be called
to approve the proposed changes.
- An amendment to a benchmark component during the review
period must be unanimously accepted. If not, the review period shall be
- It is the option of any future SPECviewperf Viewset author(s)
to require passing of selected conformance tests prior to submission of
results for that viewset.
- Benchmark Code Versioning
- Benchmark code is defined as the set of source code required
to build and run a benchmark executable (e.g. SPECviewperf and SPECglperf).
- SPECglperf Benchmark code uses the following version
coding: M.m.p (e.g. 3.1.2) M is the major release number, m is the minor
release number and p is the patch level.
- The major release number is only incremented when large
amounts of code are changed and the scripting language is dramatically
changed as a result -- backward compatibility is highly unlikely when
moving scripts or data sets between major releases (e.g. running v2
scripts on a v3 executable would almost certainly fail).
- The minor release number is bumped if some small set
of code is replaced or removed - but the standard, unchanged scripts
and data sets, as a whole, must run on the new version (but perhaps
with different performance).
- Patch releases can contain additions of new properties
and additions of new attributes to existing properties, but cannot change
or remove any existing properties, attributes or functionality. These
are typically used for bug fixes, small enhancements and so forth.
- SPECviewperf Viewset Versioning
- The version of a SPECviewperf viewset should be incremented
- changes to SPECviewperf affect the performance of the
- or changes to the Viewset script affect performance,
- or if the viewset data changes,
- or if rule changes affect the acceptance criteria.
- New results for the previous version of a Viewset will
no longer be published.
- SPECglperf Script Versioning
- The version of a SPECglperf script should be incremented
- changes to SPECglperf affect the performance of the
- or changes to the SPECglperf script can affect performance,
- or if rule changes affect the acceptance criteria.
- Benchmark Run Rules
- Benchmark Run Rules
- The system under test must perform all of the OpenGL
functionality requested by the benchmark with the exception that the
system does not have to support dithering.
- The systems under test must be OpenGL Conformant
for the pixel format or visual used by the benchmark.
- Settings for environment variables, registry variables
and hints must not disable compliant behavior.
- No interaction is allowed with the system under
test during the benchmark, unless required by the benchmark.
- The system under test can not skip frames during
the benchmark run.
- It is not permissible to change the system configuration
during the running of a given benchmark. For example, one can not
power off the system, make some changes, then power back on and run
the rest of the benchmark.
- Screen grabs for SPECviewperf will be full window
- The monitor must support the stated resolution and
refresh rate and must fully display all of the benchmark tests being
- Results to be made public must be run by official
scripts that may not be changed, with the following exceptions (which
must be documented if not the default):
- In SPECviewperf:
- specific selection of visual/pixel format
on a per-test basis
- the multithreading flag (-th) on approved
- Visual/pixel format required:
- May be selected on a per-test basis by submission
of the viewset script.
- If RGB visual/pixel format is requested,
it must have at least eight bits of red, eight bits of green
and eight bits of blue.
- If destination alpha is requested, it must
have at least 1 bit.
- If depth buffer is requested, it must have
at least 16 bits of resolution.
- Screen resolution must be large enough to run the
individual tests at their requested window size, with no reduction
or clipping of test window.
- Tests may be run with or without a desktop/window
manager, but must be run on some native windowing system.
- Submission and Review Rules
- Submission Preparation Rules
- The rules for the submission and review cycle to be used
are those posted on the SPECopc web site two weeks prior to the submission
- The benchmark versions to be used for a submission are
those posted on the SPECopc web site two weeks prior to the submission
- All benchmark sources for a submission must be the same
as that posted on the SPECopc web site two weeks prior to the submission
- Members who wish not to review the submission of other
specific members due to conflict of interest must submit that list to
the SPEC office prior to the submission deadline. The SPEC office will
hold the list in confidence from other members.
- Submission Content Rules
- The information supplied must reflect the system as tested.
- All fields in the configuration description file must
- A date must be supplied for 'General Availability' that
is accurate for the entire system - hardware, software, O/S, drivers,
- If the system as tested is not available from the submitter
through the normal ordering process, the "Comments" area of
the results page must describe how the system may be acquired.
- Date fields must always contain a valid date. "Now" is
not valid in a date field.
- A SPECviewperf submission can be for one or more viewsets
- Price includes system and monitor as tested.
- The color depth used must be at least 24 bits (true color).
- The display raster resolution must be at least 1280 pixels
by 1024 pixels.
- The monitor refresh rate must be at least 75Hz. This
requirement does not apply to digital flat panel displays.
- Alternate currency from the US dollar can be submitted
as price and the submission will be sorted separately on the summary pages
for Price and Price/Performance.
- The submitter is required to declare sufficient information
to reproduce the performance claimed. This includes but is not limited
- non-default environment variables,
- non-default registry variables,
- compiler name and version,
- compiler command line,
- changes to the standard makefiles.
- Any information required to be reported such as non-default
environment variables, registry variables or hints, that does not have
a predefined field must be documented in the "Comments" area of the results
- Valid submissions must include screen captures if required
by the benchmark.
- The SPECviewperf binary must be submitted if it is not
one of the standard binaries.
- Results previously published for a system can be resubmitted.
- Previously published results being re-submitted can only
have price changes.
- The SPECviewperf submission upload file must have the
structure defined in Figure 4.03-1.
- Each member company must ensure that the upload file
contains data for all the new configurations and existing published configurations
they wish to continue publishing.
- Standardized cache nomenclature are as follows:
- (D+I) is a Unified instruction and data cache
- (D/I) is a for separate instruction and data caches
- A number followed by KB or MB can be used to describe
the size of the cache.
- Caches dedicated to a processor are listed as per processor
- Caches shared by multiple processors are listed by
- Each component of the submitted software configuration
(including the graphics driver) shall be:
- uniquely identified,
- available to SPECopc members, upon demand, by the submission
deadline and for the duration of the review process,
- available to the public by the publication date, with
continued availability at least through the next submission deadline.
- Subsequent to publication, replacing elements of a submitted
configuration must not result in more than a 5% performance degradation
in any of the submitted benchmark results. The submitted results for this
configuration will be removed from the SPEC public website, if this requirement
is not met.
- On or before the date of publication, the submitted configuration
shall be available for purchase by the public, for the specified price
or less, with a firm delivery date.
- Submission Process Rules
- Each benchmark (SPECviewperf, etc.) is considered a separate
- Submissions of each benchmark (SPECviewperf, etc.) must
be in separate tar/zip files.
- The submission file names must contain opc_v for SPECviewperf
and opc_g for SPECglperf, contain all lower case letters and not contain
'.' except prior to the zip or tar file extension (e.g. intel_opc_v_jun99_v0.zip).
The file version is denoted prior to the file extension. The initial file
version is v0. Resubmitted files must increment the version number.
- A submitter of SPECopc benchmark results must upload
their submission to the proper server location by the submission deadline
date. The submitter must not create any new directories on the server
when uploading their submission.
- The submitter must notify SPEC Office after a submission
is uploaded to the server prior to the submission deadline with contact
information for questions about the submission.
- The submitter must contact the SPEC office if they have
attempted to upload their submission and were not successful.
- The SPEC office will not
disclose who has submitted results until the submission deadline has passed.
- Submissions will not be accepted after the submission
- The upload directory will be set to write-only until
the submission deadline has passed. Then it is set to read-write (not
modify) after the submission deadline.
- If a submitter is notified that their submission format
is incorrect, they must re-send their submission in proper format within
3 business days of notification.
- Abuse of the resubmission allowance is grounds for rejection
of a submission.
- Review Period Rules
- SPECopc members must keep other members' submitted results
SPECopc-confidential until they are publicly available.
- SPEC Office pairs reviewers to submitters.
- SPECviewperf review pools will be independent of each
other. The SPEC office will send the list of contact information for the
submissions under review.
- All members will have access to all benchmark submissions
once the review period begins.
- The review period shall be 10 calendar days.
- Submissions cannot be withdrawn during the review process.
- If a primary reviewer has a question with a submission
they must pose the question to the submitter first.
- Any reviewer who has questions with a submission must
- Pose these questions to the submitter and cc the primary
- Pose these questions to the primary reviewer. The primary
reviewer must then pose these questions to the submitter OR,
- Pose these questions to
an officer of the SPECopc. The officer of the SPECopc must then pose
these questions to the submitter and cc the primary reviewer.
- The submitter can request that their submission be rejected
on stated technical grounds.
- With public permission of the primary reviewer, a submitter
may resubmit their submission. The submitter must notify the gpcopc mailing
list with the date and version of the resubmitted file.
- The submitter must provide the primary reviewer access
to the system under test at the submitter's facilities if requested by
the reviewer during the review period. The reviewer must state prior to
the visit what part of the submission is going to be verified. Travel
expenses are the responsibility of the reviewer.
- Previously published results being re-submitted can only
be reviewed for consistency with the previous submission, and price changes.
- Price can be challenged. If so, the submitter must provide
documentation that the system can be purchased for the price quoted. Price
must be valid for 90 days from date of publication. Quantity 1 pricing
must be used.
- Reviewers will decide if the image quality of the submission
is sufficiently adherent to the OpenGL specification to satisfy the intended
end user's expectations. If a reviewer rejects the quality of an image
for a stated reason, the submitter can ask for a vote of the full SPECopc
electorate. In case of a tie the submission is rejected.
- By the end of the review period, the primary reviewer
of a submission must approve it without comment, approve it with comment,
or reject it with comment. The submitter may appeal a rejection as described
in Section 4.06.
- Comments for rejection of a submission received after
the end of the review period will not delay publication of the submission.
- The appeal period shall be 2 weeks, and immediately follow
the review period.
- Any submitter of a rejected submission can make their
case on the gpcopc email alias during the appeal period.
- At the end of the appeal period, if there is no resolution,
the Chair shall call a vote to approve or reject the submission.
- The whole SPECopc electorate votes on approval or rejection
of an appealed submission. A simple majority of the SPECopc electorate
is required to approve or reject the appeal. In case of a tie the submission
- Challenging Approved Results
- Any member may challenge approved results at any time.
- archived results,
- currently published results.
- The burden of proof that the result should be modified
is on the member who is challenging the result.
- The challenge must be ratified by a majority vote of
- The Chair will call a special review cycle in the event
that there is a ratified challenge to currently published results .
- A ratified challenge to archived results can only result
in annotation, not removal or modification. The annotation will be determined
by the majority of the committee.
- SPECopc Publication
- Benchmark results for publication by the SPECopc
must adhere to Articles I, IV and V.
- Unofficial Publication
- Benchmark results for publication elsewhere (e.g.
industry journals, vendor web sites, analyst reports) must adhere
to Articles I and IV.
- The SPECopc or any SPECopc member reserves the right
to request and receive evidence that the published results have been
achieved in accordance with the rules and that published information
- SPECopc metrics may be estimated. Metrics shall
not be estimated for configurations that are capable of running the
benchmark. All estimated metrics must be clearly identified as estimated.
Licensees are encouraged to publish actual SPECopc metrics as soon
as possible. Proper trademark usage for estimated results would be
in the following forms:
- SPECviewperf Awadvs-04 estimated score of 30
Adopted by the SPECopc on April 08, 1999.
Changes for version 1.1 adopted June 10, 1999.
Changes for version 1.2 adopted January 12, 2000.
V1.4 changes -- 5.02 (d) added
V1.5 changes -- 4.01.i.2(2), 4.01.i.2(4), 5.02.w