Q. Why was the APC project group formed?
A. Within the Graphics Performance Characterization (GPC) group there
was a strong belief that it is important to benchmark graphics performance
based on actual applications. Application-level benchmarks exist, but they
are not standardized and they often do not fully represent the graphics
functionality required by the actual application. The APC group feels that
end users will benefit from a broad-ranging set of standardized benchmarks
for graphics-intensive applications.
Q. What companies are members of the APC project group?
A. Companies that have indicated they will work on developing
benchmarks and pay membership dues include Compaq, Digital Equipment Corp.,
Hewlett-Packard, IBM, Intel, Intergraph, Real 3D, Silicon Graphics and Sun
Microsystems.
Q. Is it appropriate that vendors should drive this effort?
A. Industry vendors have the highest level of interest in developing
these benchmarks, but the development process will stem from interaction
with user groups, publications, application developers and others. There is
sometimes a perception that "vendor-driven" benchmarks are less
objective. In fact, these types of consortium-developed benchmarks are
likely to be the most objective, since there is a natural system of checks
and balances due to the specific market interests of different vendors.
Q. Why did the APC project group decide to join the GPC Group?
A. This initiative stemmed from members of the GPC Group seeing a
common need for better application benchmarks. The GPC Group, with its
affiliation with SPEC (Standard Performance Evaluation Corp.), already has
systems in place for running such a project and ensuring standardization of
the workloads, measurement criteria, review processes, and reporting formats.
This saves a great deal of administrative time and resources, allowing the
project group to concentrate more fully on developing and maintaining
benchmarks.
Q. How will the group identify applications in various market segments?
A. We will use existing market data, member company knowledge, and
ISV input to determine relevant market segments and important applications
in those segments. These applications will be updated over time as user
needs change and new applications become available.
Q. Will the application benchmarks be required to run on all platforms?
A. They should run on a reasonable number of platforms, but they do
not have to run on all platforms. Many applications are targeted to
a specific level of hardware, such as PCs or high-end workstations. It
would not be appropriate to require a vendor to run a benchmark that is not
designed for its platform.
Q. How will the group identify appropriate workloads for selected
applications?
A. APC project group members will sponsor applications and work with
end users, user groups, publications and ISVs to select and refine workloads,
which will consist of data sets and benchmark script files. Workloads will
be driven by end users and ISVs, not vendors. And, they will evolve over
time in conjunction with end users’ needs.
Q. What are the APC project group’s priorities in selecting benchmarks?
A. Number one is to select benchmarks that will be useful to end
users who are evaluating and selecting platforms. Secondly, the applications
must use graphics prominently in at least some phase of user interaction.
The workload would be selected to capture this phase. There are several
existing application benchmarks that are called "graphics-oriented,
" but whose test results do not reflect the benefits that come from
improvements vendors are making in raw graphics performance. The APC group
focuses on benchmarks that tax the graphics subsystem, including hardware,
software and the OS.
Q. How will the group define, publish, measure, report and review
results?
A. This is where our GPC Group affiliation pays benefits. We will
adopt existing GPC Group processes as much as possible to minimize work and
reduce the learning curve. Performance measurements must be accurate and
repeatable across diverse computing environments. Reporting schemes will
be designed to meet end users’ needs.
Q. How will APC results be verified and published?
A. Again, we will take advantage of standardized practices used
within the GPC Group. We intend to develop a review process, and a set of
universal reporting and display tools for publishing results in the GPC
News, the Web site that serves as the official publication of the GPC
Group.
Q. Will benchmark tools be freely available to the public?
A. Yes. We will make certain a wide audience has easy access to the
workloads and the benchmark results published by the APC project group.
These tools will be made available for free
downloading through the GPC News Web site, which will allow
independent reproduction of results and analysis of workloads. Obviously, anyone who
wants to run the benchmarks will need copies of the applications.
Q. Applications often support different features on different
platforms. This is especially true on the high end, where vendors seek to
add value by providing advanced features. How will the APC group accommodate
advanced feature sets in its application benchmarks?
A. The project group wants to make certain that we avoid "
least-common-denominator" tests. We envision developing benchmarks
based on "core" functionality, which represents the end users’
"musts" for running the application. All vendors reporting on a
benchmark must support this functionality, although it is the reporting
vendor’s choice about whether to implement the functions through hardware or
software. In addition to core functionality, we envision an "
enhanced" aspect of the benchmark. The enhanced aspect would make it
clear that there is extended functionality and performance available that
has value to end users. An example of enhanced functionality might be the
ability to do 3D texture mapping.
Q. Where is more information about membership and application benchmarking
available?
A. Information is available through the APC project group’s e-mail alias: gpcapc-info@spec.org.