The Multimedia Benchmark Committee (MBC) was formed to develop a wide range of standardized benchmark suites that will help vendors and users compare and improve multimedia system performance. The group operates under SPEC's Graphics Performance Characterization (GPC) group. Its first benchmark will measure MPEG-2 decoder performance in terms of speed and quality. Frame-rate performance will be measured up to the required frame rate (such as 30 frames per second). The MBC is still working on a method to measure performance when the system reaches the required rate.
Ways and Means
The mechanism for running MBC benchmark tests and processing results is called a harness. The MBC has licensed VBench software from Chromatic Research and is reviewing how it can be used as a harness to measure the correct timing of frames being displayed, jitter, frame drops and other performance areas.
The MBC has licensed software from MIT to be used as part of its quality measurement system. The video fidelity metric under development by the MBC compares a reference sequence with a sequence decoded by the system under test. It accounts for display device calibration, color appearance, and spatio-temporal sensitivity. The process starts when both video signals are gamma-corrected and mapped into a perceptual color space. Then, the difference between the two signals is computed. The next step is to assess the visibility of the difference signal. The signal is split into two paths, one called excitatory and the other inhibitory. In each path, the signal is filtered by a temporal and spatial filter. Each filter corresponds to a sensitivity mechanism. The difference between the two paths - the visible signal - is computed and pooled spatially and temporally to compute a distortion figure for the sequence.
Content for testing is being collected by the MBC from actual video broadcast material. Three hours of content has been licensed to the MBC from CNN to use for testing sequences. Encoding is being done on a volunteer basis from MBC member companies.
Complex Challenge
The term multimedia - always ill-defined within the marketplace - hints at the complex challenges that face the MBC. Multimedia merges video, graphics, processing and sound, making it much more challenging for benchmark developers than other technology areas. Adding to the complexity is the MBC approach to benchmarking. The group is dedicated to developing a systems-level, multi-platform benchmark, one that will allow a wide range of participants. It will also be designed to accommodate vendors with different quality. Within this realm, functionality might be measured by "good enough for the money," "better-quality picture without dropping frames," "how many frames a machine can pump through without dropping a frame," or any number of other measurements being decided upon by MBC members.
"We are doing everything possible to achieve consensus so that this benchmark will be accepted throughout the industry," says Ahmad Zandi of Sun Microsystems, the MBC chair. "This and future MBC benchmarks will measure both performance and quality across a wide range of platform and architectures. These benchmarks will benefit the industry by being repeatable, scalable, open and fair to everyone."
Faced with its sizable challenge, the MBC has decided that its first benchmark will focus on MPEG-2 video only. The benchmark will concentrate primarily on Main Profile at Main Level (MP@ML) attributes that support up to 15 Mbps and will support Low profiles as well. There will be a maximum of 720 horizontal pixels for MP@ML. In the computer space, the MBC is examining 256, 65.5K and 16-million colors, and MPEG-1 and MPEG-2 resolutions for NTSC and PAL. In the consumer space, the group is looking at 12 bits/pixel (4:2:0) color at the same resolutions as those under consideration for computers.
For more information on MBC membership, send e-mail to: gpcmbc-info@spec.org, and list name, title, company name, address, e-mail address and phone number. Questions should be in English.
For more detailed information on the MBC, go to the MBC FAQ document.