|
SPECjbb2000 User's Guide |
This document is a practical guide for setting up and running a SPEC Java Business Benchmark (SPECjbb2000) test. To submit SPECjbb2000 results the benchmarker must adhere to the rules contained in the "Run and Reporting Rules" contained in the kit. For an overview of the benchmark architecture, see the SPECjbb2000 whitepaper.
This document is targetted at people trying to run the SPECjbb2000 benchmark in order to accurately measure their Java system, comprised of a JVM and an underlying operating system and hardware.
SPECjbb2000 is a Java program emulating a 3-tier system with emphasis on the middle tier. Random input selection represents the first (user) tier. SPECjbb2000 fully implements the middle tier business logic. The third tier is represented by binary trees rather than a separate database.
The motivation behind SPECjbb2000 is that it is, and will continue to be, common to use Java as middleware between a database and customers. SPECjbb2000 is representative of a middle tier system, with simplifications to isolate it for benchmarking. This strategy saves the benchmarker the expense of having to invest in a fast database system in order to measure various JVM systems. The implication is that combining a JVM which proves to be fast on SPECjbb2000, with a database system which is proven to be fast on TPC-C, will provide a business with a fast and robust overall multi-tier environment. The decomposition into a separate middleware test makes testing accessible to more people.
SPECjbb2000 is inspired by the TPC-C benchmark and loosely follows the TPC-C specification for its schema, input generation, and transaction profile. SPECjbb2000 replaces database tables with Java classes and replaces data records with Java objects. SPECjbb2000 does no disk IO.
SPECjbb2000 runs in a single JVM in which threads represent terminals in SPECjbb2000, where each thread independently generates random input before calling transaction specific logic. There is no network IO in SPECjbb2000.
A warehouse is a unit of stored data. It contains roughly 25MB of data stored in many objects in many Btrees. A thread represents a terminal user within a warehouse. There is a one-to-one mapping between warehouses and threads, plus a few threads for SPECjbb2000 main and various JVM functions. As the number of warehouses increases during the full benchmark run, so does the number of threads.
A "point" represents a two-minute measurement at a given number of warehouses. A full benchmark run consists of a sequence of measurement points with an increasing number of warehouses (and thus an increasing number of threads).
This section describes the steps necessary to set up SPECjbb2000. The setup instructions for most platforms are very similar to these generic setup instructions.
Alternately, to run without the GUI, use the following command:
There should now be several jar files (jbb.jar, jbb_no_precompile.jar, check.jar, and reporter.jar), documentation in doc, and source and class files in src/spec/jbb and src/spec/reporter.
Do NOT recompile (javac).You are expected to use the bytecodes provided. Recompiling the benchmark will cause validation errors.
Try running the benchmark using either
or
as appropriate to your operating system. These are provided as examples,
and may require minor modifications for your particular environment. Go to
the directory containing the benchmark.
Alternately, set
CLASSPATH=./jbb.jar:./jbb_no_precompile.jar:./check.jar:.\reporter.jar:$CLASSPATH
for Unix or
CLASSPATH=.\jbb.jar;.\jbb_no_precompile.jar;.\check.jar;.\reporter.jar;%CLASSPATH%
for Windows.
To run the benchmark up to 8 points type:
java -ms256m -mx256m spec.jbb.JBBmain -propfile SPECjbb.props
The benchmarker will probably not want to keep these numbers, since the heap is fairly small and the properties files ( SPECjbb.props and SPECjbb_config.props) don't yet reflect the system under test.
Having done a trial run the benchmark, the benchmarker should now learn what the changeable parameters are, and where to find and change them in the properties files. Also, the documentation (and publishability) of the runs will be improved by editing the descriptive property file described below.
SPECjbb2000 takes two properties files as input: a control properties file and a descriptive properties file. The control properties file is used to modify the operation of the benchmark, for example altering the length of the measurement interval. The descriptive properties file is used to document the system under test; this documentation is required for publishable results, and is reflected in the output files. The values of the descriptive properties do not affect the workload.
The default name for the control properties file is SPECjbb.props, but may be overridden with a different name by using the -propfile command line option (see section 3 for an example). The name for the descriptive properties file is specified in the control properties file, using the input.include_file property. The default name, as distributed in the sample control properties file included with the kit, is SPECjbb_config.props. See below for a brief description of how to modify properties in these files.
A sample control properties file and a sample descriptive properties file are distributed with the SPECjbb2000 kit, using the default names. Before modifying these files, first make a copy of the originals to enable easy recovery of the default benchmark configuration. Also, if running on several platforms, the benchmarker will probably want to take advantage of the naming features described in the previous paragraph (e.g., SPECjbb_config.ALPHA_400.props, SPECjbb_config.NT_500.props).
Each line of a properties file is either blank, a comment, or a property assignment. Comments begin with the # character. Each property assignment is of the form name=value. "name" is the property identifier; "name" and "property" are often used synonymously. Property names are specific to the benchmark and must not be changed. See below for a discussion of the control properties that the benchmarker is likely to want to change and for a list of control properties that must not be changed in order for the SPECjbb2000 result to be publishable. See section 4 for examples of the specification of property values.
The control properties file allows modification of three distinct benchmark behaviors: length of run, warehouse sequence, and garbage collection behavior. The benchmarker may experiment with any of these behaviors, but for publishable results, there are restrictions on the modifications. See the following paragraphs for these restrictions. The control properties file also contains a property specifying the benchmark name, but this should not be changed.
Length of run is controlled by two properties: input.ramp_up_seconds and input.measurement seconds. input.ramp_up_seconds specifies, in seconds, the length of the warmup period that will precede each measured interval. input.measurement specifies, in seconds, the length of the measured interval used to produce a point (throughput value at a particular number of warehouses). For a result to be publishable input.ramp_up_seconds must be 30 and input.measurement_seconds must be 120.
Warehouse sequence may be controlled in either of two ways. The usual method for specifying warehouse sequence is the set of three properties, input.starting_number_warehouses, input.increment_number_warehouses, and input.ending_number_warehouses , which causes the sequence of warehouses to progress from input.starting_number_warehouses to input.ending_number_warehouses, incrementing by input.increment_number_warehouses. The alternative method of specifying warehouse sequence is input.sequence_of_number_of_warehouses, which allows specification of an arbitrary list of positive integers in increasing order. For a publishable result the warehouse sequence must begin at 1 and increment by 1. See Section 4.1 for a discussion of requirements on the total number of warehouses that must be run.
There is limited control over garbage collection behavior. Specifically, the benchmarker may choose to force a garbage collection between measurement intervals. This is controlled by the Boolean property input.forcegc, where the value "true" means that there will be a forced GC between measurement intervals and "false" means that no GC will be forced. Note that Boolean values must be lower case. The choice of value for this property has no effect on publishability of the result.
After completing setup as described in the previous section, the benchmark is ready to run.
Try running the benchmark using either
run.sh
or
run.bat
as appropriate to your operating system. These are provided as examples, and
may require minor modifications for your particular environment.
Alternately, set CLASSPATH:
CLASSPATH=./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:$CLASSPATH
or
CLASSPATH=.\jbb.jar:.\jbb_no_precompile.jar:.\check.jar:.\reporter.jar:%CLASSPATH%
then run the following line:
java -ms<min> -mx<max> spec.jbb.JBBmain -propfile SPECjbb.props
Most operating systems should be able to use a similar command line. Specifying more heap is allowed, and will probably help to get higher scores. Some Java 2 JVMs use -Xms/-Xmx instead of -ms/-mx to specify heap size.
The benchmark output appears in the results directory by default. The output types are raw, results, html, and ascii. The files are tagged with a sequential number, as
respectively. The jpeg also produces SPECjbb.<num>.jpg, which SPECjbb.<num>.html refers to.
Usually one doesn't need to run the Reporter manually, as the benchmark automatically creates all the file formats. However, if needed, a raw file can be processed with the Reporter tool to create: html with an html graph, html with a jpg graph, or nicely-formatted text. The html with a jpg graph requires Java 2, and (under unix) DISPLAY to be set to a working X display. (This may require setting xhost + on the display.) The html with an html graph is provided for those without Java 2, or without a display device. When the benchmark calls the Reporter, it will default to that which is available in the environment.
Additionally, the Reporter can be used to compare two results.
The most common way of running the Reporter is
java spec.reporter.Reporter -e -r results/SPECjbb.<num>.raw -o
results/SPECjbb.<num>.html
There are a number of other options:
Usage: java spec.reporter.Reporter [options] Options are: -a Plain ASCII text output default: generate HTML output with JPG graph -e Do NOT echo raw results properties in HTML output default: raw results inserted as HTML comments -h Create graph in HTML rather than JPG default: use JPG if have Java 2 and a DISPLAY -l Label Label to infix into the JPG name: GraphImage.label.jpg default: a random number -o Output Output file for generated HTML default: written to System.out -r Raw A SPEC raw file, generated by a benchmark run. May be in a mail message with mail headers. default: read from System.in -c Second raw file, to compare default: none -v Verbose. List extra config.testx.* properties default: extra properties are not reported
So, comparing two results is done as follows:
java spec.reporter.Reporter -e -r results/SPECjbb.<num>.raw -c
results/SPECjbb.<other-num>.raw -o results/compare.html
Results submitted to SPEC will appear on the SPEC website in the asc and html(-jpg) formats. The reporter output has to be redirected or specified with -o filename. The Reporter will use only the last value when the same point ( number of warehouses ) is repeated multiple times
Section 2 introduced the properties files. These can be modified to control operation of the benchmark. There are two properties files: SPECjbb.props and SPECjbb_config.props. Their relationship is described in more detail in Section 2. Section 2 also describes the general format of the properties lines, and how to change them.
------------------------------SPECjbb.props-------------------------------- ################################################################################ # # # Control parameters for SPECjbb2000 benchmark # # # ################################################################################ # # This file has 2 sections; changable parameters and fixed parameters. The # fixed parameters exist so that you may run tests any way you want, however # in order to have a valid, reportable run of SPECjbb2000, you must reset # them to their original values. # # ################################################################################ # # # Changable input parameters # # # ################################################################################ # Warehouse sequence may be controlled in either of two ways. The more # usual method for specifying warehouse sequence is the triple # input.starting_number_warehouses, input.increment_number_warehouses, # and input.ending_number_warehouses, which causes the sequence of # warehouses to progress from input.starting_number_warehouses to # input.ending_number_warehouses, incrementing by # input.increment_number_warehouses. # The alternative method of specifying warehouse sequence is # input.sequence_of_number_of_warehouses, which allows specification of # an arbitrary list of positive integers in increasing order. # For a publishable result the warehouse sequence must begin at 1, increment by # 1 and go to at least warehouses input.starting_number_warehouses=1 input.increment_number_warehouses=1 input.ending_number_warehouses=8 #input.sequence_of_number_of_warehouses=1 2 3 4 5 6 7 8 # # 'forcegc' controls whether to garbage collect between each number of # warehouses. # input.forcegc=true # # 'include_file' is the name for the descriptive properties file. # On systems where the file separator is \, use \\ as the file separator here # # Examples: # input.include_file=SPECjbb_config.props # input.include_file=/path/to/SPECjbb_config.props # input.include_file=c:\\path\\to\\SPECjbb_config.props # input.include_file=config.props # # directory to store output files # On systems where the file separator is \, use \\ as the file separator here # # Examples: # input.include_file=results # input.include_file=/path/to/results # input.include_file=c:\\path\\to\\results # input.output_directory=results ################################################################################ # # # Fixed input parameters # # # # YOUR RESULTS WILL BE INVALID IF YOU CHANGE THESE PARAMETERS # # # ################################################################################ # DON'T CHANGE THIS PARAMETER, OR ELSE !!!! input.suite=SPECjbb # # If you need to collect stats or profiles, it may be useful to increase # the 'measurement_seconds'. This will, however, invalidate your results # # Amount of time to run each point prior to the measurement window input.ramp_up_seconds=30 # Time of measurement window input.measurement_seconds=120 ------------------------------end of SPECjbb.props-------------------------------- ------------------------------SPECjbb_config.props-------------------------------- # # SPECjbb2000 properties file # This is a SAMPLE file which you can use to specify characteristics of # a particular system under test, and to control benchmark # operation. You can reuse this file repeatedly, and have a version for # each system setup you use. You should edit the reporting fields appropriately. # All of this can still be edited in the output properties file after # you run the test, but putting the values in here can save you some # typing for attributes which do not change from test to test. # ################################################################################ # #System Under Test hardware # ################################################################################ # Company which sells the hardware config.hw.vendor=Neptune Ocean King Systems # Home page for company which sells the hardware config.hw.vendor.url=http://www.neptune.com # What type of system was used config.hw.model=TurboBlaster 2 # What type of processor(s) the system had config.hw.processor=ARM # MegaHertz rating of the chip. Usually an integer config.hw.MHz=300 # Number of CPUs in the system, config.hw.ncpu=1 # Amount of physical memory in the system, in Megabytes. DO NOT USE MB or # GB, IT WILL CONFUSE THE REPORTER config.hw.memory=4096 # Amount of level 1 cache for instruction and data on each CPU config.hw.primaryCache=4KBI+4KBD # Amount of level 2 cache, for instruction and data on each CPU config.hw.secondaryCache=64KB(I+D) off chip # Amount of level 3 cache (or above) config.hw.otherCache= # The file system the class files reside on config.hw.fileSystem=UFS # Size and type of disk on which the benchmark and OS reside on config.hw.disk=1 x 4GB SCSI (classes) 1 x 12GB SCSI (OS) # Any other hardware you think is performance-relative. That is, you would # need this to reproduce the test config.hw.other=AT&T Rotary Telephone config.hw.available=Jan-1997 ################################################################################ # # System Under Test software # ################################################################################ # The company that makes the JVM software config.sw.vendor=Phobos Ltd # Home page for the company that makes the JVM software config.sw.vendor.url=http://www.phobos.uk.co # Name of the JVM software product (including version) config.sw.JVM=Phobic Java 1.2.2 # Date when the JVM software product is shipping and generally available # to the public config.sw.JVMavailable=Jan-1997 # How many megabytes used by the JVM heap. "Unlimited" or "dynamic" are # allowable values for JVMs that adjust automatically config.sw.JVMheapInitial=1024 config.sw.JVMheapMax=1024 # Command line to invoke the benchmark # On systems where the file separator is \, use \\ as the file separator here config.sw.command_line=java -ms256m -mx1024m -spec.jbb.JBBmain -propfile Test1 # Name of precompiler used config.sw.precompiler=Phobic Java Compiler # Command line to invoke the precompiler # On systems where the file separator is \, use \\ as the file separator here config.sw.precompiler_command_line=phobic-jc spec/jbb/Jbbmain.java -exclude class.list # Method or command used to exclude the methods not allowed to be optimized # or precompiled (see Run Rules section 2.1.1) config.sw.precompiler_class_excluder_method=11 classes listed in file class.list # Operating system (including version) config.sw.OS=Phobos DOS V33.333 patch-level 78 # Date when the OS version is shipping and generally available to the public config.sw.OSavailable=May-2000 # State of the system, such as "single-user mode", or "minimal boot" config.sw.systemState=normal # Free text description of what sort of tuning one has to do to either # the OS or the JVM to get these results. This is where kernel tunables # belong. Use HTML list tags, if you want a list on the report page config.sw.tuning=Operating system tunings<UL><LI>bufcache=1024</LI> <LI>maxthreads_per_user=65536</LI> </UL> # Any additional software that you think is need to reproduce the performance # measured on this test config.sw.other=Neptune JIT Accelerator 2.3b6 # Date when the other software is shipping and generally available to the public config.sw.otherAvailable=Feb-98 ################################################################################ # # Tester information # ################################################################################ # The company that ran and submitted the result config.test.testedBy=Neptune Corp. #The person who ran and submitted this result (name does not go on public pages config.testx.testedByName=Willie the Mailboy # A web page where people within the aforementioned company might get more # information # On systems where the file separator is \, use \\ as the file separator here config.test.internalReference=http://pluto.eng/specpubs/mar97/ # The company's SPEC license number config.test.specLicense=50 # Physically, where the results were gathered config.test.location=Santa Monica, CA ------------------------------end of SPECjbb_config.props--------------------------------
These parameters can be categorized as:
These are the parameters which specify the workload.
Note: note that publication requires all the points from 1 to 2*n warehouses, where n is number of warehouses at the peak throughput, and that the metric is computed from the points n to 2*n. In some cases, where the system under test is unable to successfully run all points up to 2*N, the runs may still be valid and publishable. See section 2.3 of the run and reporting rules.
The next two are "constants". Normally, you will not change these. They are provided for information only. Any changes to the benchmark constants will invalidate the run. SPEC does not endorse such changes, and obtained with modified benchmark constants are not publishable.
The examples shown are the default and also the required values for valid runs.
The Configuration description section should contain a full description of the testbed. It should have enough information to repeat the test results, including all necessary hardware, software and tuning parameters. All parameters with non-default values must be reported in the Notes section of the description.
The configuration description has the following categories.
Each category contains variables for describing it. For example, Hardware contains variables for CPUs, caches, controllers, etc. If a variable doesn't apply, leave it blank. If no variable exists for a part you need to describe, add some text to the notes section. The notes sections can contain multiple lines.
The properties file included in the kit contains examples for filling out these fields.
Hardware:
Software:
And, the test information:
In order to be a publishable result, or directly comparable to existing published results, a run must pass several runtime validity checks:
Assuming a run is valid (or invalid in a way in which the metric is still meaningful, although not publishable), the metric is the numerical representation of the performance of the system.
The SPECjbb2000 metric is calculated as follows:
The reporting tool contained within SPECjbb2000 produces a graph of the throughput at all the measured points with warehouses on the horizontal axis and throughputs on the vertical axis. All points from 1 to the minimum of 8 or 2*N are required to be run and reported. Missing points in the range N+1 to 2*N will be reported to have a throughput of 0 ops/second. The points being averaged for the metric will be marked on the report.
Benchmark results are available in several forms:
All of these are just different ways of presenting the same basic information. All include the overall score, scores at individual numbers of warehouses, validation or error messages, information about the configuration, information about the input parameters, and details about each point.
For this reason, these are described in detail only once, in the Html Result Pages section. The descriptions of the other formats will refer back to classifications of information described in the Html section. Sample output pages are included with the kit ( ASCII, HTML-jpeg, HTML-only, raw, results. ) Errors in the creation of any format (other than raw) do not prevent publication.
How to create the other formats from the raw file is described in Section 3.3 .
The SPECjbb.<num>.raw file contains all the inputs from the properties files and results from the test. The reporter tool uses this file to generate all the other file formats. The raw file is submitted to SPEC.
If you intend to postprocess the results, this is the best file to start from. All information is stored in field=value pairs. It can be read in to a Java program using the Properties class, or handled by Perl, awk, or shell using pattern match or grep.
The HTML result pages contain the following elements.
If a thread ran out of memory, it will be reported at the point in the output where the condition occurred.
This is automatically generated by the benchmark run. You can also run
java spec.reporter.Reporter -a -e -r results/SPECjbb.<num>.raw
to get the ASCII report format. This is one of the formats available on the SPECjbb2000 result pages website. It is very similar to the html, except that it is lacking the chart. It is primarily for viewing/printing on a system without a browser.
The benchmark outputs data to the screen as it runs. This output also appears in the results/SPECjbb.<num>.results file. It provides the same information as the HTML file, but in the order in which the benchmark has information available. It is therefore somewhat more verbose. The following sections appear on the screen.
The conformance check and its results comes out on the screen, but does not go into the .results file. Next, the parameters scroll down the screen.
The results from each point in the benchmark print out the following:
Calculating results
Minimum transactions by a thread = 213695
Maximum transactions by a thread = 214847
Minimum transactions by a warehouse = 213695
Maximum transactions by a warehouse = 214847
Difference = 1152 (0.5390861%)
===============================================================================
TOTALS FOR: COMPANY with 2 warehouses and 1 terminals each
................... SPECjbb2000 1.0 Results (time in seconds) ...................
Count Total Min Max Avg Heap Space
New Order: 186319 140.30 0.000 0.200 0.001 total 778.6MB
Payment: 186326 59.38 0.000 0.180 0.000 used 108.5MB
OrderStatus: 18632 6.52 0.000 0.020 0.000
Delivery: 18632 9.60 0.000 0.020 0.001
Stock Level: 18633 16.04 0.000 0.180 0.001
SPECjbb2000 result = 3570.59
===============================================================================
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The equivalent of the ascii report is printed to the screen and .results file at the end of the run, summarizing it.
All the descriptive properties are echoed to the screen and results file. Also, system and JVM properties acquired by System.getProperties() are echoed.
Error messages are explained in greater detail below, listed by key words in the message.
Messages from the benchmark:
Messages in the reports:
Still have a problem that makes no sense? Contact support@spec.org .
To tune the benchmark, analyze the workload and look for possible bottlenecks in the configuration. There are a number of factors or bottlenecks that could affect the performance of the benchmark on the system.
The following tuning tips are for configuring the system to generate the best possible performance numbers for SPECjbb2000. Please note that these should not necessarily be used to configure and size real world systems.
Upon a successful run, the results may be submitted to the SPEC Java sub-committee for review by mailing the SPECjbb.<num>.raw file to subjbb2000 @ spec.org. When mailing the output properties file, include it in the body of the mail message, don't send it as an attachment, and mail only one result per email message.
Note: The SPECjbb.<num>.raw file uses the configuration and parameter information in the properties file. Please edit the properties files with the correct information prior to running the benchmark for submission.
Every submission goes through a two-week review process. During the review, members of the sub-committee may ask for further information/clarifications on the submission. Once the result has been reviewed and approved by the sub-committee, it is displayed on the SPEC web site at www.spec.org .
