SPECvirt Client Harness User's Guide

These instructions also assume you have read the SPECvirt Design Document and are familiar with concepts and terminology introduced there that are used in these instructions.

opt
  - SPECimap
  - SPECjAppServer2004
  - SPECpoll
  - SPECptd
  - SPECvirt
  - SPECweb2005

A SPECvirt benchmark run has two measurement phases: a "loaded" phase and an "unloaded" phase. During the loaded phase, SPECpoll is used to poll the idle server VM while the other threee workloads are creating request-generated load against their corresponding VMs. During the no-load, or "active idle" phase, SPECpoll is used to poll all VMs. (Please refer to the SPECpower Methodology for more information about "active idle" power measurement.)

There are two different sets of setup instructions: those for the VMs, and those for the clients.

1. Copy the SPECpoll directory to each of the VMs in the desired location (you can actually just copy pollme.jar if you prefer).
2. Determine the port you wish to listen on. This must be consistent with the RMI_PORT value specified in Test.config on the SPECpoll client that communicates with the VM. (We assume port 8001 in our example)
3. Determine the network interface on which this process will listen. For all except the dbserver and infraserver VMs, this is the network interface used for communication between the client and server VM. For the database server VM, it is the interface used for communication between the application server and the database server. For the infraserver VM, it is the interface used for communication between the web server and the BeSim server.
4. From the SPECpoll directory, execute "java -jar pollme.jar [-n <hostname/interface>] -p 8001"

1. IDLE_SERVER: This is the hostname of the idle server VM (value used only for idle server polling; ignored for active-idle polling)
2. RMI_PORT: This is the port on which the pollme class on the VMs is listening. This must be the same port for all VMs (in our example, 8001)

If you want to run all workloads on all tiles at a higher or lower load level than the default, changing the value of the property LOAD_SCALE_FACTORS allows you to do this. (Of course, for higher load levels, you need to assure that the corresponding workload VM datasets are built to support the higher load levels.) Note that the comma-delimited string of numbers in the non-indexed LOAD_SCALE_FACTORS property determines the number of measurement intervals to run in a single test. Also note that the ",0" at the end of the LOAD_SCALE_FACTORS string only applies to power measurement runs. In the case of power-included runs, wherever a "0" value is included in the LOAD_SCALE_FACTORS string, during that interval the prime controller runs an active idle measurement. However, all "0" values are ignored for non-power tests. The only reason to remove the active idle-specific load reference in this string is if you wanted to skip an active idle measurement in a benchmark run that includes power measurement.

By default, tile index numbers must start at 0 and increase by one for each added tile. And because the datasets for each tile are tile number-specific, using this default methodology requires that you first set up and run Tile 0, and then set up and add Tile 1, etc. Further, by default each tile can only be run along with all of the lower numbered tiles. That is, you cannot run Tile 1 by itself because the default ordering scheme expects the first tile to be Tile 0.

This is where the TILE_ORDINAL property may come in handy. Using the TILE_ORDINAL property supercedes the default ordering scheme. However, if tile ordinals are used, then they must be specified for all tiles used in a benchmark run. For example, if you use TILE_ORDINAL for a four-tile run, the harness will expect TILE_ORDINAL[0] through TILE_ORDINAL[3] defined in Control.config. (It will ignore any values for indexes greater than 3.)

The simplest and perhaps most common case for using TILE_ORDINAL is when you have just set up your second tile and want to test only that tile in a benchmark run. In that case, you set "TILE_ORDINAL[0] = 1" and then make sure all other tile index references in Control.config for Tile 1 are consistent with that tile (e.g. assure the PRIME_HOST[1][w] values point to the hostnames and ports for Tile 1, etc). When the prime controller begins benchmark execution, it will then see that you want Tile 1 to be your first tile, and will execute accordingly.

With TILE_ORDINALs, the only expectation is that the TILE_ORDINAL indexes start at 0 and increase by one for each additional tile. The values used for the tile numbers and their ordering are not bound by such constraints. For example, assuming you had four tiles set up and wanted to run two of them at a time, in addition to running:

TILE_ORDINAL[0] = 2
TILE_ORDINAL[1] = 3

you could run:

TILE_ORDINAL[0] = 1
TILE_ORDINAL[1] = 3

or even:

TILE_ORDINAL[0] = 3
TILE_ORDINAL[1] = 0

Thus the TILE_ORDINAL property allows running any tile in any order in a benchmark run, provided the corresponding tile indexes for the other properties in Control.config are consistent. For example, in a four-tile run using the TILE_ORDINAL property, LOAD_SCALE_FACTORS[3] no longer also refers to the fourth tile in a run. It now refers specifically to Tile 3. So if Tile 3 was not included as one of the values in the TILE_ORDINAL list, it would skip this tile-specific load scaling and would instead run all tiles at the default LOAD_SCALE_FACTORS rate.

If this synchronization is performed via NTP, then you must assure that time synchronization does not occur in the middle of a benchmark run, as time shifts during a run can compromise response time measurements on the clients as well as compromise the jappserver workload's ability to accurately perform post-run database checks.

For example, for PRIME_HOST[0][0] = "myhostname:1098", from the SPECvirt directory on myhostname, start a client manager process as follows:

java -jar clientmgr.jar -p 1098 -log

Repeat this for each PRIME_HOST entry in Control.config. For a compliant benchmark, you will start four of these process for each tile (one each for jappserver, specweb, specimap, and specpoll).

ex. For WORKLOAD_CLIENTS[0] = "myhostname:1091" and CLIENT_LISTENER_PORT = "1088", from the SPECvirt directory on myhostname, I would start a client manager process as follows:

java -jar clientmgr.jar -p 1088 -log

Repeat this for each unique client host. Note that you do not use the port specified in WORKLOAD_CLIENTS -- that port is for the workload client to use to listen for RMI commands from its prime client. The CLIENT_LISTENER_PORT (1088) is used for communication between the SPECvirt prime controller and the client manager process.

The following example also assumes the daemons are being started on a "unix-like" prime controller and communication betwen the prime controller and the daemons occurs via the controller's serial ports. However, these daemons need not be local to the controller, and there are Windows executable files available for daemons connected to Windows systems.

Within the installation directory containing the SPECvirt and workload directories (/opt by default for a Unix/Linux environment) is a "SPECptd" directory that contains the ptd executable and script/batch files for starting the power and temperature daemons. The format for starting the (Linux) ptd is:

./ptd-linux-x86 [options] <device-type-#> <device-port>

From the /opt/SPECptd directory, running "./ptd-linux-x86" displays the invocation options for this executable. For communicating with a supported power meter, you can find the number that corresponds to your meter in this output. ("0" starts the ptd in dummy mode.) Of the paramter options listed in the output, the "-t" option (runs the ptd in temperature mode) and the "-p port" option are the most commonly used. Since the ptd by default tries to use port 8888, you must use the "-p port" option to override this value if that port is already in use by another ptd or other process.

As an example:

./ptd-linux-x86 -p 8890 8 /dev/ttyS0

starts a ptd daemon in power mode using port 8890 and communicates with a Yokogawa WT210 power meter connected to /dev/ttyS0 (COM1) of a (Linux-based) prime controller. Alternatively:

./ptd-linux-x86 -t -p8890 1000 /dev/ttyS0

runs the ptd in "temperature mode" with the ptd returning "dummy" temperature data.

Once the PTD executables are able to communicate with the power meters correctly when started, the next step is to tell the prime controller about these PTD settings in Control.config. The first parameter to change is to set USE_PTDS to "1". Once so set, the controller uses all PTDs defined via a PTD_HOSTS[x] entry. PTD_HOST is the hostname of the system running the PTD. For this example, since the three PTDs are running on the prime controller, we can simply set:

PTD_HOST[0] = localhost
PTD_HOST[1] = localhost
PTD_HOST[2] = localhost

Next tell the prime controller what port each of the PTDs are listening on. This must match the ports specified when invoking the PTD:

PTD_PORT[0] = 8888
PTD_PORT[1] = 8889
PTD_PORT[2] = 8890

Lastly tell the prime controller what the specified PTD is measuring: server power (SUT), external storage power(EXT_STOR), or in the case of ambient temperature measurement, which component the temperature sensor is near:

PTD_TARGET[0] = "SUT"
PTD_TARGET[1] = "EXT_STOR"
PTD_TARGET[2] = "SUT"

For the temperature daemon the PTD_TARGET is either SUT or EXT_STOR, depending on where ambient temperature is being measured. SAMPLE_RATE_OVERRIDE and OVERRIDE_RATE_MS should generally not be modified. LOCAL_HOSTNAME and LOCAL_PORT specify the local network interface and port to use to connect with the PTD_HOST. (In most cases, specifying these two properties is unnecessary, and they can be left commented out.) The following picture shows the addition of three power/temperature daemons, listening for commands on their respective RMI ports.

Stage 2: The power and temperature daemons are started.

The last step in configuring the PTDs is to link a specific PTD with a specific power meter description. This is done through the PWR.PTD_INDEX[] and TMP.PTD_INDEX[] properties in Testbed.config. For each power or temperature meter listed in Testbed.config there must be a PTD_INDEX value that corresponds to one of the PTD_HOST indexes in Control.config.

java -jar specvirt.jar -l

Stage 3: The specvirt prime controller is started.

The SPECvirt prime controller (specvirt) next tells the client managers that host the workload clients to start the workload clients:

Stage 4: The specvirt prime controller starts workload clients.

The prime controller then waits PRIME_START_DELAY seconds and then tells the client managers hosting the workload prime clients to start their prime clients.

Stage 5: The specvirt prime controller starts workload prime clients.

With all of the PTD, workload client, and prime client processes running, the benchmark has started and the remainder of the benchmark run involves communication between these processes and the SPECvirt prime controller (specvirt) as illustrated in the following figure.

Stage 6: Client-side benchmark runtime communication

If you have multiple values in LOAD_SCALE_FACTORS, it iterates through that number of load points as independent workload runs of a single SPECvirt benchmark run. That is, results from all iterations are reported in a single SPECvirt benchmark raw file. At the end of each iteration, after all prime clients have reported that their runs have ended, the prime controller cleans everything up and terminates the run. If the client and prime client processes have exited correctly, you should see three extra line-feeds after the "Done killing procs ..." message on each client manager console. These extra lines are added as demarcation points between run intervals, but also provide a quick way of determining whether the client manager process cleaned up everything correctly. If you don't see these line feeds, stop and restart the client manager process before attempting another benchmark run.

Between load point intervals, the prime controller waits QUIESCE_SECONDS and then starts the next load interval. Polling data as well as the performance/QOS-related data from each run interval is included in the specvirt raw file (specvirt-*.raw) in the SPECvirt results subdirectory.

Once the benchmark has ended, to start another run on the SPECvirt prime controller, invoke:

java -jar specvirt.jar -l

The client manager processes on the benchmark clients remain running, as do any ptd processes, so you only need to restart the SPECvirt prime controller. Note that while the number of clientmgr processes increases with increasing numbers of tiles (i.e. increasing load), there is always just one prime controller and one set of power/temperature daemons, regardless of the number of tiles.

If the report requires editing, modify the properties in the raw file rather than in Control.config or Testbed.config. Within the raw file, however, RESULT_TYPE is the only editable property from Control.config. Other than this, only the properties contained in Testbed.config may be edited in a raw file.

Once edited, to regenerate the formatted results using the edited raw file, invoke the reporter by passing it the name of the raw file as a parameter. Ex:

java -jar reporter.jar -r <raw_file_name>

For a complete list of reporter invocation options, pass the reporter the "-h" argument.

java -jar reporter.jar -r <raw_file_name> [-t [1-7]]

If you wish to change the type of formatted result files generated without changing the RESULT_TYPE property in the raw file, override the value in the raw file by passing the “-t” parameter with the corresponding result type to the reporter. Otherwise, you can omit this parameter from the invocation string.

If you have a submission file and want to recreate the raw file from which it was generated, you can invoke:

java -jar reporter.jar -s <sub_file_name>

and it will strip out the extra characters from the submission file so that you can view or work with the original raw file. This is the recommended method for editing a file post-submission because it assures you are not working with an outdated version of the corresponding raw file and potentially introducing previously corrected errors into the "corrected" submission file.

1. Create separate, tile-specific master configuration file directories on the prime controller. You accomplish this by creating master directories for workload configuration files on the prime controller (for example, /opt/master_confs/tile2 and /opt/master_confs/tile3 as described in detail below) and putting the customized master configuration files for each workload in these directories.
   
   
2. The client processes of the four workloads for the first tile need separate configuration files from those for the second tile on the client and, in addition, separate workload result directories for each tile. You accomplish this by separating the workload home directories on the client to create a clear mapping of the workload result directories for each tile and by editing the Control.config file to match the resulting mappings. 
   Specifically, create separate workload home directories on the client as complete copies (for example, /opt/SPECimap and /opt/SPECimap_B). For SPECimap and SPECjAppServer also define different port numbers for RMI communication in the configuration files for both instances of each workload. For SPECjAppServer also define tile-specific result directories in order to get distinct directories for each tile. Finally, edit Control.config to reflect these new directory mappings as well as to assign tile indexes and workload indexes for each specific workload and client. 
   
   
3. The SPECjAppServer2004 aliases specdelivery and specemulator for the appserver VM are no longer used and instead require the explicit VM host names. You accomplish this by editing the master configuration files for this workload and editing the hosts file on the client.