SPEC OMP2001 Config files

Last updated: 16 May 2001 4pm bkp (
To check for possible updates to this document, please see http://www.spec.org/hpg/omp2001 )


  1. Introduction
  2. Config file options for runspec
  3. Config file options for specmake
  4. Config file options for the shell
  5. Config file options for the reader
  6. Files output during a build
  7. Troubleshooting

Detailed Contents

  1. Introduction
    1. What is a config file? (Background: benchmark philosophy.)
    2. What does a config file affect?
      1. runspec
      2. specmake
      3. the shell
      4. the reader of the results
    3. Config file structure
      1. Comments and whitespace
      2. Header section
      3. Named sections
        1. Precedence for the benchmark specifier
        2. Precedence for the tuning specifier
        3. Precedence for the extension specifier
        4. Combining specifier types
        5. Precedence among section types
      4. MD5 section
      5. Variable Substitution
        1. By runspec
        2. By the shell (example: submitting to multiple nodes)
        3. By specmake
      6. Shell-like "here documents" and continued lines
      7. Included files
  2. Config file options for runspec
    1. Options for runspec that can also be used on the command line
    2. Options for runspec that can only be used in a config file
  3. Config file options for specmake
    CC, CXX, FC
  4. Config file options for the shell
  5. Config file options for the reader
    1. Pre-defined fields
      hw_parallel - May be blank, None, OpenMP, Automatic, or "Both automatic and OpenMP"
    2. Free form notes
  6. Files output during a build
    1. Automatic backup of config files
    2. The log file and verbosity levels
    3. Log file example: Feedback-directed optimization.
    4. Help, I've got too many logs!
    5. Finding the build directory
    6. Files in the build directory
    7. For more information
  7. Troubleshooting

I. Introduction

This document assumes that you have already read "runspec.html". There, you learned about the primary user interface for running SPEC OMP2001; with this document, attention turns more toward how things work inside.

A. What is a config file? (Background: benchmark philosophy.)

A config file contains:

(Readers who just want to get on with the technical description should skip the remainder of this section; those who are interested in benchmark design philosophy should continue reading.)

A key decision that must be made by designers of a benchmark suite is whether to allow the benchmark source code to be changed when the suite is used.

If source code changes are allowed:

+ The benchmark can be adapted to the system under test.
+ Portability may be easier.
- But it may be hard to compare results between systems, unless some formal audit is done to ensure that comparable work is done.

If source code changes are not allowed:

+ Results may be easier to compare.
- It may take more time and effort to develop the benchmark, because portability will have to be built in ahead of time.
- Portability may be hard to achieve, at least for real applications. (Simple loops of 15 lines can port with little effort, and such benchmarks have their uses. But real applications are more complex.)

SPEC has chosen not to allow source code changes for the OMP2001 suite, except under very limited circumstances described below. By restricting source code changes, we separate the activity of porting benchmarks (which has a goal of being performance neutral), from the activity of using the benchmarks (where the goal is not neutrality, it's Get The Best Score You Can.) Prior to the first use of OMP2001, SPEC therefore invested substantial effort to port the suite to as many platforms as practical.

Are source code changes ever allowed? Yes, for Peak; Normally no, for base. If you discover a reason why you believe such a change is required, SPEC wants to hear about it, and will consider such requests for a future revision of the suite. SPEC will normally not approve publication of OMP2001 results using modified source code for base, unless such modifications are unavoidable for the target environment. All modification must be submitted to SPEC, are made available to all users of the suite, and are formally approved by a vote.

So, if source code changes are not allowed for base, but the benchmarks must be compiled in a wide variety of environments, can the users at least write their own Makefiles, and select "-D" options to select different environments? The answer to these two questions are "no", and "yes", respectively:

* No, you can't write your own Makefiles.
* But yes, you can select portability options (such as "-D").

You do this in the config file, which contains a centralized collection of all the portability options and optimization options for all the benchmarks in a OMP2001 suite. The SPEC tools then automatically generate the Makefiles for you.

The config file contains places where you can specify the characteristics of both your compile time and run time environments. It allows the advanced user to perform detailed manipulation of Makefile options, but retains all the changes in one place so that they can be examined and reproduced.

The config file is one of the key ingredients in making results reproducible. For example, if a customer would like to run the OMP2001 suite on her own SuperHero Model 4 and discover how close results are in her environment to the environment used when the vendor published a OMP2001 result, she should be able to do that using only 3 ingredients:

B. What does a config file affect?

A config file contains directives that are addressed to four different consumers: runspec, specmake, the shell, and the reader of the results:

1. runspec
Various aspects of the operation of runspec can be set within the config file. For example, if michael.cfgincludes the lines:
output_format = asc,ps
tune = base
reportable = 1
runlist = medium
then the defaults for the runspec command would change as specified. A user who types either of the following two commands would get precisely the same effect:
runspec --config=michael
runspec --config=michael --output=asc,ps --tune=base --reportable medium
2. specmake
The tool 'specmake' is simply gnu make renamed to avoid any possible conflicts with other versions of make that may be on your system. Config file lines such as:
CC = cc
are written to the Makefile set that is ultimately used to build the benchmark, and these lines are interpreted by specmake.
3. The shell
Certain commands are interpreted by the shell, for example:
fdo_pre0 = mkdir /tmp/greg; rm -f /tmp/greg/*
Runspec passes the above command to the shell prior to running a training run for feedback directed optimization, and the shell actually carries out the requested commands. It is therefore important to peruse a config file you are given before using it.
4. The reader of the results
A SPEC OMP2001 result, when submitted to http://www.spec.org, is expected to contain all the information needed so that readers can understand exactly what was tested. Config file lines such as these are addressed to that reader:
test_date = Nov-2001
hw_avail = Apr-2001
sw_avail = May-2001
notes015 = Note: Feedback directed optimization was not used
In addition, the config file itself is available to the reader at http://www.spec.org. The config file is presented in its entirety, with one exception described below under "Comments and whitespace". The reason that the config file is made available is because it is so important to reproducing results, as described in the Introduction. The config file is saved on every run, as a compressed portion of the rawfile.

C. Config file structure

1. Comments and whitespace

Comment lines begin with a "#", and can be placed anywhere in a config file. When the config file is saved (as an encoded portion of the rawfile), the comments are included. But if a comment line begins with "#>", it will not be saved in the rawfile. Thus you could use "#" for most of your comments, and use "#>" for proprietary information, such as:

#> I didn't use the C++ beta test because of Bob's big back-end bug.

Blank lines can be placed anywhere in a config file.

Spaces within a line are almost always ignored. Of course, you wouldn't want to spell OPTIMIZE as OPT I MIZE, but you are perfectly welcome to do either of the following:


The one place where spaces are considered significant is in notes, where the tools assume you are trying to line up your comments in the full disclosure reports.

2. Header section

A config file contains the following:

Most attempts to address runspec itself must be done in the header section. For example, if you want to set "reportable=1", you must do so before any occurrences of section markers.

The next several pages are devoted to sections that you name; the MD5 section is explained after that.

3. Named Sections

A "section marker" is a four-part string of the form:


These are referred to below as the 4 "section specifiers". The allowed values for the section specifiers are:

   	 benchmark:  default
                    Any individual benchmark, such as 310.wupwise_m

        tuning:     default

        extension:  default
                    any arbitrary string, such as "cloyce-genius"

        machine:    default (As of the date of this documentation,
                            this feature is only partly implemented.
                            Leave it at default unless you feel
                            particularly courageous.)

Trailing default sections may be omitted. Thus all of the following are equivalent:


Section markers can be entered in any order. Section markers can be repeated; material from identical section markers will automatically be consolidated. That is, you are welcome to start one section, start a different one, then go back and add more material to the first section. But please note that since there is no marker for the header section, you cannot go back to it.

a. Precedence for the benchmark specifier

By constructing section markers, you specify how you would like your options applied, with powerful defaulting and over-riding capabilities. Let's examine these capabilities by way of examples. We start with a simple config file:

		 % cat tmp.cfg
        runlist       = swim_m
        size          = test
        iterations    = 1
        tune          = base
        output_format = asc
        teeout        = 1

        OPTIMIZE = -O2

        % runspec --config=tmp | grep swim.f
        f90 -c -o swim.o            -O2   swim.f

The config file above is designed for quick, simple testing: it runs only one benchmark, namely 312.swim_m, using the smallest (test) workload, runs it only once, uses only base tuning, outputs only the text (ASCII) format report, and displays the build commands to the screen. To use it, we issue a runspeccommand, and pipe the output to grep to search for the actual generated compile command. (Alternatively, on NT, we could use findstron the generated log file). And, indeed, the tuning applied was the expected -O2.

Now consider this example, where a section marker is added that has the first specifier set to "medium", for the medium suite:

     	 % cat tmp.cfg
        runlist       = swim_m
        size          = test
        iterations    = 1
        tune          = base
        output_format = asc
        teeout        = 1

        OPTIMIZE = -O2

        OPTIMIZE = -O3

        % runspec --config=tmp | grep swim.f
        f90 -c -o swim.o            -O3   swim.f

The second OPTIMIZE command is used above because the reference to the floating point suite is considered to be more specific than the overall default.

Furthermore, we can add a specifier that mentions swim_m by name:

        % cat tmp.cfg
        runlist       = swim_m
        size          = test
        iterations    = 1
        tune          = base
        output_format = asc
        teeout        = 1

        OPTIMIZE = -O2

        OPTIMIZE = -O3

        OPTIMIZE = -O4

        % runspec --config=tmp | grep swim.f
        f90 -c -o swim.o            -O4   swim.f

The third OPTIMIZE command wins above, because it is included in the section that is considered to be the most specific. But what if we had said these in a different order?

     	 % cat tmp.cfg
        runlist       = swim_m
        size          = test
        iterations    = 1
        tune          = base
        output_format = asc
        teeout        = 1

        OPTIMIZE = -O4

        OPTIMIZE = -O2

        OPTIMIZE = -O3

        % runspec --config=tmp | grep swim.f
        f90 -c -o swim.o            -O4   swim.f

Notice above that the order of entry is not significant; it's the order of precedence from least specific to most specific.

Note: when a specifier is listed more than once at the same descriptive level, the last instance of the specifier is used. Consider this case:

       OPTIMIZE = -O4

       OPTIMIZE = -O3

The ending value of OPTIMIZE for 312.swim_m is '-O3', not '-O4'.

b. Precedence for the tuning specifier

The tuning specifier is the the second in the group of four, taking the value "default", "base", or "peak". Here is an example of its use:

      	 % cat tmp.cfg
        runlist       = swim_m
        size          = test
        iterations    = 1
        tune          = base,peak
        output_format = asc
        teeout        = 1

        FC       = f90

        FC       = kf90

        % runspec --config=tmp | grep swim.f | grep swim.o
        kf90 -c -o swim.o                swim.f

In the above example, we compile swim_m twice: once for base tuning, and once for peak. Notice that in both cases the compilers defined by the more specific section markers have been used, namely kf90, rather than the f90 from default=default=default=default.

c. Precedence for the extension specifier

Now let's examine the third section specifier, extension:

		 % cat tmp.cfg
        runlist       = swim_m
        size          = test
        iterations    = 1
        tune          = base
        output_format = asc
        teeout        = 1

        LIBS = -lm

        LIBS = -lm_ev6

        % runspec --config=tmp | grep swim.o
        f90 -c -o swim.o                swim.f
        f90         swim.o  -lm   -o swim_m
        % runspec --config=tmp --extension=ev6 | grep swim.o
        f90 -c -o swim.o                swim.f
        f90         swim.o  -lm_ev6   -o swim_m

        % cd $SPEC/benchspec/OMPM2001/312.swim_m/exe
        % ls -lt | head -3
        total 3400
        -rwxr-xr-x   1 john     users      65536 Dec 10 02:47 swim_m_base.ev6
        -rwxr-xr-x   1 john     users      49152 Dec 10 02:47 swim_m_base.none

Notice above that two different versions of swim_m were built from the same config file, and both executables are present in the exe directory for swim_m.

d. Combining specifier types

What if more than one section applies to a particular benchmark? Consider this example:

		 % cat tmp.cfg
        runlist       = swim_m
        size          = test
        iterations    = 1
        tune          = peak
        output_format = asc
        teeout        = 1

        OPTIMIZE = -O2
        FC       = f90
        LIBS     = -lm

        OPTIMIZE = -O4

        FC      = kf90

        LIBS = -lm_ev6

        % runspec --config=tmp --extension=ev6 | grep swim.o
        kf90 -c -o swim.o            -O4   swim.f
        kf90     -O4   swim.o  -lm_ev6   -o swim_m

Notice above that all three sections applied: the section specifier for 312.swim_m, the specifier for peak tuning, and the specifier for extension ev6.

e. Precedence among section types

If sections contradict each other, the order of precedence is:

 		highest     benchmark
       lowest      extension

And this order can be demonstrated as follows:

		 % cat tmp.cfg
        size          = test
        iterations    = 1
        output_format = asc
        teeout        = 1

        OPTIMIZE = -O0
        OPTIMIZE = -O1
        OPTIMIZE = -O3
        OPTIMIZE = -O4
        OPTIMIZE = -O5

        % runspec --conf=tmp swim_m | grep swim.f
   [1]  f90 -c -o swim.o            -O3   swim.f
        % runspec --conf=tmp --tune=peak swim_m | grep swim.f
   [2]  f90 -c -o swim.o            -O3   swim.f
        % runspec --conf=tmp --extension=ev6 swim_m | grep swim.f
   [3]  f90 -c -o swim.o            -O3   swim.f
        % runspec --conf=tmp --tune=base mgrid | grep mgrid.f
   [4]  f90 -c -o mgrid.o            -O1   mgrid.f
        % runspec --conf=tmp --tune=peak mgrid | grep mgrid.f
   [5]  f90 -c -o mgrid.o            -O1   mgrid.f
        % runspec --conf=tmp --extension=ev6 mgrid | grep mgrid.f
   [6]  f90 -c -o mgrid.o            -O1   mgrid.f
        % runspec --conf=tmp --tune=base gzip | grep gzip.c
   [7]  cc -c -o gzip.o            -O0   gzip.c
        % runspec --conf=tmp --tune=peak gzip | grep gzip.c
   [8]  cc -c -o gzip.o            -O4   gzip.c
        % runspec --conf=tmp --extension=ev6 gzip | grep gzip.c
   [9]  cc -c -o gzip.o            -O5   gzip.c
        % runspec --conf=tmp --tune=peak --extension=ev6 gzip | grep gzip.c
   [10] cc -c -o gzip.o            -O4   gzip.c

Notice above that the named benchmark always wins: lines [1], [2], and [3]. If there is no section specifier that names a benchmark, but there is a section specifier that names a suite, then the suite wins: lines [4], [5], and [6]. If there are no applicable benchmark or suite specifiers, then tuning or extension can be applied: lines [8] and [9]. But if both tuning and extension are applied, tuning wins [10].

4. MD5

The final section of your config file is generated automatically by the tools at compile time, and looks something like this:

	# Last updated Sun Nov 14 04:49:15 2001

	# Last updated Sun Nov 14 04:49:22 2001

The "MD5" is a checksum that ensures that the binaries referenced in the config file are in fact built using the options described therein. For example, if you edit the config file to change the optimization level for 255.vortex, the next time the file is used the tools will notice the change and will recompile vortex.

You can optionally disable this behavior, but doing so is strongly discouraged. See the sarcastic remarks in the description of "check_md5", below.

If you would like to see what portions of your config file are used in computing the MD5 hash, runspec with --debug=30 or higher, and examine the log file.

5. Variable Substitution

You can do variable substitution using your config file. But, as described in the introduction, a config file can affect runspec itself, specmake, and the shell. Therefore, effective use of variable substitution requires you to be aware of which software is doing the substitution:

a. By runspec

Substitution by runspec itself uses perl variable interpolation. Only perl scalars (denoted by a leading $) can be interpolated. For example, notes001 below uses the log file number (generated by the tools) and the hardware availability date (which was set directly):

 		  % cat tmp.cfg
         size          = test
         iterations    = 1
         output_format = asc
         teeout        = 1
         expand_notes  = 1
         runlist       = swim_m
         tune          = base
         hw_avail      = May-2001
         notes001      = This run is from log.$lognum with hw_avail $hw_avail

         % runspec -c tmp | grep asc
         Identifying output formats...asc...config...html...pdf...ps...raw...
                 format: ASCII -> /greg/result/OMPM2001.101.asc
         % grep with /greg/result/OMPM2001.101.asc
              This run is from log.101 with hw_avail May-2001

As another example, let's say you want to go for minimal performance. You might want to do this with the 'nice' command. You can say:

submit=nice 20 '$command'

and the $command gets expanded to whatever would normally be executed but with 'nice 20' stuck in front of it.

If you happen to need to put some text immediately after a variable, that's possible as well. In order to make it possible for the parser to see the variable that you want, you'll have to use braces:

 		  % tail -2 tmp.cfg
         notes001 =You have done ${lognum}x runs tonight, aren't you tired yet?

         % runspec -c tmp | grep asc
         Identifying output formats...asc...config...html...pdf...ps...raw...
                 format: ASCII -> /john/result/OMPM2001.103.asc
         % grep done /john/result/OMPM2001.103.asc
              You have done 103x runs tonight, aren't you tired yet?

Perhaps a more useful example is this one, which directs the shell to clean files related to the current executable in the temporary directory, before doing a training run for feedback directed optimization:

fdo_pre0 = mkdir /tmp/pb; rm -f /tmp/pb/${baseexe}*

NOTICE in this example that although the commands are carried out by the shell, the variable substitution is done by runspec.

If you'd like a complete list of the variables that you can use in your commands (relative to the config file you're using), set runspec's verbosity to 80 or higher (-v 80) and either do a run that causes a command substitution to happen, or run with expand_notes set to 1.

b. By the shell (example: submitting to multiple nodes)

Because perl variables look a lot like shell variables, you need to specially protect shell variables if you want to prevent perl from trying to interpret them. Notice what happens with the protected and unprotected versions:

		 % cat tmp.cfg
        size                 = test
        iterations           = 1
        output_format        = asc
        teeout               = 1
        expand_notes         = 1
        runlist              = swim_m
        tune                 = base,peak
        use_submit_for_speed = 1

        submit = echo "home=$HOME; spec=$SPEC;" >/tmp/chan; $command

        submit = echo "home=\$HOME; spec=\$SPEC;" > /tmp/nui; $command

        % runspec --config=tmp > /dev/null
        % cat /tmp/chan
        home=; spec=;
        % cat /tmp/nui
        home=/usr/users/chris; spec=/omp2001;

Here is a more complex example which uses both substitution by runspec and substitution by the shell (line wraps added for readability):

 				 submit= let "MYNUM=$SPECUSERNUM" ; 
                let "NODE=\$MYNUM/2"; 
                export NODE=/hw/nodenum/\$NODE; 
                let "CPU=2*((\$MYNUM+1)/2)-\$MYNUM "; 
                export CPU; 
                /usr/sbin/dplace -place \$SPEC/submit.pf -mustrun $command

Runspec substitutes the current user number for $SPECUSERNUM, and then passes the above command to the shell which does the substitutions for CPU and NODE. This example originated on an SGI Origin system, where there are two CPUs per node. Suppose that runspec is about to submit the copy for user number 17. In that case:

              let "MYNUM=$SPECUSERNUM" ;             # i.e. 17
              let "NODE=\$MYNUM/2";                  # So, NODE=8
              export NODE=/hw/nodenum/\$NODE;        # Now NODE=/hw/nodenum/8
              let "CPU=2*((\$MYNUM+1)/2)-\$MYNUM ";  # and CPU=1
              export CPU; 
              /usr/sbin/dplace                       # So we execute dplace
                -place \$SPEC/submit.pf              # using $SPEC/submit.pf
                -mustrun $command                    # and the expected command

The desired $command is run under the control of submit.pf, with $NODE=/hw/nodenum/8 and $CPU=1. Here are the contents of the placement file, submit.pf:

			   memories 1 in topology physical near $NODE
              threads 1
              run thread 0 on memory 0 using cpu $CPU

Two important notes about the above submit command:

  1. $SPECUSERNUM must be spelled exactly as it is in the examples above. It is a limitation of the tools that saying things like '${SPECUSERNUM}' just won't work.
  2. Only the first occurrence is substituted. That's why the example above immediately assigns the result to MYNUM, which can then be used repeatedly.

Please note that, by default, 'submit' is only applied to rate runs. To use it for speed runs as well, you would set

use_submit_for_speed = yes

in your configuration file.

c. By specmake

For an example of variable substitution handled by specmake, see


Search that file for LIBS, and note the long comment which provides a walk-through of a complex substitution handled by specmake.

d. Limitations on variable substitution

Once runspec hands control over to specmake or to the shell, the results of further substitution are invisible to runspec. For this reason, you can't say:

		   wrong:  MYDIR = /usr/paul/compilers
                  FMY    = $(MYDIR)/f90
                  notes001 = compiler: $(FMY)

6. Shell-like "here documents" and continued lines

Shell-style "here documents" are supported for setting variables to multi-line values. Continued lines (with \) are also supported:

 		  $ cat tmp2.cfg

         foo = <<EOT
         This +
          is a +
          test +

         bar = \
                    and +\
                    so  +\
                    is  +\
         notes01 = $foo
         notes02 = $bar
         $ runspec --config=tmp2 --size=test --iterat=1 swim_m | grep asc
         Identifying output formats...asc...config...html...pdf...ps...raw...
                 format: ASCII -> /omp2001/result/OMPM2001.024.asc
         $ grep + ../result/*24.asc
              This +
              is a +
              test +
                        and +
                        so  +
                        is  +

7. Included files

It is possible to include another file in your config file. A typical use for this feature might be to keep all the software information in the main config file, but to include the hardware information about the current System Under Test (SUT) in another file. For example:

	   % cat tmp.cfg
      sw_compiler = GoFast C/C++ V4.2
      sw_avail    = Mar-2001
      include: SUT.inc

      OPTIMIZE = -O4

      % cat SUT.inc
      hw_model = SuperHero IV
      hw_avail = Feb-2001

      % runspec --config=tmp --iterations=1 --size=test swim_m

After the above command is completed, we end up with a report that mentions both hardware and software dates:

	... Hardware availability: Feb-2001
	... Software availability: Mar-2001

II. Config file options for runspec

This section documents options that control the operation of runspec itself.

A. Options for runspec that can also be used on the command line

The following items can be specified in a config file, and have the same meaning as if they are specified on the runspec command line. Please see runspec.html for details. If an item is specified in both places, the command line wins.

All of these should be specified in your config file header section, unless the description states that the item can be applied to individual benchmarks.

    option              default     Meaning
    -----------------   ---------   -------------------------------------------
    action              validate    What to do.  

    deletework          0           If set to 1, always delete existing 
                                    benchmark working directories.  An
                                    extra-careful person might want to 
                                    set this to ensure no unwanted leftovers 
                                    from previous benchmark runs, but 
                                    the tools are already trying to enforce 
                                    that property.

    ext                 none        Extension for executables created
    ignore_errors       0           Ignore certain errors.  Very useful when
                                    debugging a new compiler and new set of

    iterations          2           Number of iterations to run

    mach                default     Default machine id

    make_no_clobber     0           Don't delete directory when building 
                                    executables.  This option should only be 
                                    used for troubleshooting a problematic 
                                    compile.  It is against the run rules
                                    to use it when building binaries for an 
                                    actual submission.

    max_active_compares <# users>   Max number of parallel compares.  Useful 
                                    when doing large SPECrate runs on a 
                                    system with lots of CPUs.  Do not set 
                                    a value larger than the number of users[NS].

    output_format       all         Format for reports.  Valid options are
                                    asc (ASCII text), html, pdf, and ps.  You 
                                    might prefer to set this to "asc" if you're
                                    going to be doing lots of runs, and only
                                    create the pretty reports at the end of
                                    the series.  See runspec --output_format 
                                    and --rawformat.

    reportable          0           Strictly follow reporting rules.
                                    You must set reportable, in order to 
                                    generate a valid run suitable for 
                                    submission to SPEC. 
    rate                0           Rate vs Speed  (1=rate, 0=speed)[NS]
    rebuild             0           Rebuild binaries even if they exist 

    runlist             none        What benchmarks to run

    setprocgroup        1           Set the process group.  On Unix-like
                                    systems, improves the chances that ^C
                                    gets the whole run, not just one of 
                                    the children

    size                ref         Size of input set.  If you are in the
                                    early stages of testing a new compiler
                                    or new set of options, you might set
                                    this to test or train.  

    table               1           In ASCII reports, include information 
                                    about each execution of the benchmark.

    tune                base        default tuning level.  In a reportable run,
                                    must be one of "all" or "base".

    unbuffer            0           Unbuffer STDOUT.  For CPU95, there were
                                    occasional complaints about redundant
                                    output when a child process would flush
                                    its buffer.  If similar problems occur
                                    for OMP2001, try setting this to 1.
    users               1           Number of users (see runspec --users).  
                                    This option can also be used for specific
                                    benchmarks - e.g. you could decide to
                                    run 64 copies of all benchmarks except
                                    gcc, which would run only 63.   For 
                                    base, the number of copies must be the
                                    same for all benchmarks, but for peak
                                    it is allowed to vary.[NS]

    verbose             5           Verbosity level.  Select level 1 through
                                    99 to control how much debugging info
                                    runspec prints out.  For more information,
                                    see the section on the log file, elsewhere
                                    in this document.

    [NS]: This option is not supported.

B. Options for runspec that can only be used in a config file

The following options control the operation of runspec, but are not specified on the command line. Instead, they must be specified in the config file.

All of these should be specified in your config file header section, unless the description states that the item can be applied to individual benchmarks.

    option              default     Meaning
    -----------------   ---------   -------------------------------------------
    backup_config       1           When updating the MD5 hashes in the 
                                    config file, make a backup copy first.
                                    Highly recommended to defend against
                                    full-file-system errors, system crashes, 
                                    or other unfortunate events.

    basepeak            0           Use base binary and/or base result for
                                    peak.  If applied to the whole suite (in
                                    the header section), then only base is
                                    run, and its results are reported for both
                                    the base and peak metrics.  If applied to 
                                    a single benchmark, the same binary will 
                                    be used for both base and peak runs, and
                                    the lower of the two medians will be 
                                    reported for both.
    check_md5           1           Runspec uses MD5 hashes to verify that 
                                    executables match the config file that 
                                    invokes them, and if they do not, runspec
                                    forces a recompile.  You can turn that
                                    feature off by setting check_md5=0. 
                                         WARNING: If you turn this feature 
                                    off, you effectively say that you are 
                                    willing to run a benchmark even if you 
                                    don't know what you did or how you did
                                    it -- that is, you lack information as 
                                    to how it was built!  Since SPEC requires 
                                    that you disclose how you built it, such 
                                    a run would not be submittable to SPEC.  

    difflines           10          Number of lines of differences to print
                                    when comparing results.

    env_vars            0           Allow environment to be overridden by ENV_*
                                    For example:


                                    will cause the tools to spawn gzip (or 
                                    anything related, like monitor_prebench)
                                    with OMP_NUM_THREADS=16

    expand_notes        0           If set, will expand variables in notes.
                                    This capability is limited because
                                    notes are NOT processed by specmake, so you
				    cannot do repeated substitutions. 

    feedback            1           Normally, feedback is controlled by the
                                    presence of one or more PASSn options
                                    (see the documentation of make variables).
                                    An additional control is provided by this
                                    config file option, which can be used to
                                    selectively turn feedback off for
                                    individual benchmarks.  Note: in a base
                                    run, all benchmarks in a given language
                                    must use the same settings for feedback.

    ignore_sigint       0           Ignore SIGINT.  If this is set, runspec
                                    will attempt to ignore you when you press
                                    ^C.  (It is not clear why one might want
                                    to set this feature.)

    line_width          0           line wrap width for screen

    locking             1           Try to lock files

    log_line_width      0           line wrap width for logfiles.  If your
                                    editor complains about lines being too
                                    long when you look at logfiles, try
                                    setting this to some reasonable value,
                                    such as 80 or 132.

    make                specmake    Name of make executable.  Note that  
                                    the run rules require use of specmake for
                                    reportable results.  

    makeflags           ''          Extra flags for make (such as -j).  Use
                                    only if you are familiar with gnu make.  

    mean_anyway         0           Calculate mean even if invalid.
                                    DANGER this will write a mean to all
                                    reports even if no valid mean can
                                    be computed (e.g. half the benchmarks
                                    failed!)  DANGER

    minimize_rundirs    0           Try to keep working disk size down. 
                                    Cannot be used in a reportable run.
    minimize_builddirs  0           Try to keep working disk size down 
                                    during builds

    peaksrc             ''          Name of subdirectory under
                                    <benchmark>/src/peak from which to draw
				    approved source code modifications. This
				    is only used for peak runs, and can be
				    set on a per-benchmark basis.

    submit              undef       Commands used to submit jobs; typically
                                    used for large rate runs.  See the
                                    section on variable substitution to
                                    discover how to make this feature useful.

    teeout              0           Run output [from build] through tee 
                                    so you can see it on the screen
    use_submit_for_speed 0          If set, use submit commands for speed
                                    runs as well as rate runs.  Handy for
                                    running the benchmarks on a simulator, etc.

III. Config file options for specmake

For a complete list of options that are used by specmake, please see $SPEC/docs/makevars.txt and notice which variables are documented as OK to mention in a config file.

Here are the commonly used variables:

    CC          How to invoke your C Compiler
    CXX         How to invoke your C++ Compiler
    FC          How to invoke Fortran (the one that handles both .f and .f90) 

    CLD         How to invoke the Linker when compiling C programs
    CXXLD       How to invoke the Linker when compiling C++ programs
    FLD         How to invoke the Linker when compiling Fortran programs

    ONESTEP     If set, build from sources directly to final binary.  
                See the discussion in rule of 
                $SPEC/docs/runrules.html or %SPEC%\docs.nt\runrules.html

    OPTIMIZE    Optimization flags to be applied for all compilers
    COPTIMIZE   Optimization flags to be applied when using your C compiler
    CXXOPTIMIZE Ditto, for C++
    FOPTIMIZE   Ditto, for Fortran

    PORTABILITY    Portability flags to be applied no matter what the compiler
    CPORTABILITY   Portability flags to be applied when using your C compiler
    CXXPORTABILITY Ditto, for C++
    FPORTABILITY   Ditto, for Fortran

    RM_SOURCES  Remove a source file.  Should only be used for library
                substitutions that comply with run rule 2.1.2

    PASSn_CFLAGS    Flags for pass "n" C compilation when doing profile-
                    directed feedback.  Typically n is either 1 or 2, for 
                    the compile done before the training run and the compile 
                    done after the training run.  Search for the word
                    "feedback" in $SPEC/docs/runrules.html for more info;
                    especially rule 2.2.3.  See the feedback examples in
                    example-medium.cfg and example-advanced.cfg from
                    $SPEC/docs/ or %SPEC%\docs.nt\ .
    PASSn_CXXFLAGS  Ditto, for C++
    PASSn_FFLAGS    Ditto, for Fortran

Note that you can also make up your own variable names, which specmake will use (and perform substitution on). For an example of this feature, see $SPEC/docs/example-advanced.cfg.

IV. Config file options for the shell

The following config file options will cause their contents to be executed by the shell (or the Windows/NT command interpreter):

    fdo_pre0        commands to be executed before starting a feedback
                    directed compilation series 
    fdo_preN        commands to be executed before pass N
    fdo_make_cleanN commands to be executed for cleanup at pass N
    fdo_pre_makeN   commands to be done prior to Nth compile
    fdo_make_passN  commands to actually do the Nth compile
    fdo_post_makeN  commands to be done after the Nth compile
    fdo_runN        commands to be used for Nth training run
    fdo_postN       commands to be done at the end of pass N

    submit          commands to be used for distributing jobs across
                    a multiprocessor system.  See the detailed example  
                    in the section on "Variable Substitution", above.

For an example of one of the fdo_ options, see the "Log file example" later in this document.

V. Config file options for the reader

Whether or not you submit your result to SPEC, you should fully disclose how you achieved the result. If it requires the installation of the GoFastLinker, you should say so. By setting the appropriate fields in the config file, you can cause information about the GoFastLinker to appear in all reports.

A. Pre-defined fields

Here are the pre-defined fields that you can set:

       company_name   The company performing the tests.  Will not be printed
                      in reports unless it differs from the field hw_vendor.
       hw_avail       Date hardware first shipped.  If more than one date
                      applies, use the LATEST one.
       hw_cpu         CPU type
       hw_cpu_mhz     Speed of the CPUs, in MHz
       hw_disk        Disk subsystem
       hw_fpu         Floating point unit
       hw_memory      Size of main memory
       hw_model       Model name 
       hw_ncpu        Number of CPUs configured.  Note that if your system
                      has the ability to turn CPUs off, for example through 
                      a firmware setting, then it is acceptable to report 
                      "1" here if only 1 CPU was enabled on an SMP system.  
                      But beware -- you need to ensure that your method is 
                      effective and that you are not silently getting help
                      (or interference!) from the allegedly turned-off CPUs.
       hw_ncpuorder   Valid number of CPUs orderable for this model.
                      For example, "2 to 16".
       hw_ocache      4th level or other form of cache
       hw_other       Any other performance-relevant hardware
       hw_pcache      1st level (primary) cache
       hw_scache      2nd level cache
       hw_tcache      3rd level cache
       hw_vendor      Name of manufacturer for hardware
       license_num    Your SPEC license number
       machine_name   Machine name: not currently used, leave blank
       prepared_by    Is never output.  If you wish, you could set this to
                      your own name, so that the rawfile will be tagged with
                      your name but not the formal reports. 
       display_order  The order that you want your outputs displayed.  Sorted
                      alphanumeric value. This is useful in displaying
                      multiple runs of the same system in order of the number
                      of cpus for example, or to put the newest results
                      at the top.  An example would be start out with the
                      first submission as 5, the next could be 55 (coming after)
                      or 45 coming before.
       sw_avail       Availability date for the software used.  If more than
                      one date, use the LATEST one.
       sw_compiler    Name and version of compiler 
       sw_file        File system (nfs, ufs, etc)
       sw_os          Operating system name and version
       sw_state       Multi-user, single-user, default, etc
       test_date      When you ran the tests
       tester_name    Your employer

Note that these fields are scoped by section headers - that is, you can vary them according to the type of run that you are doing. You can also continue any of these fields to another line by appending a numeral to it. Here is an example of both these features:

         sw_compiler  = Compaq C X6.2-259-449AT 
         CC           = cc -v      

         sw_compiler2 = DIGITAL C++ V6.1-029-408B6
         CXX          = cxx -v

         sw_compiler2 = Compaq Fortran V5.3
         sw_compiler3 = KAP Fortran V4.2
         FC           = kf90 -v

B. Free form notes

In addition, you can write as many free-form notes as you wish. Start your notes with the characters "notes", and then add numbers and/or underscores. The notes will be sorted before being printed. For example:

		% cat tmp.cfg
        size                 = test
        iterations           = 1
        output_format        = asc
        teeout               = 1
        runlist              = swim_m
        tune                 = base

        notes01_1 = ++ how
        notes02   = ++ you?
        notes01_2 = ++ are
        notes01   = ++ Alex,
        notes000  = ++ hi


        % runspec --config=tmp > /nev/dull
        % ls -t ../result/*asc | head -1
        % grep ++ ../result/OMP2001.111.asc
             ++ hi
             ++ Alex,
             ++ how
             ++ are
             ++ you?

You can also use notes to describe software or hardware information with more detail beyond the predefined fields. For an example of where this might be useful, see $SPEC/docs/example-medium.cfg and search for "patch".

VI. Files output during a build

A. Automatic backup of config files

It was mentioned above that the MD5 section of the config file is written automatically by the tools. Each time your config file is updated, a backup copy is made. Thus your config directory may soon come to look like this:

	% ls /home/jim/OMP2001/config/tmp.c*
	tmp.cfg             tmp.cfg.20011210aq  tmp.cfg.20011210j
	tmp.cfg.20011210    tmp.cfg.20011210ar  tmp.cfg.20011210k
	tmp.cfg.20011210a   tmp.cfg.20011210as  tmp.cfg.20011210l
	tmp.cfg.20011210aa  tmp.cfg.20011210at  tmp.cfg.20011210m
	tmp.cfg.20011210ab  tmp.cfg.20011210au  tmp.cfg.20011210n
	tmp.cfg.20011210ac  tmp.cfg.20011210av  tmp.cfg.20011210o
	tmp.cfg.20011210ad  tmp.cfg.20011210aw  tmp.cfg.20011210p
	tmp.cfg.20011210ae  tmp.cfg.20011210ax  tmp.cfg.20011210q
	tmp.cfg.20011210af  tmp.cfg.20011210ay  tmp.cfg.20011210r
	tmp.cfg.20011210ag  tmp.cfg.20011210az  tmp.cfg.20011210s
	tmp.cfg.20011210ah  tmp.cfg.20011210b   tmp.cfg.20011210t
	tmp.cfg.20011210ai  tmp.cfg.20011210ba  tmp.cfg.20011210u
	tmp.cfg.20011210aj  tmp.cfg.20011210c   tmp.cfg.20011210v
	tmp.cfg.20011210ak  tmp.cfg.20011210d   tmp.cfg.20011210w
	tmp.cfg.20011210al  tmp.cfg.20011210e   tmp.cfg.20011210x
	tmp.cfg.20011210am  tmp.cfg.20011210f   tmp.cfg.20011210y
	tmp.cfg.20011210an  tmp.cfg.20011210g   tmp.cfg.20011210z
	tmp.cfg.20011210ao  tmp.cfg.20011210h
	tmp.cfg.20011210ap  tmp.cfg.20011210i

If this feels like too much clutter, you can disable the backup mechanism, as described under "backup_config", below. Note that doing so may leave you with a risk of losing the config file in case of a filesystem overflow or system crash. A better idea may be to periodically remove the clutter, for example by typing:

 rm *.cfg.200112*

B. The log file and verbosity levels

$SPEC/result (Unix) or %SPEC%\result (NT) contains reports and log files, as mentioned in runspec.html. When you are doing a build, you will probably find that you want to pay close attention to the log file. Depending on the verbosity level that you have selected, it will contain enormous amounts of information about how your build went.

The OMP2001 tool suite provides for varying amounts of output about its actions during a run. These levels range from the bare minimum of output (level 0) to copious streams of information almost certainly worthless to anyone not developing the tools themselves (level 99). Note: selecting one output level gives you the output from all lower levels, which may cause you to wade through more output than you might like.

The 'level' referred to in the table is the <n> in 'runspec --verbose <n>'.

Levels higher than 99 are special; they are always output to your log file. You can also see them on the screen if you set verbosity to the specified level minus 100. For example, the default log level is 3. This means that on your screen you will get messages at levels 0 through 3, and 100 through 103. Additionally, in your log file, you'll find messages at levels 104 through 199.

   Level      What you get
   --------   ---------------------------------------------------------------
     0        Basic status information, and most errors.  These messages can
              not be turned off.

     1        List of the benchmarks which will be acted upon.

     2        A list of possible output formats, as well as notification
              when beginning and ending each phase of operation (build,
              setup, run, reporting).

     3 (default)   A list of each action performed during each phase of 
              operation (e.g. "Building 310.wupwise_m", "Setting up 312.swim_m")

     4        Notification of benchmarks excluded

    10        Information on basepeak operation.

    12        Errors during discovery of benchmarks and output formats.

    24        Notification of additions to and replacements in the list of

    30        A list of options included in the MD5 hash of options used to
              determine whether or not a given binary needs to be

    35        A list of key=value pairs that can be used in command and
              notes substitutions.

    40        A list of 'submit' commands for each benchmark.

    70        Information on selection of median results.

    99        Gruesome detail of comparing MD5 hashes on files being copied
              during run directory setup.

--- Messages at the following levels will always appear in your log files  ---

   100        Error message if the 'pdflib' module can't be loaded.

   103        A tally of successes and failures during the run broken down
              by benchmark.

   106        A list of runtime and calculated ratio for each benchmark run.

   107        Dividers to visually block each phase of the run.

   110        Error messages about writing temporary files in the config
              file output format.

   120        Messages about which commands are being issued for which

   125        A listing of each individual child processes' start, end, and
              elapsed times.

   130        A nice header with the time of the runspec invocation and the
              command line used.

   140        General information about the settings for the current run.

   145        Messages about file comparisons.

   150        List of commands that will be run, and details about the
              settings used for comparing output files.  Also the contents
              of the Makefile written.

   155        Start, end, and elapsed times for benchmark run.

   160        Start, end, and elapsed times for benchmark compilation.

   180        stdout and stderr from commands run

   191        Notification of command line used to run specinvoke.

C. Log file example: Feedback-directed optimization.

Let's consider the log file for a compile with feedback-directed optimization (FDO). This will serve both as an example of how you accomplish such a compile and how to interpret a log file. [Note: this section uses an actual log file, but white space has been liberally adjusted, and much extraneous output has been removed.]

Feedback-directed optimization typically means that we want to compile a program twice: the first compile creates an image with certain instrumentation. Then, we run the program, and data about that run is collected (a profile). When the program is re-compiled, the collected data is use to improve the optimization.

First, let's look at the config file entries that enabled FDO. The log file tells us what was written to the Makefile.spec, along with lots of state information:

      Building 330.art_m ref base nov14a default
      Wrote to makefile 
      PASS1_CFLAGS     = -prof_gen_noopt    -prof_dir /tmp/pb
      PASS2_CFLAGS     = -prof_use_feedback -prof_dir /tmp/pb
      baseexe          = parser
      fdo_pre0         = mkdir /tmp/pb; rm -f /tmp/pb/${baseexe}*
      feedback         = 1

To tell the tools that we wanted to use FDO, all we had to do was set PASSn_CFLAGS. If the tools see any use of PASSn_xxxxx, they will perform multiple compiles. The particular compiler used in this example expects to be invoked twice: once with "-prof_gen_noopt" and then again with "-prof_use_feedback". Note that we have also requested special processing before we start, in the fdo_pre0 step. The variable ${baseexe} is substituted with the name of the generated executable, minus any extensions or directories:

  Issuing fdo_pre0 command 'mkdir /tmp/pb; rm -f /tmp/pb/parser* 

Below, the first compile is done. Notice that specmake is invoked with FDO=PASS1. If you want to understand exactly how this affects the build, read $SPEC/benchspec/Makefile.defaults, along with the document $SPEC/docs/makevars.txt. Briefly, the fact that FDO=PASS1 causes the switches from PASS1_CFLAGS to be used.

 		 Output from fdo_make_pass1 
        'specmake  FDO=PASS1 build > fdo_make_pass1.out 2> fdo_make_pass1.err':
        cc -v  -prof_gen_noopt  -prof_dir /tmp/pb  -DSPEC_OMP2001  -v -arch ev6 
        -fast analyze-linkage.c and.c build-disjuncts.c extract-links.c ...

The next section shows how specinvoke runs the benchmark for the training run, according to the commands in speccmds.cmd. For more information on specinvoke, see $SPEC/docs/utility.html. Basically, the instrumented parser is called using the training data set as input.

      Training 330.art_m
      Commands to run: 
       -u /omp2001/benchspec/OMPM2001/330.art_m/run/00000002
       -i train.in -o train.out -e train.err ../00000002/parser 2.1.dict -batch

      Specinvoke: /omp2001/bin/specinvoke -d 
         -e speccmds.err -o speccmds.out -f speccmds.cmd

Finally, the compiler is run a second time, this time to use the profile feedback and build a new executable. Notice that this time, specmake is invoked with FDO=PASS2, which is why the compile picks up the PASS2_CFLAGS:

 	   Output from fdo_make_pass2 
      'specmake  FDO=PASS2 build > fdo_make_pass2.out 2> fdo_make_pass2.err':
      cc -v  -prof_use_feedback -prof_dir /tmp/pb -DSPEC_OMPM2001 -v -arch ev6  
        -fast analyze-linkage.c and.c build-disjuncts.c extract-links.c ...

        Compile for '330.art_m' ended at:Thu Dec 16 23:35:48 2001 (945405348)
        Elapsed compile for '330.art_m': 00:02:45 (165)

      Build Complete

And that's it. The tools did most of the work; the user simply set PASS1_CFLAGS, PASS2_CFLAGS, and fdo_pre0 in the config file.

D. Help, I've got too many logs!

If you do a very large number of builds and runs, you may find that your result directory gets far too cluttered. If it does, you should feel free to issue commands such as these on Unix systems:

 		 cd $SPEC
        mv result result_old
        mkdir result

On NT, you could say:

 		 cd %SPEC%
        rename result result_old
        mkdir result

E. Finding the build directory

As described under "About Disk Usage" in runspec.html, the OMP2001 tools do the actual builds and runs in numbered run directories. The benchmark sources are never modified in the src directory.

To find the build directory, look for the word "build" in run/list. If more than one build directory is present, you can pipe the output to search for the specific extension that you want.

For example:

  	  F:\omp2001> cd %SPEC%\benchspec\OMPM2001\310.wupwise_m\run

     F:\omp2001\benchspec\OMPM2001\310.wupwise_m\run>findstr build list
     00000001 dir=F:/omp2001/benchspec/OMPM2001/310.wupwise_m/run/00000001 
     ext=oct14a lock=0 type=build username=Administrator

     F:\omp2001\benchspec\OMPM2001\310.wupwise_m\run>cd *01

F. Files in the build directory

A variety of files are output to the build directory. Here are some of the key files which can usefully be examined:

	 Makefile.spec      - the components for make that were generated
                         for the current config file with the current
                         set of runspec options 

    options.out        - For 1 pass compile: build options summary
    options1.out       - For N pass compile: summary of first pass
    options2.out       - For N pass compile: summary of second pass

    make.out           - For 1 pass compile: detailed commands generated 
    fdo_make_pass1.out - For N pass compile: detailed commands
                         generated for 1st pass
    fdo_make_pass2.out - For N pass compile: detailed commands
                         generated for 2nd pass

    *.err              - The output from standard error corresponding to
                         the above files.

G. For more information

For more information about how the run directories work, see the descriptions of specinvoke, specmake, and specdiff in

  		$SPEC/docs/utility.html (Unix) or
       %SPEC%\docs.nt\utility.html (NT)

VII. Troubleshooting

When something goes wrong with a build, here are some things to check:

1) Are there any obvious clues in the log file? Search for the word "Building". Keep searching until you hit the next benchmark AFTER the one that you are interested in. Now scroll backward one screenful.

2) Did your desired switches get applied? Go to the build directory, and look at options*out.

3) Did the tools or your compilers report any errors? Look in the build directory at *err.

4) What happens if you try the build by hand? See the section on specmake in utility.html.

5) If an actual run fails, what happens if you invoke the run by hand? See the information about "specinvoke -n" in utility.html.

VII. New Features

    SPEC_CPU2000_NO_RUNDIR_NUKE   If this environment variable is set
                                  to a non-empty value, runspec will *never*
                                  touch a used rundir.   This feature might
                                  be useful when trying to preserve
                                  evidence about an intermittent failure.
                                  Note: Do *not* use this feature lightly!
                                  It causes severe benchmark tree bloat!

    SPEC_NEVER_TRY_PDF            If set to a non-empty value, PDF will
                                  not be attempted.  By default, PDF is

The following new config file directives have been added:

    command_add_redirect          If set, the generated command will have
                                  ">", "<", and "2>" redirection operators
                                  added to it, so that the shell will do
                                  I/O re-direction.  For more information,
                                  and an example of where this feature is
                                  useful, see config.html (section on Variable
                                  Substitution by runspec).

   srcalt                         If set, the tools will look for an
                                  approved alternate source in the
                                  named subdirectory.