SPEC MPI2007 Config Files

Version 2.00

(To check for possible updates to this document, please see http://www.spec.org/mpi2007/Docs/ )

Overview

Click one of the following to go to the detailed contents about that item:

I. Introduction

II. Config file options for runspec

III. Config file options for specmake

IV. Config file options for the shell

V. Config file options for readers

VI. Deprecated options

VII. The config file preprocessor

VIII. Output files - and how they relate to your config file

IX. About Alternate Sources

X. Troubleshooting

Contents

I. Introduction

A. What is a config file? (Background: benchmark philosophy.)

B. What does a config file affect?

1. runspec

2. specmake

3. The shell

4. Readers of the results

5. The config file preprocessor

C. Config file structure

1. Comments and whitespace

2. Header section

3. Named sections

a. Precedence of the benchmark specifier

Default benchmark

Named suite

Named benchmark

Order of differing sections does not matter

Order of the same section does matter

b. Precedence of the tuning specifier

c. Precedence of the extension specifier

Extension found in config file

Extension not found in config file

Extension override

d. Combining specifier types

e. Precedence among section types

4. MD5 section

5. Shell-like "here documents" and continued lines

6. Included files

D. Variable substitution

1. By runspec, at startup: $[variable]

2. By runspec, during a run: $variable and ${variable}

2. By the shell: \$VARIABLE

4. By specmake: $(VARIABLE)

5. Example: controlling rank counts in the AIX MPI driver

6. Limitations on variable substitution

7. Unsetting a variable with %undef%

II. Config file options for runspec

A. Options that can also be used on the runspec command line

action check_version delay deletework ext flagsurl http_proxy http_timeout ignore_errors info_wrap_columns iterations mach make_no_clobber nobuild notes_wrap_columns output_format ranks reportable rebuild runlist setprocgroup size table tune verbose

B. Options for runspec that can only be used in a config file

allow_extension_override backup_config basepeak build_in_build_dir check_md5 command_add_redirect difflines env_vars expand_notes expid fail fail_build fail_run ignore_sigint inherit_from ___ keeptmp line_width locking log_line_width _ log_timestamp mailcompress mailmethod mailport mailserver mailto mail_reports make makeflags mean_anyway minimize_rundirs minimize_builddirs no_monitor no_input_handler notes_wrap_indent output_root preenv section_specifier_fatal sendmail src.alt strict_rundir_verify sysinfo_program teeout version_url

III. Config file options for specmake

CC, CXX, FC
CLD, CXXLD, FLD
ONESTEP
OPTIMIZE, COPTIMIZE, CXXOPTIMIZE, FOPTIMIZE
PORTABILITY, CPORTABILITY, CXXPORTABILITY, FPORTABILITY...
RM_SOURCES

IV. Config file options for the shell

_ bench_post_setup _ ______ post_setup monitor_pre_bench monitor_post_bench monitor_wrapper monitor_specrun_wrapper submit

V. Config file options for the output report

A. Field scoping

B. Continued versus fixed-format fields

C. Free-form notes

D. Links in notes sections

E. Attachments

F. Test description

hw_avail license_num notes_test prepared_by sw_avail system_class system_name system_vendor tester test_sponsor

G. Benchmark description

notes_base notes_comp notes_peak notes_port sw_base_ptrsize sw_c_compiler sw_cxx_compiler sw_f_compiler sw_mpi_library sw_mpi_other sw_notes sw_other sw_peak_ptrsize

H. Component attributes

I. Node description(s)

node_tag_label node_tag_order node_tag_count node_tag_hw_vendor node_tag_hw__model node_tag_purpose node_tag_hw_cpu_name node_tag_hw_ncores node_tag_hw_ncoresorder node_tag_hw_ncoresperchip node_tag_hw_nchips node_tag_hw_nthreadspercore node_tag_hw_cpu_char node_tag_hw_cpu_mhz node_tag_hw_fpu node_tag_hw_pcache node_tag_hw_scache node_tag_hw_tcache node_tag_hw_ocache node_tag_hw_memory node_tag_sw_os node_tag_sw_state node_tag_hw_disk node_tag_sw_local_file node_tag_sw_shared_file node_tag_sw_other node_tag_hw_other node_tag_notes

J. Adaptor description(s)

node_tag1_hw_adaptor_tag2_model node_tag1_hw_adaptor_tag2_slot_type node_tag1_hw_adaptor_tag2_data_rate node_tag1_hw_adaptor_tag2_count node_tag1_hw_adaptor_tag2_ports_used node_tag1_hw_adaptor_tag2_driver node_tag1_hw_adaptor_tag2_firmware

K. Interconnect description(s)

interconnect_tag_hw_purpose interconnect_tag_hw_model interconnect_tag_hw_notes interconnect_tag_hw_switch interconnect_tag_hw_switch_count interconnect_tag_hw_switch_ports interconnect_tag_hw_switch_data_rate interconnect_tag_hw_switch_firmware interconnect_tag_hw_topo interconnect_tag_hw_vendor interconnect_tag_label interconnect_tag_order

VI. Deprecated options

A. SPEC/CPU rate runs

B. Feedack-directed optimization

C. Old-style descriptive formats

D. Obsolete hardware designs

E. Automatic thread-level parallelization

VII. The config file preprocessor

A. Defining macros

B. Un-doing macro definition

C. Using macros

D. Conditionals

1.%ifdef .. %endif

2.%ifndef .. %endif

3.%if .. %endif

4.%else

5.%elif

E. Informational directives

1.%warning

2.%error

VIII. Output files - and how they relate to your config file

A. Automatic backup of config files

B. The log file and verbosity levels

1. Search Strings

2. Temporary Debug Logs

3. Definitions of verbosity levels

C. Help, I've got too many logs

D. Finding the build directory

E. Files in the build directory

F. For more information

IX. About Alternate Sources

A. Example: Applying a src.alt

B. Developing a src.alt (brief introduction)

X. Troubleshooting

I. Introduction

SPEC MPI2007 config files provide very detailed control of testing. Before learning about these details, most users will find it helpful to begin with:

Note: links to SPEC MPI2007 documents on this web page assume that you are reading the page from a directory that also contains the other SPEC MPI2007 documents. If by some chance you are reading this web page from a location where the links do not work, try accessing the referenced documents at one of the following locations:

  • www.spec.org/mpi2007/Docs/
  • The $SPEC/Docs/ (Unix) or %SPEC%\Docs\ (Windows) directory on a system where SPEC MPI2007 has been installed.
  • The Docs/ directory on your SPEC MPI2007 distribution DVD.

The runspec document discusses the primary user interface for running SPEC MPI2007; with this document, attention turns more toward how things work inside.

q. This document looks big and intimidating. Where do I start?

a. Don't start here. Start with runspec.html. But, once you do read this document, please be sure to notice:

  • Section I.B, which explains that a config file contains options that are used by five (5) different consumers.
  • The table of contents, which lists the options accepted by each of those consumers. For example, the shell options are listed here.

If you keep track of which options are addressed to which consumer, you will considerably ease your learning curve.

I.A. What is a config file? (Background: benchmark philosophy.)

A config file contains:

A key decision that must be made by designers of a benchmark suite is whether to allow the benchmark source code to be changed when the suite is used.

If source code changes are allowed:

 +  The benchmark can be adapted to the system under test.
 +  Portability may be easier.
 –  But it may be hard to compare results between systems, unless some formal audit is done to ensure that comparable work is done.

If source code changes are not allowed:

 +  Results may be easier to compare.
 –  It may take more time and effort to develop the benchmark, because portability will have to be built in ahead of time.
 –  Portability may be hard to achieve, at least for real applications. Simple loops of 15 lines can port with little effort, and such benchmarks have their uses. But real applications are more complex.

SPEC has chosen not to allow source code changes for the MPI2007 suite, except under very limited circumstances.

By restricting source code changes, SPEC separates the activity of porting benchmarks, which has a goal of being performance neutral, from the activity of using the benchmarks, where the goal is getting the best score possible. Prior to the first production use of MPI2007, SPEC invested substantial effort to port the suite to as many platforms as practical, including 32 and 64-bit systems; little-endian and big-endian hardware; Unix, Linux, and Microsoft Windows systems.

Are source code changes ever allowed? Normally, no. But if you discover a reason why you believe such a change is essential, SPEC wants to hear about it, and will consider such requests for a future revision of the suite. SPEC will normally not publish MPI2007 results using modified source code, unless such modifications are unavoidable for the target environment, are submitted to the SPEC HPG committee for review, are made available to all users of the suite, and are formally approved by a vote of the SPEC HPG committee.

So, if source code changes are not allowed, but the benchmarks must be compiled in a wide variety of environments, can the users at least write their own makefiles, and use -D options to select different environments? The answer to these two questions are "no", and "yes", respectively:

You do this in the config file, which contains a centralized collection of all the portability options and optimization options for all the benchmarks in the MPI2007 suite. The SPEC tools then automatically generate the makefiles for you.

The config file contains places where you can specify the characteristics of both your compile time and run time environments. A general notation is used to express platform-specific conventions for compiling and running the benchmarks. It allows the advanced user to perform detailed manipulation of makefile options, but retains all the changes in one place so that they can be examined and reproduced.

The config file is one of the key ingredients in making results reproducible. For example, if a customer would like to run the MPI2007 suite on her own SuperHero Model 4 and discover how close results are in her environment to the environment used when the vendor published a MPI2007 result, she should be able to do that using only 3 ingredients:

  1. The SPEC MPI2007 media, ordered from SPEC.
  2. The software and hardware as specified in the full disclosure when the vendor published the original results.
  3. The config file from the vendor's submission.

I.B. What does a config file affect?

A config file contains options targetting five distinct consumers:

  1. The runspec tool, which provides overall control of the test,
  2. The tool that compiles benchmarks: specmake,
  3. Your shell, which is used for certain operations such as submiting to specific processors,
  4. Readers of the results, and
  5. The config file preprocessor.

To understand how to write a config file effectively, you need to understand which consumer you are addressing at any given point.

The above point seems worth emphasizing:

To understand how to write a config file effectively, you need to understand which consumer you are addressing at any given point.

This section gives you an overview of the consumers; more detail is included in later sections.

I.B.1. runspec

Various aspects of the operation of runspec can be affected by setting options within a config file. You'll find a list of these options in the table of contents for section II - including some that are available both on the runspec command line and some that can only be set within a config file.

For example, if michael.cfg includes the lines:

   output_format = text,ps
   tune          = base
   reportable    = 1
   runlist       = medium

then the defaults for the runspec command would change as specified. A user who types either of the following two commands would get precisely the same effect:

   runspec --config=michael
   runspec --config=michael --output=text,ps --tune=base --reportable medium

I.B.2. specmake

The tool specmake is simply GNU make renamed to avoid any possible conflicts with other versions of make that may be on your system. The options commonly used for specmake are listed in the table of contents for section III.

For example, these config file lines:

   CC           = cc
   CPORTABILITY = -DSPEC_SINGLE_UNDERSCORE
   OPTIMIZE     = -O4

are written to the makefile set that is ultimately used to build the benchmark, and are interpreted by specmake.

I.B.3. The shell

Some config file lines define commands that are handed off to the shell or, on Windows, the command interpreter. The list of these is in the table of contents for section IV.

For example, consider a config file that contains:

submit = let "MP_PROCS=$ranks"; poe $command

When using this config file, runspec will pass the above command to the shell to run the corresponding benchmark binary. It is the shell that actually carries out the requested commands, not runspec, so the syntax of the command depends on whether you are using the Unix shell (/bin/sh) or the Windows command interpreter (cmd.exe).

Because runspec can cause arbitrary commands to be executed,
it is therefore important to read a config file you are given before using it.

I.B.4. Readers of the results

If a SPEC MPI2007 result is published (for example, at http://www.spec.org/), it is expected to contain all the information needed for a reader to understand exactly what was tested. Fields that are informative for readers are listed in the table of contents for section V.

For example, config file lines such as these are addressed to the human reader:

   test_date = Nov-2008
   hw_avail  = Apr-2008
   sw_avail  = May-2008
   node_notes_015  = Note: Node 0 is used as the file server for the cluster.

In addition, for results published by SPEC, the config file itself is available to readers at http://www.spec.org/mpi2007/. The config file is presented as you wrote it, with three exceptions (protected comments, the MD5 section, and rawfile corrections for reader fields). The config file is made available because it is so important to reproducing results, as described in the Introduction. The config file is saved on every run, as a compressed portion of the rawfile, and can be accessed with runspec --rawformat --output_format=config <rawfile>.

I.B.5. The config file preprocessor

There is also a config file preprocessor, which is addressed via lines that begin with % in the first column. The config file preprocessor is a new SPEC tool, introduced with CPU2006 and MPI2007.

I.C. Config file structure

A config file contains:

About scope: Every line of the config file is considered to be within the scope of one of the above three. Lines prior to the first section marker are in the scope of the header section. All other lines are either in the scope of the most recently preceding user-defined section marker, or else in the MD5 section.

A line within the scope of a named section may be overridden by a line within the scope of a different named section, according to rules that are described below.

I.C.1. Comments and whitespace

A comment begins with a #, and can be placed anywhere in a config file, either as a trailing end-of-line comment or as a whole line:

#
#   Header comment
#
output_format   = rsf,text,html,csv      # End-of-line comment.
When the config file is saved as an encoded portion of the rawfile, the comments are included. But if a comment begins with #>, it is a "protected comment" that will not be saved in the rawfile. Thus you could use # for most of your comments, and use #> for proprietary information, such as:

   #> I didn't use the C++ beta version because of Bob's big back-end bug.

Blank lines can be placed anywhere in a config file.

Trailing spaces and tabs are stripped, unless they are preceeded by a backslash:

   CC_PATH=/path/with/no/trailing/spaces

That is turned into "/path/with/no/trailing/spaces". To preserve those trailing spaces, you'd simply add a backslash:

      CC_PATH=/path/with/trailing/spaces\ 

That is turned into "/path/with/trailing/spaces ".

Spaces within a line are usually ignored. Of course, you wouldn't want to spell OPTIMIZE as OPT I MIZE, but you are perfectly welcome to do either of the following:

   OPTIMIZE=-O2
   OPTIMIZE   =  -02

One place where spaces are considered significant is in notes, where the tools assume you are trying to line up your comments in the full disclosure reports. (Notes are printed in a fixed-width font.)

Spaces at the beginning of lines are ignored, except when attempting to address the preprocessor. Preprocessor directives always begin with a percent sign (%) in the first column. You can put spaces after the percent sign, if you wish, as shown by the examples below.

I.C.2. Header section

The header section is simply the first section, prior to the first occurence of a named section.

Most attempts to address runspec itself must be done in the header section. For example, if you want to set reportable=1, you must do so before any occurrences of section markers.

I.C.3. Named Sections

A named section is a portion of the config file that begins with a section marker and continues until the next section marker or the MD5 section is reached. The contents of the named section are applied based upon the precedence rules described in the following sections.

A "section marker" is a four-part string of the form:

   benchmark[,benchmark]*=tuning[,tuning]*=extension[,extension]*=machine[,machine]*:

These are referred to below as the 4 "section specifiers". The allowed values for the section specifiers are:

benchmark: default
mpi2007
medium
large
medium_all_cpp(C++ benchmarks in the medium suite)
medium_all_c(C benchmarks in the medium suite)
medium_all_mixed(benchmarks in the medium suite, mixing C and Fortran)
medium_all_fortran(Fortran benchmarks in the medium suite)
medium_all_f9x(benchmarks in the medium suite, using Fortran 90 and beyond)
large_all_cpp(C++ benchmarks in the large suite)
large_all_c(C benchmarks in the large suite)
large_all_mixed(benchmarks in the large suite, mixing C and Fortran)
large_all_fortran(Fortran benchmarks in the large suite)
large_all_f9x(benchmarks in large suite, using Fortran 90 and beyond)
Any individual benchmark, such as 104.milc
tuning: default
all
base
peak
extension: default
An arbitrary string, such as "cloyce-naturalblonde"
machine: default
An arbitrary string [*]

[*] The "machine" specifier works similarly to the extension specifier, but it does not affect the name of the executable produced, which, in many environments, makes it less useful than the extension specifier. This document does not describe the usage of the "machine" specifier, other than to note that it exists; if you feel particularly courageous, you can experiment with it.

But please be aware that if you use a single config file to build with two different machine settings, you will likely overwrite the original binaries on the second build, since the machine specifier does not affect the name of the generated executable.

Lists may be used in any of the slots in the section marker:

       104.milc,122.tachyon = base,peak = optLevel1,optLevel2 = arch1,arch2:

This capability is new with the MPI2007 and CPU2006 tool sets.

Trailing default section specifiers may be omitted from a section marker. Thus all three of these section markers are equivalent:

   104.milc=base=default=default:
   104.milc=base=default:
   104.milc=base:

Section markers can be entered in any order. Section markers can be repeated; material from identical section markers will automatically be consolidated. That is, you are welcome to start one section, start a different one, then go back and add more material to the first section. But please note that since there is no marker for the header section, you cannot go back to it.

By constructing section markers, you specify how you would like your options applied, with powerful defaulting and overriding capabilities. The next several sections walk through examples to demonstrate precedence, including how sections interact with each other.

I.C.3.a. Precedence of the benchmark specifier

For the benchmark specifier, the precedence is:

  highest   named benchmark
            suite or bset name
  lowest    default

Using default as the benchmark specifier

For example, consider this config file that only mentions default for the benchmark specifier:

   $ cat tmp.cfg
   runlist        = lammps
   size           = mtest
   iterations     = 1
   tune           = base
   output_format  = text
   teeout         = 1
   ranks          = 32

   default=default=default=default:
   CXX            = mpCC_r
   OPTIMIZE       = -O3
   EXTRA_LDFLAGS  = -q64
   EXTRA_CXXFLAGS = -q64
   submit         = poe $command -procs $ranks

   $ runspec --config=tmp | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O3 -q64 lammps.cpp
   $ 

The config file above is designed for quick, simple testing: it runs only one benchmark, namely 126.lammps, using the smallest (mtest) workload, runs it only once, uses only base tuning, outputs only the text-format (ASCII) report, and displays the build commands to the screen (teeout). To use it, we issue a runspec command, and pipe the output to grep to search for the actual generated compile command. (Alternatively, on Windows, we could use findstr on the generated log file).

The careful reader may ask, "Why does the runlist reference lammps rather than 126.lammps?" The answer is that the runlist can use benchmark numbers or any abbreviation that is sufficient for uniqueness; any of the following would have the same effect: 126.lammps, 126, or lammps. This is the same rule as for the corresponding option on the command line.

The results show that the tuning applied was the expected -O3 (which mean optimization level 3, for the IBM mpCC compiler). The tools have automatically added -c -o lammps.o to specify where the object file is to be written. The switch -DFFT_NONE was also added automatically.

Using a bset list as the benchmark specifier

The next example differs from the previous one by adding a section marker with medium_all_cpp, for the C++ benchmarks, as the first section specifier:

   $  cat tmp.cfg
   runlist       = lammps
   size          = mtest
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1
   ranks         = 32

   default=default=default=default:
   CXX            = mpCC_r
   OPTIMIZE       = -O3
   EXTRA_LDFLAGS  = -q64
   EXTRA_CXXFLAGS = -q64
   submit         = poe $command -procs $ranks

   medium_all_cpp=default=default=default:
   OPTIMIZE = -O4

   $ runspec --config=tmp | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O4 -q64 lammps.cpp
   $

The second OPTIMIZE line is used above because the reference to the C++ benchmarks is considered to be more specific than the overall default.

Using a named suite as the benchmark specifier

A named suite is one case of a bset, with the property that it contains all the benchmarks necessary for a reportable run. The example below uses the medium section marker as the first section specifier, to denote the entire medium sized data suite:

   $  cat tmp.cfg
   runlist       = lammps
   size          = mtest
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1
   ranks         = 32

   default=default=default=default:
   CXX            = mpCC_r
   OPTIMIZE       = -O3
   EXTRA_LDFLAGS  = -q64
   EXTRA_CXXFLAGS = -q64
   submit         = poe $command -procs $ranks

   medium=default=default=default:
   OPTIMIZE = -O4

   $ runspec --config=tmp | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O4 -q64 lammps.cpp
   $

The second OPTIMIZE line is used above because the reference to the medium suite is considered to be more specific than the overall default.

Using a named benchmark as the benchmark specifier

Furthermore, we can add a specifier that mentions lammps by name:

   $  cat tmp.cfg
   runlist       = lammps
   size          = mtest
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1
   ranks         = 32

   default=default=default=default:
   CXX            = mpCC_r
   OPTIMIZE       = -O3
   EXTRA_LDFLAGS  = -q64
   EXTRA_CXXFLAGS = -q64
   submit         = poe $command -procs $ranks

   medium_all_cpp=default=default=default:
   OPTIMIZE = -O4

   126.lammps=default=default=default:
   OPTIMIZE = -O5 -qstrict

   $ runspec --config=tmp | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O5 -qstrict -q64 lammps.cpp
   $

The third OPTIMIZE line wins above, because it is included in the section that is considered to be the most specific.

Order of differing sections does not matter:

But what if we had said these in a different order?

   $ cat tmp.cfg
   runlist       = lammps
   size          = mtest
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1
   ranks         = 32

   126.lammps=default=default=default:
   OPTIMIZE = -O5 -qstrict

   default=default=default=default:
   CXX            = mpCC_r
   OPTIMIZE       = -O3
   EXTRA_LDFLAGS  = -q64
   EXTRA_CXXFLAGS = -q64
   submit         = poe $command -procs $ranks

   medium_all_cpp=default=default=default:
   OPTIMIZE = -O4

   $ runspec --config=tmp | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O5 -qstrict -q64 lammps.cpp

Notice above that the order of entry is not significant; it's the order of precedence from least specific to most specific.

Order of the same section does matter:

When a specifier is listed more than once at the same descriptive level, the last instance of the specifier is used. Consider this case:

   126.lammps=default=default=default:
   OPTIMIZE = -O4

   122.tachyon=default:
   OPTIMIZE = -O5

   126.lammps=default=default=default:
   OPTIMIZE = -O3

The ending value of OPTIMIZE for 126.lammps is -O3, not -O4.

I.C.3.b. Precedence of the tuning specifier

For the tuning specifier, either base or peak has higher precedence than default.

Here is an example of its use:

   $ cat tmp.cfg
   runlist       = lammps
   size          = mtest
   iterations    = 1
   tune          = base,peak
   output_format = text
   teeout        = 1
   ranks         = 32

   default=default=default=default:
   CXX            = mpCC_r
   OPTIMIZE       = -O3
   EXTRA_LDFLAGS  = -q64
   EXTRA_CXXFLAGS = -q64
   submit         = poe $command -procs $ranks

   default=base=default=default:
   OPTIMIZE       = -O4

   default=peak=default=default:
   OPTIMIZE       = -O5 -qstrict

   $ runspec --config=tmp -T base | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O4 -q64 lammps.cpp
   $ runspec --config=tmp -T peak | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O5 -qstrict -q64 lammps.cpp
   $ 

In the above example, we compile 126.lammps twice: once for base tuning, and once for peak. Notice that in both cases the optimizations defined by the more specific section markers have been applied, namely -O4 and -O5 -qstrict, rather than -O3 from default=default=default=default.

I.C.3.c. Precedence of the extension specifier

For the extension specifier, any named extension is at a higher precedence level than the default.

Using an extension found in the config file

Analogous to the example above, the choice of extension specifier determines the level of optimization that is applied:

   $ cat tmp.cfg
   runlist       = lammps
   size          = mtest
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1
   ranks         = 32

   default=default=default=default:
   CXX            = mpCC_r
   OPTIMIZE       = -O3
   EXTRA_LDFLAGS  = -q64
   EXTRA_CXXFLAGS = -q64
   submit         = poe $command -procs $ranks

   default=base=myke=default:
   OPTIMIZE       = -O4

   default=base=yusuf=default:
   OPTIMIZE       = -O5 -qstrict

   $ runspec --config=tmp --extension=myke | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O4 -q64 lammps.cpp
   $
   $ runspec --config=tmp --extension=yusuf | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O5 -qstrict -q64 lammps.cpp
   $ 
   $ cd $SPEC/benchspec/MPI2007/126.lammps/exe
   $ ls -lt | head -3
   total 9344
   -rwxrwxr-x   1 spec     staff       2330870 Jun 13 09:50 lammps_base.yusuf
   -rwxrwxr-x   1 spec     staff       2446066 Jun 13 09:34 lammps_base.myke
   $

Notice above that two different versions of 126.lammps were built from the same config file, and the default OPTIMIZE setting was overridden the extension rule in each case. Both executables are present in the exe directory for 126.lammps.

Using an extension that is not found in the config file

The previous section demonstrated use of the runspec switch --extension to select among extensions defined in the config file. But what if the extension on the command line is not mentioned in the config file? The example below continues immediately from the example just above:

   $ runspec --config=tmp --extension=yusoff
   runspec v4200 - Copyright 1999-2007 Standard Performance Evaluation Corporation
   ...
   ERROR: The extension 'yusoff' defines no settings in the config file!
          If this is okay and you'd like to use the extension to just change
          the extension applied to executables, please put
       allow_extension_override = yes
          into the header section of your config file.

By default, if you mention an extension on the runspec command line that does not exist in the config file, the tools refuse to build.

Extension override

But if you add allow_extension_override=yes to the top of the config file, then the tools will build or run with the extension you specified, using the same settings as they would have used if no extension had been entered on the runspec command line.

The next example continues with a config file that adds allow_extension_override to the previous example config file:

   $ diff tmp.cfg tmp2.cfg
   0a1
   > allow_extension_override=yes
   $
   $ runspec --config=tmp2 --extension=yusoff | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O3 -q64 lammps.cpp
   $
   $ cd $SPEC/benchspec/MPI2007/126.lammps/exe
   $ ls -lt | head -2
   total 12472
   -rwxrwxr-x   1 spec     staff       1599589 Jun 13 10:02 lammps_base.yusoff

Notice above that although the tools now consent to build with the requested extension, the library settings now falls back to the default, since yusoff does not match any section markers.

(For CPU2000/OMP2001/HPC2002, the tools silently accepted any extension, including typos, which sometimes led to surprising results. MPI2007 tries to reduce surprises by rejecting your typos, unless you set allow_extension_override.)

I.C.3.d. Combining specifier types

If more than one section applies to a particular benchmark without disagreement among them, then all are applied.

Consider this example:

   $ cat tmp.cfg
   runlist       = lammps
   size          = mtest
   iterations    = 1
   tune          = base
   output_format = text
   teeout        = 1
   ranks         = 32

   default=default=default=default:
   CXX            = mpCC_r
   OPTIMIZE       = -O3

   126.lammps=default=default=default:
   submit         = poe $command -procs $ranks

   default=peak=default=default:
   OPTIMIZE       = -O5 -qstrict

   default=default=AIX64=default:
   EXTRA_LDFLAGS  = -q64
   EXTRA_CXXFLAGS = -q64

   $ runspec --config=tmp --tune=peak --ext=AIX64 | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O5 -qstrict -q64 lammps.cpp
   $ 

Notice above that all three sections applied: the section specifier for 126.lammps, the specifier for peak tuning, and the specifier for extension AIX64.

I.C.3.e. Precedence among section types

If sections conflict with each other, the order of precedence is:

     highest     benchmark
                 suite or bset
                 tuning
                 extension
     lowest      machine

And this order can be demonstrated as follows:

    $  cat tmp.cfg
     runlist       = lammps
     size          = mtest
     iterations    = 1
     tune          = base
     output_format = text
     teeout        = 1
     makeflags     = -j30
     ranks         = 32
  
     default=default=default=default:
     CXX            = mpCC_r
     CC             = mpcc_r
     FC             = mpxlf95_r
     OPTIMIZE       = -O3
     EXTRA_LDFLAGS  = -q64
     EXTRA_CXXFLAGS = -q64
     EXTRA_CFLAGS   = -q64
     submit         = poe $command -procs $ranks
  
     medium_all_c=default=default=default:
     OPTIMIZE       = -O4
  
     126.lammps=default=default=default:
     OPTIMIZE       = -O5 -qstrict
  
     default=peak=default=default:
     OPTIMIZE       = -O5
  
     default=default=AIX64=default:
     OPTIMIZE       = -O
     $
     $ runspec --config=tmp lammps | grep lammps.cpp  
[1]  mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O5 -qstrict -q64 lammps.cpp
     $ runspec --config=tmp --tune=peak lammps | grep lammps.cpp
[2]  mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O5 -qstrict -q64 lammps.cpp
     $ runspec --config=tmp --extension=AIX64 lammps | grep lammps.cpp
[3]  mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O5 -qstrict -q64 lammps.cpp
     $ 
     $ runspec --config=tmp --tune=base tachyon | grep main.c
[4]  mpcc_r -c -o main.o -DSPEC_MPI -DNDEBUG -O4 -q64 main.c
     $ runspec --config=tmp --tune=peak tachyon | grep main.c
[5]  mpcc_r -c -o main.o -DSPEC_MPI -DNDEBUG -O4 -q64 main.c
     $ runspec --config=tmp --extension=AIX64 tachyon | grep main.c
[6]  mpcc_r -c -o main.o -DSPEC_MPI -DNDEBUG -O4 -q64 main.c
     $
     $ runspec --config=tmp --tune=base GemsFDTD | grep GemsFDTD_MPI.fppized.o
[7]  mpxlf95_r -c -o GemsFDTD_MPI.fppized.o -O3 -q64 GemsFDTD_MPI.fppized.f90
     $ runspec --config=tmp --tune=peak GemsFDTD | grep GemsFDTD_MPI.fppized.o
[8]  mpxlf95_r -c -o GemsFDTD_MPI.fppized.o -O5 -q64 GemsFDTD_MPI.fppized.f90
     $ runspec --config=tmp --extension=AIX64 GemsFDTD | grep GemsFDTD_MPI.fppized.o
[9]  mpxlf95_r -c -o GemsFDTD_MPI.fppized.o -O -q64 GemsFDTD_MPI.fppized.f90
     $ runspec --config=tmp --tune=peak --extension=AIX64 GemsFDTD | grep GemsFDTD_MPI.fppized.o
[10] mpxlf95_r -c -o GemsFDTD_MPI.fppized.o -O5 -q64 GemsFDTD_MPI.fppized.f90
     $ 

Notice above that the named benchmark always wins: lines [1], [2], and [3]. If there is no section specifier that names a benchmark, but there is a section specifier that names a suite or bset, then the suite or bset wins: lines [4], [5], and [6]. If there are no applicable benchmark or suite specifiers, then tuning or extension can be applied: lines [8] and [9]. But if both tuning and extension are applied, tuning wins [10].

I.C.4. MD5 Section

The final section of your config file is generated automatically by the tools when the benchmarks are compiled, and looks something like this:

   __MD5__
   126.lammps=base=none=default:
   # Last updated Thu May 11 08:53:12 2007
   optmd5=a0104c0975ee3b341e73437843df57a9
   baggage=
   compile_options=\
   @eNpz9vcNsFJITlbQBaJ8hfyCksz8vGK9fAVdl+AAV+d454BQINPPxdUp1F1BQUG3wt9YQbdcAR3Y\
   FOeXFiWn2nE5Wyk4O9sqJScrgZn+Tl7+ASG2SiimK3EBrQQa7Obj6B4MlMO0SYnL30oBqNHT1zPK\
   FagCYqsSl4+nnzfYtVgcYpOflJWaXFJshySGsBPkGh8XiMOwmw1W4B8KciySSwGYpEmr
   exemd5=9b42a21f12b1fbb4e87cd9bacd781c43

The "MD5" is a checksum that ensures that the binaries referenced in the config file are in fact built using the options described therein. For example, if you edit the config file to change the optimization level for 126.lammps, the next time the file is used for lammps, the tools will notice the change and will recompile it.

You can optionally disable this behavior, but doing so is strongly discouraged. Not only is the result not reportable, but the binaries you run may not be compiled under the conditions you think they are compiled under. See the acerbic remarks in the description of check_md5, below.

If you would like to see what portions of your config file are used in computing the MD5 hash, runspec with --debug=30 or higher, and examine the log file.

For published results, the published config file (from rawformat --output_format=config) does not include the MD5 section.

I.C.5. Shell-like "here documents" and continued lines

Shell-style "here documents" are supported for setting variables to multi-line values. Continued lines (with \) are also supported:

   $ cat tmp2.cfg
   expand_notes  = 1
   size          = mtest
   runlist       = lammps
   iterations    = 1
   output_format = text

   foo =<<EOT

   This +
   is a +
   test +
   EOT

   bar = \
         and +\
         so  +\
         is  +\
         this+

   notes01 = $foo
   notes02 = $bar

   $ runspec --config=tmp2 | grep txt
           format: ASCII -> /ailuropoda/raj/spec/result/MPIM2007.159.mtest.txt
   $ grep + ../result/*159*txt
       This +
       is a +
       test +
             and +
             so  +
             is  +
             this+
   $

Note: although continued lines are supported, they are rarely used. The more common method of continuation is by appending a number to a field, as described in the section "Field scoping and continuation".

I.C.6. Included files

It is possible to include another file in your config file. A typical use for this feature might be to keep all the software information in the main config file, but to include the hardware information about the current System Under Test (SUT) in another file. For example:

   $ cat tmp.cfg
   output_format = text
   iterations    = 1
   size          = mtest
   sw_compiler   = myC V1.0
   sw_avail      = Mar-2007
   runlist       = lammps
   include: SUT.inc

   default=base:
   OPTIMIZE = -O
   $ cat SUT.inc
   hw_model = SuperHero IV
   hw_avail = Feb-2008

   $ runspec --config tmp | grep txt
           format: ASCII -> /manas/spec/result/MPI2007.160.mtest.txt
   $ grep avail ../result/*160.mtest.txt
    [...]       Hardware availability: Feb-2008
    [...]       Software availability: Mar-2007
   $ 

Notice above that the report mentions both the hardware and software dates.

I.D. Variable Substitution

You can do variable substitution using your config file. But, as described in the Introduction, the contents of a config file are directed to various consumers. Therefore, effective use of variable substitution requires you to be aware of which software is doing the substitution. Differing syntax is used for each.

%{macro_name}

The config file preprocessor does macro substitution. Macros to be substituted always use a percent sign and curly brackets: %{macro_name} See the preprocessor section for more information.

$[variable]

A dollar sign and square brackets cause runspec to do substitution immediately after reading the config file. This is useful for variables that don't change during a run, for example $[top] to refer to the directory where SPEC MPI2007 has been installed.

$variable
${variable}

A dollar sign and the name of a variable cause Perl variable interpolation to be used, for example $hw_avail. Curly brackets can be used to help the parser avoid confusion with contiguous other content: ${hw_avail}ish.

\$VARIABLE

If runspec generates an external command which is handed off to your shell (or command interpreter), the variable will be interpreted according to the rules of your shell. Such a variable might look like this: \$SPEC.

$(VARIABLE)

If the variable is handed off to specmake, it is interpreted by the usual rules of GNU make. Such variables might look like this: $(FOPTIMIZE).

q. Wait a minute... all these choices for substitution? Which one do I want?

a. You probably want the preprocessor.

Have a look at the example at the top of Section VII. If that looks like what you want, you're all set; otherwise, you'll have to think through which consumer you are addressing (runspec, specmake, or the shell) and pick your syntax accordingly.

I.D.1. By runspec, at startup: $[variable]

Substitution for variables of the form $[variable] happens immediately after the config file is read by runspec. Any value that's set in the config file and is visible in the scope where the variable is used can be substituted. Because of the named section scoping restriction, if you want to use variable substitution to note your optimization flags, the notes for the individual benchmarks must be in those benchmarks' sections:

122.tachyon=peak=default=default:
PEAKFLAG=-gofast
# The following will turn into what you expect
notes_peak_400_1=I use $[PEAKFLAG]

122.tachyon=base=default=default:
BASEFLAG=-besafe
# The following will turn into what you expect:
notes_base_400_1=I use $[BASEFLAG]
# The following will NOT WORK:
notes_base_400_2=My brother likes $[PEAKFLAG]

You can't substitute variables that don't exist or whose values aren't known when the config file is read.

Wrong:

ext = foo

default:
OPTIMIZE = -xO2
notes01 = my ext is $[ext]

default=default=bar:
OPTIMIZE = -xO1
notes02 = my ext is $[ext]

This doesn't work because the sorting of which extensions to use doesn't happen until after the config file is processed. In this particular example, it's obvious (to you) what the value should be, but the tools aren't as clever as you are.

Perhaps the most useful variable is the one for the top of the SPEC MPI2007 tree, $[top], often found in contexts such as:

    flagsurl = $[top]/myflagsdir/myflagsfile.xml

Variables of possible interest might include:

configpath The location of your config file
dirprot protection that is applied to directories created by runspec
endian 4321 for big endian, 1234 for little
flag_url_base directory where flags files are looked up
OS unix or windows
os_exe_ext exe for windows, nil elsewhere
realuser the user name according to the OS
top the top directory of your installed SPEC MPI2007 tree
username the username for purposes of tagging run directories
uid the numeric user id

You can substitute with most options that you enter into the config file, including: action, allow_extension_override, backup_config, basepeak, check_md5, check_version, command_add_redirect, config, delay, deletework, difflines, env_vars, expand_notes, expid, ext, fake, flagsurl, http_proxy, http_timeout, ignore_errors, ignore_sigint, info_wrap_columns, iterations, line_width, locking, log_line_width, mach, mail_reports, mailcompress, mailmethod, mailport, mailserver, mailto, make, make_no_clobber, makeflags, mean_anyway, minimize_builddirs, minimize_rundirs, no_input_handler, no_monitor, notes_wrap_columns, notes_wrap_indent, output_format, output_root, rawformat, rebuild, reportable, runlist, section_specifier_fatal, sendmail, setprocgroup, size, strict_rundir_verify, sysinfo_program, table, teeout, tune, username, verbose, version_url.

You can also print out the value of additional variables that you may have created.

Here is a sample config file that illustrates square bracket variable substitution:

$ cat x.cfg
expand_notes = 1
runlist      = medium
action       = validate
myfriend     = jamiemeow
output_root  = /tmp
flagsurl     = $[top]/Docs/flags/flags-advanced.xml

notes01      = Today, I am running $[runlist] in $[top] on a $[OS] system 
notes02      = Today the flags file is $[flagsurl]
notes03      = Today, my favorite friend is $[myfriend]

$ runspec --config=x  --fakereportable | grep txt
        format: ASCII -> /tmp/result/MPI2007.002.txt
$ grep Today /tmp/result/MPI2007.002.txt
     Today, I am running medium in /spec/mpi2007 on a unix system 
     Today the flags file is /spec/mpi2007/Docs/flags/flags-advanced.xml
     Today, my favorite friend is jamiemeow

You can't use square brackets to substitute variables whose value changes during a run or build.

Wrong:

default=default=default=default:
#        I executed ''
notes0 = I executed '$[command]'

What did you expect?

Don't worry -- you'll receive a warning if you use a variable that the tools don't know about. It's up to you to heed them.

I.D.2 By runspec, during a run: $variable, ${variable}

The second round performed by runspec uses Perl variable interpolation. Only Perl scalars (denoted by a leading $) can be interpolated. For example, notes001 below uses the log file number (generated by the tools) and the hardware availability date (which was set directly):

$ cat tmp.cfg
runlist       = lu
tune          = base
size          = mtest

iterations    = 1
output_format = text

expand_notes  = 1
hw_avail      = May-2007
notes001      = This run is from log.$lognum with hw_avail $hw_avail

$ runspec -c tmp | grep txt
        format: ASCII -> /spec/david/result/MPIM2007.029.mtest.txt
$ grep with ../result/*029*txt
     This run is from log.029 with hw_avail May-2007
$ 

In this case, $hw_avail could also have been substituted in the first round by writing it as $[hw_avail]. In general, for variable interpolation, the earlier the better.

To put text immediately after a variable, you need to make it possible for the parser to see the variable that you want, by using braces:

   % tail -2 tmp.cfg
   notes001 =You have done ${lognum}x runs tonight, go to bed.

   % runspec -c tmp | grep txt
      format: ASCII -> /john/result/MPIM2007.103.mtest.txt
   % grep done /john/result/MPIM2007.103.mtest.txt
      You have done 103x runs tonight, go to bed.

Interpolation won't always do what you wish it might do: for example, some variables are only defined at certain times, and your submit or notes line might be interpolated at a different time. When debugging a config file that uses variable interpolation, you will probably find --size mtest useful.

Some things that you might choose to interpolate include:

baseexe The first part of the executable name, which is <baseexe>_<tune>.<ext>. For example, in "lammps_base.foo", baseexe is "lammps".
benchmark The number and name of the benchmark currently being run, for example 126.lammps
benchname The name of the benchmark currently being run, for example lammps
benchnum The number of the benchmark currently being run, for example 126
benchtop The top directory for the benchmark currently being run, for example /spec/mpi2007/benchspec/MPI2007/126.lammps
command The current command, for example ../run_peak_test_foo.0001/lammps_peak.foo dryer.jpg 2 > dryer.jpg.out 2>> dryer.jpg.err
commandexe The executable for the current command, for example ../run_peak_test_foo.0001/lammps_peak.foo
ext The extension for the benchmark being run
iter The current iteration number
logname The complete log file name, for example /spec/mpi2007/result/MPI2007.168.log
lognum The log file number, for example 168
ranks The number of ranks to be used to run each benchmark. The value is required to be set, either in the config file or on the runspec command line, so the reports will be formatted correctly. Use $ranks to pass the count to the MPI driver, as shown in the example below.
tune The tuning for the benchmark being run (base or peak)
workload The current workload number (within the iteration)

Variable interpolation is most commonly used to paramaterize the MPI driver when a benchmark is invoked by the submit directive. In the two examples below, $command is replaced with the invocation of each benchmark, including the name of the executable and its parameters, and $ranks is replaced by the number of ranks that is to be used with that benchmark for the base or peak run:

submit = poe $command -procs=$ranks

or

env_vars     = 1
ENV_MP_PROCS = $ranks
submit = poe $command

If you'd like a complete list of the variables that you can use in your commands (relative to the config file you're using), set runspec's verbosity to 35 or higher (-v 35) and do a run that causes a command substitution to happen, with expand_notes=1.

I.D.3. By the shell: \$VARIABLE

Substitution by the shell - or by the windows command interpreter - uses backslash dollar sign. Because Perl variables look a lot like shell variables, you need to specially protect shell variables if you want to prevent Perl from trying to interpret them. Notice what happens with the protected and unprotected versions:

$ cat tmp.cfg
runlist              = mcf
size                 = mtest
tune                 = base,peak
iterations           = 1

output_format        = text 
teeout               = 1
expand_notes         = 1

default=peak=default=default:
submit = echo "home=$HOME; spec=$SPEC;" > /tmp/chan; $command

default=base=default=default:
submit = echo "home=\$HOME; spec=\$SPEC;" > /tmp/nui; $command
$ runspec --config=tmp > /dev/null
$ cat /tmp/chan
home=; spec=;
$ cat /tmp/nui
home=/home/chris; spec=/spec/mpi2007;
$ 

In the first submit command, $HOME and $SPEC were gobbled up by runspec. But since those are not the names of variables that can be interpolated, empty strings were the result. In the second command, the backslashes prevented runspec from interpreting the variables, so they were seen by the shell instead.

I.D.4. By specmake: $(VARIABLE)

Variables with a dollar sign and parentheses, aka "round brackets", are substituted by specmake. For example:

    COMPILER_DIR=/usr/local/bin/
    CC=$(COMPILER_DIR)cc

For a more extensive example of variable substitution handled by specmake, see the SPEC CPU2000 example at www.spec.org/cpu2000/docs/example-advanced.cfg. Search that file for LIBS, and note the long comment which provides a walk-through of a complex substitution handled by specmake. Note: a MPI2007 version of that example will be provided at a later date; but the concepts, from the CPU2000 example, are expected to work in a similar fashion for MPI2007.

Deprecated feature alert: Although it is also possible to pass information to specmake using curly brackets: ${SPECMAKE}, this is not recommended. Instead, you should consistently use curly brackets to address runspec and round brackets to address specmake. It is possible that a future version of runspec may insist on interpolating curly brackets itself, rather than allowing specmake to do so.

I.D.5. Example: controlling rank counts in the AIX MPI driver

Here is a simple but useful example of controlling the rank counts in the MPI benchmarks. First, the benchmarks are invoked using the form

	submit = poe $command

The poe is the MPI driver for the AIX operating system. The command is the variable containing the invocation of the benchmark executable with its parameters, and is assigned by runspec in the Perl variable interpolation phase. Given that the ranks count has been assigned on the command-line:

	runspec --ranks 32 ...

or at the top of the config-file:

	ranks = 32

here are several ways that the ranks can be specified to the poe driver:

[1]      submit = poe $command -procs $ranks           # Ranks are set by an explicit parameter to "poe".

[2]      env_vars = 1                                  # "poe" reads the ranks from the environment variable "MP_PROCS".
         ENV_MP_PROCS=$ranks
         submit = poe $command

[3]      submit = let "MP_PROCS=$ranks"; poe $command  # Here the environment variable is set in the shell-execution stage.

[4]      submit = let "COUNT=$ranks"; poe $command -procs \$COUNT   # Here assigned to an intermediate shell variable.

The first three cases rely strictly on variable interpolation. In case [4], the $ranks and $command are expanded in the variable-interpolation stage, while the $COUNT is not expanded because of its preceding '\'. The string that is passed to the shell has the form

	     COUNT=32; poe ... -procs $COUNT

where ... is the expansion of the $command parameter.

I.D.6. Limitations on variable substitution

Once runspec hands control over to specmake or to the shell, the results of further substitution are invisible to runspec. For this reason, you can't say:

Wrong:

MYDIR         = /usr/gretchen/compilers
FC            = $(MYDIR)/f90
notes_comp001 = compiler: $(FC)

However, there are a couple of ways to get around this restriction. The best way for global settings is to use the preprocessor:

%define MYDIR /usr/gretchen/compilers
FC            = %{MYDIR}/f90
notes_comp001 = compiler: %{MYDIR}/f90

That leaves a little to be desired, though, doesn't it? If your Fortran compiler is changed to 'f2001', you still need to remember to change it in two places. You could of course define a whole macro for this:

%define MYFC /usr/gretchen/compilers/f90
FC            = %{MYFC}
notes_comp001 = compiler: %{MYFC}

But what if you have a config file where FC might be set in multiple places, and you really want to know how it was set right here? In that case, use a combination of the preprocessor and variable substitution:

$ cat right_here.cfg 
%define MYDIR /usr/gretchen/compilers

default=base:
FC            = %{MYDIR}/f2001
notes_comp001  = For base, we used this Fortran: $[FC] 

127.wrf2=peak:
FC            = %{MYDIR}/f77
notes_comp002 = For 127.wrf2 peak, we used this Fortran: $[FC]

$ runspec -c right_here.cfg --fakereportable fp | grep txt       
        format: ASCII -> /usr/gretchen/spec/result/MPI2007.179.txt
$ tail -15 ../result/*179.txt

                          Compiler Invocation Notes
                          -------------------------
     For base, we used this Fortran: /usr/gretchen/compilers/f2001 
     For 127.wrf2 peak, we used this Fortran: /usr/gretchen/compilers/f77

    SPEC and SPECfp are registered trademarks of the Standard Performance
    Evaluation Corporation.  All other brand and product names appearing
    in this result are trademarks or registered trademarks of their
    respective holders.
-----------------------------------------------------------------------------
For questions about this result, please contact the tester.
For other inquiries, please contact webmaster@spec.org.
Copyright 2007 Standard Performance Evaluation Corporation
Generated on Wed Aug  2 09:39:53 2007 by SPEC MPI2007 ASCII formatter v4626
$ 

I.D.7. Unsetting a variable with %undef%

It is sometimes useful to be able to undo the setting of a variable that is defined in a lower-precedence section. This is easily accomplished using the special value %undef%:

  $ cat gnana.cfg
   teeout = yes
   action = build
   runlist = lammps

   default=default:
   OPTIMIZE  = -O3
   COPTIMIZE = -O5

   126.lammps=peak:
   COPTIMIZE = %undef%

   $ runspec --config=gnana --tune=base | grep lammps.cpp
   c++ -c -o lammps.o -DFFT_NONE -O3 -O5 lammps.cpp
   $ runspec --config=gnana --tune=peak | grep lammps.cpp
   c++ -c -o lammps.o -DFFT_NONE -O3 lammps.cpp
   $ go lammps
   /spec/gnana/kit91/benchspec/MPI2007/126.lammps
   $ cd run/build_peak_none.0000/
   $ grep OPT Make*
   Makefile.spec:COPTIMIZE        = 
   Makefile.spec:OPTIMIZE         = -O3

As you can see in the peak compilation, the -fast flag was not present because the setting for COPTIMIZE had been deleted.

II. Config file options for runspec

This section documents options that control the operation of runspec itself.

II.A. Options that can also be used on the runspec command line

The following items can be specified in a config file, and have the same meaning as if they are specified on the runspec command line. Please see runspec.html for details.

In general, if options can be specified on both the runspec command line and in a config file, the command line will win vs. items specified in the header section, but will not win over items specified in named sections. Effectively, the order of precedence is:

   named sections (highest)
   command line
   header section (lowest)

In the table that follows, the "Use In" column indicates where the option can be used:

H use only in header section
N use in a named section.
H,N can be used in both the header section and in named sections. The item can therefore be applied on a global basis, and/or can be applied to individual benchmarks.

SPEC MPI2007 V1.1 note: if you select basepeak=1 for a benchmark, the number of ranks in peak will be forced to be the same as in base. In SPEC MPI2007 V1.0, selection of the number of ranks could differ, but this was deemed to be an error. If you want to run the same tuning in both base and peak, while changing the number of ranks, you will need to build two binaries with the same compiler switches.

Option Use In Default Meaning
action H validate What to do.
check_version H 0 (1 for
reportable
runs)

When set, before doing a reportable run, runspec will download a small file (~15 bytes) from www.spec.org containing the current version of the suite and the date it was released, and check your copy vs. that file. In this way, you can be notified if the version of the suite that you're using is out-of-date.

Setting this variable to 0 will disable this check.

If you'd like to check a local file instead, you can modify version_url to point to your internal copy.

If you would like to check your version for a NON-reportable run, you will need to add --check_version to your command line. Setting check_version=1 in the config file only causes the check for reportable runs. The check_version is a new SPEC feature with CPU2006 and MPI2007.

delay H,N 0 Insert a delay of the specified number of seconds before and after benchmark execution. This delay does not contribute to the measured runtime of the benchmark. This delay is also not available in a reportable run. The delay is a new SPEC feature with CPU2006 and MPI2007.)
deletework H,N 0 If set to 1, always delete existing benchmark working directories. An extra-careful person might want to set this to ensure no unwanted leftovers from previous benchmark runs, but the tools are already trying to enforce that property.
ext H none Extension for executables created. This may not be set to any value that contains characters other than alphanumerics, underscores, hyphens, or periods.
flagsurl H none If set, retrieve the named URL or filename and use that as the "user" flags file. If the special value "noflags" is used, runspec will not use any file and (if formatting previously run results) will remove any stored file. Automated prcoessing of flags is new with MPI2007, and is explained in flag-description.html.
http_proxy H

In some cases, such as when doing version checks and loading flag description files, runspec will use HTTP or FTP to fetch a file. If you need to specify the URL of a proxy server, this is the variable to use. By default, no proxy is used. Note that this setting will override the value of the http_proxy environment variable. For example, one might set:

   http_proxy     = http://webcache.tom.spokewrenchdad.com:8080 

Note: if an FTP proxy is needed, it must be set in the ftp_proxy environment variable; there is no corresponding config file setting.

Config files as posted at www.spec.org/mpi2007 will not include whatever you put on this line. Support for http proxies is new with CPU2006 and MPI2007.

http_timeout H 30 This is the amount of time (in seconds) to wait while attempting to fetch a file via HTTP or FTP. If the connection cannot be established in this amount of time, the attempt will be aborted. Support for http timeout is new with CPU2006 and MPI2007.
ignore_errors H 0 Ignore certain errors which would otherwise cause the run to stop. Very useful when debugging a new compiler and new set of options: with this option set, you'll find out about all the benchmarks that have problems, instead of only finding out about the first one.
info_wrap_columns H 50 When set to a value greater than 0, attempts to split non-notes informational lines such that they are no longer than info_wrap_columns columns wide. Lines are split on whitespace, and newly created lines are guaranteed to have at least the same indentation as the original line. If a line contains an item that is longer than info_wrap_columns, a warning is logged and the original line is left unchanged. Automatic line wrapping is a new SPEC feature with CPU2006 and MPI2007.
iterations H 3 Number of iterations to run.
mach H default   Default machine ID. This may not be set to any value that contains characters other than alphanumerics, underscores, hyphens, or periods.
make_no_clobber H,N 0 Don't delete directories when building executables. This option should only be used for troubleshooting a problematic compile. The tools will not allow you to use this option when building binaries for a reportable result. Note that you could issue multiple successive runspec commands with this option set (either in the config file, or with the --make_no_clobber switch), and the build directories will be preserved. But once you remove make_no_clobber (allowing it to default back to 0), then the tools will attempt a normal build with a fresh build directory.
nobuild H 0 Do not attempt to build benchmarks. Useful to prevent attempts to rebuild benchmarks that cannot be built. Also comes in handy when testing whether proposed config file options would potentially force an automatic rebuild. The --nobuild is a new SPEC feature with CPU2006 and MPI2007.
notes_wrap_columns H 0 When set to a value greater than 0, attempts to split notes lines such that they are no longer than notes_wrap_columns columns wide. Lines are split on whitespace, and newly created lines are guaranteed to have at least the same indentation as the original line. If a line contains an item that is longer than notes_wrap_columns, a warning is logged and the original line is left unchanged. Automatic line wrapping is a new SPEC feature with CPU2006 and MPI2007.
output_format H all Format for reports. Valid options are listed at runspec.html under --output_format; major options include text (ASCII text), html, pdf, and ps. You might prefer to set this to text if you're going to be doing lots of runs, and only create the pretty reports at the end of the series. See also the information in runspec.html about --rawformat.
ranks H,N 0 The number of ranks to use in a base run. Note that the submit directive must be set up to use this variable. Note also that the peak rule for an individual benchmark may set this variable differently.
reportable H 0 Strictly follow reporting rules. You must set reportable to generate a valid run suitable for publication and/or submission to SPEC.
rebuild H 0 Rebuild binaries even if they exist.
runlist H none What benchmarks to run. Names can be abbreviated, just as on the command line. See the long discussion of run order in runspec.html.
setprocgroup H 1 Set the process group. On Unix-like systems, improves the chances that ^C gets the whole run, not just one of the children.
size H mref Size of input set. If you are in the early stages of testing a new compiler or new set of options, you might set this to mtest or mtrain.
table H 1 In ASCII reports, include information about each execution of the benchmark.
tune H base default tuning level. In a reportable run, must be either all or base.
verbose H 5 Verbosity level. Select level 1 through 99 to control how much debugging info runspec prints out. For more information, see the section on the log file, below.

II.B. Options for runspec that can only be used in a config file

The following options control the operation of runspec, but can not be specified on the command line. Instead, they must be specified in the config file.

In the table that follows, the "Use In" column indicates where the option can be used:

H use only in header section
N use in a named section.
H,N can be used in both the header section and in named sections. The item can therefore be applied on a global basis, and/or can be applied to individual benchmarks.

Note that requesting a parallel build with makeflags = -j N causes multiple processors to be used at build time. It has no effect on how multiple processors are used at run time, and so does not affect how you report on parallelism.

Option Use In Default Meaning
allow_extension_override H 0

The runspec command can use --extension to select among different options in a config file, as mentioned above. But what if the extension mentioned on the runspec command does not occur in any section marker? What should be done then?

  • If allow_extension_override is 0 (or no or false), the tools will print an error message and exit.
  • If allow_extension_override is 1 (or yes or true), then the tools will attempt to build (or run) the same binary they would have built by default, but will name the binary with specified extension. See the example above.

The allow_extension_override is a new SPEC feature with CPU2006 and MPI2007.

backup_config H 1 When updating the MD5 hashes in the config file, make a backup copy first. Highly recommended to defend against full-file-system errors, system crashes, or other unfortunate events.
basepeak H,N 0 Use base binary and/or base result for peak. If applied to the whole suite (in the header section), then only base is run, and its results are reported for both the base and peak metrics. If applied to a single benchmark, the same binary will be used for both base and peak runs, and the median of the base run will be reported for both.
build_in_build_dir H 1

When set, put build directories in a subdirectory named

benchspec/CPU2006/nnn.benchmark/build/build... (Unix) or
benchspec\CPU2006\nnn.benchmark\build\build... (Windows)

Specifying '0' will cause the build directories to be the same as in CPU2006 V1.0:

benchspec/CPU2006/nnn.benchmark/run/build... (Unix)
benchspec\CPU2006\nnn.benchmark\run\build... (Windows)

Why are build directories separated? Benchmarks are now built in directories named nnn.benchmark/build rather than under the benchmark's run subdirectory in order to make it easier to copy, backup, or delete build and run directories separately from each other. It may also make problem diagnosis easier in some situations, since your habit of removing all the run directories will no longer destroy essential evidence 10 minutes before the compiler developer says "Wait - what exactly happened at build time?".

If you prefer the V1.0 behavior, you can revert to it by setting build_in_build_dir to 0.

The build_in_build_dir feature is new with CPU2006 V1.1.

check_md5 H 1 Runspec uses MD5 hashes to verify that executables match the config file that invokes them, and if they do not, runspec forces a recompile. You can turn that feature off by setting check_md5=0.

Warning: It is strongly recommended that you keep this option at its default, '1' (that is, enabled). If you disable this feature, you effectively say that you are willing to run a benchmark even if you don't know what you did or how you did it -- that is, you lack information as to how it was built!

The feature can be turned off because it may be useful to do so sometimes when debugging (for an example, see env_vars, below), but it should not be routinely disabled.

Since SPEC requires that you disclose how you build benchmarks, reportable runs (using the command-line switch --reportable or config file setting reportable=1) will cause check_md5 to be automatically enabled.

command_add_redirect H 0 If set, the generated $command will include redirection operators (stdout, stderr), which are passed along to the shell that executes the command. If this variable is not set, specinvoke does the redirection itself. This option is commonly used when using the submit command; please see the example in section I.D.1.d.
difflines H,N 10 Number of lines of differences to print when comparing results.
env_vars H,N 0

If this variable is set to 1, then environment settings can be changed for benchmarks using ENV_* options in the config file. For example, consider the following executable, which fails because the library directory has been removed:

   $ ls -l bwaves_base.tmp3
   -rwxrwxr-x   1 david ptg       304964 May 16 10:16 bwaves_base.tmp3
   $ ldd bwaves_base.tmp3 | head -3
           libmvec.so.1 =>  /lib/libmvec.so.1
           libfui.so.2 =>   (file not found)
           libfai.so.3 =>   (file not found)
   $ 

But if we add the following to the config file:

   check_md5=0
   env_vars=1
   410.bwaves:
   ENV_LD_LIBRARY_PATH=/spec/david/lovecraft/lib

then bwaves gets an LD_LIBRARY_PATH which points to the new library directory, and the benchmark runs successfully.

Note that MPI2007 allows env_vars to be set for reportable runs while CPU2006 does not. The CPU2006 run rules require that the environment be set prior to the start of runspec. Since control of some MPI libraries is only possible by adjusting environment variables, this feature is allowed in reportable MPI2007 runs.

Which environment? If you are attempting to communicate settings from your shell environment into runspec, this is not the feature that you are looking for. Try the config file preprocessor instead. The env_vars option and ENV* are about communication from config file to environment of the invoked benchmark.

When developing a config file that uses env_vars, you may find it useful to set the verbosity level to 35 (or higher), which will cause the tools to log environment settings. MPI2007 newly provides logging of env_vars settings.

expand_notes H 0 If set, will expand variables in notes. This capability is limited because notes are NOT processed by specmake, so you cannot do repeated substitutions. You'll find some suggestions above.
expid H If set to a non-empty value, will cause executables, run directories, results, and log files to be put in a subdirectory (with the same name as the value set) in their normal directories. For example, setting expid = CDS will cause benchmark binaries to end up in exe/CDS, run directories to end up in run/CDS, and results and logs in $SPEC/result/CDS. The expid is a new SPEC feature with CPU2006 and MPI2007.
fail H,N 0 If set, will cause a build or run to fail. The ability to force a run to fail is new with MPI2007
fail_build H,N 0

If set, will cause a build to fail. For example, you could say something like this:

122.tachyon=default:
#> I am posting this config file for use by others in the
#> company, but am forcing it to fail here because
#> I want to force users to review this section.
#>
#> Once you find your way here, you should test whether
#> bug report 234567 has been fixed, by using the first
#> line below.  If it has not been fixed, then use the
#> second.  In either case, you'll need to remove the
#> fail_build.
#>
#>   - Pney Guvaxre
#>     Boomtime, the 66th day of Confusion in the YOLD 3172

# OPTIMIZE = -Osuperduper
# OPTIMIZE = -Omiddling
fail_build = 1

In the example above, the build is forced to fail until the user examines and modifies that section of the config file. Notice that Pney has used protected comments to cause the comments about the internal bug report to disappear if the config file were to be published as part of a reportable run.

The ability to force a build to fail is a new SPEC feature with CPU2006 and MPI2007
fail_run H,N 0 If set, will cause a run to fail. The ability to force a run to fail is a new SPEC feature with CPU2006 and MPI2007
ignore_sigint H 0 Ignore SIGINT. If this is set, runspec will attempt to continue running when you interrupt one of its child processes by pressing ^C (assuming that you have ^C mapped in the common way). Note that this does NOT cause runspec itself to ignore SIGINT.
inherit_from N ''

If set within a benchmark section, allows explicit inheritance of settings from another benchmark's section. The section to be inherited from is referenced using colons between the four section specifiers. Other inheritance mechanisms continue to work. Effectively, the referenced benchmark is the second highest priority -- second only to items specifically mentioned in the referring section. An example may help to clarify these points:

   $ cat -n tmp8.cfg
        1  iterations = 1
        2  size       = mtest
        3  teeout     = yes
        4
        5  default=default:
        6  OPTIMIZE = -O1 
        7
        8  medium_all_c,medium_all_cpp=default:
        9  OPTIMIZE = -O2
       10
       11  122.tachyon=default:
       12  OPTIMIZE = -O3 
       13  CC = mpcc_r
       41
       15  126.lammps=peak:
       16  inherit_from = 122.tachyon:default:default:default
       17  CXX = mpCC_r
   $ runspec --config tmp8 --tune base lammps | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O2 lammps.cpp
   $ runspec --config tmp8 --tune peak tachyon | grep lammps.cpp
   mpCC_r -c -o lammps.o -DSPEC_MPI -DNDEBUG -DFFT_NONE -O3 lammps.cpp
   

In the above example,

  • The first runspec command uses the tuning for medium_all_c and medium_all_cpp benchmarks, from the config file lines 8-9.
  • The second runspec command:
    • Does not apply the optimization level on line 9, because it is overridden by other options. Line 16 says that we are to inherit from the 122.tachyon section, so the optimization level at line 12 (-O3) is used instead.

Line 15 above could have been simplified. Just as trailing default section specifiers can be omitted at the original definition points, as explained above, they can also be omitted on an inherit_from option. The inherit_from is a new SPEC feature with CPU2006 and MPI2007.

keeptmp H 0

Whether or not to keep various temporary files. If you leave keeptmp at its default setting, temporary files will be automatically deleted after a successful run. If not, temporary files may accumulate at a prodigious rate, and you should be prepared to clean them by hand. Temporary files include:

  • MPI2007.nnn.log.debug in your result directory ($SPEC/result, %SPEC%\result, or the corresponding location under your output_root).

  • A directory named tmp under your top level SPEC directory, or under your output_root.

  • If tmp is not usable, you might find temporaries under tmp.mpi2007 or, possibly, tmp.mpi2007.n, where n may range from 1 to 15.

The keeptmp feature is new with SPEC CPU2006 and MPI2007 v1.1.

line_width H 0 Line wrap width for screen. If left at the default, 0, then lines will not be wrapped and may be arbitrarily long.
locking H 1 Try to use file locking to avoid race conditions, e.g. if more than one copy of runspec is in use. Although performance tests are typically done with only one copy of runspec active, it can be handy to run multiple copies if you are just testing for correctness, or if you are compiling the benchmarks.
log_line_width H 0 Line wrap width for logfiles. If your editor complains about lines being too long when you look at logfiles, try setting this to some reasonable value, such as 80 or 132. If left at the default, 0, then lines will not be wrapped and may be arbitrarily long.
log_timestamp H 0 Whether or not to prepend time stamps to log file lines. The log_timestamp feature is new in CPU2006 V1.1
mailcompress H 0 When using the 'mail' output format, turning this on will cause the various report attachments to be compressed. Report mailing is a new SPEC feature with CPU2006 and MPI2007.
mailmethod H smtp When using the 'mail' output format, this specifies the method that should be used to send the mail. On UNIX and UNIX-like systems, there are three choices: 'smtp' (communicate directly with an SMTP server over the network), 'mail' (try using mail(1) if available), and 'sendmail' (try invoking sendmail directly). On Windows systems, only 'smtp' is available. SMTP is the recommended setting. Report mailing is a new SPEC feature with CPU2006 and MPI2007.
mailport H 25 When using the 'mail' output format, and when the mailmethod is 'smtp', this specifies the port to use on the mail server. The default is the standard SMTP port and should not be changed. Report mailing is a new SPEC feature with CPU2006 and MPI2007.
mailserver H 127.0.0.1 When using the 'mail' output format, and when the mailmethod is 'smtp', this specifies the IP address or hostname of the mailserver through which to send the results. Report mailing is a new SPEC feature with CPU2006 and MPI2007.
mailto H ''

The address or addresses to which results should be sent when using the 'mail' output format. If multiple addresses are specified, they should be separated by commas or whitespace. Each address should consist only of the name@domain part (i.e. no "full name" type info). The addresses are not checked for correct formatting; if a mistake is made, the results may be sent to an unknown location. Think: comp.arch. OK, probably not there, but seriously be careful about security on this one.

Config files as posted at www.spec.org/mpi2007 will not include whatever you put on this line (thus, spambots will not see the contents of this field).

Note that to get your reports mailed to you, you need to specify both mail as an output_format and an address to which they should be mailed. For example:

   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail

If no addresses are specified, no mail will be sent. Report mailing is a new SPEC feature with CPU2006 and MPI2007.

mail_reports H all The list of report types to mail. The format and possible values are the same as for output_format, with the addition of log, which will cause the current log file to be sent. The default is for all files associated with the run to be mailed (so, this will include what you listed as your desired output_format plus log (the log file) and rsf (the rawfile). You can cut your email down to the bare essentials with something like this:
   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail
   mail_reports=text
If none of the requested report types were generated, no mail will be sent. Report mailing is a new SPEC feature with CPU2006 and MPI2007.
make H,N specmake Name of make executable. Note that the tools will enforce use of specmake for reportable results.
makeflags H,N '' Extra flags for make (such as -j). Set this to -j n where n is the number of concurrent processes to run during a build. Omitting n or setting it to zero unlimits the number of jobs that will be run in parallel. (Use of -j in conjunction with ONESTEP will still result in successful builds, but they will be necessarily serialized unless your compiler implements the parallelism itself.) Use with care! Other flags should be used here only if you are familiar with GNU make.
mean_anyway H 0 Calculate mean even if invalid. DANGER: this will write a mean to all reports even if no valid mean can be computed (e.g. half the benchmarks failed). A mean from an invalid run is not "reportable" (that is, it cannot be represented in public as the SPEC metric).
minimize_rundirs H 0 Try to keep working disk size down. Cannot be used in a reportable run.
minimize_builddirs H 0 Try to keep working disk size down during builds.
no_monitor H,N '' Exclude the listed workloads from monitoring via the various monitor_* hooks. The no_monitor feature is new with MPI2007.
no_input_handler H,N close

Method to use to simulate an empty input. Choices are:

  • close - Just close stdin before executing the benchmark (this is the default).
  • zero - Create a zero-length file and attach it to stdin.
  • file - Synonym for zero.
  • null - Open the null device (/dev/null or NUL:) and attach it to stdin.

Normally, this option should be left at the default; it was actually added to the tools for the benefit of a different SPEC suite that needed the feature. If a reportable run for MPI2007 uses this feature, an explanation should be provided as to why it was used. The no_input_handler is a new SPEC feature with CPU2006 and MPI2007.

notes_wrap_indent H ' ' When line wrapping is enabled (see notes_wrap_columns), this is the string that will be prepended to newly created lines after the indentation from the original line is applied. The default is four spaces, but it can be set to any arbitrary string. Automatic wrapping is a new SPEC feature with CPU2006 and MPI2007.
output_root H

When set to a non-empty value, causes all files written (other than config files) to be rooted in the directory named by the value, instead of being rooted in $SPEC. For example, setting output_root = /tmp/foo will cause results and logs to be deposited in /tmp/foo/result. This also applies to benchmark binaries and run directories.

This feature can be used to easily allow multiple people to access a single benchmark installation to which (with one exception) they do not need write access. The exception is $SPEC/config. Because the setting for the location comes from the config file, the config files still live under $SPEC. On Unix systems, you might choose to set the permissions on the config subdirectory to 1777 (which is the same as /tmp on many systems).

For an example of the use of output_root, see the section on it in runspec.html. The output_root is a new SPEC feature with CPU206 and MPI2007.

preenv H 1

Use preENV_ lines in the config file. When this option is set (the default), lines of the form preENV_<variable> = <value> will cause runspec to set the specified environment variable to value and re-exec runspec to perform the run. The restart is done in order to enforce the "unchanging environment" run rule.

Multiple preENV_ settings may appear in the config file.

For an example of the use of preENV settings, see the discussion of $LD_LIBRARY_PATH in the description of make_bundle.

The preenv option is new with SPEC CPU2006 and MPI2007 V1.1

section_specifier_fatal H 1 While parsing the config file, if a section specifier is found that refers to an unknown benchmark or benchset, an error is output and the run stops. Set section_specifier_fatal=0 in the header section of your config file to convert this error into a warning and allow the run to continue. Prior to MPI2007, section specifier errors were silently ignored - which sometimes led to surprising results for testers. The new suite, by default, prevents surprises by making the errors very visible.
sendmail H /usr/sbin/
sendmail
When using the mail output format, and when the mailmethod is sendmail, this specifies the location of the sendmail binary. The ability to mail reports is a new SPEC feature with CPU2006 and MPI2007.
src.alt N '' Name of subdirectory under <benchmark>/src/src.alt/ from which to draw approved source code modifications. Set this in the named section for the benchmark(s) where you wish to have src.alt(s) applied. This should be set only where required, on a per-benchmark basis. Multiple src.alts may be specified; the names should be separated by commas. If the specified alternate sources are not for the version of the suite in use, or if they are corrupt, the build will fail.

You may also spell this option as srcalt.
Any use of this feature is automatically noted in reports.
See also the chapter "About Alternate Sources", below.

Automatic reporting of src.alt's is a new SPEC feature with CPU2006 and MPI2007.
strict_rundir_verify H 1

When set, the tools will verify that the file contents in existing run directories match the expected MD5 checksums. Normally, this should always be on, and reportable runs will force it to be on. Turning it off might make the setup phase go a little faster while you are tuning the benchmarks. CPU2000, OMP2001 and HPC2002 provided fewer controls over the degree of verification.

Developer notes: setting strict_rundir_verify=0 might be useful when prototyping a change to a workload or testing the effect of differing workloads. Note, though, that once you start changing your installed tree for such purposes it is easy to get lost; you might as well keep a pristine tree without modifications, and use a second tree that you convert_to_development.

sysinfo_program H '' Specifies the name of an executable program or script that delivers system configuration information about the system under test for use in the reports. Please see the comments in sample-sysinfo-program.pl for complete details. (Note: some combinations of servers/browsers will refuse to open files of type .pl; if the link does not work for you, try looking on your SPEC distribution media, or you might be able to right-click and save the copy posted at www.spec.org/mpi2007/Docs.) The sysinfo_program is a new SPEC feature with CPU2006 and MPI2007.
teeout H 0 Run output through tee so you can see it on the screen. Primarily affects builds, but also provides some information about progress of runtime, by showing you the specinvoke commands.
version_url H http://
www.
spec.org/
mpi2007/
current_
version
If version checking is enabled, this specifies the location from which the version information should be fetched.

III. Config file options for specmake

Most commonly used variables:

Name Meaning
CC How to invoke your C compiler.
CXX How to invoke your C++ compiler.
FC How to invoke your Fortran compiler. The MPI2007 Fortran benchmarks are expected to be compliant with Fortran 90 so there are no special options for invoking Fortran 77. If you wish to use a Fortran 77 compiler for a benchmark in peak, set FC for that benchmark, not the F77 which had been the convention in SPEC CPU2000, OMP2001, and HPC2002.
   
CLD How to invoke the linker when compiling C programs.
CXXLD How to invoke the linker when compiling C++ programs.
FLD How to invoke the linker when compiling Fortran programs.
   
ONESTEP If set, build from sources directly to final binary. See the discussion in rule 2.3.7 of runrules.html.
   
PORTABILITY Portability flags to be applied when compiling and linking all source files.
CPORTABILITY Portability flags to be applied when compiling and linking C source files.
CXXPORTABILITY Portability flags to be applied when compiling and linking C++ source files.
FPORTABILITY Portability flags to be applied when compiling and linking Fortran source files.
FPPPORTABILITY Portability flags to be applied when pre-processing Fortran sources. Note that since preprocessing is not a part of standard Fortran, SPEC supplies a copy of the freely available filepp, with minor modifications, as specpp. Using specpp ensures that the preprocessing is done consistently across platforms. If you need to define Fortran preprocessor variables, do not put them in FPORTABILITY. Instead, put them in FPPPORTABILITY or PORTABILITY.
   
OPTIMIZE Optimization flags to be applied when compiling all source files. This is typically set in single-benchmark rules.
COPTIMIZE Optimization flags to be applied when compiling C source files.
CXXOPTIMIZE Optimization flags to be applied when compiling C++ source files.
FOPTIMIZE Optimization flags to be applied when compiling Fortran source files.
   
EXTRA_LDFLAGS Additional flags to pass to your linker.
LDCFLAGS Flags to add to link lines used for C builds.
LDCXXFLAGS Flags to add to link lines used for C++ builds.
LDFFLAGS Flags to add to link lines used for Fortran benchmarks or benchmarks that use both Fortran and C.
   
LIBS Libraries to link into the final executable.
   
RM_SOURCES Ignore these source files. Its usage must be explained to the HPG committee, and is typically used to substitute calls to native system libraries for libraries that are packaged with the MPI2007 sources, compliant with run rule 2.2.2.

Other build variables:

Name Meaning
EXTRA_CFLAGS Additional flags to pass to your C compiler.
EXTRA_CXXFLAGS Additional flags to pass to your C++ compiler.
EXTRA_FFLAGS Additional flags to pass to your f90 compiler.
FPPFLAGS Flags to be added to Fortran preprocessor.
EXTRA_FPPFLAGS Additional flags to pass to Fortran preprocessor.
LDOPT Options to supply to the linker for all builds.
   
EXTRA_PORTABILITY Additional portability flags to pass to all compilers.
EXTRA_CPORTABILITY Additional portability flags to pass to your C compiler.
EXTRA_CXXPORTABILITY Additional portability flags to pass to your C++ compiler.
EXTRA_FPORTABILITY Additional portability flags to pass to your Fortran compiler.
LDPORTABILITY Portability flags supplied to the linker for all builds.
   
EXTRA_OPTIMIZE Additional flags to pass to all compilers.
EXTRA_COPTIMIZE Additional optimization flags to pass to your C compiler.
EXTRA_CXXOPTIMIZE Additional optimization flags to pass to your C++ compiler.
EXTRA_FOPTIMIZE Additional optimization flags to pass to your Fortran compiler.
LDOPTFLAGS Optimization flags to pass to the linker.
   
OS_LIBS List of OS-specific libs to link with.
EXTRA_LIBS Extra libraries to appear at the end of the link command.
EXTRA_CLIBS Additional libraries to link in to C benchmarks.
EXTRA_CXXLIBS Additional libraries to link in to C++ benchmarks.
EXTRA_FLIBS Additional libraries to link in to Fortran and mixed-language benchmarks.
   
PORTABILITY_LIBS Portability libraries to link in with all benchmarks.
PORTABILITY_CLIBS Portability libraries to link in with C benchmarks.
PORTABILITY_CXXLIBS Portability libraries to link in with C++ benchmarks.
PORTABILITY_FLIBS Portability libraries to link in with Fortran and mixed-language benchmarks.
   
OPTIMIZATION_LIBS Libraries used for optimization purposes, to be added to all builds.
OPTIMIZATION_CLIBS Libraries used for optimization purposes, to be added to C builds.
OPTIMIZATION_CXXLIBS Libraries used for optimization purposes, to be added to C++ builds.
OPTIMIZATION_FLIBS Libraries used for optimization purposes, to be added to Fortran and mixed C and Fortran builds.
   
MATHLIBOPT Set of math libraries (if any) to link into the final executable.
   
OBJOPT Option to compile and name an output file. Defaults to "-c -Fo$@" on Windows and "-c -o $@" elsewhere.
COBJOPT Special OBJOPT for C. Defaults to same as OBJOPT.
CXXOBJOPT Special OBJOPT for C++. Defaults to same as OBJOPT.
FOBJOPT Special OBJOPT for f90. Defaults to same as OBJOPT.
   
CONESTEP Whether or not to compile source files separately or all at once for C benchmarks. Optimization flag.
CXXONESTEP Whether or not to compile sources separately or all at once for C++ benchmarks. Optimization flag.
FONESTEP Whether or not to compile source files separately or all at once for Fortran or mixed-language benchmarks. Optimization flag.
   
EXTRA_SOURCES Sources to add when doing a build. Must be explained to SPEC!
EXTRA_RMFILES Additional files to be deleted before a build.
EXTRA_TARGETS Additional targets to make when doing a build. These extra targets would be added using the vendor_makefile mechanism; there are no extra targets defined by the tools.
   
OBJ The file name extension for object files on your system. Defaults to '.obj' on Windows and '.o' elsewhere.
EXT Desired extension for the object files. Set via 'ext' in config file or '--ext' on command line.
   
FLD Name of linker to use for F90 programs. It is assumed that the compiler knows how to call the linker.
LDOUT Linker flag used to specify an output file. Defaults to '-Fe$@' on Windows and '-o $@' elsewhere.
LDOUT How to spell "-o <output file>" on your system. Defaults to "-o $@" on Unix, "-Fe$@" on Windows.
   
ECHO How to spell "echo" command. Defaults to 'echo'.
NEEDATFILE Variable to set to indicate that lists of files be stored in a file to reduce command line length.
RMRF The name of a program (with arguments, if necessary) that can be used to delete a directory tree.

User-defined variables:

You can also make up your own variable names, which specmake will use (and perform substitution on). A good example of this is for managing compilers that are located under different paths:

C_COMPILER =
F_COMPILER =
#C_COMPILER = /c/dev_050810/vacpp
#F_COMPILER = /c/dev_050810/xlf

CC   = ${C_COMPILER}/usr/bin/mpcc_r
FC   = ${F_COMPILER}/usr/bin/mpxlf95_r

The CC and FC compilers will be found at the paths determined by the C_COMPILER and F_COMPILER variables. As stated, the example above uses the default paths under /usr/bin/. To use the alternate paths under /c/dev_050810/vacpp/usr/bin/ and /c/dev_050810/xlf/usr/bin/, uncomment the second set of assignment lines prior to invoking runspec.

Assigned variables:

A long list of variables are defined and set in the benchspec/Makefile.defaults file, the individual benchspec/MPI2007/*/Spec/object.pm files, and the SPEC tools. Setting these variables in your config-file is not supported and is unlikely to accomplish your desired goal. The settings of these variables are only interesting some technical cases, for example:

Here is the list of variables, and where their values are set:

Source Name Meaning
object.pm BENCHLANG Benchmark language; one of C, CXX, F.
(tools) BENCHMARK The formal benchmark name, e.g. 104.milc.
object.pm BENCH_CFLAGS Benchmark specific C flags predefined by SPEC.
object.pm BENCH_CXXFLAGS Benchmark specific C++ flags predefined by SPEC.
object.pm BENCH_FFLAGS Benchmark specific Fortran flags predefined by SPEC.
object.pm BENCH_FLAGS Benchmark specific flags predefined by SPEC.
object.pm BENCH_FPPLAGS Benchmark specific Fortran preprocessor flags predefined by SPEC.
(tools) CPUFLAGS Macros defined on a suite-wide basis for C, C++, and preprocessed Fortran codes.
(tools) CXXC Same as CXX. A minor coding trick uses this to help figure out how to invoke linker.
(tools) DOONESTEP Whether this is a ONESTEP complilation.
object.pm EXEBASE Name of the base benchmark executable (without extension or tuning information).
(tools) FINAL_CFLAGS Full list of flags passed to C compiler.
(tools) FINAL_CXXFLAGS Full list of flags passed to C++ compiler.
(tools) FINAL_FFLAGS Full list of flags passed to F90 compiler.
(tools) FINAL_FPPFLAGS Full list of flags passed to Fortran preprocessor.
(tools) FINAL_LDOPT Full list of flags passed to linker.
(tools) FINAL_LIBS Full list of libraries.
(tools) FINAL_SOURCES Full list of source files to be used for compilation.
(tools) FPP_USED Does this benchmark use the Fortran preprocessor?
(tools) LD The name of your linker. Do not set this directly; set the language specific value (CLD, FLD, etc).
(tools) LDOPTFLAGS Optimization flags supplied to the linker.
(tools) MATH_LIBS Set of math libraries (if any) to link into the final executable (just $MATHLIBOPT again).
object.pm NAME The benchmark name, e.g. milc.
object.pm NEED_MATH Flag telling whether or not the benchmark needs to be linked with the math libraries.
object.pm NUMBER The benchmark number, e.g. 121.
(tools) OBJS List of object files needed to link the final executable.
(tools) OS Name of OS being used. Supplied automatically by specmake.
object.pm SOURCES What the benchmark/src/Makefile expects will normally be used.
(tools) TUNE The tuning level to which the benchmark is being built, i.e. base or peak.

About Automatic Rebuilds

Since the release of SPEC CPU2000, the SPEC CPU, OMP and HPC toolsets have attempted to keep config files and binaries in synch with each other (unless you have set check_md5=0, which is not recommended). This generally means that edits to your config file have often resulted in the binaries being rebuilt, sometimes to the dismay of testers who found that rebuilds were attempted at inconvenient times. The inconvenience was compounded by the fact that the first step in a rebuild is to delete the old binary, so even a very fast interrupt of the run wouldn't help: once the rebuild would start, your binary was gone.

New with CPU2006 and MPI2007 V1.1, unexpected rebuilds have been reduced.

In SPEC MPI2007 V1.0, the tools were much more likely to trigger a rebuild of the benchmark binaries than they are in V1.1, because unrecognized options (e.g. a mis-spelled CXXOPTIMZIE, or a user-defined option such as MY_OPTS) would be passed to specmake, and the tools had no way to know what specmake did with such options. Now, the tools record only the options actually used by specmake, plus the options that are sent to the shell (e.g. via submit), described in the next section). With this more careful recording, config file changes are less likely to trigger rebuilds, unless they actually affect the generated binary.

The tools do not promise to entirely eliminate unexpected rebuilds. Occasionally, they may make assumptions that seem conservative about config file options that might affect the generated binary.

Testing option sensitivity: If you would like to test whether a config file option change might cause a rebuild, without actually DOING the rebuild, here is a recommended procedure:

  1. copy the config file to a test version (let's call it: mytest.cfg)
  2. make the edits in the mytest.cfg
  3. make sure that mytest.cfg does NOT change the default for check_md5
  4. enter this command: runspec --config mytest --size mtest --iterations 1 --noignore_errors --nobuild ...

In the above command:

IV. Config file options for the shell

Some directives in your config file cause commands to be executed by your shell (/bin/sh) or by the Windows command interpreter (cmd.exe). The most essential case is

submit = ... $command ... Command to be used to invoke the benchmark executable. See the several   detailed   examples in the section on Variable Substitution, above.

Because runspec can cause arbitrary commands to be executed,
it is therefore important to read a config file you are given before using it.

In addition to this are directives that can invoke performance measurement tools before and after each benchmark is set up or invoked:

bench_post_setup

Command to be executed after each benchmark's run directory setup phase. The run rules say that this feature may be used to cause data to be written to stable storage (e.g. sync). The command must be the same for all benchmarks. It will be run after each benchmark is setup, and for all workloads (mtest/mtrain/mref).

The bench_post_setup option may appear only in the header section.

The bench_post_setup option is new with SPEC CPU2006 and MPI2007 V1.1.

post_setup

Command to be executed after all benchmark run directories have been set up. The run rules say that this feature may be used to cause data to be written to stable storage (e.g. sync).

monitor_pre_bench Command that is invoked prior to each benchmark being activated by specinvoke.
monitor_post_bench Command that is invoked after to each benchmark completes.

A good example of the monitor_pre_bench / monitor_post_bench is to read a counter register before a benchmark is run and again after it finishes, or to start tracing before the benchmark runs and to stop traces after it finishes:

monitor_pre_bench  = trcon          # Trace OS events into a file.
submit             = ...
monitor_post_bench = trcoff         # Stop tracing.

As an alternative to using these directives, the submit form can be used to invoke a script, which in turn runs the performance measurement tool in conjuction with the benchmark $command. The result of the $command must be returned as the stdout of the wrapper. Using such a wrapper may not be acceptable in a submittable run, since the benchmark invocation can not be inferred from the config file. On the other hand, the monitor conventions may not work:

monitor_pre_bench  = vmstat $interval 1>>$report 2>>$logfile &; PID=$!		# Start monitor in background and record the Process ID.
submit             = ...
monitor_post_bench = kill $PID												# Kill the monitor process.

The problem is that we can derive the PID of the performance monitor at the point we invoke it, but have no easy way to capture it and pass it to the kill command after the benchmark finishes running. This may not be an issue with some monitoring tools, but in complex cases it is better to wrap both the monitor management and benchmark invocation into a single submit script which is passed $command as a parameter:

vmstat $interval 1>>$report 2>>$logfile &
PID=$!

$*                                                      # Run the program.

kill $PID

Directives that can assist in this are:

monitor_wrapper = ... $command ... Setting this directive overrides the submit directive and instead invokes the specified parameter. Note the the benchmark $command must be executed or the benchmark run will not validate.
monitor_specrun_wrapper = ... $command ... This works like monitor_wrapper except $command contains the call to specinvoke as well as the benchmark itself. This removes the issue of managing the output of the benchmark separately from other output streams in the wrapper body, since specinvoke redirects the benchmark output appropriately, but also the time spent running specinvoke will be included in the duration of the monitoring.

V. Config file options for the output report

The result of a SPEC MPI2007 run is a human-readable report that is generated in various formats. Whether or not you submit your result to SPEC, the report should be in a consistent format and should correctly describe the system and the conditions under which the measurement was made. The report is broken into the following sections:

  1. Measurement results
  2. System summary description
  3. Benchmark description
  4. Node description(s)
  5. Interconnect description(s)

Each section consists of a formatted table of information followed by a free-form notes section. The config-file includes a set of fields that specify the contents of each section, other than fields that are generated by the run, such as the measurement results and date. These fields and the conventions for assigning them are described in the following sections.

V.A. Field scoping

Note that the descriptive fields are scoped by config-file section markers - that is, you can vary them according to the type of run that you are doing. For example:

   medium_all_c,medium_all_mixed=default=default=default:
   sw_compiler1 = Compaq C X6.2-259-449AT 
   CC           = cc -v      

   medium_all_cpp=default=default=default:  
   sw_compiler2 = DIGITAL C++ V6.1-029-408B6
   CXX          = cxx -v

   medium_all_fortran,medium_all_mixed=default=default=default:
   sw_compiler3 = KAP Fortran V4.2
   FC           = kf90 -v

In the above example, the information about the C, C++, or Fortran compilers will be printed if you run a benchmark that is written (or partly written) in that language. For example, if you only run 126.lammps, only the DIGITAL C++ V6.1-029-408B6 will appear in the report. Note that a reportable run of medium includes all the benchmarks, so all the compiler details will be printed.

V.B. Continued versus fixed-format fields

In several cases a field is used to describe complex information and can be continued to span multiple lines. The above example uses the forms

	sw_compiler  = 
	sw_compiler1 =
	sw_compiler2 =
	sw_compiler3 =

to generate a 2-to-3-line compiler description in the output report. Fields which allow this kind of continuation are identified specially in the lists of fields in the sections below. You can append any sequence of numbers onto the field and there is no set limit on how long it can be, although in extreme cases like notes_plat782348320742972403 you just might encounter the dreaded (and highly unusual) out of memory error, so don't do that. Ex-BASIC programmers might naturally use line number sequences like 1000, 1010, 1020 or 1030 etc. because they leave room to insert lines in between.

In fact these suffixes will be replaced with consecutive three-digit numbers, with leading zeros as necessary, in the rsf file generated by the run. Editing these files may be necessary to finalize system details prior to submitting the report to SPEC/HPG for review.

Fields that do not allow continuation, usually expect a certain fixed format for their value, like a date, a number, fixed range of values such as one of {base,peak,all}, or a short string such as a system or vendor name.

V.C. Free-form notes

In addition to the pre-defined fields, you can write as many notes as you wish. The benchmark, node and file server sections include notes with special suffixes, which are printed in the corresponding section of the report, using a fixed-width font:

  1. notes_comp -- Notes about compiler invocation.
  2. notes_port -- Notes about portability options.
  3. notes_base -- Notes about base optimization options.
  4. notes_peak -- Notes about peak optimization options.
  5. notes -- General notes.

These may in turn be followed by an additional alphanumeric suffix, which is a generalization of the numeric suffixes described in the previous section:

   $ cat tmp.cfg
   size                 = mtest
   iterations           = 1
   output_format        = text
   teeout               = 1
   runlist              = tachyon
   tune                 = base

   notes_part_greeting_011  = ++ how
   notes_part_greeting_20   = ++ you?
   notes_part_greeting_012  = ++ are
   notes_part_aname_1       = ++ Alex,
   notes_part080            = ++ hi

   $ runspec --config tmp > /nev/dull
   $ cd ../result
   $ ls -t *mtest.txt | head -1
   MPIM2007.101.mtest.txt
   $ grep ++ *101*txt
        ++ hi
        ++ Alex,
        ++ how
        ++ are
        ++ you?
   $ 

You can optionally include an underscore just before the numeric part of the suffix, but beware: if you say both notes_plat_105 and notes_plat105, both are considered to be the same line. The last one mentioned will replace the first, and it will be the only one output.

You generally use notes to describe software or hardware details that go beyond what is captured by the predefined fields. For an example of where this might be useful, see example-medium.cfg and search for "patch". Note: some combinations of servers/browsers will refuse to open files of type .cfg; if the link does not work for you, you might be able to right-click and save the file. If the file is not found, you might try using a terminal window (aka command window, terminal emulator) to look in your installed SPEC tree, or on your SPEC distribution media in the config directory.

If no notes are filled in for a particular section, no notes are output, so you don't need to worry about making sure you have something in each section.

V.D Links in notes sections

You can mention URLs in your notes section. New with CPU2006 and MPI2007 V1.1, html reports will correctly render them as hyperlinks. For example:

notes_plat_001 = Additional detail may be found at
notes_plat_002 = http://www.turboblaster.com/servers/big/green/

If you like, you can use descriptive text for the link by preceding it by the word LINK and adding the descriptive text in square brackets:

LINK url AS [descriptive text]

The brackets may be omitted if your descriptive text is a single word, without blanks.

For example:

notes_plat_001 = Additional detail may be found at 
notes_plat_002 = LINK http://www.turboblaster.com/servers/big/green/ AS [TurboBlaster Servers]

When the above appears in an html report, it is rendered as:

Platform Notes

 Additional detail may be found at 
 TurboBlaster Servers

And in a text report, it appears as:

                                Platform Notes
                                --------------
     Additional detail may found at 
     TurboBlaster Servers (http://www.turboblaster.com/servers/big/green/)

Since the text report is not a context in which the reader can click on a link, it is spelled out instead. Note that because the text report spells the link out, the text line is wider than in HTML, PS, and PDF reports. When deciding where to break your notes lines, you'll have to pick whether to plan line widths for text (which may result in thin-looking lines elsewhere) or plan your line widths for HTML/PS/PDF (which may result in lines that fall of the right edge with text). The feature notes_wrap_columns won't help you here, since it is applied before the link is spelled out.

LINK AS is new with CPU2006 and MPI2007 V1.1

V.E Attachments

You can cause files to be attached to a result with this syntax:

ATTACH url AS [descriptive text]

Unlike links (described in the previous section), if you use ATTACH, the mentioned file is actually copied into your result directory.

For example, the notes below will cause /Users/john/Desktop/power.jpg and /Users/john/Desktop/fan.jpg to be copied into the result directory:


notes_plat_110 = Stated performance depends on proper cooling, as shown in the 
notes_plat_120 = ATTACH file:///Users/john/Desktop/power.jpg AS [power supply photo] 
notes_plat_130 = and ATTACH file:///Users/john/Desktop/fan.jpg AS [fan diagram] 

When the above notes are used with an HTML report, they appear as:

Platform Notes

 Stated performance depends on proper cooling, as shown in the
 power supply photo 
 and fan diagram

And in a text report, they appear as:

                                Platform Notes
                                --------------
     Stated performance depends on proper cooling, as shown in the
     power supply photo (MPI2007.011.mtest.jpg) 
     and fan diagram (MPI2007.011.mtest.1.jpg)

In the text report, you can see that when fan.jpg and power.jpg were copied into the result directory, they were given names to show which result they correspond to: MPI2007.011.mtest.jpg and MPI2007.011.mtest.1.jpg. Note that since the text report spells the name of the attachment out, the text line is wider than in HTML, PS, and PDF reports. When deciding where to break your notes lines, you'll have to decide whether to plan line widths for text (which may result in thin-looking lines elsewhere) or plan your line widths for HTML/PS/PDF (which may result in lines that fall off the right edge with text). The feature notes_wrap_columns won't help you here, since it is applied before the attachment name is spelled out.

ATTACH is new with CPU2006 and MPI2007 V1.1

V.F Test description

Test information describes who performed the test and when. The other sections describe how the test was performed and on what system.

license_num The SPEC license number for either the tester or the test_sponsor.
notes_test Free-form notes about other details. Notes can be continued by suffixing the notes variable name.
prepared_by Is never output. If you wish, you could set this to your own name, so that the rawfile will be tagged with your name but not the formal reports.
hw_avail The availability date of the system hardware.
system_name The name assigned to the kind of system. Usually this name collectively describes the key components of the system, e.g. SGI Altix 4700 Cluster (4 nodes at 1600 MHz) with NumaLink4, but in some cases the name may not derive from its components: Our Super Duper Cluster Computer.
system_vendor The Vendor of the particular system. Usually this is the same as the manufacturer of the node used, but a system may also be constructed from 3rd-party commodity boards and the Vendor is the manufacturer of the integrated system rather than the component boards.
system_class The type of compute system:
HomoHomogeneous cluster (all nodes identical).
HeteroHeterogeneous cluster (different kinds of nodes).
SMPSymmetric Multi-Processor (one node containing multiple CPU's).
sw_avail The availability date of the system software.
tester The entity actually carrying out the tests. An optional field; if not specified, defaults to test_sponsor. An example is given in the test_sponsor section.
test_date When the tests were run.
test_sponsor

The entity sponsoring this test. An optional field; if not specified, defaults to system_vendor.

For example, suppose that the Genius Compiler Company wants to show off their new compiler on the TurboBlaster 9000 computer, but does not happen to own a maxed-out system with eight thousand processors. Meanwhile, the Pawtuckaway State College Engineering department has just taken delivery of such a system. In this case, the compiler company could contract with the college to test their compiler on the big machine. The fields could be set as:

      test_sponsor = Genius Compilers
      tester       = Pawtuckaway State College
      system_vendor    = TurboBlaster

V.G. Benchmark description

This section of the report describes the compiling and running of the benchmarks. The first set of parameters are assigned values that have fixed formats:

sw_base_ptrsize Size of pointers in base. Report:
  • "32-bit" if all benchmarks in base are compiled with switches that request only 32-bit pointers.
  • "64-bit" if all benchmarks in base are compiled with switches that request only 64-bit pointers.
  • "32/64-bit" if there is a mix of 32-bit and 64-bit. For example, if the C++ benchmarks use 32-bit pointers, while the Fortran and C benchmarks use 64-bit, then you would report "32/64-bit" here.
sw_peak_ptrsize Size of pointers in peak. Report:
  • "32-bit" if all benchmarks in peak are compiled with switches that request only 32-bit pointers.
  • "64-bit" if all benchmarks in peak are compiled with switches that request only 64-bit pointers.
  • "32/64-bit" if there is a mix of 32-bit and 64-bit.

The second set of parameters can be continued to multiple lines by repeating the variable name multiple times with different suffixes:

notes_base Free-form notes about base optimization options.
notes_comp Free-form notes about compiler invocation.
notes_peak Free-form notes about peak optimziation options.
notes_port Free-form notes about portability options.
sw_notes Free-form notes about other details.
sw_preprocessors Name and version of any preprocessors used.
sw_c_compiler Name and version of the C compiler.
sw_cxx_compiler Name and version of the C++ compiler.
sw_f_compiler Name and version of the Fortran compiler.
sw_mpi_library The type and version of the MPI library used.
sw_mpi_other Other information on the MPI implementation or settings.
sw_other Any other performance-relevant non-compiler software used, including third-party libraries, accelerators, pre-processors, etc.

V.H. Component attributes

A system running MPI may be a standalone SMP or a cluster of nodes with dedicated file servers. The MPI2007 reporting conventions allow you to describe highly heterogeneous systems by specifying the attributes of multiple kinds of nodes, interconnects and file servers.

The output report will contain a separate section describing each of these components. Each one of these sections consists of a formatted table derived from the fixed-field entries in the config-file, followed by free-form notes derived from the notes entries of the config-file.

Each component is described with a prefix that defines its type, followed by the field to be assigned:

	node_IBM655_label = IBM 655 Server                      # This system has only one kind of node
	node_IBM655_count = 16                                  # and uses 16 of them.
	node_IBM655_count = compute,fileserver					# One of the nodes is also the file server.
	node_IBM655_notes = Node 0 is the file server for the cluster.

	node_HP2000_label = HP 2000 Server
	node_HP3000_label = HP 3000 Server                      # This system has two kinds of nodes
	node_HP2000_count = 12                                  # with different numbers of each.
	node_HP3000_count = 20
	node_HP2000_purpose = compute
	node_HP3000_purpose = compute

	interconnect_GBEthernet_data_rate = 1Gb/sec                      # Details of the GB Ethernet interconnect.
	interconnect_InfiniBand_data_rate = 1Gb/sec                      # Details of the InfiniBand interconnect.

	node_SUN1000_label = SUN 1000 Disk Array Server   # Details of the SUN 1000 file server.
	node_SUN1000_count = 1                            # Only one of them.
	node_SUN1000_purpose = fileserver

If you don't declare any components of a given type, no subsection will appear in the report. For example, on an SMT system there will be only one kind of node, and no file server, since the node uses its own local file system. The interconnect is neither replaceable nor configurable so there are no deatils to report, either.

V.I. Node description(s)

An SMP system consists of one node of one type. A heterogeneous cluster may consist of many nodes of several types. Most nodes are used for computation but others may act as file servers or provide other utilities. The fields in this section describe the hardware and software configuration of a given node, and are assigned as follows:

	node_HP2000_vendor = Hewlett-Packard Corporation

The available fixed-format fields are as follows:

node_tag_label The name of the node type.
node_tag_order The output report contains a subsection for each component in the system. The node_order parameter specifies the position of this subsection relative to the others in the report.
node_tag_count The number of nodes in the system, of this type.
node_tag_hw_vendor The hardware vendor. An example of usage of this and related fields is given in the test_sponsor section.
node_tag_hw_model The model of the node type.
node_tag_purpose What the node is used for:
compute Used for computation.
fileserverA file server.
head Initiates processes but is not otherwise used for computation.
other Fills some other purpose, such as a TTY-server.
node_tag_hw_cpu_name Manufacturer-determined formal processor name.
node_tag_hw_ncores Number of CPU cores configured/enabled in the node. See the discussion of CPU counting in the run rules.
node_tag_hw_ncoresorder Valid number of processor cores orderable for this model of node, including a unit. For example: 2, 4, 6, or 8 chips.
node_tag_hw_ncoresperchip The number of processor cores per CPU chip.
node_tag_hw_nchips Number of CPU chips configured. See the discussion of CPU counting in the run rules.
node_tag_hw_nthreadspercore Number of hardware threads per core. See the discussion of CPU counting in the run rules.
node_tag_hw_cpu_char Technical characteristics to help identify the processor. (You'll find more information about the intended use of hw_cpu_name vs. hw_cpu_char in the run rules.)
node_tag_hw_cpu_mhz Speed of the CPUs, in MHz.
node_tag_hw_fpu Floating point unit.
node_tag_hw_pcache 1st level (primary) cache.
node_tag_hw_scache 2nd level cache.
node_tag_hw_tcache 3rd level cache.
node_tag_hw_ocache 4th level or other form of cache.
node_tag_hw_memory Size of main memory (and other performance-relevant information about memory, as discussed in the run rules.)
node_tag_sw_os Operating system name and version.
node_tag_sw_state Multi-user, single-user, default, etc.

Second, continuable fields:

node_tag_hw_disk

Disk subsystem for the SPEC run directories. Three important Notes:

  1. Only a single file system may be used for the run directories, as described in the run rules at Rule 3.1.1.
  2. If the output_root feature is used, its disk subsystem should be the one described here.
  3. Disks used for other purposes - such as the system disk - are usually not a performance-relevant variable, so need not be described. But if you wish to do so, you can extend this field using the same continuation rules as other fields (so, you could use hw_disk01, hw_disk02, etc).
node_tag_sw_local_file Local file space, used perhaps to store intermediate data files. Local file systems are not usually described since (a) the SPEC run-directory must be shared, and (b) file systems other than the SPEC run directory are usually not a performance-relevant variable.
node_tag_sw_shared_file

File system (ntfs, ufs, nfs, etc) for the SPEC run directories. Three important Notes:

  1. Only a single file system may be used for the run directories, as described in the run rules at Rule 3.1.1.
  2. If the output_root feature is used, its file system should be the one described here.
  3. File systems used for other purposes - such as for the operating system - are usually not a performance-relevant variable, so need not be described.
node_tag_sw_other Any other performance-relevant software to describe for the node. (Note that compiler and library details are covered in the Benchmark (REFERENCE) section).
node_tag_hw_other Any other performance-relevant hardware to describe for the node.
node_tag_notes Free-form notes describing other details of the node type. Note that you can use the continuation convention to separate this into node_notes_hw, node_notes_os, etc.

V.J. Adaptor description(s)

The nodes in a compute-cluster use adaptors to attach to the available interconnects. The disclosure convention provides the following fields to declare multiple adaptors of different kinds for each given node type, using a second level of tags

            node_HPCX_hw_adaptor_InfiniBand_model = PCI-Express DDR InfiniBand HCA
            node_HPCX_hw_adaptor_InfiniBand_count = 1

that allow for there being different kinds of adaptors. The defined fields are

node_tag1_hw_adaptor_tag2_model Adaptor model and vendor.
node_tag1_hw_adaptor_tag2_slot_type PCIe x8, HTX, etc.
node_tag1_hw_adaptor_tag2_data_rate Per port nominal data-rate and units, e.g. 1Gb/sec.
node_tag1_hw_adaptor_tag2_count Number of tag2 adaptors attached to nodes of type tag1.
node_tag1_hw_adaptor_tag2_ports_used Number fo cables connecting this kind of adaptor to the interconnect switches.
node_tag1_hw_adaptor_tag2_driver Adaptor driver.
node_tag1_hw_adaptor_tag2_firmware Adaptor firmware.

V.K. Interconnect description(s)

The nodes of a compute-cluster are joined via one or more interconnects that will transmit various kinds of data communication. The report will contain one section for each interconnect. An SMT system with no external file server will not use an interconnect for communication and file transfers, so none of the below fields will be filles in and the report will not contain any interconnect sections. The attributes of each interconnect are prefixed as follows:

	interconnect_GBEthernet_label = Gigabit Ethernet             # If more than one interconnect.
	interconnect_mpiComm_label    = MPI traffic interconnect     # It may be named by what it's used for.

First, the fixed-format fields:

interconnect_tag_label The name of this interconnect type.
interconnect_tag_order The output report contains a subsection for each component in the system. The interconnect_order parameter specifies the position of this subsection relative to the others in the report.
interconnect_tag_hw_vendor The hardware vendor. An example of usage of this and related fields is given in the test_sponsor section.
interconnect_tag_hw_model The model of the interconnect type.
interconnect_tag1_hw_switch_tag2_count Number of switches of this type in the interconnect.
interconnect_tag1_hw_switch_tag2_ports Ports per switch of this type.
interconnect_tag1_hw_switch_tag2_data_rate Descriptive phrase for switch port speed.

Second, field names continuable as described in section V.B.

interconnect_tag_hw_switch_tag2 The vendor and model of the internal aggregation or leaf switch(es) composing this interconnect.
interconnect_tag_hw_topo Description of the interconnect arrangement.
interconnect_tag1_sw_switch_tag2_firmware Switch firmware version.
interconnect_tag_purpose Major use for this interconnect.
interconnect_tag_notes Free-form notes describing interconnect details.

VI. Deprecated options

This section lists the configuration-file options which may be allowable in other benchmark suites that are not allowed in MPI2007.

VI.A SPEC/CPU rate runs

The MPI2007 benchmark suites deviate from the SPEC/CPU suites in that only speed runs are defined, where the time of each benchmark is measured as a single execution, as opposed to a rate run which executes multiple copies of the benchmark simultaneously. The following configuration-file options produce undefined behavior in MPI2007 and are not permitted in a submittable run:

bind List of values to substitute for the $BIND variable with each invocation of the corresponding submit directive.

It is possible that the SPEC/CPU rate run conventions might be able to be used to control the execution of the binaries in an MPI application, as an alternative to invoking an MPI driver like mpich or poe. We will consider adding these or other constructs back into MPI2007 if we find it to be important.

BIND Variable stating the binding index of each benchmark invocation in a rate run.
copies Number of copies to execute in a rate run.
max_active_compares Controls parallel comparison of results from parallel jobs.
rate Specify 1 for a rate run instead of a speed run.
use_submit_for_speed In SPEC/CPU, submit directives only apply to rate runs unless this flag is set to 1. In MPI2007, the submit directives apply to all runs.

VI.B Feedback directed optimization

The MPI2007 benchmark suites deviate from the CPU, OMP, and HPC suites in that feedback-directed compiler optimizations are not allowed. The following configuration-file options are illegal in MPI2007:

fdo_pre0 Commands to be executed before starting a feedback directed compilation series.
fdo_preN Commands to be executed before pass N.
fdo_make_cleanN Commands to be executed for cleanup at pass N.
fdo_pre_makeN Commands to be done prior to Nth compile.
fdo_make_passN Commands to actually do the Nth compile.
fdo_post_makeN Commands to be done after the Nth compile.
fdo_runN Commands to be used for Nth training run.
fdo_postN Commands to be done at the end of pass N.
feedback The feedback option applies an on/off switch for the use of feedback directed optimization (FDO), without specifying how the feedback will be done.
PASSn_CFLAGS Flags for pass "n" C compilation when using feedback-directed optimization (FDO). Typically n is either 1 or 2, for the compile done before the training run and the compile done after the training run.
PASSn_CXXFLAGS Flags for pass "n" C++ compilation when using FDO.
PASSn_FFLAGS Flags for pass "n" Fortran compilation when using FDO.
PASSn_LDFLAGS Flags to use with the linker in pass "n" when using FDO.

VI.C Old-style descriptive formats

A number of descriptive fields were included in earlier benchmark suites that no longer serve any purpose in generating the output reports:

company_name You are probably looking for hw_vendor, tester, or test_sponsor. See the example under test_sponsor. If you do use company_name, the effect will be mostly harmless, except when it surprises you or your users by causing unexpected rebuilds.
machine_name If you do use it, the effect will be mostly harmless, except when it surprises you or your users by causing unexpected rebuilds.
tester_name You are probably looking for hw_vendor, tester, or test_sponsor. See the example under test_sponsor. If you do use tester_name, the effect will be mostly harmless, except when it surprises you or your users by causing unexpected rebuilds.
VENDOR You are probably looking for hw_vendor. If you do use it, the effect will be mostly harmless, except when it surprises you or your users by causing unexpected rebuilds.

VI.D Obsolete hardware designs

At least one field is a holdover from early versions of the SPEC/cpu suite, and no longer pertains to modern microprocessors:

hw_fpu Used to indicate that floating-point co-processor chips were separate from or integrated into the main CPU chip or chipset. All modern general-purpose microprocessors include on-chip floating-point units so the value is always integrated.

If an mpi2007 result submission uses one of these older-style system designs, the SPEC committee may ask that the details be described in the notes section of the node.

VI.E Automatic thread-level parallelization

Some compilers provide an auto parallelization that converts loops into separate threads that execute in parallel. This is not allowed with SPEC/MPI2007, so the field

sw_auto_parallel Were multiple cores employed by a parallelizing compiler? Note that a speed run that uses a parallelizing compiler causes a single instance of a benchmark to run using multiple CPUs; this is different from a rate run, which typically distributes N instances over N CPUs.

introduced in SPEC/cpu2006 and the first release of SPEC/mpi2007, is no longer used.

VII. The config file preprocessor

New with MPI2007, the tools implement a configuration file macro preprocessor. The preprocessor can be used in a variety of ways; for example to communicate settings from your shell environment into runspec:

$ cat > jan.cfg
notes01 = Today, I am happily running in directory %{MYDIR} on system %{HOST}
$ runspec --config jan --define MYDIR=$PWD --define HOST=`hostname` \
  --fakereportable fp --output_format text | grep txt
        format: ASCII -> /jan/mpi2007/result/MPI2007.090.txt
$ grep Today /jan/mpi2007/result/MPI2007.090.txt
     Today, I am happily running in directory /jan/mpi2007/config on system civilized-03
$ 

The config file preprocessor is called a macro processor because it allows you to define macros, which are brief abbreviations for longer constructs. If you've ever used the C preprocessor, the concepts will be familiar, though the syntax is slightly different.

The preprocessor is automatically run whenever you use runspec. Or, you can run it separately, as configpp, which is documented in utility.html.

Preprocessor directives begin with the percent (%) character. This character must be the first character on the line. Any amount of spaces or tabs may separate the percent from the directive.

The following are okay:

%define foo
%    define bar
%		undef hello!

The following are not okay:

# Space in the first column
 %define foo

# Tab in the first column
	%define foo

# This isn't CPP!
#define foo

VII.A Defining macros

The preprocessor is all about macros. There are no macros defined by default, so unless you define some macros, the preprocessor can't do much for you.

Macros can be defined in two ways:

  1. On the command line, using the --define switch, or
  2. In the configuration file, using the '%define' preprocessor directive:
    # Define a simple macro
    %define foo bar
    # Now the macro called 'foo' has the value 'bar'
    
    # It's not necessary for a macro to have a value to be useful
    %define baz
    # Now the macro called 'baz' is defined, but it has no value.
    

    Note that no quoting is necessary when specifying the names of macros or their values.

Macros defined in both ways are entirely equivalent. Because ones set on the command-line are defined first, it's not possible to use the command line to override a macro definition that occurs in the config file itself. It may help to think of a series of '%define' directives, one per --define, as being prepended to the config file.

The values assigned to macros do NOT follow the same quoting rules as variables in the config file. In particular, you may NOT use line continuation, line appending, or block quotes. You may have a value of arbitrary length, but in the interests of config file readability and maintainability, please keep them relatively short.

You will receive a warning if a previously defined macro is re-defined.

Macro names may only be composed of alphanumeric characters, underscores, and hyphens, and they ARE case-sensitive.

A macro that has not been defined will not be substituted. A macro that has been defined, but which has not been assigned a value, has the value "1".

VII.B Un-doing macro definition

Sometimes you want to make the preprocessor forget about a macro that you taught it. This is easily accomplished.

Macros can be undefined in two ways:

  1. On the command line, using the --undef switch, or
  2. In the configuration file, using the '%undef' preprocessor directive.
    %define foo bar
    # Now the macro called 'foo' has the value 'bar'
    
    %undef foo
    # Now it doesn't
    

    Note that no quoting is necessary when specifying the names of macros.

Like macro definition, the undefinition requests can't affect macros set in the config file because those definitions effectively happen after the un-definition. For this reason, command-line undefinition is basically useless; it can only undo macros also set on the command line.
So why was such a useless ability added to the tools? The writer likes orthogonality.

VII.C Using macros

By now you're probably over that initial euphoric rush that comes from wantonly defining and undefining macros, and you're looking for something more. This is it!

When you want to use the value of a macro, you refer to it by name. Unfortunately, the syntax for this is not as simple as you might hope. It's not too complicated, though; to have the preprocessor expand the macro 'foo', just write

  %{foo}

in the place where you'd like it to appear. Given the following config file snippet:

%define foo Hello!
%define bar baz
%define foobar Huh?
%define foobaz What?
%define Hello!baz Please don't do this

Here's a handy table to see the various ways you can reference these values:

Macro reference Value
%{foo} Hello!
%{bar} baz
%{foobar} Huh?
%{foobaz} What?
%{Hello!baz} Please don't do this

Easy, right? The following is also possible:

%{foo%{bar}} What?
%{%{foo}%{bar}} Please don't do this

Because macro values can only be one line long, it's not possible to use the preprocessor to macro-ize large chunks of your config file at once, as may be common practice for advanced users of CPP.

A macro that has not been defined will not be substituted. Thus the following case is the expected behavior:

$ cat tmp.cfg
OPTIMIZE = %{FOO}
$ runspec -c tmp -i mtest -T base tachyon
...
cc -c -o spec.o -DSPEC_MPI -DNDEBUG   %{FOO}   spec.c

Your C compiler probably won't know what to do with with %{FOO}. If you want to substitute an empty string, then assign it one:

$ runspec -c tmp -i mtest -T base --define FOO="" tachyon

Note that the following are NOT equivalent:

   --define FOO=""
   --define FOO

The former sets FOO to an empty string. The latter, effectively, sets it to "1".

VII.D Conditionals

Defining, undefining, and expanding macros is quite an enjoyable activity in and of itself, and can even be useful on occasion. However, conditionals add an entirely new dimension to config file processing: the ability to include and exclude entire sections of text based on macros and their values.

VII.D.1. %ifdef .. %endif

The %ifdef conditional provides a way to determine whether or not a particular macro has been defined. If the named macro has been defined, the conditional is true, and the text to the matching %endif is included in the text of the config file as evaluated by runspec. Note that the matching %endif may not necessarily be the next %endif; conditionals may be nested.

For example, given the following section of a config file:

%define foo
%ifdef %{foo}
This text will be included
%endif

%ifdef %{bar}
This text will not be included
%endif

The preprocessor would produce the following output:

This text will be included

Note especially the quoting used for the macro names in the conditional; the only time macro name quoting may be omitted is when defining or undefining it.

VII.D.2. %ifndef .. %endif

The %ifndef conditional is the converse of %ifdef; If the named macro has not been defined, the conditional is true, and the text to the matching %endif is included in the text of the config file as evaluated by runspec. Note that the matching %endif may not necessarily be the next %endif; conditionals may be nested.

Given a slightly modified version of the example from earlier:

%define foo
%ifndef %{foo}
Now THIS text will not be included
%endif

%ifndef %{bar}
This text WILL be included
%endif

The preprocessor would produce the following output:

This text WILL be included

VII.D.3. %if .. %endif

Checking whether or not a macro is defined is quite useful, but it's just a subset of the more general conditional facility available. This general form is

%if expression
...
%endif

The expression is evaluated using a subset of the Perl interpreter, so the possibilities for testing values are fairly broad. For example,

%ifdef %{foo}
...
%endif

is exactly equivalent to

%if defined(%{foo})
...
%endif

Likewise,

%ifndef %{foo}
...
%endif

is exactly equivalent to

%if !defined(%{foo})
...
%endif

Using the general form, it's possible to string conditionals together:

%if defined(%{foo}) && !defined(%{bar}) || %{baz} == 0
...
%endif

If a macro contains a string value, you must supply quotes:

%if '%{foo}' eq 'Hello, Dave.'
...
%endif

You may also perform basic math on macro values:

%if %{foo} * 2004 > 3737
...
%endif

More precisely, the Perl operations allowed are the :base_core and :base_math bundles, with the ability to dereference and modify variables disallowed. For more details, see the source code for config.pl (the eval_pp_conditional subroutine) and Perl's own Opcode documentation.

VII.D.4. %else

It's possible to get by without the "else" part of the classic "if .. then .. else" trio, but it's not any fun. It works as you'd expect:

%define foo
%ifndef %{foo}
This text will not be included
%else
This text WILL be included (from the else clause)
%endif

The preprocessor would produce the following output:

This text WILL be included (from the else clause)

Only one %else per conditional is allowed.

VII.D.5. %elif

%elif is another convenience that's been added. For those not familiar with CPP, it's an "else if" construct. You may have as many of these as you'd like. Given:

%define foo Hello!

%if !defined(%{foo})
This text will not be included
%elif defined(%{bar})
This text won't be included either
%elif '%{foo}' eq 'Hello!'
This text WILL be included (from the second elif clause)
%else
Alas, the else here is left out as well.
%endif

The preprocessor would produce the following output:

This text WILL be included (from the second elif clause)

VII.E Informational Directives

It's often helpful to be able to warn or abort on certain conditions. Perhaps there's a macro that must be set to a particular value, or maybe it's just very highly recommended.

VII.E.1. %warning

%warning does just what you'd expect; when the preprocessor encounters this directive, it prints the text following to stderr, along with its location within the file being read, and continues on.

Consider:

%if !defined(%{somewhat_important_macro})
% warning
% warning You have not defined somewhat_important_macro!
% warning
%endif

When run through the preprocessor, this yields the following output:

  $ configpp -c warning.cfg
  runspec v2.20 - Copyright 1999-2003 Standard Performance Evaluation Corporation
  Loading standard modules.................
  Loading runspec modules..............
  Locating benchmarks...found 13 benchmarks in 1 benchset.
  Locating output formats: text...config...html...mail...pdf...ps...raw...screen...
  WARNING: 
  WARNING: You have not defined somewhat_important_macro!
  WARNING: 
  Pre-processed configuration file dump follows:
  --------------------------------------------------------------------
  # Invocation command line:
  # /Users/cloyce/SPEC/mpi2007/bin/configpp --configpp -c warning.cfg
  ############################################################################

  --------------------------------------------------------------------

  The log for this run is in result/MPI2007.013.log

  runspec finished at Tue Apr  1 08:55:36 2003; 5 total seconds elapsed

The blank warning lines were added to make the warning stand out a little; as you can see, it's easy for it to be lost in the rest of the output.

VII.E.2. %error

Like %warning, %error logs an error to stderr and the log file. Unlike %warning, though, it then aborts the run.

Consider a slightly modified version of the previous example:

%if !defined(%{REALLY_important_macro})
% error You have not defined REALLY_important_macro!
%endif

When run through the preprocessor, this yields the following output:

  $ configpp -c error.cfg
  runspec v2.20 - Copyright 1999-2003 Standard Performance Evaluation Corporation
  Loading standard modules.................
  Loading runspec modules..............
  Locating benchmarks...found 13 benchmarks in 1 benchset.
  Locating output formats: text...config...html...mail...pdf...ps...raw...screen...
  ERROR: You have not defined somewhat_important_macro!

  The log for this run is in result/MPI2007.014.log

  runspec finished at Tue Apr  1 09:00:06 2003; 4 total seconds elapsed

  $ echo $?
  1

The ERROR line also tends to get a little lost in the visual clutter, but unlike a warning, it will be close to the last thing output. As you can see from the output of 'echo $?', runspec has exited with an error code 1.

VIII. Output files - and how they relate to your config file

This section describes how the location and contents of several kinds of output files are influenced by your config file.

VIII.A. Automatic backup of config files

It was mentioned above that the MD5 section of the config file is written automatically by the tools. Each time your config file is updated, a backup copy is made. Thus your config directory may soon come to look like this:

   $ cd $SPEC/config
   $ ls tmp.cfg*
   tmp.cfg                   tmp.cfg.2007-05-11_0842   tmp.cfg.2007-05-11_1631
   tmp.cfg.2007-05-10_2007   tmp.cfg.2007-05-11_0847   tmp.cfg.2007-05-11_1632
   tmp.cfg.2007-05-10_2010   tmp.cfg.2007-05-11_0853   tmp.cfg.2007-05-11_1731
   tmp.cfg.2007-05-10_2048   tmp.cfg.2007-05-11_0854   tmp.cfg.2007-05-11_1731a
   tmp.cfg.2007-05-10_2051   tmp.cfg.2007-05-11_0855   tmp.cfg.2007-05-12_0921
   tmp.cfg.2007-05-10_2054   tmp.cfg.2007-05-11_0856   tmp.cfg.2007-05-13_0846
   tmp.cfg.2007-05-10_2058   tmp.cfg.2007-05-11_0857   tmp.cfg.2007-05-13_0846a
   tmp.cfg.2007-05-10_2105   tmp.cfg.2007-05-11_0858   tmp.cfg.2007-05-13_0849
   tmp.cfg.2007-05-10_2105a  tmp.cfg.2007-05-11_0903   tmp.cfg.2007-05-13_0850
   tmp.cfg.2007-05-10_2106   tmp.cfg.2007-05-11_0904   tmp.cfg.2007-05-16_0957
   tmp.cfg.2007-05-10_2125   tmp.cfg.2007-05-11_0905   tmp.cfg.2007-05-18_1133
   tmp.cfg.2007-05-10_2125a  tmp.cfg.2007-05-11_0906   tmp.cfg.2007-05-19_1626
   tmp.cfg.2007-05-10_2126   tmp.cfg.2007-05-11_1348   tmp.cfg.2007-05-19_1627
   tmp.cfg.2007-05-10_2127   tmp.cfg.2007-05-11_1349   tmp.cfg.2007-05-19_1634
   tmp.cfg.2007-05-11_0811   tmp.cfg.2007-05-11_1349a  tmp.cfg.2007-05-19_1638
   tmp.cfg.2007-05-11_0823   tmp.cfg.2007-05-11_1349b  tmp.cfg.2007-05-19_1718
   tmp.cfg.2007-05-11_0835   tmp.cfg.2007-05-11_1553   tmp.cfg.2007-05-19_1720
   tmp.cfg.2007-05-11_0836   tmp.cfg.2007-05-11_1556   tmp.cfg.2007-05-19_1731
   tmp.cfg.2007-05-11_0838   tmp.cfg.2007-05-11_1557   tmp.cfg.2007-05-21_0611
   tmp.cfg.2007-05-11_0839   tmp.cfg.2007-05-11_1627   tmp.cfg.2007-05-21_0622
   tmp.cfg.2007-05-11_0840   tmp.cfg.2007-05-11_1629   tmp.cfg.2007-05-21_0652
   $

If this feels like too much clutter, you can disable the backup mechanism, as described under backup_config. Note that doing so may leave you with a risk of losing the config file in case of a filesystem overflow or system crash. A better idea may be to periodically remove just portions of the clutter, for example by typing:

   $ rm tmp.cfg.2007-05-1*
   $ ls tmp.cfg*
   tmp.cfg                  tmp.cfg.2007-05-21_0622
   tmp.cfg.2007-05-21_0611  tmp.cfg.2007-05-21_0652
   $ 

VIII.B. The log file and verbosity levels

$SPEC/result (Unix) or %SPEC%\result (Windows) contains reports and log files. When you are doing a build, you will probably find that you want to pay close attention to the log file. Depending on the verbosity level that you have selected, it will contain enormous amounts of information about how your build went.

The MPI2007 tool suite provides for varying amounts of output about its actions during a run. These levels range from the bare minimum of output (level 0) to copious streams of information almost certainly worthless to anyone not developing the tools themselves (level 99). Note: selecting one output level gives you the output from all lower levels, which may cause you to wade through more output than you might like.

VIII.B.1. Useful Search Strings

When you are trying to find your way through a log file, you will probably find these (case-sensitive) search strings useful:

runspec:The runspec command for this log.
RunningPrinted at the top of a run of a benchmark.
#Printed at the top of a run of a benchmark for runs with multiple iterations. Useful for finding the ref workloads in reportable runs.
runtimePrinted at the end of a benchmark run.
CopyTimes for individual copies in a rate run.
BuildingPrinted at the beginning of a benchmark compile.
Elapsed compilePrinted at the end of a benchmark compile.

VIII.B.2. About Temporary Debug Logs

There are also temporary debug logs, such as MPI2007.001.log.debug. A debug log contains very detailed debugging output from the SPEC tools, as if --verbose 99 had been specified.

For a successful run, the debug log will be removed automatically, unless you specify "--keeptmp" on the command line, or "keeptmp=1" in your config file.

For a failed run, the debug log is kept. The debug log may seem overwhelmingly wordy, repetitive, detailed, redundant, repetitive, and long-winded, and therefore useless. Suggestion: after a failure, try looking in the regular log first, which has a default verbosity level of 5. If your regular log doesn't have as much detail as you wish, then you can examine the additional detail in the debug log.

If you file a support request, you may be asked to send in the debug log.

Temporary debug logs are new with CPU2006 and MPI2007 V1.1.

VIII.B.3. Definitions of verbosity levels

The 'level' referred to in the table below is selected either in the config file verbose option or in the runspec command as in 'runspec --verbose n'.

Levels higher than 99 are special; they are always output to your log file. You can also see them on the screen if you set verbosity to the specified level minus 100. For example, the default log level is 5. This means that on your screen you will get messages at levels 0 through 3, and 100 through 105. In your log file, you'll find find the same messages, plus the messages at levels 106 through 199.

Level What you get
0 Basic status information, and most errors. These messages can not be turned off.
1 List of the benchmarks which will be acted upon.
2 A list of possible output formats, as well as notification when beginning and ending each phase of operation (build, setup, run, reporting).
3 A list of each action performed during each phase of operation (e.g. "Building 127.wrf2", "Setting up 137.lu")
4 Notification of benchmarks excluded
5 (default) Notification if a benchmark somehow was built but nevertheless is not executable.
6 Time spent doing automatic flag reporting.
7 Actions to update SPEC-supplied flags files.
10 Information on basepeak operation.
12 Errors during discovery of benchmarks and output formats.
15 Information about certain updates to stored config files
24 Notification of additions to and replacements in the list of benchmarks.
30 A list of options which are included in the MD5 hash of options used to determine whether or not a given binary needs to be recompiled.
35 A list of key=value pairs that can be used in command and notes substitutions, and results of env_var settings.
40 A list of 'submit' commands for each benchmark.
70 Information on selection of median results.
89 Progress comparing run directory MD5s for executables.
90 Time required for various internal functions in the tools.
95, 96, 97, 98 Flag parsing progress during flag reporting (progressively more detail)
99 Gruesome detail of comparing MD5 hashes of files being copied during run directory setup.
--- Messages at the following levels will always appear in your log files ---
100 Various config file errors, such as bad preprocessor directives, bad placement of certain options, illegal characters...
102 Information about output formats that could not be loaded.
103 A tally of successes and failures during the run broken down by benchmark.
106 A list of runtime and calculated ratio for each benchmark run.
107 Dividers to visually block each phase of the run.
110 Elapsed time for each portion of a workload (if an executable is invoked more than once).
120 Messages about which commands are being issued for which benchmarks.
125 A listing of each individual child processes' start, end, and elapsed times.
130 A nice header with the time of the runspec invocation and the command line used. Information about what happened with your sysinfo program
140 General information about the settings for the current run.
145 Messages about file comparisons.
150 List of commands that will be run, and details about the settings used for comparing output files. Also the contents of the makefile written.
155 Start, end, and elapsed times for benchmark run.
160 Start, end, and elapsed times for benchmark compilation.
180 stdout and stderr from commands run
190 Start and stop of delays
191 Notification of command line used to run specinvoke.

VIII.C. Help, I've got too many logs!

If you do a very large number of builds and runs, you may find that your result directory gets far too cluttered. If it does, you should feel free to issue commands such as these on Unix systems:

   cd $SPEC
   mv result result_old
   mkdir result

On Windows, you could say:

   cd %SPEC%
   rename result result_old
   mkdir result

VIII.D. Finding the build directory

As described under "About Disk Usage" in runspec.html, the MPI2007 tools do the actual builds and runs in the run directories. The benchmark sources are never modified in the src directory.

The build directories for a benchmark are located underneath that benchmarks' top-level directory, typically $SPEC/benchspec/MPI2007/nnn.benchmark/build (Unix) or %SPEC%\benchspec\MPI2007\nnn.benchmark\build (Windows).

(If you are using the output_root feature, then the first part of that path will change to be your requested root instead of SPEC; and if you change the default for build_in_builddir, then the last part of that path will be run instead of build.)

New with MPI2007, the build directories have logical names, typically of the form build_<tune>_<extension>.0000. For example, after the command runspec --config jun09a --action build --tune base leslie3d, the following directory was created:

$ cd $SPEC/benchspec/MPI2007/107.leslie3d/run
$ pwd
/spec/joelw/benchspec/MPI2007/107.leslie3d/run
$ ls -ld build*jun*
drwxrwxr-x   2 joelw ptg    1536 Jun 10 14:49 build_base_jun09a.0000
$ 

On Windows, you would say cd %SPEC%\benchspec\MPI2007\107.leslie3d\run followed by dir build*.

If the directory build_<tune>_<extension>.0000 already exists when a new build is attempted for the same tuning and extension, the directory will be re-used, unless:

In such cases, the 0000 will be incremented until a name is generated that is available. You can find locked directories by searching for lock=1 in the file $SPEC/benchspec/MPI2007/<nnn.benchmark>/run/list (Unix) or %SPEC%\benchspec\MPI2007\<nnn.benchmark>\run\list (Windows).

When more than one build directory has been created for a given tuning and extension, you may need to trace the directory back to the specific build attempt that created it. You can do so by searching for the directory name in the log files:

$ grep Building *log | grep build_base_jun09a.0001
MPI2007.380.log:  Building 107.leslie3d mref base jun09a default: (build_base_jun09a.0001)
$ 

In the above example, the grep command locates log #380 as the log that corresponds to this run directory. On Windows, of course, you would use findstr instead of grep.

VIII.E. Files in the build directory

A variety of files are output to the build directory. Here are some of the key files which can usefully be examined:

Makefile.spec The components for make that were generated for the current config file with the current set of runspec options.
*.err The output from standard error corresponding to the above files.

VIII.F. For more information

For more information about how the run directories work, see the descriptions of specinvoke, specmake, and specdiff in utility.html.

IX. About Alternate Sources

Sometimes a portability issue may require use of alternative source code for a benchmark, and SPEC may issue the alternate as a "src.alt". The effect of applying a src.alt is to modify the sources in the build directory. This chapter provides an example of applying a src.alt, and introduces development of src.alts.

IX.A. Example: Applying a src.alt

This example shows the effect of a src.alt, 400.perlbench.solaris_gcc.cpu2006.v1.0.tar.bz2, which was approved for SPEC CPU2006 V1.0. The README for the src.alt says, in part:

This change is necessary when building 400.perlbench using GCC (or GCC-
compatibile) compilers on newer versions of Solaris.  

The diff is quite large, but the change is easy to describe.  All of the 
code in the PerlIOStdio_invalidate_fileno() function in perlio.c which 
was protected by __sun__ is replaced by the following line:

  return 0;

This makes this section of the code consistent with the code as it stands
in the standard distribution of Perl 5.10.0, and the behavior identical to
the behavior when compiled with Sun Studio.

Having downloaded the src.alt file from www.spec.org/cpu2006/src.alt, we apply it per the instructions (found in the same README from which the above paragraphs were excerpted):

$ . ./shrc

$ specbzip2 -dc 400.perlbench.solaris_gcc.cpu2006.v1.0.tar.bz2 | spectar -xf  -

Let's set up two build directories - one without the change:

$ cat $SPEC/config/without.cfg
ext          = without
$

$ runspec --action buildsetup -T base -c without -v 1 400.perlbench

runspec v4662 - Copyright 1999-2006 Standard Performance Evaluation Corporation
Using 'solaris-sparc' tools
Reading MANIFEST... 17935 files
Loading runspec modules...............
Locating benchmarks...Reading config file '/bench/cpu2006/1.0.1/config/without.cfg'
Benchmarks selected: 400.perlbench

The log for this run is in /bench/cpu2006/1.0.1/result/CPU2006.014.log

runspec finished at Thu Mar 20 18:41:51 2008; 10 total seconds elapsed
$ 

And a build directory with the change:

$ cat $SPEC/config/with.cfg
ext          = with
400.perlbench:
srcalt      = solaris_gcc
$ 

$ runspec --action buildsetup -T base -c with -v 1 400.perlbench
runspec v4662 - Copyright 1999-2006 Standard Performance Evaluation Corporation
Using 'solaris-sparc' tools
Reading MANIFEST... 17935 files
Loading runspec modules...............
Locating benchmarks...Reading config file '/bench/cpu2006/1.0.1/config/with.cfg'
Benchmarks selected: 400.perlbench
400.perlbench (base): Applied solaris_gcc src.alt.

The log for this run is in /bench/cpu2006/1.0.1/result/CPU2006.013.log

runspec finished at Thu Mar 20 18:41:32 2008; 9 total seconds elapsed
$

If we go look at those directories, the "with" directory has the expected change from the src.alt. As mentioned in the README above, a segment of code is removed (lines marked "-"), and return 0; is substituted (the line marked "+"):

$ diff -u build_base_without.0000/perlio.c build_base_with.0000/perlio.c

--- build_base_without.0000/perlio.c    Mon Jun  6 23:30:23 2005
+++ build_base_with.0000/perlio.c       Thu Mar 20 18:12:51 2008
@@ -2847,31 +2847,7 @@
     f->_fileno = -1;
     return 1;
 #  elif defined(__sun__)
-#    if defined(_LP64)
-    /* On solaris, if _LP64 is defined, the FILE structure is this:
-     *
-     *  struct FILE {
-     *      long __pad[16];
-     *  };
-     *
-     * It turns out that the fd is stored in the top 32 bits of
-     * file->__pad[4]. The lower 32 bits contain flags. file->pad[5] appears
-     * to contain a pointer or offset into another structure. All the
-     * remaining fields are zero.
-     *
-     * We set the top bits to -1 (0xFFFFFFFF).
-     */
-    f->__pad[4] |= 0xffffffff00000000L;
-    assert(fileno(f) == 0xffffffff);
-#    else /* !defined(_LP64) */
-    /* _file is just a unsigned char :-(
-       Not clear why we dup() rather than using -1
-       even if that would be treated as 0xFF - so will
-       a dup fail ...
-     */
-    f->_file = PerlLIO_dup(fileno(f));
-#    endif /* defined(_LP64) */
-    return 1;
+    return 0;
 #  elif defined(__hpux)
     f->__fileH = 0xff;
     f->__fileL = 0xff;
@@ -5056,11 +5032,3 @@
     return result;
 }
 #endif

IX.B. Developing a src.alt (brief introduction)

If you are trying to create a new alternative source, you should become familiar with how to work in a sandbox, temporarily abandoning the tools during your development phase. (Or, you can use convert_to_development to make the whole installation into one big sandbox.) Once you have developed your alternative source, you'll want to package it up with makesrcalt, and you'll need to contact SPEC to get the source approved. The convert_to_development utility is new with CPU2006 and MPI2007 V1.1, and makesrcalt is newly documented. Both are much more fully discussed in utility.html.

X. Troubleshooting

When something goes wrong with a build, here are some things to check:

  1. Are there any obvious clues in the log file? Search for the word "Building". Keep searching until you hit the next benchmark AFTER the one that you are interested in. Now scroll backward one screen's worth of text.

  2. Did your desired switches get applied? Go to the build directory, and look at options*out.

  3. Did the tools or your compilers report any errors? Look in the build directory at *err.

  4. What happens if you try the build by hand? See the section on specmake in utility.html.

  5. If an actual run fails, what happens if you invoke the run by hand? See the information about "specinvoke -n" in utility.html

  6. Do you understand what is in your path, and why? Sometimes confusion can be greatly reduced by ensuring that you have only what is needed, avoiding, in particular, experimental and non-standard versions of standard utilities.

  7. Try asking the tools to leave more clues behind, with keeptmp.


Copyright © 1999-2010 Standard Performance Evaluation Corporation
All Rights Reserved