DRAFT DRAFT DRAFT
Submitting a result to SPEC for publication on its website

Last updated: $Date: 2012-07-17 06:37:35 -0400 (Tue, 17 Jul 2012) $ by $Author: JohnHenning $

This document provides information about what needs to be done to get a benchmark run ready for submission to SPEC for publication on its website.

DRAFT this document is subject to change.

Outline
   Published results must follow rules in full detail
    - www.spec.org/cpu2006/Docs/runrules.html
    - all submitters should read the entire doc
    - note in particular section 4.2 about config disclosure
    - fields frequently in need of correction
       - describe the cpu caches correctly
       - if you adjusted bios (e.g. turn off hyper
         threading) you must say so
       - if compiler features require runtime settings,
         you must say what you did at runtime
       - if something was previously submitted with
         one spelling, please spell it the same way

    You need two flags files  
    - Ok to reference existing one for compiler
    - Need to write your own platform flags file

    Email address for submissions
    Email address for corrections and how to do them

    Requests from SPEC for field consistency
        Memory Descriptions (draft is below)
        Compiler versions (plaintext draft below)
        Phrasing of MHz and MHz changes (e.g. Turbo Boost)

Format for Memory Descriptions

Required unless tester asserts a reason not to use it

As an aid to customers, and to improve uniformity when searching SPEC's pages, the SPEC CPU Subcommittee expects the following format to be used for results published on SPEC's web site, unless the tester asserts to the subcommittee that there is a reason not to do so. Examples of such reasons may include:

• Memory organization is of a different type (e.g. not a DIMM) and the fields below are not applicable.
• Tester has evidence that a choice is not performance relevant, per the exception in 4.2.2(k).
• Memory is available only through the same vendor as the system, and choices are not allowed (perhaps for supportability reasons, or for business reasons, the vendor requires use of parts that have passed its own qual procedures).

Although the subcommittee will typically accept such reasoning by testers, SPEC intends to nevertheless encourage use of the following format, even when not technically required by the run rule.

The format

XX ss (YY x gg ss
   {eRxff PChv-wwwwwm-aa <from JEDEC standard [1]>}
   {, ECC <if supported>}
   {, per node <for clustered SUTs>}
   {, running at xxxx MHz <if memory is clocked at a speed other than the 
                          speed implied by the PChv-wwwwm-aa specification>}
      and CLy <if memory is clocked at a different speed than nominal>}
   {, other comments} [2]
   )

{} indicates items that are included only if appropriate.
<> indicates commentary about this format. Do not include in submission.

Although the above string has many fields, in practice it is usually brief, as the following examples show:

• Example #1: 24 GB (6 x 4 GB 2Rx4 PC3-10600R-9, ECC)
• Example #2: 512 GB (64 x 8 GB 2Rx4 PC3L-10600R-9, ECC, running at 1066 MHz and CL8)

Notice that the main string "gg ss eRxff PChv-wwwwwm-aa" can be read directly from the label on the memory itself for all vendors who use JEDEC standard labels.

Notes:

• [1] The appropriate JEDEC labeling standard, such as "DDR3 DIMM Label", PRN09-NM4, October 2009, http://www.jedec.org/standards-documents/docs/pr-n09-nm1 which see for a more complete explanation of several of the fields.

  The wikipedia page on DDR3 is not normative, but may nevertheless be helpful in understanding the fields: http://en.wikipedia.org/wiki/DDR3_SDRAM

• [2] The other comments area can be used for:

  • other descriptions that may be required from the run rule that do not fit the above format (e.g. arrangement of memory modules across memory carriers, if performance relevant)
  • future descriptions that the subcommittee may require
  • data that might be used by other subcommittees, should they choose to use the same format

References

JEDEC:

• "DDR3 DIMM Label", PRN09-NM4, October 2009, http://www.jedec.org/standards-documents/docs/pr-n09-nm1

Wikipedia:

• http://en.wikipedia.org/wiki/DDR2_SDRAM
• http://en.wikipedia.org/wiki/DDR3_SDRAM

Identification of compilers

   WHEREAS it is desirable to improve compiler naming for
   results published on the SPEC website, in order to:

     * 1. Put the most important information first.
     * 2.  Make  it easier for readers to understand what
       compilers were used.
     * 3.  Make it easier for reviewers to verify correct
       content.
     * 4. Make it possible to write tools that parse version
       strings and compare results.

THEREFORE: 

   (Part A) The SPEC CPU
   submission  handler will reject submissions unless the
   compiler string matches the following:

       language: Version n.n.n of name [Build id] [; repeat pre vious]

   where:

      []         indicates optional content
      "language" is a compiler identification string
                 as defined below
      "Version"  is a literal string
      "n.n.n"    is a version identifier as defined below
      "of"       is a literal string
      "name"     is a name of up to 60 characters as defined below
      id         is optional; up to 20 characters, for example
                 "Build 20110304-patch"

   Several of the fields above are defined using Perl regular
   expression syntax below.

  $language = '(C|C\+\+|Fortran)(/C\+\+)?(/Fortran)?';
  $language = "($language|Libraries)";
  $version  = '([\d_-.]+)( Update [\d_-.]+)?';
  $name     = '([^;]){3,60}';
  $build    = '\sBuild [^;]{1,20}';
  $lvn      = "$language: Version $version of $name($build)?";

  $re       = qr/^$lvn(; $lvn)?\$/;

   Notes:
     * 1 The language field contains either a slash-separated
       list of languages, or the word "Libraries". If more than
       one language is mentioned, the order for them is C, C++,
       Fortran.

     * 2 The version field contains digits, underscores, and
       dots, plus an optional string "Update" followed by more
       digits, underscores, and dots.

     * 3 The above definition is not intended to specify the
       exact  Perl  code  for the submission handler. The
       implementer may choose use other syntax to achieve the
       same effect.

     * 4 The checks requested by this proposal will be added to
       the checks already in the submission handler.

     * 5 The submission handler will not attempt to validate
       the name field for the compiler other than checking its
       length, and demanding that it not contain a semi-colon.
       See Part B.

     * 6 The Build field is, from a syntax point of view,
       optional. It may nevertheless be mandatory from a run
       rules point of view: see About Reproducibility, below.

THEREFORE:

   (Part  B) Starting 11-Oct-2011, during review of results,
   the compiler name will be adjusted if it 
   differs from an 'expected' spelling. The expected spelling
   for each compiler name and version will be determined by the
   CPU subcommittee, typically by following the recommendation
   of the compiler vendor, if the vendor is a member of the
   subcommittee.

   The expected spelling will take into account:

     * The run rule (4.1.2)
     * The naming intent of the brand
     * Clarity for readers of SPEC results
     * Reproducibility

About Reproducibility

   If  the  name  and version string as used for ordering
   purpopses are ambiguous with regard to performance-relevant
   updates, additional information shall be included to resolve
   such ambiguity. Examples:

     * The version string for ordering purposes is "12.2" but
       performance changes over time. The "cc -v" command
       prints "12.2.37.9". The subcommittee may decide to
       require the version as printed by "cc -v" for this
       compiler.

     * The version string is always "12.2" (for both ordering
       and from "cc -v"); but performance changes with build
       id. The subcommittee may decide to require inclusion of
       the build id for this compiler.