Skip navigation

Standard Performance Evaluation Corporation

 
 

SPEC MPI Accelerator Benchmark Search Program

SPEC holds to the principle that better benchmarks can be developed from actual applications. SPEC is once again seeking to encourage those outside of SPEC to assist us in locating applications that could be used in the next high performance computing benchmark suite, currently under development within SPEC's High Performance Group (HPG) and currently designated as SPEC MPI ACCEL.

The SPEC MPI Accelerator Search Program is sponsored by the Standard Performance Evaluation Corporation (SPEC), a non-profit consortium; identified as "Sponsor" in this document.

What is the SPEC MPI Accelerator Search Program?

The Search Program is designed to encourage and recognize achievements of members of the academic and industrial community who provide and develop application code and datasets that can be used as components of a new benchmark suite which combine components of the SPEC MPI and SPEC ACCEL benchmark suites.

For each step of the process that a submission passes, SPEC/HPG will compensate the Program Submitter (in recognition of the Submitter's effort and skill). A submission that passes all the steps and is included in the next SPEC MPI ACCEL benchmark suite will receive $5000 overall and a licensed copy of the new benchmark suite when released.

What is SPEC looking for?

Overall, the SPEC ACCEL and SPEC MPI benchmarks have provided performance metrics for comparing systems on known compute-intensive workloads. SPEC ACCEL emphasized the measurement of a single acceleration device, accelerator programming ABIs, device drivers, and compilers. SPEC MPI emphasized multi-node performance including inter-connects, CPUs, and memory architectures. SPEC is expecting to continue this trend by merging the MPI and ACCEL benchmark suites to facilitate support for homogeneous and heterogeneous multi-node systems.

SPEC has held to the principle that better benchmarks are based on applications, and these applications can come from any area of work. For example, the current SPEC ACCEL and SPEC MPI suites includes applications from the following areas:

  • Astrophysics
  • Bioinformatics / DNA Matching
  • Computational Fluid Dynamics
  • Finite Difference
  • Graph Traversal
  • Hydrodynamics
  • Medicine
  • Molecular Dynamics
  • Ocean Modeling
  • Quantum Chromodynamics
  • Seismic Wave Modeling
  • Signal processing
  • Thermodynamics
  • Weather Prediction

SPEC/HPG is not limiting consideration to just these areas. A proposed application may come from any area or field of work.

Criteria considerations important for the next SPEC MPI ACCEL benchmark suite

A good benchmark candidate:

  • Uses MPI+X, where "X" is a node-level parallel programming model such as OpenACC, OpenMP, CUDA, OpenCL, Kokkos, TBB, etc. Additional programming models will be considered for "X" provided the model is widely available and supported. Preferably the candidate will support multiple models. For example, the candidate can be compiled to use either OpenACC or OpenMP depending upon the target system.
  • Includes a MPI only version that is portable across multiple systems. For example, if the candidate uses MPI+OpenCL, the candidate must be able to compile with only MPI for systems without OpenCL support.
  • Is Computationally intensive.
  • Is Highly parallel.
  • Represents the state of the art for the given field.
  • Is derived from a representative application. Full applications are also encouraged provided they can be packaged and contain few library dependencies.
  • Is capable of solving problems of varying sizes allowing for scaling from a single node to many nodes.
  • Is reasonably predictable as to its code path. For example, minor differences in floating point accuracy across platforms should not cause the program/application to do different work on those platforms.
  • Has limited stress on I/O.
  • Must have results validated by numerically comparing to a reference output.

The SPEC MPI Accelerator Search Program Official Rules

  • No purchase is necessary to participate.
  • In all cases, the burden of proof is on the Submitters.
  • SPEC/HPG will evaluate the submitted benchmarks for inclusion in the suite.
  • All evaluation and awards are within the sole discretion of SPEC/HPG.

Who May Submit Benchmarks to the Search Program?

  • VOID WHERE PROHIBITED BY LAW. Not open to any employees or contractors of any SPEC Participating organizations and their immediate families.
  • Entries may be submitted on an individual entrant basis (defined as a natural person), or on a group or team basis, in which case the registration and submission will identify a leader of such group or team.
  • Corporate entities are not eligible.
  • The award program is sponsored by Standard Performance Evaluation Corporation, (SPEC), 7001 Heritage Village Plaza, Suite 225, Gainesville, VA 20155, U.S.A.

Participation Constitutes Agreement to Official Rules

  • Participation in the Search Program constitutes Submitter's full and unconditional agreement to and acceptance of these Official Rules.

Timing

  • Submissions for the first step in the search program will be accepted by SPEC beginning May 22, 2017 and ending December 31, 2017 (11:59 pm, Pacific Standard Time).
  • Status updates through the rest of the process will be provided periodically to the Submitters until the release of the next SPEC ACCEL MPI suite.

How To Enter

Step 1: Complete the legal paperwork

Submit the SPEC MPI Accelerator Search Program Proposal.

All Submitters must provide the following information using this form: http://www.spec.org/hpg/search/entry_form.html

  • the Submitter's name and contact information (email address and shipping address),
  • the name of the program/application,
  • the programming language(s) used in the program/application and approximate lines of code,
  • a description of what the program/application does, the programming model and how it is used, and
  • a statement declaring that the Submitter has the authority to give SPEC the right to distribute the program/application as part of a SPEC benchmark suite.

Upon receipt of the proposal, SPEC/HPG will acknowledge receipt within seven (7) working days. Within four (4) weeks, the HPG committee will notify the Submitter as to whether SPEC/HPG wishes to further evaluate the entry.

If SPEC/HPG wishes to further evaluate the entry, SPEC Headquarters will provide the Submitter with a SPEC "Permission to Use" form and the necessary IRS tax documentation. The Submitter must provide the signed tax form along with a signed "Permission to Use" form for the submitted code, input sets any other material provided to SPEC. Once the signed tax form and the signed Permission to Use form are received by SPEC, technical work may commence.

Step 2: Initial porting ($1000 upon successful completion)

If the submission is accepted for entry into Step 2, SPEC/HPG will provide the Submitter with a copy of the SPEC benchmark framework and information on how to prepare the program/application.

The Submitter, with the assistance of a SPEC/HPG primary contact, must now prepare the Submitter's program/application code for submission and submit it to SPEC.

To complete this step, the Submitter must complete the following:

  • Ensure that the program/application can be built and run within the SPEC/HPG toolset ("runspec", described at www.spec.org/accel/docs/runspec.html).
  • Ensure that any Platform dependent code (e.g. gcc-specific extensions, GNU-specific header files, assembly code) within the program/application is disabled, unless needed for portability. The use of -D compilation switches for portability is allowed, where needed. In some cases, this may require a significant effort to re-work the benchmark.
  • Include an initial input data set that demonstrates compute-bound behavior. At this stage, any data set will suffice that runs for a few minutes when compiled with basic optimization, such as "-O" and two MPI ranks. Note that later steps will require other problem sizes and scaling to larger numbers of MPI ranks.
  • Include a set of expected outputs that demonstrate correct operation with the supplied input data set.
  • Demonstrate that correct answers are obtained on a system running Unix (such as AIX, BSD, HP-UX, Linux, Mac OS X, or Solaris) when compiled with basic optimization (e.g. "-O").
  • Provide a completed SPEC Benchmark Description file, including a description of how workloads can be created for the program/application
  • For C/C++ programs:
    • return an exit code of 0 if it runs successfully, or non-zero otherwise.
    • for the main routine, take one of these two forms:
          int main(void); 
               or 
          int main(int argc, char * argv[]); 
          

A copy of the toolset will be provided, and SPEC can assist with the process of converting code to use the tools.

Note that SPEC's approach to porting benchmarks does not allow the use of an automatic configuration process such as GNU autoconf or Metaconfig for each target platform.

If, in SPEC's sole judgment, the submission does not meet these criteria, the Submitter will be notified as to what expectations were not met.

If the submitted code meets these criteria, the Submitter is awarded $1000 and the submission moves to Step 3.

Note that SPEC/HPG, in its sole judgment, may decide at any time to discontinue work on a submission.

Step 3: Provide workloads and demonstrate profile ($500 upon successful completion)

During Step 3, the Submitter will need to provide four (4) workloads:

  • A proposed test input(s) with validation output. The test input should provide a short test (perhaps 60 seconds) to confirm that the compiled program/application is functional. The pure MPI version should be able to run with one or two ranks.
  • A proposed small input(s) with validation output. The small workload should be sized to run approximately 15 to 30 minutes on a single dual-socket x86 "Haswell" system with 32 cores. The pure MPI version should be able to run with 32 ranks.
  • A proposed medium input(s) with validation output. The medium workload should be sized to run approximately 15 to 30 minutes on multiple dual-socket x86 "Haswell" systems each with 32 cores. The pure MPI version should be able to run with at least 2048 ranks.
  • A proposed large input(s) with validation output. The large workload should be sized to run approximately 15 to 30 minutes on multiple dual-socket x86 "Haswell" systems each with 32 cores. The pure MPI version should be able to run with at least 8192 ranks.

Note that SPEC is likely to adjust requirements for both run time and memory footprint of the workload, based on cost of benchmarking, and final requirements for the benchmark suite. Flexibility as to problem size, and a variety of input sets, is advantageous for a benchmark candidate.

When the above is completed, SPEC/HPG will profile the benchmark with the supplied workload, and expects that:

  • At least 95% of the time is compute bound. This requirement will not be waived.
  • At least 95% of the execution time is spent in the supplied code. SPEC/HPG may consider granting waivers for this requirement where it can be demonstrated that library code - such as exp, BLAS, FFT - is typical of the application area. Such waivers will be solely at SPEC's discretion.

If, in SPEC/HPG's sole judgment, the submission does not meet these criteria, the Submitter will be notified as to what expectations were not met.

If the submitted code meets these criteria, the Submitter is awarded $500 and the submission moves to Step 4.

Note that SPEC/HPG, in its sole judgment, may decide at any time to discontinue work on a submission.

Step 4: Complete Code Testing and Benchmark Infrastructure ($1000 upon successful completion)

During Step 4, the candidate must:

  • Demonstrate to SPEC/HPG's satisfaction that the benchmark runs on a wide variety of additional systems, including (but not limited to) little endian and big endian systems, as well as one or more accelerator devices.
  • Demonstrate to SPEC/HPG's satisfaction that the benchmark is capable of being ported to more than one accelerator programming model. It is not a requirement that the code be ported to multiple models, only that there is nothing that would prevent the code to be ported to additional models in the future.
  • Respond to issues that arise when compilers employ more advanced optimization techniques since it is typical to discover that high optimization exposes problems that are not seen with low optimization. At this stage, analysis is done to try to determine whether failures are due to the benchmarks, or due to the compiler/OS.
  • Resolve standards problems to SPEC/HPG's satisfaction. SPEC always prefers to use code that conforms to the relevant language standards. It is understood that in rare cases this may not be practical (e.g. for entrenched user practices that are, in fact, commonly accepted by compilers.)
  • Complete remaining items in the SPEC/HPG infrastructure, such as "flags files", and ensure that all remaining portability problems are documented and/or fixed.
  • As needed, adjust the workload to meet SPEC's requirements.

If, in SPEC/HPG's sole judgment, the submission does not meet these criteria, the Submitter will be notified as to what expectations were not met.

If the submitted code meets these criteria, the Submitter is awarded $1000 and the submission moves to Step 5.

Note that SPEC/HPG, in its sole judgment, may decide at any time to discontinue work on a submission.

Step 5: Acceptance into the SPEC MPI ACCEL Suite ($2500 if accepted)

If the program/application is recommended to and is accepted by SPEC/HPG, in its sole discretion, then the program/application is included in the suite and the Submitter will receive $2500 and a license for the suite when it is released.

General Information and Legal Terms

Number of Entries and Return of Entries

Submitters may enter as many times as they wish. However, each entry must be accompanied by a separate registration. Sponsor will not return any entries to the Submitters.

Personal Data

Sponsor will use its best effort to use personal data about Submitter only for the administration of the award program, in accordance with these Official Rules and for internal marketing purposes. Submitter agrees to such use of the personal data.

Important Terms

Sponsor is not responsible for lost, late, incomplete, illegible or misdirected mail or e-mail, for failed, partial or garbled computer transmissions, or for technical failures of any kind. Sponsor reserves the right to cancel or modify the awards if fraud, technical failures or other events or circumstances impair the integrity of the award program as determined by Sponsor in its sole discretion.

Submitter agrees that Sponsor's only obligations are to submit entries from eligible submissions, per the procedures and criteria set forth in these Official Rules, and to award the compensation set forth herein, to Submitters as criteria in Sponsor's sole judgment are met, subject to the terms and conditions herein.

Submitter will be held harmless from any liability incurred from the intended or unintended use of the application as a benchmark or application.

Submitter agrees that Sponsor's obligations are solely to fairly and adequately consider any entry submitted, and that Submitter is not entitled to and shall not seek any further compensation.

Submitter hereby grants Sponsor a perpetual, non-exclusive license to copy, modify, display, distribute, and sub-license the entry without geographical limitations or further compensation to Submitter of any kind, provided that Sponsor's sole use of the entry will be for the purposes of inclusion of the entry in Sponsor's MPI Accelerator benchmark suite. Submitter agrees that e-mail shall satisfy any writing requirement, which may apply to intellectual property licenses. Upon request of Sponsor, Submitter agrees to obtain written consent from the owner of the copyright in the program/application, if that person is not Submitter, and to execute any documents required to effectuate the terms of these Official Rules.

Submitter warrants that it has the right to submit this program/application and information to Sponsor and to grant the license rights granted herein, and that to Submitter's knowledge use of the program/application will not infringe or violate any intellectual property rights (including but not limited to patent, trademark, trade secret, and copyright) or license rights of any third party. Submitter agrees to notify Sponsor of any claim of such infringement or violation within fifteen (15) days of its receipt of the claim. Submitter shall defend, indemnify and hold harmless Sponsor, its successors, assigns and licensees for all costs (including attorneys' fees) arising out of or related to any breach of these warranties and representations.

All issues and questions concerning the construction, validity, interpretation and enforceability of these Official Rules, or the rights and obligations of Submitter and Sponsor in connection with the Award Program, shall be governed by, and construed in accordance with, the laws of the Commonwealth of Virginia, U.S.A., without giving effect to any choice of law or conflict of law rules or provisions (whether of the Commonwealth of Virginia, U.S.A., or any other jurisdiction), which would cause the application of the laws of any jurisdiction other than the Commonwealth of Virginia, U.S.A.

Taxes, customs duties, fees, freight charges and other related charges, on prizes are the sole responsibility of the winners. Winners may waive the right to receive compensation.

Submittal Notification

Submitters will be notified by electronic means of each successful completion of each step and may be required to complete an affidavit of eligibility and publicity release (except where prohibited by law), which must be returned within 20 days of receipt by Submitter. Failure to sign and return the affidavit or release or to comply with any term or condition in these Official Rules may result in disqualification and forfeiture of compensation.

This Program is subject to all applicable laws. Void where prohibited.

All questions about the search program can be directed to SPEC at info@spec.org.