aocc6_1699-flags AMD Optimizing C/C++ Compiler Suite Version 6.0 #1318 Staging Flag Descriptions Using numactl to bind processes and memory to cores

For multi-copy runs or single copy runs on systems with multiple sockets, it is advantageous to bind a process to a particular core. Otherwise, the OS may arbitrarily move your process from one core to another. This can affect performance. To help, SPEC allows the use of a "submit" command where users can specify a utility to use to bind processes. We have found the utility 'numactl' to be the best choice.

numactl runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for a command and inherited by all of its children. The numactl flag "--physcpubind" specifies which core(s) to bind the process. "-l" instructs numactl to keep a process's memory on the local node while "-m" specifies which node(s) to place a process's memory. For full details on using numactl, please refer to your Linux documentation, 'man numactl'

Note that some older versions of numactl incorrectly interpret application arguments as its own. For example, with the command "numactl --physcpubind=0 -l a.out -m a", numactl will interpret a.out's "-m" option as its own "-m" option. To work around this problem, we put the command to be run in a shell script and then run the shell script using numactl. For example: "echo 'a.out -m a' > run.sh ; numactl --physcpubind=0 bash run.sh"

]]> numactl --interleave=all runcpu

numactl --interleave=all runcpu executes the SPEC CPU command runcpu so that memory is consumed across NUMA nodes rather than consumed from a single node. This helps prevent local node out-of-memory conditions which can occur when runcpu is executed without interleaving. For full details on using numactl, please refer to your Linux documentation, 'man numactl'

Transparent Huge Pages (THP)

THP is an abstraction layer that automates most aspects of creating, managing, and using huge pages. It is designed to hide much of the complexity in using huge pages from system administrators and developers. Huge pages increase the memory page size from 4 kilobytes to 2 megabytes. This provides significant performance advantages on systems with highly contended resources and large memory workloads. If memory utilization is too high or memory is badly fragmented which prevents huge pages being allocated, the kernel will assign smaller 4k pages instead. Most recent Linux OS releases have THP enabled by default.

THP usage is controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/enabled. Possible values:

The SPEC CPU benchmark codes themselves never explicitly request huge pages, as the mechanism to do that is OS-specific and can change over time. Libraries such as amdalloc which are used by the benchmarks may explicitly request huge pages, and use of such libraries can make the "madvise" setting relevant and useful.

When no huge pages are immediately available and one is requested, how the system handles the request for THP creation is controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/defrag. Possible values:

An application that "always" requests THP often can benefit from waiting for an allocation until those huge pages can be assembled.
For more information see the Linux transparent hugepage documentation.

ulimit -s <n>

Sets the stack size to n kbytes, or unlimited to allow the stack size to grow without limit.

ulimit -l <n>

Sets the maximum size of memory that may be locked into physical memory.

powersave -f (on SuSE)

Makes the powersave daemon set the CPUs to the highest supported frequency.

/etc/init.d/cpuspeed stop (on Red Hat)

Disables the cpu frequency scaling program in order to set the CPUs to the highest supported frequency.

LD_LIBRARY_PATH

An environment variable that indicates the location in the filesystem of bundled libraries to use when running the benchmark binaries.

sysctl -w vm.dirty_ratio=8

Limits dirty cache to 8% of memory.

sysctl -w vm.swappiness=1

Limits swap usage to minimum necessary.

sysctl -w vm.zone_reclaim_mode=1

Frees local node memory first to avoid remote memory usage.

kernel/numa_balancing

This OS setting controls automatic NUMA balancing on memory mapping and process placement. NUMA balancing incurs overhead for no benefit on workloads that are already bound to NUMA nodes.

Possible settings:

For more information see the numa_balancing entry in the Linux sysctl documentation.

kernel/randomize_va_space (ASLR)

This setting can be used to select the type of process address space randomization. Defaults differ based on whether the architecture supports ASLR, whether the kernel was built with the CONFIG_COMPAT_BRK option or not, or the kernel boot options used.

Possible settings:

Disabling ASLR can make process execution more deterministic and runtimes more consistent. For more information see the randomize_va_space entry in the Linux sysctl documentation.

vm/drop_caches

The two commands are equivalent: echo 3> /proc/sys/vm/drop_caches and sysctl -w vm.drop_caches=3 Both must be run as root. The commands are used to free up the filesystem page cache, dentries, and inodes.

Possible settings:

MALLOC_CONF

The amdalloc library is a variant of jemalloc library. The amdalloc library has tunable parameters, many of which may be changed at run-time via several mechanisms, one of which is the MALLOC_CONF environment variable. Other methods, as well as the order in which they're referenced, are detailed in the jemalloc documentation's TUNING section.

The options that can be tuned at run-time are everything in the jemalloc documentation's MALLCTL NAMESPACE section that begins with "opt.".

The options that may be encountered in SPEC CPU 2017 results are detailed here:

PGHPF_ZMEM

An environment variable used to initialize the allocated memory. Setting PGHPF_ZMEM to "Yes" has the effect of initializing all allocated memory to zero.

GOMP_CPU_AFFINITY

This environment variable is used to set the thread affinity for threads spawned by OpenMP.

OMP_DYNAMIC

This environment variable is defined as part of the OpenMP standard. Setting it to "false" prevents the OpenMP runtime from dynamically adjusting the number of threads to use for parallel execution.

For more information, see chapter 4 ("Environment Variables") in the OpenMP 4.5 Specification.

OMP_SCHEDULE

This environment variable is defined as part of the OpenMP standard. Setting it to "static" causes loop iterations to be assigned to threads in round-robin fashion in the order of the thread number.

For more information, see chapter 4 ("Environment Variables") in the OpenMP 4.5 Specification.

OMP_STACKSIZE

This environment variable is defined as part of the OpenMP standard and controls the size of the stack for threads created by OpenMP.

For more information, see chapter 4 ("Environment Variables") in the OpenMP 4.5 Specification.

OMP_THREAD_LIMIT

This environment variable is defined as part of the OpenMP standard and limits the maximum number of OpenMP threads that can be created.

For more information, see chapter 4 ("Environment Variables") in the OpenMP 4.5 Specification.

]]>
Compilers: AMD Optimizing C/C++ Compiler Suite ]]>
-O Set the optimization level to -O2.

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -O0 Means "no optimization". This level compiles the fastest and generates the most debuggable code.

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -O1 Somewhere between -O0 and -O2.

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -O2 Moderate level of optimization which enables most optimizations. This is the default when no "-O" option is specified, or if no value is specified (i.e. "-O").

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -O3 Like -O2, except that it enables optimizations that take longer to perform or that may generate larger code (in an attempt to make the program run faster).

If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.

]]> -march=znver6 Specify that Clang should generate code for a specific processor family member and later. For example, if you specify -march=znver1, the compiler is allowed to generate instructions that are valid on AMD Zen processors, but which may not exist on earlier products. -march=znver4 enables AVX 512 ISA for Genoa (znver4) processors.

]]> -flto Generate output files in LLVM formats suitable for link time optimization. When used with -S this generates LLVM intermediate language assembly files, otherwise this generates LLVM bitcode format object files (which may be passed to the linker depending on the stage selection options).

]]> -m32 Generates code for a 32-bit environment. The 32-bit environment sets int, long and pointer to 32 bits and generates code that runs on any i386 system. The compiler generates x86 or IA32 32-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target platform specified is 64-bit, otherwise the default is 32-bit.

]]> -m64 Generates code for a 64-bit environment. The 64-bit environment sets int to 32 bits and long and pointer to 64 bits and generates code for AMD's x86-64 architecture. The compiler generates AMD64, INTEL64, x86-64 64-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target platform specified is 64-bit, otherwise the default is 32-bit.

]]> -ffast-math Enables a range of optimizations that provide faster, though sometimes less precise, mathematical operations that may not conform to the IEEE-754 specifications. When this option is specified, the __STDC_IEC_559__ macro is ignored even if set by the system headers.

]]> -ffinite-math-only -ffinite-math-only, which is implied by -ffast-math and -Ofast, allows optimizations for floating-point arithmetic that assume that arguments and results are not NaNs or +-Infs. Setting -fno-finite-math-only does the opposite: the compiler must prepare for the possible presence of NaNs and infinities.

]]> -fassociative-math -fassociative-math allows the compiler to reassociate floating-point expressions. This means the compiler may change the grouping of operations (for example, transforming (a + b) + c into a + (b + c)) in order to enable better optimization. Such transformations may change numerical results due to rounding differences.

This option is implied by -ffast-math and -Ofast. Using -fno-associative-math preserves strict evaluation order and IEEE-compliant rounding behavior.

]]> -freciprocal-math -freciprocal-math allows the compiler to replace floating-point division operations with multiplication by a reciprocal (for example, replacing x / y with x * (1.0 / y)). This can significantly improve performance on some architectures, but may reduce numerical accuracy.

This option is implied by -ffast-math and -Ofast. Using -fno-reciprocal-math forces the compiler to preserve exact division semantics.

]]> -fopenmp Enable handling of OpenMP directives and generate parallel code. The openmp library to be linked can be specified through -fopenmp=library option.

]]> -lm Instructs the compiler to link with system math libraries.

]]> -lamdlibm Instructs the compiler to link with AMD-supported optimized math library.

]]> -lamdalloc amdalloc is AMD's memory allocator based on jemalloc library and is available as a part of AOCC binary package.

]]> -lamdalloc-ext amdalloc-ext is AMD's memory allocation library optimized for performance. You can link the library only on Linux platforms. This library provides optimized implementation of allocation routines malloc(), calloc(), realloc() and free(). It is C99 standard compliant. This library is not thread safe. It should not be used in multi threaded application.

]]> -std=gnu89 Selects the C language dialect.

]]> -std=c++98 Selects the C++ language dialect.

]]> -std=f2008 -std=f2018 -lomp Instructs the compiler to link with the OpenMP runtime libraries.

]]> -lflang Instructs the compiler to link with flang Fortran runtime libraries.

]]> -mllvm -enable-lv-late-costing=false -static -fremap-arrays -fenable-readonly-thp=16M -mllvm -fuse-tile-inner-loop -fapprox-func -fno-math-errno -mllvm -vectorize-gslp=true -mllvm -enable-loop-fusion -mllvm -enable-loop-blocking Enables dead virtual function elimination optimization. Requires -flto=full.

]]> Configures the C++ compilation to use the LLVM standard C++ library (libc++) instead of the default system library (usually libstdc++).

]]> -pthread -fdo-concurrent-to-openmp=host Turns on LLVM's instrumentation based profiling.

]]> Enables loop strip mining optimization. This optimization breaks a large loop into smaller segments or strips to improve temporal and spatial locality.

]]> Uses the profiling files generated from a program compiled with -fprofile-instr-generate to guide optimization decisions.

]]> -Wl,-mllvm -Wl,-inline-threshold=100 Sets the compiler's inlining threshold level to the value passed as the argument. The inline threshold is used in the inliner heuristics to decide which functions should be inlined.

]]> -mllvm -unroll-threshold=100 Sets the limit at which loops will be unrolled. For example, if unroll threshold is set to 100 then only loops with 100 or fewer instructions will be unrolled.

]]> -mllvm -advanced-loop-opt=auto -fveclib=AMDLIBM Use the given vector functions library.

]]> -Wno-unused-command-line-argument Do not warn about unused command line arguments.

]]> -fvisibility=hidden Set the default symbol visibility for all global declarations.

]]> -fno-plt Do not use the Procedure Linkage Table (PLT) for external function calls. Instead, generate code to load the function address directly from the Global Offset Table (GOT) and branch to it. This can improve performance by avoiding the overhead of branching through the PLT stubs.

]]> -fno-PIE Do not generate a position independent executable. Usually this option will be used to compile code that will be linked using the -no-pie linker option.

]]> -no-pie Don't produce a dynamically linked position independent executable (Linker option).

]]> -Wl,-u,fmod Treat the named symbol as undefined before the link begins, so that the archive member defining it is extracted during the initial library scan (Linker option). Used with a static link under link time optimization, where a call the backend generates only during code generation would otherwise be emitted after the linker has finished pulling members out of the static libraries.

]]> -Wl,--defsym=fmod=__ieee754_fmod Define a symbol at link time as an alias for another symbol (Linker option). Used here to supply the public name of a math routine that the 32-bit static C library omits, by pointing it at the IEEE core routine that the same library does provide.

]]> -Wl,-mllvm -Wl,-extra-inliner Schedule inlining optimization after few other optimizations.

]]> Generate instrumented code to collect execution counts.

]]> Use instrumentation data for profile-guided optimization.

]]> -mllvm -enable-lv-late-costing=true This option enables late cost modeling for the loop vectorizer. Late costing allows the vectorizer to make more informed decisions about vectorization profitability by evaluating costs later in the optimization pipeline, potentially improving the quality of vectorization decisions.

]]> -fno-fast-math Disables optimizations that allow the compiler to relax the IEEE 754 standard for floating-point arithmetic. This ensures more predictable and repeatable numerical results by preventing reassociations or simplifications that might lose precision.

]]> -fno-finite-math-only -ffinite-math-only, which is implied by -ffast-math and -Ofast, allows optimizations for floating-point arithmetic that assume that arguments and results are not NaNs or +-Infs. Setting -fno-finite-math-only does the opposite: the compiler must prepare for the possible presence of NaNs and infinities.

]]> -fno-associative-math -fassociative-math allows the compiler to reassociate floating-point expressions. This means the compiler may change the grouping of operations (for example, transforming (a + b) + c into a + (b + c)) in order to enable better optimization. Such transformations may change numerical results due to rounding differences.

This option is implied by -ffast-math and -Ofast. Using -fno-associative-math preserves strict evaluation order and IEEE-compliant rounding behavior.

]]> -fno-reciprocal-math -freciprocal-math allows the compiler to replace floating-point division operations with multiplication by a reciprocal (for example, replacing x / y with x * (1.0 / y)). This can significantly improve performance on some architectures, but may reduce numerical accuracy.

This option is implied by -ffast-math and -Ofast. Using -fno-reciprocal-math forces the compiler to preserve exact division semantics.

]]> Specifies size of off_t data type.

]]> -funsigned-char This option instructs the compiler to treat char type as unsigned.

]]> clang clang is a C compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking. Depending on which high-level mode setting is passed, Clang will stop before doing a full link.

]]> clang++ clang++ C++ compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking. Depending on which high-level mode setting is passed, Clang will stop before doing a full link.

]]> flang flang is a Fortran compiler which encompasses parsing, optimization, code generation, assembly, and linking. Depending on which high-level mode setting is passed, Flang will stop before doing a full link.

]]> -L/path/to/libs Specifies a directory to search for libraries. Use -L to add directories to the search path for library files. Multiple -L options are valid. However, the position of multiple -L options is important relative to -l options supplied.

]]> -I /path/to/include Specifies a directory to search for include files. Use -I to add directories to the search path for include files. Multiple -I options are valid.

]]>