SPEC CPU2017 Platform Settings for Lenovo Systems

Operating System Tuning Parameters

sched_cfs_bandwidth_slice_us

This OS setting controls the amount of run-time(bandwidth) transferred to a run queue from the task's control group bandwidth pool. Small values allow the global bandwidth to be shared in a fine-grained manner among tasks, larger values reduce transfer overhead. The default value is 5000 (ns).

sched_latency_ns

This OS setting configures targeted preemption latency for CPU bound tasks. The default value is 24000000 (ns).

sched_migration_cost_ns

Amount of time after the last execution that a task is considered to be "cache hot" in migration decisions. A "hot" task is less likely to be migrated to another CPU, so increasing this variable reduces task migrations. The default value is 500000 (ns).

sched_min_granularity_ns

This OS setting controls the minimal preemption granularity for CPU bound tasks. As the number of runnable tasks increases, CFS(Complete Fair Scheduler), the scheduler of the Linux kernel, decreases the timeslices of tasks. If the number of runnable tasks exceeds sched_latency_ns/sched_min_granularity_ns, the timeslice becomes number_of_running_tasks * sched_min_granularity_ns. The default value is 8000000 (ns).

sched_wakeup_granularity_ns

This OS setting controls the wake-up preemption granularity. Increasing this variable reduces wake-up preemption, reducing disturbance of compute bound tasks. Lowering it improves wake-up latency and throughput for latency critical tasks, particularly when a short duty cycle load component must compete with CPU bound components. The default value is 10000000 (ns).

numa_balancing

This OS setting controls automatic NUMA balancing on memory mapping and process placement. NUMA balancing incurs overhead for no benefit on workloads that are already bound to NUMA nodes. Possible settings:

0: disables this feature
1: enables the feature (this is the default)

For more information see the numa_balancing entry in the Linux sysctl documentation.

Transparent Hugepages (THP)

THP is an abstraction layer that automates most aspects of creating, managing, and using huge pages. It is designed to hide much of the complexity in using huge pages from system administrators and developers. Huge pages increase the memory page size from 4 kilobytes to 2 megabytes. This provides significant performance advantages on systems with highly contended resources and large memory workloads. If memory utilization is too high or memory is badly fragmented which prevents hugepages being allocated, the kernel will assign smaller 4k pages instead. Most recent Linux OS releases have THP enabled by default.
THP usage is controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/enabled. Possible values:

never: entirely disable THP usage.
madvise: enable THP usage only inside regions marked MADV_HUGEPAGE using madvise(3).
always: enable THP usage system-wide. This is the default.

THP creation is controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/defrag. Possible values:

never: if no THP are available to satisfy a request, do not attempt to make any.
defer: an allocation requesting THP when none are available get normal pages while requesting THP creation in the background.
defer+madvise: acts like "always", but only for allocations in regions marked MADV_HUGEPAGE using madvise(3); for all other regions it's like "defer".
madvise: acts like "always", but only for allocations in regions marked MADV_HUGEPAGE using madvise(3). This is the default.
always: an allocation requesting THP when none are available will stall until some are made.

An application that "always" requests THP often can benefit from waiting for an allocation until those huge pages can be assembled.
For more information see the Linux transparent hugepage documentation.

cpupower

The OS 'cpupower' utility is used to change CPU power governors settings. Available settings are:

Performance: Run the CPU at the maximum frequency.
powersave: Run the CPU at the minimum frequency.
ondemand: Scales the frequency dynamically according to current load.

tuned-adm

The tuned-adm tool is a commandline interface for switching between different tuning profiles available to the tuned tuning daemon available in supported Linux distros. The default configuration file is located in /etc/tuned.conf and the supported profiles can be found in /etc/tune-profiles. Some profiles that may be available by default include: default, desktop-powersave, server-powersave, laptop-ac-powersave, laptop-battery-powersave, spindown-disk, throughput-performance, latency-performance, enterprise-storage. To set a profile, one can issue the command "tuned-adm profile (profile_name)". Here are details about relevant profiles:

throughput-performance: Server profile for typical throughput tuning. This profile disables tuned and ktune power saving features, enables sysctl settings that may improve disk and network IO throughput performance, switches to the deadline scheduler, and sets the CPU governor to performance.
latency-performance: Server profile for typical latency tuning. This profile disables tuned and ktune power saving features, enables the deadline IO scheduler, and sets the CPU governor to performance.
enterprise-storage: Server profile to high disk throughput tuning. This profile disables tuned and ktune power saving features, enables the deadline IO scheduler, enables hugepages and disables disk barriers, increases disk readahead values, and sets the CPU governor to performance

dirty_background_ratio

Set through "echo 40 > /proc/sys/vm/dirty_background_ratio". This setting can help Linux disk caching and performance by setting the percentage of system memory that can be filled with dirty pages.

dirty_ratio

Set through "echo 8 > /proc/sys/vm/dirty_ratio". This setting is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk.

ksm/sleep_millisecs

Set through "echo 200 > /sys/kernel/mm/ksm/sleep_millisecs". This setting controls how many milliseconds the ksmd (KSM daemon) should sleep before the next scan.

swappiness

The swappiness value can range from 1 to 100. A value of 100 will cause the kernel to swap out inactive processes frequently in favor of file system performance, resulting in large disk cache sizes. A value of 1 tells the kernel to only swap processes to disk if absolutely necessary. This can be set through a command like "echo 1 > /proc/sys/vm/swappiness"

Zone Reclaim Mode

Zone reclaim allows the reclaiming of pages from a zone if the number of free pages falls below a watermark even if other zones still have enough pages available. Reclaiming a page can be more beneficial than taking the performance penalties that are associated with allocating a page on a remote zone, especially for NUMA machines. To tell the kernel to free local node memory rather than grabbing free memory from remote nodes, use a command like "echo 1 > /proc/sys/vm/zone_reclaim_mode"

Free the file system page cache

The command "echo 3> /proc/sys/vm/drop_caches" is used to free pagecache, dentries and inodes.

Firmware / BIOS / Microcode Settings

Workload Profile (Default="General Computing - Power Efficiency")

Select the profile based on your preference. The selected workload profile will automatically change lower-level settings per the profile selected and will not allow them to be changed individually.

"General Computing - Power Efficiency" "Power Efficiency" profiles are comprised of settings comparable to Intel's Optimized Power Mode (OPM).
"General Computing - Peak Frequency" This workload profile is defined by the requirement to drive the highest core frequencies out of a processor accross a subset of cores available - not for all cores active. This workload profile benefits workloads requiring either high per core and / or overall CPU package frequency.
"General Computing - Max Performance"This workload profile maximizes the absolute performance of the system without regard for power savings.
"High Performance Computing" This profile is for customer running large-scale scientific and engineering workloads. These environments tend to be clustered environments where where each node performs at maximum utilization for extened periods of time, and the application is Non-uniform Memory Access (NUMA) aware.
"Custom", may be selected after any of the other presets allowing low-level settings, which otherwise are preset and unchangeable, to be individually modified

Page Policy:

Adaptive Open Page Policy can improve performance for applications with a highly localized memory access pattern; Closed Page Policy can benifit applications that access memory more randomly. Default is Closed.

CPU P-state Control:

Select the method used to control CPU P-states (performance states). "None" disables all P-states and the CPUs run at either their rated frequency or in turbo mode (if turbo is enabled). When [Legacy] is selected, the CPU P-states will be presented to the operating system (OS) and the OS power management (OSPM) will directly control which P-state is selected. With [Autonomous], the P-states are controlled fully by system hardware. No P-state support is required in the OS or VM. [Cooperative] is a combination of Legacy and Autonomous. The P-states are still controlled in hardware but the OS can provide hints to the hardware for P-state limits and the desired setting. With [Cooperative without Legacy], uses Intel native hardware p-states without Legacy p-state control (No OS controlled p-states). Starts with Autonomous mode until the OS swtiches to Cooperative. With [Cooperative with Legacy], uses Intel native hardware p-states but Legacy p-state control. Starts with Legacy p-states until the OS swtiches to Cooperative. When a preset mode is selected, the low-level settings are not changeable and will be grayed out. If user would like to change the settings, please choose [Custom] in "Workload Profile" located under "System Setting" submenu. Default is Autonomous.

Memory Speed:

Select the desired memory speed. [Maximum Performance] mode maximizes performance. [Balanced] mode offers a balance between performance and power. [Minimal power] mode maximizes power savings. When a preset mode is selected, the low-level settings are not changeable and will be grayed out. If user would like to change the settings, please choose [Custom] in "Workload Profile" located under "System Setting" submenu. Default is Maximum Performance.

C-States:

C-states reduce CPU idle power. There are two options in this mode: Legacy, Disabled. Default is Legacy.

Legacy: When [Legacy] is selected, the operating system initiates the C-state transitions. For E5/E7 CPUs, ACPI C1/C2/C3 map to Intel C1/C3/C6. For 6500/7500 CPUs, ACPI C1/C3 map to Intel C1/C3 (ACPI C2 is not available). Some OS SW may defeat the ACPI mapping (e.g. intel_idle driver).
Disabled: When "Disabled" is selected, only C0 and C1 are used by the OS. C1 gets enabled automatically when an OS autohalts.

When a preset mode is selected, the low-level settings are not changeable and will be grayed out. If user would like to change the settings, please choose [Custom] in "Workload Profile" located under "System Setting" submenu.

C1 Enhanced Mode:

Enabling C1E (C1 enhanced) state can save power by halting CPU cores that are idle. Default is Enabled.

Turbo Mode:

Enabling turbo mode can boost the overall CPU performance when all CPU cores are not being fully utilized. A CPU core can run above its rated frequency for a short perios of time when it is in turbo mode. When a preset mode is selected, the low-level settings are not changeable and will grayed out. If user would like to change the settings, please choose [Custom] in "Workload Profile" located under "System Settings" submenu. Default is Enabled.

Hyper-Threading:

Enabling Hyper Threading let operating system addresses two virtual or logical cores for a physical presented core. Workloads can be shared between virtual or logical cores when possible. The main function of hyper-threading is to increase the number of independent instructions in the pipeline for using the processor resources more efficiently. Default is Enabled.

DCA:

DCA capable I/O devices such as network controllers can place data directly into the CPU cache, which improves response time. Default is Enabled.

Power/Performance Bias:

Power/Performance bias determines how aggressively the CPU will be power managed and placed into turbo. With [Platform Controlled], the system controls the setting. Selecting [OS Controlled] allows the operating system to control it. Not all OSes support this feature. Default is Platform Controlled.

Platform Controlled Type:

Controls how aggressively the processor's Power Control Unit (PCU) will engage power management and how the CPU cores are placed into turbo mode. Default is [Efficiency-Favor Performance].

Maximum Performance: Turbo mode can be engaged opportunistically before it is requested and uncore power management features (Memory, UPI, C-state demotion, I/O bandwidth limit and UFS) are aggressively disabled.
Efficiency-Favor Performance:Turbo is quickly engaged but not opportunistically. Some light uncore and UPI power management features are engaged to provide low cost power savings with an overall weighting towards per core and per package performance.
Efficiency-Favor Power:Turbo is engaged more slowly but uncore and UPI power management policies are quickly engaged to balance the systems towards power savings.
Minimal Power:Turbo is not engaged and aggressive power management policies on the uncore and UPI are engaged to drive lowest performance / watt.

CPU Frequency Limits:

When the CPU Frequency Limit parameter is set to the Restrict Maximum Frequency setting, the maximum frequency (turbo, AVX, and non turbo) can be restricted to a frequency that is between the maximum turbo frequency for the CPU installed and 1.2GHz. This can be useful for synchronizing CPU task. Note, the max frequency for N+1 cores cannot be higher than N cores. If an illegal frequency is entered, it will automatically be limited to a legal value. If the CPU frequency limits are being controlled through application software, leave this menu item at the default ([Full turbo uplift]), please choose [Custom] in "Workload Profile" and [Enable] in "Turbo Mode" located under "System Setting" submenu. Default is Full turbo uplift.

Uncore Frequency Scaling:

When enabled, the CPU uncore will dynamically change speed based on the workload. All miscellaneous logic inside the CPU package is considered the uncore. Default is Enabled.

MONITOR/MWAIT:

MONITOR/MWAIT instructions are used to engage C-states. Some operating systems will re-enable C-states even when they are disabled in setup. To prevent this, disable MONITOR/MWAIT. Please chkoose [Custom] in "Workload Profile" and [Disabled] in "C-States" located under "System Setting" submenu. Default is Enabled.

Sub-NUMA Cluster (SNC):

Sub_NUMA Cluster (SNC) partitions the cores and last level cache (LLC) into clusters with each cluster bound to a set of memory controllers in the system, dividing each CPU package into 2 or 4 NUMA nodes. This can improve average latency to the last level cache and memory. Default is Disabled.

Disabled:the processor socket is treated as one cluster. No partitioning occurs.
SNC2:each socket is divided into two groups, each with its own cores, LLC, and memory controller. System address is divided into two regions, one for each SNC. Required memory configuration to be left/right symmetric.
SNC4:each socket is divided into four groups, each with its own cores, LLC, and memory controller. System address is divided into four regions, one for each SNC. Required memory configuration to be fully symmetric (left/right and top/bottom).

XPT Prefetcher

When enabled, enables a read request that is sent to the processor last level cache to speculatively issue a copy of that read request to the processor memory controller prefetcher to reduce latency. The performance benefit can vary so the end user must determine whether or not enabling the XPT prefetcher benefits their specific application environment. Default is Enabled.

UPI Prefetcher

When enabled, memory reads on the memory bus can be started earlier by the processor memory controller to reduce latency. The performance benefit can vary so the end user must determine whether or not enabling the UPI prefetcher benefits their specific application environment. Default is Enabled.

Patrol Scrub:

Enable/Disable "Patrol Scrub" which proactively searches the system memory to repair correctable errors. when [Enabled] is selected, Patrol Scrub would take effect at the end of POST. Default is Enabled.

DCU Streamer Prefetcher:

DCU (Level 1 Data Cache) streamer prefetcher is an L1 data cache prefetcher. Lightly threaded applications and some benchmarks can benefit from having the DCU streamer prefetcher enabled. Default setting is Enabled.

Stale AtoS

The in-memory directory has three states: invalid (I), snoopAll (A), and shared (S). Invalid (I) state means the data is clean and does not exist in any other socket`s cache. The snoopAll (A) state means the data may exist in another socket in exclusive or modified state. Shared (S) state means the data is clean and may be shared across one or more socket`s caches. When doing a read to memory, if the directory line is in the A state we must snoop all the other sockets because another socket may have the line in modified state. If this is the case, the snoop will return the modified data. However, it may be the case that a line is read in A state and all the snoops come back a miss. This can happen if another socket read the line earlier and then silently dropped it from its cache without modifying it. Values for this BIOS option can be:

Disable: Disabling this option allows the feature to process memory directory states as described above.
Enable: In the situation where a line in A state returns only snoop misses, the line will transition to S state. That way, subsequent reads to the line will encounter it in S state and not have to snoop, saving latency and snoop bandwidth.

Stale AtoS may be beneficial in a workload where there are many cross-socket reads. Default is Auto.

LLC dead line alloc

In some Intel CPU caching schemes, mid-level cache (MLC) evictions are filled into the last level cache (LLC). If a line is evicted from the MLC to the LLC, the core can flag the evicted MLC lines as "dead." This means that the lines are not likely to be read again. This option allows dead lines to be dropped and never fill the LLC if the option is disabled. Values for this BIOS option can be:

Disable: Disabling this option can save space in the LLC by never filling MLC dead lines into the LLC.
Enable: Opportunistically fill MLC dead lines in LLC, if space is available.

Default is Enable.

Adjacent Cache Prefetch:

When enabled, fetches both cache lines that make up a 128 byte cache line pair even if the requested data is only in the first cache line. Lightly threaded applications and some benchmarks can benefit from having the adjcent cache line prefetch enabled. Default is Enabled.

DCU IP Prefetcher:

A processor L1 instruction cache prefetcher that uses a predictive instruction prefetching scheme. This setting fetches the next cache line into the L1 data cache if there is a sequential load history of cache line accesses. The next cache line is fetched from either L2 or main memory.

Enabled: When enabled, this parameter fetches the next cache line into the L1 data cache if there is a sequential load history of cache line accesses.
Disabled: Experiments have been run selectively disabling the prefetchers one at a time and performance is improved when disabled.

Default is Enabled.

Workload Configuration:

I/O sensitive should be used with expansion cards that require high I/O bandwidth when the CPU cores are idle to allow enough frequency for the workload. Default is Balanced.

Memory Power Management:

Allows the platform to put the memory into a lower power consumption state. Performance may be reduced. [Disabled] provides maximum performance but minimum power savings. [Automatic] is suitable for most applications. When a preset mode is selected, the low-level settings are not changeable and will grayed out. If user would like to change the settings, please choose [Custom] in "Workload Profile" located under "System Settings" submenu. Default is Disabled.

UPI Link Disable:

Limiting the QPI/UPI connections to minimum number can save power. Note, for some highly NUMA optimized workoads, maximum performance may be achieved by setting 'Minimum Number of Links Enabled'. Default is Enabled ALL Links.

UPI Link Frequency:

Select the desired CPU UPI link frequency. Default is Maximum Performance.

Maximum Performance: maximizes performance.
Balanced: offers a balance between performance and power.
Minimal Power: maximizes power savings.

LLC Prefetch:

LLC prefetcher is an additional prefetch mechanism on the top of the existing prefetchers that prefetch data into the core DCU amd MLC. Enabling LLC Prefetch gives the core prefetcher the ability to prefetch data directly into the LLC without necessarily filling into the MLC. Default is Enabled.

AMP Prefetch:

The Adaptive Multipath Probability (AMP) prefetch is a new data prefetch mechanism for the Golden Cove architecture. The AMP prefetch operates alongside the pre-existing L2 prefetch mechanisms: the L2 streamer and the L2 spatial prefetchers. Set to enable or disable MLC AMP prefetch (MSR 1A4h [4]). Some benchmarks can benefit from having this Mid-Level-Cache (MLC) AMP prefetch enabled. Default is Disable.

Optimized Power Mode:

The feature can save power while having minimal impact on performance for certain workloads that tend to run 30-40% CPU utilization.

Enable: Enable Fast C1E, fix the uncore max frequency to 2.2GHz, set the active idle uncore frequency to 1.4GHz, and set the utilization point (UP) = 6.
Disable (default): Disable Fast C1E, set the uncore max frequency to 2.5GHz, set the active idle uncore frequency to default, set the ubilization point (UP) = 0.

Package C State:

This feature selects the processor's lowest idle package power state (C-state) that is enabled. The processor will automatically transition into the package C-states based on the Core C-states, in which cores on the processor have transitioned. The higher the package C-state, the lower the power usage of that idle package state. Package C6 (retention) is the lowest power idle package state supported by the processor). Values for this setting can be:

C6(Retention) state (default): All cores have saved their architectural state and have had their core voltages reduced to zero volts. The LLC retains context, but no accesses can be made to the LLC in this state, the cores must break out to the internal state package C2 for snoops to occur.
C6(non Retention) state: All cores have saved their architectural state and have had their core voltages reduced to zero volts. The LLC does not retain context, and no accesses can be made to the LLC in this state, the cores must break out to the internal state package C2 for snoops to occur.
No Limit: All cores are in an active state and have not entered any power saving state.

Memory Hierarchy:

In a system with High Bandwidth Memory and regular DDR memory,

Flat: Flat mode means both High Bandwidth Memory and DDR memory are used as contiguous address space.
Cache: Cache mode means acts as a buffer for the DDR memory.

If Cache mode configuration fails, or system only has High Bandwidth Memory, then the system will fail back to Flat mode. Default is Flat.

HBM Refresh Mode:

Disabled: Disable refresh mode, same as Single.
Auto (default): Support all refresh mode, could save power, lower system noise level and is useful to mitigate rowhammer issue, but that might have a performance impact.
Single: Force single refresh.
Double: Force double refresh, could save power, lower system noise level and is useful to mitigate rowhammer issue, but that might have a performance impact.

SNC2 is not supported on High Bandwidth Memory (HBM) processors i.e., Intel Xeon CPU Max Series SKU Stack.