Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• Firmware settings used:
• IMM.ThermalModePolicy=Performance
• Memory.PatrolScrub=Disable
• OperatingModes.ChooseOperatingMode=Maximum Performance
• Power.PlatformControlledType=Maximum Performance
• Processors.C1EnhancedMode=Disable
• Processors.PackageACPIC-StateLimit=ACPI C2
• Processors.ProcessorPerformanceStates=Disable

Storage Notes

• The System Under Test has a 200GB internal SSD and contains the hypervisor OS that is booted.
• The SUT is connected to the external IBM Storwize V7000 controller (via two CN4093 10GB switches) using one port on each quad-port Emulex CN4054R adapter.
• The 12 disks in the V7000 are configured into two 6-disk RAID5 arrays. Each array is allocated to its own storage pool.
• Each storage pool contains 12 logical volumes for VM images, enough for 24 tiles, though only 22 are used.
• Each of the VM logical volumes has 6 partitions, 1 for each of the 6 VM types in a tile.

Virtualization Software Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline hpet=disable clocksource=tsc tsc=reliable nohz=on highres=off cgroup_disable=cpu,cpuacct,memory,blkio,hugetlb rdshell intel_iommu=on iommu=pt pci=realloc nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce nmi_watchdog=0
• The SUT booted a RHEL 7.2 based kernel from Red Hat that contained a fix for Red Hat bug 1329357.
• The following services are configured to be disabled during runlevel initialization:
• abrtd, abrt-oops, atd, avahi-daemon, chronyd, crond, dbus, firewalld, iprdump, iprinit, iprupdate, ksmtuned, libstoragemgmt, lvm2-lvmetad, NetworkManager, polkit, postfix, rhsmcertd, rngd, smartd
• The following items were added in /usr/lib/sysctl.d/specvirt.conf:
• kernel.sched_min_granularity_ns = 10000000
• kernel.sched_wakeup_granularity_ns = 10000000
• kernel.sched_latency_ns = 32000000
• kernel.sched_migration_cost = 250000
• kernel.numa_balancing = 0
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• Network configuration is as follows:
• The Emulex network (elx-lnvgy_dd_nic_15c-10.6.228.32-1_rhel7_32-64) & FCoE (elx-lnvgy_dd_fc_15c-10.6.228.31-1_rhel7_32-64) drivers, were downloaded from the official Emulex support web site and installed.
• 116 virtual functions were used for all the VMs.
• Ethernet devices ens1f0, ens1f1, ens2f0, ens2f1, enp21s0f0, enp21s0f1 are the 6 physical functions that are used by the host to provide physical transport for the virtual functions.
• Ethernet devices enp11s0f0, enp11s0f1 are the 2 physical functions that are used by the host to provide physical transport for FCoE.
• Virtual Functions were started using the following command for each port: echo ## > /sys/bus/pci/devices/0000:xx:00.x/sriov_numvfs
• ## represents the number of virtual functions on the xx:00.x port pci address.
• 5 ports each had 21 VFs, 1 port had 11 VFs.
• VM configurations are as follows:
• All VMs are managed with libvirt and created using the virsh utility.
• All VMs are created with the following XML template (exceptions noted below template):
  • <domain type='kvm'>
  •   <name>$vm_name</name>
  •   <memory>$vm_memory</memory>
  •   <currentMemory>$vm_memory</currentMemory>
  •   <vcpu cpuset='$host-cpu-list'>$vm_vcpus</vcpu>
  •   <os>
  •    <type arch='x86_64' machine='pc'>hvm</type>
  •    <boot dev='hd'/>
  •   </os>
  •   <features>
  •    <acpi/>
  •    <apic/>
  •    <pae/>
  •   </features>
  •   <cpu match='exact'>
  •    <model>SandyBridge</model>
  •    <feature policy='force' name='x2apic'/>
  •   </cpu>
  •   <clock offset='utc'>
  •    <timer name='pit' tickpolicy='delay'/>
  •   </clock>
  •   <on_poweroff>destroy</on_poweroff>
  •   <on_reboot>restart</on_reboot>
  •   <on_crash>restart</on_crash>
  •   <devices>
  •    <emulator>/usr/libexec/qemu-kvm</emulator>
  •    <disk type='block' device='disk'>
  •     <driver name='qemu' type='raw' cache='none' io='native'/>
  •     <source dev='$vm_disk'/>
  •     <target dev='vda' bus='virtio'/>
  •    </disk>
  •    <hostdev mode='subsystem' type='pci' managed='no'>
  •     <source>
  •      <address domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
  •     </source>
  •    </hostdev>
  •    <serial type='pty'>
  •     <target port='0'/>
  •    </serial>
  •    <console type='pty'>
  •     <target type='serial' port='0'/>
  •    </console>
  •    <input type='mouse' bus='ps2'/>
  •    <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
  •    <memballoon model='none'/>
  •   </devices>
  • </domain>
• VMs are bound to host CPUs via their XML configuration in the following manner:
• All VMs of tiles 1-4, 9-12, 18, 20, and 21 are bound to NUMA node 0.
• All VMs of tiles 5-8, 13-16, 17, 19, and 22 are bound to NUMA node 1.
• App, Db, & Web VMs are reniced -19.
• Infra, Mail, and Batch VMs are reniced +19.

Web Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio
• The following services are disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, mcelogd, mdmonitor, messagebus, netconsole, nfs, nfslock, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• The rhel distro (4.6.62.0r) be2net driver was used.
• root file system options: barrier=0,noatime
• /etc/rc.local was modified to include:
• echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• ethtool -C eth0 adaptive-rx off rx-usecs 128
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_tw_recycle = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• The php-fpm package was installed. /etc/php-fpm.d/www.conf changes:
• user = nginx
• group = nginx
• /etc/nginx/nginx.conf changes:
• worker_processes 4;
• worker_cpu_affinity 00000001 00000001 00000010 00000010;
• worker_rlimit_nofile 65535;
• #error_log /var/log/nginx/error.log warn;
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• # log_format main '$remote_addr - $remote_user [$time_local] "$request" '
• # '$status $body_bytes_sent "$http_referer" '
• # '"$http_user_agent" "$http_x_forwarded_for"';
• log_format common '$remote_addr - $remote_user [$time_local] $request'
• '"$status" $body_bytes_sent ';
• access_log /var/log/nginx/access.log common gzip buffer=4m flush=1m;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• open_file_cache_errors on;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• /etc/nginx/conf.d/default.conf changes:
• listen 80 backlog=81920;
•   root /var/www/html;
•   index index.html index.htm index.php;
• location = /error.log {
•   root /var/log/nginx;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
•   root /var/www/html;
•   expires 100d;
• }
• location ~ ^/images/ {
•   root /var/www/html/support;
•   expires 100d;
• }
• location ~ .php$ {
•   root html;
•   fastcgi_pass 127.0.0.1:9000;
•   fastcgi_index index.php;
•   fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
•   include fastcgi_params;
• }
• /etc/nginx/conf.d/example_ssl.conf changes:
• listen 443 ssl;
• ssl on;
• ssl_certificate /etc/pki/tls/certs/localhost.crt;
• ssl_certificate_key /etc/pki/tls/private/localhost.key;
• ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
• ssl_ciphers ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW;
• ssl_prefer_server_ciphers on;
• location / {
•   root /var/www/html;
•   index index.html index.htm index.php;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
•   root /var/www/html;
•   expires 100d;
• }
• location ~ ^/images/ {
•   root /var/www/html/support;
•   expires 100d;
• }
• location ~ .php$ {
•   root html;
•   fastcgi_pass 127.0.0.1:9000;
•   fastcgi_index index.php;
•   fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
•   include fastcgi_params;
• }
• Web VM used NFS to access files from infra VM.
• After the Infra VM is booted and its NFS service is up, the Web VM mounts the NFS file system via test automation software.
• The datastore is mounted on /var/log/httpd

Application Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio
• The following services are disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, ip6tables, iptables, irqbalance, kdump, mcelogd, mdmonitor, messagebus, netconsole, nfs, nfslock, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• The rhel distro (4.6.62.0r) be2net driver was used.
• root file system options: barrier=0
• /etc/rc.local was modified to include:
• echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• ethtool -C eth0 adaptive-rx off rx-usecs 400
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator.
• Heap size for AppSrv01 has been changed to 2048 MB.
• The following changes were made to the defaults in run.properties:
• runLO=4
• warmUpStyle = 1
• linearWarmUpPercentage = .60

Mail Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio
• The following services are disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, ip6tables, iptables, irqbalance, kdump, mcelogd, mdmonitor, messagebus, netconsole, nfs, nfslock, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• The rhel distro (4.6.62.0r) be2net driver was used.
• root file system options: barrier=0,noatime,data=writeback
• /etc/rc.local was modified to include:
• echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• ethtool -C eth0 adaptive-rx off rx-usecs 1000
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• Dovecot was compiled with default options, and the following dovecot.conf was used:
• base_dir = /var/run/dovecot/
• ssl = no
• ssl_cert =
• ssl_key =
• disable_plaintext_auth = no
• listen = *
• log_path = /var/log/dovecot.log
• mail_debug = no
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 8192
• maildir_very_dirty_syncs = yes
• mbox_very_dirty_syncs = yes
• mbox_write_locks = fcntl
• mmap_disable = yes
• mail_fsync = never
• passdb {
•   args = scheme=plain /etc/passwd.dovecot
•   driver = passwd-file
• }
• protocols = imap
• service auth {
•   user = root
• }
• service imap-login {
•   inet_listener imap {
•     port = 143
•   }
•   process_min_avail = 10
•   service_count = 0
• }
• service imap {
•   process_limit = 1024
• }
• userdb {
•   driver = passwd
• }
• dict {
• }

Database Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio
• The following services are disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, mcelogd, mdmonitor, messagebus, netconsole, nfs, nfslock, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• The rhel distro (4.6.62.0r) be2net driver was used.
• root file system options: noatime,barrier=0,data=writeback
• /etc/rc.local was modified to include:
• echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• ethtool -C eth0 adaptive-rx off rx-usecs 666
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• kernel.randomize_va_space=0
• kernel.sem = 500 64000 400 512
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• 13312 2MB huge-pages were allocated prior to starting the database instance.
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 6553600
• db2set DB2_LARGE_PAGE_MEM=DB
• db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5898240
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING
• alter tablespace C_parts_space NO FILE SYSTEM CACHING
• alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• alter tablespace M_parts_space NO FILE SYSTEM CACHING
• alter tablespace M_bom_space NO FILE SYSTEM CACHING
• alter tablespace M_wo_space NO FILE SYSTEM CACHING
• alter tablespace M_lo_space NO FILE SYSTEM CACHING
• alter tablespace M_inv_space NO FILE SYSTEM CACHING
• alter tablespace O_cust_space NO FILE SYSTEM CACHING
• alter tablespace O_ords_space NO FILE SYSTEM CACHING
• alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• alter tablespace O_item_space NO FILE SYSTEM CACHING
• alter tablespace S_comp_space NO FILE SYSTEM CACHING
• alter tablespace S_sc_space NO FILE SYSTEM CACHING
• alter tablespace S_po_space NO FILE SYSTEM CACHING
• alter tablespace S_site_space NO FILE SYSTEM CACHING
• alter tablespace S_supp_space NO FILE SYSTEM CACHING
• alter tablespace U_space NO FILE SYSTEM CACHING
• update database config for specdb using LOGFILSIZ 65536
• update db cfg for specdb using logprimary 64 logsecond 4

Infraserver VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio
• The following services are disabled:
• abrt-ccpp, abrtd, atd, auditd, autofs, certmonger, crond, cups, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, mdmonitor, messagebus, netconsole, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, winbind, ypbind
• The rhel distro (4.6.62.0r) be2net driver was used.
• root file system options: noatime,barrier=0,data=writeback
• /etc/rc.local was modified to include:
• echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• ethtool -C eth0 adaptive-rx off rx-usecs 300
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness=0
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• httpd.conf changes:
• ServerLimit 10240
• MaxClients 10240
• Listen 81
• fcgi package was built from Fedora 14 source RPM, which includes patch to compile with GCC 4.4.
• The following items were added to/etc/httpd/conf/fcgi.conf:
• LoadModule fastcgi_module modules/mod_fastcgi.so
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgiIpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin">
•   AllowOverride None
•   Options +ExecCGI -Includes
•   SetHandler fastcgi-script
•   Order allow,deny
•   Allow from all
• </Directory>
• AddHandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• Standard Linux NFS server is enabled to serve files to Web VM.
• /etc/exports file contains:
• /support-download *(rw,async,no_subtree_check,crossmnt,no_root_squash)

Batch Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio
• The following services are disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, certmonger, cgconfig, cgred, crond, cups, haldaemon, ip6tables, iptables, irqbalance, kdump, mcelogd, mdmonitor, messagebus, netconsole, nfs, nfslock, numad, oddjobd, portreserve, postfix, psacct, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, sssd, svnserve, winbind, ypbind
• The rhel distro (4.6.62.0r) be2net driver was used.
• root file system options: barrier=0
• /etc/rc.local was modified to include:
• echo "tsc" >/sys/devices/system/clocksource/clocksource0/current_clocksource
• ethtool -C eth0 adaptive-rx off rx-usecs 500
• The tuned-0.2.19 package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000

Client Driver Notes

• Two physical client systems (System x3550 M5) host six prime client VMs using Red Hat Enterprise Linux 6.6; the other two client systems host 5 prme client VMs. One client system (IBM System x3550 M2) running Red Hat Enterprise Linux 6.6 is the prime controller.
• Each prime client VM has 5 VCPUs, 7936 MB memory, and a 16 GB disk. 1 virtio_net device and 1 virtio_blk device are used in all client VMs.

Other Notes

• 4 x Flex System Fabric CN4093 10Gb Converged Scalable Switches are installed in switch bays 1,2,3 and 4 of the Flex System Enterprise Chassis.
• The system under test (SUT) is a blade system that plugs into the Flex System Enterprise Chassis. The NICs in the SUT connect to the switches via the backplanes within the chassis.
• The external client systems are connected to the switches using 10Gb direct connect cables.
• The IBM Storwize V7000 controller is connected to the switches using 10GB fiber cables and running FCoE.

Performance Details: