Performance Summary:

Power Summary:

Power Details:

Validation Errors:

•   No Validation Errors Found

Power Measurement Configuration:

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• Firmware settings used:
• OperatingModes.ChooseOperatingMode=Maximum Performance

Storage Notes

• The System Under Test used 8 x 3.84 TB SSDs. Detailed configuration is as follows:
• Create 2 x RAID10 arrays, each with 4 x SSD drives
• Create 19 x logical disks for one RAID array, create 18 x logical disks for the other RAID array
• The first logical disk has the OS installed. The other logical disks were partitioned for the VMs of 1 tile

Virtualization Software Notes

• Boot parameters set: splash=silent audit=0 elevator=deadline hpet=disable clocksource=tsc tsc=reliable nohz=on highres=off cgroup_disable=cpu,cpuacct,memory,blkio,hugetlb intel_iommu=on iommu=pt pci=realloc pci=pcie_bus_safe nosoftlockup ipv6.disable=1 intel_idle.max_cstate=1 processor.max_cstate=1 mce=ignore_ce nmi_watchdog=0 quiet showopts
• The following services were disabled:
• atd, auditd, autofs, autoyast-initscripts, blk-availability, bmc-snmp-proxy, brltty, console-getty, console-shell, cron, debug-shell, dm-event, dmraid-activation, dnsmasq, exchange-bmc-os-info, gpm, grub2-once, ipmi, ipmievd, irqbalance, iscsid, iscsiuio, kdump-rebuild-initrd, kdump, kexec-load, klog, libvirt-guests, lunmask, lvm2-lvmetad, lvm2-monitor, multipathd, nfs-blkmap, nfs-server, nfs, nfsserver, ntp-wait, ntpd, numad, openct, postfix, radvd, rpcbind, rsyncd, rtkit-daemon, saslauthd, serial-getty@, slpd, SuSEfirewall2, SuSEfirewall2_init, sysstat, systemd-bootchart, systemd-journal-upload, systemd-nspawn@, systemd-timesyncd, vsftpd, wickedd-auto4, wickedd-dhcp4, wickedd-dhcp6, wickedd-nanny, winbind, xinetd, ypbind
• /usr/lib/sysctl.d/specvirt.conf was updated with the following lines:
• fs.aio-max-nr = 400000
• kernel.perf_event_max_sample_rate = 1000
• net.ipv4.tcp_mem = 24718578 32958104 49437156
• net.ipv4.udp_mem = 49437156 65916208 98874312
• kernel.sched_min_granularity_ns = 10000000
• kernel.sched_wakeup_granularity_ns = 15000000
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• kernel.sched_latency_ns = 20000000
• kernel.sched_migration_cost_ns = 150000
• kernel.numa_balancing = 0
• Network configuration is as follows:
• One BCM5719 NIC port(eth0) was used for host access
• The 4 x X710 SFP+ ports were assigned to bridges (br0,br1,br2,br3)
• Add scripts for i40evf settings in /etc/rc.d/boot.local as follows,
• echo 64 > /sys/class/net/eth4/device/sriov_numvfs
• echo 64 > /sys/class/net/eth5/device/sriov_numvfs
• echo 64 > /sys/class/net/eth6/device/sriov_numvfs
• echo 64 > /sys/class/net/eth7/device/sriov_numvfs
• Set blacklist in /etc/modprobe.d/blacklist.conf as follow,
• blacklist ipmi_si ipmi_msghandler ipmi_devintf
• blacklist i40evf
• All VMs were assigned with a VF
• VM configurations were as follows:
• All VMs were managed with libvirt
• All VMs were created with the following XML template:
• <domain type='kvm'>
• <name>$vmname</name>
• <memory>$vmem</memory>
• <currentMemory>$vmem</currentMemory>
• <vcpu placement='static' cpuset='$host_cpu_list'>$vcpus</vcpu>
• <os>
• <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <acpi/>
• <apic/>
• <pae/>
• </features>
• <cpu mode='custom' match='exact'>
• <model>SandyBridge</model>
• <vendor>Intel/>vendor>
• <topology sockets='1' cores='$vcpus' threads='1'/>
• </cpu>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>restart</on_crash>
• <devices>
• <emulator>/usr/bin/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vdisk'/>
• <target dev='vda' bus='virtio'/>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </disk>
• <interface type='hostdev' managed='yes'/>
• <mac address='$vmac'/>
• <source>
• <address type='pci' domain='0x0000' bus='0x$bus' slot='0x$slot' function='0x$func'/>
• </source>
• </interface>
• <serial type='pty'>
• <target port='0'/>
• </serial>
• <console type='pty'>
• <target type='serial' port='0'/>
• </console>
• <memballoon model='none'/>
• </devices>
• </domain>
• VMs' nice value was set after startup via the renice command to the following settings:
• appservers = -3
• dbserver = -15
• webserver = 18
• infraserver = 19
• mailserver = 18
• batchserver = -19
• All VMs of tiles 1-16, 33 and 36 were bound to NUMA node 0
• All VMs of tiles 17-32, 34 and 35 were bound to NUMA node 1

Web Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• Add ethtool -C eth0 adaptive-tx off adaptive-rx off rx-usecs 1500 in /etc/rc.local
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, php-fpm, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind, cpuspeed, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• Contents of nginx configuration file /etc/nginx/nginx.conf
• user nginx;
• worker_processes 4;
• worker_cpu_affinity 00000001 00000001 00000010 00000010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request"$status" $body_bytes_sent ';
• access_log /var/log/access.log common gzip buffer=4m flush=1m;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• }
• Contents of nginx configuration file /etc/nginx/conf.d/default.conf
• server {
• listen 80 backlog=8192;
• server_name localhost;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ \.php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Contents of nginx configuration file /etc/nginx/conf.d/example_ssl.conf
• server {
• listen 443 ssl;
• server_name localhost;
• ssl on;
• ssl_certificate /etc/pki/tls/certs/localhost.crt;
• ssl_certificate_key /etc/pki/tls/private/localhost.key;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
• ssl_ciphers ALL:!kEDH!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ \.php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Changes made to /etc/php-fpm.d/www.conf:
• listen = /dev/shm/php-cgi.sock
• listen.owner = nginx
• listen.group = nginx
• user = nginx
• group = nginx
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 800000
• * hard nofile 800000
• * soft nproc 102400
• * hard nproc 102400

Application Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, crond, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, mdmonitor, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind, cpuspeed, lv2-monitor, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• root file-system options: barrier=0
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator
• Heap size for AppSrv01 has been changed to 2048 MB
• The following change was made to the defaults in run.properties
• runLO=4
• warmUpStyle = 1
• Add ethtool -C eth0 adaptive-tx off adaptive-rx off rx-usecs 250 in /etc/rc.local
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 130172
• * hard nofile 130172
• * soft nproc 130172
• * hard nproc 130172
• * soft memlock 8388608
• * hard memlock 8388608

Mail Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• Add ethtool -C eth0 adaptive-tx off adaptive-rx off rx-usecs 1000 in /etc/rc.local
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind, cpuspeed, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• root file-system options: barrier=0
• mail store file-system options: noatime,barrier=0,data=writeback
• Contents of dovecot configuration file /etc/dovecot/dovecot.conf:
• dict {
• }
• !include conf.d/*.conf
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• log_path = /var/log/dovecot.log
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mbox_write_locks = fcntl
• mail_fsync=never
• maildir_very_dirty_syncs = yes
• mbox_very_dirty_syncs = yes
• passdb {
• args = scheme=plain /etc/dovecot/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = postmaster@example.com
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
• ssl_key = </etc/pki/dovecot/private/dovecot.pem
• userdb {
• driver = passwd
• }
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400

Database Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, postgresql-9.2, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind, cpuspeed, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• kernel.sem = 500 64000 400 512
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• root file-system options: barrier=0
• database store file-system options: noatime,barrier=0,data=writeback
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• sysctl -w kernel.randomize_va_space=0
• sysctl -w vm.nr_hugepages=0
• sysctl -w vm.nr_hugepages=12288
• sysctl -w vm.nr_overcommit_hugepages=512
• Add ethtool -C eth0 adaptive-tx off adaptive-rx off rx-usecs 250 in /etc/rc.local
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 6300000
• db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5700000
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING
• alter tablespace C_parts_space NO FILE SYSTEM CACHING
• alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• alter tablespace M_parts_space NO FILE SYSTEM CACHING
• alter tablespace M_bom_space NO FILE SYSTEM CACHING
• alter tablespace M_wo_space NO FILE SYSTEM CACHING
• alter tablespace M_lo_space NO FILE SYSTEM CACHING
• alter tablespace M_inv_space NO FILE SYSTEM CACHING
• alter tablespace O_cust_space NO FILE SYSTEM CACHING
• alter tablespace O_ords_space NO FILE SYSTEM CACHING
• alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• alter tablespace O_item_space NO FILE SYSTEM CACHING
• alter tablespace S_comp_space NO FILE SYSTEM CACHING
• alter tablespace S_sc_space NO FILE SYSTEM CACHING
• alter tablespace S_po_space NO FILE SYSTEM CACHING
• alter tablespace S_site_space NO FILE SYSTEM CACHING
• alter tablespace S_supp_space NO FILE SYSTEM CACHING
• alter tablespace U_space NO FILE SYSTEM CACHING
• update database config for specdb using LOGFILSIZ 65536
• update db cfg for specdb using logprimary 54 logsecond 4
• The following lines were added to /etc/security/limits.conf:
• * soft nproc 578528
• * hard nproc 578528
• * soft nofile 331072
• * hard nofile 331072
• * soft memlock 33554432
• * hard memlock 33554432

Infraserver VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind, cpuspeed, sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• backend store file-system options: noatime,barrier=0,data=writeback
• httpd.conf changes:
• Timeout 20
• #KeepAliveTimeout 15
• #Include conf.d/*.conf
• #MaxKeepAliveRequests 100
• CustomLog logs/access_log common
• Listen 81
• ServerName besim:80
• <IfModule prefork.c>
• StartServers 10
• MinSpareServers 10
• MaxSpareServers 10
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• </IfModule>
• LoadModule fastcgi_module modules/mod_fastcgi.so
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgiIpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin ">
• AllowOverride None
• Options +ExecCGI -Includes
• SetHandler fastcgi-script
• Order allow,deny
• Allow from all
• </Directory>
• AddHandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• NFS export options:
• rw,async,no_subtree_check,no_root_squash
• Add ethtool -C eth0 adaptive-tx off adaptive-rx off rx-usecs 333 in /etc/rc.local
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400

Batch Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind, cpuspeed, sysstat
• tuned virtual-guest profile applied to VM
• root file-system options: barrier=0
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400

Client Driver Notes

• All prime clients were VMs running on Red Hat Enterprise Linux Server 6.5 KVM
• The prime controller ran on the first vClient
• Two vClients were used for each tile. One vClient per tile ran the webserver workload, the other vClient per tile ran the other workloads
• Each vClient has 4 vCPUs, 6GB memory, 16GB disk, 1 virtio_blk device and 1 virtio_net device
• The first vClient also acted as prime controller and therefore had one more virtio_net device for management
• The physical client has 4 x 10 Gbps ports and were configured as software bridges for the vClients
• Create one raid10 array with 4 x 800GB SSDs. It contains one logical volume with all VM images deployed on it

Power Configuration Notes

• PTD[0] was connected to SUT only.
• APSSY64K160H-PD was used, power analyzer connected between UPS and SUT

Other Notes

• 1 x Extreme Network Summit X670-48x-SFP+ switch was used to connect the clients and the SUT

Performance Details: