Performance Summary:

Performance Details:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• Firmware settings used:
• TurboModeEnable=Enable
• TurboBoost=Traditional
• HardwarePrefetcher=Disable
• Virtualization-VTDx-Enable
• MemorySpeed=MaxPerformance
• ThermalMode=Performance
• Power=CappingDisabled
• OperatingModes=Custom
• TurboBoostPowerOptimization=Traditional

Storage Notes

• All storage is accessed with 1 x 8Gb Fibre Channel link. Channel A of DS4800 FC Controller was attached to 6 EXP810 enclosures containing 16 136GB FC disks, 7 EXP810 enclosures containing 16 73GB FC disks each. Total FC drive count is 208. There are 22 FC RAID5 LUNS, each consisting of 8 drives. Each LUN contains the VM image and data for one tile, with two exceptions. First, the hypervisor boot disk is located on a 50GB LUN created on the Tile1 disk array. A second 50GB LUN is created on the Tile1 disk array, but it is not used. Secondly, the Mail Data for all 22 tiles is located on one 32 disk RAID10 array. Each LUN defined for a Tile is divided into 11 logical volumes (2 per VM, with the exception of the mail-data logical volume) using LVM. The Mail Data LUN is divided into 22 logical volumes. The HX5 connects to the SAN via 1 port on a IBM Qlogic 8Gb Fibre Channel Switch Module. A second connection is configured but is not used.

Virtualization Software Notes

• The following boot options were used:
•   intel_iommu=on iommu=pt
• The following services are disabled:
•   dnsmasq, iptables, ip6tables
• Network configuration is as follows:
•   One software bridge is created, br0 and the second Broadcom
•   Ethernet port is assigned to the bridge (eth1 to br0). First Broadcom
•   port is for host access. Every VM's virtIO network device is assigned to br0.
•   Kernel module for Intel x520 10 Gigabit Server adapter is loaded with
•   these options to create 124 virtual functions: max_vfs=62,62.
•   70 of these virtual functions are assigned (one per VM) to the Web, App
•   DB, Mail and Infra VMs. When a virtual function is present on a VM, a script
•   is executed to shut down the first virtio_net device and migrate the IP
•   settings to the virtual function device. This is done to facilitate
•   easy IP/dns management when using a mix of VMs with and without
•   virtual functions. The script, convert_to_eth.sh is available in the
•   disclosure archive. On the host, Ethernet devices eth0 to eth1 are the two physical
•   functions for the active Intel 10Gbps Ethernet ports. These
•   are only used by the host to provide physical transport for the virtual functions.
• Web, App, Idle, Infra and Mail VMs are launched with nice utility (-1,-6,10,4,4, respectively) to adjust their scheduling priority
• All VMs of odd tiles are bound to NUMA node 1 and have a preferred memory allocation node of 1
• All VMs of even tiles are bound to NUMA node 0 and have a preferred memory allocation node of 0
• All VMs are created with the following command line format (options in [] are for VMs which have PCI device assignment):
•   /usr/libexec/qemu-kvm -name $name -M rhel6.0.0 -enable-kvm -drive file=$dev1,if=virtio,boot=on,cache=none
•   -drive file=$dev2,if=virtio,boot=off,cache=none -netdev tap,id=$name,ifname=$name,script=/etc/kvm/qemu-ifup-br0,vhost=off
•   -device virtio-net-pci,netdev=$name,mac=$macaddr [-pcidevice host=$pcidev] -m $mem -vnc $vnc_port -cpu qemu64,+x2apic
•   -no-kvm-pit-reinjection -smp $cpus,sockets=$cpus,cores=1,threads=1 -monitor telnet::$monitor_port,server,nowait
•   -serial telnet::$serial_port,server,nowait -daemonize
•   After boot, all block I/O devices changed to use deadline I/O scheduler via sysfs interface

Web Server VM Notes

• The following boot options were used:
•   selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• httpd.conf changes:
•   KeepAlive on
•   MaxKeepAliveRequest 0
•   ServerLimit 10240
•   MaxClients 10240
•   MaxRequestsPerChild 1000
•   CustomLog logs/access_log common
• Web VM used NFS to access files from infra VM
• NFS mount options: /support_downloads: rw, no_subtree_check,crossmnt,no_root_squash
• root file-system options: barrier=0
• ixgbevf module option used: InterruptThrottleRate=2000
• The following non-default sysctl.conf values were used:
• net.ipv4.neigh.default.gc_thresh1 = 4096
• net.ipv4.neigh.default.gc_thresh2 = 8192
• net.ipv4.neigh.default.gc_thresh3 = 8192
• net.ipv4.neigh.default.base_reachable_time = 86400
• net.ipv4.neigh.default.gc_stale_time = 86400

Application Server VM Notes

• The following boot options were used:
•   selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator
• Heap size for AppSrv01 has been changed to 1280 MB
• root file-system options: barrier=0
• ixgbevf module option used: InterruptThrottleRate=2000
• The following non-default sysctl.conf values were used:
• net.ipv4.neigh.default.gc_thresh1 = 4096
• net.ipv4.neigh.default.gc_thresh2 = 8192
• net.ipv4.neigh.default.gc_thresh3 = 8192
• net.ipv4.neigh.default.base_reachable_time = 86400
• net.ipv4.neigh.default.gc_stale_time = 86400

Mail Server VM Notes

• The following boot options were used:
•   selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• Dovecot was compiled with default options and the following dovecot.conf was used:
•  base_dir = /var/run/dovecot/
•  ssl = no
•  disable_plaintext_auth = no
•  listen = *
•  log_path = /var/log/dovecot
•  mail_debug = yes
•  mail_location = dbox:/mailstore/dovecot/maildir/%u
•  mdbox_rotate_size = 2048
•  passdb {
•   args = /etc/passwd.dovecot
•   driver = passwd-file
•  }
•  protocols = imap
•  service auth {
•   user = root
•  }
•  service imap-login {
•   inet_listener imap {
•    port = 143
•   }
•   process_min_avail = 10
•  }
•  userdb {
•   driver = passwd
•   driver = passwd
•  }
• root file-system options: barrier=0
• ixgbevf module option used: InterruptThrottleRate=2000
• The following non-default sysctl.conf values were used:
• net.ipv4.neigh.default.gc_thresh1 = 4096
• net.ipv4.neigh.default.gc_thresh2 = 8192
• net.ipv4.neigh.default.gc_thresh3 = 8192
• net.ipv4.neigh.default.base_reachable_time = 86400
• net.ipv4.neigh.default.gc_stale_time = 86400

Database Server VM Notes

• The following boot options were used:
•   hugepages=416 selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• Database tuning options:
•   db2 update db cfg for specdb using DATABASE_MEMORY 200000
•   db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
•   alter bufferpool IBMDEFAULTBP size 160000
•   alter tablespace roll_space NO FILE SYSTEM CACHING
•   alter tablespace C_cust_space NO FILE SYSTEM CACHING
•   alter tablespace C_supp_space NO FILE SYSTEM CACHING
•   alter tablespace C_site_space NO FILE SYSTEM CACHING
•   alter tablespace C_parts_space NO FILE SYSTEM CACHING
•   alter tablespace C_custinv_space NO FILE SYSTEM CACHING
•   alter tablespace M_parts_space NO FILE SYSTEM CACHING
•   alter tablespace M_bom_space NO FILE SYSTEM CACHING
•   alter tablespace M_wo_space NO FILE SYSTEM CACHING
•   alter tablespace M_lo_space NO FILE SYSTEM CACHING
•   alter tablespace M_inv_space NO FILE SYSTEM CACHING
•   alter tablespace O_cust_space NO FILE SYSTEM CACHING
•   alter tablespace O_ords_space NO FILE SYSTEM CACHING
•   alter tablespace O_ordl_space NO FILE SYSTEM CACHING
•   alter tablespace O_item_space NO FILE SYSTEM CACHING
•   alter tablespace S_comp_space NO FILE SYSTEM CACHING
•   alter tablespace S_sc_space NO FILE SYSTEM CACHING
•   alter tablespace S_po_space NO FILE SYSTEM CACHING
•   alter tablespace S_site_space NO FILE SYSTEM CACHING
•   alter tablespace S_supp_space NO FILE SYSTEM CACHING
•   alter tablespace U_space NO FILE SYSTEM CACHING
•   update db cfg for specdb using logprimary 64
• root file-system options: barrier=0
• ixgbevf module option used: InterruptThrottleRate=2000
• The following non-default sysctl.conf values were used:
• net.ipv4.neigh.default.gc_thresh1 = 4096
• net.ipv4.neigh.default.gc_thresh2 = 8192
• net.ipv4.neigh.default.gc_thresh3 = 8192
• net.ipv4.neigh.default.base_reachable_time = 86400
• net.ipv4.neigh.default.gc_stale_time = 86400

Infraserver VM Notes

• The following boot options were used:
•   selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• httpd.conf changes:
•   Listen port to 81
•   ServerLimit 10240
•   MaxClients 10240
•   CustomLog logs/access_log common
• /etc/httpd/conf.d/fcgi.conf added:
•   LoadModule fastcgi_module modules/mod_fastcgi.so
•   ScriptAlias /fcgi-bin/ /var/www/fcgi-bin/
•   FastCgiIpcDir /etc/httpd/fastcgi
•   <Directory /var/www/fcgi-bin>
•     AllowOverride None
•     Options +ExecCGI -Includes
•     SetHandler fastcgi-script
•     Order allow,deny
•     Allow from all
•   </Directory>
•   AddHandler fastcgi-script fcgi
•   FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• Standard Linux NFS server is enabled to serve files to Web VM
• /etc/exports file contains:
•  /support-download *(rw,no_subtree_check,crossmnt,no_root_squash)
• root file-system options: barrier=0
• ixgbevf module option used: InterruptThrottleRate=2000
• Fcgi package built from Fedora 14 source RPM, which includes patch to compile with GCC 4.4
• The following non-default sysctl.conf values were used:
• net.ipv4.neigh.default.gc_thresh1 = 4096
• net.ipv4.neigh.default.gc_thresh2 = 8192
• net.ipv4.neigh.default.gc_thresh3 = 8192
• net.ipv4.neigh.default.base_reachable_time = 86400
• net.ipv4.neigh.default.gc_stale_time = 86400

Idle Server VM Notes

• The following boot options were used:
•   selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• root file-system options: barrier=0
• The following non-default sysctl.conf values were used:
• net.ipv4.neigh.default.gc_thresh1 = 4096
• net.ipv4.neigh.default.gc_thresh2 = 8192
• net.ipv4.neigh.default.gc_thresh3 = 8192
• net.ipv4.neigh.default.base_reachable_time = 86400
• net.ipv4.neigh.default.gc_stale_time = 86400

Client Driver Notes

• All prime clients are VMs hosted with Red Hat Enterprise Linux 6.0. The first two client systems each host client VMs 1-7 and 8-14, respectively. The third client system hosts prime client VMs 15-22. The fourth client hosts the prime controller. Each VM has 2 vCPUs, and prime client VMs' vCPUs are bound to sibling threads in a processor core. Paravirtual clock is used for all client VMs. 1 virtio_net device, and 1 virtio_blk device are used in all client VMs. For the host systems, there are 6 1Gbps Ethernet ports (Clients 1-3 have 6 ports active, the Prime Controller client has 1 port active) on each physical client. On all clients, the first Ethernet port is used for host access and NFS. The remaining 5 active ports are used for software bridges, up to 2 VMs per bridge. Client VM images are located on a 2 disk RAID1 Array of 73GB SAS disks located inside the host. The Client ports connect into the Network via a CISCO Catalyst 6500 switch, containing a 48-port, Gigabit Ethernet Catalyst 6500 blade.
• All clients are running Red Hat Enterprise Linux 6.0 (64-bit) except for the Prime Controller which is running RedHat Enterprise Linux 5.5 (64-bit, kernel 2.6.18-194)
• The following non-default sysctl.conf values were used:
• net.ipv4.neigh.default.gc_thresh1 = 4096
• net.ipv4.neigh.default.gc_thresh2 = 8192
• net.ipv4.neigh.default.gc_thresh3 = 8192
• net.ipv4.neigh.default.base_reachable_time = 86400
• net.ipv4.neigh.default.gc_stale_time = 86400

Other Notes

• A CISCO Catalyst 6500 switch containing 2 blades (1Gb and 10Gb) is used to provide ethernet connectivity between the HX5 and the client systems. The HX5 connects to the Catalyst 6500 switch using 2 10Gb Ethernet ports, and 1Gb Ethernet port. Two BNT Virtual Fabric 10Gb Switch Modules provide access to the 10Gb adapter ports inside the HX5. One IBM Cisco Catalyst Switch 3110G provides access to a 1gb adapter port inside the HX5. One IBM Cisco Catalyst Switch 3110G and one 1Gb Ethernet port inside the HX5 are unused. The clients connect to the Catalyst 6500 switch using 22 Gb Ethernet ports.