Performance Summary:

Performance Details:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Minimum Processor Idle Power Core State set to C3 State
• Power Profile set to Maximum Performance
• Thermal Configuration set to Increased Cooling
• Virtualization Technology set to Enabled
• Processor Power and Utlization Monitoring set to Disabled

Storage Notes

• Each port of the SUT's QLogic QMH2562 8Gb Fibre Channel HBA was connected to
• a Brocade 8Gb SAN Switch for HP BladeSystem c-Class module. Each SAN Switch
• module was connected via one 8Gb Fibre Channel link to 1 x StorageWorks
• 8/24 SAN switch. 6 x StorageWorks P2000 storage enclosures were connected
• via 2 x 8Gb Fibre Channel links to the StorageWorks SAN switch. 4 x P2000's
• contained 24 x 146 GB SFF 15 K SAS drives and were partitioned into 4 x vdisks.
• 2 x P2000s contained 24 x 72 GB SFF 15 K SAS drives and were partitioned into 4 x vdisks.
• 1 x P2000 with 72 GB drives was not used for the measurements.
• Each vdisk was configured as RAID10 with 6 drives and contained one tile's VMs.

Virtualization Software Notes

• The following services are disabled:
• dnsmasq, iptables, ip6tables
• ulimit -n and ulmit -u were set to 102400 in /etc/bashrc.
• The Network configuration is as follows: 14 virtual bridges were created. 12 were assigned to ethernet ports
• and 2 were used as internal networks for inter-VM traffic. One physical ethernet port was used for host
• access. virbr1 and virbr2 (internal networks) were configured with MTU=9000. The network bridges and network
• ports were laid out as follows:
• eth0 assigned to br2, tiles 5 and 7 connected,
• eth1 assigned to br3, tiles 6 and 8 connected,
• eth2 assigned to br6, tiles 13 and 15 connected,
• eth3 assigned to br7, tiles 14 and 16 connected,
• eth4 assigned to br10, no tiles connected,
• eth5 assigned to br11, no tiles connected,
• eth6 and eth7 not used,
• eth8 used for host access,
• eth9 assigned to br0, tiles 1 and 3 connected,
• eth10 asigned to br1, tiles 2 and 4 connected,
• eth11 assigned to br5, tiles 10 and 12 connected,
• eth12 assigned to br4, tiles 9 and 11 connected,
• eth13 assigned to br9, tiles 18 and 19 connected,
• eth14 assigned to br8, tile 17 connected,
• eth15 not used,
• virbr0 not used,
• virbr1 connected to tiles 1, 3, 5, 7, 9, 11, 13, 15, and 17 internal ports
• virbr2 connected to tiles 2, 4, 6, 8, 10, 12, 14, 16, 18, and 19 internal ports
• Web, App, Db, Idle, Infra and Mail VMs are launched with nice utility (-1,-4,-4, 19,1,6, respectively) to
• adjust their scheduling priority.
• All VMS of odd tiles are bound to NUMA node 1 and have a preferred memory allocation node of 1
• except for tile 19 which is bound to NUMA node 0 and have a preferred memory allocation node of 0.
• All VMs of even tiles are bound to NUMA node 0 and have a preferred memory allocation node of 0.
• All VMs are created with the following command line format :
• /usr/libexec/qemu-kvm -name $name -M rhel6.0.0 -enable-kvm/
• -m $mem -mem-prealloc -mem-path /dev/hugepages /
• -cpu qemu64,+x2apic -no-kvm-pit-reinjection/
• -smp $cpu,sockets=1,cores=$cpu,threads=1/
• -drive file=$dev1,if=virtio,boot=on,cache=none,aio=native/
• -drive file=$dev2,if=virtio,boot=on,cache=none,aio=native/
• -netdev tap,id=vnet_$VM$TILENO_0,ifname=vnet_$VM$TILENO_0,script=/etc/libvirt/qemu-ifup-br$BRIDGE,vhost=on/
• -device virtio-net-pci,netdev=vnet_$VM$TILENO_0,mac=$macaddr1/
• -netdev tap,id=vnet_$VM$TILENO_1,ifname=vnet_$VM$TILENO_1,script=/etc/libvirt/qemu-ifup-virbr$VBRIDGE,vhost=on/
• -device virtio-net-pci,netdev=vnet_$VM$TILENO_1,mac=$macaddr2/
• -vnc 0.0.0.0:114 /
• All VMs are initialized with hugepages.
• The following entry was added to /systcl.conf:
• vm.nr.hugepages=240000
• Hugespages were mounted on boot with the following entry in /etc/fstab:
• nodev /dev/hugepages hugetlbfs defaults 0 0

Web Server VM Notes

• Boot parameters set: divider=10, selinux=0, audit=0, notsc.
• Internal NIC set to use jumbo frames by adding 'MTU=9000' to /etc/sysconfig/network-scripts/ifconfig-eth1
• VM setting CPU shares was set to 'high' via vConsole
• VM setting Memory Reservation was set to '20480' via vConsole
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• Add line in httpd.conf to 'Include conf/extra/httpd-default.conf'
• Add line in httpd.conf to 'Include conf/extra/httpd-mpm.conf'
• Change values in extra/httpd-default.conf:
• MaxKeepAliveRequest 0
• Change values in extra/httpd-mpm.conf:
• StartServers 20
• MinSpareServers 20
• MaxSpareServers 20
• ServerLimit 1536
• MaxClients 1536
• MaxRequestsPerChild 4000

Application Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline.
• Internal NIC set use jumbo frames by adding 'MTU=9000' to /etc/sysconfig/network-scripts/ifconfig-eth1
• The following lines were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.nr_hugepages = 768
• Huge pages were mounted by using the following line in /etc/fstab:
• nodev /mnt/hugepages hugetlbfs defaults 0 0
• The following lines were added to /etc/security/limits.conf:
• soft memlock 2048000
• hard memlock 2048000
• The following lines were added to /opt/glassfish/domains/spec2004-1/domain.xml:
• To jdbc-connection-pool line parameter 'max-pool-size = 64' was set
• To request-processing line parameter 'initial-thread-count = 90' was set
• To ejb-container line, parameter 'max-pool-size=64' was set
• To mdb-container line, parameter 'max-pool-size = 64' was set
• Line '-XX:+UseLargePages' was added
• jvm-option '-Xms1024m' was set
• jvm-option '-XmX1024m' was set

Mail Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline.
• The following lines were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• dovecot was upgraded to 2.0.4-1_119.el5.x84_64. The dbox format was used.
• Contents of dovecot.conf file:
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• listen = *, ::
• log_path = /var/log/dovecot
• mail_debug = no
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 2048
• passdb {
• args = /etc/passwd.dovecot
• driver = passwd-file
• }
• protocols = imap
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• }
• ssl = no
• userdb {
• driver = passwd
• }
• protocol lda {
• postmaster_address = postmaster@example.com
• mail_fsync = optimized
• }

Database Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline.
• Internal NIC set use jumbo frames by adding 'MTU=9000' to /etc/sysconfig/network-scripts/ifconfig-eth1
• The following lines were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1

Infraserver VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline.
• Internal NIC set use jumbo frames by adding 'MTU=9000' to /etc/sysconfig/network-scripts/ifconfig-eth1
• The following lines were added to sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1

Idle Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline.
• The following lines were added to sysctl.conf:
• fs.file-max = 500000"VM.IDLE.NOTES[12] = " net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1

Client Driver Notes

• Each physical client runs 3 virtual clients. Each virtual client uses 1 x vCPU, 2 GB memory, and 1 x 1Gb NIC port.
• The first virtual client on the first pysical client is dedicated as the prime controller
• One virtual client was used for each tile.

Other Notes

• A single ProCurve 2910al 48-port switch was used to connect clients to SUT.
• The 10GbE ports were attached to a dual port 10 GbE SFP+ expansion module (J9008A)
• All 4 of the 10Gb NIC ports were used for networking for the benchmarking. A
• single port on one NC464m was used to access the vConsole for administration.
• 1 x StorageWorks 8/24 SAN Fibre switch was used to connect the SUT to the external storage.