Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• The test sponsor attests, as of date of publication, that CVE-2017-5754 (Meltdown) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2017-5753 (Spectre variant 1) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2017-5715 (Spectre variant 2) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2018-3620 and CVE-2018-3646 (Foreshadow) are mitigated in the system as tested and documented:
• Yes: hosts, guests

Storage Notes

• The System Under Test has 14 7.68TB NVMe SSDs configured as one RAID10 arrays in OS by mdadm
• The System Under Test has one NVMe SSD contains the logical volume for the hypervisor OS that was booted.

Virtualization Software Notes

• The following kernel boot options were used:
• crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet selinux=0 audit=0 elevator=deadline clocksource=tsc amd_iommu=on iommu=pt pci=realloc nosoftlockup ipv6.disable=1 mce=ignore_ce mds=off pti=off default_hugepagesz=1G hugepagesz=1G hugepages=1859
• The following services were disabled:
• kdump irqbalance firewalld crond multipathd smartd
• The following items were added in /etc/sysctl.conf:
• fs.aio-max-nr = 1572864
• kernel.perf_event_max_sample_rate = 250
• kernel.sched_rt_runtime_us = 950000
• kernel.numa_balancing = 0
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• Network configuration was as follows:
• Ethernet devices ens2f0np0, ens2f1np1, ens6f0np0, ens6f1np1 were the physical functions that were used by the host to provide physical transport for the virtual functions.
• A script allocates the virtual functions.
• 69 virtual functions were configured for each port.
• All VMs were assigned a VF that is routed to the NUMA node to which the VM's VCPUs were pinned.
• After the system booted, 1GB huge pages were manually allocated on the NUMA nodes. 914 1GB huge pages were allocated on NUMA node 0, 945 1GB huge pages were allocated on NUMA node 1.
• VM configurations were as follows:
• All VMs were created with qemu.
• App, Web and Batch VMs were created with the following cmd template:
• cd $DIR && $DIR/vfio-pci-bind.rb $vm_nic && numactl -m $vm_numa_id -N $vm_numa_id /usr/libexec/qemu-kvm -machine pc-q35-rhel9.4.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off -enable-kvm -cpu host,migratable=off,+invtsc,-vmx,+tsc-deadline,pmu=off -smp $vm_vcpu_count -m $vm_memory_size -mem-prealloc -mem-path /dev/hugepages1G -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 -device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 -device virtio-blk-pci,scsi=off,bus=pci.3,drive=hd,write-cache=on,bootindex=1 -drive if=none,id=hd,file=$vm_disk_path,format=raw,cache=none,aio=native -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.7,addr=0x0 -name $vm_name,process=$vm_name,debug-threads=on -device vfio-pci,host=$vm_nic -display none --nic none -daemonize -kernel /boot/vmlinuz.guest -initrd /boot/initramfs.guest -append 'root=/dev/mapper/rhel-root resume=/dev/mapper/rhel-swap ro rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet'
• Db, Infra and Mail VMs have this following cmd template for their data disk:
• cd $DIR && $DIR/vfio-pci-bind.rb $vm_nic && numactl -m $vm_numa_id -N $vm_numa_id /usr/libexec/qemu-kvm -machine pc-q35-rhel9.4.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off -enable-kvm -cpu host,migratable=off,+invtsc,-vmx,+tsc-deadline,pmu=off -smp $vm_vcpu_count -m $vm_memory_size -mem-prealloc -mem-path /dev/hugepages1G -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 -device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 -device virtio-blk-pci,scsi=off,bus=pci.3,drive=hd,write-cache=on,bootindex=1 -drive if=none,id=hd,file=$vm_disk_path,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.4,drive=hd2,write-cache=on -drive if=none,id=hd2,file=$vm_disk_data_path format=raw,cache=none,aio=native -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.7,addr=0x0 -name $vm_name,process=$vm_name,debug-threads=on -device vfio-pci,host=$vm_nic -display none --nic none -daemonize -kernel /boot/vmlinuz.guest -initrd /boot/initramfs.guest -append 'root=/dev/mapper/rhel-root resume=/dev/mapper/rhel-swap ro rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet'
• VMs were bound to host CPUs via their scripts in the following manner:
• All VMs of tiles 01, 02, 03, 04, 09, 10, 11, 12, 17, 18, 19, 20, 25, 26, 27, 28, 33, 34, 35, 36, 41, 42, 43, 44, 49, 50 were bound to NUMA node 0.
• All VMs of tiles 05, 06, 07, 08, 13, 14, 15, 16, 21, 22, 23, 24, 29, 30, 31, 32, 37, 38, 39, 40, 45, 46, 47, 48, 51, 52 were bound to NUMA node 1.
• Db VMs of tiles 01, 9, 17, 25, 33, 41 were bound to NUMA node 0.
• Db VMs of tiles 05, 13, 21, 29, 37, 45, 49 were bound to NUMA node 1.
• App VMs were reniced -20.
• Db VMs were reniced -12.
• Web VMs were reniced -8.
• Mail VMs were reniced 19.
• Infra VMs were reniced 19.
• Batch VMs were reniced 19.
• After the system booted, power of CPU was set manually with the command: cpupower frequency-set -g performance.
• The tuned package was installed and activated with the virtual-host profile.
• The following items were added to rc.local:
• cpupower frequency-set -g performance
• tuned-adm profile virtual-host
• echo 24000000 > /sys/kernel/debug/sched/latency_ns
• echo 5000000 > /sys/kernel/debug/sched/migration_cost_ns
• echo 10000000 > /sys/kernel/debug/sched/min_granularity_ns
• echo 15000000 > /sys/kernel/debug/sched/wakeup_granularity_ns
• The following items were added to limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited

Web Server VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers atd auditd blk-availability chrony-wait console-getty cpupower crond debug-shell ebtables firewalld gssproxy httpd httpd@ io.podman iprdump iprinit iprupdate irqbalance iscsid iscsiuio kdump kpatch lvm2-monitor mcelog mdmonitor nfs-blkmap nfs-server nftables nginx php-fpm psacct qemu-guest-agent rdisc rhnsd rhsm-facts rhsm rhsmcertd rngd saslauthd serial-getty@ smartd sshd-keygen@ sssd systemd-resolved tcsd
• root file system mount options: defaults,noatime
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 81920
• net.core.net_dev_budget = 3000
• net.core.net_dev_max_backlog = 30000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_fin_timeout = 20
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_keepalive_probes = 4
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_max_syn_backlog = 20000
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_tw_recycle = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_max_tw_buckets = 1500000
• vm.swappiness = 0
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• The php-fpm package was installed. /etc/php-fpm.d/www.conf changes:
• user = apache
• group = apache
• pm.max_children = 700
• pm.start_servers = 250
• pm.min_spare_servers = 100
• pm.max_spare_servers = 300
• /etc/nginx/nginx.conf contains:
• user apache apache;
• worker_processes 4;
• worker_cpu_affinity 0001 0001 0010 0010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request'
• '"$status" $body_bytes_sent ';
• access_log off;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• keepalive_requests 10000;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• open_file_cache_errors on;
• read_ahead 4096;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• }
• /etc/nginx/conf.d/default.conf contains:
• upstream fastcgi_backend {
• server 127.0.0.1:9000;
• keepalive 128;
• }
• server {
• listen 80 backlog=81920;
• server_name localhost;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• location = /error.log {
• root /var/log/nginx;
• }
• location ~ .*/.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ /.php$ {
• root html;
• fastcgi_pass fastcgi_backend;
• fastcgi_keep_conn on;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• /etc/nginx/conf.d/ssl.conf contains:
• server {
• listen 443 ssl backlog=81920;
• server_name webserver:443;
• ssl on;
• ssl_certificate /etc/pki/tls/certs/hostname.crt;
• ssl_certificate_key /etc/pki/tls/private/hostname.key;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
• ssl_ciphers ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*/.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ /.php$ {
• root html;
• fastcgi_pass fastcgi_backend;
• fastcgi_keep_conn on;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Web VM used NFS to access files from infra VM.
• After the Infra VM was booted and its NFS service was up, the Web VM mounted the NFS file system via test automation software.
• The datastore was mounted on /var/www/html/support/downloads
• The datastore was mounted on /var/www/html/support/downloads with the following parameters:
• rw,noatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,acregmin=14400,acregmax=14400,acdirmin=14400,acdirmax=14400,hard,nocto,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none

Application Server VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers blk-availability brltty canberra-system-bootup canberra-system-shutdown-reboot canberra-system-shutdown chrony-wait console-getty cpupower cups-browsed debug-shell dnsmasq ebtables firewalld gssproxy initial-setup-reconfiguration initial-setup io.podman iprdump iprinit iprupdate iscsid iscsiuio kpatch libvirt-guests ndctl-monitor netcf-transaction nfs-blkmap nfs-server nftables numad ostree-finalize-staged ostree-remount psacct qemu-guest-agent radvd ras-mc-ctl rasdaemon rdisc rdma rhnsd rhsm-facts rhsm saslauthd serial-getty@ speech-dispatcherd sshd-keygen@ switcheroo-control systemd-nspawn@ systemd-resolved tcsd upower vmtoolsd-init wpa_supplicant
• root file system mount options: defaults
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 8192
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 500000
• vm.swappiness = 0
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• There were two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator.

Mail Server VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers atd auditd blk-availability chrony-wait console-getty cpupower crond debug-shell ebtables firewalld io.podman iprdump iprinit iprupdate irqbalance iscsid iscsiuio kdump kpatch lvm2-monitor mcelog mdmonitor nftables psacct qemu-guest-agent rdisc rhnsd rhsm-facts rhsm rhsmcertd rngd serial-getty@ smartd sshd-keygen@ sssd systemd-resolved tcsd
• root file system mount options: defaults,noatime
• Mail storage file system mount options: defaults,noatime,data=writeback
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 8192
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 500000
• vm.swappiness = 0
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• The following dovecot.conf was used:
• base_dir = /var/run/dovecot/
• ssl = no
• ssl_cert =
• ssl_key =
• disable_plaintext_auth = no
• listen = *
• log_path = /var/log/dovecot.log
• mail_debug = no
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 8192
• maildir_very_dirty_syncs = yes
• mbox_very_dirty_syncs = yes
• mbox_write_locks = fcntl
• mmap_disable = yes
• mail_fsync = never
• mailbox_list_index = yes
• mailbox_list_index_very_dirty_syncs = yes
• auth_cache_size = 5000
• passdb {
• args = scheme=plain /etc/dovecot/passwd.dovecot
• driver = passwd-file
• }
• protocols = imap
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_min_avail = 10
• }
• userdb {
• driver = passwd
• }
• dict {
• }

Database Server VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers atd auditd blk-availability chrony-wait chronyd console-getty cpupower crond debug-shell ebtables firewalld gssproxy io.podman iprdump iprinit iprupdate irqbalance iscsid iscsiuio kdump kpatch libstoragemgmt mcelog nfs-blkmap nfs-server nftables psacct qemu-guest-agent rdisc rhnsd rhsm-facts rhsm rhsmcertd rngd rsyslog serial-getty@ smartd sshd-keygen@ sssd systemd-resolved tcsd
• root file system mount options: defaults,noatime
• Database storage file system mount options: noatime,data=writeback
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.randomize_va_space = 0
• kernel.sched_features = 7275
• net.core.somaxconn = 32384
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 1000000
• vm.swappiness = 0
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• kernel.sched_migration_cost_ns = 150000
• kernel.randomize_va_space=0
• vm.nr_hugepages=0
• vm.nr_hugepages=13312
• vm.nr_overcommit_hugepages=512
• The following lines were added to /etc/security/limits.conf:
• db2inst1 soft nofile 1048576
• db2inst1 hard nofile 1048576
• db2inst1 soft nproc 1000000
• db2inst1 hard nproc 1000000
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• 13312 2MB huge pages were allocated prior to starting the database instance.
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 6553600
• db2set DB2_LARGE_PAGE_MEM=DB
• db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5898240
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING
• alter tablespace C_parts_space NO FILE SYSTEM CACHING
• alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• alter tablespace M_parts_space NO FILE SYSTEM CACHING
• alter tablespace M_bom_space NO FILE SYSTEM CACHING
• alter tablespace M_wo_space NO FILE SYSTEM CACHING
• alter tablespace M_lo_space NO FILE SYSTEM CACHING
• alter tablespace M_inv_space NO FILE SYSTEM CACHING
• alter tablespace O_cust_space NO FILE SYSTEM CACHING
• alter tablespace O_ords_space NO FILE SYSTEM CACHING
• alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• alter tablespace O_item_space NO FILE SYSTEM CACHING
• alter tablespace S_comp_space NO FILE SYSTEM CACHING
• alter tablespace S_sc_space NO FILE SYSTEM CACHING
• alter tablespace S_po_space NO FILE SYSTEM CACHING
• alter tablespace S_site_space NO FILE SYSTEM CACHING
• alter tablespace S_supp_space NO FILE SYSTEM CACHING
• alter tablespace U_space NO FILE SYSTEM CACHING
• update database config for specdb using LOGFILSIZ 65536
• update database config for specdb using logprimary 64 logsecond 4

Infraserver VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers atd auditd blk-availability chrony-wait console-getty cpupower crond debug-shell ebtables firewalld gssproxy httpd@ io.podman iprdump iprinit iprupdate irqbalance iscsid iscsiuio kdump kpatch lvm2-monitor mcelog mdmonitor nfs-blkmap nftables psacct qemu-guest-agent rdisc rhnsd rhsm-facts rhsm rhsmcertd rngd saslauthd serial-getty@ smartd sshd-keygen@ sssd systemd-resolved tcsd
• root file system mount options: defaults
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 81920
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamp = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness = 0
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• httpd.conf changes:
• Listen 81
• StartServers 8
• MinSpareServers 5
• MaxSpareServers 20
• ServerLimit 10240
• MaxClients 10240
• MaxRequestsPerChild 4000
• StartServers 2
• MaxClients 150
• MinSpareThreads 25
• MaxSpareThreads 75
• ThreadsPerChild 25
• MaxRequestsPerChild 0
• LoadModule fcgid_module modules/mod_fcgid.so
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/
• FcgidIPCDir /etc/httpd/fcgid
• AllowOverride None
• Options +ExecCGI -Includes
• SetHandler fcgid-script
• Order allow,deny
• Allow from all
• AddHandler fcgid-script fcgi
• Standard Linux NFS server was enabled to serve files to the Web VM.
• /etc/exports file contains:
• /home/webfiles/downloads webserver-int(rw,async,no_subtree_check,no_root_squash)

Batch Server VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers atd auditd blk-availability chrony-wait console-getty cpupower crond debug-shell ebtables firewalld io.podman iprdump iprinit iprupdate irqbalance iscsid iscsiuio kdump kpatch lvm2-monitor mcelog mdmonitor nftables psacct qemu-guest-agent rdisc rhnsd rhsm-facts rhsm rhsmcertd rngd serial-getty@ smartd sshd-keygen@ sssd systemd-resolved tcsd
• root file system mount options: defaults,noatime
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 8192
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 500000
• vm.swappiness = 0
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000

Client Driver Notes

• Each prime client VM has 8 VCPUs, 16 GB memory, and a 12 GB disk. 1 vfio_pci and 1 virtio_blk device were used in all client VMs.
• All prime client VMs run Red Hat Enterprise Linux 8.0.

Other Notes

• An external 100Gb H3C S6850-56HF switch was used to connect the client systems to the system under test (SUT).
• The SUT used the two dual-port 100Gb PCIe NICs for communication with the VMs.
• The SUT NICs were connected to the switch using 100Gb QSFP28 DAC cables, the external client systems were connected to the switch using 25Gb SFP28 DAC cables.

Performance Details: