Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Power Profile set to Maximum Performance
• Intel(R) VMD Technology: Enabled
• The test sponsor attests, as of date of publication, that CVE-2017-5754 (Meltdown) is mitigated in the system as tested and documented:
• Yes: hosts, No: guests
• The test sponsor attests, as of date of publication, that CVE-2017-5754 (Meltdown) is mitigated in the system as tested and documented:
• Yes: hosts, No: guests
• The test sponsor attests, as of date of publication, that CVE-2017-5754 (Meltdown) is mitigated in the system as tested and documented:
• Yes: hosts, No: guests
• The test sponsor attests, as of date of publication, that CVE-2017-5754 (Meltdown) is mitigated in the system as tested and documented:
• Yes: hosts, No: guests

Storage Notes

• The System Under Test has 2 480GB SATA SSDs configured as one RAID1 arrays in BIOS by Broadcom LSI MegaRAID 9560-8i SAS3908 raid controllerfor the hypervisor OS that was booted.
• The System Under Test has 8 3.2TB NVMe SSDs configured as one RAID5 arrays in BIOS by Intel(R) Virtual RAID on CPU for vm images and datastores.
• There are total 420 logical volume for vm.

Virtualization Software Notes

• The following lines were added to /usr/lib/sysctl.d/01-specvirt.conf:
• kernel.perf_event_max_sample_rate = 250
• kernel.sched_min_granularity_ns = 30000000
• kernel.sched_wakeup_granularity_ns = 30000000
• kernel.sched_latency_ns = 100000
• kernel.sched_migration_cost_ns = 13000
• kernel.sched_rt_runtime_us = 960000
• kernel.numa_balancing = 0
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• fs.aio-max-nr = 1572864
• fs.inotify.max_user_watches = 65536
• net.ipv6.conf.all.disable_ipv6 = 1
• net.core.netdev_max_backlog = 500000
• net.ipv4.conf.default.promote_secondaries = 1
• net.ipv4.conf.all.promote_secondaries = 1
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• The following boot options were used:
• intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=800 nohz=on cgroup_disable=cpu,cpuacct,memory,blkio,hugetlb clocksource=tsc tsc=reliable highres=off nosoftlockup intel_idle.max_cstate=0 processor.max_cstate=1 mce=ignore_ce nmi_watchdog=0 nowatchdog audit=0 selinux=0 pci=pcie_bus_safe hpet=disable nopku pti=off nopcid systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller pcie_aspm=off nospectre_v2 nospec_store_bypass_disable nospectre_v1 mitigations=off nokaslr nopti
• The following services are disabled:
• atd auditd, autovt@, blk-availability, crond, debug-shell, dnsmasq, ibacm, irqbalance, iscsi-onboot, iscsi, kvm_stat, mcelog, nfs-server, nfs-client, smartd, sssd, sysstat, mdcheck, mdmonitor, serial-getty@
• Network configuration is as follows:
• 2 Mellanox Technologies MT2894 Family [ConnectX-6 Lx] for internal communication between WEBserver & INFRAserver, 32 virtual functions were configured for each port.
• 3 Mellanox Technologies MT2894 Family [ConnectX-6 Lx], 1 Intel Corporation Ethernet Controller E810-XXV for SFP for the Web, App, DB, Mail,Infra and Batch VMs, 64 virtual functions were configured for each port.
• All the guest VMs use Virtual Functions.
• Each VM is given a virtual function that is routed to the NUMA node to which the VM's VCPUs are pinned.
• VMs are bound to host CPUs via their XML configuration in the following manner:
• All VMs of tiles 01, 02, 03, 04, 09, 10, 11, 12, 17, 18, 19, 20, 25, 26, 27, 28, 33, 34, 35, 36, 41, 42, 43, 44, 49, 50, 51 and 52 were bound to NUMA node 0.
• All VMs of tiles 05, 06, 07, 08, 13, 14, 15, 16, 21, 22, 23, 24, 29, 30, 31, 32, 37, 38, 39, 40, 45, 46, 47, 48, 53, 54, 55 and 56 were bound to NUMA node 1.
• Db VMs of tiles 01, 09, 17, 25, 33, 41 and 49 were bound to node 0
• Db VMs of tiles 05, 13, 21, 29, 37, 45 and 53 were bound to node 1
• VM configurations are as follows:
• All VMs are managed with libvirt and created using virsh create
• All VMs are created with the following xml template, exceptions noted below template:
• <domain type='kvm'>
• <name>$vm_name</name>
• <memory>$vm_memory</memory>
• <currentMemory>$vm_memory</currentMemory>
• <vcpu cpuset='$host-cpu-list'>$vm_cpu</vcpu>
• <numatune>
• <memory mode='strict' nodeset='$vm_node'/>
• </numatune>
• <os>
• <type arch='x86_64' machine='pc'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <acpi/>
• <apic/>
• <pae/>
• </features>
• <cpu mode='host-passthrough' check='none'>
• <topology sockets='1' cores='$vm_vcpus' threads='1'/>
• </cpu>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>destroy</on_crash>
• <devices>
• <emulator>/usr/libexec/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk' />
• <target dev='vda' bus='virtio'/>
• </disk>
• <interface type='hostdev' managed='yes'>
• <mac address='$mac_address'/>
• <driver name='vfio'/>
• <label>'ethx' </label>
• <source>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </interface>
• <memballoon model='none'>
• </devices>
• </domain>
• The following section is repeated for the second disk used in the infra, mail and db VMs for their data:
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vm_disk'/>
• <target dev='vdb' bus='virtio'/>
• </disk>
• The following section is added for app and db VMS:
• <memoryBacking>
• <hugepages>
• <page size='1048576' unit='KiB'/>
• </hugepages>
• </memoryBacking>
• VMs' nice values were setted after startup via the renice command to the following settings:
• app='-4'
• db='-8'
• web='-2'
• infra='19'
• mail='18'
• batch='19'
• After the system booted, the number of hugepages on numa nodes are set manually with the command: echo 364 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages, echo 364 > /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages
• The following items were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following item was added to /etc/rc.d/rc.local:
• ulimit -n 100000 > /dev/null 2>&1
• The following items were added to /etc/sysctl.conf:
• kernel.msgmax = 65536
• kernel.msgmnb = 65536
• kernel.shmall = 4294967296
• kernel.shmmax = 68719476736
• net.core.netdev_max_backlog = 32768
• net.core.rmem_default = 8388608
• net.core.rmem_max = 16777216
• net.core.somaxconn = 32768
• net.core.wmem_default = 8388608
• net.core.wmem_max = 16777216
• net.ipv4.conf.all.rp_filter = 1
• net.ipv4.conf.default.accept_source_route = 0
• net.ipv4.conf.default.rp_filter = 1
• net.ipv4.ip_forward = 1
• net.ipv4.ip_local_port_range = 32768 65000
• net.ipv4.ip_no_pmtu_disc = 1
• net.ipv4.neigh.default.gc_stale_time=120
• net.ipv4.neigh.default.gc_thresh2 = 1024
• net.ipv4.neigh.default.gc_thresh3 = 4096
• net.ipv4.tcp_fin_timeout = 1
• net.ipv4.tcp_max_orphans = 3276800
• net.ipv4.tcp_max_syn_backlog = 262144
• net.ipv4.tcp_max_syn_backlog = 65536
• net.ipv4.tcp_max_tw_buckets = 6000
• net.ipv4.tcp_mem = 94500000 915000000 927000000
• net.ipv4.tcp_synack_retries = 1
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_tw_reuse = 1
• net.ipv6.conf.all.disable_ipv6 = 1
• net.ipv6.conf.default.disable_ipv6 = 1
• vm.swappiness = 0
• kernel.numa_balancing = 0
• net.ipv4.conf.all.accept_redirects = 0
• kernel.core_uses_pid = 1
• kernel.panic_on_oops = 1
• net.ipv4.ip_local_reserved_ports=49152-49216

Web Server VM Notes

• The following boot options were used:
• nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce audit=0 elevator=noop clocksource=tsc tsc=reliable watchdog=0 hpet=disable nohz=on cgroup_disable=memory,cpu,blkio
• The following services are disabled:
• auditd,cloud-init,ebtables,httpd,irqbalance,kdump,rdisc,sssd,sysstat,tcsd,vmtools
• /etc/fstab changes:
• root file-system options: noatime,barrier=0
• nfs file-system defaults options: async,noatime,noacl,actimeo=14400,nocto
• MTU=9000 for eth1 (internal network).
• Contents for nginx configuration file /etc/nginx/nginx.conf:
• user nginx;
• worker_processes 4;
• worker_cpu_affinity 00000001 00000001 00000010 00000010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request'
• '"$status" $body_bytes_sent ';
• access_log off;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• keepalive_requests 10000;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• open_file_cache_errors on;
• read_ahead 4096;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• }
• Contents of nginx configuration file /etc/nginx/conf.d/default.conf:
• server {
• listen 80 backlog=8192;
• server_name localhost;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• Location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Contents of nginx configuration file /etc/nginx/conf.d/ssl.conf:
• server {
• listen 443 ssl http2 default_server;
• server_name localhost;
• ssl_certificate /etc/pki/nginx/new/localhost.crt
• ssl_certificate_key /etc/pki/nginx/new/localhost.key;
• ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_ciphers ALL:!kEDH!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• The following lines were changed in /etc/php-fpm.d/www.conf:
• listen = /dev/shm/php-cgi.sock
• listen.owner = nginx
• listen.group = nginx
• user = nginx
• group = nginx
• pm = static
• pm.max_children = 64
• pm.start_servers = 5
• pm.min_spare_servers = 5
• pm.max_spare_servers = 35
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness = 0
• net.ipv4.tcp_fin_timeout=60
• net.ipv4.tcp_tw_recycle=1
• net.core.netdev_budget = 3000
• net.core.netdev_max_backlog = 30000
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400
• The eth0 used adaptive-rx off rx-usecs 300 rx-frames 512; The eth1 used adaptive-rx off rx-usecs 300 rx-frames 512

Application Server VM Notes

• The following boot options were used:
• nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce clocksource=tsc selinux=0 elevator=noop audit=0 cgroup_disable=memory,cpu,blkio hpet=disable highres=off watchdog=0 tsc=realiable
• The following services are disabled:
• atd,auditd,avahi-daemon,brltty,cloud-init,cups,dnsmasq,ebtables,firewalld,irqbalance,iscsi,kdump,lvm2-monitor,mdmonitor,multipathd,podman,qemu-guest-agent,rdisc,rhsm,smartd,sssd,sysstat,tcsd,vdo,vgauthd,vmtools
• /etc/fstab changes:
• root file-system options: barrier=0
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator
• Heap size for AppSrv01 has been changed to 2880 MB
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• /etc/security/limits.conf changes:
• hard and soft memlock set to 50000000
• hard and soft nproc set to 100000
• * - nofile 1048576

Mail Server VM Notes

• The following boot options were used:
• nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce audit=0 elevator=noop clocksource=tsc tsc=reliable watchdog=0 hpet=disable nohz=on cgroup_disable=memory,cpu,blkio
• The following services are disabled:
• auditd,cloud-init,ebtables,irqbalance,kdump,rdisc,sssd,sysstat,tcsd,vmtools
• /etc/fstab changes:
• root file-system options: barrier=0
• datadisk file-system defaults options:noatime,barrier=0,data=writeback
• Dovecot was compiled with default options and the following dovecot.conf was used:
• protocols = imap pop3
• base_dir = /var/run/dovecot/
• the following 10-auth.conf was used:
• disable_plaintext_auth = no
• the following 10-logging.conf was used:
• log_path = /var/log/dovecot.log
• mail_debug = no
• the following 10-mail.conf was used:
• mail_location =dbox:~/
• mmap_disable = yes
• mail_fsync = never
• mdbox_rotate_size = 8192
• the following auth-system.conf.ext was used:
• passdb {
• args = scheme=plain /usr/local/etc/dovecot/passwd.dovecot
• driver = passwd-file:
• }
• userdb {
• driver = passwd
• }
• the following 10-master.conf was used:
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• service auth {
• user = root
• }
• the following 15-lda.conf was used:
• postmaster_address = postmaster@example.com
• Following items were added to /etc/sysctl.conf:
• net.ipv6.conf.all.disable_ipv6=1
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• ulimit -u 10240
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Database Server VM Notes

• The following boot options were used:
• nosoftlockup mce=ignore_ce selinux=0 audit=0 elevator=noop clocksource=tsc tsc=reliable watchdog=0 hpet=disable cgroup_disable=memory,cpu,blkio
• The following services are disabled:
• auditd,cloud-init,ebtables,firewalld,irqbalance,kdump,rdisc,sssd,sysstat,tcsd,vmtools
• /etc/fstab changes:
• root file-system options: barrier=0
• datadisk file-system defaults options:noatime,nodiratime,nobarrier
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness=0
• vm.nr_hugepages = 13312
• vm.hugetlb_shm_group = 901
• vm.nr_overcommit_hugepages=512
• kernel.sem = 500 64000 400 512
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• kernel.randomize_va_space=0
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• Database tuning options:
• db2set DB2_LARGE_PAGE_MEM=DB
• db2 alter bufferpool IBMDEFAULTBP size 5500000
• db2 alter tablespace roll_space NO FILE SYSTEM CACHING
• db2 alter tablespace C_cust_space NO FILE SYSTEM CACHING
• db2 alter tablespace C_supp_space NO FILE SYSTEM CACHING
• db2 alter tablespace C_site_space NO FILE SYSTEM CACHING
• db2 alter tablespace C_parts_space NO FILE SYSTEM CACHING
• db2 alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• db2 alter tablespace M_parts_space NO FILE SYSTEM CACHING
• db2 alter tablespace M_bom_space NO FILE SYSTEM CACHING
• db2 alter tablespace M_wo_space NO FILE SYSTEM CACHING
• db2 alter tablespace M_lo_space NO FILE SYSTEM CACHING
• db2 alter tablespace M_inv_space NO FILE SYSTEM CACHING
• db2 alter tablespace O_cust_space NO FILE SYSTEM CACHING
• db2 alter tablespace O_ords_space NO FILE SYSTEM CACHING
• db2 alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• db2 alter tablespace O_item_space NO FILE SYSTEM CACHING
• db2 alter tablespace S_comp_space NO FILE SYSTEM CACHING
• db2 alter tablespace S_sc_space NO FILE SYSTEM CACHING
• db2 alter tablespace S_po_space NO FILE SYSTEM CACHING
• db2 alter tablespace S_site_space NO FILE SYSTEM CACHING
• db2 alter tablespace S_supp_space NO FILE SYSTEM CACHING
• db2 alter tablespace U_space NO FILE SYSTEM CACHING
• db2 update database config for specdb04 using LOGFILSIZ 65536
• db2 update db cfg for specdb04 using logprimary 54 logsecond 4
• db2 update dbm cfg using DIAGLEVEL 0
• db2 update dbm cfg using NOTIFYLEVEL 0
• db2 update dbm cfg using MAX_QUERYDEGREE 1
• db2 update dbm cfg using CPUSPEED 2.676618e-07
• db2 update db cfg for specdb04 using CATALOGCACHE_SZ 2560
• db2 update db cfg for specdb04 using LOGBUFSZ 2560
• db2 update db cfg for specdb04 using UTIL_HEAP_SZ 364378
• db2 update db cfg for specdb04 using LOCKTIMEOUT 60
• db2 update db cfg for specdb04 using LOGFILSIZ 65536
• db2 update db cfg for specdb04 using LOGPRIMARY 54
• db2 update db cfg for specdb04 using MINCOMMIT 10
• db2 update db cfg for specdb04 using AUTO_RUNSTATS OFF
• db2 update db cfg for specdb04 using DATABASE_MEMORY 6100000
• /etc/security/limits.conf changes:
• db2inst1 soft data -1
• db2inst1 hard data -1
• db2inst1 soft nofile 65536
• db2inst1 hard nofile 65536
• db2inst1 soft fsize -1
• db2inst1 hard fsize -1
• db2inst1 soft nproc -1
• db2inst1 hard nproc -1
• * - nofile 1048576

Infraserver VM Notes

• The following boot options were used:
• nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce audit=0 clocksource=tsc tsc=reliable watchdog=0 hpet=disable nohz=on cgroup_disable=memory,cpu,blkio
• The following services are disabled:
• auditd,cloud-init,ebtables,firewalld,irqbalance,kdump,rdisc,sshd,sysstat,tcsd,vmtools
• /etc/fstab changes:
• root file-system options: barrier=0,noatime
• datadisk file-system defaults options:noatime,nodiratime
• MTU=9000 for eth1 (internal network).
• /etc/httpd/conf/httpd.conf changes:
• Listen 81
• ServerName besim:80
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.conf.all.arp_notify = 1
• net.ipv4.conf.default.arp_notify = 1
• net.ipv4.conf.eth0.arp_notify = 1
• net.ipv4.conf.lo.arp_notify = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness=0
• The following items were added to /etc/bashrc:
• ulimit -n 100000 > /dev/null 2>&1
• The eth0 used adaptive-rx off rx-usecs 300 rx-frames 512
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400
• The infra vms Used xfs file system (all other workload VMs used ext4).

Batch Server VM Notes

• The following boot options were used:
• nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce audit=0 elevator=noop clocksource=tsc tsc=reliable watchdog=0 highres=off hpet=disable nohz=on cgroup_disable=memory,cpu,blkio
• The following services are disabled:
• auditd,cloud-init,crond,ebtables,firewalld,irqbalance,kdump,rdisc,sssd,sysstat,tcsd,vmtools
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• /etc/security/limits.conf changes:
• hard and soft nofile set to 102400
• hard and soft nproc set to 102400

Client Driver Notes

• The client systems VMs using CCLinux release 21.08.2
• Two virtual clients were used for each tile,One virtual client per tile ran the webserver workload and the other virtual client per tile ran the other workloads.
• The first virtual client is the prime controller.
• Each prime client VM has 8 GB memory, and a 20 GB disk,1 Mellanox mlx5Gen Virtual Function device and 1 virtio_blk device are used.
• Each prime client has 4 VCPUs.
• Each webserver client VM has 6 GB memory, and a 20 GB disk,1 Mellanox mlx5Gen Virtual Function device and 1 virtio_blk device are used.
• Each web client has 4 VCPUs.

Other Notes

• An external 25Gb FutureMatrix 6865-48S8CQ-SI switch was used to connect the client systems to the system under test (SUT)
• The SUT used the six dual-port 25Gb PCIe NICs for communication with the VMs.
• The SUT NICs and the external client systems were connected to the switch using Fiber optic network cable

Performance Details: