SPECsfs2008_nfs.v3 Result

Avere Systems, Inc. : FXT 3800 (3 Node Cluster, Local ZFS Storage Config)
SPECsfs2008_nfs.v3 = 180538 Ops/Sec (Overall Response Time = 0.88 msec)


17859 0.3
35758 0.4
53680 0.5
71595 0.6
89506 0.7
107481 0.8
125649 1.0
143458 1.3
161102 1.7
180538 2.7
Performance Graph

Product and Test Information

Tested By Avere Systems, Inc.
Product Name FXT 3800 (3 Node Cluster, Local ZFS Storage Config)
Hardware Available July 2013
Software Available March 2014
Date Tested April 2014
SFS License Number 9020
Licensee Locations Pittsburgh, PA

The Avere Systems FXT 3800 Edge filer running AvereOS V3.2 provides NAS storage that enables performance scalability at the edge while leveraging both Legacy NAS and object-based Cloud storage capacity at the core. The AvereOS Hybrid NAS software dynamically organizes hot data into RAM, SSD and SAS tiers, retaining active data on the FXT Edge filer, and placing inactive data on the Core filer. The FXT Edge filer managed by AvereOS software provides a global namespace, clusters to scale to as many as 50 nodes, supports millions of I/O operations per second, and delivers over 100 GB/s of I/O bandwidth. The FXT 3800 is built on a 64-bit architecture that provides sub-millisecond responses to NFSv3 and CIFS client requests consisting of read, write, and directory/metadata update operations. The AvereOS Local Directories functionality enables the Avere Edge filer to immediately acknowledge all filesystem requests, and flush inactive directory data and user data to the Core filer. The tested Edge filer configuration consisted of (3) FXT 3800 nodes. The system also included (1) OpenIndiana ZFS server in the role of a single Core filer mass storage system exporting a single filesystem. In this configuration, Avere's AOS presents a single filesystem backed by the Core filer on the local network.

Configuration Bill of Materials

Item No Qty Type Vendor Model/Name Description
1 3 Storage Appliance Avere Systems, Inc. FXT 3800 Avere Systems Hybrid NAS Edge filer appliance running Avere OS V3.2 software. Includes (13) 600 GB SAS Disks and (2) 400GB SSD Drives.
2 1 Server Chassis Genstor CSRM-4USM24ASRA 4U 24-Drive Chassis for OpenIndiana Core filer storage NFS Server.
3 1 Server Motherboard Genstor MOBO-SX8DTEF Supermicro X8DTE Motherboard for OpenIndiana Core filer storage NFS Server.
4 2 CPU Genstor CPUI-X5620E Intel Westmere Quad Core 2.4 GHz 12MB L3 CPU for OpenIndiana Core filer storage NFS Server.
5 2 CPU Heat Sink Genstor CPUI-2USMP0038P SM 2U 3U P0038P Heat Sink for OpenIndiana Core filer storage NFS Server.
6 8 Memory Arrow MT36JSZF1G72PZ-1GD1 Micron DRAM Module DDR3 SDRAM 8GByte 240 RDIMM for OpenIndiana Core filer storage NFS Server.
7 1 Disk Drive Seagate ST500NM0011 Seagate 500GB Constellation ES SATA2 System Disk for OpenIndiana Core filer storage NFS Server.
8 9 Disk Drive Seagate ST4000NM0033 Seagate 4TB Constellation ES.3 SATA2 Disks for OpenIndiana Core filer storage NFS Server.
9 1 RAID Controller LSI LSI00209 LSI 4 internal/4 external port 6Gb/s SATA+SAS RAID card, x8 PCIe 2.0 host interface for OpenIndiana Core filer storage NFS Server.
10 1 Network Card Intel E10G42BTDA Intel 10 Gigabit AF DA Dual Port Server Adapter for OpenIndiana Core filer storage NFS Server.

Server Software

OS Name and Version Avere OS V3.2
Other Software Core filer storage server runs OpenIndiana SunOS 5.11 oi_151a, this package is available for download at http://dlc.openindiana.org/isos/151a/oi-dev-151a-x86.iso
Filesystem Software Avere OS V3.2

Server Tuning

Name Value Description
Writeback Time 12 hours Files may be modified up to 12 hours before being written back to the Core filer storage system.
buf.autoTune 0 Statically size FXT memory caches.
buf.InitialBalance.smallFilePercent 57 Tune smallFile buffer to use 57 percent of memory pages.
buf.InitialBalance.largeFilePercent 10 Tune largeFile buffers to use 10 percent of memory pages.
cfs.randomWindowSize 32 Increase the size of random IOs from disk.
cluster.dirMgrConnMult 12 Multiplex directory manager network connections.
dirmgrSettings.unflushedFDLRThresholdToStartFlushing 40000000 Increase the directory log size.
dirmgrSettings.maxNumFdlrsPerNode 120000000 Increase the directory log size.
dirmgrSettings.maxNumFdlrsPerLog 1500000 Increase the directory log size.
dirmgrSettings.balanceAlgorithm 1 Balance directory manager objects across the cluster.
tokenmgrs.geoXYZ.fcrTokenSupported no Disable the use of full control read tokens.
tokenmgrs.geoXYZ.trackContentionOn no Disable token contention detection.
tokenmgrs.geoXYZ.lruTokenThreshold 15400000 Set a threshold for token recycling.
tokenmgrs.geoXYZ.maxTokenThreshold 15500000 Set maximum token count.
vcm.readdir_readahead_mask 0x3000 Optimize readdir performance.
vcm.disableAgressiveFhpRecycle 1 Disable optimistic filehandle recycling.
vcm.readdirInvokesReaddirplus 0 Disable optimistic trigger of client readdir calls to readdirplus fetches.
initcfg:cfs.num_inodes 28000000 Increase in-memory inode structures.
initcfg:vcm.fh_cache_entries 16000000 Increase in-memory filehandle cache structures.
initcfg:vcm.name_cache_entries 18000000 Increase in-memory name cache structures.

Server Tuning Notes


Disks and Filesystems

Description Number of Disks Usable Size
Each FXT 3800 node contains (13) 600 GB 10K RPM SAS disks. All FXT data resides on these disks. 39 21.3 TB
Each FXT 3800 node contains (2) 400 GB eMLC SSD disks. All FXT data resides on these disks. 6 2.2 TB
Each FXT 3800 node contains (1) 250 GB SATA disk. System disk. 3 698.9 GB
The Core filer storage system contains (9) 4 TB SATA disk. The ZFS file system is used to manage these disks and the FXT nodes access them via NFSv3. 9 25.4 TB
The Core filer storage system contains (1) 500GB system disk. 1 465.7 GB
Total 58 50.0 TB
Number of Filesystems 1
Total Exported Capacity 22016 GiB (OpenIndiana Core filer storage system capacity)
Filesystem Type TFS (Tiered File System)
Filesystem Creation Options Default on FXT nodes.
OpenIndiana Core filer storage server ZFS filesystem created with commands:
  zpool create vol0 raidz2 <drive1-9 4TB from chassis>
  zfs create -o quota=21.5t -o sharenfs=root=@ vol0/cloudboi
Filesystem Config 7D+2P RAID6 (RAIDZ2) configuration on OpenIndiana Core filer storage server, with 1 RAID set in the zpool.
Fileset Size 20920.2 GB

Network Configuration

Item No Network Type Number of Ports Used Notes
1 10 Gigabit Ethernet 3 One 10 Gigabit Ethernet port used for each FXT 3800 Edge filer appliance.
2 10 Gigabit Ethernet 1 One 10 Gigabit Ethernet port used for the Core filer storage system.

Network Configuration Notes

Each FXT 3800 was attached via a single 10 GbE port to one Gnodal GS7200 72 port 10 GbE switch. The load generating client was attached to the same switch. The Core filer storage server was also attached to the same switch. A 1500 byte MTU was used throughout the network.

Benchmark Network

An MTU size of 1500 was set for all connections to the switch. The load generator was connected to the network via a single 10 GbE port. The SUT was configured with 3 separate IP addresses on one subnet. Each cluster node was connected via a 10 GbE NIC and was sponsoring 1 IP address.

Processing Elements

Item No Qty Type Description Processing Function
1 6 CPU Intel Xeon CPU E5645 2.40 GHz Hex-Core Processor FXT 3800 Avere OS, Network, NFS/CIFS, Filesystem, Device Drivers
2 2 CPU Intel Xeon E5620 2.40 GHz Quad-Core Processor OpenIndiana Core filer storage systems

Processing Element Notes

Each file server has two physical processors.


Description Size in GB Number of Instances Total GB Nonvolatile
FXT 3800 System Memory 144 3 432 V
Core filer storage system memory 64 1 64 V
FXT 3800 NVRAM 2 3 6 NV
Grand Total Memory Gigabytes     502  

Memory Notes

Each FXT node has main memory that is used for the operating system and for caching filesystem data. Each FXT contains two (2) super-capacitor-backed NVRAM modules used to provide stable storage for writes that have not yet been written to disk.

Stable Storage

The Avere filesystem logs writes and metadata updates to the PCIe NVRAM modules. Filesystem modifying NFS operations are not acknowledged until the data has been safely stored in NVRAM. In the event of primary power loss to the system under test (SUT), the charged super-capacitor provides power to the NVRAM circuitry to commit the NVRAM log data to persistent flash memory on the NVRAM card. The NVRAM log data is indefinitely persisted to on-board flash memory. When power is restored to the SUT, the NVRAM log data is restored from the persistent on-board flash to the NVRAM subsystem, and during startup, AvereOS replays this log data to the on-disk filesystem thus assuring a stable storage implementation. The above described implementation handles cascading power failures in the same manner: persist NVRAM to stable flash upon power failure, restore NVRAM log data from flash and replay logs to filesystem upon power-on. The Core filer is ZFS-based, thus the Stable Storage requirement is guaranteed by the ZFS Intent Log (ZIL) which logs writes and other filesystem changing transactions to the disk drives. Writes and other filesystem changing transactions are not acknowledged until the data is written to stable storage.

System Under Test Configuration Notes

The system under test consisted of (3) Avere FXT 3800 nodes. Each node was attached to the network via 10 Gigabit Ethernet. Each FXT 3800 node contains (13) 600 GB SAS disks and (2) 400GB eMLC SSD drives. The OpenIndiana Core filer storage system was attached to the network via a single 10 Gigabit Ethernet link. The Core filer storage server was a 4U Supermicro server configured with a software RAIDZ 7+2 RAID6 array consisting of (9) 4TB SATA disks.

Other System Notes


Test Environment Bill of Materials

Item No Qty Vendor Model/Name Description
1 1 Supermicro SYS-1026T-6RFT+ Supermicro Server with 48GB of RAM running CentOS 6.4 (Linux 2.6.32-358.0.1.el6.x86_64)
2 1 Gnodal GS7200 Gnodal 72 Port 10 GbE Switch. 72 SFP/SFP+ ports

Load Generators

LG Type Name LG1
BOM Item # 1
Processor Name Intel Xeon E5645 2.40GHz Hex-Core Processor
Processor Speed 2.40 GHz
Number of Processors (chips) 2
Number of Cores/Chip 6
Memory Size 48 GB
Operating System CentOS 6.4 (Linux 2.6.32-358.0.1.el6.x86_64)
Network Type Intel Corporation 82599EB 10-Gigabit SFI/SFP+

Load Generator (LG) Configuration

Benchmark Parameters

Network Attached Storage Type NFS V3
Number of Load Generators 1
Number of Processes per LG 768
Biod Max Read Setting 2
Biod Max Write Setting 2
Block Size 0

Testbed Configuration

LG No LG Type Network Target Filesystems Notes
1..1 LG1 1 /zfsLocal LG1 node is connected to the same single network switch.

Load Generator Configuration Notes

The load generator client was mounted against the single filesystem on all FXT nodes.

Uniform Access Rule Compliance

The load-generating client hosted 768 processes. The assignment of 768 processes to 3 network interfaces was done such that they were evenly divided across all network paths to the FXT appliances. The filesystem data was evenly distributed across all disks and FXT appliances and Core filer storage servers.

Other Notes


Config Diagrams

Generated on Wed May 14 12:18:20 2014 by SPECsfs2008 HTML Formatter
Copyright © 1997-2008 Standard Performance Evaluation Corporation

First published at SPEC.org on 14-May-2014