Glossary of terms (sorted alphabetically)

Application Instance

A group of instances created to run a single workload together. An application instance comprises a workload driver instance and set of instances, which are stressed by the workload driver. SPEC Cloud (tm) IaaS 2016 benchmark uses multiple application instances during specific phases to determine elasticity and scaling.

Baseline phase

In baseline, peak performance for each workload is determined in five separate test runs. During each workload run, instances are provisioned, data set is generated, load generator is started, results are accumulated, and the instances are destroyed. The workloads are run in a sequential fashion. Data from the baseline phase is used to establish parameters for the Elasticity + Scalability phase.

Benchmark phases

Benchmark has two phases, namely baseline and elasticity + scalability.

Blackbox Cloud

A cloud-provider provides a general specification of the SUT, usually in terms of how the cloud consumer may be billed. The exact hardware details corresponding to these compute units may not be known. This will typically be the case if the entity benchmarking the cloud is different from a cloud provider.

Cloud (system or service)

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

Cloud Consumer

A person or organization that is a customer of a cloud; note that a cloud customer may itself be a cloud and that clouds may offer services to one another.

Cloud Provider

An organization that provides cloud services to customers who pay for only the computing time and services actually used.

Elasticity

Elasticity measures whether the work performed by application instances scales linearly in a cloud. That is, for statistically similar work, the performance of N application instances in a cloud must be the same as the performance of application instances during baseline phase when no other load is introduced by the tester. Elasticity is expressed as a percentage (out of 100).

Elasticity + scalability phase

In the Elasticity + Scalability phase, new application instances are created, and they run the workloads concurrently to determine the elasticity and scalability metrics. The benchmark reports are generated at the end of elasticity + scalability phase.

Infrastructure as a Service (IaaS)

The Service Provider gives the Cloud-Consumer the capability to the provision processing, storage, network, and basic computing resources. They can also deploy and run arbitrary operating systems. The End-Consumer does not manage or control the underlying physical cloud infrastructure, but has control over the operating system, assigned storage, deployed applications, and limited control of select networking components (e.g., host firewalls).

Instance

An instance is an abstracted execution environment which presents an operating system (either discrete or virtualized). The abstracted execution environment presents the appearance of a dedicated computer with CPU, memory and I/O resources available to the operating system. In SPEC Cloud, an instance consists of a single OS and the application software stack that supports a single SPEC Cloud component workload. There are several methods of implementing an instance, including physical machines, virtual machines, or containers. An instance is created or destroyed using an API provided by an IaaS cloud.

Instance image

An image on the disk from which an instance is provisioned. Common formats for instance image include QCOW2 (Qemu copy on write 2), RAW, or AMI (Amazon machine image)

Physical machine

A set of connected components consisting on one or more general purpose processors (CPUs), memory, network connectivity, and mass storage either local (disk) or remote (network attached, block storage). The physical machine can have its own physical packaging or be installed as a blade in a blade chassis. An example would be a multi-core server with 4 GB of memory, 250 GB disk, and 1 Gb/s network adapter.

Provisioning Time

The measured time needed to bring up a new instance, or add more resources (like CPU or storage) to an existing instance.

Private Cloud

The cloud infrastructure is provisioned for exclusive use by a single organization comprising single or multiple consumers (e.g., business units). It may be owned, managed, and operated by the organization, a third party, or some combination of them, and it may exist on or off premises.

Public Cloud

The cloud infrastructure is provisioned for open use by the general public. It may be owned, managed, and operated by a business, academic, or government organization, or some combination of them. It exists on the premises of the cloud provider.

Quality of Service (QoS)

The minimum percent (e.g., 95%) of collected values that complete within a predefined threshold in each measurement category. The Run and Reporting Rules document contains specific QoS limitations for each workload and benchmark.

Response Time

The time between when the work item request is issued until the corresponding completion condition. This definition is identical to the YCSB Latency metric.

Scalability

Scalability measures the total amount of work performed by application instances running in a cloud. The aggregate work performed by one or more application instances should linearly scale in an ideal cloud. It answers the question: How much more gets done if N instead of one (1) application instances? Ideally, each additional application instance will contribute 100% of its individual capability, but this may not be the case as the cloud gets loaded with multiple application instances or other workloads (for public clouds). Scalability is reported as a unit-less number @ the number of valid application instances. The number is an aggregate of workloads metrics across all application instances running in a cloud normalized by workload metrics in a reference cloud platform. The reference platform metrics are an average of workload metrics taken across multiple cloud platforms during the benchmark development.

SUT

One or more cloud services under test. This includes all hardware, network, base software and management systems used for the cloud service.

Variability

The difference between different runs of the benchmark. Typically, variability arises due to factors such as multi-tenancy and time of day execution, and may be more pronounced on public clouds.

Whitebox Cloud

The SUT’s exact engineering specifications including all hardware and software are known and under the control of the tester. This will typically be the case for private clouds.