Before Setting up the Benchmark

The tester must consider the following before running the benchmark.

Where does the CBTOOL Run?

CBTOOL and benchmark drivers must be set up together on the same machine. This machine is hereafter referred to as the benchmark harness machine.

If the cloud under test is whitebox, the benchmark harness machine must not be part of the cloud.

If the cloud under test is blackbox, the benchmark harness machine should not share a physical machine with the instances to the extent possible.

A diagram depicting the harness and instances is shown below.

Timezone and NTP Server For Instances and Benchmark Harness Machine

It is important that the benchmark harness machine and the benchmark test instances have a very similar idea of “time.” By default, benchmark instances will be configured to get time via NTP (Network Time Protocol) from the benchmark harness machine. The benchmark harness machine must therefore be able to serve time via NTP, which means it too must be time synchronized - for example by being configured to obtain time from some number of NTP servers.

If you wish to have benchmark test instances obtain time from NTP server(s) other than the benchmark harness machine, that is accomplished by editing the osgcloud_rules.yaml configuration file present in ~/osgcloud/driver/ directory. A comma separated list of NTP servers can be specified in the following parameter:

timeserver: NTP_Server1_IP_Address_Or_Hostname,NTP_Server2_IP_Address_Or_Hostname

It is highly recommended that the timezone of the benchmark harness machine and instances be UTC. Please note that to setup timezones in instance images, a separate configuration is required that may vary across operating system distributions.

Caution about NTP servers

The benchmark test instances must be able to synchronize time with one or more NTP servers. If the instances are unable to reach the NTP server(s), or synchronize time with them, then “baseline” and “elasticity” phases of the benchmark may hang.

A Typical Flow for Running the Benchmark

A tester may use the following flow as a reference for the overall testing process.