.. _elasticity_measurements:
*******************************************************************
Running Elasticity + Scalability For the First Time With Your Cloud
*******************************************************************
Setting up Parameters
=====================
Provision Retries
-----------------
If cloud under test is baremetal or instances take a long time to provision,
the tester can adjust the update_attempts in osgcloud_rules.yaml::
vm_defaults:
update_attempts: 60
update_frequency: 5
The values for update_attempts and update_frequency determine whether
benchmark considers the provisioning to have failed. A tester can set
them to a small value to force provisioning failures for testing.
During a compliant run, the value for update_attempts is automatically
calculated based on the max of the average AI provisioning time of YCSB
and KMeans.
Using Workloads
---------------
The tester can configure which workloads to run during testing of elasticity + scalability phase
by modifying the following parameter::
workloads: [ ycsb, kmeans ]
Setting Max AIs and Ignoring QoS
--------------------------------
The tester can set the maximum AIs to be provisioned to a specific value or number of AIs from which results is received to a specific value, and ignore the stopping conditions during testing of elasticity + scalability phase. These parameters are controlled as follows::
maximum_ais: 24
reported_ais: 14
ignore_qos_when_max_is_set: false
Running
=======
Elasticity + scalability phase is run as follows. It assumes that CBTOOL is already
running and is connected with your cloud.::
python osgcloud_elasticity.py --exp_id SPECRUN
If you run the elasticity + scalability phase as::
python osgcloud_elasticity.py --exp_id SPECRUNID
It assumes that a baseline_SPECRUNID.yaml files exists in ~/results/SPECRUNID/perf .
The results and logs of elasticity + scalability phase are present in the following directory.::
ls ~/results/SPECRUNID/perf
Following files and directories are generated as a result of a successful
elasticity + scalability phase run::
elasticity_SPECRUNID.yaml
osgcloud_elasticity_SPECRUNID-20150811234204UTC.log
SPECRUNIDELASTICITY20150811234204UTC
Where the timestamp in file and directory names will change based on the
date/time the elasticity + scalability phase was run.
Preparing Environment Parameters for Submission File
====================================================
The tester must set appropriate values in osgcloud_environment.yaml file.
The key/value pairs from this file are dumped into the submission file.
Generating Submission File
==========================
The program osgcloud_fdr.py operates on the performance data stored
in ``~/results/SPECRUN/perf`` , and the environment file.
The tester should fill out the details of its cloud environment (e.g.,
physical machines for whitebox cloud) in osgcloud_environment.yaml .
The file should be present in ``~/results/SPECRUN/perf`` .
The submission file generation program is then run as follows (assuming
your SPECRUN id was SPECRUNID)::
python osgcloud_fdr.py --baseline_yaml ~/results/SPECRUNID/perf/baseline_SPECRUNID.yaml --elasticity_results_path ~/results/SPECRUNID/perf/SPECRUNIDELASTICITY*** --exp_id SPECRUNID --environment_yaml osgcloud_environment.yaml
Where *** indicates the timestamp that forms of the elasticity results
directory name.
The output of this phase is as the following::
osgcloud_fdr_SPECRUNID-20150811234502UTC.log
fdr_ai_SPECRUNID.yaml
sub_file_SPECRUNID.txt
Generating HTML Report
======================
Assuming the results directory was ``~/result``s , the submission file is present
in the following directory::
~/results/EXPID/perf/sub_file_EXPID.txt
If the elasticity experiment was run successfully, the HTML report
can be generated as::
cd ~/osgcloud/driver
python osgcloud_fdr_html.py --exp_id EXPID
For whitebox clouds, the tester must include an architecture diagram of
the cloud in the PNG format. It can be included in the final HTML report
as follows::
python osgcloud_fdr_html.py --exp_id EXPID --networkfile arch.png
The resulting file is generated in::
~/osgcloud/driver
The name of the file is::
FDR_EXPID.html
Tips on Running the Elasticity + Scalability Phase
==================================================
* Before starting elasticity + scalability phase, it is best to restart CBTOOL if another
elasticity + scalability phase was run. There is no need to restart CBTOOL if only
baseline phases were run.
* When testing elasticity + scalability phase, it is best to start with small number of AIs. Set the number of AIs to 6 or 8.
* It is not necessary to run baseline phase every time before elasticity + scalability phase.
However, elasticity + scalability phase needs the output of baseline phase (as a YAML) file
to determine the stopping condition.
Assuming the experiment ID was EXPID and the result directory was ``~/results`` ,
and the baseline was already run, the tested can do the following to
repeat the elasticity + scalability phase::
cd ~/results/EXPID/perf/
rm -rf *ELASTICITY*; rm -f *elasticity*.log
* The instance provisioning time of a cloud under test may be long. The
time until CBTOOL waits to check for instance provisioning can be
adjusted using update_attempts parameter in osgcloud_rules.yaml file.
It should be set to a value
such that ( update_attemps x update_frequency ) never exceeds three times
the maximum
baselne AI provisioning time of YCSB or KMeans workloads. For a compliant run,
it must be set before the start of the experiment.
* If any errors are encountered during an elasticity + scalability phase run, they can be checked in CBTOOL log::
cd /var/log/cloudbench
grep ERROR *
The tester can check for errors for specific AI by searching for an AI number.
* The errors can also occur during an AI_run for a workload. Examples of errors include::
Cassandra: Create, Remove, List keyspace fail, or seeds fail to form a cluster.
YCSB: Data generation fails
Hadoop: Hadoop slaves fail to form a cluster
KMeans: Data generation fails
* Instance names also include AI numbers. This can come handy when you
are debugging elasticity + scalability phase in your cloud.
* The tester can check for failed AIs by running the following command at the CBTOOL prompt::
MYSIMCLOUD> ailist failed
.. _testing_instance_supporting_evidence:
Testing Instance Supporting Evidence Collection
===============================================
To test that instance supporting evidence collection works properly, please follow these steps:
* Start CBTOOL and ensure that it can connect to your cloud and launch instances.
* Ensure that workload images have been created.
* Launch an application instance of Cassandra or Hadoop::
aiattach cassandra_ycsb
aiattach hadoop
* Determine the Linux username and SSH key path for instances. Assuming the
Linux user name is cbuser and SSH key path is ``~/osgcloud/cbtool/credentials/cbtool_rsa``
* Test supporting evidence collection for an instance. Run the supporting evidence instance script on CBTOOL machine
to collect supporting evidence for an instance.
Create a directory where the results are stored::
mkdir /tmp/instance -p
Run the supporting evidence collection script::
cd ~/osgcloud/driver/support_script/
./collect_support_data.sh remote_vm_sysinfo 10.146.5.41 cbuser ~/osgcloud/cbtool/credentials/cbtool_rsa /tmp/instance/
SCRIPT INSTANCE/CASSANDA/HADOOP IPADDR IPOFINSTANCE LINUXUSER SSHKEYPATH TARGETDIR
OUTPUT:
tree /tmp/instance
|-- date.txt
|-- df.txt
|-- dpkg.txt
|-- etc
| |-- fstab
| |-- hosts
| |-- iproute2
| | |-- ematch_map
| | |-- group
| | |-- rt_dsfield
| | |-- rt_protos
| | |-- rt_realms
| | |-- rt_scopes
| | `-- rt_tables
| |-- nsswitch.conf
| |-- security
| | `-- limits.conf
| `-- sysctl.conf
|-- hostname.txt
|-- ifconfig.txt
|-- lspci.txt
|-- mount.txt
|-- netstat.txt
|-- ntp.conf
|-- proc
| |-- cmdline
| |-- cpuinfo
| |-- devices
| |-- meminfo
| |-- modules
| |-- partitions
| |-- swaps
| `-- version
|-- route.txt
`-- var
`-- log
`-- dmesg
* Test supporting evidence collection for YCSB and Cassandra.
Find the IP address of instance with YCSB role from CBTOOL (by typing vmlist on CBTOOL CLI). Then run the following commands::
mkdir /tmp/ycsb -p
./collect_support_data.sh remote_vm_software 10.146.5.41 cbuser ~/osgcloud/cbtool/credentials/cbtool_rsa /tmp/cassandra cassandra_ycsb
OUTPUT from machine with YCSB role:
$ tree /tmp/ycsb
/tmp/ycsb/
|-- javaVersion.out
|-- role
`-- YCSB
|-- custom_workload.dat
`-- workloads
|-- workloada
|-- workloadb
|-- workloadc
|-- workloadd
|-- workloade
|-- workloadf
`-- workload_template
Find the IP address of an instance with SEED role from CBTOOL (by typing vmlist on CBTOOL CLI). Then run the following commands::
mkdir /tmp/seed -p
./collect_support_data.sh remote_vm_software 10.146.5.41 cbuser ~/osgcloud/cbtool/credentials/cbtool_rsa /tmp/seed cassandra_ycsb
OUTPUT:
$ tree /tmp/cassandra
/tmp/
|-- cassandra
| |-- du_datadir
| |-- du_datadir_cassandra
| |-- du_datadir_cassandra_usertable
| |-- nodetool_cfstats
| `-- nodetool_status
|-- cassandra_conf
| |-- cassandra-env.sh
| |-- cassandra-rackdc.properties
| |-- cassandra-topology.properties
| |-- cassandra-topology.yaml
| |-- cassandra.yaml
| |-- commitlog_archiving.properties
| |-- logback-tools.xml
| |-- logback.xml
| `-- triggers
| `-- README.txt
|-- javaVersion.out
`-- role
* Testing supporting evidence collection from Hadoop.
Find the IP address of instance with HADOOPMASTER role from CBTOOL (by typing vmlist on CBTOOL CLI). Then run the following commands::
mkdir /tmp/hadoop -p
./collect_support_data.sh remote_vm_software 10.146.5.41 cbuser ~/osgcloud/cbtool/credentials/cbtool_rsa /tmp/hadoop hadoop
OUTPUT from machine with HADOOPMASTER role:
tree /tmp/hadoop/
/tmp/hadoop/
|-- hadoop
| |-- datahdfs
| |-- dfsadmin_report
| |-- du_datanodedir
| |-- du_namenodedir
| |-- input_hdfs_size
| |-- output_hdfs_size
| `-- version
|-- hadoop_conf
| |-- capacity-scheduler.xml
| |-- configuration.xsl
| |-- container-executor.cfg
| |-- core-site.xml
| |-- hadoop-env.cmd
| |-- hadoop-env.sh
| |-- hadoop-metrics2.properties
| |-- hadoop-metrics.properties
| |-- hadoop-policy.xml
| |-- hdfs-site.xml
| |-- httpfs-env.sh
| |-- httpfs-log4j.properties
| |-- httpfs-signature.secret
| |-- httpfs-site.xml
| |-- kms-acls.xml
| |-- kms-env.sh
| |-- kms-log4j.properties
| |-- kms-site.xml
| |-- log4j.properties
| |-- mapred-env.cmd
| |-- mapred-env.sh
| |-- mapred-queues.xml.template
| |-- mapred-site.xml
| |-- mapred-site.xml.template
| |-- masters
| |-- slaves
| |-- ssl-client.xml.example
| |-- ssl-server.xml.example
| |-- yarn-env.cmd
| |-- yarn-env.sh
| `-- yarn-site.xml
|-- javaVersion.out
`-- role