One important consideration for deploying an application in a new environment is performance. In the case of Amazon Web Services, many organizations are considering building new applications there, migrating existing applications, or using AWS as a target environment for disaster recovery. In all of these cases, you’ll want to do a minimum set of performance testing, for two big reasons.
- To ensure that your application’s performance will meet your needs
- To help determine the most cost effective (cheapest!) AWS services you can use
The goal of this post is to help you quickly get started with performance testing of Amazon’s block storage service. To get the test running quickly and with minimal steps, we will use AWS CloudFormation to build the test environment and run the tests.
Note: Not all storage performance tests are created equal! This is a good starting point, but storage performance is a deep subject. Ultimately, the best way to test is by measuring the performance of your application.
- An AWS account in which you can afford to accrue a few dollars in usage charges
- Internet access and your local machine’s public IP address
Amazon Elastic Block Store (EBS)
Amazon has quite a few storage related services. The most general purpose of these is EBS, which presents a block device to your Elastic Compute Cloud (EC2) instances. (“EC2 instance” is Amazon’s name for a virtual machine).There are currently three types of EBS volumes, which can be up to 16TB in size and have varying IOPS characteristics. (IOPS, or Input/Output Operations per Second, is a common unit of measurement for storage performance.)
- Magnetic (“standard”) – Magnetic disk backed volumes with a fixed cost per GB and a variable cost for IOPS used
- General Purpose (“gp2″) – SSD backed volumes with 3 IOPS per GB (burstable up to 3000 IOPS at volume sizes less than 1TB) and a fixed cost per GB.
- Provisioned IOPS (“io1″) – SSD backed volumes with consistent IOPS (up to 20,000 per volume), a per-GB cost, and an IOPS level cost.
See the EBS documentation to learn more about volume types, such as how bursting works for gp2 volumes.
EC2 Instance types
“EC2 Instance” is Amazon’s term for virtual machine. (“EC2” stands for “Elastic Compute Cloud”). EC2 Instance types are configured with different levels and ratios of CPU, memory, network performance, I/O performance, and local storage options. There are currently over 40 instance types to choose from, so to keep things simple we will use a small subset of them here.
Since we don’t have a particular application in mind for this test, we’ll gear it to learning how different combinations of EBS volume types and EC2 instance types perform. In particular, we’ll test all three EBS volume types, with the Provisioned IOPS volume set to the maximum possible IOPS; and we’ll test with EC2 instance types at three different performance levels.The following table shows the EC2 instance types we’ll test with.
|EC2 instance type||VCPUs||Memory||I/O||EBS optimized||cost per hour|
|c3.2xlarge||8||15GB||High / 1000Mbps||yes||$0.42|
The next table shows the EBS volume types we’ll test with.
|EBS volume type||size||IOPS (burst IOPS)||cost per hour|
Finally, here’s the table we will populate with our results, which for this test will be in MB/s (megabytes per second).
Our test is very simple: it uses the unix dd tool to copy 1GB of data from /dev/zero into a test file, using a 1MB block size. This test should give results that are close to the highest possible throughput for a single process.
We can create the EC2 instances and run the tests using the AWS CLI or one of the many SDKs, but for this post we will use the AWS Console. Once you are logged in to console, pick a region from the drop-down menu in the upper right, or just use the default.
Run the tests
For each of the three instance types (m3.medium, c3.2xlarge, c3.8xlarge), we will do the following
- Create the CloudFormation stack
- Record the results
- Delete the CloudFormation stack
Create the CloudFormation stack
The bulk of the work is done by AWS CloudFormation. Using a custom template, CloudFormation will do the following for us:
- Create an EC2 instance, including the three EBS volume types we want to test
- Install the Apache web server
- Create and mount filesystems on the three EBS volumes
- Run a set of simple throughput tests
- Record the test results in a place that Apache can serve them
- Provide a link to the test results
Here’s the procedure for creating the CloudFormation stack:
- Download the aws-storage-test.template CloudFormation template to your local machine
- From the AWS Console main menu, select CloudFormation from the Services menu in the top left
- Click the blue “Create Stack” button in the top left.
- In the “Name” input, type “Perftest1″ for the first test, “Perftest2″ for the second, etc.
- Under Source, select “Upload a template to Amazon S3″, click “Choose File”, and find the aws-storage-test.template file that was downloaded above
- In the “Specify Parameters” screen, enter your local machine’s public IP address followed by /32 in the AllowedIpPrefix field, and select the InstanceType from the drop-down
Note: you only need to enter a KeyPairName if you want to be able to log in to the instance for troubleshooting or other admin tasks.
- Move through the next two screens by clicking Next, Next and Create.
After clicking Create, we arrive at the list of stacks. The most recent stack (“Perftest1″, “Perftest2″ or “Perftest3″) is marked “CREATE_IN_PROGRESS”. If we select our stack, we can monitor its build progress in the Events table in the bottom pane.
The stack is done once the EC2 instance boots, completes some configuration tasks, and finishes running performance tests on all three volumes. The entire process should take anywhere from 5 to 10 minutes.
Here are our test results.
- standard: 1024 MB in 47.62 seconds, 21.50 MB/s
- gp2: 1024 MB in 33.48 seconds, 30.58 MB/s
- io1: 1024 MB in 33.12 seconds, 30.91 MB/s
- standard: 1024 MB in 30.80 seconds, 33.24 MB/s
- gp2: 1024 MB in 10.54 seconds, 97.15 MB/s
- io1: 1024 MB in 9.75 seconds, 105.02 MB/s
- standard: 1024 MB in 33.47 seconds, 30.59 MB/s
- gp2: 1024 MB in 9.76 seconds, 104.91 MB/s
- io1: 1024 MB in 4.79 seconds, 213.77 MB/s
Delete CloudFormation stacks
Once the tests are done, let’s not forget to delete our CloudFormation stacks, otherwise we’ll continue to accrue hourly charges for the EC2 instances and EBS volumes!
Back in the AWS Console, simply select each stack name and click the “Delete Stack” button.
Here are the results in MB/s (megabytes per second).
|Instance type ->
Performance generally increases as the instance size increasesHere are some observations we can draw from the tests.
- Throughput for standard, gp2 and io1 volumes appears to be constrained by the m3.medium instance
- Throughput for io1 volumes appear to be constrained by the c3.2xlarge instance
And some conclusions.
- It may not make sense to use gp2 or io1 volumes on smaller instance types since we won’t get the full benefit of their performance
- Moving to larger instance sizes doesn’t help much when using standard volumes, or in some cases even gp2
Finally, here are some next steps we might want to take to learn more.
- Repeat the same tests to see if we get consistent performance
- Test with more EC2 instance sizes to determine the most cost effective instance we need to get full throughput from gp2 and io1 volumes
- Test in different regions and availability zones to determine if there are any differences
- Record IOPS in addition to throughput
- Adjust the block size to be the same as your application (or a variety of block sizes if you don’t know yours), because smaller block sizes generally lead to higher IOPS but lower throughput.