The Tests
For over two years, Nasuni has been testing the largest cloud service providers with a series of five cascading tests. CSPs had to pass the first test in order to move on to the next. CSPs that failed a test dropped out of the field of contenders.
Who Passed
Amazon S3AT&T Synaptic Storage as a ServiceMicrosoft Azure NirvanixPeer1 HostingRackspace Cloud
API Integration Test
The most fundamental test put to the cloud storage field was whether or not it was possible to build a connector for the service. Nasuni developed a custom connector in Python according to each CSP’s API documentation and tested within Python’s framework to see if it worked.
API Integration Results
Only one CSP failed the API test, but Nasuni reports that a number of others took multiple tries and extensive work with CSP tech support over the course of several months to get the CSPs’ APIs to work.
Unit Testing Test
Nasuni broke down large software components into units and tested for inputs, outputs and error cases, looking for whether the CSPs could write and read different file sizes, offer access from multiple locations at the same time, and write to the cloud with one connection and read the same file back from a different connection.
Unit Testing Results
Five of the remaining field failed this test, many because they were unable to carry out a simple read-write-delete operations. Among those who dropped out at this stage, many were those built with architectures designed for less transactional operations, such as archiving files or storing photos.
Performance Test
The third test measured response time and throughput, with different variables tested such as concurrency, object size and workload type. To pass this benchmarking test, CSPs could not break under testing or ask Nasuni to stop the testing because it was overstressing their infrastructure.
Performance Results
Among the remaining ten CSPs to make it this far, two more failed due to performance levels that would be unacceptable to typical corporate users. Amazon S3 posted the most consistently fast service across all file types, though Nirvanix was 17 percent faster than Amazon at reading large files and Microsoft Azure was 12 percent faster at writing files.
Stability Test
This test checked out long-term reliability by repeating the following three steps every five minutes for an extended period of time: writing a file to the cloud, infrequently reading back a file from a previous write and comparing it against expected data and deleting the old file. CSPs had to perform with no data loss or unplanned outages to pass.
Stability Results
None of the eight providers that made it to this test failed. Amazon S3 and Microsoft Azure had the fastest response time, and Nirvanix the slowest. Amazon S3 also had essentially 100 percent availability, with Microsoft Azure, Peer1 and Rackspace coming in with 99.9 percent availability. Nirvanix and AT&T had 99.8 percent and 99.5 percent availability, respectively, with the former experiencing a high number of outages compared to the rest and AT&T having fewer but longer outages.
Scalability Testing
Nasuni wrote 1KB files with many threads of concurrency for several weeks or until the number of objects reached 100 million objects and then tracked error rates and performance. If CSPs crossed the 100 million object threshold without performance issues, they passed.
Scalability Testing
Two more CSPs failed at this final stage of testing. Four of the service providers–Amazon, Microsoft Azure, Nirvanix and Rackspace–experienced a write error rate of zero. AT&T and Peer 1 had a miniscule 0.01 percent write error rate. Amazon was the only CSP to experience a zero read error rate, with Microsoft close behind at 0.01 percent. Rackspace was the laggard in this category, experiencing a 59.63 percent read error rate.