August 24, 2006
Web Technology Department
Amazon Simple Storage Service (Amazon S3)
Everybody says Google is going to build a web-based operating system, but Amazon seems to be doing it too, with their Amazon Web Services line of products.
For example, there's Amazon Simple Storage Service (Amazon S3), which implements a very simple store-a-file/get-a-file service. Amazon says it's fast and designed for 99.99% availability, with data centers in two locations. Storage costs you 15 cents per gigabyte per month, and bandwidth costs 20 cents for every gigabyte transferred in or out. There's no minimum charge.
Some companies have already decided that buying bandwidth and space from Amazon is cheaper than expanding their data centers. The smugmug photo sharing service uses S3 for storing redundant copies of all images, and Altexa Software used S3 to implement an over-the-internet data backup service. Both of them added S3 to their services in just a few days.
Calls to the S3 API can be made using REST or SOAP interfaces, and are authenticated with strong encrypted signing of the HTTP header, including the timestamp field to prevent replay attacks.
Each file stored in S3 can have its own security policy. If a file is set to be readable by everyone, then signed headers are unnecessary and S3 will serve up the file in response to an ordinary HTTP request, qualified by the "Host" header, allowing S3 to be used for virtual hosting of static content.
The image of the Chicago lakefront at the top of this page is currently hosted at Amazon. It took me a couple of hours of mucking around with Perl scripts to write a program to put the file out there, but somebody who actually knows Perl could probably have done it in a few minutes. (I could have done it in a few minutes in .NET, but I wanted Perl scripts so they'll run on the Windypundit host system.)
If you need to serve large files to lots of people, the S3 servers will also act as a seeder site for BitTorrent.
I think anybody implementing a web business that stores large amounts of static data is going to have to include S3 (and its eventual competitors) in their planning.
.
Leave a comment