S3 can't process files, so it's not a place to place scripts that need server-side execution. Thankfully, Amazon provides Elastic Computer Cloud (EC2). For $0.10 an hour (plus some nominal bandwidth charges) you get a fully accessible linux box that you can configure to your heart's content. I recommend starting with the brilliant Eric Hammond's public Ubuntu server images from alestic.com and configure from there. Once you have box you can save it and launch instances as needed.

Until recently EC2 had some major flaws: when an instance went down you'd lose all data (making it less than suitable for database use) and you weren't guaranteed to get the same IP address with a new machines (meaning additional downtime while DNS propagates with a new IP). Amazon has since solved these issues with Elastic Block Storage (essentially a 1GB-1TB external drive that can be attached to any instance and persists even if the instance goes down) and Elastic IP Addresses (IP addresses assigned to your account that can be pointed at any running instance, so your DNS stays valid when an instance dies). Check out Eric Hammond's article on combining EBS with MySQL and the ElasticDB concept for combining CouchDB, EC2 and EBS.

Even with these additions there are some parts of the server-in-the-cloud equation that are still missing from Amazon's offering. Since Christmas is right around the corner, I've complied an AWS Wishlist for the jolly elves in Seattle.

1. Tiny EC2 Instance
Currently Amazon offers instances from $0.10 to $0.80 an hour. For a dime you get a Small Instance that comes with 1.7 GB of memory, 1 EC2 Compute Unit (which Amazon says is the equivalent of a "1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor"), and 160 GB of instance storage. This is plenty powerful enough for the early stages of a small startup. For a proof of concept, demo machine, or student project it's too powerful and far too expensive: those dimes, plus bandwidth, add up to between $70 and $90 a month.

I'd love to see a Tiny Instance with 512 MB of memory, .25 EC2 Compute Unit, and 10 GB of instance storage for $5 a month, or possibly free with a limit of three Tinys per account. Three free Tinys would be a shot across Google's bow, competing with their announced price structure and spur a lot of interest for AWS as a development sandbox.

2. Load Balancer
One of the amazing benefits of AWS/EC2 is the ability to scale application machines as needed. Get dugg and are being hit hard? Spin up 10 extra boxes. When the peak is over kill those boxes without dealing with the hassle of service reps or contracts. But, how do you properly balance to the new machines (and then stop balancing to them when they're destroyed)? You could set up an instance for load balancing, but that's a bit excessive. I'd love to see the next step in Elastic IPs and be able to specific a number of machines to attach to an external IP and choose a balancing strategy via a web services API.

3. DNS
With load balancing and DNS Amazon could offer an end-to-end platform solution. Register your domain with godaddy and point it at AWS. Then you'd be able to control it all with their APIs or build yourself a nice web interface for connecting domains, IP addressess, services, machines, and load balancers.

4. Email
Yes, each of us could launch an instance and configure it as a mail server. But why bother? Typical uses for mail in a webapp are sending out a few notifications, creating accounts, and resetting passwords. I'd gladly pay some nominal fee to avoid complicating my server setup with a machine to send and receive a few messages.

5. SMS
Amazon already has the ability to send text messages. Extending this to their AWS users would be great for developers who want to integrate mobile phones into their business.

Oh Amazon elves I've been a very good boy this year! Feel free to drop me a line and let me know when I can expect my new AWS toys!

A mi modo de ver esta jugada es muy inteligente porque sin inversión en infraestructura y mantienendo su tradicional forma de engordar sus arcas a través del licenciamiento de sus productos, aprovechan el auge del mercado del cloud computing eludiendo una de las características mas interesantes del "as-a-service" el pago a través de las suscripciones de usuarios.  Además permiten que los actuales clientes puedan utilizar su licencia actual en EC2 sin coste ninguno ayudando a que se incrementen los ingresos de Amazon a través del uso de las EC2. 

El público objetivo de este movimiento serán las empresas que quieran crear su paas(platform as a service) en Amazon, las ISV de saas y los departamentos de IT, y los dos primeros podrán montar su precio a través de suscripciones ya que las licencias de Oracle sobre EC2 están referenciadas a las máquinas y no al número de usuarios. 

Desde el punto de vista del cloud computing es una gran noticia porque aparte de ser un nuevo producto bajo una marca de reconocido prestigio que apuesta por la nube, aumenta la oferta de productos del submercado de las bases de datos que en la nube es muy poco competitivo (Mysql y Postgre son la BBDD que están utilizando).

Now, hosted databases are not entirely new, with various methods having been offered in the past, but this marks a new form of easily customer-deployable (but yet still requiring management) solution, that can even leverage your existing Oracle licenses.

As the accessibility of these solutions advances, it raises a few questions around which I'd like feedback, if you're considering (or have made) a move to hosted databases:

• Hybrid or pure deployment? - When you look at the architecture for using a cloud database, would you consider it as a primary resource, or to complement existing onsite deployments? Furthermore, would you migrate existing resources to the new hosted instances?
• Security - Now that your database resource is no longer within a firewall (or VPN), as many are, what new security measures will you be considering? Does this mean a new headache for you of deploying VPN resources into your cloud (or modifying usage models to use different protocols)? Or is this a moot point for the applications you are considering?
• Intended Usage - A significant number of AWS services are positioned around startups and web services that are bringing direct-to-the-market offerings to fruition. Do you fall in this camp, or are you considering AWS/Oracle for internal business use?
• Backup/long term archiving - Obviously, we believe in the reliability and promise of online backup, and Amazon has provided a solution for the Oracle instances using S3. Is this something you would consider as your primary means of backup, or supplement it by storing additional copies elsewhere, in the cloud or on premise?

As each day goes by, more and more of your primary data is starting (or has the opportunity) to move offsite, both in the form of unstructured data (like identity and general web app usage), and now more and more structured and critical data (like databases and file systems).  Having a backup copy of this data could mean the difference between a crisis or a nuisance in the case of a disaster, either on premise or off.

How is this changing model impacting your decisions around how to protect your data?

Nelle prossime settimane lanceremo sul mercato italiano, in partnership con un'importante azienda statunitense, una soluzione di backup on line basata proprio su AWS. E' giunto quindi il tempo di un secondo breve approfondimento ...

Nel marzo 2006, Amazon lancia il suo servizio di online storage; mettendo a disposizione del pubblico l'infrastruttura che essa stessa utilizza.

Il servizio è concepito per essere minimale: Caricamento, Accesso e Cancellazione dei file sono ideati per essere il più semplici possibile.

Può essere memorizzato un numero ipoteticamente infinito di file (da un byte a 5 GB ognuno) protetti da accessi non autorizzati tramite meccanismi di autenticazione.

Amazon S3 fornisce una solida protezione contro gli attacchi più tradizionali (DoS, IP Spoofing,Port Scanning, Man in the Middle), mentre sarà l'utente stesso ad inserire la protezione aggiuntiva che ritiene necessaria, per esempio crittografando il traffico di informazioni sensibili tramite HTTPS.

Il servizio è molto abbordabile ($0.15 per ogni GB caricato) ed estemamente flessibile grazie all'intefaccia http oppure BitTorrent™ (in futuro verranno rese disponibili altre interfacce).

Con questo suo sistema Amazon vuole liberare i clienti dai costi fissi e di mantenimento delle più tradizionali soluzioni di memorizzazione dei dati, permettendo loro di investire maggiormente nel loro business.

For enterprise customers that want to operate their software on multiple operating systems or on multiple platforms the news could be compelling. Essentially, RightScale is offering customers a one-stop-shop for managing and provisioning different types of clouds. With such an offering it's as easy to run applications in Windows-based clouds offered by GoGrid as in Linux-based clouds as offered by Amazon.

This will help with the problems of ensuring reliability and the pain of dealing with platform specific clouds, issues I wrote about a few months ago in why enterprises are not ready to trust the cloud.

Basically, PyAuthTicket is a tiny package that allows you to create an HMAC digest of a "ticket". A "ticket" is defined as a combination of a message (in App3's case this was the full request, ie "GET /"), a timestamp (when the ticket was generated), and a secret key (such as an API key) which should be kept secret by both sides.

The ticket object lets you create the ticket on one end, send the message, timestamp, and digest to a remote server, and let that server recreate the ticket to check if it's valid. When creating the ticket, it will also do a check based on a threshold (in seconds) of how old the ticket can be and still remain valid. (Note that if the user tries to fib the plain text timestamp it passes, the digest won't match up and the ticket will still be invalid.)

I've posted the project and package on both PyPI and GoogleCode (there is a code example on the GoogleCode page):

• http://pyauthticket.googlecode.com/
• http://pypi.python.org/pypi/pyauthticket/

Was ist EC2?

Die EC2-Plattform gibt jedem die Möglichkeit eigene virtuelle Server auf von Amazon bereitgestellter Hardware zu booten. Dazu lassen sich eigene System-Images auf der S3-Plattform von Amazon speichern, die dann direkt geladen werden. Jede Instanz erhält, solange sie läuft, einen eindeutigen Public-DNS-Eintrag der Form ec2-XXX-XXX-XXX-XXX.compute-X.amazonaws.com, über den auf die Instanz zugegriffen werden kann. Dabei bildet der Bereich XXX-XXX-XXX-XXX die IP der Instanz.

Was macht EC2-Instanzen zu einem lohnenden Angriffsziel?

Da gerade Web-Plattformen ihre Webserver mit EC2 entlasten, findet man dort oftmals Quelltexte verschiedenster Plattformen, deren Verkauf oder Veröffentlichung sicherlich für einige Probleme bei diesen Firmen sorgen kann. Da auch Datenbanken gern dann auf diesen Instanzen laufen, kommt man so auch leicht an die kompletten Kundenstämme der betroffenen Plattform und andere sensible Daten. Laufen auf den EC2-Instanzen keine eigenen Datenbanken, findet man fast immer bestehende Tunnel in die DMZ der angegriffenen Plattform.

Was macht den Angriff auf EC2-Instanzen einfacher als direkte Angriffe auf die zugehörige Plattform?

Schaut man sich mal in der Liste der öffentlich verfügbaren AMIs (Amazon Machine Images) etwas genauer um, findet man dort Unmengen veralteter Systeme. Natürlich erstellen die meisten Admins eigene Images, aber dies geschieht oftmals auf Basis von bestehenden Public-AMIs. Erstellt man ein eigenes AMI aus einem vorhandenen öffentlichen, kopiert man sein X.509-Zertifikat und die zugehörige Schlüsseldatei nach /mnt und erstellt mittels der EC2-Tools ein neues AMI, das man dann auf seinen S3-Speicher kopiert und als AMI für seine UserID registriert. Dieses AMI ist dann 'private' und kann nur über den zugehörigen Account genutzt werden. Bei der Erstellung von AMIs werden aber offenbar immer wieder ein paar typische Fehler gemacht.

Zuerstmal wird zumeist nicht beachtet, dass alle Public-AMIs bereits für den root-Account einen SSH-Key in der /root/.ssh/authorized_keys liegen haben, dessen Private-Key auch zumeist irgendwo im System rumschwirrt. Jemand, der sich also das zugrunde liegende Public-AMI bootet, kann so einen passenden Key für den root-Account erlangen. Weiterhin wird bei der Absicherung des Systems nicht beachtet, dass keine echte DMZ aufbaubar ist. Vergleichbar ist das mit Rootservern oder VServern, wie man sie bei jedem Hoster mieten kann. Dadurch werden viele DoS-Lücken nicht gefixt, die sonst die Firewall oder das IDS/IPS abfangen würde. (D)DoS-Angriffe auf eine EC2-Instanz können aber für den zugehörigen Account ziemlich teuer werden, denn dort gibt es keine Traffic-Flatrates.

Und das Problem der fehlenden DMZ erstreckt sich noch weiter... Um auf die Datenbanken (ich bleibe einfach mal beim Beispiel der Webplattformen) zuzugreifen, muss auf die DMZ der Plattform zugegriffen werden. Dabei spielt es keine Rolle, ob eine replizierte Datenbank auf der Instanz selbst läuft (meiner Erfahrung nach die häufigste Methode, da die Performance damit besser ist) oder ob direkt auf die Datenbankserver der Plattform zugegriffen wird. Zumeist werden für die Zugriffe auf die DMZ SSH- oder VPN-Tunnel verwendet. Nun muss man nicht lange raten um sich zu denken, dass die Keys dafür zumeist in den AMIs eingebunden und somit direkt auf der laufenden Instanz verfügbar sind. Hat man also Zugriff auf die Instanz, kommt man auch ganz schnell an alle anderen statischen Teile der zugehörigen Plattform.

Doch damit nicht genug... Auffällig ist auch, dass die Images offenbar nicht so schnell aktualisiert werden, wie es bei den zugehörigen statischen Teilen der Plattformen der Fall ist. Macht man bei einer EC2-Instanz ein Update, muss aus der Instanz ein neues AMI erstellt werden, damit es beim nächsten Booten auch wieder aktualisiert ist. Dies kostet natürlich Zeit, weswegen die Updates von EC2-Instanzen meist ziemlich hinterher hinken. Man findet sogar Instanzen, deren AMIs schon seit Monaten nicht aktualisiert wurden, so dass man passende Exploits auf jeder schlechten Cracker-Seite findet.

Zusammengefasst liegen die Schwächen von EC2-Instanzen in folgenden Punkten

• schlechte Einrichtung des AMI
• schlechte Update-Politik
• Nicht-Beachtung des Fehlens einer DMZ und externer Schutzsysteme wie Firewall, IDS/IPS usw.
• Zugriffe auf die DMZ einer Plattform über Tunnel

Wie bekommt man raus welches Image einem AMI zugrunde liegt?

Dies ist zumeist über die Fingerprints der Systeme möglich, wie man sie z.B. mit nmap ermitteln kann. Zuerstmal ermittelt man allgemein das System, das auf dem AMI läuft. Anhand der Kernel-Version sucht man nach passenden Public-AMIs, die man sich dann bootet und deren Fingerprints mit dem Ziel vergleicht. Auf diese Weise kann man recht gut das Public-AMI ermitteln, das als Grundlage des Instanz-AMI diente. Der Rest verlangt dann zumeist nur noch ein paar Grund-Skills um sich Zugang zur Instanz zu verschaffen.

Was sollte ein Admin, der mit EC2 arbeitet, beachten?

• bei der Einrichtung der AMIs kein public AMI verwenden oder dieses gründlich säubern
• zusätzlicher Schutz vor Angriffen muss eingerichtet werden
• Keys für notwendige Tunnel in die DMZ sollten von einem Remote-Medium geholt werden, das ausgehängt wird, wenn der Tunnel aufgebaut wurde
• SSH-Port verlegen (sollte man eh bei jedem Server tun)
• tmp-Verzeichnis sollte von einem Remote-Server (z.B. Rootserver) kommen und noexec gemountet werden, per Default haben EC2-Instanzen keine extra tmp-Partition
• Updates regelmässig machen und nach jedem Update ein neues AMI erstellen
• sobald ein Reboot notwendig ist immer das zuletzt erstellte AMI booten, keinen normalen Reboot machen

Naja... es gäbe noch einiges mehr, das man erwähnen könnte, aber ich brauch jetzt erstmal eine Tüte Schlaf. Wer Rechtschreibfehler findet, kann sie behalten. :D

Yesterday was 9/11.. 7 year anniversary apparently :|

uhm let's see.. I went to orientation today~! It took forEVER! I was trying to keep eye contact the whole time but I couldn't help that my eyes would squint.. then go big again. I got my uniform @_@ but the manager gave me the guys one by accident LOL. The people there seem nice... oh shit I better be allowed to wear my shoes. I DO NOT want to have to buy another pair.

I got bored after orientation so I said, why not? I'll just volunteer for the book sale, like I did yesterday, although I wasn't registered for today. They didn't know.

Demain je dois aller a la class de francais a` vancouver. Apres, je vais avec ma soeur (et peut-etre mes parents) a metrotown pour manger avant ma soeur doit aller a la rendez-vous dentiste. Avant 5 je dois aller a surrey pour aller a lieu de travail

(I'm a french fob D:)!!

For those not too familiar with AWS, here's a brief summary of the services that are discussed in the AWS security whitepaper:

• Elastic Computing Cloud (EC2) - A service that permits a customer to create a virtual OS disk image (in a proprietary Amazon format) that will be executed by a web service running on Amazon's hosted web infrastructure.
• SimpleDB - A service that provides a simple (non-relational) database hosted on Amazon's hosted web infrastructure. Data is created and accessed using SQL-like queries.
• S3 - A service that essentially provides an Internet accessible remote network share.

All of the AWS services are accessed using an API provided by Amazon that is encapsulated in HTTP requests (with the exception of guest OSs created in EC2 where customers are free to create their own applications running in the 'cloud'). More details on AWS can be found here.

The AWS security whitepaper focuses it's analysis on the traditional security dimension triad: confidentiality, integrity, and availability. While there are plenty of other security dimensions to consider (as readers of my series on X.805 will know), these 3 are generally well understood and cover the majority of customer security concerns.

The whitepaper starts by scoping the security analysis to the physical and management access of the cloud computing infrastructure, the EC2, Simple DB, and S3 services (which makes me wonder if there will be another round of analysis of the other AWS services done at some later date). Next, the paper discusses certifications that Amazon is pursuing for it's infrastructure and services (Sarbanes-Oxley is mentioned, as well as other auditing standards. No specific security standards are mentioned here, but that in itself is not a cause for alarm. Amazon's intent to subject itself to audits implies that they have an information security program around the systems. Of course, the devil is in the details for actually implementing the IS program, but I give them a star for this point. One point they raised that I didn't like was that they mentioned that a customer has decided to use the S3 service for building HIPAA compliant applications. I don't think this is a good vote of confidence because it's not a good idea to base a security program around how another customer chooses to interpret a standard.

The next section discusses the physical security of the infrastructure that hosts AWS. This section is a little light on details but basically describes the typical physical security measures you'd find in pretty much any datacenter environment (with perhaps the exception of the two-factor authentication required for Amazon employees to access the datacenter). I don't put a lot of faith in a two-factor system for physical access because most systems I've seen are simply pin-pads protecting doors.

Next, backup policies are described. Nothing too noteworthy in this section; they acknowledge following well-known backup principles like offsite backups, keeping backups in seperate locations etc. I liked the fact that they clearly explained what is in the scope of their backups and what is not (e.g. they do not perform backups of the data created in virtual OSs in EC2 instances for example, but they do backup data stored directly in S3).

The next 3 sections delve into the security details surrounding the 3 services. The first section focuses on EC2. The analysis starts with a discussion of the multiple security layers that exist within EC2: host OS security, guest OS security, firewalls, and signed API calls. The host OS security primarily focuses on the security used by Amazon personnel to access the host OS. I thought that this section missed some pretty obvious security concerns such as policies used for applying security patches, a description of an incident response program in case a security breach occurs, and auditing of changes made to the host OSs by Amazon personnel. Guest OS security analysis is basically punted off to the customer (which I agree with). This section just describes what customers should do to secure their guest OSs running on EC2. The firewall section is a little confusing to me because it is mentioned as a capability of EC2, but it is not clear what this firewall applies to. Is the firewall instance tied to each EC2 instance that the customer has purchased? In any case, there is nothing noteworthy here except the fact that changing the firewall rules requires the customer to use a X.509 certificate for credentials to modify the firewall ruleset. The choice of an X.509 certificate for a authentication credential in this case is a little bizarre to me; after all the security of PKI system from a client perspective relies entirely on a client keeping their X.509 certificate private key safe (and not sharing the passphrase used to protect the private key with anyone). So I'm not sure how this choice is more secure than a seperate username/password required to modify the firewall rulesets. Finally, the section delves into the details of how the signed API calls work. Essentially, API calls to modify instances on EC2 must be signed (again using a X.509 cert or a 'Amazon Secret Key'). This seems to be a good idea; it's the equivalent of having integrity protection applied to signalling data. The remaining analysis of EC2 is focused on describing their implemention of Xen Hypervisor to manage the different virtual OS images/instances. After the hypervisor discussion, there is a brief analysis of the network security measures in place. This section was pretty good in that it covered some of the common network security threats such as DDoS, MITM attacks, and packet sniffing. I didn't see anything here that raised any flags, but again there wasn't a lot of detail in how the security measures are implemented.

The next AWS service analyzed is the S3 service. I found that the analysis of this section  was very light in details. They focused the discussion only on threats of other customers accessing your data, security of your data at rest, and security of your data in transit. The paper provided a little detail on the ACLs used to restrict access to data based on 'buckets'  that data is placed into. I did like the fact that they discussed how the ACLs are administered (there is an ACL limiting access to ACL administration to bucket creators only). The paper also acknowledged that encrypted filesystems are not provided by default, so the customer has to encrypt their data before storing it on S3. I can understand this decision by Amazon because key escrow is difficult, and taking responsibility for keys that essentially 'lock' customer data is a big responsibility. I like that they highlighted this. Finally, there is a little discussion on how data is secured in transit. Reference is made back to the SSL encrypted API calls.

The final section of the paper discusses the security of the SimpleDB service. This is another service where the bulk of the security responsibility lies with the customer. Amazon provides a 'domain' under which the data is created. The domain administrator is given tools such as ACLs to limit who can access the data. Again, reference is made back to the SSL encrypted API calls that are used to protect the data in transit. SimpleDB databases again are not stored in an encrypted fashion, so Amazon makes it clear that customers should encrypt data before storing it on SimpleDB if they wish to limit access to their data.

In summary, I was impressed with the analysis done of the EC2 service, but the S3 and SimpleDB security analysis was too light in details. I also felt that Amazon didn't do a very good job in explaining how data stored on the S3 and SimpleDB services is safe from viewing by Amazon personnel. I understand the position that it is largely a customer responsibility for ensuring that your data is safe, I feel that Amazon pushed too much responsibility onto the customer without providing a better explanation of what they do to protect your data at rest. In any case, I'm happy to see that Amazon published this paper and hope that this starts a trend by other 'cloud computing' vendors and vendors in general.

The Amazon Web Services crew are back on the road with their AWS Startup Tour, and Brad Hintze and I will be joining them tomorrow in Salt Lake City.

I'll make a brief presentation around 3:30-ish, speaking about Bungee Connect and how Bungee Labs use Amazon Web Services. So if you're in the area, please swing by to give Brad and me a hello.

SQLite is no longer the default DB in rails, but for my dev PC I don't want the whole hog. So back to SQLite it was. I consider myself a bit of a windows guru, so I wasn't expecting any real problems. I downloaded  ruby & started to get up & running, installing gems, capistrano, ec2onrails, & finally sqlite3. I downloaded the SQLite3 DLL & command tool into SQLite3, added this to the path, & then tried to install the gem:

D:\ruby\bin>gem install sqlite3-ruby
Building native extensions.  This could take a while...
ERROR:  Error installing sqlite3-ruby:
ERROR: Failed to build gem native extension.

d:/ruby/bin/ruby.exe extconf.rb install sqlite3-ruby
checking for fdatasync() in rt.lib... no
checking for sqlite3.h... no

nmake
'nmake' is not recognized as an internal or external command,
operable program or batch file.

not good!! nmake shouldn't be necessary for installing a gem. Turns out we need to instead use:

D:\ruby\bin>gem install --version 1.2.3 sqlite3-ruby
Successfully installed sqlite3-ruby-1.2.3-x86-mswin32
1 gem installed
Installing ri documentation for sqlite3-ruby-1.2.3-x86-mswin32...
Installing RDoc documentation for sqlite3-ruby-1.2.3-x86-mswin32...

which works a treat. I expect this will trip up a fair few people so hope this helps the happy googlers.

Amazon har netop udgivet et whitepaper om netop dette emne:
http://s3.amazonaws.com/aws_blog/AWS_Security_Whitepaper_2008_09.pdf

Så er der lidt weekend læsning :)

The contest is open to any site or service that uses one of the AWS services are part of its solution: S3, EC2, SimpleDB, SQS, FPS, DevPay, Alexa Web Services, or even Mechanical Turk. You've got until October 3 to enter, and if you're in the top 5, they'll fly you out to Seattle for a dog-and-pony show.

Infrastructure Services

• S3 - Simple Storage System.  Web-based storage.
• EC2 - Elastic Compute Cloud - On-demand processing power.
• SQS - Simple Queue Service - Messaging service for transferring work between computers.
• SimpleDB - Database version of S3.

Payment & Billing Services

• FPS - Flexible Payment System - System for accepting payments
• DevPay - Tool for developers to be paid for AWS applications

On-Demand Workforce

• Mechanical Turk - Ability to farm out units of work to humans.

Amazon also operates four Alexa services (web search, web information search, top sites, and site thumnail).  These services are tied to the Alexa web search engine.  Finally, Amazon operates two other web services fulfillment web service and associates web service.  Both of these services are e-commerce services related to the Amazon store.

There don't seem to be many books on the subjet.  The best looking book is "Programming Amazon Web Services: S3, EC2, SQS, FPS, and SimpleDB" by James Murty the author of JetS3t.  A number of the reviews say that its a "Ruby" book.  Another interesting book is "Amazaon.com Mashups" from Francis Shanahan.  Both books were written some time ago while Amazon keeps upgrading their services.

There are a couple of seemingly good examples of how to program AWS on the Amazon site.

Sample application to get started with Amazon SQS and Amazon EC2 - http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1457&categoryID=85. The client examples in C#, server examples are in Java.

Browser Uploads to S3 using HTML POST Forms - http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1434.  The examples are in Ruby, Python, or Java.

JetS3t is a free, open-source Java toolkit and application suite for S3.  The JetS3t toolkit provides Java programmers with a powerful yet simple API for interacting with S3 and managing data stored there.  (Description lifted from the web site.)

To use AWS you need to be prepared to pay for everything you do.  Its not clear to me why this not free to developers like Salesforce.com developer accounts and Google's AppEngine.

It's a great time to program for the cloud, no matter what Ted Dziuba's entertaining but barely coherent rants have to say (will someone get that guy some experience?). Amazon and Google are going toe-to-toe, with Amazon's addition of sorting in Simple DB bringing it up to par with Google App Engine's Datastore API. Sorting was the biggest missing piece in Simple DB and the most compelling reason to choose the Datastore API instead. No longer.  

But Google App Engine (GAE) and the Datastore API still win. Here's why:

1. The Datastore API is projected to be 10x cheaper. $0.15-$0.18 per GB-month sounds a lot better than Simple DB's $1.50 per GB-month.
2. GQL. GAE's SQL subset is just brain dead simple. As adept as programmers are at learning new frameworks, it's nice to have something brain dead every once in awhile. Simple DB takes a few more cycles to learn (brain cycles that is -- more coffee and such. Modafinil perhaps? Anyone tried it? I'm curious).
3. GAE has better Object Relational Mapping (ORM). GAE basically uses Django's sweet ORM system. You've got to jump through a lot more hoops to get something as nice with Simple DB. 
4. GAE automatically scales the web application, not just the database. With Amazon, you have to add load balancing and bring machines up and down yourself, even if you're using Simple DB. While there are third-party tools to help, they're not built-in. Again, GAE is brain dead here.  

Sure, App Engine only supports Python. The ultimate question, though, is what functionality can you get in the end? For web apps, App Engine gives you more, particularly for scaling (which is kind of the whole point). Don't know Python? Learn it. It will save you time in the end. Instead of endlessly fiddling with your load balancer and custom scripts for bringing instances up and down, you'll spend your time adding the next killer feature your users will love.

In the end, the Amazon/Google "main event" is a huge win for you, me, and our users. The sorting announcement from Amazon comes on the heals of a flurry of other new features from both companies, including Amazon's impressive persistent storage addition for EC2 called the Elastic Block Store, querying by attributes on Simple DB, GAE's support for 10 applications per user instead of 3, GAE's batch writes, etc. Neither one is pulling any punches, and the tools at our disposal as developers are progressing at a breathtaking pace as a result.

Amazon's is clearly the more complete offering (you can do anything on it, in any language), but it needs to learn from Google's focus on the dominant deployment scenarios.  Amazon could easily win if it does the following:

1. Makes Simple DB pricing competitive with Google's projected prices.
2. Adds a query language for Simple DB along the lines of GQL.
3. Adds automatic scaling for web applications, not just the database.
4. Offers complete deployment solutions for the dominant web applications frameworks, from Tomcat/Spring/Hibernate to Django and Zend, with ORM models already adapted to Simple DB, instances automatically replicated with traffic, etc. Basically the same thing as App Engine for more web app frameworks than App Engine supports and adapted to the Amazon platform. Sure, there are third-party solutions for some of this stuff, but those will never be trusted as much as something offered directly from Amazon.

I'm a big fan of Amazon and Werner Vogels (one of the most innovative people in the industry, and also apparently a pretty nice guy), but Amazon desperately needs to learn from what Google has done. It's ultimately a question of "usability" for developers. The originators of "one-click shopping" are losing in the game they practically invented. 

Amazon needs to turn on the one-click cloud.

The best thing with EBS is that data is persistent and volumes have redundancy built-in. If a drive on their SAN fails, you'll still have your data. It's not as reliable as S3, but it beats putting your data on EC2, where your files are lost once as instance is destroyed.
Just like EC2 instances, EBS volumes exist in a particular Availability Zone. The not-so-good thing about this is that you can only attach an EBS volume to an EC2 instance in the same Availability Zone, unlike S3. But you can't mount an S3 bucket, right? ;)

You can also make snapshots of EBS volumes. A snapshot is a backup of an EBS volume in a specific point in time. Snapshots are stored in S3. Making backups this way is more efficient than making a big tarball of the files you need to backup and sending them to S3. Your EBS volume is "frozen" the moment you call the snapshot command, unlike other means of transferring to S3 where data can be modified in the middle of a transfer. Making snapshots is also resource-effective since it does not use cpu cycles since it is not the instance that executes the creation of the snapshot. Snapshots are also incremental, which means you'll save time and space. You can't see EBS snapshots using the S3 API.

From http://wiki.rightscale.com/2._References/EBS/01-Overview_of_Elastic_Block_Storage_(EBS)

The costs of EBS will be similar to the pricing structure of data storage on S3. There are three types of costs associated with EBS.

Storage Cost + Transaction Cost + S3 Snapshots = Total Cost of EBS

• Storage Costs
  The cost of an EBS Volume is $0.10/GB per month. You are responsible for paying for the amount of disk space that you reserve, not for the amount of the disk space that you actually use. If you reserve a 1TB volume, but only use 1GB, you will be paying for 1TB.
  • • $0.10/GB per month of provisioned storage
• Transaction Costs
  In addition to the storage cost for EBS Volumes, you will also be charged for I/O transcations. The cost is $0.10 per million I/O transactions, where one transaction is equivalent to one read or write. This number may be smaller than the actual number of transactions performed by your application because of the Linux cache for all file systems.
  • • $0.10 per 1 million I/O requests
• Snapshot Costs
  Snapshot costs are compressed and based on altered blocks from the previous snapshot backup. Files that have altered blocks on the disk and then been deleted will add cost to the Snapshots for example. Remember, snapshots are at the data block level.
  • • $0.15 per GB-month of data stored
      $0.01 per 1,000 PUT requests (when saving a snapshot)
      $0.01 per 10,000 GET requests (when loading a snapshot)

NOTE: Payment charges stop the moment you delete a volume. If you delete a volume and the status appears as "deleting" for an extended period of time, you will not be charged for the time needed to complete the deletion.

Elasticfox has just been recently updated to add support to EBS.

Creating a new volume is fairly easy. Just click on the + button on the Volumes pane and a window will appear.

Enter the size (in GB) of the volume you're creating and the Availability Zone of your EC2 instances. Since we're creating a volume from scratch, Snapshot ID should be set to .

The next step is attach it to an EC2 instance. Select an instance from the drop-down box and enter the device name (what it will appear as within the EC2 instance).

Your EBS volume is now attached to your EC2 instance.

To make a snapshot of an EBS volume, right click and select Create a new snapshot from this volume. This might take a while depending on the size of the volume you created.

Since it is impossible to attach an EBS volume to more than 1 EC2 instance, you can clone it instead. Right click on the snapshot and select Create a new volume from this snapshot.

Enter the size of the volume and click Create.

The newly created volume will now appear under the Volumes pane with a SNAP ID this time.

All in all, Elastic Block Store is a great product from Amazon. We rely on EC2 for our non-production deployments with the fear of losing our data once an instance is killed. With the release of EBS, we can now put our data (usually webapps) to a volume instead of inside an EC2 instance .

What I need now is a strategy to cost-effectively use EBS. Our modified data (the ones not in the bundle) are scattered: virtualhosts definitions in /etc, Java webapps in /opt, RoR and PHP webapps in /var. At first I was thinking of creating symlinks. Let's say one of our instances die. After launching another one from a bundle, I'll just attach and mount the volume and create the necessary symlinks (ln -s /mnt/ebs/html /var/www/html).

This way we do not have to make constant backups from EC2 to S3. It is more efficient to create volume snapshots than to tar your /var/www/html and send it to S3 via s3cmd. Eventhough your data in EBS is persistent and has some form of redundancy, it's still a good idea to make snapshots to S3.

I'm wondering how much Amazon will charge us if I create a volume out of a Linux distribution and chroot / it :)

Quoting Yanskie, "Thanks Amazon! You're the best!"

The cloud computing industry is still in its infant stage (yes really!), even with everyone and their mum now calling their service cloud computing these days. (Buzzwords, gotta love em.)

One of the major factors that will start to hold it back as time goes on is inability to move swiftly and easily between different platforms. Without this ability customers feel locked in and thus much more hesitant to try and use cloud computing. This applies throughout the various *aaS's that relate to cloud computing (software, hardware, infrastructure, platform etc), but as you can imagine my main focus is on hardware/infrastructure.

Building your entire application so that it can only work on one cloud is foolish, and it's irrelevant who's cloud that might be - if you are locked in, what do you do when things go wrong? If the cloud has specific features that no-one else has, or has a particular niche or audience (SalesForce is the first one that comes to mind), then I can certainly see the sense in that, although you should still be able to pull all of your data out in an easy and legible way. However, when it comes to hardware/infrastructure, why would you want to be locked in?

We have several customers now who are splitting their infrastructure between ourselves and Amazon EC2/S3, and we think this is brilliant, as the customers can scale either up or down as needed, and have removed their reliance on one platform. This is a great example of how having (nearly) interoperable systems enables customers in general to be less scared of moving to a new technology, which is great for everyone involved as it means the industry can and will grow quicker than it would do if it was only a handful of individual companies providing distinct services that weren't compatible with each other.

We are sticking a flag in the ground and saying that Interoperability and Portability are absolutely key to the future development of the cloud computing industry, and we as a company will be doing everything we can to promote this, including open sourcing various parts of our technology as we grow to help standardise the technology, and using existing open source standards and technology wherever possible.

Having a standard API so people can work automatically with your systems is certainly a good step (and frankly, fundamental to any cloud computing platform), but it doesn't make a platform truly open. This was the subject of conversation at Structure 08 where myself and Jason Hoffman from Joyent debated with Christopher Bisciglia from Google on whether BigTable from Google (as used in Google App Engine) is open. (He says it is, we say it isn't!).

It will also be mentioned in a debate between Jeff Barr (from Amazon Web Services) and myself at FoWA this October. The schedule for FoWA is "here", and for those of you reading this that haven't heard of FoWA, it's *the* most relevant expo/conference for web/application developers in the UK (and it's also great fun). If you get the chance to go, jump at it!

Our current platform is already built on established and well regarded standards (you can port an application to it from a traditional dedicated server as fast as you can copy the files, no other work to be done), however, there's still a lot of innovation going on in this area. So from now on, wherever possible we will be open sourcing or giving as much public information as we can on how our platform works. We'll even be releasing some code that will work with and aid interoperability with other platforms in an effort to promote standardisation, though of course we'll have to keep some bits to ourselves :)

Watch this space..........

Tony.

Picked a pair of DG, but it couldn't work for us.. Blah blah then stayed a pair of Vogue, saving her $70. Holy shit.. I never knew lenses cost so much! Way more than the frame itself most of the time. Grand totale: a wooping 350 WITH the 140$ discount and extra $20 the manager gave us cause she's so awesome! (there was a special.. buy a pair of frames over 179.95 and get free lenses! -- which was exactly how much the frames cost in our case..)The service at lenscrafter is greaaaaaaaaaaaat. I'm so going there once I need glasses.
P.S. I got hired at AWS! 3 days only though :(

Kdo chtěl na AWS donedávna provozovat svou databázovou aplikaci, měl v zásadě dvě možnosti:

Buďto spravovat svá data pomocí Amazoních služeb S3 a SimpleDB, což mohlo narazit na několik omezení:

• S3 je jen key/value storage
• SimpleDB běží pouze v neveřejném módu "limited beta"
• Obě služby fungují dost jinak než by mohl očekávat člověk zvyklý na relační databáze a SQL

Druhá možnost je pustit na EC2 instanci svou oblíbenou databázi. Člověk potom sice nemá k dispozici neomezeně skálující úložiště jako které nabízí výše zmíněné Amazoní služby či Googlí BigTable, ale to se dá obejít za pomocí patternů prošlápnutých Flickrem, eBayí (zminěna onehdá i na Jabloku) a mnoha dalšími.

Potíž je ale v tom, že EC2 je primárně navržena pro aplikační servery, a nestará se o persistenci dat, jinými slovy co si na EC2 instanci nahrajete, to vám tam vydrží dokud instance nespadne - tedy třeba dokud se server, na kterém běží, nezrestartuje. A to se může stát přeci jen docela často oproti tomu, jak často se u "normální" fyzické krabice rozbije harddisk.

Ani toto není neřešitelný problém - aplikace jako Scalr (dostupný ve zdrojácích i jako služba) můžou udržovat a synchronizovat farmu EC2 serverů hostujících master a slave instance MySQL, v případě výpadků roli mastera předávat "slejvům" a průběžně zálohovat do S3.

Ale přeci jen, obnovení databáze ze zálohy vezme nějaký čas, a pro malé projekty (rozumněj projekty, kterým se nevyplatí mít zapnuté a tedy platit příliš mnoho EC2 instancí najednou) s objemnými databázemi to zjevně příliš spolehlivé řešení nebude.

Člověk si říká: "tohle je ten slavný cloud-computing? To bych teda čekal něco lepšího..." Třeba mít nezávislé virtuální disky, které si člověk může ke své EC2 instanci namountovat aniž by riskoval ztrátu ulozených při shození instance.

A právě toto řeší nově spuštěný Elastic Block Store. Sláva.

Co?

Elastic Block Store (EBS) nabízí možnost vytvářet si perzistentní svazky o velikosti od 1GB do 1TB, které je možné namountovat k jedné EC2 instanci (ta naopak může mít připojeno ECB svazků několik). Vůči uživateli se pak tváří jako neformátované blokové zařízení, prostě jako pevný disk.

Navíc elegantně řeší otázku zálohování a replikace: EBS samotné sice nabízí spolehlivost několikrát vyšší oproti běžným diskům, ale spolehlivosti S3 zdaleka nedostahují. Uživatel si ale kdykoli může vytvořit snapshot, který se transparentně uloží do bezpečného úložiště S3, odkud se může použít pro obnovení nebo pro vytvoření kopie svazku. Snapshoty se vytváří inkrementálně, takže by to mělo celkem švihat.

Kvůli zpoždění sítě budou EBS svazky přirozeně pomalejší oproti lokálním diskům. Ale záleží na tom, s čím člověk srovnává. Na RightScale blogu naměřili 70MB/s s tím, že běžný průtok bude asi horší. Není to sice žádný zázrak, ale žít se s tím dá.

A za kolik?

Účtování služby SBS má dvě složky: platby za velikost svazku a za datový tok. Za alokovaný GB je účtováno $0.10 měsíčně a k tomu se počítá dalších $0.10 za každých milion I/O operací.

Pro ilustraci jsem si spočítal hypotetické náklady na používání této služby pro případnou migraci diskusního serveru Okoun.cz (zobrazuje zhruba 3 miliony dynamických stránek za měsíc):

• MySQL má data na partition o velikosti 10GB (zaplněná ze 70%, řekněme, že to nějakou dobu bude  stačit). Tedy dolar měsíčně.
• Podle iostatu server dává na databázové partition cca 80 I/O operací za sekundu, což měsíčně odpovídá zhruba dvaceti dolarům
• Tedy $21 měsíčně celkem, což při aktuálním kurzu koruny dělá 340 Kč. To není úplně zlé, když uvážíme, co si tím můžeme ušetřit za opruz

Myslím, že se serverovým hardware si brzo přestanu dělat starosti.

Zdroje

• Werner Vogels: Amazon EBS - Elastic Block Store has launched
• Popis služby Amazon Elastic Block Store (EBS) na stránkách Amazonu
• Amazon’s Elastic Block Store explained na RightScale blogu

First a couple of words about what EBS is. In short it's a SAN (Storage Area Network) in the cloud. You can allocate a disk volume of 1GB to 1TB in size from what is now an endless SAN in the cloud and attach it to an instance of yours running in EC2. The volume is stored on redundant disks (i.e. with some form of RAID) and has a lifetime separate from any instance on which it is mounted, so you can unmount it and later remount it on another instance. You can also perform a snapshot backup of a volume to S3, where it is stored with the redundancy and durability of all objects on S3. Moreover, successive snapshots are incremental providing a very powerful and efficient incremental backup capability for volumes.

All this and much more is explained in detail in my other post and there's yet more detailed EBS information on our support site. The official EBS announcement is on the EC2 detail page, Werner Vogels provides some background, and Jeff Barr's blog entry has links to many other related announcements.

The RightScale dashboard supports all the features of EBS and offers a number of additional goodies such as configuring volumes to automatically be attached to servers when these launch and keeping track of the ancestry of a volume or snapshot.

What does EBS enable? In short: traditional processing on large datasets and reliable storage for many servers. Let's look at these two areas one-by-one.

Large datasets

Amazon Web Services are designed for scale. EC2, S3, SQS, and SDB are ideally suited for building large systems that process huge data volumes. The catch has been that they are geared towards modern service oriented systems that can use storage accessed via HTTP PUTs and GETs (Amazon S3), can work using a non-relational database like Amazon SDB, and thrive on large numbers of simple servers (EC2). Users that have more traditional applications, such as relational databases, that require large datasets stored in a file system with a POSIX interface have had difficulties in meeting all their requirements for operating in AWS. While an EC2 X-large instance comes with about 1.4TB of local disk it is rather difficult to actually use this disk space in a production system. Populating the disk with data at boot time can take hours and backups, replication and restoring the data in case of an instance failure are all sore points. For up to 100GB the timescales are all workable, but beyond that it gets difficult.

With EBS the processing of large datasets contained within a file system becomes easily accessible. First of all, volumes can be up to 1TB in size and beyond that it is possible to mount multiple volumes on the same instance such that file systems of 10TB are practical. The volumes can further be backed-up to S3 using the snapshots and they can be replicated by creating new volumes from the snapshots. What is particularly nice is that a volume can be created in any availability zone (think datacenter) of a region from a snapshot, so copying a large volume across datacenters can be off-loaded to EBS and is done very efficiently.

Many virtual appliance servers

EBS also really enables SaaS vendors that use a single-tenant "virtual appliance" model. Many software vendors have approached us with use-cases where they would like to run individual servers on behalf of their customers. Often these servers are co-managed between customer and software vendor or have other properties that make the service inappropriate for multi-tenant SaaS implementation. In these use-cases the end-customer is storing important data on these servers and requires a robust data safeguarding architecture, in particular for database storage. While we today have a very effective mysql replication and backup solution, it is really geared at multi-server set-ups and doesn't fit the price and complexity budget of cookie-cutter single-server virtual appliances. For those use-cases EBS brings the desired performance and reliability and drops the complexity and price.

With EBS the canonical reliable single-server virtual appliance can be implemented with the following architecture: an EC2 instance whose type is chosen for the cpu and memory required, an EBS volume sized appropriately for the data set, a revolving set of frequent snapshots providing disaster recovery backups, and periodic application-level "export" of backups to S3 for archiving and off-cloud backups. In case of a total failure of the EC2 instance and the EBS volume (e.g. datacenter fire) a new instance and volume can be allocated in another availability zone from the last revolving snapshot.

When it comes time to upgrade the virtual appliance to a new software version it becomes relatively easy for the software vendor to spin-up a second instance and volume with the upgraded software for important customers so they can test-drive the new version on their data and train their internal users before committing to the upgrade.

Try it out for yourself!

We've been busy integrating support for this new storage system for months so that you can start using it immediately. And our RightScale Dashboard support for EBS is available as part of our free Developer Edition! To learn more about EBS and RightScale's support for it, check out my detailed technical review, read our EBS tutorials at wiki.rightscale.com, or register for our upcoming RightScale EBS Webinar. Or just drop us a line at sales@rightscale.com.

The Basics

EBS starts out really simple: you create a volume from 1GB to 1TB in size and then you mount it on a device (like /dev/sdj) on an instance, format it, and off you go. Later you can detach it, let it sit for a while, and then reattach it to a different instance. You can also snapshot the volume at any time to S3, and if you want to restore your snapshot you can create a fresh volume from the snapshot. Sounds simple, eh? It is but the devil is in the detail!

Reliability

EBS volumes have redundancy built-in, which means that they will not fail if an individual drive fails or some other single failure occurs. But they are not as redundant as S3 storage which replicates data into multiple availability zones: an EBS volume lives entirely in one availability zone. This means that making snapshot backups, which are stored in S3, is important for long-term data safeguarding.

I know that folks at Amazon have thought long and hard how to characterize the reliability of EBS volumes, so here's their explanation taken from the EC2 detail page:

Amazon EBS volumes are designed to be highly available and reliable. Amazon EBS volume data is replicated across multiple servers in an Availability Zone to prevent the loss of data from the failure of any single component. The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% - 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives.

From a practical point of view what this means is that you should expect the same type of reliability you get from a fully redundant RAID storage system. While it may be technically possible to increase the reliability by, for example, mirroring two EBS volumes in software on one instance, it is much more productive to rely on EBS directly. Focus your efforts on building a good snapshot strategy that ensures frequent and consistent snapshots, and build good scripts that allow you to recover from many types of failures using the snapshots and fresh instances and volumes.

Volume performance

Our performance observations are based on the pre-release EBS volumes, thus some variations on the production systems should be expected. On the one hand our pre-release tests were probably running on a small infrastructure with fewer users, but on the other hand many of these users were also running stress tests, so it's really hard to tell how all this will carry over. Only time will tell.

EBS volumes are network attached disk storage and thus take a slice off the instance's overall network bandwidth. The speed of light here is evidently 1GBps, which means that the peak sequential transfer rate is 120MBytes/sec. "Any number larger than that is an error in your math." We see over 70MB/sec using sysbench on a m1.small instance, which is hot! Presumably we didn't get much network contention from other small instances on the same host when running the benchmarks. For random access we've seen over 1000 I/O ops/sec, but it's much more difficult to benchmark those types of workloads. The bottom line though is that performance exceeds what we've seen for filesystems striped across the four local drives of x-large instances.

With EBS it is possible to increase the I/O transaction rate further by mounting multiple EBS volumes on one instance and striping filesystems across them. For streaming performance this doesn't seem worthwhile as the limit of the available instance network bandwidth is already reached with one volume, but it can increase the performance of random workloads as more heads can be seeking at a time.

Snapshot backups

Snapshot backups are simultaneously the most useful and the most difficult to understand feature of EBS. Let me try to explain. A snapshot of an EBS volume can be taken at any time, it causes a copy of the data in the volume to be written to S3 where it is stored redundantly in multiple availability zones (like all data in S3). The first peculiarity is that snapshots do not appear in your S3 buckets, thus you can't access them using the standard S3 API. You can only list the snapshots using the EC2 API and you can restore a snapshot by creating a new volume from it. The second peculiarity is that snapshots are incremental, which means that in order to create a subsequent snapshot, EBS only saves the disk blocks that have changed since previous snapshots to S3.

How the incremental snapshots work conceptually is depicted in the diagram below. Each volume is divided up into blocks. When the first snapshot of a volume is taken all blocks of the volume that have ever been written are copied to S3, and then a snapshot table of contents is written to S3 that lists all these blocks. Now, when the second snapshot is taken of the same volume only the blocks that have changed since the first snapshot are copied to S3. The table of contents for the second snapshot is then written to S3 and lists all the blocks on S3 that belong to the snapshot. Some are shared with the first snapshot, some are new. The third snapshot is created similarly and can contain blocks copied to S3 for the first, second and third snapshots.

There are two nice things about the incremental nature of the snapshots: it saves time and space. Taking subsequent snapshots can be very fast because only changed blocks need to be sent to S3, and it saves space because you're only paying for the storage in S3 of the incremental blocks. What is difficult to answer is how much space a snapshot uses. Or, to put it differently, how much space would be saved if a snapshot were deleted. If you delete a snapshot, only the blocks that are only used by that snapshot (i.e. are only referenced by that snapshot's table of contents) are deleted.

Something to be very careful about with snapshots is consistency. A snapshot is taken at a precise moment in time even though the blocks may trickle out to S3 over many minutes. But in most situations you will really want to control what's on disk vs. what's in-flight at the moment of the snapshot. This is particularly important when using a database. We recommend you freeze the database, freeze the file system, take the snapshot, then unfreeze everything. At the file system level we've been using xfs for all the large local drives and EBS volumes because it's fast to format and supports freezing. Thus when taking a snapshot we perform an xfs freeze, take the snapshot, and unfreeze. When running mysql we also "flush all tables with read lock" to briefly halt writes. All this ensures that the snapshot doesn't contain partial updates that need to be recovered when the snapshot is mounted. It's like USB dongles: if you pull the dongle out while it's being written to "your mileage may vary" when you plug it back into another machine...

Snapshot performance appears to be pretty much gated by the performance of S3, which is around 20MBytes/sec for a single stream. The three big bonuses here are that the snapshot is incremental, that the data is compressed, and that all this is performed in the background by EBS without affecting the instance on which the volume is mounted much. Obviously the data needs to come off the disks, so there is some contention to be expected, but compared to having to do the transfer from disk through the instance to S3 it is like night and day.

Availability Zones

EBS volumes can only be mounted on an instance in the same availability zone, which makes sense when you think of availability zones as being equivalent to datacenters. It would probably be technically possible to mount volumes across zones, but from a network latency and bandwidth point of view it doesn't make much sense.

The way you get a volume's data from one zone into another is through a snapshot: You snapshot one volume and then immediately create a new volume in a different zone from the snapshot. We have really gotten away from the idea that we're unmounting a volume from one instance and then remount it on the next one: we always go through a snapshot for a variety of reasons. The way we think and operate is as follows:

• You create a volume, mount it on an instance, format it, and write some data to it.
• Then you periodically snapshot the volume for backup purposes.
• If you don't need the instance anymore, you may terminate it and, after unmounting the volume you always take a final snapshot. If the instance crashes instead of properly terminating, you also always take a final snapshot of the volume as it was left.
• When you launch a new instance on which you want the same data, you create a fresh volume from your snapshot of choice. This may be the last snapshot, but it could also be a prior one if it turns out that the last one is corrupt (e.g. in the case of an instance crash or of some software failure).

By creating a volume from the snapshot you achieve two things: one, you are independent of the availability zone of the original volume, and second, you have a repeatable process in case mounting the volume fails, which can easily happen especially if the unmount wasn't clean.

Now, of course, in some situations you can directly remount the original volume instead of creating a new volume from a snapshot as an optimization. This applies if the new instance is in the same availability zone, the volume corresponds to the snapshot that we'd like to mount, and the volume is guaranteed not to have been modified since (e.g. by a failed prior mount). The best is to think of the volume as a high-speed cache for the snapshot.

Price

Estimating the costs of EBS is really quite tricky. The easy part is the storage cost of $0.10 per GB per month. Once you create a volume of a certain size you'll see the charge. The $0.10 per million I/O transactions are much harder to estimate. To get a rough estimate you can look at /proc/diskstats on your servers. This will include something like this:

   8  160 sdk 9847 77 311900 56570 1912664 3312437 160672914 211993229 0 1597261 212049797
   8  176 sdl 333 86 4561 1538 895 51 19002 20131 0 4043 21669

which is just a pile of numbers. Following the explanation for the columns you should sum the first number (reads completed) and the fifth number (writes completed) to arrive at the number of I/O transactions (9847+1912664 for /dev/sdk above). This is not 100% accurate but should be close (I believe subtracting the 2nd and 6th numbers gets you closer yet, but I prefer an over-estimate). As a point of reference, our main database server is pretty busy and chugs along at an average of 17 transactions per second, which should total to around $4.40 per month. But our monitoring servers, prior to some recent optimizations, hammered the disks as fast as they would go at over 1000 random writes per second sustained 24x7. That would end up costing over $250 per month! As far as I can tell, for most situations the EBS transaction costs will be in the noise, but you can make it expensive if you're not careful.

The cost of snapshots is harder to estimate due to their incremental nature. First of all, only the blocks written are captured on S3 (i.e. blocks on the volume that have never been written are not stored on S3). Second it's tricky to talk about the cost of a snapshot due to their incremental sharing.

Summing it up

All in all it's amazing how simple EBS is, yet how complex a universe of options it opens. Between snapshots, availability zones, pricing, and performance there are many options to consider and a lot of automation to provide. Of course at RightScale we're busy working out a lot of these for you, but beyond that it is not an overstatement to say that Amazon's Elastic Block Store brings cloud computing to a whole new level. I'll repeat what I've said before: if you're using traditional forms of hosting it's gonna get pretty darn hard for you to keep up with the cloud, and you've probably already fallen behind at this point!

anyway I have such a big headache right now... took so advil and it's getting better. Might go to PNE tomorrow to watch my sister and her dance crew perform. Is it really the last day tomorrow?