The ACID test for Clouds

For the last year (it feels more like five) my company has been working on a product to support transactions in the cloud. It all started when I was in a GigaSpaces meeting about their "application platform" capability and there was no mention of their transactional capability. Well, it turned out there was :

• confusion over the term "transaction" meaning different things to DBMS and Gigaspaces;
• no transaction capability that old men with an ACID database would recognise.

Pretty much all my life, I’ve wanted to go bigger and faster … first of all it was motorbikes, then VLSI chips, local networks, enterprise systems, distributed systems ... and now clouds. So, just when I thought the next step in speed and scalability of applications was at hand ... there was nothing.

The more I looked for a solution to allow application programmers to easily write big, fast applications on the cloud using transactions, the more it seemed there wasn't one, and no appetite for one either. There were three stances from the cloud camp:

• "You don't understand" - you don't need to save in-memory data grid transactions to a database, because the grid is so reliable. Right - tell that to your CIO;
• "Eventual, non-consistent save is good enough". Well, that's at least got the 'D' part of ACID ... but nothing else;
• Junk the RDBMS - solve the scalability and consistency problem in a different paradigm. Shame about the databases and information feeds that drive commerce and decision-making today.

And from the non-cloud camp, there's the impression that distributed transactions won't work either - they're either too slow and complicated (Höller) or too slow and unreliable in clouds (Helland).

On the other hand, it seemed like a valuable approach, to provide a transaction layer :

• to match the speed and scalability of the cloud;
• matching up with all the existing data and add-ons around RDBMSs;
• in a way that normal developers can use.

As it happened, working on the problem was slightly trickier than we thought! (more on that later).

Now we are on the point of launching CloudTran**, which supports as-fast-as-possible transactional applications in grids and clouds, with a scalable, fast and reliable transaction mechanism. Our first performance tests showed around 300 (distributed cloud) transactions per second, and the key machine - a single Intel i7 920- was only at 8% CPU utilization, which implies we'll get 1000's of transactions per second.

This series of blogs will explain some of the assumptions that make this possible - i.e. define the space for CloudTran**, compared with native and distributed transactions - and describe its unique aspects.
**Previously known as CloudSave.