Live from PDC'09 - it's quite cloudy out there

The first keynote for the PDC'09 ended up minutes ago. Although it was not such a surprise, Azure was pervasive in virtually every talk being made this morning. Azure is definitively a top priority for Microsoft much like Windows or Office.

For my small company, it's very good news because we are banking a lot on this technology. Also, I really like the vision of Microsoft that includes tooling as a core of the cloud experience.

Key news in the session:

•  Windows Azure won't be in production before Jan 1st.
• .NET Services finally renamed / federated as AppFabric.
• Caching layer is now part of AppFabric.
• VM with various sizes are now available.
• Configurable VM images in Azure will come in 2010.

Windows Azure deserves a public roadmap

Last week, I had the chance to meet in person with Steve Marx and Doug Hauger, two key people part of the Windows Azure team at Microsoft.

First of all, I have been really pleased, those folks are brilliant. My own little company is betting a lot on Windows Azure. When I tell people (partners, investors, customers) about the amount of work involved to migrate Lokad toward the cloud, the most frequent feedback is that I am expecting way too much from Microsoft, that Lokad is taking way too much risk too rely on unproved Microsoft products, that Microsoft failed many times before, ...

My own belief in that matter is that Microsoft is a large company, with loads of talented people and loads of not so talented people too. Yet it seems clear to me now that Microsoft has gathered a top notch team on Windows Azure, and this alone is a very healthy sign concerning the future of Windows Azure.

In particular, Doug Hauger spend a lot time to explain to me his vision about the future of Windows Azure. Again, it was brilliant. Unfortunately, due to NDA, I won't be able to discuss here the most salient aspects of this roadmap. It's a bit sad because I am pretty sure that most of the Azure community would be thrilled - like I am - if this vision was openly shared.

Among all projects going on at Microsoft, on team that I like a lot is the C# team. In my humble opinion, C# is about one of the finest product ever released by Microsoft; and one thing that I appreciate a lot about the C# team is that they openly discuss their roadmap. C# 4.0 is not even released, and that have already started to discuss features that lies further ahead. If C# is such a good product, I believe it's precisely because every feature get openly discussed so much.

Back to Windows Azure, I think everybody would agree that cloud computing is, as a technology, about several order of magnitude more complex than any programming language (even C#). My own experience - reading questions asked on the Windows Azure Forums - is that many developers still fails to understand the cloud, and keep asking for the wrong features (ex: Remote Desktop). A roadmap would help people to avoid such pitfall, as it would make it much more obvious to see where Azure is heading.

Then, when we started migrating Lokad toward Azure about 6 months ago, we build our architecture upon a lot of guesses about the features that were most likely to be shipped with Windows Azure. So far, we have been really lucky, and Doug Hauger just confirmed me last week loads of things that we were only guesstimating so far. Yet, I would have been 10x more confident in the roadmap had been available from the start. You can't expect people to be that lucky at doing forecasts as a line of business.

The world is vast, and no matter how dedicated is the Azure team, it does not seems reasonable to expect they will be able to spend hours with every partner to enlight them with their secret roadmap. Private roadmaps just don't scale. Considering that Microsoft is a late entrant in the cloud computing market (Amazon EC2 has been in production for more than 2 years), a public disclosure of their roadmap seems unlikely to profit to any competitor (or rather the profit will be very marginal).

In the other hand, an Azure roadmap would heavily profit in very certain ways to all the partners already investing on Windows Azure; plus it would also help convincing other partners that Azure is here to stay, not just cover fire.

Azure Management API concerns

Disclaimer: this post is based on my (limited) understanding of the Azure Management API, I did start reading the docs only a few hours ago.

Microsoft has just released the first preview of their Management API for Windows Azure.

As far I understand the content of the newly released API (check the MSDN reference), this API just let you automates what was done manually through the Windows Azure Console so far.

At this point, I have two concerns:

1. No way to adjust your instance count for a given role.


2. Auto-management (*) involves loads of quirks.

(*) Auto-Management: the ability for a cloud app to scale itself up and down depending on the workload.

I am not really satisfied by this Management API as it does not seem to address basic requirements to easily scale up or down my (future) cloud app.

Being able to deploy a new azure package programmatically is nice, but we were already doing that in Lokad.Cloud. Thanks to the AppDomain restart trick, I suspect we will keep deploying that way, as the deployment through Lokad.Cloud is likely to be still 100x faster.

That being said, the Management API is powerful, but it does not seem to address auto-management, at least not in a simple fashion.

The single feature I was looking forward was being able to adjust the number of instances on-demand through a very very simple API that would have let me do three things:

1. Create new instance for the current role.


2. Shut down current instance.


3. Get the status of instances attached to the current role.

That's it!

Notice that I am not asking here to deploy a new package, or to change the production/staging status. I just need to be able tweak the instance count.

In particular, I would expect a Non-SSL REST API to do those limited operations, much like the other REST API available for the cloud storage.

Indeed, security concerns related to the instance count management are nearly identical to the ones related to the cloud storage. Well, not really, as in practice securing your storage is way much more sensitive.

Table Storage or the 100x cost factor

Until very recently, I was a bit puzzled by the Table Storage. I couldn't manage to get a clear understanding how the Table Storage could be a killer option against the Blob Storage.

I get it now: Table Storage can cut your storage costs by 100x.

At outlined by other folks already, I/O costs typically represents more than 10x the storage costs if your objects are weighting less than 6kb (the computation has been done for the Amazon S3 pricing, but the Windows Azure pricing happens to be nearly identical).

Thus, if you happen to have loads of fine grained objects to store in your cloud, say less-than-140-characters tweets for example, you're likely to end-up with an insane I/O bill if you happen to store those fine-grained items in the Blob Storage.

But don't lower your hopes, that's precisely the sort of situations the Table Storage has been designed for, as this service lets you insert/update/delete entities by batches of 100 through Entity Group Transactions.

This fine-grained item orientation is reflected in the limitations that apply to entities:

• A single entity should not weight more than 1MB.


• A single group transaction should not weight more than 4MB.


• A single entity property should not weight more than 64kb.

Situations where loads of small items end-ups being processed - threshold being at 60kb - by your cloud apps are likely to be good candidate for the Table Storage.

We will definitively try to reflect this in our favorite O/C mapper.

Thinking the Table Storage of Windows Azure

Disclaimer: I am not exactly a Table Storage expert. In this post, I am just trying to sort out my own thoughts about this service offered with Windows Azure. Check my follow-up post.

Soon after the release announcement of the release of our new O/C mapper (object to cloud) named Lokad.Cloud, folks on the Azure Forums raised the question of the Table Storage.

Although it might be surprising, Lokad.Cloud does not provide - yet - any support for Table Storage.

At this point, I feel very uncertain about Table Storage, not in the sense that I do not trust Microsoft to end-up with finely tuned product, but rather at the patterns and practices level.

Basically, the Table Storage is an entity storage that features three special system properties:

• PartitionKey: a grouping criterion - data having the same PartitionKey being kept close.


• RowKey: the unique identifier for the entity.


• Timestamp: the equivalent of Blob Storage ETag.

So far, I got the feeling that many developers feel attracted toward the Table Storage for the wrong reasons. In particular, Table Storage is not a substitute of your old plain SQL tables:

• No support for transactions.


• No support for keys (let alone foreign keys).


• No possible refactoring (properties are frozen at setup).

If you are looking for those features, you're most likely betting on the wrong horse. You should be considering SQL Azure instead.

Then, some might argue that SQL Azure won't scale above 10GB (at least considering the current pricing plans offered by Microsoft). Well, the trick is Table Storage won't scale either, at least not unless you're not very cautious with your queries.

AFAIK, the only indexed column of the Table Storage is the RowKey. Thus, any filtering criterion based on custom entity properties is likely to get abyssal performance as soon your Table Storage get large.

Well, sort of, the most probable scenario is like to to be worse as your queries are just going to timeout after exceeding 60s.

Again, my goal here is not to bash the Table Storage, but it must be understood that the Table Storage is clearly not a magically scalable equivalent of the plain old SQL tables.

Back to Lokad.Cloud, we did not consider adding Table Storage because we did not feel the need either although our forecasting back-end is probably very high in the currently complexity spectrum of the cloud apps.

Indeed, the Blob Storage is surprisingly powerful with very predicable performance too:

• Storing complex objects is a non-issue with a serializer at hand.


• A blob name prefix is a very efficient substitute to the PartitionKey.

Basically, it seems to me that any Table Storage operation can be executed with the same performance with the Blob Storage for now. Later on, when the Table Storage will start supporting secondary indexes, this situation is likely to evolve, but meantime I still cannot think a single situation that would definitively support Table Storage over Blob Storage.