Author

Portrait of Joannes Vermorel

I am Joannes Vermorel, founder at Lokad. I am also an engineer from the Corps des Mines who initially graduated from the ENS.

I have been passionate about computer science, software matters and data mining for almost two decades,

Meta

Entries in best practices (3)

Monday
Jan112010

Scaling-down for Tactical Apps with Azure

Cloud computing is frequently quoted for unleashing the scalability potential of your apps, and the press goes wild quoting the story of XYZ a random Web 2.0 company that as gone from a few web servers to 1 zillion web servers in 3 days due to massive traffic surge.

Yet, the horrid truth is: most web apps won’t ever need to scale, and an even smaller fraction will even need to scale out (as opposed of scaling up).

A pair of servers already scales up to millions of monthly visitors for an app as complex as StackOverflow. Granted, people behind SO did a stunning job at improving the performance of their app, but still, it illustrates that moderatly scaling up already brings you very far.

At Lokad, although we direly need our core forecasting technology to be scalable, nearly all  other parts do not. I wish we had so many invoices to proceed that we would need to scale out our accounting database, but I don’t see that happen any time soon.

Actually, over the last few months, we have discovered that cloud computing have the potential to unleash yet another aspect in the software industry: the power of scaled-down apps.

There is an old rule of thumb in software development that says that increasing the complexity of a project by 25% increases the development efforts by 200%. Obviously, it does not look too good for the software industry, but the bright side is: if you cut the complexity by 20% then you halve the development effort as well.

Based on this insight, I have refined the strategy of Lokad with tactical apps. Basically, a tactical app is a stripped-down web app:

  • not core business, if the app crashes, it’s not a showstopper.
  • features are fanatically stripped down.
  • from idea to live app in less than 2 weeks, single developer on command.
  • no need to scale, or rather very unlikely.
  • addresses an immediate need.

Over the last couple of weeks, I have released 3 tactical apps based on Windows Azure:

Basically, each app took me less than 10 full working days to develop, and each app is addressing some long standing issues in its own narrow yet useful way:

  • Website localization had been a pain for us from the very beginning. Formalized process where tedious, and by the time the results were obtained, translations were already outdated. Lokad.Translate automates most of the mundane aspect of website localization.
  • Helping partners figuring out their own implementation bugs while they were developing against our Forecasting API was a slow painful process. We had to spend hours guessing what could be the problem in partner's code (as we typically don’t have access to the code).
  • Helping prospects to figure out how to integrate Lokad in their IT, we end-up facing about 20 new environments (ERP/CRM/MRP/eCommerce/…) every week, which is a daunting task for a small company such as Lokad. Hence, we really need to plug partners in, and Lokad.Leads is helping us to that in a more straightforward manner.

Obviously, if we were reach 10k visits per day for any one of those apps that would be already a LOT of traffic.

The power of Windows Azure for tactical apps

Tactical apps are not so much a type of apps but rather a fast-paced process to deliver short-term results. The key ingredient is simplicity. In this respect, I have found that the combination of Windows Azure + ASP.NET MVC + SQL Azure + NHibernate + OpenID is a terrific combo for tactical apps.

Basically, ASP.NET MVC offers an app template that is ready to go (following Ruby on Rails motto of convention over contention). Actually, for Lokad.Translate and Lokad.Debug, I did not even bother in re-skinning the app.

Then, Windows Azure + SQL Azure offer an ultra-standardized environment. No need to think about setting up the environment, environment is already setup, and it leaves you very little freedom to change anything which is GREAT as far productivity is concerned.

Also, ultra-rapid development is obviously error-prone (which is OK because tactical apps are really simple). Nevertheless, Azure provides a very strong isolation from one app to the next (VM level isolation). It does not matter much if one app fails and dies suffering some terminal design error, damage will be limited to app itself anyway. Obviously, it would not have been the case in a shared environment.

Finally, through OpenID (and its nice .NET implementation), you can externalize the bulk of your user management (think of registration, confirmation emails, and so on).

At this point, the only major limitation for tactical apps is the Windows Azure pricing which is rather unfriendly to this sort of apps, but I expect the situation to improve over 2010.

·         not part of your core business, if the app crashes, it’s annoying, but not a showstopper.

·         features are fanatically stripped down.

·         from idea to live app in less than 2 weeks, single developer on command.

·         no need to scale, or rather, the probability of needing scalability is really low.

·         addresses an immediate need, ROI is expected in less than a few months.

Monday
Nov302009

Continuous Localization or l10n 2.0

There is nothing is easier to sell globaly than software. Yet, it's still surprising to me to see how few efforts are made on average by small software companies toward localization.

Disclaimer: I am not saying that selling software anywhere is easy. Some places are really tough. I am just saying that selling about anything else worldwide level is just 10x harder.

Translation is (relatively) cheap

Localization (l10n in short) is easier and cheaper than you think. In a past project of mine, a few years ago, I managed to translate a web app (freelance marketplace) in 13 languages for less than $2000. Yes, that's right, it was roughly $150 per language.

The first thing to understand here is that freelance translation is cheaper than usually thought. Generic freelance considerations apply, but compared to the massive efforts needed to actually develop and maintain any small piece of software, translation is just dirty cheap.

... but management is not ...

Yet, if translation is unexpensive, managing translators is not. Each time I took over localization works, I relealized that managing half a dozen of remote translators on an ongoing basis was nearly requiring a full-time commitment from my side.

If think, most companies realize this effect intuitively. The bottom line result is that localization is typically performed in big batches.

Big batches seem to be the archetype of the non-agile process. Every two years, package all documents and hand them over to some translation agency. Wait for two months. Publish the (already outdated) documents, and wait more. Two years later, once documents are desperatly outdated, repeat.

Although, I can't blame the community doing it that way, as I was doing no better. Yet, this process felt wrong. Since localization is such a big time-consuming mess, we do it only once in a while and meantime prospects and customers suffer outdated materials on an ongoing basis, which, somehow, is even worse than poor-quality translation.

... hence Continuous Localization

Among all good practices in software development, I have found continuous integration to be one of the few breakthrough that have significantly improved agility in project management. The core idea being continuous integration is that integration becomes part of your daily process.

Instead of updating the deployment logic once every 18months, you do it on an ongoing basis, so that the software is already ready to ship. Yet, continuous integration comes with a gotcha: you can't do it by hand. It takes another layer of automation: the integration server.

Thinking about it, localization is similar to integration

The simple idea of incremental localization without some automation seemed doomed as it would require insane communication efforts between manager and translators.

Then, what about adding this automation layer to simplify the process?

Let see how the localization process could be automated:

  1. (automated) Get all source documents and incremental updates.
  2. (automated) Map updates to every target languages.
  3. (manual) Apply corresponding incremental updates to target documents.
  4. (automated) Keep track of the amount of work made by each translator.
  5. (automated) Keep track of work batch to get translators paid.

Obviously, the one step that cannot be automated this translation operation itself; but then all other steps can be vastly automated.

This idea has been the starting point of a project codenamed Lokad.Translate. This project is nothing more than a webapp playing the role of a localization server and providing all the automation that we can get to speed-up the localization process - both on the management side, but on the translator side as well.

Tech note: Lokad.Translate is ASP.NET MVC + NHibernate on top of Azure.

Since we did not want to reinvent the wheel, we decided to leverage the capabilities of the wiki powering our own company website. In particular, in order to retrieve the list of incremental changes, we using nothing else but the web feed generated by the wiki (RSS in present case, although it does not matter much) that represents recent changes. The nice thing about web feeds is that most webapps are already providing one (think blogs).

Then, concerning document management (both originals and translations), there is a gotcha: there is no need to manage documents themselves, as managing the URLs pointing to the documents is enough. Once the URL is known, if a REST API is provided by the wiki, all other commands (view/diff/edit/...) could be inferred with simple REGEX.

My objective would be to achieve a near continuous localization of the content posted on our website with say, no more than 2 weeks of delay between initial post and its localization with minimal overhead both on the management and translator. We will soon start deploying Lokad.Translate for our internal need, we will see how it goes.

Then, depending of community, we will probably provide release Lokad.Translate one way or another. Stay tuned for more (and don't hesitate to contact me if you're interested).

Thursday
Sep172009

Table Storage or the 100x cost factor

Until very recently, I was a bit puzzled by the Table Storage. I couldn't manage to get a clear understanding how the Table Storage could be a killer option against the Blob Storage.

I get it now: Table Storage can cut your storage costs by 100x.

At outlined by other folks already, I/O costs typically represents more than 10x the storage costs if your objects are weighting less than 6kb (the computation has been done for the Amazon S3 pricing, but the Windows Azure pricing happens to be nearly identical).

Thus, if you happen to have loads of fine grained objects to store in your cloud, say less-than-140-characters tweets for example, you're likely to end-up with an insane I/O bill if you happen to store those fine-grained items in the Blob Storage.

But don't lower your hopes, that's precisely the sort of situations the Table Storage has been designed for, as this service lets you insert/update/delete entities by batches of 100 through Entity Group Transactions.

This fine-grained item orientation is reflected in the limitations that apply to entities:

  • A single entity should not weight more than 1MB.

  • A single group transaction should not weight more than 4MB.

  • A single entity property should not weight more than 64kb.

Situations where loads of small items end-ups being processed - threshold being at 60kb - by your cloud apps are likely to be good candidate for the Table Storage.

We will definitively try to reflect this in our favorite O/C mapper.