Author

Portrait of Joannes Vermorel

I am Joannes Vermorel, founder at Lokad. I am also an engineer from the Corps des Mines who initially graduated from the ENS.

I have been passionate about computer science, software matters and data mining for almost two decades. (RSS - ATOM)

Meta
Tags
« Telling the difference between cloud and smoke | Main | Big Wish List for Windows Azure - PDC10 update »
Wednesday
Dec222010

A few tips for Web API design

During the iteration in spring 2010 that has lead Lokad to release its Forecasting API v3, I have been thinking a lot about how to design proper Web APIs in this age of cloud computing.

Designing a good Web API is very surprisingly hard. Because Remote Procedure Call has been around forever, one might think that designing API is a well-known established practice, and yet, suffering the defects of Forecasting API v1 and v2, we learned the hard way it wasn't really the case.

In this post, I will try cover some of gotchas that we learned the hard-way about Web API design.

1. Full API stack ownership

When you expose an API on the web, you tend to rely on building-blocks such as XML, SOAP, HTTP, ... You must be prepared to accept and to support a full ownership of this stack: in order words, problems are likely to happen at all levels, and you must be ready to address them even if the problem arises in one of those building blocks.

For example, at Lokad, we initially opted for SOAP, and I now believe it was a mistake from Day 1. There is nothing really wrong with SOAP itself (*), the problem was that the Lokad team (myself included) was not ready to support SOAP to in full. Whenever a client was raising a subtle question some obscure XML namespace or SOAP version matters, we were very far from our comfort area. In short, we were relying on a complex SOAP toolkit which proved to be a leaky abstraction.

When it comes to public API, you have to be ready to support all abstraction leaks from the TCP, HTTP, XML, SOAP... That why, in the end, for our Forecasting API v3, we settled for POX (Plain Old Xml) over HTTP with a basic REST philosophy. Now, when an issue arises, we can trully own the problem along with its solution, instead of being helpless facing a leaky abstraction that we don't have the resource to embrace.

(*) Although there are definitely many gray areas with SOAP (just look at SOAP Exceptions for example).

2. Be minimalistic

Dealing with all the software layers that sit between the client code and the Web API itself is bad enough, if the API itself adds to complexity of its own, the task quickly becomes maddening.

While refactoring our Forecasting API v2 into the Forecasting API v3, I reduced the number of web methods from +20 to 8. Looking back, it was probably one of the best insights I had.

When developing a software library, it is usually convenient for the client developer to benefit from many syntactic sugars that is to say method variants that achieve the purpose following various coding style. For a Web API, the reverse is true: there should be one and only one way to achieve each task.

Redundancy within methods only cause confusion and extra-friction when developing against the API, and while cause an endless stream of questions whether the method X should be favored against the method X* while both achieve essentially the same thing.

3. Idempotence

Network is unreliable. With a low frequency, API calls will fail because of network glitches. The easiest way to deal with those glitches is simply to have retry policies in place: if the call fails, try again. If your API semantic is idempotent, retry policies will integrate seamlessly with your API. If not, each network glitch is likely to wreak havoc within the client app.

For example, coming from a SQL background, most developer might consider having both INSERT (only for new items) and UPDATE (only for existing items) methods. INSERT is a bad choice, because if the method is attempted twice because of a retry, the second call will fail probably triggering some unexpected exception on the client side. Instead you should rather adopt UPSERT, ie UPDATE or INSERT, semantics which play much nicer with retry policies.

4. Explicit input/output limitations

No web request or response should be allowed to be arbitrarily large as it would cause a lot of problem server side to maintain decent performance across concurrent web requests. It means that from Day 1 every message that goes in or out your API is explicitly capped: don't let your users discover your API limitations through reverse engineering. In practice, it means that no array, list or string should be left unbounded: max length should always be specified.

Then, unless you precisely want to move plain text around, I suggest to keep tight limitation on any string being passed on your API. In the Forecasting API, we have opted for the Regex pattern ^[a-zA-Z0-9]{1,32}$ for all string tokens. Obviously, this is rather inflexible, but again, unless your API is intended for plain text / binary data storage there is no point is supporting the whole UTF-8 range which can prove very hard to debug and trigger SQL-injection-like problems.

 

References (43)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Response
    epson printer driver download
  • Response
    fujitsu.com
  • Response
    Response: hp drivers update
    how to download hp drivers
  • Response
    lexmark drivers
  • Response
    winbook drivers download
  • Response
    Response: Cheat App Tools
    A few tips for Web API design - - - Joannes Vermorel's blog
  • Response
    Response: windows phone
    [...]A few tips for Web API design - - - Joannes Vermorel's blog[...]
  • Response
    Wise restore Outlook PST is well-known recovery tool for outlook. It provides advanced utility to retrieve Outlook PST files as well as to extract entire emails and contacts from corrupted PST files. Wise how to recover Outlook PST helps you to fix corrupted or damaged PST files; you can just open ...
  • Response
    MKV To MP4 Converter is a professional video converter which is specialized in converting videos from one format to another, especially from MKV to MP4.
  • Response
    Smart repair Explorer Not Responding error Pro is an perfect computer error fixer and Windows registry cleaner. Running this tool on a scheduled basis, will work out a stable system and will greatly enhance system performance. The scan technology helps the user quickly [identify|locate errors and invalid entries that cause the ...
  • Response
    A few tips for Web API design - - - Joannes Vermorel's blog
  • Response
    Response: driver download
    kodak printer drivers windows 7 Utility is capable of displaying a detailed report about all installed drivers (versions, release dates) and offer you with Windows Experience Index rating information before you buy a new hardware.
  • Response
    mmc card recovering is a perfect, well-known data recovery software which can help you restore all lost data in a MMC card.
  • Response
    Smart why is my pc running slow Pro is a popular downloading tool to fix registry errors. It performs comprehensive and deep analysis for Windows Registry, and repair invalid entries or PC slowdowns.Smart why is my pc running slow Pro supply you with Registry Backup, System Backup, Favorites Backup and Folder ...
  • Response
    sms recovery android Pro is available which is especially designed and developed to recover missing or deleted sms from smartphones or tablets.
  • Response
    Smart how can i speed up my computer Pro is a top downloading tool to fix registry errors. It performs comprehensive and deep analysis for Windows Registry, and repair invalid entries or PC slowdowns.Smart how can i speed up my computer Pro provides you with Registry Backup, System Backup, Favorites Backup ...
  • Response
    SDHC card recovery pro has powerful and various functions. This attracts so many people to undelete important files via recover sdhc card pro. recover sdhc card pro is easy to operate. Just a few clicks you will undelete your lost of SDHC. Using corrupt sdhc card recovery pro is safe and ...
  • Response
    Response: Zip Disk recovery
    Wise how to recover Zip Disk files is considered as the most efficient and effective restore Zip Disk data which is capable of recovering deleted files from zip disk. A zip disk is a computer disk, similar to a floppy disk but capable of storing a mass of data. It can ...
  • Response
    Response: usb data recovery
    Wise Recovering USB Files is regarded as a multi-functional program which is designed with powerful algorithms that helps to identify various types of files such as audios, videos, images, documents etc and recovers them easily. undeleting deleted files from USB drive after accidental or intentional deletion and deleted files during antivirus ...
  • Response
    Wise recovering Windows deleted data is accessible in a portable version making installation unnecessary - a very important advantage. I especially love the simple Folders and File Types views available in Wise recovering Windows deleted data which make finding the file you'd like to undelete much easier than in some other ...
  • Response
    A few tips for Web API design - - - Joannes Vermorel's blog
  • Response
    how to recover deleted files from sd card
  • Response
    Response: sd recovery
    how to recover deleted photos from sd card
  • Response
    Response: photo recovery
    sd card picture recovery
  • Response
    how to recover notes from ipad
  • Response
    memory stick data recovery
  • Response
    picture recovery
  • Response
    android recovery
  • Response
    how to recover deleted photos from sd card
  • Response
    Response: data recovery mac
    how to recover photos from android
  • Response
    Response: telecharger skype
    A few tips for Web API design - - - Joannes Vermorel's blog
  • Response
    A few tips for Web API design - - - Joannes Vermorel's blog
  • Response
    A few tips for Web API design - - - Joannes Vermorel's blog
  • Response
    sd card picture recovery pro is a great data recovery program which can help you undelete any missing data in a SD card. how to recover pictures from sd card pro recovers lost or deleted datas in 3 Steps. recover pictures from sd card pro supports a hot of camera memory ...
  • Response
    A few tips for Web API design - - - Joannes Vermorel's blog
  • Response
    Response: siemens-ag.com.ua
    A few tips for Web API design - - - Joannes Vermorel's blog
  • Response
    I found a great...
  • Response
    I found a great...
  • Response
    Response: senuke
    A few tips for Web API design - - - Joannes Vermorel's blog
  • Response
    how to recover pictures from sd card pro is a perfect data recovery software which can help you restore any lost data in a SD card. how to recover deleted pictures from sd card pro recovers lost or deleted files in 3 Steps. recover pictures from sd card pro supports a ...
  • Response
    A few tips for Web API design - - - Joannes Vermorel's blog
  • Response
    Response: Mark Galliano
    A few tips for Web API design - - - Joannes Vermorel's blog
  • Response
    A few tips for Web API design - - - Joannes Vermorel's blog

Reader Comments (7)

5. You can always add, but you can never remove

Good read: http://vermorel.com/journal/2010/12/22/a-few-tips-for-web-api-design.html

December 22, 2010 | Unregistered CommenterBjörn Raupach

"probably once of the best insights I had" perhaps should be "probably one of the best insights I had"

December 22, 2010 | Unregistered CommenterFriendly Grammar Guy

"If you API semantic is idempotent" perhaps should be "If your API semantic is idempotent"

December 22, 2010 | Unregistered CommenterFriendly Grammar Guy2

UPSERT? More commonly known as SET.

December 23, 2010 | Unregistered CommenterMatthew Holloway

Nice post. I like #3. I currently have insert and update separated, maybe in v2 of my current project I could combine those. I also agree about the simplicity in #1. I use rest whenever it might fit because all the verbs are already known and documented, error codes are already known and documented, its almost to the point where a client can just make calls and look at the data they get back ;) I try to just have client get/post/etc json objects, it keeps things simple.

December 23, 2010 | Unregistered Commenterryan day

you might want to double-check that regex to make sure it's anchored at the start of the string as well, or it will just let anything that ends with alphanumerics through. like ^[a-zA-Z0-9]{1,32}$

December 23, 2010 | Unregistered CommenterChris

Thanks to you all for the various suggested corrections, including the Regex one (well actually, it's already implemented that way within Lokad).

Concerning the suggestion 5. You can always add, but you can never remove: it seems very close to the event sourcing idea that we implement in several products at Lokad (Salescast being the flagship here), and indeed I would mostly agree. Maybe at some point we will end-up rewriting our Forecasting API with an event sourcing + CQRS viewpoint.

December 23, 2010 | Registered CommenterJoannes Vermorel

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
All HTML will be escaped. Hyperlinks will be created for URLs automatically.