Applying Software Patterns to the Cloud and the Internet Operating System

The MVC for the Cloud meme got me thinking about other principles that might apply to programming the Web as a whole. A lot of the principle's we routinely apply to web apps to great effect seem to have analogies when taken beyond individual apps. While MVC seems a clear fit, here are some others:

• DRY - Don't Repeat Yourself!
• Dynamic Languages
• Convention over Configuration
• Data and Control Abstraction
• Continuous Integration

Taken together with MVC these also make a lot of sense applied to the cloud. On the first one, I think it's fair to say we're already getting there:

• DRY (or DIE): the last 20 years have seen an amazing ecosystem of open source software become available and most programming frameworks have a truly awesome ecosystem of plugins and code available to make the core frameworks more powerful. Now this is starting to be replicated at the service level - there are very few reasons now to develop your own credit card processing, email sending, exchange rate checking etc. systems - these are all freely available via external services which can be integrated by API. The trend seems to be accelerating with even user authentication management (Facebook Connect), file storage (Amazon S3), and a large volume of external data sources (event databases, weather, finance etc.) all starting to become externally available. It's becoming the rule that before you code anything or gather anything it makes sense to say "Is there an API for that?".

This isn't an argument that there should be only ONE API for it - diversity is crucial, but one perhaps the may just be ONE unified common interface. For the next, if we look hard enough I'd say the Web is already the best example of a runtime computing system based on dynamic languages: 

• There is no fixed type system: While SOAP Web Services have taken hold within the enterprise, on the Internet REST is King and there are few formal specifications used for API Interfaces (WSDL, WADL and others do see use - but it's still sparse) - although this will change over time and we encourage our customers at 3scale to provide good specs for their APIs, we're a long way from a formal type system since there are no shared object definitions which are commonly agreed. (Semantic Web efforts are helping change this is also - but I suspect they'll take a long time to break through).
• Even if there were a fixed type system there would still be no notion of "compilation" in the cloud for applications which span multiple services because different services are operated by different companies and individuals and could change at any time.
• End services can also be written in almost any web language from C++ or Ada to Django or Java - even if everything could be frozen and "compiled" there would be no way to do it in the traditional sense. 

"It's about the Interface Stupid" might apply well here because amazingly things still work - today's API providers are realizing the importance of taking care of their interfaces over time. In the long run, at 3scale we see this as evolving into a Contract First environment where we don't just see languages like WSDL & WADL define what an interface does functionally, but what the precise commitments are from the provider to the users of their services (see some great work on this in the European Commission funded IST-CONTRACT project btw).

Seeing the Cloud as a big Operating System based on dynamic languages means the following two memes are all the more important, but are only just emerging:

• Convention over Configuration: Frameworks such as Ruby on Rails have a huge advantage over other web development frameworks by using the idea that if a set of "normal ways of doing things" was established then in many cases there would be no need to maintain complex configuration files everywhere (JSPs - we're looking at you..), what are often a few logical lines of code in a rails app required multiple lines in other configuration files in a JSP driven App. In the cloud it's fair to say this idea is slowly emerging - c.f. the adoption of Twitter's API structure by a range of other services. REST itself is also something which follows a Convention pattern with it's 7 core operations applied to RESTfull data objects. We're a still a long way from being able to integrate 3rd party services almost without checking the documentation. 
• Data and Control Abstraction: within modern programming frameworks ORM (Object Relational Mappings) provide a powerful layer of abstraction over database entries that allow developers to think in terms of objects rather than lower level structures. Again in the cloud REST helps us generate some of this abstraction - new NoSql database even make some mappings even clearer (check out this amazing app - Jquery + CouchDB - look no middle layer.). Web APIs are by their very nature an abstraction of data and functionality - without this layer we would be writing raw SQL queries into linkedIn's database rather than using their API. What hasn't yet emerged is a standard model for representing many of the objects which are being shared - again something progress of the Semantic Web may change.

Programmable Web now lists nearly 5000 mashups of 2000+ Public APIs - this is a huge number of integrations - and is testament to the hardworking developers than made this happen (each one of those links took a lot of debugging) - conventions and abstractions can come fast enough to make this all easier. Those 5000 mashups also highlight something which relates to the last of the 5 memes I picked on - continuous integration:

• Continuous Integration: applying continuous integration to software development is a no brainer for almost any complex system - the technique relies on the creation of batteries of software tests for all aspects of the software application which can be executed in an automated fashion (often the tests are written even before the application code) and runs these tests on any new build of the system when it becomes available. This means that if new code breaks the development build there's immediate awareness of the problem and it can be fixed. Continuos Integration allows a dev team to keep track of how the code matches up to how it is supposed to behave at all time (we keep 3scale's Hudson Installs pretty busy :-) ). In the cloud context, we're still only taking baby steps towards an equivalent of automated testing and continuous integration. Most tests are still before deployment only and after that we hand of to far less detailed monitoring systems - despite the fact that many apps depend on external services. Continuous Integration will likely eventually need to be really continuous and operate at runtime against live deployed systems - not just pre-deploy time.

All in all, we're likely to want to replicate all the power and flexibility we have available for building individual applications to Applications which depend on Multiple cloud services. The scope of the problem changes - but the metaphors still seem to be highly valid. 

I especially like the dynamic languages and DRY analogies - it's great to see the heterogeneity of available Web APIs and amazing to see that stable applications still emerge - Web API interfaces provide a critical glue to make this work and it's interesting to see that they are emerging "bottom up", rather than via top down de jure standards.