Learning to Let Go – Self Organising Systems

We like to think that things that work well come about through good organisation, rigorous planning and strong front led management. But is this really true ? If we look at nature there’s some excellent examples of self-organising systems that “just seem to work”. They manage to co-ordinate hundreds of thousands of individuals into building some phenominal structures. Can we learn from this and learn to “let go” ?

It turns out we probably can and there are various methodologies and frameworks actively being used to organise teams to solve some very complex problems indeed. In this article we look at a few aspects of this.

What is a “Self Organising System”

Firstly let us define what we mean by a “self organising system” :

Self-organisation is a process where some form of global order or coordination arises out of the local interactions between the components of an initially disordered system. This process is spontaneous: it is not directed or controlled by any agent or subsystem inside or outside of the system. The resulting organization is wholly decentralized or distributed over all the components of the system. As such it is typically very robust and able to survive and self-repair substantial damage or perturbations.

To some this might sound like chaos and a recipe for disaster. How can anything that is remotely complex be successfully built using such a seemingly chaotic and disordered system.

Examples in Nature

Flocking

Starlings Flocking

Flocking behaviour is the behaviour exhibited when a flock of birds are foraging or in flight. There is no one leader or organiser of these complex movements they come about from the self-organising behaviour of each individual within the flock obeying certain simple rules of interaction.

In Cologne in Germany, biologists have even demonstrated a flock like behaviour in humans. The group of people exhibited a very similar behavioural pattern to that of a flock, where if 5% of the flock would change direction the others would follow suit.

There are also parallels with the shoaling behaviour of fish, the swarming behaviour of insects, and herd behaviour of land animals.

Termite Cathedrals

A Termite Cathedral

In northern Australia termite colonies build huge above ground structures termed cathedrals. They are large, complex and extremely sophisticated.

These structures aren’t actually lived in by the termites themselves, they live below the surface. These edifices act as giant air conditioning units. They are built like an aeroplane wing so that an area of low pressure is formed when the wind blows across them serving to ventilate the colony below ground.

A mature cathedral mound can be around 5m tall. Given it is built by insects around 5mm long this is equivalent to humans getting together and building a massive skyscraper over a kilometre high and covering many city blocks.

Again they are built from the termites self-organising themselves with no discernible plan or a hierarchical organisational structure directing the building efforts.

Harnessing Self-Organisation in Software Development

Software by it’s very nature is complex and putting it all together is even more complex when you think that hundreds of humans may all be working on that same piece of software.

Take for example a computer’s operating system like Windows or Mac OS. This piece of software not only has to provide a user interface but has to display items on screen, communicate with the outside world via the internet, send items to a printer and remember things in memory. To make things even more complicated no two PCs are the same so that graphics chip on your PC maybe different to another graphics chip on another PC but they both are running the same version of Windows.

You might think therefore that a strict and rigorous control system with lots of design and analysis up front would be needed. However there are a growing number of methodologies and frameworks that actively discourage this approach and encourage Self-Organization. Together these methodologies are collectively called “Agile Software Development”. It may further surprise you to know that Windows 7 was built using an agile framework called “Scrum”.

A lot of software has traditionally been built using a technique called the “Waterfall methodology”. You analyse the requirements for a client, produce detailed specifications and designs, build the software and then test it. After all this it then gets delivered to the client. For complex systems the problem with big analysis and design up front is that, by the time it gets delivered to market, a considerable amount of time will usually have expired. In addition when it does arrive it’s often out of date as it may have taken years to build and the market has moved on or something in the requirements was missed and the cost of changing it so late in the day after everything has been tested is high.

Agile methodologies seek to solve these problems. The main principles behind these Agile methodologies, also called the “Agile Manifesto”, are shown opposite in the side-bar. However in a nutshell …

• The team implementing the solution organize and liaise directly with a product owner (ie the business) on priorities & planning
• The team self-organises based on these priorities & estimates accordingly
• The team self-organises the specifications & implementation details just for the goals defined
• The teams efforts are ‘time-boxed’ ie they have a fixed cycle of development at eth end of which working software HAS to be delivered.
• The ‘time boxes’ are designed to focus development to produce something tangible after every period (usually 2 weeks to 1 month). Thus time to delivery of working software is reduced and functional roll-out staged.
• Responding to changing requirements or changing market condition is not longer so painful.