The Known Unknowns Matrix

I.T. is not the only industry to have happily latched onto the the former Secretary of State's famous phrase, "the unknown unknowns". It's a good phrase if you must plan or estimate anything because planning & estimating always involve risk.

But we should really consider the full matrix. There are pitfalls in at least two of the quadrants:

	Known	Not Known
Knowns	Things we know, and we know we know them	Things we know but don't realise we know them. Tacit knowledge that we take for granted. Becomes a problem if we are responsible, and fail, to communicate them to people who don't know. Also a problem when we start work in a new context and do not realise that what we ‘know’ is no longer valid here, so they become unknown unknowns.
Unknowns	Things we know that we don't know. We can record the risk, and estimate a cost for investigation & discovery	Things we don't know that we don't know. This is the quadrant most likely to shipwreck plans.

Concerning the unknown unknowns, my experience with doing novel software is that when budgeting for development you should estimate for development, plus learning time, plus developing the things you learned about, plus solving problems you didn't know you'd have. A rule of thumb for novel systems might be, multiply your estimate by ten to cope with the unknowns. And/Or, have clear “abandon the project” criteria, even months into the project. Don't be a dupe for the sunk-cost fallacy.

Less dramatically, my takeaway from this is to use this quadrant when listing risks and assumptions. Just having a space for the possibility of unknown knowns & unknowns can be an impetus to discuss, “risk-storm” & consult, to help your team discover the as-yet-unknowns.

P.S.

I've just read the brief and brilliant mcfunley.com/choose-boring-technology which points out that you can't afford too much novelty. He suggests that for any new project you should grant yourself a limit of 3 novelty chips. When you've spent them, you get no more.

Well, not unless you really can overshoot your budget and timescales by over 1,000%.

Kudos

A slideshare by Danni Mannes on Agile Architecture pointed out to me all quadrants are worth some of our time.

Coplien & Bjørnvig : Lean Architecture For Agile Developers. A Review

Four years after this book came out, Agile Architecture has at last become a Thing. But as the nuance of its title hints, this book is not fad-driven. It is a carefully-thought out exposition of what architects can learn from lean and agile ideas, and what they can do better as a result.

Well. It's partly that. If you are a practising architect, it is actually four must-read books in one.

If you are not, you might dismiss this book for two reasons. The first, that judging by other reviews & my own experience, the homespun style of the first half does not suit a bullet-pointed gimme-the-headlines-now generation. The second is that if you have not experienced the pitfalls of architecting and doing software in a real organisation with actual people in it, then Coplien & Bjørnvig's pearls of wisdom may impress you much as the agile manifesto might impress a cattle rancher.

It is not a beginner's book. It is the mature distillation of the sweat-soaked notebooks of a fellow-traveller who has stumbled over rocky terrain, been through the tarpits and has some hard-bloody-earned (and, academically researched) wisdom to share as a result.

I said four must-reads in one. They are:

1) The (literally) decades of experience of a leading practitioner & thinker in the field.

2) A thought-through answer to the questions, what can we learn from lean & agile. Whilst value-chains and some technique feature, the authors' secret conviction is surely that Technology Is All About Human Beings. "Everybody, all together, from early on" is their Lean Secret. "Deferring interaction with stakeholders [users, the business, customers, domain experts, developers], or deferring decisions beyond the responsible moment slows progress, raises cost and increases frustration. A team acts like a team from the start."

3) Which leads to that which for me, as a more techy-focused reader, was the marvel of the book. Clements et al in "Software Architecture in Practise" offered attribute-driven design, a partitioning of the system based on a priority ordering of, primarily, technical quality requirements. Coplien & Bjørnvig all but deduce a partitioning based on the priority ordering of people: Users first, Development team next.

"The end-users' mental model" is the refrain on which they start early and never stop hammering. From this they suggest that the first partitioning is What the System Is from What the System Does. I am tempted to paraphrase as, Domain Model from Use Cases. But their point is partly that this also neatly matches the primary partition by rate of change, because the last thing to change in a business is, the business of the business. If you're in retail, you might change what and how and where you sell, but your business is still Selling Stuff. And at lower levels inside the organisation too: an accounting department, though there has been five thousand years of technology change, still does accounting. The users know this. They understand their domain and if the fundamental form of your software system matches the end user mental model then it can survive–nay, enable!–change and stay fundamentally fit for purpose.

Second, don't fight Conway's law. "Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations". The deduction for agile teams is, partition so as to maximise the long-term autonomy of your self-organising teams. Even when the team divisions are imposed for non-technical reasons–geography, politics, whatever–still allow that fact to trump more 'technical' considerations. This may not match your vision of technical perfection, but it will still be the best way. Ruth Malan recently paraphrased Conway's law as, "if the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins". Don't be a loser.

These points seem to me obvious in hindsight, yet they turn traditional approaches to high-level designs on their head. After considering People, yes we can considering rates of change, quality attributes, technology areas. But it's always People First.

4) And finally. This is the first book-sized exposition of DCI architecture, which I would describe as an architectural pattern for systems that have users. Having separated What the System Is from What the System Does, DCI provides the design pattern for how What the System Does (the Use Cases) marshals the elements of the What the System Is. There are at least three notable outcomes.

Firstly that use cases are mapped closely to specific pieces of code; in the best case each use case can be encapsulated in its own component.

Secondly, that the relationship between Domain elements and Use Cases are expressed as "In Use Case X Domain element Y plays the Role of Z". This brings significant clarity to both, and is part of the key to 'componentising' use cases; the Roles needed for a use case become, in the code, its public dependencies. In UML-speak, the Required Interfaces for such a UseCaseComponent are the roles needed to 'play' the use case, and those roles are Interfaces which are implemented by Classes in the Domain Model. Coupling is reduced, cohesion is gained, clarity is achieved.

Thirdly, the simpleness of the mapping from business architecture to code is greatly increased. Suddenly one can draw simple straight lines between corresponding elements of architecture and code.

The authors say of their work, "This book is about a Lean approach to domain architecture that lays a foundation for agile software change". To my mind, this hits the agile architecture nail on the head. Agile software development always only ever succeeded at scale because the people doing it either knew, or had given to them, enough architecture to make it work. Software Agility, just like every other software -ility, must either be supported by the architecture or it ain't gonna happen.

But the best thing I got from this book was the proof, before my very eyes, that correct technical design flows from knowing how to put the human beings central.

Where to Buy

UK

ebay (UK): Lean Architecture for Agile Software Development

Amazon (UK): Lean Architecture for Agile Software Development

USA

Amazon (USA): Lean Architecture for Agile Software Development

A Single Deployment Target is not a Monolith

Since the dawn of software architecture, we mostly knew that 'Monolithic Architecture' was more or less a synonym for ball of mud architecture. It mostly referred to code which was so entangled and coupled that it could not be separated out into components (or layers, or modules, or libraries, or equivalent). God classes, lack of information hiding, a change in one place means 20 other changes. That kind of thing. A failure of Modularity.

Recently the term Monolith has been taken to mean having a Single Deployment Target at runtime. This is a quite different meaning.

If you think that Monolith as described in para 1 above is the same as Monolith in para 2 above, then I suggest that you have confused, not separated, your concerns.

This is easy to understand if you are comfortable with architecture views. The para 1 definition is about the logical and (in 4+1 lingo) development views: the structure and relationships of classes, components, packages, and other kinds of modularity. The para 2 definition is only about the runtime deployment view (in 4+1: physical view).

The point is that you are at liberty in pretty much any operating system, runtime or language devised in the last 30 years, to structure your code and components as carefully and modularly as you like, whilst choosing your runtime deployment scenario independently of that modularity: it's okay for 2 uncoupled components to run on the same machine. Honestly. *nix does it all the time. Ooh, so does Windows. And .Net and Java and Android and iOS and ....

The CTO at intilery.com showed me a couple of years ago how their server codebase can be deployed as either a single .war for a single webserver or split as separate .wars for separate machines by flicking a switch in the build config.

It's not rocket science, it's Separation of Concerns: the codebase is not the runtime.

Estimates and NoEstimates

We had a debate&discussion at XP-Man on NoEstimates for which I did some notes. Reading the NoEstimates stuff, I was most attracted to the sense of "Let's not be satisfied with second rate" and of a thirst for continuous improvement.
I was left with the sense (possibly because I already believed it) that there are contexts in which NoEstimates works, and contexts in which it doesn't. But I was very glad to be provoked to ask in each case, "What value if any is our estimate/planning effort adding?" and "Isn't there a better way to deliver that value?"

What is an Estimate?

An estimate for a Project is (1) a list of things to work on; (2) a cost-range for those things; and (3) a list of risks, that is (3a) of Dependencies that 1&2 rely on, and (3b) of things that might cause significant change.

An estimate or plan for a sprint is (1) a list of things to work on, (2) a "cost" (eg story points) for those things and (3) a list of things we are uncertain about, or (4) need to get help with.

The value of a project estimate is to feed-in to (1) A go/no-go decision and (2) seeing things we want to see sooner rather than later (e.g. should we hire more people, do we need help from specific 3rd parties, is releasing in time for Christmas possible)

The value of a sprint estimate is, to see things we need to ask for in advance (ie external help or resources); to give everyone a sense of confidence about what we're doing; to fail-faster, that is to see sooner what we can't achieve.