Tag Archives: software

Three Historical Definitions of the Open/Closed Principle and a Claim that it’s Pointless

Bertrand Meyer first published the OCP in his influential 1980’s book Object Oriented Software Construction:

  • “A module is said to be open if it is still available for extension. For example, adding new fields, or performing new functions.
  • A module will be said to be closed if it is available for use by other modules.”

It’s a neat double-definition, not least because the definition of Closed is both useful, and one that might not contradict the definition of Open. But Meyer’s proposed technique—use subclassing to achieve Openness of a Closed module—is widely ignored. Many of us have discovered the pain of working with inheritance hierarchies, so we savour the Gang of Four’s sage dictum: “prefer composition over inheritance.”

Dynamic languages like javascript can do the open/closed trick quite easily. The danger is that, in doing so, you develop inscrutable code and and are left with system that, when it works, you don’t know how it works; and so when it breaks you don’t know how to fix it.

Uncle Bob all-but-redefined the Open/Closed principle by using Interfaces as his technique. The interface is fixed and Closed; modules that depend on it can rely on it not changing. But the Implementation is Open: it can be changed without breaking the interface.

It is worth mentioning a word of wisdom from the .Net team’s Framework Design Guidelines: the weakness of interfaces in Java and .Net is precisely that they are 100% closed. There can be no version 2. Or rather, if there is a InterfaceV2 then it can usually have no useful relationship to InterfaceV1. You might as well call it ICompletelyUnrelatedInterface. (Or perhaps one could put the versioning at the namespace level).
This versioning problem is widely felt in service oriented systems with public interfaces. It is often addressed by creating a new endpoint for a new version of the service. Offering two versions of a service becomes on the whole precisely as expensive as offering two services, which is to say twice as expensive. This is unfortunate.

Contrast this with Meyer’s vision of OCP: On Meyer’s subclassing approach, version 1 clients and version 2 clients would call the same service and get the same responses. Version 2 clients would recognise, and so be able to use, the enhanced v2 capabilities; whereas version 1 clients would only recognise the version 1 capabilities. But here I see a second problem with Meyer’s vision: I’ve almost never seen systems (or even parts of systems) that can achieve it in practise. It’s a beautiful dream. But unachievable. It is a pipedream.

More recently (Dec 2016), Michael Feathers has offered an updated version, towards the bottom of the the page at Towards a galvanizing definition of technical debt:
“our code is better to the degree that we don’t have to change it much when we add features. We should be able to make modifications primarily by adding new classes and functions rather than changing existing ones”
This is much ‘softer’ formulation than Meyer’s or Bob Martin’s and you could take it as just a rule of thumb; something to weigh in the balance against other factors. Feathers implementation in this case (and I’m left with the impression that in a different codebase he’d be happy with a different implementation) is doing event driven code as most people think it should be done: use an AddEventListener() interface, which makes the code Open to all kinds of extension.
This AddEventListener() is exactly the approach used in the HTML spec and other GUI frameworks of the past 20 years. The downside is that the ‘closed’ bit of the interface is so small and weakly-typed that it’s almost non-existent. The interface tells you nothing about the semantics. (What kind of events can I listen to? What information do I get about each event? What can I do with them? I can only find out by reading the HTML spec, which turns out to be quite hard going, or turning to MDN, or, the first port of call for many, StackOverflow. In a bespoke codebase replace this with “ask for documentation; find it is incomplete; and then hunt through the code for examples of how I can use it”).
Strongly typed interfaces are at least somewhat self-documenting—they offer a definitive list of all syntactically valid calls to the service—even if that documentation depends heavily on how well the developers chose their method and parameter names.

These three examples leave me with mixed feelings. OCP seems like trying to square the circle, and Meyer’s choice of name was a well-chosen contradiction. Yet the goals—Openness for extension, Closedness for reliability—are unavoidable.

Dan North, amongst others, has suggest that OCP, and indeed all the SOLID principles, are of limited value and we should drop them in favour of something else. I sympathise—I think that SOLID is a mishmash of mixed value—but I’m willing to wrestle for a couple more years with OCP before I admit defeat.

I’d rather have the above three technique, and others, in my toolkit because my software design still has to address the two contradictory requirements that Meyer identified in the 80s:
–Because my software is still evolving, it has to be open for evolution: it has to change.
–Because my software is already is use, and hence being depended on by some other software or person, it has to be reliable and therefore can’t change.

Conway’s Law & Distributed Working. Some Comments & Experience

The eye-opener in my personal experience of Conway’s law was this:

A company with an IT department on the 1st floor, and a marketing department on the 2nd floor, where the web servers were managed by the marketing department (really), and the back end by the IT department.

I was a developer in the marketing department. I could discuss and change web tier code in minutes. To get a change made to the back end would take me days of negotiation, explanation and release co-ordination.

Guess where I put most of my code?

Inevitably the architecture of the system became Webtier vs Backend. And inevitably, I put code on the webserver which, had we been organised differently, I would have put in a different place.

This is Conway’s law: That the communication structure – the low cost of working within my department vs the much higher cost of working across a department boundary – constrained my arrangement of code, and hence the structure of the system. The team “just downstairs” was just too far. That distance was composed of gaps & differences in priorities, release schedules, code ownership, and personal acquaintance.

Conway’s Law vs Distributed Working

Mark Seemann has recently argued that successful, globally distributed, OSS projects demonstrate that co-location isn’t all it’s claimed to be. Which set me thinking about communication in OSS projects.

In my example above, I had no ownership (for instance, no commit rights) to back end code and I didn’t know, and hence didn’t communicate with, the people who did. The tools of OSS—a shared visible repository, the ability to ‘see’ who is working on what, public visibility of discussion threads, being able to get in touch, to to raise pull requests—all serve to reduce the cost of communication.

In other words, the technology helps to re-create, at a distance, the benefits enjoyed by co-located workers.

When thinking of communication & co-location, I naturally think of talking. But @ploeh‘s comments have prodded me into thinking that code ownership is just as big a deal as talking. It’s just something that we take for granted in a co-located team. I mean, if your co-located team didn’t have access to each other’s code, what would be the point of co-locating?

Another big deal with co-location is “tacit” knowledge, facilitated by, as Alistair Cockburn put it, osmotic communication. When two of my colleagues discuss something, I can overhear it and be aware of what’s going on without having to be explicitly invited. What’s more, I can quickly filter out what isn’t relevant to me, or I can spontaneously join conversations & decisions that do concern me. Without even trying, everyone is involved when they need to be in a way that someone working in a separate room–even one that’s right next door–can’t achieve.

But a distributed project can achieve this too. By forcing most communication through shared public channels—mailing lists, chatrooms, pull request conversations—a distributed team can achieve better osmotic communication than a team which has two adjacent rooms in a building.

The cost, I guess, is that typing & reading is more expensive (in time) than talking & listening. Then again, the time-cost of talking can be quite high too (though not nearly as a high as the cost of failing to communicate).

I still suspect that twenty people in a room can work faster than twenty people across the globe. But the communication pathways of a distributed team can be less constrained than those same people in one building but separated even by a flimsy partition wall.

References

The Yes, The No and the Painful: using, and failing to use, estimates for a no-go decision

@AgileKateOneal recently asked for examples of effective estimate use in medium/long-term planning, and making a no-go decision sprang instantly to mind.

Many such decisions are casual and quickly forgotten: the back of an envelope calculation which says that an idea is well beyond what we can afford; and the conversation moves on. An NoEstimator might object that one could profitably try out something rather than nothing, such is sometimes true but creative thinkers in commerce & IT can always generate a hundred more ideas than a team can try out; you can’t try out everything.

But a couple of more specific examples from recent work:

Example 1: Small UK charity looking at CRM options

in November last year I worked with a small UK charity, www.redinternational.org who were badly in need of some kind of CRM software to keep in touch with supporters and project partners. They were running largely on spreadsheets built from downloaded reports from virginmoneygiving.com / mydonate.bt.com etc. They also had an Access database with a fair amount of donor & similar data in it.

Question: Is it better to pay for a CRM solution — typical charity starting price £10,000 going up to easily £100k – or get someone to do enough work on the Access database to make it a usable solution?
My Answer: I first spent some time discovering and documenting their main use-cases (to clarify: their ‘business’ use cases, that is the things the charity had to do whether manually or with IT). I gave that picture to the CRM providers so that they could give us a sensible proposal. And I worked out an estimate for extending/developing the Access database. Based on that, we could see that a CRM consultancy/solution looked like £10-£20k (5 year cost) and the DIY-option about 200-400 developer days.

Even with this level of accuracy it was good enough to see that DIY should be a no-go. I did not expect this. I thought that the charity’s actual requirements were sufficiently small that we could do something useful for a few thousand pounds. But two hours spend going through their use-cases on my estimating spreadsheet showed me that I was wrong. So, I recommended the best value CRM option.

This, I think, is planning 101: a couple of hours working through the detail on paper is a lot cheaper than running the experiment; but can be enough to make a probably-good decision.

Example 2: Provide a system to automate a small team’s manual processes for a capped price

This was for a financial services company in 2013. The team were working on PPI claims for an insolvency practitioner (obliged to pursue potential claims that might bring in some money for their clients’ accounts) and had about ten thousand potential claims with hundreds per month being added. They had been working manually on spreadsheets for over a year.

I spent 4 days on analysis and listed a set of use-cases that covered the processes end-to-end; and I estimated that a suitable system could be done for about 40 days development work. The estimate cost about 3 or 4 hours on top of the analysis.

The contract to provide was capped-price. The customer was not open to a no-estimates approach. And I accepted being bargained down to below my estimate (Doh! I hear you say. Quite so). The actual cost came out close to (but above) my original estimate, but could have used another week’s work to make it user-friendlier.

The better course would have been to use the estimate/budget mismatch to declare a no-go rather than accept a reduced budget. This might have resulted in the client agreeing to go ahead anyway (which might in turn have led to a no-estimates approach to the work). Or it might have led to no contract. Either way would have been less painful and more controlled than over-running the budget.

The Known Unknowns Matrix

I.T. is not the only industry to have happily latched onto the the former Secretary of State’s famous phrase, “the unknown unknowns”. It’s a useful phrase to ponder if you’re responsible for planning or estimating anything because planning & estimating always involve risk. A recent slideshare by Danni Mannes on Agile Architecture pointed out to me that one should really consider the full matrix:

Known Not Known
Knowns Things we know, and we know we know them Things we know but don’t realise we know them; tacit knowledge that we take for granted. Become a problem if we are responsible, and fail, to communicate them to people who don’t know. Also a problem when we start work in a new context and don’t realise that what we ‘know’ is no longer valid, so they become unknown unknowns.
Unknowns Things we know that we don’t know. We can record the risk, and estimate a cost for investigation & discovery Things we don’t know that we don’t know. This is the quadrant most likely to shipwreck plans.

My personal takeaway from this is that I will try using this quadrant when listing risks. Just having a space for the possibility of unknown knowns & unknowns can be an impetus to do a little risk-storming & consultation, to help you discover the as-yet-unknowns.

P.S.

I’ve just read the brief and brilliant mcfunley.com/choose-boring-technology

Coplien & Bjørnvig : Lean Architecture For Agile Developers. A Review

Four years after this book came out, Agile Architecture has at last become a Thing. But as the nuance of its title hints, this book is not fad-driven. It is a carefully-thought out exposition of what architects can learn from lean and agile ideas, and what they can do better as a result.

Well. It’s partly that. If you are a practising architect, it is actually four must-read books in one.

If you are not, you might dismiss this book for two reasons. The first, that judging by other reviews & my own experience, the homespun style of the first half does not suit a bullet-pointed gimme-the-headlines-now generation. The second is that if you have not experienced the pitfalls of architecting and doing software in a real organisation with actual people in it, then Coplien & Bjørnvig’s pearls of wisdom may impress you much as the agile manifesto might impress a cattle rancher.

It is not a beginner’s book. It is the mature distillation of the sweat-soaked notebooks of a fellow-traveller who has stumbled over rocky terrain, been through the tarpits and has some hard-bloody-earned (and, academically researched) wisdom to share as a result.

I said four must-reads in one. They are:

1) The (literally) decades of experience of a leading practitioner & thinker in the field.

2) A thought-through answer to the questions, what can we learn from lean & agile. Whilst value-chains and some technique feature, the authors’ secret conviction is surely that Technology Is All About Human Beings. “Everybody, all together, from early on” is their Lean Secret. “Deferring interaction with stakeholders [users, the business, customers, domain experts, developers], or deferring decisions beyond the responsible moment slows progress, raises cost and increases frustration. A team acts like a team from the start.”

3) Which leads to that which for me, as a more techy-focused reader, was the marvel of the book. Clements et al in “Software Architecture in Practise” offered attribute-driven design, a partitioning of the system based on a priority ordering of, primarily, technical quality requirements. Coplien & Bjørnvig all but deduce a partitioning based on the priority ordering of people: Users first, Development team next.

“The end-users’ mental model” is the refrain on which they start early and never stop hammering. From this they suggest that the first partitioning is What the System Is from What the System Does. I am tempted to paraphrase as, Domain Model from Use Cases. But their point is partly that this also neatly matches the primary partition by rate of change, because the last thing to change in a business is, the business of the business. If you’re in retail, you might change what and how and where you sell, but your business is still Selling Stuff. And at lower levels inside the organisation too: an accounting department, though there has been five thousand years of technology change, still does accounting. The users know this. They understand their domain and if the fundamental form of your software system matches the end user mental model then it can survive–nay, enable!–change and stay fundamentally fit for purpose.

Second, don’t fight Conway’s law. “Organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”. The deduction for agile teams is, partition so as to maximise the long-term autonomy of your self-organising teams. Even when the team divisions are imposed for non-technical reasons–geography, politics, whatever–still allow that fact to trump more ‘technical’ considerations. This may not match your vision of technical perfection, but it will still be the best way. Ruth Malan recently paraphrased Conway’s law as, “if the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins”. Don’t be a loser.

These points seem to me obvious in hindsight, yet they turn traditional approaches to high-level designs on their head. After considering People, yes we can considering rates of change, quality attributes, technology areas. But it’s always People First.

4) And finally. This is the first book-sized exposition of DCI architecture, which I would describe as an architectural pattern for systems that have users. Having separated What the System Is from What the System Does, DCI provides the design pattern for how What the System Does (the Use Cases) marshals the elements of the What the System Is. There are at least three notable outcomes.

Firstly that use cases are mapped closely to specific pieces of code; in the best case each use case can be encapsulated in its own component.

Secondly, that the relationship between Domain elements and Use Cases are expressed as “In Use Case X Domain element Y plays the Role of Z”. This brings significant clarity to both, and is part of the key to ‘componentising’ use cases; the Roles needed for a use case become, in the code, its public dependencies. In UML-speak, the Required Interfaces for such a UseCaseComponent are the roles needed to ‘play’ the use case, and those roles are Interfaces which are Implemented by Classes in the Domain Model. Coupling is reduced, cohesion is gained, clarity abounds.

Thirdly, the simpleness of the mapping from business architecture to code is greatly increased. Suddenly one can draw simple straight lines between corresponding elements of architecture and code.

The authors say of their work, “This book is about a Lean approach to domain architecture that lays a foundation for agile software change”. To my mind, this hits the agile architecture nail on the head. Agile software development always only ever succeeded at scale because the people doing it either knew, or had given to them, enough architecture to make it work. Software Agility, just like every other software Ility, must either be supported by the architecture or it ain’t gonna happen.

But the best thing I got from this book was the proof, before my very eyes, that correct technical design flows from knowing how to put the human beings central.

Where to Buy

UK



ebay (UK): Lean Architecture for Agile Software Development


Amazon (UK): Lean Architecture for Agile Software Development

USA



Amazon (USA): Lean Architecture for Agile Software Development