The problem of representation: meh architecture and system diagrams for software

Have a quick look at a random collection of system diagrams then come back to this page (search link).

It is truly horrible how many 'so so' or just plain bad architecture and system diagrams for software there are out there.

What do we use architecture diagrams for? Do they serve a purpose or are they just eye candy? I'll elaborate on the assumption that these diagrams serve a purpose.

Architecture diagrams in general are convincing aides, to explain, to generate and answer questions.

• How do I...?
• How do you interact with our system? 
• How does our stuff work with other stuff?
• How does the software map across hardware?
• What is going on...
• What are the important parts of our system?
• What parts need to go on this machine, on that machine, on certain devices etc?
• What other software stacks do we use, interact with, depend on?
• What needs to be backed up?
• etc...

We use these diagrams as sharing objects for answering questions.

• As a way of explaining the relationship between our systems and the customer's environment.
• As a tool for simply describing our stuff, which after all is digital, virtual, intangible.
• As a tool for helping to sell our products (which after all are digital, virtual, intangible).
• As a tool to help training, troubleshooting, designing, configuring etc.

However we really should acknowledge that depicting and describing complex digital technology architecture is a really really really really difficult problem! But that doesn't mean we have to continue to punch ourselves in the face with meh diagrams.

My issue is, I expect more. Perhaps it is just an issue of style. I'd like diagrams that tie in with the best policies for information mapping,using the css, themes, interface and interaction metaphors that are currently reconfiguring and improving user experiences on new digital media services and devices.

So what is possible? Who has done anything better? Have I actually ever seen a beautiful architecture diagram that ticks all the boxes? In answer to that question I am curating the following, representing for me at least, a collection of software architecture diagrams that come close to looking 'beautiful'.

Is this the real thing?

The NoUML narrative has been gathering popularity

Colour: white background, alternating colouration for elements. Font: any. Connectors: any. Shape and connector meanings socially determined. Composition: free-form.
See the Simon Brown's NoUML notes at InfoQ or sections from the Software Architecture for Developers book by Simon Brown.

But why limit ourselves to blocks?

From a discussion on Stackoverflow

Colour: thematic set, mixed, and complementary colours used for elements. Font: Sans serif. Connectors: none. Ambiguous message/relationship/connection. Composition: top-bottom linear, rectilinear.

Simple can be better. 

A "typical e-commerce system" architecture.

Colour: white background, black normal, red focus. Font: Sans serif. Connectors: solid, no overlap. Connector meaning is digital message/connection, zigzag denotes internet. Composition: balanced, central, radial.

A flat and simple diagram can be quite elegant and something a graphic designer can elaborate on. A single theme or style may also be employed as an in-house corporate style, tying in with the CSS for a website, application UI.

Maybe do more with less?

Google's use of white space on its home page has spawned an appreciation for negative space in general.

Negative space is the space around and between the subjects. Stop thinking about "one big diagram", instead think of each diagram as a use-case, use it to achieve one goal rather than 20. Perhaps a simplified index diagram that opens up detailed sub-diagrams. Designers using negative space won't clutter the screen. They leave space by intention!

Colour: blue/grey background, semi-transparent elements, white. Font: Sans serif. Connectors: mixed, railway track. Connector meaning is digital message/connection. Boundary lines dashed. Icons various Composition: asymmetrical, linear, left-right.

Employ a small palette sparingly, consistently.

The Windows technology ecosystem

Colour: thematic set, blue background, 3 colours. Colouration used to categorise. Font: Sans serif. Connectors: none. Connection implied by position, but ambiguous message/relationship/connection. Composition: grid style asymmetrical, top-bottom, linear, rectilinear.

Flat & clean is the hallmark of the current trend in interface representation and graphic design that could seen to have started with Bootstrap from Twitter, that has influenced Microsoft's Metro interface guidelines & chrome, and even so far as to informing Apple's anti-skewmorphic post-retina display 'flat' iOS7 interface.

But don't loose the point by overdoing the beautiful elegance.

Product design + supply chain element

Colour: thematic set, dark grey background, white and complementary colours used for elements. Font: Sans serif. Connectors: railway track style. Ambiguous message/relationship/connection. Composition: left-right linear, rectilinear.

And what happened to all that useful negative space!

A simple model of architectural constructs

Colour: grey/dark background, alternating colouration for elements. Font: Sans serif. Connectors: dashed, no overlap. Connector meaning is digital message/relationship/connection. Composition: balanced, central, radial.

Acquiring technical skills

On the topic of acquiring or refreshing technical skills I had mentioned codecademy.com

They deliver excellent self-paced community generated free technical skills training content.

Quite interesting too to appreciate how they have integrated the principles of Gamification into the service (badges, notifications, reputation etc) as a driver towards engagement, continuation and completion of tasks.

What Tech employers really want?

From Silicon Republic (link)

Experienced techies are in high demand...
A mix of financial software consultants...
Computer engineers, experienced and graduates...
Understanding the full range of a business...
A new venture demands understanding a problem...
Understanding customers...
Switchers from product management, sales and marketing...
Particularly interested in generic skills, to learn quickly, to fit in, attitudinal ...
Seeking people with a mixture of technology and business skills...
Technical skills, but also accountants and business...
Seeking people across the spectrum, sales operations, customer operations, supply chain management...
From commercial experience to engineering experience...
Data analysts, project managers, project roles...
'Smart people', who understand new ways of using technology...
Creative, innovative, flexible problem solvers...

Talk to customers

Sunil Kumar offers a nice quote on the risk taken with a design-in-the-dark strategy (interview in Silicon Republic)

"I’ve said this to loads of people: moving from an IT perspective to a business perspective can be achieved but definitely the easiest way to do it is talk to customers, because they say things you don’t expect, and it gets you thinking.
I’ve taken a technical decision in the past where I’ve thought something was a really great feature – coded it, and went out to try and sell it, and potential customers were not interested. All of a sudden you’ve made this technical decision based on nothing but your ability to code it and you get a zero return on investment."

The best designs disappear from mind

There is a view that good design should disappear from view, should not direct itself to mind, should be invisible. Good designs, designed things, tools and things that get used, should not impose themselves on their users. Or rather, our interaction with the designed object should be so natural and seamless that we do not need be present-aware that we are using the tool in attaining our goal. Use-in-work (and enjoyment-in-use?) with a well designed tool, product or service should, in a sense, occur beneath direct perception, acting as a natural extension of our physical and sensory engagement as we are engrossed in the activity or goal we are directed towards.

The article in Wired (link) reminded me of these design principles, and it introduced me to Dieter Rams' less but better (weniger, aber besser) and his other "ten principles of good design". It set me thinking about how these principles apply to software design...

Another topology of research methods for design

Another depiction of the spectrum of possible research methods, a way of classifying them, a way of understanding their application and use. (link to the Interaction Design Foundation article)

From the article on UX Research Communication – Report Writing

Cinematic views of software, high-tech, and philosophy

Expanding on a list of key documentary and cinematic views of working in software and high-tech industry (informed by Codemate's post).

Being in the World (2010)
A surprisingly beautiful presentation of the philosophy of human experience, how we apprehend ourselves, others, space, time, action, activity. The film's director and speakers attempt to convey a foundational understanding of our place in the world, our embodied existence. What is true about my being a person, how are we beings, how do I manifest, what it is to be or to exist in the world, how do we encounter and act in the world? Directed by Tao Ruspoli, interviews with the philosophers Hubert Dreyfus, Sean Kelly and many others.

Wittgenstein: A Wonderful Life (1989)

From VHS. A BBC 'Horizon' film about the remarkable life of the Austrian philosopher Ludwig Wittgenstein. Directed by Christopher Sykes.
Code: Debugging the Gender Gap
Documentary dealing with gender imbalance in the tech industry

Triumph of Nerds 1-3
by Robert X Cringely

Something Ventured
by Dan Geller

id software history
by machinima.com

Pirates of Silicon Valley
by Noah Wyle

Revolution OS
by Linus Torvalds

Coding Culture
by Gautam Sonti

Aardvark'd: 12 weeks with Geeks
by Lerone D. Wilson

Indie Game: The Movie
by Merge Games
After watching you will be inspired to play or re-play some of the most influential indie and commercial computer games of the last decades referenced in the film: ICO, Limbo, Zelda (all platforms, all versions), World of Goo, Portal, Slender, etc.

The Social Network
by David Fincher

Designing Interactions
by Bill Moogridge

The Pixar Story
by Leslie Iwerks

Connecting: a documentary about UX and ID
by Basset and Partners

The Design Makers - Inside Ford Design
by the Ford Design Center

Helvetica
by Gary Hustwit

Objectified
by Gary Hustwit

Crocodile in the Yangtze (2012)
by Porter Erisman

---

Fallacies of design and development projects

We have varying experiences, over time, of technology, systems, and products. There are commonly held beliefs and folklore about technologies, how they are developed, what they can and cannot do. When we use systems, make system, we build up pet theories and assumptions about the objects and others involved.

Software is magic

• The possibilities for software are unlimited (you can make anything you desire if you have the ambition)
• Your team members are Wizards; although they will be very smart, well educated, experienced, amazing wonderful people.
• Good design appears 'out of the blue'; it doesn't, it is the product of many cycles of both perspiration and inspiration. This fallacy sometimes pops up as a counter-argument to the principle of customer driven design, often by appealing to a design icon like the iPhone. The counter-counter-argument should highlight that the iPhone 'appeared' after more than 10 years of product revision and market evolution.

You can have all you can eat

• You can have anything and everything you want; 
• Designs are infinitely malleable; this is one of the assumptions behind the 'all you can eat' fallacy.
• Redesigns are cheap; this is one of the assumptions behind the 'all you can eat' fallacy.
• All requirements are equally important; this is a corollary to the 'all you can eat' fallacy.

People know what they want

• Requirements are the same as user goals; they aren't!
• Users know what they want; they don't, but they do know what they like when they see it and use it for a while.
• Designers know what users want; they don't, not always anyway, and of course designers may be users too, just be careful that the designer isn't always designing what he or she wants instead of what your market needs/wants etc; he/she is not a Wizard, just an incredibly diligent, sensitive, smart, well educated, experienced, amazing wonderful person.
• The project manager knows what everyone should be doing, is doing, has done (what, when and where); he/she is not a Wizard, just an incredibly diligent, sensitive, smart, well educated, experienced, amazing wonderful person.

People are stupid

• Customers/users/developers/managers etc "are stupid"; labelling groups is a symptom of other underlying problems.
• Someone else is at fault; results in finger pointing, when really the disagreement between expectations and actuality is more likely the result of different starting point assumptions (from Alex Cowan)
• I must be stupid because I can't use the software; any time you experience an 'I don't get it' moment, rephrase the feeling to 'the designer doesn't get me' (from Alex Cowan).

Software is free

• Software is free; Nothing is free free, free Apps have costs, even Free/Open Source Software costs.
• Your time is free; This is the working assumption behind 'death march' projects. Also the working assumption behind designs that impose unnecessary cognitive loads on end users.

One size fits all

• The needs of the many outweigh the needs of the few.
• The need of the one is the same as the need of the many.

Lord of the Rings Management

• Give me the one best solution; the solution could be product, service, system, method, management framework; the fallacy is that everything can be made simple when instead some things are intrinsically difficult or complex.

I know you know

• But it's in the documentation!; the perfect communication fallacy, 'the documentation' might be nearly anything, from a Market Requirements Document to a bug report.
• But I told you yesterday!; the perfect communication fallacy. Communication is rarely perfect i.e. what you think you said, what you actually said, what was heard, what was understood.
• RTFM; Assuming I took the time to read the ** manual. Assuming the information needed is in the manual. Assuming it is well written, comprehendible etc.

Failure is not an option

• Well actually, it is! In fact it is a very real possibility. That's why we have pivots and product readiness, project reviews
• Don't get sucked into following this piece of classic leadership claptrap.

Questioning: Collective Creativity

Help Seeking: Help Giving: Reflective Reframing: Reinforcing:
How to introduce such behaviours?
What to do when other behaviours detract from what you want to achieve?
When and when do problem recognition, problem solving, and design insight take place?
Who takes responsibility for work in such an environment?
Must organisations leave it to 'spontanaeity' if they want creative solutions?
Are prior social relations necessary before another would participate on someone else's problem solving?
What kinds of organisational features would allow HS:HG:RR:R occur?
What kinds of organisational features would restrict the occurrence of HS:HG:RR:R?
Can the knowledge and learning of another be fully captured in a report or presentation?
Is intrinsic motivation sufficient to sustain HS:HG:RR:R?
What reward structures support or undermine HS:HG:RR:R?
Minority dissent, high participation, performance pressure are all associated with increased innovation.
Social loafing, social anxiety, blocking, downward comparison are associated with decreased innovation.
Brainstorms and transitory teams are two organisational responses for supporting the behaviours.
Organisation's values, perceived and explicitly stated imply reinforcing features.
Knowledge management systems, documents and databases have a supporting role if used or useable.

Can Agile teams work to contract?

Contracts are often pointed to as one of the structures that appear to act as obstacles to the adoption of Agile process and practice adoption in teams. Yet at the same time 'contractual thinking' seems to be behind the use of User Stories, Product Backlogs, Unit Tests and Test Driven Development, all of which are key practices on SCRUM, XP and Agile teams.
Mike Cohn of Mountain Goat Software writes about these contractual aspects of Agile teams but also, how to write contracts that work well in these environments (link)

Definition for Digital Product Management

Product Owner - Product Development 
Outward facing responsibilities
You take responsibility for product line business development and reporting to the management team.
You help steer the development of the product through client feedback.
You provide Sales with the tools needed to sell the product (user guides, promotional materials etc.).
You prepare the demo kit for/with the sales team
Write and commission advertisements and article copy
You are involved in managing consumer communications via social media networks like Twitter / Facebook
You monitor and manage customer feedback / issues.
You will care about User Experience and User Interface design for consumer touch points – app, website etc, helping to develop, manage and maintain the company’s websites, market research, reporting revenue / costs, account management, product direction, PR, events, and marketing.
You will manage definition of the MVP (minimum viable product), identify pivots, engage the whole product development team, engage customers / users / players.

Inward facing responsibilities
You are involved in and manage product requirements elicitation and management, identifying feature goals (in-order-to's), and values of.
You manage the product backlog process.
You are involved in enhancing all aspects of the working environment.

Areas of concern for digital product management

I believe that the following areas are those that should be the concern of, and areas of action for, digital product management.
Spotting where problems are in software, in code, in tests, in systems, in processes and procedures.
Collaborating with others to propose or identify or create solutions.
Needing to have a mental picture and idea of the whole product environment in order to understand technical possibilities and limitations.
Being able to follow if not run the full release cycle, usually with other's involvement and help, to produce a workable copy of the software and operate it.
There are whole specialisms and domains of knowledge that you need to be aware of if not expert in, for example:

• Creating digital media including but not limited to; graphics, video, audio, fonts, text, formatted text, electronic books, styles, models, and many more.
• Programming in various computer languages. Knowing the distinction between interpretable and executable software, between scripts, byte code and machine code, the necessary extras (libraries) that allow the software to run.
• Wireframes, prototyping, paper prototypes, incremental and iterative development.
• Testing, as a of broad specialism that also includes programming.
• Source code control systems and document versions using github or subversion or cvs or other systems.
• Packaging, installation, and package management.
• Issues of legacy systems; in one way anything becomes a legacy system the moment it gets released. This area can be approached in different ways, data versus execution or running environments.
• Localisation (aka l10n) and Internationalisation (i18n) are really important areas, unless of course you want to ignore international customers/users. Localisation is highly specialised but increasingly easier to address through the enhanced support offered by software development environments. So no more need to 'hard code' translations or language rules, these are now provided for by standard development libraries. So to, it is now a commonplace for computer fonts and character sets to support extended and unicode character sets allowing the display and use of text in any language, type, and linguistic framework. For example, left to right, right to left, top bottom, sentence patterns used in european, middle east and asian languages.
• Online help systems including pop-up and context sensitive help, often the first and only documentation provided with software.
• The components of and services provided by operating systems.
• Managing vendors and other third party contributions.
• Data visualisation.
• Help desk and frontline support systems.
• Sales and customer relationship management.
• Quality management systems; not usually an area that software development prides itself on but essential and necessary nonetheless.
• General technological infrastructure like internet protocol, ip addresses, networks, network protocols, subnets, routers, switches, wired and wireless, firewalls, packets, packet sniffing, shared drives, cloud storage, routers, domain control, ftp, port management, virtual machines, password managers.
• Web hosting, website builders, domain registration and management, email server setup and settings, canonical name record (CNAME) and the Domain Name System (DNS) and IP addresses.
• Content management systems, workflow and web services (things like Wikis, Wordpress, Joomla, Drupal and many others).
• Databases, ranging over flat file, to relational, sql, object databases and post-relational databases.
• Web services, evolving capabilities of HTML, REST, and other important API services.
• LAMP architecture (i.e. Linux, Apache, MySQL and PHP) 
• MEAN architecture (i.e. MongoDB, Express, AngularJS, Node.js)