Designing software (by alaric)

One thing that has often annoyed me in my career as a software designer is coming into conflict with people who want me to do the worst possible job that is "just good enough".

This often comes out in statements like:

• "There isn't time to design it properly, just hack something so we can start demoing it"
• "We'll have to rewrite it anyway, requirements always change"
• "Supporting more features than we need right now is a waste, add them when we need them"
• "Can't we just do something quick and simple?"

Reading between the lines, they seem to think that "more designing" will mean more complicated software with more features, that will take more time to build.

I think the problem comes from how product management thinks of software - they want a feature, they request the engineers add it (ideally specifying the feature well enough that they actually describe what they want), they get that feature. And there's often some discussion about reducing the scope of the feature by removing some criteria to get it implemented sooner. It seems very much like "more features with more acceptance criteria equals more work".

I'd like to dispell that myth.

I assert that better-designed software.will take less time to write, and will be better due to being more flexible rather than through having more features, and through being easier to extend in future when new requirements come up.

There's a paragraph in the Scheme language standard known as the "Prime Clingerism", and it reads thus:

Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary. Scheme demonstrates that a very small number of rules for forming expressions, with no restrictions on how they are composed, suffice to form a practical and efficient programming language that is flexible enough to support most of the major programming paradigms in use today.

It has been my experience that this approach applies to all forms of software design, not just programming languages. As a software architect, I see designing a feature as more like Michelangelo making the statue of David (famously, he said that the process of making a statue is just to get a big enough block of stone, and remove all the bits of it that aren't the statue).

Rather than thinking in terms of each acceptance criterion in the specification for the feature as a thing that will have to be done separately (so more ACs means more work), I like to work the other way: what's the simplest piece of software that can do all of these things? Most problems have a "core" that it can be boiled down to, plus some "details". I try to design software that has to meet specific requirements by making the "core" with the fewest assumptions and then adding "details" to tailor it to the exact requirements - which tends to mean I can add more details later to fulfill more requirements.

An example

For instance, many years ago, I worked at a CRM SaaS company. A big part of the app was sending out HTML emails to subsets of a user's customers to offer them some special offer, serving up the images in the email from our HTTP server (and tracking who saw the email by including the email message ID and the recipient ID in the image URL), and tracking what links they clicked by replacing URLs in the email body with URLs hosted by our HTTP server that would log the email message ID and recipient ID then redirect the user.

I was given a feature request: people were occasionally forwarding the emails to their friends, and when they did so, their viewing of the email images and their clicking of links would be logged against the original recipient, as their ID was in all the URLs. Our server would see the same person viewing the email lots of times and clicking the links lots of times. Learning how users responded to these messages was a big priority for our customers (we had extensive reporting systems to analyse this data!), so they were tetchy about this, and desired the ability to put "forward to a friend" forms in the email HTML. The recipient would put their friend's email address into this form and hit a button, and our system would send a fresh copy of the email to their friend, with their own recipient ID - so our user would know the forward had happened, and could see the friends' response separately.

Now, the "simplest" solution would be to add an extra feature to our HTTP server, accepting a form submission to a URL containing the recipient and message IDs, extracting an email address from the form submission in the usual manner, creating a new recipient in the DB with the email address (or finding an existing one), recording a forward from the original recipient ID to the "friend"'s ID, and sending a copy of the email to the "friend" - then responding to the form submission with either a hardcoded thankyou page or responding with custom thankyou-page HTML uploaded by our user, or a redirect to a URL set by the user.

A quick chat with the boss revealed that being able to host custom thankyou pages was a very desirable feature, and as that would involved embedded CSS and images (and maybe javascript), I clearly needed some kind of generic user-uploaded-content HTTP serving infrastructure. So I threw that together, letting customers created websites in the application and upload files, and an HTTP server that would server resources from them with a URL containing the website ID and the path of the file "within" that website - and with the ability to tag URLs with email message and recipient IDs for tracking purposes. We already had an engine to "personalise" HTML message content for a specific recipient, which also handled modifying all the URLs to embed email message and recipient IDs, so I used that again for hosting textual content such as HTML, meaning that if a link came in from an email message with message and recipient IDs, they would be fed into all further URLs returned by the system. To prevent spoofing of the IDs in the URLs, I reused the method we used on the image and redirect server URLs: the URL itself was "signed" with a short cryptographic signature when the software generated it.

But rather than hardcoding the forward-to-a-friend feature into that, I did something that took perhaps ten minutes more programming: I allowed an arbitrary number of named "commands" to be included in a URL. The only named command I implemented was "forward" that would do the forward-to-a-friend logic (finding all form submission fields called "forward-to" and using them as recipient email addresses, so you could make forms that accepted lots of addresses; making it loop over all fields with that name rather than just looking for one was the first assumption I removed to simplify the software while making it more flexible) then proceed with handling the HTTP request in the normal way.

But when a requirement came in to support unsubscribe forms, I just added an "unsubscribe" command, which took all of five minutes. Adding an "update" command handler to let people update their details in the master DB took a little longer, just because I had to list all the supported field names in the documentation and include a more lengthy example. And then, because I'd implemented the recipient and email message IDs in the URLs as optional things from the start, adding a "subscribe" command handler that created a new recipient in the DB from scratch, along with a tweak to the UI to let the user get an anonymous URL for any file in their website took an hour or so - and meant that users could now create subscription forms and generate a URL they could publish. I think I also added a "send" command handler to send a copy of a specific email message to the recipient ID in the tracking context; as the "subscribe" command put the new recipient's ID in the tracking context, a URL with a "subscribe" command followed by a "send-" command would handle a subscription and then send a welcome email to the new subscriber...

I added a few other command types that did various other things in the system, and all was good.

Now, I didn't "set out" to design an "overcomplicated super-flexible" HTTP server system that could do all these things when I was asked to add the forward-to-a-friend system. I just spotted that the following assumptions would be really easy to NOT bake into the core of the implementation:

• Forward to a friend is always to a single friend; it's easy to create multiple form fields with the same name in an HTML form, and easy to make my server loop over any it finds.
• Every visit to the new generic HTTP server will have a recipient ID and an email message ID; although forward-to-friends will as they will be linked from an email, it's easy to imagine that we might want to host subscription forms in future. So making the tracking IDs in the email optional (or allowing for other kinds of tracking IDs in future) by making the URL structure by a series of tagged fields (it looked something like http://domain/sss/Rxxx/Eyyy/Wzzz/foo.html for a request for file foo.html in website ID zzz for recipient xxx from email yyy, with sss being the anti-tamper signature) was worth the few minutes' work.
• Requests to the web server will be forward-to-friend requests or just static resources in support of a response page; it was easy (given we already had an ordered list of tagged components in the URL) to added an ordered list of arbitrary commands (with URL path components like forward, subscribe, etc; any component before the Wzzz/filename part was considered a command or tag, and tags started with an uppercase letter).

The "extra work" required to do things that way rather than the "simple" way was tiny. But it meant that the core of the HTTP server was simple to read and didn't need to change as we extended it with more and more commands (by adding them to a lookup table); it made a powerful system that was easy to extend, easy to understand, and capable of things we didn't have to "add as features" (some or our users did quite creative things by stacking commands!). As the HTTP server core and the commands were small separate modules with a clearly-defined interface, they could be understood individually and were well isolated from each others' implementation details, the benefits of which are well documented.

So please help me fight the assumption that putting thought into designing software means it'll be complicated, more effort to implement, and that "it'll need rewriting anyway so there's no point"! Let me do my job well 🙂

1 Comment

• 

  By John Cowan, Sun 29th Dec 2019 @ 6:14 pm

  I think the answer is that project managers want predictability above all: they are not willing to count on flashes of insight that may or may not come. Hence the focus on "estimating" "points", which really means guessing wildly and then being held to your guesses as if they had something behind them.

Other Links to this Post

RSS feed for comments on this post.

Leave a comment

*Name

*Email (not published)

Website