In my opinion, one of the most insidious form of technical debt is what I like to call future-coding. You might also call it over-engineering, second-system syndrome, etc. It’s code written in an overly elaborate and general way in order to, the future-coder reasons, pre-emptively handle use cases that we’ll “probably see in the future”.
I find it more treacherous than plain “stupid code” because it predominantly affects smart people. It’s mostly smart people with limited experience – once you see the downsides of future coding a few times, you tend to knock it off (although there are powerful confirmation biases at work) – but still, these are bright people you’d want to hire and keep around in you organization because they’ll do great things in the future.
Let me illustrate my point using a hypothetical example with two equally smart programmers, where one just has a little bit more experience than the other and is less prone to future-coding.
The Tale of Two Street Lights
Let’s say you’re making a game where a large portion of the time the player is navigating an urban environment. One day the artists ask for a feature to make it easier to place street lights in a level. They say: “Can we have a way to click a path through the level and have the game place lights at an even spacing along it?” This seems like an easy enough request so you start implementing it.
Here’s where our story forks into two divergent paths.
First option – the pragmatic coder
The pragmatic coder writes some quick UI in the editor where the artist clicks on points in the level that gets stored into an array.
On the runtime side he writes a dozen-line for-loop that simply walks the path segments at equal spacing and spawns a street light each time. Job done, the pragmatic coder moves on to the next task.
Second option – the future-coder
The future-coder writes the same UI as the pragmatic coder, but for the runtime he decides upon a different approach. After all, he’s read books about design patterns and large scale software design, and he knows about the importance of the Single Responsibility Principle, and keeping code factored for changes that might occur in the future.
Thus, he decides to abstract over the exact type of path used. Sure, the artists only asked for a simple path today, but in the future they might want a B-spline or some other kind of curve, so he pulls out the logic for the path into its own Path object that can step along the curve in some kind of iterator fashion. Of course, that’s not enough, in order to change the behavior of the curve he puts this Path object behind an interface and adds a factory to spawn the right kind of path based on some kind of configuration parameter. He also updates the UI to allow for future kinds of paths. Maybe he even implements a B-spline option just so that he can test this properly with more than one option.
Now, the code to spawn street lights first creates an abstract path from a factory, but how does he determine the spacing of the objects? The artists asked for simple equal spacing today, but in the future they might want something different, so he writes another abstract class that handles the spacing policy (with a factory to go with it).
Street lights in your game happens to be reasonably symmetric, so today just spawning them with their default orientation works fine, but in the future the artists might want to use this system to spawn objects where the orientation needs to depend on the curve, so our future-coder updates the path object to also supply tangents and bi-tangents, and writes an abstract class to handle the specific placement policy in order to take this new information into account.
Phew, that was a lot of work – he’s added a lot of new classes and glue code, and the code that actually uses it is a lot more opaque and hard to read than what the pragmatic programmer came up with. It will be worth it in the future, however, when this extra flexibility will pay off. The future-coder goes home late today, but feels good that he’s written an elegant system that will stand the test of time, rather than just a dumb one-off implementation.
Reality Kicks In!
Let’s go through a few scenarios of what might happen next.
Good news, no changes!
Turns out this simple system was all the artists ever needed. They never ask for any changes, and the code is shipped unchanged from the original implementation.
The pragmatic coder feels good that his code did exactly what was needed and never caused any issues or technical debt in the project.
The future-coder feels bad that he spent all this time on an extensible system and nobody ever made the most of the flexibility in his elegant design, but at least it was used at all and the artists liked it, so it’s not too bad.
The feature gets cut
A week later the artists decide that they don’t mind placing street lights manually after all, since it gives them extra control. The automatic street light placing feature goes unused, and is eventually deleted.
The pragmatic coder barely remembers the feature and couldn’t give a damn.
The future-coder feels bad that his elegant system was completely deleted. He grumbles to the other programmers about the awful artists and designers who always keep changing the requirements and wasting so much of his time asking for stuff that they end up not using.
Requirements change
The artists are happy with the feature, but they have one minor request: “Yes, I know we asked for equal spacing of the street lights, but actually we want to make sure that there’s always a street light on each corner of the path, and that the spacing within each path segment is slightly adjusted to ensure this”. To the artists this feels like a simple change so they don’t feel too bad about asking for it.
For the pragmatic coder this is a trivial change. He updates his dozen-liner to first compute the number of street lights required for each segment by dividing the length by the desired spacing and rounding it to the nearest integer, then computing an updated spacing by dividing the segment length by this count. This takes a few minutes of his time and he moves on to other things.
The future-coder starts thinking about how to accommodate this new request into his streetlight-placing framework. Shit-shit-shit, he thinks, a frustrated panic starts to spread in his mind. He never anticipated this change and can’t really see how to update his system to handle it. What does it even mean to place a light at “each corner” when the path is completely abstract? After all, if a future path is defined with a spline, the “corner” may not have a control point on it.
And even if you did have a way to define what a “corner” means, how do you feed that data to the object that governs the spacing policy, now that it’s completely decoupled from the path object? This will require some major re-architecting he thinks, and postpones the task until he can find a larger slot in his schedule. Later that night the future-coder sits in a bar with a programmer coworker, complaining about those damn artists that keep changing the specs on him, oblivious to the irony that it’s precisely his overly complicated attempt to be robust to spec-changes that is causing him problems in the first place.
Ah, a solution! He decides to define “corners” in terms of the curvature of the path, with a configurable threshold, and then feeds this data to the spacing policy object so it can use it to compute the appropriate spacing. The code gets kind of messy since the path and spacing classes are now getting more coupled, and there’s a lot of tunable parameters involved, but it does the job so he checks it in and goes home.
The next day the artists informs him that his solution doesn’t work at all. They want to use this as way to ensure exact placement of streetlights even on straight-line stretches of road, by placing dummy control points in the path. So this notion of using curvature to define street light positions just doesn’t work.
Let’s leave the future-coder to work out a solution to this self-inflicted problem. I have no idea what he could be doing to rescue his system at this point, but whatever he comes up with it’s unlikely to be pretty, leading to even more technical debt and complexity.
Requirements change – redemption for our future-coder!
In this scenario the requirements change in a different way. The artists now want to have the option of using either a straight path with sharp corners, or a smooth B-spline path.
The pragmatic coder adds a flag to the UI, and a switch in his runtime code that either calls the old placement code, or calls some new code that interprets the control points as a B-spline and steps through them using his math library’s B-spline code. The code is still easy to understand, almost certainly contains no bugs, and it didn’t take too long to update.
The future-coder feels redeemed! He may even have had B-spline option already implemented so he can simply point the artist at the appropriate UI option. But even if he has to implement the B-spline option from scratch it’s a simple matter of implementing a new Path object that iterates over a B-spline, and update all the relevant factories and configuration options. Yes it’s more code than what the pragmatic coder needed to write to make this change, and it’s still much harder to read and navigate, and it’s not at all as clear that it doesn’t contain bugs. But look at the UML-diagram! It’s so elegant, and follows the single responsibility principle, and dammit we’ve already seen that we’ve needed one additional option already, so in the future when the number of options grows to a dozen or so we’ll really see the benefit of this elegant framework!
The future-coder wins the lottery
This time the scenario is that the artists keep asking for more and more kinds of paths over time, with increasingly elaborate spacing options, and it becomes one of the most reused systems in the whole project.
After the a few different path options have been added, each having a few different spacing options, the pragmatic coder starts seeing the limits of his original design, and decides that a more abstract and generic design will have greater benefits than downsides. So he refactors his code so that the path is specified by an abstract path object, and pulls the spacing policy out into its own object too. At the end of the day he ends up with a system much like the future-coder, but doesn’t feel too bad because he spent almost no time on the original systems in the first place, and the simplicity of them meant he could get the initial system going quickly, and respond to change-requests faster than the future-coder could’ve.
The future-coder feels like a goddamn programming genius. He totally predicted this and it paid off! He is now more certain than ever that just spending some time up front predicting the future will pay off down the line.
He’s blind to the fact that for every time he happens to get it right, there are a hundred other instances where his failed predictions led to unnecessarily complicated code, time wasted debugging because you can’t tell that the code obviously has no deficiencies, and additional development effort because the extra scaffolding required by his general systems made it harder to address changes that weren’t anticipated.
It’s like he’s won at the roulette table in Vegas – he’s convinced that he’s just that good, that he’s found a system. He’ll keep spending his money at the casino, without ever really doing the math on whether or not his overall winnings is higher than his losses. Everyone says they’re up on average in Vegas right? It’s a mystery that Casinos still make money!
Conclusion
We probably all have a bit of the future-coder in us, but we must resist him! The bottom line is that the future-coder almost never wins. In reality, predicting the future is virtually impossible, so don’t even try. It hardly ever pays off. Best case, you’ve just wasted a lot of time, but in reality abstraction has cognitive overhead throughout the project when reading or debugging code, and can make it harder to deal with unanticipated changes (ironically), as well as cause performance issues. And even when it does pay off it doesn’t actually buy you all that much and is actually a net negative because it skews your intuition into thinking that future-coding is a good thing.
The human brain didn’t evolve to think in abstract terms. You will probably have noticed this when trying to understand a new mathematical concept – it’s much easier if you start with concrete examples rather than the fully general definitions and proofs! Simple code that just does one thing in the dumbest way possible takes very little time for someone to understand, whereas if you have to jump through abstract classes and factories (each named in suitably abstract ways) it can take an absurd amount of time to understand something that really isn’t all that complicated.
All these extra virtual calls act as barriers to inlining, in addition to the overhead of the virtual function calls themselves, which cause performance issues. Yes, I know you’ve been told that you shouldn’t worry about performance for code that isn’t in a hotspot, but the reality is that for many applications there aren’t any real hotspots anymore. It’s depressing to look at the profiler for gameplay code and see that your warmest “hotspot” takes 5% of your frame time, and has already been optimized to death. By that time it’s often too late – performance has been siphoned off by tens of thousands of small inefficiencies added over many years, all because people didn’t want to worry about minor performance hits for code that wasn’t on the mythical “hot-path”. Simple code tends to be cheaper to execute, most of the time, in addition to being better in almost every other way as well.
So am I saying that we should just hack things together in the quickest way possible, with no regard for future maintainability? No, of course not. I’m advocating doing things in the simplest way possible, not the quickest way possible. Often, the simplest way requires removing a bunch of cruft that has accumulated over time, deleting stateful flags, pulling logic out into clearly named functions etc. Sometimes, it even requires an abstract factory.
You should optimize for simplicity, readability and modifiability. If you do, you will also end up with code that’s more maintainable, easy to modify and optimize in the future. Take the example above where our heroes needed to add support for a B-spline. Let’s assume that there was no library for interpolating a B-spline already so that they had to add it. I’d expect both types of programmers to pull that code out into its own function. My motivation for doing this isn’t that I’m trying to predict the future, though! You pull it out into its own function because doing so makes the code easier to read today, by making sure the B-spline functionality is hidden behind a sensibly named function instead of cluttering up the “flow” of the main function with unnecessary details. The minor cognitive cost of the extra function call is vastly outweighed by the increased readability of hiding unimportant details. It’s not about predicting the future, it’s about increasing readability.
Sensibly named function calls without side effects almost always add more to readability then they cost, if they’re virtual less so, if they have side effects even less so, if they’re in a class even less so, and so on. It’s all a tradeoff, and the break-even point for any given abstraction will change depending on the specifics of what you’re doing.
Very few abstraction techniques have no merit at all, but the road to programmer hell is paved with best-practices, applied too early. You should postpone applying these abstraction techniques until the cognitive and other costs of using them are clearly outweighed by their benefits (e.g. when you have demonstrated reusability for a component). There are often genuine systems even in game engines where it’s perfectly reasonable to apply many of the abstraction techniques you’ve read about in books about design patterns and large scale software design etc., but the vast majority of code is better off if you keep things simple, and avoid trying to predict the future.
A Random Walk Through Geek-Space 
I love this article..
Thanks a lot..:-)
One point that you missed is that well-structured code is actually easier to read and understand than a pragmatic write-and-forget hack. This doesn’t mean that you always need an interface, a factory and an XML configuration file, but I’ve seen far too many examples where the approach like «add a flag to the UI, and a switch in runtime code» eventually led to unmaintainable hundreds-lines-long monster functions. Also, past some threshold of ugliness the broken windows effect kicks in.
So probably this pragmatic programmer should have created two subclasses instead of a switch. But no earlier, that’s true.
I’m not advocating hacks, I’m advocating simple code when complexity is unwarranted (the vast majority of the time). This includes being well-structured.
Creating two subclasses instead of a simple switch would count as over-engineering IMO. By doing so you’ve added three more classes where before there was none (base class, and two subclasses), and made things significantly harder to read (instead of a simple switch you’re seeing a lot of boiler plate constructing classes defined in different files, and then making virtual calls), for no real benefit. What’s the *value* you gain that you’re going to use pay for that extra cognitive overhead?
You’re paying a cost to readability, modifiability, and performance, and you’re not gaining anything.
Having classes does not introduce as much overhead as doubling method length. Experienced programmer can easily ignore the boilerplate, especially when using a good IDE.
The value of splitting code into self-contained single-purpose modules (e.g. classes) is that you can work on a module without thinking about how it’s used as long as you conform to the interface.
When you have code in a single module (e.g. a single method), on the other hand, you always have to think how your changes affects the code around it.
Probably your lights-placing method looks like this:
putLights(Point[] path) {
float interval = askUserForInterval();
List lightPoints = new SomeKindOfList(estimatedCount);
Mode mode = askUserForLineMode();
switch (mode) {
case STRAIGHT: { /* fill lightPoints, 10 lines */ }
case BEZIER: { /* fill lightPoints, 10 lines */ }
}
addLightsAt(lightPoints);
}
The programmer modifying this method must read it fully and understand what it does, and it does a lot. Now let’s say that we introduced a LightsPlacer interface with Straight and Bezier implementations. This is how the method looks now:
putLights(Point[] path) {
float interval = askUserForInterval();
addLightsAt(lightsPlacer.generateLightPoints(path, interval));
}
Would you argue that it’s not easier to understand now?
No, I would vehemently disagree that you have improved anything, and would argue that you have significantly harmed the code base (as much as you can in this tiny amount of code).
No, the original code would be:
putLights(Point[] path){
switch (mode) {
case STRAIGHT: { putLightsStraight(path); }
case BEZIER: { putLightsSmooth(path); }
}
}
Your example method doesn’t actually include the additional complexity that your approach causes. The added complexity is in whatever code you have to select the current “lightsPlacer”, as well as the definition of the base classes, and sub classes.
In your approach you’ve:
1) Tripled the amount of code just due to the boilerplate of wrapping trivial dozen-line functions into an abstract class hierarchy, thereby drastically reduced the signal:noise ratio.
2) Added three new classes that does nothing but wrap up a couple of dozen-liner methods, which clutters up your source tree and adds extra “navigational overhead” to read your code.
3) When reading your method the reader doesn’t actually learn anything about the code. He first needs jump to wherever “lightsPlacer” is set and by which condition, then read how the base class is defined and what subclasses there are. That’s a lot of extra work needed to fully understand the system, compared to the absolutely trivial original code.
4) You’ve added the cost of a virtual function call.
5) You’ve significantly hampered the compilers ability to inline the methods.
6) lightsPlacer is an abstract base class, which means it needs to be allocated (can’t store it ‘in place’, since it has no known size). Or possibly stored as static constants. Either way we now have the the complexity of needing to worry about object ownership, allocation and deallocation. This is a price you pay both when writing the code, and when reading and debugging it.
7) Holy crap are we now paying the cost of an allocation just to switch between two different dozen-liner methods?
8) Adding another mode now involves writing a brand new class instead of a single method, again reducing readability due to the extra boilerplate getting in the way of the actual meat.
So, quite honestly I can’t see that your approach has any benefits at all, and readability isn’t even a contender.
Firstly: you mention costs of virtual calls and memory allocation. If these are a problem, then you’re working in a rare domain where you need to select abstractions carefully. In most modern application they are negligible, and are a very small price to pay for advanced abstraction capabilities.
As I already mentioned above, boilerplate is not a problem. You learn to ignore it pretty fast, and good IDEs even tend to collapse unnecessary parts of it. Also, they let you navigate modules and contracts, so that you don’t have to worry about what is located in what file.
Full understanding of a system is not really possible in case of larger systems. This is why it’s important to separate code into small self-contained single-responsibility modules that interact through strictly enforced contracts.
By hiding specifics of light placement behind an interface, and mode selection behind method dispatch mechanism, I let the reader concentrate on how the input data for this action is obtained. If the reader is interested in the specifics of drawing, he can look into corresponding module and, again, not worry about how this implementation of LightPlacer is used.
In your snipped you overlooked the code preceding and following the switch. How the “mode” is obtained? Do methods putLightsStraight/Smooth call actual placing routine at the end of their code? How are you going to test them? How this code would be wired into the encompassing UI framework? How and where are you going to log at what points the lights were placed?
Even if you don’t care about performance, why not keep it simple and performant when there’s no reason not to?
If you don’t think boiler plate matters then I don’t think you and I will find common ground on anything relating to readability. Having to scan through reams of useless code, and jarringly jump from one unnecessary file to another, etc., just to gain the equivalent information of a 4 line method is the very definition of awful readability.
The fact that big systems can be hard to understand is no reason to make it harder in the small. Keep things simple and readable, don’t add complexity and barriers to understanding.
My code is still single-responsibility, it has two completely independent implementations, each is a simple method. Then it has a trivial top-level method to select between them based on a single enum value. Complicating that adds no value.
Look, when evaluating readability it helps to view your code as documentation. Take my method first, after reading it you will know 1) There are two different modes for placing lights 2) One places them in a “straight” path, 3) One places them in a “smooth” path, 4) It’s trivial where you need to look for more details (press F12 on one of the methods). Not bad for four lines of code.
In your method you don’t really learn anything – it basically just passes the buck to a virtual method call, and the selection of which method *actually* gets called is determined somewhere completely different. That’s action at a distance – bad for readability. So while my code answers questions, yours only raises them: What’s this lightPlacer thing? How many subclasses are there? Where is the current lightsPlacer object set? Who owns lightsPlacer? Who’s responsible for deallocating it? Why is this code so complicated? Where is my whiskey?
Why do you think that boilerplate is so harmful for readability? It’s a well-documented feature of our brains to stop noticing repetitive unimportant fluff after some exposure to it.
I’m all pro keeping code simple, and simple to me means that understanding it requires as little context as possible. I totally agree that if you need to jump between several code locations open to understand a piece of code, then it’s not well-written.
My second snippet was about choreographing input, calculation and output. Essentially, it reads as «given the path, ask the user about interval, calculate placement* and put lights. (*) for more info about calculation see $here$». Consequently, if the reader goes $there$ (LightPlacer hierarchy), he can totally and thoroughly forget about where the data comes from and how it’s used later. On the other hand, the reader can concentrate on the actual sequence, ignoring the calculation details.
Your code, on the other hand, reads as “first switch on mode, then do the rest described $here$”, and the reader is left unsure whether $here$ contains something important or not, whether to continue reading or not, because there potentially might be some important stuff (like output, for example). And in fact there is: those methods handle both calculation and output (but not input/switching). This doesn’t look like a single responsibility to me, and more like an arbitrarily selected fraction of “let user put lights along pre-selected path” responsibility.
Regarding raising questions: why do you care how many subclasses are there? Let’s say there’s one, or two, or ten — ok, what now? If you want to add a new curve — go see where this lightPlacer comes from and forget about this putLights method. On the other hand, if you are interested in modifying UI sequence, don’t worry about where this lightPlacer comes from.
Also, there’s not that many languages out there that do not use automatic memory management. These concerns about memory allocation, ownership and deallocation are their own specifics and are not really related to logical code arrangement that we are discussing.
I think boilerplate is harmful to readability because of experience and common sense. If you have to scan through reams of code that looks mostly the same before you get to the code that actually *does something* it’ll take you longer to understand it. I can’t believe you’re even making this argument, to be honest.
Look, I don’t see how a reasonable individual could think that introducing an abstract class hierarchy (and all the extra complexity it entails) just to choose between two different dozen-liner methods is anything other than utter madness. I feel like I’ve clearly demonstrated how my approach is an order of magnitude more readable than yours, and if you still disagree I don’t really think we’ll get any further, and I’ll just have to trust that other readers will see my point better.
I’m not disputing the value of abstraction. As I say in the post, abstractions have value, but they also have costs, and the costs often affect the exact same thing that their value does (readability, maintainability, performance). You have to find where the break-even point is, where the cost of the abstraction is outweighed by its benefit (an abstract class harms readability, but there are cases where not using one harms it even more). As an example where additional abstraction is warranted see the last scenario. If you had 4 different paths, and 4 different spacing policies, then a simple switch would require *sixteen* different methods containing mostly duplicated code to cover all the combinations. That would obviously be utter madness too, and at that point the added cognitive overhead and complexity of having two abstract class hierarchies that can be mixed-and-matched at runtime is clearly the lesser evil.
It seems to me that coming from a different background (I assume your primary language is C++? you mentioned memory management and virtual calls) you might be used to different kinds of artifacts in code.
For me creating or understanding an abstract class, or an interface, or a hierarchy of those, often supported by a DI container, doesn’t seem such a big deal: I’m doing that daily and I can navigate them with ease. That’s why your reservations about isolating concepts into classes look weird to me.
It’s not about not understanding abstract classes (they’re used pretty frequently in C++ as well). It’s about the fact that you, objectively, have to read, write and maintain *tons* more code, where logic is spread out in different places, and depends on “action at a distance”. Where you can’t “see the forest for the trees” because for every line of code that *does* something there’s three that are just in the way.
And all you really needed was a simple switch and two methods.
Take your reasoning at face value, and every function inside System.Math in C# should be its own class – it’s just silly, sometimes a function is the right level of abstraction.
This discussion is a good example of over engineered comments. You’re arguing about an example when you’re missing the point :p
This almost reads like satire. Saying that full understanding is unattainable, so you should retreat to your island, sounds like an excuse and a prequel to CYA.
Simpler the better. Shared programming is a weird art. Simpler makes it as unweird as it can manage to be.
I think what you’re actually describing is the difference between an application developer and a library designer. Neither approach will be optimal for the other situation. A library needs to be highly general to accommodate unforeseen use cases without patching the library itself. An application developer on the other hand has complete control over the code and needs no consideration for any “downstream” consumers.
Pretty much. The future coder is someone who is an applications’ developer but thinks he’s a library designer. Very few people need to design APIs, and it’s really frickin’ hard to do. The future coder writes all his code as if it’s a piece of code that will remain immutable for all eternity and therefore needs to be fully generic and abstract – but of course in reality he’ll probably abstract over the wrong things because it’s really frickin’ hard to anticipate exactly what you’ll need to modify in the future. And in the process of introducing this unnecessary generality and abstraction (that actually ends up being in the way most of the time), he’s also reduced simplicity and therefore readability.
First, I want to agree with the general principle that you should be pragmatic, and do the simplest thing you can to make something work as it needs to today. That said, this is utter nonsense:
> Very few people need to design APIs, and it’s really frickin’ hard to do.
Everybody needs to design APIs, and it’s not very hard to do. Please watch this. I know it says JavaScript, but virtually everything in it can be applied to any language:
There’s a reason people who make libraries for wide distribution, programming language libraries etc., spend hundreds of hours on API review. It *is* hard to get right. Every time you generalize something, you lose the flexibility that comes from specificity, and also add complexity to the API (check out a high level UI library in Haskell sometimes – they’re often wonderful and elegant and extremely general, but they’re also a significant accomplishment to understand).
Future-coding isn’t really beneficial to library design either. In fact I’d say it’s arguably worse.
Libraries have the additional penalty that when you fail to predict the future, you now have to maintain the failed attempt as well as the solution, or cause your customers grief (aka. time and money) to adapt their software to your new solution. Significant abstraction only makes this worse, as changes to class hierarchies can ripple through customer code. Eventually you end up maintaining multiple custom builds to provide specific upgrades without disrupting the stability of their software.
Simple, constrained APIs are far easier to maintain, leaving more time to spend on improvements to actual use-cases from customers.
Thanks for this write up. I enjoyed it a great deal.
Good read, but you didn’t consider something pretty important: often, you’re working with files. And unlike code, files can’t be easily updated for future versions. Then you don’t really have the choice: you must abstract everything from the beginning. Nobody wants to check for version 1 or version 2 or version 3 everywhere in their code.
Yes, I’m not against writing robust solutions when you know they’re needed (in fact see my big, and somewhat unreadable brain-dump on game data format for some thoughts on exactly this – making your files robust to code changes). The point is that you *actually* know what requirements you have, you’re not making guesses about how requirements may change in the future, trading of readability and flexibility in the present in the process.
Great Article!
In my experience the hard thing to learn is, knowing the right time when changing from just solving the problem at hand to abstract the problem the right way. I’ve seen both extremes: over-engineering by creating overly complex system hoping to solve everything there might come and never doing the step of moving from just coding stuff down to refactor into a more systematic approach. Both are not good.
Another aspect future coders always miss: make the common case fast the uncommon case work. This totally applies to coding and software: optimize your design for the common case. Systems don’t have to support everything. The uncommon case should somehow work but it may be involved/ugly because it is the uncommon case.
@leastfixedpoint: I totally disagree. fewer source code to read/debug/refactor == better. Glue code just distracts from solving problems and make your live more painful. It is like cancer: spreading through your code base and becoming worse over time. It increases the “surface” of your design/code leading to more coupling and complexity making it harder to understand/use/debug and refactor. Keeping it to a minimum is definitely a good idea.
btw: One thing I wonder is how to best test on this Future Coder / Pragmatic Coder Aspect in job interviews. I find it really hard without seeing source code…
> I find it really hard without seeing source code…
That’s why I always let candidates write code during interviews. On the whiteboard. I clearly explain to them that it’s not quiz with a “right” and “wrong” answer that will get them hired or not, but a tool to elicit a discussion about coding. It’s very interesting to see the wide variety of approaches to the same programming problem employed by different candidates.
Coding during job interviews doesn’t sound like a great idea to me.. to solve a problem elegantly you need some time and peace of mind – you usually have neither in a job interview. Especially when the coding task is big enough to discuss style afterwards.
Maybe a small coding task that’s handed in before the interview is better suited for that.
Apart from that, as Sebastian wrote “but still, these are bright people you’d want to hire and keep around in you organization because they’ll do great things in the future.” – so even if it turns out during a job interview that someone tends to over-engineer, it’s no reason not to hire him. Just make sure (e.g. through code reviews) that he adapts his style of code.
I find rules of thumb hard. It’s really kind of a global optimizer that has to run in your head all the time – selecting when the best bang for the buck requires more up-front design. You could call it “taste”, I suppose.
For stuff in the small, I guess a good rule of thumb would be: abstract only when it makes code more readable. A lot of text books and university courses seem to tell students to abstract *always* whenever there’s anything even remotely independent going on. Those kinds of hard rules are nonsense. An elegant UML-diagram is a non-goal. What matters is if it makes things actually better (i.e. more readable). I think that would get you most of the way there – there aren’t *that* many “systems” in game engines anyway (requiring the extra up front design and forethought), and I think we could all probably just list them ahead of time by now that maybe you don’t need any rules of thumb to identify them.
HULK say OOP bad KISS good
Nice post!, over-engeneering and premature optimization are things that we are attracted to but are (most of the time) more harmful than good.
I think the “future” / “pragmatic” division among programming styles is misleading. In designing a piece of software the goal is to create code that meets all required features with the most readable code in the fewest lines. The creation process involves first writing refined thought out pragmatic features (not quick hacks) and at every step checking for similarities / repetitions in other features and packaging shared code into larger classes. A programmer can then test the usefulness of an abstraction to see if the added complexity reduced the lines of pragmatic code or just added useless sophistication.
I’m not sure I understand why you think it’s misleading. The first sentence of your comment doesn’t seem to be supported by the rest of the comment (which I agree with).
You are creating a false dichotomy. For each new feature added a series of steps need to be followed: first refactor or remove any code the new feature will overlap, then write the new code pragmatically, finally abstract if needed. By defining simply a programmer into two categories you have missed the point of the process.
Do you become a mistaken “futurist” if you see the repeated code ahead of time and make said classes in the second step instead of waiting to find repeated patterns in the third? If not, then the “futurist” you mean in the article is someone who takes it upon themselves to create unrequested features which is more a team management problem and less a coding one.
Factoring out repeated functionality is just plain old coding. I don’t think you’ll find me arguing against that anywhere. What I’m arguing against is unnecessary generality. I.e. you need to do something specific, but you’re trying to be clever and see that this is really an instance of problem class X, so you solve it more generally instead.
The issue with that approach is, of course, that the initial problem may well be an instance of problem class X, but that doesn’t mean that the next version of the problem will fit cleanly into the X bucket. So by generalizing up front, you’ve actually added constraints on your implementation, making it much harder to update it when the actual problem changes. See the third scenario above for an example of how this happens.
So yes, if you “skip a step” by abstracting before you have reason to do so (i.e. evidence), you’re paying for something you’re not using (in readability, in flexibility). All those extra payments add up over time and is pure technical debt (unless you get lucky and guess right).
My claim is that the future coder very rarely wins the lottery, and that the example I gave is fairly representative of both the kinds of over-engineered crap you end up with when people try to be too clever, as well as the problems you run into (failure to accommodate unanticipated changes).
If the programmer truly did prevent repeated code then he should be able to have a concrete reason for the abstraction which should be testable at a common sense level with evidence in his reasoning (e.g did grouping this data together reduce lines of code and complexity?). If the lines are more then without abstraction he has failed with his ability to recognize patterns and abstract effectively.
In your example the programmer is not reducing any complexity or lines of code — there is one instance of the code and he is making more to do extra things that nobody really wants — he would not pass the common sense test.
The skill of the programmer can be measured in their ability to not waste time by writing as little code as required to meet feature requests. If they write lots of repeated code before seeing abstractions or they create bad abstractions without testing their reasoning they are wasting their time with their inexperience.
The notions that a pragmatic programmer is a hacker and a “future programmer” is an over-generalist are conflations. It’s possible to generalise badly while focussing on immediate requirements. It is possible write hacky code for future purposes. The main point here is: code what is required now, to the best of your ability. Leave the future to sort itself out.
Yes, but IME the motivation behind over-generalization is that “we’ll need it in the future”, and the main result of someone trying to predict the future tends to be over-generalization.
Thank you for this article. It does a good job demonstrating the perils of speculative generality.
What bothers me about the article, though, is the implication that abstraction is a way to trade simplicity or readability for generality. The two are not at odds, and well-chosen abstractions can do wonders to *improve* readability (as you seem to concede in the comments).
Consider Bob Martin’s definition of abstraction as “elimination of the irrelevant and amplification of the essential”. From this point of view, abstraction is all about precise and clear communication (i.e. readability) and has nothing to do with anticipation of the future.
The human brain has no problem with abstraction as a way to clarify meaning. You won’t think twice if you see a fruit salad on a restaurant’s menu, even though it’s a combination of two abstract concepts. You’d have more trouble figuring out what the deal is with “pineapple, strawberry, kiwi, banana, and orange cut into chunks and mixed together”, despite the added concreteness (and really because of it).
What’s relevant is of course a judgment call, but in your example I consider the relevant things to be: (1) placing (2) street lights (3) evenly (4) along a path. These are the things that well-abstracted code brings to forefront, while hiding everything else. We do need to iterate over path segments and interpolate between their end points, but that’s just a chore we have to take care of. Going into details like that would dilute the connection between our current (not anticipated) requirements and the code that fulfills them, so we abstract them away (e.g. into a Path class).
We do get some added flexibility (it’s easy to add new path types) and reusability (we can walk paths for other purposes without writing new code), but that happens as a side-effect of simplicity (as the opposite of entanglement [1]), precision, and clarity – not at their expense.
Having to jump through class after seemingly useless class to understand what a piece code does is not a sign of too much abstraction. If anything, it means that what’s essential has been buried in a mess of irrelevance, and is thus a sign of complete failure at abstraction.
[1] Rich Hickey – Simple Made Easy: http://www.infoq.com/presentations/Simple-Made-Easy
Abstraction does have a cost to readability, that doesn’t mean they’re at odds. I go over this a lot in the post. But here’s another way of thinking about it: If pulling something out into a function call costs 1 “readability point” (because you now have to make a “jump” to see what it does), then that’s often offset if you can save 5 “readability points” by hiding unnecessary detail behind a well-named abstraction. Other abstractions have far higher costs, so should only be used when the readability improvements they offer are correspondingly larger. The key thing to understand is that abstractions aren’t free (w.r.t. readability, and other things), and if you think you’re improving readability when abstracting something too early, you are mistaken. Too many people have been taught that abstractions have “zero cost” w.r.t. readability, so they apply them mechanically just because it’s possible.
Not all abstraction is about predicting the future (see above – sometimes it’s about readability), but when you take something simple and make it *general* then the motivation is usually that it’ll be needed in the future. Actually, the motivation is probably that it’s just a reflex because they’ve learnt that *any* independent functionality *must* be abstracted, and *anything* that could vary in the future *must* be abstracted, and so on. But when you press them on it, future-proofing tends to be the go-to-explanation.
“2) Added three new classes that does nothing but wrap up a couple of dozen-liner methods, which clutters up your source tree and adds extra “navigational overhead” to read your code.”
While I understand the cognitive overhead of adding new methods and classes (and often eschew those in favor of inline code), the converse is ending up with a lot of totally unrelated methods with completely separate functionality in a single class. Then, navigating the class becomes a worse pain than navigating a complex hierarchy.
As long as you keep boilerplate to a minimum, classes can increase organization by allowing you to compartmentalize behavior.
Of course, creating a class is a tempting opportunity for a lot of added complexity, from defining and exposing a public interface, to constructors, factories, state and so on. Many developers dive straight in and don’t think about it but I’d argue most of these factors can be controlled. For example, even a compromise of just turning these into static methods inside a non-instantiable utility class would increase code clarity and organization.
I agree with the core of your argument. All I’m saying is we shouldn’t treat classes as the bogeyman when the real enemy is complexity.
Pingback: Links & reads for 2013 Week 33 | Martin's Weekly Curations
Premature generalization is one of the easiest ways to add unnecessary complexity to your code, on that we agree. But generalization and abstraction aren’t the same thing. You’re conflating decomposing a procedure into a sequence of more abstract functions with building a system capable of handling use cases other than the one at hand. While the former may often work in service of the latter, it’s not the primary objective; enhancing maintainability should be the goal.
I grant you that maintainability isn’t always a desired virtue. When the project is uncertain or has a reasonably certain, short lifetime and is well-specified or at least well-understood, the short-term hacker you call a pragmatist probably has the right idea. In my experience, these projects are in the minority.
Let’s put it this way. I’ve often looked back and lamented that my methods did too many things; didn’t have good test coverage because they were too hard to test; that I speculatively generalized in the wrong directions. I don’t think I’ve ever looked back and lamented too many layers of abstraction.
As I say in the post, abstraction in service of readability (maintainability etc.) is fine. The key point though is that abstraction has readability “overhead”. If you use it when there’s not a lot of benefit, then the extra cognitive overhead of the abstraction itself dominates and you’re worse off.
Wonderful article. I think one difficult part (the most difficult part?) of being a senior programmer is knowing when to future-code. For example, if you’re designing a database, the simplest is to use incrementing integer IDs. But if you think, in the future (in italics), you might need to merge objects created on different machines, you might want to use UUIDs. It’s a bit harder now, but fixing this later is really hard on a live database. Freshmen never future-code. Sophomores always future-code. After that it gets more subtle.
Your analysis was fantastic. It reminds me of the phrase, “You think to much” I remember a town where I lived that needed a light at a street corner. There was so much debate, committees formed, analysis productions given and finally after five deaths at this particular corner it was built. It took three years for all the paper work. A sane person would have just built it and fixed the problem. But when we over think, we over compensate.
Love your picture! Looks like my husband, how did you get my husband’s picture, ha ha!
Both type of programemrs are plain stupid, they just need to evolve to a next level. You build the software conforming a spec. If the spec changes, apply the changes to your code. The artists are also dumb guys. One thing is what you call future programming and another is to have a well-organized code.
You know, the problem isn’t really future-coding. Coding for the future is fine, and a good idea. It’s that these are the kind of people who are using idiotic data to imagine what the future looks like. For your traffic light spacing example, you don’t code for things that your audience has ZERO chance of wanting. You need to use psychology — and honest psychology, not the “I’m better than everyone else because I read Atlas Shrugged” psychology that’s so popular in the techie world. Honest gut-level knowledge of human nature to imagine what features your users are likely to want. Then, you write sockets where that junk will get plugged in, but you don’t bother writing the junk yet.
Using the fact that you just read a book on some cool garbage that no one but a hardcore geek would even know exists as a way to divine what your users will want in times to come is like trying to judge whether or not a pair of dice will roll 5 or 3 next by checking the weather. Seriously, what you’ve described above is not future-coding. It’s using the excuse of “I’m trying to plan for the future!” to justify “I’m writing this cool bit of useless code to solve a problem that no one else cares about because I’m bored.”
Future-coding is very necessary. But you need to do something that most coders won’t do, can’t do, or are scared to do: socialize with other humans and get a feel for how they behave — note that I said other humans, not ones that consider it witty to answer a knock on the bathroom door with, “Compiling!”
How would you have written the code differently?
I don’t think the future-coder in my example made a terrible guess, he didn’t commit to any changes, he just tried to “add sockets” as you say, in order to slot in new code in the future. The problem is that even doing that is too much, too early. Just the fact that he chose to generalize something to be “robust to changes” meant that he lost the ability to be specific about that thing. The solution isn’t “get better at predicting”, the solution is to stop predicting. The value when you get it right is minimal, and the odds are overwhelmingly against you getting it right anyway. So you might as well take the hit of generalizing when you actually need it and then you never pay for something prematurely (or multiple times).
Well written Sebastian.. What I feel is most developers don’t end up maintaining their code and since most of us are lazy, we end up in the pragmatic lot..
I’ve worked with such an architect. His code was so overly complex and hard to maintain / manage, only he himself could ever change it. The future-coder often correlates with the you’re-not-smart-enough-to-understand-my-superior-design coder. Obviously, the whole system was full of bugs, because of all those not-smart-enough “average” developers (some of which highly skilled) made so many “obvious” mistakes, which “he” had to fix in late shifts…
A simple design could’ve helped the whole team get the job done much more quickly with better quality.
The future-coder breaches an important Xtreme Programming rule. Never Add Functionality Early:
http://www.extremeprogramming.org/rules/early.html
Great article Sylvan. Although you’re not the first one to point this out, I think you said it well.
In the comments for almost any post you do, there are people telling you that you shouldn’t have to worry about performance in the way you are. At least _most_ people shouldn’t have to worry. I think it’s very clear that you’re not joking around, and that performance in the little parts does matter to you. Hey, it even matter for me too and I’m just a simple desktop application developer.
When I read in the comments that future-coding is good — it’s just implemented wrong, I get scared. I totally agree with you that it is wrong and bad and sinful. I think the key here is that the sensible opposite to future-coding isn’t stacking a bunch of simple hacks until the code starts looking like the spaghetti incident. We add simple solutions until we have enough on our feet to make a good abstraction or generalization and then we refactor the code. This is lean, and agile, and just the intelligent thing to do.
Now, I know you don’t think TDD is applicable to your domain but TDD demands the simple solution. When adding the feature you would first write one or a few tests that will fail. Then you add code to make the tests green. If the future-coder adds the more complex code to make the tests green, he or she adds code that isn’t needed for the tests and therefore isn’t covered by tests. That code could and should simply be removed.
TL; DR; This is a fan boy comment and TDD agrees with you.
Yeah. If you’re writing games, or just application in general in a world where unnecessary cycles translates to shorter battery life, you really have to worry about both the performance of your inner-loop algorithm, as well as all the little systematic performance drains that infects everything. Often, the latter actually dominates because it’s harder to optimize after-the-fact.
I have no practice in any coding but I have read your article with great interest. While reading your post I have glimpsed at my inner mirror and discovered all three mindsets involved in your story- I was the artist and the both programers at the same time. That was worthy of pondering deeper to grasp the soul of a man…Thank you for the illuminating mental journey. I would greatly appreciate your visit to my blogs and the comments. Thank you once again. http://arthiker.wordpress.com/2013/08/19/freedom-to-choose-2/
Reblogged this on Printing Robots.
Thank you for illuminating me regarding advantages and disadvantages of this increasingly popular trend in programming. I code LESS css and javascript and not only once but many times I found myself optimizing modules for later use when it just wasn’t the case. The way you made your point with the two examples and a real project situation was so elegant. it made me reconsider the way I think about code from now on. @nedzen
Just writing this to say that I agree with every word you are saying, Sebastian. In fact, if it wasn’t so well-written, I would have thought I had written this post myself.
Good article although the author seems to be rather biased (possibly coming more from a C background as opposed to C++). While I agree that one should not jump to some huge complex system based on his assumption of future complexity, I do not agree that throwing everything into a switch statement is the best way to go either. The better solution here would have been for programmer B to start simple (KISS) only adding complexity when it was actually needed.
A feature such as the one described should not be thought of as a system from the get go but as features are added it should move more from the realm of function to system. Using the above scenario as an example there would almost certainly be a great deal of commonality between each new feature which could and SHOULD be abstracted or at least moved into separate functions. When this happens, the original feature is now becoming more of a system (wikipedia: “A system is a set of interacting or interdependent components forming an integrated whole…”). With this in mind it now makes more sense to move this code into it’s own file(s).
In short: “Remember kids, a design pattern is only a single tool in your toolbox. A hammer shouldn’t be used where a wrench is more suited”.
“The better solution here would have been for programmer B to start simple (KISS) only adding complexity when it was actually needed. ”
That’s exactly what the Pragmatic Coder did in the example.
“That’s exactly what the Pragmatic Coder did in the example.”
Sure but my point is that there needs to be balance between the two. If you immediately jump into a problem head first without any sort of planning there’s an equal chance you’ll end up throwing away your code later anyway or at least rewriting a good portion of it. If you take the time to look at the task at hand and think early on about possible portions in which could be abstracted out and coding with that in mind, you’ll A. still get the task done quickly with minimal effort and B. have a system which can be more easily extended and or built on top of later.
My point is that this is a fallacy. Don’t be afraid of throwing code away. Simple code is easier to modify (and less expensive to throw away). I understand that people prematurely abstract and generalize because they *think* it makes the code easier to extend, but my claim is that this isn’t true in practice. Keeping it simple makes it easier to extend, prematurely adding infrastructure based on a prediction on what you think you may want to extend in the future generally lands you in trouble. Paraphrasing someone on reddit: the most flexible code is the code that isn’t there – it imposes no constraints on future changes.
Reblogged this on My Blog and commented:
The Tale of Two Street Lights
I’m curious on your meaning of prediction- in this scenario, it sounds like you are putting forth the idea that both developers are coming in pretty fresh- they don’t understand how the tool set they are working on is used, so the pragmatist just writes to what he is told, and the futurist tries to guess at the future when he doesn’t understand how the level layout will go.
I do believe there is a good balance that you can maintain- there is definitely such a thing as over abstraction, and it can turn the code into an unreadable mess that can’t really be successfully maintained by someone outside of the system(lovely eye-bleeding has been had >_<). I do believe that good abstraction comes from a combination of understanding how the tools you are making should be used, as well as using it to enhance legibility and avoid repeated code.
Also, just because I tend to be a horrible person and like poking holes in scenarios- Usually there would be abstraction well before you get to this point anyways. If you are working on making a level designer, there would most likely be an abstraction of placement algorithms already, if not- abstracting it out correctly wouldn't be a bad idea simply because that code would almost certainly get used elsewhere.
Basically trying to predict inherently unpredictable things, like spec-changes (if they could be predicted, they would be in the spec!). Choosing to split some system into pieces A, B and C, because you assume ahead of time that this will be the correct split for future modifications, even though there’s usually plenty of other choices that are also reasonable. Plus, generalization comes at a cost of loss of flexibility (as my example shows) – once you have a general value instead of a specific one, you are much more limited in what you can do with it.
I don’t think I’ve ever felt more guilty. Thanks for the heads-up.
Superb article, finely defining why the KISS principle is so useful in programming 🙂
great post, love it
Please don’t confuse trying to predict the future with building something that’s easy to change / extend.
Finish that sentence and you’ll see why they’re the same: “…. building something that’s easy to change/extend in the future“. Planning for future changes/extension is future coding, unless you actually have data about what will be needed in the future (e.g. specs). My argument is that keeping things readable and simple now in practice ends up making it easier to change/extend for more possible futures, than any “active” attempt to make things general/extensible.
Please don’t confuse building something that’s easy to change with building something that is easy to extend.
Straight-forward code is easy to change. Code designed for extension is much harder.
Reblogged this on Mind's Eye .
Thank you!
Honestly, I couldn’t agree more with your article. Having been the future-coder a couple years ago. I feel like the moment I learned design patterns I became a future-coder, trying to follow the book to the nose, and abstract away everything. I’ve since realized that for most things, those end up being unnecessary abstractions that place the meat of the code too far away from the actual need. I now consider myself to be the pragmatic programmer that first solves the immediate problem, or implements the feature via the shortest possible path. Then identifies repetition and abstracts to suit.
So what do you make of highly abstract Haskell libraries like lenses? “It makes imperative code so simple!”, proponents say. But the price is that to understand what your “simple” code is doing you have to grok an incredibly complex machinery that’s underneath. I don’t know if the gains are worth it.
I love Haskell (in a certain light), but I do think there’s a lot of intellectual masturbation going on in that community too. I don’t think people should be stacking monads together to solve their problems. It’s cool that they come up with so much cool stuff there, but I do think that if you’re adding a monadic style feature you’re better off actually making it a first class citizen in the language itself (with its own syntax etc.). E.g. the way LINQ and async/await works in C#. Yeah that means you can’t come up with new monadic options as user, but the lower complexity (not to mention learning curve) is totally worth it for a mainstream language, IMO.
I’ve seen the future coder in action too many times and yes, it almost never pays off, this article is right on the money!