Simplifying Software Development
It took me a while to work it out, I will be the first to admit that.
Looking back as short a time as 3 years, I still believed that the most important part of being a developer was knowing the language. Sure, the language changed over time, C, then C++ (I like so many of those around me prided myself on an insane amount of knowledge of the functioning of vtables, everything about the STL, a dozen ways to implement a linked list (and the most efficient, at least until a better way came along). I actually started to feel the threads of the complex mass that I was trying to hold in my head beginning to unravel when Java came along and gave me new focus (of interest, I tried helping a friend with C++ fairly recently, and while I could probably patch up or maintain an application in it, I found much of the more esoteric knowledge I had is already gone).
Java, and even more so python, another language I learned at around the same time, showed me that I could write code much quicker without needing to know vtables inside out, understand the rules of name mangling, andso forth. The problem was, I didn't really listen. I started to learn Java inside out. The language was relatively simple, but look at all those libraries and frameworks to learn! Gravy! Here was a challenge worthy of my efforts.
So, in essence, I swapped intricate knowledge of an overly complex language for intricate knowledge of an overly complex platform.
Now, personally I am more fan of than detractor from the J2EE platform (or Java Enterprise Edition as it is now called). I will be the first to say that many of the technologies, EJBs in particular, deal with a pretty scary problem domain filled with threading, multi-user, scaling and network issues. It is an attempt to get a handle on all of that, and a decent attempt too.
Outside of software development, I found myself similarly obsessed with complexity and control. For example, I had a home network setup with a Web server, my own mail server, print server, and specially configured Linux-based firewall, all of which I thought I needed because I wanted things just so, and all of which extracted an enormous toll in time and effort.
What Fixed Me?
Simple: So Much To Do, So Little Time
As time went on, results became more of a focus than knowing a language and platform inside out.
It sounds like heresy, but it is something that a lot of developers have lost sight of. Language facts and examples can be gleaned from a book, or google. There is no need to try and hold them all in your head anymore.
On top of my work, extra responsibilities started drawing on my time. It was all stuff I wanted to do, but it's amazing how being behind all the time will focus you on what really matters.
Meanwhile, every time there was a network glitch, or something changed in my DSL provider's configuration, I could end up spending an evening trying to fix the problem. Meanwhile we were usually not receiving e-mail, the Web sites were down, and the pressure was on.
I would like to say I had an epiphany—that would certainly be the cool thing to say—but the truth was that this constant grind slowly wore me down. It was a gradual realization, not a sudden one, although I can remember the final point that something clicked in my head (I promise this wordy explanation has a point and it is worth the journey):
Like many people, I had received an invitation to gmail, and had started an account. Also like many people, I had created the account, messed with it for a few hours, thought "that's cool," and promptly forgotten about it.
Then one evening, my DSL provider changed its policy on port 25 traffic and closed the port (SMTP for those in the know) and suddenly I could not get e-mail. I ranted down the phone at the DSL provider and asked why it was necessary, I tried to find ways around the blockage, and eventually I implemented a fetchmail cron job to go and grab my mail off of their server, which, as it turned out, was flaky at best.
Then, I started thinking about the reason I actually ran my own mail server. The main reason was that during the course of a day, I end up using as many as seven different computers, and in order to not have to deal with mail more than once on each, or worse—not having access to mail that I dealt with on another machine—I set up an IMAP server that I controlled, so that I had centralized e-mail. Then, I remembered gmail again. I thought, it will never do everything my IMAP server can do, and I was right. It doesn't, but here is the crucial point, it did what I needed.
So, I dusted off the gmail account, redirected my domain mail to it, and never looked back. As time went on, I discovered all manner of features in gmail that make life much easier—and this is also an important point—things that I did not know to look for in my e-mail previously.
I went on, happy at not having to maintain my IMAP server or postfix any longer, converted my wife over to gmail as well, and showed her the stuff it could do, which she loved. Then, I thought nothing more of it for a while.
However, as time went on, I suddenly realized how much time I was saving now, and I started to look at how I could apply the lessons I learned in other parts of my life, both professional and otherwise.
OK, I really don't want to be trite. I don't expect these lessons will work for everyone or that you even want to hear them, but this was my experience and I wanted to share it anyway:
- It saves a lot of time and effort if you can get someone else to do the work of tasks which, for want of a better term, are not "core" to your interests. For example, I had no interest in running a mail server at home; I merely wanted the benefits of doing so. That Google was offering to do the hard work for me, and for free even, made it a no brainer.
- 90% of the features for only doing 10% of the effort is a good trade. There were a couple of minor things I lost when I let my IMAP server go, but it was a good trade. On the other hand, there were things I did need from my e-mail, and I did not want to lose sight of those necessary features.
- If you are willing to take the plunge to try a new tool or service, you may find that it brings its own benefits over what you know already.
- Getting back to "core" interests, letting gmail provide my mail service gave me more time to concentrate on the things I really wanted to. I have no problem getting into deep complexity if the situation warrants it, but you only have so much time and can't spend it everywhere.
- The cost of switching (and there is always a cost) is often minor compared with the gains that result.
Applying the Lessons
So what does this mean to development? Well, not much, yet, but the relevance is coming.
I reasoned that if I could simplify my development world, use as much in the way of services and tools to cut out the work that I didn't need to do, I could concentrate on the tasks that I needed to solve because other people have not already.
For example, NewEnergy Associates, the company I work for, has some unique "crown jewels" like the formula service engine. This is like the Excel formula engine on steroids. It models the passage of time, acquisitions and mergers, and so forth. This is clearly custom work and value add in the industry—it is core to our business.
To display the data, we use a cell grid—much like an Excel spreadsheet. We also have to feed data and formulas into the grid, which is held in a database. Writing data display grids and persistence engines to solve these two is clearly not core. That much is clear and I doubt whether many companies would take it upon themselves to write these components when off-the-shelf ones can be bought (or open source ones used)?
Grabbing a few components to do the parts that are obviously plug-and-play is not the full story. I wonder how many of you have a policy of always looking for pre-written components before embarking on designing and coding your own. Beyond that, how many times have you had to write your own because of just a couple of things that the pre-written ones couldn't do?
Now we get into the meat. Could those pre-written bought (or open sourced) components be used if you accepted a few limitations? Perhaps the persistence library can't store one of your more esoteric data types. Can you get by with dropping that data type and making it use more common data storage types? Think about whether it is worth the tradeoff.
Likewise, can you make do without a couple of features from the display grid? If there is a pre-written display grid that is easy to use, but doesn't allow pasting a range of cells into it, can you get by without the paste functionality, or come up with an inferior but workable alternative, or are you going to sit down and code your own display grid instead?
Tools can be a similar source of inefficiency. Many companies have a "recommended" tool base, and this is generally a good thing because it is a set of tools that you can train everyone for and have a standards skillsets. The problems I see with this approach generally are:
- To minimize the number of tools selected and offer the broadest possible development options from the least number of tools, very flexible but complex tools tend to be selected. In other words, do-it-all tools are selected over simple, focused ones.
- The expense of selecting a fixed set of tools can be high, so once they are set they tend not to change for a while. Meanwhile, new versions or better tools are coming out that could greatly simplify and speed development, but are being ignored because they are not approved.
Taking things a little further, and this is a decision you will have to make, maybe it is better to select simpler, targeted tools that suit projects well and can be picked up quickly, rather than one Swiss Army knife of a tool that takes weeks to learn properly and even then slows development by being over-complicated. A decent developer will be able to pick up a simple new IDE very quickly in my experience. IDEs look similar at the most basic levels (project navigator, editor window, compile, debug, breakpoints, and so forth all work very similarly among IDEs). Using an IDE that suits the project and minimizes distractions immediately makes for simple and faster development. By dropping some features you don't really need and switching tools, you can make life much easier for yourself and possibly find some new must-have features that you didn't even know you were missing.
So What's to be Done?
Based on my experience, some of the longest to write, hardest to maintain and most complex code has resulted from those projects that were rushed. It is an irony that in the development world, more haste really does mean less speed.
When a project is rushed, the temptation is to get coding. There have been many diatribes about this subject and I am sure you are already familiar with the idea that good process will save you time and produce simpler and more manageable code.
Suffice it to say that I have personal experience that will back it up. Rushed projects that lacked the proper process have resulted in complex code bases with subsystems that were difficult to test in isolation, components that were written when they could have been bought or open-source equivalents used, and complexity that arose from trying to close the last 5% gap any way that would work, however ugly.
In general, I consider the following points to be important for simplifying development long before you get to the coding phase:
- Understand the problem domain fully
- Look for outlying requirements that appear to complicate matters or prevent bought or freely available components from being used, and see whether they can be dropped or modified
- Identify those parts of the task that are core to the value you or your company is trying to add, and try and ensure that those are the parts you will spend your time on
- See what third-party components can be obtained. (Remember to include the reputation of the source as a factor in these decisions. I will more readily use a released project from Apache than a small project from a group I have never heard of—unless that project looks to be of very high quality and has passed a stringent examination.)
- For large systems, understand the architecture and responsibilities of the subsystems and figure out the APIs needed between those systems. Then, you can test those APIs thoroughly, isolating any bugs rather than trying to track them down through an entire stack. (This point is critical—poor architecture always causes extra complexity in my experience.)