What Happened to Software Engineering?
Over the past few years there has been an evolutionary shift in the world of software development. Not very long ago, the dominant Software Development Life Cycle (SDLC) methodology was the Waterfall Method with very specific phases that separated the construction phase from phases like design and test. The software development industry, still very new, was striving to find a repeatable, predictable process for developing software.
The best model for this seemed to be the physical sciences, like civil engineering and architecture. Artifacts like detailed requirements, design documents, and technical specifications were written and signed off on long before a single line of code was developed, similar to the process used in construction of physical structures like bridges, buildings, roads, and dams.
To further align with the physical sciences, job titles like “Software Engineer” and “Solutions Architect” were adopted.
This style of project management has been very successful in construction. Yet a significant number of software projects were failing outright, and many more went significantly over budget and/or missed deadlines. This was due to several factors, but probably the most significant have been both the speed of change in software and hardware and the speed of change in business needs. These changes in the software industry would be similar to that of having brand new vehicles requiring a complete redesign of the roads they drive on about every 18 months.
When civil engineers are asked to build a bridge across a river to join two roads together, the engineers building the roads have fairly exact coordinates defining where the roads will come together at the river, and vehicles haven’t changed significantly over the years. The bridge engineers merely have to join the two roads together using tried and true construction techniques that have been employed thousands of times before.
In software systems, it’s not unusual for technology or changing business needs to significantly change the requirements during construction (after all of the requirements and design documents have been completed). To put it in the bridge building analogy, it is like having one of the roads moved six miles downstream after the bridge foundation was already in place.
To counter these issues Software Engineers developed many new techniques and practices designed to refine the construction phase through improvements in software quality, code reuse, and productivity. Some of these new practices include defining (and enforcing) code standards and naming conventions, encouraging the use of proven software design patterns, using tools like unit test frameworks and techniques like Test Driven Development (TDD), followed by Behavior Driven Development (BDD), continuous integration, and pair programming. These techniques were very effective in reducing defects and improving the construction phase, becoming commonly referred to as Software Engineering Best Practices.
While practices for refining the construction phase were evolving, there was also a lot of study into refining the additional phases of software development such as requirements definition, systems design, quality assurance, and testing. These included Scrum, Extreme Programming and Kanban (the adaptation of Lean Manufacturing) to name just a few.
This reflection resulted in the development of what are referred to now as the Agile Methodologies. In fact, a significant number of the practices to improve software construction like TDD and Continuous Integration were developed alongside the process improvements that would later come to be called Agile.
Today, agile methodologies are rapidly moving from the fringes to the mainstream, penetrating even the largest enterprise software development teams. The agile revolution has brought about a lot of change, and exposed many of the problems that developers faced in using the old waterfall approach.
Conferences, Open Spaces, and classes are filled with discussions on how to be better at agile, focusing on topics like how to best manage the backlogs, run retrospectives, plan sprints, and other process oriented topics -- all extremely important items that need to be well understood and implemented correctly as teams move to agile adoption.
What tends to be left behind in all of the agile excitement is the Engineering Practices that were developed to deliver higher quality software. Most of the agile methodologies focus mainly on the process management topics and don’t discuss construction techniques in their teachings (the main exception is Kent Beck and Extreme Programming). I believe this due to the agile methodologies assuming you are already doing the technical practices!
Unfortunately, in my career as a consultant and an agile coach, I have seen all too often that those engineering practices are being left behind. This could be due to one or more of the common mistaken assumptions surrounding agile, or the lack of emphasis from some of the major certification programs in agile. Or it might just be that the term Software Engineering has developed a stigma from the waterfall days.
We grow as an industry when we learn from those that came before us. Even process reinvention takes into account what is already known, incorporating what works and eliminating what doesn’t. By removing all of those Software Engineering Practices like Test/Behavior Driven Development, Continuous Integration, and Pair Programming we are forgetting what counts at the end of the day: developing high quality software.
All of the other processes that come with the agile methodologies are important as well in achieving our goal, but don’t throw the baby out with the bathwater. I go by many titles these days, including Speaker, Agile Coach, and Writer. At the heart of it all, I am a Software Engineer. And proud of it.