The Systems Development Problem
Information system development involves coordinating many people working on many tasks over extended periods of time. Partly because it is a relatively new profession and partly because of an insane sense of urgency behind almost every systems development effort, the development efforts often fail.
One reason for failure is a complete misunderstanding (or, more charitably, underestimation) of the complexities involved in a development effort. This is a crucial issue. We rarely appreciate how complex computer software really is. According to Jerry Weinberg, modern software systems are the most complex things ever built by human beings. (Weinberg also said that if carpenters built houses the way programmers write programs, the first woodpecker to come along would destroy civilization.)
Software projects weren't always so complex. A few decades ago we were writing programs to solve specific logistical problems such as payroll, receivables aging, order entry and so on. In the process control world the programs involved logic to open and close valves or logic to adjust a control surface or activate a servo of some kind. So that is what we did. We developed independent programs to solve what we saw as more or less independent problems. Information technology had a narrower purpose in those days, one that was easier to grasp.
Since then, the fundamental problems that we concern ourselves with have changed. As time passed, information technology began to assume responsibility for groups of programs, or application systems. Instead of payroll we started implementing human resources systems which included the same basic data but extended the scope to include other programs using overlapping data. Instead of order entry programs and receivables aging programs, we started dealing with systems that include both and others as part of a single application. We started developing process control applications that managed entire power trains in automobiles, processes in chemical plants and avionics subsystems in aircraft.
Now, we are beginning to work with systems of systems that integrate applications based on shared data requirements. We can no longer deal with order entry applications separate from inventory, warehouse control and production applications. We must not only incorporate these varied applications into integrated ones, we now have to go outside the immediate organization and include the entire supply chain. We treat an entire automobile as a network of process control nodes and include external navigation and traffic data besides. This extension of scope requires us to concern ourselves with issues not previously considered to be in the domain of information technology, namely the organization itself. You cannot seriously address corporate information requirements without paying careful attention to corporate policies that affect the data and everything else.
Each of these changes in scope has increased the complexity of the problem by at least one order of magnitude. Yet, in many cases, we are attempting to solve modern systems implementation problems as though nothing had changed. We still try to use the brute force techniques that more or less worked when we were faced with the very different issue of writing stand alone programs to solve simple logistical problems. Steve McConnell calls this approach 'code and fix development'. It used to work sometimes; now it works hardly ever.
Some organizations have recognized the problem and developed more rigorous approaches to dealing with the complexities of systems development. Typically these are called software development methods rather than systems development ones but they do represent an improvement over the code and fix approach.
A number of vendors developed packaged methodologies in response to the perceived need for control of the software development process. This started in the late seventies and it is still going on. The early methodologies consisted of large sets of manuals containing numerous detailed procedures, tasks and forms whose avowed purpose was to bring order to the process of software development. Some of the methodologies proposed that you pick and choose among the options available depending on the perceived complexity of a particular project. Some of the methodologies died, others evolved into computer aided software engineering (CASE) tools and some are still going pretty much as they were initially written.
The Datamorphosis Approach
© 2011 Datamorphosis Inc. - P.O. Box 11498 Bainbridge Island WA 98110 - (206) 842-1100 www.datamorph.com