Is Your Software Project Killing You?
The Horse That Ran Down A Staircase
I am often brought in to review huge enterprise projects that have fallen completely off track. Not only are these programs over budget, but they don’t work as originally conceived. Users have grown frustrated and unproductive. Management wants to know: what went wrong?
Imagine building a skyscraper with a $1 billion (US) budget. If you make a mistake late in the construction, what would it cost to repair? Or put another way, if you had hired an architect for that project, how much would you sue them for? And would that really help? Your client would still fire you, and your reputation would be lost. You would spend years trying to achieve remedies in court – both as plaintiff (against the architect) and as defendant (against your former loyal clients). In many ways, the cost would be incalculable. This is why so many companies fail after they make major changes to their operations, including where or how they conduct their business, and certainly including major shifts in what they sell. This includes software development projects, which are some of the most complex tasks ever undertaken by a company.
The tough answer to my clients’ question is that they are like a horse down a staircase — they have committed so completely to what they are doing that they cannot back out (imagine the horse trying that) or change their neck-breaking downward trajectory.
The most common causes of a failed software project are:
- The client failed to start with an independent program architect, or hired the wrong individual for that key role.
- The client couldn’t afford one. But could they afford to completely and utterly fail in their development efforts? The answer is always, “Well, now that you mention it…” A great architect pays for him/herself in direct savings that result from proper planning, design and management.
- The client couldn’t find an architect. It is hard to reach out and locate expert help when you need it. The architect has to possess complete knowledge of an extremely complex arena that changes each day. They also must learn the company’s business so they can tailor a solution to the firm’s needs.
- The client didn’t want to deal with an “outsider”. When you open the doors each morning, and watch the familiar faces pour into your building, it’s hard to imagine that any person who is not a member of your culture could assist you. But this is the actual benefit: the architect gets to see you for the first time, without bias. They see with clear eyes. Most businesses can leverage this valuable perspective when they are committing major funding to an arena as challenging as software development.
- The client never interviewed (or qualified) their own staff based on the actual requirements of the software project.
No major human achievement has ever been successfully accomplished without an architect. But virtually every disaster has been blithely uncontaminated by one.
“We’ve got lots of smart guys in here” is the usual explanation. So when the company decides to move from its ancient web pages to an ultra-modern client-server project, who else could be more suited? Answer: anyone with direct experience in the new technology, who understands programming philosophy, and who obeys fundamentals. When was the last time you evaluated your staff on that basis? “We’ve been too busy for any of that touchy-feely stuff”…
You owe it to yourself to understand exactly who you have hired and what they are capable of. This not only affects their current assignments. It also indicates whether they can move on to ever more complex and conceptually challenging projects. In many companies, the people working there are the “leftovers” — individuals who lack the energy to advance their knowledge, but simply wish to pay their bills and keep their jobs. Business owners should not assume that those workers will somehow “rise to the occasion”. Have they exceeded expectations previously? Have you had their work reviewed by an architect? Have they show ambition and drive, or have they generally just agreed with whatever you suggested? Every business needs “a little disobedience” from its brilliant and somewhat untamed staff. If you’re not getting it, maybe your employees are asleep.
Modern bureaucracies are based on alliances rather than capabilities. Those who speak up, or exceed the mundane performance of their co-workers, are quickly isolated and expunged, or get frustrated and move on to better opportunities. Unless you have taken steps to protect high-achievers, your company has probably already devolved down into this modern-day version of Lord of the Flies. The problem with this environment is that it forms a classic downward spiral: over time, your staff realizes that the only way for them to survive is to protect each other. So they work at the level of the dumbest member, and at the pace of the slowest member. Even managers get sucked in: they build their own alliances by approving the incompetent work of their subordinates.
Part of investigating a “software nightmare” is to ask the programmers what they feel is wrong, and how to recover the project. The single most alarming fact that arises from these inquiries is that everyone can eagerly point the finger at someone else. Also, the software development process sounds eerily similar to the way one thinks of an old building: It’s always existed; it can’t be torn down; the only thing anyone can do is patch the old thing up and pray it does not collapse around them.
After this initial muddle, I talk with the managers officially assigned to the undertaking. They complain that upper management, demands a finished product without an adequate budget or a fair schedule.
Finally, I analyze the facts, and eliminate the opinions. In virtually every failed software project, this is the picture drawn by the cold, hard truth:
- The project lacked a clear development plan, including a reasonable budget and valid timeline.
- Management failed to hire a qualified architect.
- The program evolved spontaneously, using miscellaneous effort from mostly unqualified programmers.
- Because of changing requirements, the project stalled and so never achieved its originally (vaguely stated) goals. Meanwhile, the budget bloated out of control.
- Once the downward spiral had begun, no one in the company had the will to stop it. The topic of actually redesigning the software was so politically incorrect that no one had the courage to raise it.
- Anyone attempting to change the course of the project got labeled as a “malcontent”.
- Expectations plummeted.
- The company cut and eventually canceled the budget.
- The responsible personnel — especially managers who oversaw the entire process — were never punished. Instead, they received new assignments that eerily resembled the ones that had failed.
Their accounting department wasn’t of much help.
Every major corporation has an accounting department, and you would certainly think they could calculate the cost vs. benefit of the things they purchased. Yet they sometimes behave as if they don’t even know what a calculator is. Here are some juicy gems that I have witnessed personally over the past ten years:
- They approve the hiring of over a hundred programmers, but when asked to provide essential tools to help those individuals perform more effectively, they respond that such expenses are “unaffordable”. Sometimes the tools are even free; this does not seem to matter. The entire topic of acquiring tools is taboo. If you take the average programmer’s gross cost of around $120k per year, and figure that they could increase their accuracy, efficiency and effectiveness by at least 25% with proper hardware and software, it is easy to calculate the break-even: $30k per year. But the budget for new tools is nearly zero. If programmers struggle, are inefficient, and do sub-standard work that damages the company’s long-term objectives, that is like talking about next month’s weather with an accountant. It is not certain, is broad-sounding, and resides in the nebulous future, so cannot be calculated with their methodologies.
- They authorize contracts that require programmers to work long hours, including unpaid overtime, because that fits their budgets. But one day the Labor Board finds out and shuts the company down because of illegal practices. The company is then sued on behalf of the workers. The accounting department blames this on the legal department, but the entire company signed off on mistreating their workers until it bit them on the proverbial posterior.
- When a temporary programmer achieves greatness in his tasks, he is sometimes offered a salaried position. But the offer is inferior to industry standards, so the candidate refuses and moves on to work elsewhere. The company loses a valuable asset because it does not know how to quantify one person except as “one potato divided into all of the mashed potatoes we make each year”. This is another way in which the staff at major U.S. corporations devolves into incompetence over time. The workers that remain are the sheep that have learned to keep their mouths shut (except to chew their cuds).
- If one software project requires planning and design the accounting department chimes in, “Is this liberal arts stuff really necessary?” because no measurable change has taken place. Meanwhile, the same group cheerily authorizes any development assignment where no planning takes place, but instead the team charges blindly forward, announcing their progress in daily scrums and piling thousands of lines into the code base. Sure, the spontaneous project will probably fail, but that event will occur in the future. It won’t affect this year’s budget.
They failed to heed the warning signs of a project gone bad.
At the risk of sounding like a late-night talk show host, you know your software project is going terribly wrong when
- You are having trouble talking about what you are doing because you assume “it’s too complicated for normal humans”. Complexity is not a goal of software design. Simplicity is. If someone can’t describe an important aspect of your project to you, it is usually because they are testing your gullibility. Once you “pass” this test, you are doomed to receive a barrage of cryptic explanations of why nothing works and cannot be done on time or within budget.
- The program has not been completely described by requirements, or those requirements have begun to seem vague, as if they were cut-and-pasted from online samples. Your planners should have laid this out in extraordinary detail.
- You receive advice to adopt special management techniques like scrums where your workers get treated like military cadets. The goal is to “keep the boys in line”, but all this ever engenders is resentment. An efficient programming staff does not have time for unnecessary meetings, and should spend zero percent of its time trying to make management “feel better”.
- You realize that the planned approach to developing your software failed to remain open to obvious changes in technology and process within the industry. This means that your architect was an “old school” type that figured, “This is how we did it at IBM..”
- Your customers are not directly involved in the software development process from the start. These are the individuals who will have to buy and make use of the product. If they are absent, so is most of the information you need to make them happy.
- You have three or more managers in charge of the project, who spend their time in endless emails and meetings trying to demonstrate why they are essential to the process. This creates an irritating “noise” for those who are trying to produce the actual finished product. It also indicates an obsessive control disorder that many modern bureaucracies suffer from. The most happy and energetic companies I have seen actually don’t use management as corporations do. The owner is usually present, with an assistant. Anyone can talk to “the big chief”, but they also answer to them. Modern management creates a competition to identify the most obnoxious and demanding individual; the lucky winner gets to run the place.
- Your development teams are large, preventing them from acting efficiently. This also encourages managers to treat them as a mass group, without individual voices. Your programmers are the only reason you exist as a software development entity. They have a lot to say about how to become more efficient and effective, if anyone would listen.
- Your staff fails to answer emails or other forms of communication, or becomes vague in discussing the project within the team. Senior members become arrogant and distant because they feel indispensable to the outcome. You begin to feel that they are actually running your company out from underneath you (they are).
- The staff has to work overtime just to keep up. This usually means that the project is improperly planned and designed, and/or that the works don’t know what they are doing. A proper use of overtime would be to sprint to the finish of a quarterly deadline.
- New programmers take 3 months to become competent at the assignment and 6 months to become valuable. This reflects a cumbersome, unmanageable code base. If I had the choice, and some devilish attorney could make it legal, I would issue a polygraph test to every new programmer hired after 3 months. They would have to answer honestly, or would be fired. The answers would, however, be thrown into a hat so we did not know who said what.
- Is this software competently designed?
- Should the entire project be scrubbed and started over again?
- Who is valuable on the software team, and who should be fired?
- What can we do to make this program great?
- What can we do to help you maximize your potential?
- If you received a job offer from another firm, would you take it?
I would give these interviews 10x the weight of anyone else’s opinion on the staff, including my own. If the new programmers are properly selected, their impressions are priceless, if the company listens to them.
- The code base is improperly documented. You can actually judge code comments as an amateur (I’ll write about it soon). But that’s your architect’s job. The inability to describe what you are doing and why you are doing it reflects a lack of conceptual skills, which are more important than rote technical knowledge.
- When bugs arise in the program, they get recorded and fixed like isolated issues that will not take much time. In a professional environment, a bug is a “kind of problem” that is symptomatic in the location where it is found, but almost never isolated there. Virtually all bugs are design problems, and so must be addressed at that level. This is why your programmers should possess a 50/50 blend of conceptual and technical skills.
- The coders rush their work. Their goal is to achieve a result through symptomatic testing. The proper approach is to plan and design the requirements, the project, and the code as essential pre-steps to actual coding work. The faster a coder works, the sloppier they work. Yet when I ask managers to show me their “best coders”, I always get introduced to someone floating on a jagged cloud of crystal meth. The calm, quiet, deliberate workers get labeled as dullards, though they do the best overall work.
- The programmers talk about “refactoring” the code “when they find time”. This is the worst single statement any programmer can ever utter, as it indicates that they did not design their code properly in the first place, or they are covering up for someone else’s incompetence.
- The project requires complicated steps to produce and release for testing. This is the most frequently underestimated area of development because so much goes wrong so often that the staff gets numb to the problems and acts as if “we’ll handle that on release”. Yep, by your very angry clients.
- The customer must traverse a Mount Everest sized learning curve to adapt, convert, and use your program. They are too busy to begin such adventures. Besides, like all human processes, it involves variation and complexity that sucks up time like the desert sucks up moisture.
Management gets isolated at the top
Uneasy lies the head that wears a crown.
Most people fantasize about what it would be like to own a business. It’s like wanting to own a yacht. Yet there is that other saying: “The happiest two days in a sailor’s life are: (1) the day they buy their boat; and (2) the day they sell their boat.”
Running a business in the long-term is also like being an Indian King: “My slave is my master”. So I have a lot of sympathy for the founders of companies, as they want what everyone else does, but they have the courage and patience to stick with it, even when it costs them the most productive years of their lives.
Business owners (or top executives) also suffer from some interesting maladies that the rest of us can’t relate to:
- They want to achieve their vision so badly that they sometimes lose their vision — that is to say, their judgment. They can be capricious and reactive. This also occurs when you are emotionally exhausted after many years of struggling with the physical realities of running a company.
- They become so desperate to unload their burdens that they over-trust the help that they do hire, and so get taken advantage of. They hire the managers that hire the managers that hire the managers that oversee programming projects that can scuttle the entire business. In their attempts to delegate, they often lose control of very dangerous situations involving huge amounts of money, as well as their personal reputation.