Why Is Software Development So Hard?

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

“If you think it’s simple, then you have misunderstood the problem.”

— Bjarne Stroustrup, inventor of the C++ programming language

Hellish failure rates. Endless delays, miscommunication, cost overruns. Seriously peeved clients.

All of these factors are commonplace in the software development process. A recent Standish Group report indicates that only 35 percent of software projects are “successful,” meaning they met the deadline within budget and satisfied client needs. Some 19 percent were outright failures, with the rest muddling through in the middle ground.

Which begs the question: Why? Why is software development, which is done by groups of intelligent professionals, so god-awful difficult?

Scott Rosenberg, author of Dreaming in Code, a book about the development process, has mulled this question extensively. Having interviewed legions of programmers and been involved with software projects himself, he’s come to some conclusions.

“Dreaming in Code” by Scott Rosenberg

First, though development is a profoundly technical process, it’s not the technology that creates snafus. Rather, it’s the human-technology interface. Specifically, “The difficulty that people in management positions have in gaining visibility into the technical world.” And also – and more worrisome – “The assumptions that people make about software.”

For instance, compare software development with the process of building a bridge. The mix of steel and concrete that goes into bridge building is tangible to the casual observer. We can look at it and say, It’s about halfway done.

But with software? How are upper level execs supposed to know that a mass of code is halfway done?

They could, of course, ask the lead programmer, and in fact they often do. But that’s where the problems usually start.

In the mind of the lead programmer, once the project is decomposed into a series of technical problems, he can say with some assurance, “Oh, this module is halfway done, or this module, we haven’t started on that.” Those questions can be answered.

But when that programmer reports his progress to management, he’s talking to people who don’t see things as he does.

Rosenberg notes: “The assumption is, because we know it’s so easy to change – it’s code, right? We didn’t pour concrete into a foundation” – that it’s not difficult to change things.

Furthermore, “Programmers are often, to their credit, eager to please, and eager to prove their versatility and their ability to nimble. And so we don’t apply the same restraint and self-discipline when building software that we tend to use in physical world projects. We get a quarter of the way or halfway into it, and say, Let’s add some things.”

The innocent phrase “Let’s add some things” is rarely beneficial to the sanity of the programmers.

In fairness to management (a group that might not deserve such fairness), sometimes the very act of development opens new possibilities. So perhaps it’s only right and natural that they shift course in midstream.

“And maybe as it’s being built they get a clearer picture, and suddenly they see something they didn’t see before. Then they go, ‘Oh, I guess we should rebuild this part, and they go back to the engineers and say, ‘You know, we didn’t realize this.’

“And before you know it, you’re in trouble.”

Do the Programmers Share Blame?

Software and More

Who Killed the Software Engineer? (Hint: It Happened in College)

When Just Enough Is Enough To Be Fired

The 2008 IT Salary Guide

Understanding Your ‘Idiot’ Manager

FREE Tech Newsletters

Although it’s rarely the programmers who turn projects into quicksand, sometimes the classic programmer’s “personality profile” comes into play.

There’s often a particular personality type that goes into the field, Rosenberg says. “To master the technical complexity of getting the computer to do exactly what you want it to do, you have to be very precise. You have to be able to think in a way that makes that easy for you.”

“Sometimes – not always, but sometimes – if you are the kind of person who is able to think really easily about how to get a computer to do what you want, you’re not always the most imaginative person to think about how usersare going to do something. This is a classic issue in the field.”

Users (or at least certain users) are remarkably non-technical. Simple computer tasks confuse them terribly. For a technologically sophisticated programmer to put himself into the mindset of a blubbering newbie is difficult.

And, even if developers can adopt this “newbie blindness,” other project members may stymie their best efforts. The number of non-developers in any project is nearly uncountable: Usability experts, designers, executives, salespeople, the CEO’s mistress – anyone might have input.

“You’ve got all these layers of people,” Rosenberg says. “But the more layers you’ve got, the more opportunity you’ve got for miscommunication between them.”

A related aspect of the programmer’s personality plays a role. The really great achievements in programming are achievements in abstraction, Rosenberg says. “Great programmers tend to be really good at taking huge piles of specific details and finding patterns in them, and creating new abstractions that make it easier to deal with these details.”

“Dreaming in Code” by Scott Rosenberg

The downside: sometimes the person who is really good at that is a bit of a dreamer, someone whose head is in the clouds. “Someone who is not terribly focused on day-to-day, mundane matters like, ‘Oh, the product is due next week.’ Or, ‘Oh, I’m still trying to figure out the most elegant algorithm for sorting this stuff.’”

Making Software and Making War: Similar Concepts

“There’s an interesting parallel between warfare and creating software – and not just that every project is a battle,” Rosenberg says.

He points to the American experience in the Iraq war, in which Saddam Hussein was quickly toppled – yet that didn’t mean the U.S.’s true goal was achieved.

“The military people could justifiably say they won the war,” Rosenberg notes. “But if you look at the original, great theorist of war, Carl von Clausewitz, in the 19th century he said, ‘War is politics by other means.’ The only importance in a war is achieving your goal – you need to know what your goal is, and not get obsessed with details.”

“So if you conquer another country but you haven’t achieved your goal, what was your goal?”

In the same vein, a programmer could be capable of dashing off thousands of lines of pristine code in short order. “But if it doesn’t do what you were setting out to do, what good does it do?”

To keep his eye on the goal, a developer must often filter out unimportant noise. That noise might be excess detail by fellow programmers or a dose of management anxiety, but true success involves deciding what’s important and what’s not, and coding accordingly (if management allows that sane of an approach).

The Programmer in Society

The popular image of the software developer has seen an upheaval as dramatic as the change from mainframes to client-server.

Until the mid ’90s, programmers were viewed as hopeless techno-nerds, equipped with pocket protectors and the social skills of a suave seventh-grader. They were Chess Club refugees, fiddling with computers as their red-blooded classmates chased girls and played football. Bill Gates hews to this original stereotype; he became rich and famous yet remains a software geek.

But the rise of the Internet changed things. “First, it had a hugeimpact on everyone’s lives. It was something that people could see,” Rosenberg notes.

Software and More

Who Killed the Software Engineer? (Hint: It Happened in College)

When Just Enough Is Enough To Be Fired

The 2008 IT Salary Guide

Understanding Your ‘Idiot’ Manager

FREE Tech Newsletters

People understood that techno-wizards not only created the Net – envisioning this brave new world – they kept pushing it forward with imaginative, cool software. While the ex-football jock now managed the local QuickMart, the dorky programmers were making good money (in some cases verygood money). Programmers were hip, if still somewhat odd.

More recently, a small clique of programmers has pushed the public perception of developers still further ahead by blogging. It makes a difference, Rosenberg says, “The number of programmers who started writing about their work, sometimes just for technical reference, but beyond that, writing about their lives, their experience, what it’s like to be a programmer.”

In particular, software experts like Joel Spolsky and Paul Graham have become highly visible commentators about the development process.

Their Web-based writing has “given programmers a chance to see, ‘Oh, there’s other people who are like me and have similar experiences.'” And for people on the outside who want to understand, it’s cracked open the door to the mysterious process of building software.

Here’s an in-depth list of tech-related blogs.