My Three Worst Experiences in Software Development

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

(Add a comment or describe your worst software development experiences at the bottom of the page.)

“Gone?”

I almost dropped the cell phone and almost hit the car in front of me. The phone call was the result of something bad happening on a high profile data conversion project. Our team had to convert three legacy payroll and HR systems into a brand-spanking new ERP system. And we had to do it in a weekend or potentially impact the payroll of 100,000 employees.

Now, everything was going down the toilet.

I’m sure everyone in software has moments in their careers where they heard that same flushing sound and hoped it was just a project and not their career going down the commode. The fact is, it’s just part of working in IT. Like any job, things go wrong.

Although in IT, it’s kind of like working for the power company because it’s only when things go wrong that anyone pays any attention to you.

When I reminisce over my twenty year, mostly non-eventful career in IT, there are three instances I’d like to share with you, which will hopefully help you realize you aren’t alone the next time you have a really bad day.

Getting back to my conversion project, frankly, I thought it was going to be a breeze. We had run multiple simulations, which not only instilled confidence that the data would be accurate, but more important, we knew exactly how long the conversion would take to run.

But first the pre-conversion scripts had to run to scrub and prep the data for the actual conversion. Those scripts take two days to execute and verify the results. Two days.

I left work about 3AM the night (morning) before after successfully running the scripts. Everything had gone peachy. As I drove back into work at 9 AM (not sure why I even bothered going home), I receive that unwelcome cell phone call.

It was my boss and he said there was a little problem with the conversion data. “Actually”, he said, “there was a big problem because the conversion data has been wiped out. It’s gone.”

I thought “Oh crap” what did I do wrong?I went over and over the script progressions in my head, but everything had checked out.

“What do you mean, gone? How?”

Turns out our DBA had accidentally pointed the nightly backups to the directory we used for the conversion. He over wrote the data.

It was an honest (but very painful) mistake. The backups were supposed to backup the conversion data, not overwrite them. In retrospect, should have done this backup manually. We never even tested the automated backup process – which turned out to be the primary weakness in our conversion plan.

Now I had to find a way to rerun the scripts in eight hours instead of two days. I quickly identified which scripts were not dependent on each other and created a multi-threaded approach that enabled us to rerun everything just in time.

Unfortunately, haste makes waste and sure enough there were some errors in the converted data as a result of this last minute tinkering.

Ultimately, it didn’t matter. In the process of devouring an entire bottle of antacids, we were able to manually clean up the data and the first payroll run that next week went off without a hitch.

My second bad IT experience was at my first job out of college and it didn’t end so well.It involved another data conversion project, but this one was for a mammoth financial management system for one of the largest US government agencies. I knew this project wasn’t going to be any fun when I realized the conversion would happen in August, the hottest time of the year in Texas.

I remember going to lunch was torture because the car was about 120 degrees Fahrenheit. But if we didn’t go out to eat, we didn’t see the light of day.

There was a team of about twenty people that worked on the preparation for this conversion over a one year period. The last few weeks were almost nonstop conversion dry runs.

We’d work a few days straight, take a breather, and then do it all over again. The only time I spent at home (when not in sweltering Texas) was to sleep and shower.

Finally it was time to do the actual conversion under the hot Texas sun. Okay, we were really in a freezing data center under the hot Texas sun, but we were still sweating under the pressure.

Each program took many hours to crunch through and convert all the data. So it was hurry up and wait, then validate.This was another one of those weekend conversions, so the team was under the gun to finish before users logged in Monday morning.

We almost didn’t make it.

In truth we didn’t make it, at least not completely. Out of the five developers who had written conversion programs, it turned out that mine was the only one that caused a major problem. I ended up using an older version by mistake and one particularly important data mapping was not correct, causing hundreds of thousands of duplicate records to be created.

Discovering this on Monday morning at 3 AM was not a pleasant feeling. I had let the team down and put us all in jeopardy of failing.

Through teamwork we were able to clean up most of the data by 8 AM, but not everything. Therefore part of the system was not available that first week until all data was scrubbed clean.

As I walked back to the hotel, we passed the training team waiting to go in and educate a group of users. The first thing they said was “you guys stink!”

I took the comment literally and felt like socking the dude, but the reality was that we hadn’t left the data center all weekend, thus no showers. We really did stink.

I went to my hotel room and passed out for 12 hours, totally exhausted and despondent.

Looking back on it, I realize that these were great learning experiences that everyone has to go through. We all hit our rough patches. They key is that we learn from them.

I’m extra careful about backing things up and making sure I have the correct versions of software, documents, and whatever I’m delivering to a customer.

So what is my third worst experience?

That’s easy. After finishing a six month project, I had to repeat almost the exact same project for another customer.

Same custom software, just with different data sources.

I was absolutely bored out of my mind! There is nothing worse than showing up for work knowing that you won’t be challenged and that you won’t learn anything new.

I’d learned tons from the first two bad experiences and would do them over again. However, I can’t shake the feeling that any customer would rather I be bored to tears.

Eric Spiegel is CEO and co-founder of XTS, which provides software for planning, managing and auditing Citrix and other virtualization platforms.