Traditional techniques and motivating staff

Software Testing Club magazine

This article appeared in the February 2010 edition of Software Testing Club Magazine, now the Testing Planet.

I didn't post this on the website immediately because I had some reservations about the article. I wondered if I had taken a rather extreme stance. I do believe that rigid standards and processes can be damaging, and I certainly believe that enforcing strict compliance, at the expense of initiative and professional judgement, undermines morale.

However, I thought I had perhaps gone too far in dismissing the idea of any formality, and that I might be seen to be advocating software development as an improvised activity with everyone making their own rules

I have expanded on this in my blog, (see "Tom Gilb & project requirements") where people can pass comment. For now, please enjoy this polemic!

Do we follow the standards or use our initiative?

Recently I've been thinking and writing about the effects of testing standards. The more I thought, the more convinced I became that standards, or any rigid processes, can damage the morale, and even the professionalism, of IT professionals if they are not applied wisely.

The problem is that calling them "standards" implies that they are mandatory and should be applied in all cases. The word should be reserved for situations where compliance is essential, eg security, good housekeeping or safety critical applications.

I once worked for a large insurance company as an IT auditor in Group Audit. I was approached by Information Services. Would I consider moving to lead a team developing new management information (MI) applications? It sounded interesting, so I said yes.

On my first day in the new role I asked my new manager what I had to do. He stunned me when he said. "You tell me. I'll give you the contact details for your users. Go and see them. They're next in line to get an MI application. See what they need, then work out how you're going to deliver it. Speak to other people to see how they've done it, but it's up to you".

The company did have standards and processes, but they weren't rigid and they weren't very useful in the esoteric world of insurance MI, so we were able to pick and choose how we developed applications.

My users were desperate for a better understanding of their portfolio; what was profitable, and what was unprofitable. I had no trouble getting a manager and a senior statistician to set aside two days to brief me and my assistant. There was just us, a flip chart, and gallons of coffee as they talked us through the market they were competing in, the problems they faced and their need for better information from the underwriting and claims applications with which they did business.

I realised that it was going to be a pig of a job to give them what they needed. It would take several months. However, I could give them about a quarter of what they needed in short order. So we knocked up a quick disposable application in a couple of weeks that delighted them, and then got to work on the really tricky stuff.

The source systems proved to be riddled with errors and poor quality data, so it took longer than expected. However, we'd got the users on our side by giving them something quickly, so they were patient.

It took so long to get phase 1 of the application working to acceptable tolerances that I decided to scrap phase 2, which was nearly fully coded, and rejig the design of the first part so that it could do the full job on its own. That option had been ruled out at the start because there seemed to be insurmountable performance problems.

Our experience with testing had shown that we could make the application run much faster than we'd thought possible, but that the fine tuning of the code to produce accurate MI was a nightmare. It therefore made sense to clone jobs and programs wholesale to extend the first phase and forget about trying to hack the phase 2 code into shape.

The important point is that I was allowed to take a decision that meant binning several hundred hours of coding effort and utterly transforming a design that had been signed off.

I took the decision during a trip to the dentist, discussed it with my assistant on my return, sold the idea to the users and only then did I present my management with a fait accompli. They had no problems with it. They trusted my judgement, and I was taking the users along with me.

The world changed and an outsourcing deal meant I was working for a big supplier, with development being driven by formal processes, rigid standards and contracts. This wasn't all bad. It did give developers some protection from the sort of unreasonable pressure that could be brought to bear when relationships were less formal. However, it did mean that I never again had the same freedom to use my own initiative and judgement.

The bottom line was that it could be better to do the wrong thing for the corporately correct reason, than to do the right thing the "wrong" way. By "better" I mean better for our careers, and not better for the customers.

Ultimately that is soul destroying. What really gets teams fired up is when developers, testers and users all see themselves as being on the same side, determined to produce a great product.

Development isn't engineering

Reality is chaotic. Processes are perfectly repeatable only if one pretends that reality is neat, orderly and predictable. The result is strain, tension and developers ordered to do the wrong things for the "right" reasons, to follow the processes mandated by standards and by the the contract.

Instead of making developers more "professional" it has exactly the opposite effect. It reduces them to the level of, well, second hand car salesmen, knocking out old cars with no warranty. It's hardly a crime, but it doesn't get me excited.

Development and testing become drudgery. Handling the users isn't a matter of building lasting relationships with fellow professionals. It's a matter of "managing the stakeholders", being diplomatic with them rather than candid, and if all else fails telling them "to read the ******* contract".

This isn't a rant about contractual development. Contracts don't have to be written so that the development team is in a strait-jacket. It's just that traditional techniques fit much more neatly with contracts than agile, or any iterative approach.

Procurement is much simpler if you pretend that traditional, linear techniques are best practice; if you pretend that software development is like civil engineering, and that developing an application is like building a bridge.

Development and testing are really not like that all. The actual words used should be a good clue. Development is not the same as construction. Construction is what you do when you've developed an idea to the point where it can be manufactured, or built.

Traditional techniques were based on that fundamental flaw; the belief that development was engineering, and that repeatable success required greater formality, more tightly defined processes and standards, and less freedom for developers.

Development is exploring

Good development is a matter of investigation, experimentation and exploration. It's about looking at the possibilities, and evaluating successive versions. It's not about plodding through a process document.

Different customers, different users and different problems will require different approaches. These various approaches are not radically different from each other, but they are more varied than is allowed for by rigid and formal processes.

Any organisation that requires development teams to adhere to these processes, rather than make their own judgements based on their experience and their users' needs, frequently requires the developers to do the wrong things.

This is demoralising, and developers working under these constraints have the initiative, enthusiasm and intellectual energy squeezed out of them. As they build their careers in such an atmosphere they become corporate bureaucrats.

They rise to become not development managers, but managers of the development process; not test managers, but managers of the test process. Their productivity is measured in meetings and reports. Sometimes the end product seems a by-product of the real business; doing the process.

If people are to realise their potential they need managers who will do as mine did; who will say, "here is your problem, tell me how you're going to solve it".

We need guidance from processes and standards in the same way as we need guidance from more experienced practitioners, but they should be suggestions of possible approaches so that teams don't flounder around in confused ignorance, don't try to re-invent the wheel, and don't blunder into swamps that have consumed previous projects.

If development is exploration it is thrilling and brings out the best in us. People rise to the challenge, learn and grow. They want to do it again and get better at it. If development means plodding through a process document it is a grind.

I know which way has inspired me, which way has given users applications I've been proud of. It wasn't the formal way. It wasn't the traditional way. Set the developers free!