Header image
Information Technology In Business

12 Tough Questions

1. Numbers - Why isn't the improvement quantified?

We are quick to quantify those precious resources “money” and “time”. But competitiveness is not reducing cost or time alone. It is only useful if the “right” product or service is delivered. The “right stuff” is the critical question. The right stuff can be classified as “qualities” and ”benefits”, or more generally as “advantages”. Practically every advantage can be ultimately evaluated in terms of money. But, most of them need direct measures of their own, so that we can control and envisage them before the payoff.

How often do you read the words: “improved”, “better”, “enhanced”. They should be forbidden in serious management planning. They need to be replaced by two numeric points on a scale of measure, your current level of performance, and your planned level of performance in the future. So, for example, the phrase “ leading to a substantial increase in product reliability” should be replaced by “reaching 99.9% uptime during customer use, by next year, as opposed to less than 85% this year.”

I have found that “intangibles” are quantifiable. I have found that qualitative ideas can be quantified hang on! almost without exception. The concept that management must quantify to get control is not new. But most managers today still have a large number of concepts, important to their daily work, which they do not view as quantifiable. Neither do their immediate surroundings set a quantification example either. This is a combination of lack of leadership and training their boss should have insisted on quantification and shown the way. The facts on how to quantify things should be made available.  The following have been quantified in my practical work: 

  • Adaptability
  • Usability
  • Portability
  • Security for example. 

If you don’t know how, or don’t believe it is possible, then you too have something to learn [note 1]

2. Risk - What's the degree of risk or uncertainty, and why?

It is one thing to put a number on a planned quality, but quite another to quantitatively estimate how much the reality might vary.  “The requirement from our customers is 99% portability at least!” How sure are you? Well, nothing is absolutely for sure! But that is the number marketing came up with. Give me some idea of extremes. (later) Most of our customers only need 90% or less. A few new market areas probably require 99% or more.

One prospect has been identified who wants 99.99% or better. Thanks, that is useful. Treat all such numbers like that. We need to understand the variation, not just some average or maximum. You should require as the norm that all estimates include an uncertainty estimate, and a reason in writing for that uncertainty. Uncertainty is a way of expressing how wrong you would be if you only used the one number in your planning. Too good for most of your customers and much worse than needed for some others! Above all, this question makes people think about what they know, to inquire more deeply, and to plan for the variation.

3. Doubt - Are you sure? If not, why not?

People do not seem to feel responsible for what they write, propose or say. Somebody else told them. Mr. Nobody is responsible. They should learn to be sure or to be sure that they are no. Uncertainty should be stated in no uncertain terms. Most leading edge planning is done under conditions of great uncertainty. New markets. New competitions. New technology.

We cannot let these factors be an excuse for sloppy planning. They are the very reason we must introduce some discipline in establishing what is for sure and what isn’t and why it isn’t. “UNIX is the standard of the future”. Are you sure? Of course, everybody in the industry says so. What % of systems in OUR marketplace in five years will use UNIX? I don’t know! Make an estimate and break it down by customer type.(later) Less than 40% (plus or minus 20%) of our customers will require or use UNIX in five years. Thanks! I hope you will be that clear in the future.

4. Source - Where did you get that information? How can I check it out?
People too rarely bother to credit their sources. But, if sources are not documented, they cannot easily be checked. If they are not checked, they can easily be wrong, outdated or not credible. If people know that sources can and will be checked, they will take more trouble to be accurate in the first place. I use a little left arrow “fact ç source” as a simple shorthand for giving the source, and insist that every critical fact have its source documented briefly. I make it a “standard”. For example: “Our present service level is 80% çQuality Audit May this year page 65.”

Most management documents, after approval and meetings, contain (hang on again!) between 10 and 50 major problems per page (you read me right!). I know that this will cause the reader to doubt me. My source? Frequent personal involvement and measurement by deep quality control auditing for many years. It rarely fails to turn up that number no matter who does the checking [note 2] We need to make it practical and economic for people to check a document’s content without having to guess what the source is. We need to get people into the habit of making sure that their sources are reliable and that they have reliably represented the information from that source. Making them document the source is a useful first step in that direction. Checking it is the next step. The cost of checking when the source is not given, clearly exceeds the cost of getting people to annotate their source. The cost of being wrong probably exceeds the cost of noting the source and getting the data checked later.

5. Impact - How does your idea affect my goals and budget? Measurably?
Everybody has an opinion about their favorite strategy, technique or product. But, few can give you substantiated numeric facts about how their suggestion will impact all of your critical success factors: benefits, qualities, time and costs. We do not want to know “how good things are in general”; but how they affect our particular goal levels in a given time frame. My experience is that even when we search diligently, the hard facts in the relevant areas are hard to come by. But, we are speaking of potential impact of a plan or design on your critical success factors. By definition, if you don’t know what the impact is, you risk failure. So, we have to ask the question. We have to take the trouble to find out what we can. We have to get numeric estimates (not just “a little” as in Quality Function Deployment (QFD) and similar simplistic methods. How much do three “littles” and one “medium” add up to in terms of my required levels? Sometimes just getting a number and an uncertainty estimate and a source, even if shaky, is useful, For example “the impact on our reliability quality objectives is <50% of the target (±20%) çWild Guess by Jones.” Even if we only clearly establish that nobody has any knowledge of the probable effects, or are making educated guesses, we have learned a lot about the risk we take if we approve a poorly founded plan.
6. All critical factors - Did we forget anything critical to survival?

There are many dozens of factors which can cause the death of the best laid plan. There are too many factors to pretend to identify and control all of them directly. Some are so obscure and improbable, that we just have to await their first warning signs and hope we can act in time to avert failure. I know that total control over even a few critical factors (for example meeting deadlines, budgets and two critical quality areas) is actually quite difficult for most organizations. This is particularly true for ambitious, world class, competitive organizations.

For this reason I usually recommend that planners limit themselves to planning for a a “top ten” critical success factors with numeric control. Full control even of these few is beyond most organizations. So, the question “did we forget anything critical” must invariably be answered “Yes, of course!”. But the question is not to be interpreted quite so literally. I find that projects regularly fail to identify and control well known factors which constantly cause problems by being out of control. For example software projects have well known problems in the maintenance phase, which accounts for most of their lifetime costs. But it is rare to see a formal maintainability objective (like “mean time to repair of the program less than 30 minutes”) in any software project. Another example.

Most planning projects try to estimate startup costs, but ignore future training, recruitment and other operational costs bound to occur in the future. A further example. Everybody agrees that systems should be userfriendly, but try to find the measurable objective stated somewhere. We all agree about security and flexibility; again, try to find the stated measurable objective and the will to track the result. It is not enough to wave our hands shouting generally agreed buzzwords. If that be done, then we are missing the objective and we shall surely fail to implement what we might have needed to achieve. As a practical minimum, think about problems in your last project and ask if there is a lesson about benefits or costs which must be put under control in the new plan.

7. Evidence - How do you know it works that way? Did it ever?

Asking for the source of information enables you to ask some better questions about it. One critical question of the source is ”proof”. You are telling me strategy Alpha is a winner. How exactly do you know? It is amazing that many educated people do not seem to know this concept. They answer, “because it is obvious”, “because everybody knows it is so” and all manner of worthless defense. But not a shred of evidence is proffered voluntarily. We need to teach and ask clearly for relevant facts as evidence.

Where was this strategy practiced? When? By Whom? How did it work? How was this measured? How did things go in the long term? What reason have we to think it will or will not work in our particular project? Are you sure of cause and effect? Even with the best of facts and reasoning, a strategy or design may not work in our particular case. We should start our work with good reason to believe that things will work. If people cannot give you that, then clearly you are engaging in a level of risk that you should be prepared to fail with.

I teach that every estimate of any consequence, of quality or cost, must be accompanied by individual written “evidence” and the evidence must be relevant facts. This should be the rule. It is the exception. It must be cheaper to dig out the facts, than to persist in discovering them the hard way, too late.

8. Enough - Have we got a complete solution? Are all requirements satisfied?

A “complete solution” must both meet the planned levels of quality and benefit, and fall within the resource constraints. A complete planned solution needs some safety margin in case we are over optimistic about results. I recommend that this question be answered with a table (an Impact Estimation table) with critical factors on the left and strategies on the top.

The intersection between any objective and any strategy is an estimate of the impact (0% means no change from the present, 100% means it will meet our planned or budgeted level). All other estimates are relative to these two notions. The numbers can be added horizontally to get a rough feeling for the completeness of the plan, and benefit numbers can be divided by cost numbers on the vertical plane to get some impression of the relative benefit to cost ratio of each strategy.

This is the same principle as Quality Function Deployment, except that we have numbers for our objectives and numbers for the impact as the norm (not an optional extra) and we get a much better picture of the plan. I have used this device in all manner of Corporate planning (Corporate objectives versus Strategies) and at technical product design levels. It is as dramatic as putting on your first pair of glasses.

9. Profitability first - Are we planning on doing the 'profitable things' first?

Most people make the mistake of planning to do too much at once. Very little in our present planning culture encourages step by step delivery of the results we want. Big bang is the norm. The deadline (singular) is King. This is almost acceptable in a stable culture. Requirements are fixed. Technology is known. Build the bridge and tell us when it is done. But most of us are working in a very different situation. The politics change dramatically. The customers and market keep changing their minds, abetted by the competitors. New untried technology demands to be exploited for we fear lagging behind the competition or not being the best. No stability. Chaos. This demands a different approach to planning.

The long term vision has to be systematically broken down into small fractions (1% to 5% of budget) of deliverable results. This has many positive effects. Complete control over deadline and budget [note 3] Early useful results. Learning of what works and what doesn’t. Ability to modify the plans after each partial step. And, choice of which step to deliver next. This last capability is like the choice confronting the chess player at every move. How to get maximum benefit from each next move.

Each increment of results must be evaluated for benefits and costs in relation to the long term objectives and budgets. The one with the best “profitability” (benefits to cost ratio) should normally be chosen. This has a dramatic effect on cash flow and on capital budgeting. Management has to learn to insist on this type of planning.

10. Committment - Who is responsible for failure, or success?

Who is responsible for all this? For the planned levels of quality being the right ones? For the facts being correct? For the strategies being adequate? For the plans working in practice? If these things fail, will they stand responsible? Do they know it? Nobody is perfect. We will all make some mistakes. But have they been forthright about problems, or hidden them? Have they indicated the degree of risk involved or played it down? Have they built in a safety margin or taken a chance? Are they playing with your reputation or theirs? Your money of theirs? Will they put their money where their mouth is? If not, would you?

11. Proof - How can we be sure the plan is working, during the project, early?

The plan was great but the project died. How can you assure your boss, your company, that the plan is on track? Not the bureaucracy of the plan creation , the real implementation of it. How can you be sure that the money is well spent and is producing results worth the spend? Will you have to spend the entire budget, and more perhaps, to find out if it works at all? Can the project team prove their worth on a smaller scale of the project before continuing? Are they willing to be put to this acid test? If not, why not? What do they fear? Are they making excuses that the project cannot be broken down into smaller deliverable result steps? Is the first result “a long time and a great deal of money” away.

If things go wrong, will they get paid but your job will be on the line? Almost any plan, no matter how large or small (I have successfully demonstrated this on both 2 persons for 3 months, as well as a 986 person project times four years) can be chopped up into increments of benefit delivery at arbitrarily small levels (1% to 10% of total are common increments). There are two conditions for control here. You must be able to measure progress on all critical benefits and resources on a “continuous” basis. Then you must be able to learn from your experiences and change plans and designs in order to succeed, if the feedback from early delivery steps says things are not on the right track.

Finally, the individual steps must be small enough that individual failure does not threaten the larger project. You may lose a week, but not the entire year, by betting on a bad untested idea, which looked good at the time. Practical useful delivery of change, of quality improvement, of benefits “as you go” is the only reliable sign that a project is on track. Don’t listen to peoples excuses about why this cannot be done on this project. Get control, not in theory but in practice. For some managers this is a radical cultural change. For others it is just common sense. This is the Deming Cycle, this is Tom Peter’s “small wins” mentality.

12. No cure, no pay - Is 'no cure, no pay' in the contract? Why not?

If you are buying external services or products, as part of your solution and you do not get the results envisaged who pays? You or they? Do you have a contract with them making payment conditional on the results being achieved in your company? Not just their product or service delivered, but the followon effects the savings, the personnel reductions, the faster service, the greater profit.

They will argue that that is your job, not theirs. True, but you can argue why should I buy your stuff if it is not going to give me what I want. By involving them at some level they may turn in to a more active partner in helping you make those final results happen. Or, it may become clear that their product or service alone is not the guarantee of success that their salesperson claimed. You need to “buy insurance”, to “transfer risk”, to provoke the truth out into the open. They would love to take your money and run. But don’t let them. You don’t need to. Find a supplier who will take responsibility. The search is enlightening.


1. Details of the definitions of these concepts are found in Chapter 19 of Gilb: Principles of Software Engineering Management”, they apply to hardware systems as well. Most of them require breakdown into subfactors before quantification. A simple demonstration is that usability can be expressed as “mean time to learn”. If this is too simple, see the book.

2. The method used is a document inspection method developed by IBM. It is based on a small team, using about one hour per page to check and one hour per page to report and do further checking. Written standards (like “All statements shall be unambiguous”) are part of the game. For example at a very large US Corporation, in Chicago, in May 1990, three independent employee teams checked the Corporate Quality policy hanging on every meeting room wall, and signed by the Chairman. Independently of the other two teams each claimed to me that they found about 50 Major problems on the page and about three Extremely serious problems. Your own employees cannot be that consistently wrong!

3. IBM Federal Systems Division used this method from about 1970 on large military and space projects, claiming almost perfect control over deadline and costs as a result (Dr. Harlan Mills in IBM Systems Journal No. Four 1980. A typical four year project delivered to the Navy in monthly usable deliverables.

Tom Gilb


Page updated: 20-Feb-2009