False Precision


Eddie sez:

False precision is an engineering concept that basically says you cannot assume a level of precision of a result greater than the precision of the numbers used to produce that result. It is easy concept to understand with a few examples, but it is a concept many seem to be fooled by in real life.

For example, If your engines have a fuel flow of 1,000 pounds per hour on a gauge that has two digits produced by an LED that change and two painted digits that do not, you accuracy is plus or minus 100 pounds. If the meter reads "1000" the only possible choices above and below are "900" and "1100" because the numbers in the ones and tens place cannot change. If you have 11,000 pounds of fuel now and your two engines are indicating 1,100 and 1,200 PPH each, you could accurately say you are burning 2,300 PPH total. If I ask you to predict your total fuel in 30 minutes, you could say 11,000 - 0.5 (2,300) = 9,850 but you would be overstating the accuracy of your prediction. Because each gauge is only good for plus or minus 100 PPH and you are using two of them, you could be off by as much as 200 PPH. So the mathematically pure answer should be "9,850 pounds, plus or minus 200 pounds."

Is this important for a pilot? It might be or it might not. But you should understand that rounding errors only get worse as we manipulate them.

Everything here is from the references shown below, with a few comments in an alternate color.

Last revision:


False Precision

Significant Digits

Significant digits are those that carry meaning to the number in question. There are rules about these things in engineering, but these rules are ignored by most editors for the sake of readability. If you fill your car with gas and the meter finishes at 9.3 gallons, you would never record the result as 9.300 gallons because you know the meter was reporting the number with such precision. If the meter finished at 9.0 gallons, you could say you had "9 gallons" or "nine point zero" gallons. But you would be falsely accurate if you said "nine point zero, zero gallons."

So which digits are significant and which are not? In general:

  • All non-zero digits are significant
  • Zeros between non-zero digits are significant.
  • Leading zeros are never significant.
  • In a number with a decimal point, trailing zeros, those to the right of the first non-zero digit, are significant.
  • In a number without a decimal point, trailing zeros may or may not be significant. More information through additional graphical symbols or explicit information on errors is needed to clarify the significance of trailing zeros.

A Few Examples

A tour guide at a museum says a dinosaur skeleton is 100,000,005 years old, because an expert told him that it was 100 million years old when he started working there 5 years ago.

A press report in the United States said the following: "European authorities estimated that the bomb used 220 pounds of explosive." Who would estimate such an amount? The European authorities probably estimated the bomb used 100 kg of explosives and the American editor converted the number to pounds. The editor would have been better off saying: "European authorities estimated that the bomb used about 100 kg (220 lbs) of explosives."

Back in the days the 3.5 inch disk was the standard in the computer industry an American could translate these as 88.9 mm disks. But the 3.5 is a rounded number and the disks were actually 90 mm.

A Poem and a Critique

Thou shalt not be saved by works;
Thou hast been a sinner too;
Ruin'd trunks on wither'd forks, Empty scarecrows, I and you!

Fill the cup, and fill the can;
Have a rouse before the morn;
Every moment dies a man,
Every moment one is born.

—Alfred, Lord Tennyson

In your otherwise beautiful poem, one verse reads, "Every minute dies a man, Every minute one is born"; I need hardly point out to your that this calculation would tend to keep the sum total of the world's population in a state of perpetual equipoise, whereas it is a well-known fact that the said sum total is constantly on the increase. I would therefore take the liberty of suggesting that in the next edition of your excellent poem the erroneous calculation to which I refer should be corrected as follows: "Every moment dies a man, And one and a sixteenth is born." I may add that the exact figures are 1.067, but something must, of course, be conceded to the laws of metre."

—Charles Babbage

From The Thrilling Adventures of Lovelace and Babbage, Sydney Padua, 2015


See Also:

Padua, Sydney, The Thrilling Adventures of Lovelace and Babbage, 2015.