When intuition fails, how to use probability and statistics to find the real answers

Imagine there is a bus that arrives every 30 minutes on average and you arrive at the bus stop not knowing when the last bus has left. How long can you wait for the next bus? Intuitively half of the 30 minutes sounds good, but you would be very lucky to just wait 15 minutes.

Say, for example, half the time the buses arrive 20 minutes apart and half the time 40 minutes apart. The overall average is now 30 minutes. From your perspective, however, you’re twice as likely to show up during the 40-minute interval than during the 20-minute interval.

This is true in all cases, except when the buses arrive at exact intervals of 30 minutes. As the dispersion around the mean increases, the amount by which the expected wait time exceeds the mean wait also increases. This is the Inspection Paradox, which states that every time you “inspect” a process, you will likely find that things take (or last) longer than their “uninspected” average. What appears to be the persistence of bad luck is simply the laws of probability and statistics that take their natural course.

Once you become aware of the paradox, it seems to appear everywhere.

For example, suppose you want to conduct a college average class size survey. Let’s say the college has classes of 10 or 50, and there is an equal number of each. The overall average class size is therefore 30. But by selecting a student at random, it is five times more likely that he comes from a class of 50 students than of 10 students. Thus, for each student who answers “10” to your question about the size of their class, there will be five who answer “50”. The average class size recorded by your survey is therefore closer to 50 than to 30. Thus, the fact of inspecting the class sizes considerably increases the average obtained compared to the true uninspected average. The only circumstance in which the inspected and uninspected mean coincides is when each class size is equal.

We can examine the same paradox in the context of what is called length-based sampling. For example, when digging up potatoes, why does the fork go through the very large? Why is the network connection interrupted while downloading the largest file? It is not because you were born unlucky, but because these outcomes occur for a greater extension of space or time than the average extension of space or time.

Once you know the inspection paradox, the world and our perception of our place in it are never quite the same.

Another day, you line up at the doctor’s office to be tested for a virus. The test is 99% accurate and you are positive. Now what is the chance that you have the virus? The intuitive answer is 99%. But is it true? The information given to us relates to the likelihood of testing positive since you have the virus. What we want to know, however, is the likelihood of having the virus given that you test positive. Common intuition confuses these two probabilities, but they are very different. This is an example of the Reverse or the attorney fallacy.

The meaning of the test result depends on how likely you are to have the virus before you take the test. This is called the a priori probability. Essentially, we have a competition between the rarity of the virus (the baseline rate) and how often the test goes wrong. Let’s say there’s a 1 in 100 chance, based on local prevalence rates, that you will have the virus before you get tested. Now remember that the test is false once in 100. These two probabilities are equal, so the chance that you have the virus when the test is positive is 1 in 2, although the test is 99% accurate. But what if you show symptoms of the virus before you get tested? In this case, we should update the prior probability to something higher than the prevalence rate in the population tested. The chance that you will have the virus when you test positive increases accordingly. We can use Bayes’ theorem to do the calculations.

In short, intuition often lets us down. Yet by applying the methods of probability and statistics, we can challenge intuition. We can even solve what might seem to many the biggest mystery of all: why we seem to get stuck in the slowest lane or queue so often. Intuitively, we were born unlucky. The logical answer to the Slower Lane puzzle is that this is exactly where we should expect it!

When intuition fails, we can always use probability and statistics to find the real answers.