The Improbability Principle

Extremely improbable events are commonplace.

It’s an unusual day if nothing unusual happens”. This aphorism encapsulates a characteristic pattern of events called the Improbability Principle. Popularised by statistician Sir David Hand, emeritus professor at Imperial College London, it codifies the paradoxical idea that extremely improbable events happen frequently.  [TM112 or search for “thatsmaths” at irishtimes.com].

From front cover of  The Improbability Principle

We all experience coincidences on an everyday basis: you are thinking of a long-lost friend just as she phones you; on a holiday in Rome you meet your next-door neighbour; you join a bridge club to find there is already a member with the same name as your own. It seems that highly improbable events are commonplace.

Five Underlying Laws

Prof Hand appeals to five underlying strands that together make up the improbability principle. First, the “Law of Inevitability” states an obvious idea: something must happen! In a lottery draw, it is highly unlikely that you will pick the winning numbers, but quite likely that some lucky person has bought a winning ticket. After all, many tickets are sold and some particular combination of numbers must win.

Next, the “Law of Truly Large Numbers” says that, given enough chances or choices, something outlandish is bound to occur. With regular lotteries throughout the globe, the same six numbers in two consecutive draws are to be expected somewhere, sooner or later. Indeed, just such a coincidence occurred in September 2009 in the Bulgarian lottery.

What about Ireland? With 6 numbers to choose from 47, there are about 11 million possible combinations. But after 4,000 draws the chances are better than even that the same 6 numbers will have come up twice. This is very surprising; the same counter-intuitive result occurs when we consider the chances that two people in a group share the same birthday. A simple calculation shows that with just 23 people the chances of a common birthday are better than evens.

From front cover of  The Improbability Principle

Then the “Law of Selection” shows that consideration of events that have already happened gives a distorted impression: by focussing on the occurrence of the same set of winning numbers in consecutive weeks, we tend to ignore the reality of the many thousands of cases in which no such coincidence has occurred.

What Hand calls the “Law of the Probability Lever” shows how minor changes can have a major impact on probability. Changing the assumptions about the underlying probability distribution can greatly affect the likelihood of extreme events. Recognizing that events thought to be independent of each other are actually connected can change a “remarkable coincidence” to something to be expected. Finally, the “Law of Near Enough” deals with treating similar events as if they are identical. If you do not find an exact match, you look for a near match and disregard the discrepancy.

The Improbability Principle

Prof Hand weaves these five strands into a general principle that gives him the title of his book “The Improbability Principle”, which bears the subtitle “Why Incredibly Unlikely Things Keep Happening”. He applies this principle to gambling, extreme weather, medical tests and motor accidents and more widely to explain why surprising things happen regularly and how statistical analysis gives us insight into a wide range of real-world events.

Sources

Hand, David J, 2014: The Improbability Principle. Scientific American, 288pp. ISBN: 978-0374175344