Wednesday, June 2, 2010

Simpsons Paradox

Consider the following cartoonish thought experiment:

Homer and Lenny get into a week-long grudge match at the nuclear power plant over which of the two can eat more donuts. They decide to settle it once and for all in a weekend donut-eating contest: each of them gets 100 donuts, whoever eats more by the end of the weekend wins.

Lenny secretly knows Homer will be able to out-eat him, but he also knows something about statistics that he's hoping Homer doesn't. Lenny suggests, and Homer agrees, that Lisa will be in charge of moderating the match to keep things fair.

Lisa will buy 100 donuts each morning and divide the 100 donuts into two boxes, one for Homer and one for Lenny. After setting the two boxes out for the day, Lisa will return periodically to mark the percentage of each box's donuts that have been eaten. In this way Homer and Lenny know how they're faring against each other and each can adjust their eating behavior over the day to try and keep up with the other.

Here's how it goes down:
    1. On Saturday, Homer eats more of his box of donuts than Lenny eats of his box of donuts.
    2. On Sunday, Homer eats more of his box of donuts than Lenny eats of his box of donuts.
    3. On Monday, Homer is shocked to find that Lenny has won in the final tally by over a dozen donuts!

    Wait, how did that happen? Lenny didn't cheat and Lisa didn't divide them unfairly; each got the opportunity to eat 100 donuts. So why didn't Homer win when the percentages always showed him in the lead? Lisa divided each daily allotment of 100 donuts into a box of 90 donuts and a box of 10 donuts. Lenny won by weighting.

    Simpson's Paradox, as explained by a singingbanana:


    Why is Simpson's Paradox important to remember in the real world? As mentioned in the video above, direct percentage comparisons of weighted data is a risk in any field using statistical analysis, especially the social sciences. Failing to consider the meaning of statistical percentage comparisons can lead to less favorable outcomes given seemingly favorable supporting data. If you want your doctor to pick the best medicine (Drug A) for you, you'd better hope he gets the proper recommendation from the groups running the statistical analysis first. Otherwise you may get worse medication despite the availability of a more effective alternative, and neither you nor your doctor will be the wiser.

    Statistical analysis is an important way to get a holistic understanding of phenomena, but interpreting test results isn't as easy as it looks; sometimes you miss the holes staring you right in the face.
    blog comments powered by Disqus

    Think about...

    Random Thoughts

    Where Thinkers Come From
     
    Real Time Web Analytics