Telephone telepathy study: some kind words

There has been a lot of criticism (from the popular media/lay folk) of the study that is the subject of this story. The study that asks each of 63 people for the telephone numbers of 4 different friends or family members and then randomly calls those numbers and has the friend/family member call the person back. Before the call is answered, the person is asked who is calling. You would expect that the correct rate would be 25%. However, it was 45%.

Telephone telepathy - I was just thinking about you

Each person in the trials was asked to give researchers names and phone numbers of four relatives or friends. These were then called at random and told to ring the subject who had to identify the caller before answering the phone.

"The hit rate was 45 percent, well above the 25 percent you would have expected," he told the annual meeting of the British Association for the Advancement of Science. "The odds against this being a chance effect are 1,000 billion to one."

People who seem to have a problem with this study are saying that the sample size was too small. This criticism didn't make any sense to me. This wasn't a poll. They weren't trying to use a small number of people to estimate some statistic of a large number of people. This wasn't about estimation at all. Thus, I think 63 is not only sufficient, but actually really BIG. It's like the researcher says, the odds against this being chance effect are 1,000 billion to one.

Unfortunately, the criticism about sampling effect is even referenced in the article by the Reuters reporter who wrote it.

However, his sample was small on both trials -- just 63 people for the controlled telephone experiment and 50 for the email -- and only four subjects were actually filmed in the phone study and five in the email, prompting some skepticism.

This obsession with sampling size fascinates me. Was I overlooking something? You could certainly do the Bernoulli trials and calculate exactly what the probability of this being pure chance was, couldn't you? In fact, didn't the researcher do that specifically? That's what he was referring to in his quote, right? What was I missing?

So I e-mailed an ecology professor I know who does some really impressive things with statistics in his data. He's very humble about his capabilities, but we all know better. His response? (note: I think I told him 64 people instead of 63, if you want to check these numbers yourself)

An exact probability of 3.4x10-14. That's more unlikely than encountering Moses on The Oval. My criticism: the sample size was ostentatious.
I'll try to remember this example. Good for those teachable moments.

Yes, exactly! (note, "The Oval" is much like "The Quad" at other universities)

Now, I'm not quite sure exactly what he means by the sample size being ostentatious. That is, could this sample size be TOO large? Could that be causing problems? Or does he just think the researchers are being obnoxious and including far too much data?

Regardless, anyone saying that the sample size is too small needs to really think through what that would mean. Why is 63 too small in THIS case? What exactly is being "sampled?" If 1000 people are enough to poll the opinion of a few million people (or much more, actually), then how many people do you need to say something conclusive about the abilities of humans in general? (there are near 7 billion of them in existence now, but that number could change greatly with time)

It's not about sample size. 63 people is plenty.

If you're looking for something to criticize, look elsewhere.

Technorati Tags: telephone telepathy, large sample size, small sample size, sample size, Bernoulli trials, statistics, probability, trials, news, research, study, studies, Reuters