Clearly Flawed Research Conclusions

[docB]

So, it's not EMS related but since we've been debating research a lot here lately I wanted to throw this one out in the interest of helping people look critically at research. In this case the research may not be bad but the conclusions drawn by the reporter are highly questionable.

http://www.foxnews.com/story/0,2933,316796,00.html

The conclusion drawn by the reporter is that all nighters make grades worse. What's a much more likely conclusion?

How would you design a study to really test if all nighters make grades worse?

---

Members don't see this ad.

 

[pseudoknot]

What's funny is that from the quotes in the article, I'd say the PI believes the same thing as the reporter.

I'm actually surprised that no one has ever done the experimental version of this, since people have been interested in sleep and performance for so long. I suppose it would be logistically difficult, but not impossible.

 

[jbar]

It's pretty well established that sleep helps memory consolidation and the recall of info, so given that it would be highly strange if all nighters were anything but neutral or harmful to ones GPA. But here are some problems with this study.

1: Small sample size

2: Limited population, only one college

3: Looked at overall GPA, not grade in the class for which they pulled an all nighter.

4: Didn't look at why people pulled the all nighter, those that do may be less prepared, have skipped classes, are farther behind because of learning disabilities etc all of which would predict a lower GPA.

5. Also sample is limited in that it only has psych 101 students, who may be 1: Younger
(most seniors aren't taking entry level psych classes), 2: a select population (psych students may differ from those in other classes)

6: Didn't look at one all nighter history v number of all nighters pulled.

I think it may be somewhat defensible to say "at one school, students in an intro psych class who had a history of at least one all nighter were found to have a slightly lower GPA. It is unclear if all nighters lower GPA, or are a marker for another variable. Further reasearch is needed."

To "prove" that all nighters are bad you would have to set up a large, randomized trial in which people were forced to either do an all nighter or were not allowed to. It would be very difficult to actually do this, and control for previous preparation, etc. The next closest thing would be to have people study for a certain number of hours overnight v the same number of hours during a normal day and take a test after the same elapsed time. I think this research has been done.

 

[docB]

4: Didn't look at why people pulled the all nighter, those that do may be less prepared, have skipped classes, are farther behind because of learning disabilities etc all of which would predict a lower GPA.

5. Also sample is limited in that it only has psych 101 students, who may be 1: Younger
(most seniors aren't taking entry level psych classes), 2: a select population (psych students may differ from those in other classes)
...
I think it may be somewhat defensible to say "at one school, students in an intro psych class who had a history of at least one all nighter were found to have a slightly lower GPA. It is unclear if all nighters lower GPA, or are a marker for another variable. Further reasearch is needed."

To "prove" that all nighters are bad you would have to set up a large, randomized trial in which people were forced to either do an all nighter or were not allowed to. It would be very difficult to actually do this, and control for previous preparation, etc. The next closest thing would be to have people study for a certain number of hours overnight v the same number of hours during a normal day and take a test after the same elapsed time. I think this research has been done.

Excellent points. These are all true. This is an example of selection bias. I think that #4 and #5 point to the main problem which is that the people who pulled all nighters and got lower grades may just be worse students. It's reasonable to think that people who are less prepared would feel compelled to pull an all nighter and that those people would do worse than their colleagues.

To do the study you need to pick students with equal GPAs and then have them do all nighters or not. Those results would say more about the damage or help done by all nighters.

This is important for EMS. We're looking at all these studies that claim to say that patient outcomes are worse for EMS than private car, or that EMS intubation doesn't help. You can't just believe what the researchers say. They can be wrong. In many cases it just turns out that the EMS patients were sicker to begin with or had some other factor stacked against them. In some cases that's not true. But we have to look at these studies critically.