So the NATURE commentary from Joe Bast got Stan Young going on one of his favorite topics.
I told you in the most recent post on the Beisner discussion that I didn’t want to get Statistician genius Stan Young going so late at night, well he was already going on the NATURE article about p values.
So he also got to talk about the strength of randomized and controlled studies and why observational studies are unreliable.
Stan Young commented on Nature Articles
Joe Bast Sent me this summary of Nature articles with his brief comments. I liked. …
‘ “Statistical Errors,” by Regina Nuzzo, in the February 13th issue of NATURE, is a fascinating article about the value of P-values. “P values, the ‘gold standard’ of statistical validity, are not as reliable as many scientists assume.” ‘
There is new research in the area of reliability of p-values. The new research indicates they are much less repeatable/reliable than previously thought. STILL, they are relatively reliable so long as the researcher
a. posts their analysis protocol before examination of the data (in pool, call your shot)
b. there is only one question OR the analysis corrects for asking multiple questions.
c. The protocol explains exactly how the analysis will be adjusted AND does not allow the analysts flexibility in the adjustment process.
Most of the non-reliability of reported p-values comes from how the data is treated before analysis, multiple testing (point b above) and multiple modeling (point c).
Simple rule of thumb: Randomized clinical trials are relatively reliable. Observational studies, as currently conducted, are so unreliable as to be essentially worthless.
See how much I have learned from Stan, who is at the National Inst for Statistical Sciences in the Research Triangle NC.