Originally Posted by
callee
One of the problems, Will, is that much of what you present appears to be conclusion driven, i.e. they have a conclusion and then set out to marshal any argument they can find that supports that conclusion.
But it is nevertheless the antithesis of the scientific method that forms the basis of western knowledge, and in that way it is nothing but an anti-intellectual power grab.
Take the snipe above about peer-review. Why on earth would they attack the process of peer review? As someone who has published peer reviewed articles, I can tell you, the process is essential to the scientific method.
The process isn't perfect, but it generally works, and it's the best way we have to accomplish what is obviously a necessary function.
So given all that, why oh why would someone try to sniper the peer-review process?
It seems transparent that the only reason they would do so is polemic.
They disagree with the conclusions of a certain body of scientific work. Therefore they would like to discredit that work [Note: That much is indeed the purpose of some peer review -- whether for good or for ill.]- not because the work is sound or un-sound, but simply because they dislike the conclusion. But then they are told they cannot discredit that work, because it is already peer-reviewed. What to do then? Aha! That's it! Just discredit the peer-review process! Expose that hokus pokus for what it is! After that, they'll have a free and clear road to discrediting the work, and then dismissing the conclusion that they started out not liking.
That may make good politics, but it is bad science, and I think that's what Kevin might have been getting at.
I remind people that they are not compelled to read the entire post -- or to follow the links unless they have an interest in that type of content...
Ryan/Callee:
I was so stunned by that remark that it took me a while to formulate a response -- which I will try to keep simple so that other people if they wish can follow three links and see what is being discussed -- one of them will be a real-time, live "peer review" -- of sorts.
I underlined and bolded the portion we seem to agree on...
First -=- a short history of peer review is here. It agrees with my contention that formal peer review is a relatively recent practice:
http://michaelnielsen.org/blog/three...c-peer-review/
Note: The use of the word Myth is not mine -- but it could be difficult to disagree.
Myth number 1: Scientists have always used peer review
The myth that scientists adopted peer review broadly and early in the history of science is surprisingly widely believed, despite being false. It’s true that peer review has been used for a long time – a process recognizably similar to the modern system was in use as early as 1731, in the Royal Society of Edinburgh’s Medical Essays and Observations (
ref). But in most scientific journals, peer review wasn’t routine until the middle of the twentieth century, a fact documented in historical papers by
Burnham,
Kronick, and
Spier.
...
How many of Einstein’s 300 plus papers were peer reviewed? According to the physicist and historian of science
Daniel Kennefick, it may well be that only a single paper of Einstein’s was ever subject to peer review. That was a paper about gravitational waves, jointly authored with Nathan Rosen, and submitted to the journal Physical Review in 1936. The Physical Review had at that time recently introduced a peer review system. It wasn’t always used, but when the editor wanted a second opinion on a submission, he would send it out for review. The Einstein-Rosen paper was sent out for review, and came back with a (correct, as it turned out) negative report. Einstein’s indignant reply to the editor is amusing to modern scientific sensibilities, and suggests someone quite unfamiliar with peer review:
Myth number 2: peer review is reliable
Update: Bill Hooker has pointed out that I’m using a very strong sense of “reliable” in this section, holding peer review to the standard that it nearly always picks up errors, is a very accurate gauge of quality, and rarely suppresses innovation. If you adopt a more relaxed notion of reliability, as many but not all scientists and members of the general public do, then I’d certainly back off describing this as a myth. As an approximate filter that eliminates or improves many papers, peer review may indeed function well.
Every scientist has a story (or ten) about how they were poorly treated by peer review – the important paper that was unfairly rejected, or the silly editor who ignored their sage advice as a referee. Despite this, many strongly presume that the system works “pretty well”, overall.
There’s not much systematic evidence for that presumption. In 2002 Jefferson et al (ref) surveyed published studies of biomedical peer review. After an extensive search, they found just 19 studies which made some attempt to eliminate obvious confounding factors. Of those, just two addressed the impact of peer review on quality, and just one addressed the impact of peer review on validity; most of the rest of the studies were concerned with questions like the effect of double-blind reviewing. Furthermore, for the three studies that addressed quality and validity, Jefferson et al concluded that there were other problems with the studies which meant the results were of limited general interest; as they put it, “Editorial peer review, although widely used, is largely untested and its effects are uncertain”.
- George Zweig’s paper announcing the discovery of quarks, one of the fundamental building blocks of matter, was rejected by Physical Review Letters. It was eventually issued as a CERN report.
- Berson and Yalow’s work on radioimmunoassay, which led to a Nobel Prize, was rejected by both Science and the Journal of Clinical Investigation. It was eventually published in the Journal of Clinical Investigation.
- Krebs’ work on the citric acid cycle, which led to a Nobel Prize, was rejected by Nature. It was published in Experientia.
- Wiesner’s paper introducing quantum cryptography was initially rejected, finally appearing well over a decade after it was written.
Myth: Peer review is the way we determine what’s right and wrong in science
By now, it should be clear that the peer review system must play only a partial role in determing what scientists think of as established science. There’s no sign, for example, that the lack of peer review in the 19th and early 20th century meant that scientists then were more confused than now about what results should be regarded as well established, and what should not. Nor does it appear that the unreliability of the peer review process leaves us in any great danger of collectively coming to believe, over the long run, things that are false.
In practice, of course, nearly all scientists understand that peer review is only part of a much more complex process by which we evaluate and refine scientific knowledge, gradually coming to (provisionally) accept some findings as well established, and discarding the rest. So, in that sense, this third myth isn’t one that’s widely believed within the scientific community. But in many scientists’ shorthand accounts of how science progresses, peer review is given a falsely exaggerated role, and this is reflected in the understanding many people in the general public have of how science works. Many times I’ve had non-scientists mention to me that a paper has been “peer-reviewed!”, as though that somehow establishes that it is correct, or high quality. I’ve encountered this, for example, in some very good journalists, and it’s a concern, for peer review is only a small part of a much more complex and much more reliable system by which we determine what scientific discoveries are worth taking further, and what should be discarded.
The entire article is much longer -- but it summarizes the main points and much of it matches with my experience. I could write a book on that...
For a live peer review process -- follow this link.
http://climateaudit.org/2012/06/17/s...-lot-of-noise/
There has been a great deal of discussion on a recent CA thread on the efficacy of screening proxies for use in reconstructions by selecting on the size of the correlation between the proxy and the temperature during the calibration time period. During the discussion I asked Nick Stokes the following questions in a comment:Do you think that it would be appropriate to use correlation to screen tree rings in a particular site or region when doing a temperature reconstruction for that site? Would you not be concerned that the process could bias the result even if the trees did contain actual information about the temperature?
The stats may be well above most people's level, However, some of the "English" comments summarize the points rather neatly...
On Bishop Hill -- somebody made this comment regarding an article which has since been pulled...
http://bishophill.squarespace.com/bl...last-post.html
I think this explanation of their "science" sums up all that is wrong:
"Iʻm talking, yes, about the scientific method. Simply put, to have a valid argument, a scientist will hypothesize about something, letʻs say the temperature of a certain place over a specific period of time. He or she will then record temperatures for a specific length of time then write a paper and send it in to a journal publication-for example Science or Nature. Before it gets published, various other scientists read it to assess the validity of the argument. If none can falsify it or find no major inconsistencies, the paper gets published and then is used as the basis for other scientific hypotheses. So, these peer reviewed articles are what constitute the scientific fact."
In other words, if no "scientist" publishes a paper that says the world has not warmed - if no one questions warming because they don't want to rock the boat ... if for whatever reason the evidence contradicts the hypothesis or the models ... but no one publishes .... if no one sees it in their career interests to suggest the models are carp, .... then according to this absurd notion which wouldn't wash in an arts subject like literature criticism and is about as far from science as you can possibly get, .... despite the real fact that none of the models predicted the climate and the climate has not warmed in this bubble of madness it is a "scientific fact" that the world is warming and the models are all correct ... because none of them have published a paper saying otherwise.
Mike Haselers comments are in normal type -- the original authors contention is in Italics.
But again -- it's an interesting summary of the issues.
Peer review has been with us a long time in some forms -- the Hemlock of Socrates -- the banning of Galileo -- the burning of Jean D'Arc are some of the extreme examples -- where peer review was effective -- and final.
http://en.wikipedia.org/wiki/Trial_of_Socrates
The trial of Socrates and the subsequent execution of that classical Athenian philosopher took place in 399 BC. Socrates was tried on two notoriously ambiguous charges: corrupting the youth and impiety (in Greek, asebeia). More specifically, Socrates' accusers cited two "impious" acts: "failing to acknowledge the gods that the city acknowledges" and "introducing new deities.
But it was never an essential part of the scientific process, Archimedes and Aristotle might vouch of that -- not to mention Newton and Einstein.
It has become part of the publishing process -- but that is entirely another issue.
Facebook
Junk Science Week...
.jpg "Canadian Woodworking and Home Improvement Forum")
Reply With Quote
