the Air Vent

Because the world needs another opinion

Why Replicate

Posted by Jeff Id on May 19, 2009

An article on the pitfalls of ‘peer review’ was linked to here thanks to Adam Gallon.

The scientific fraudster who dazzled the world of physics

The article is quite interesting. There were several pertinent points which have AGW corrilaries.

Schön’s fraud was the largest ever exposed in physics; he ended up without a job, and was forced to leave America in disgrace. But the ease with which his fraudulent findings and grotesque errors were accepted by his peers raises troubling questions about the way in which scientists assess each other’s work, and whether there might be other such cases out there.

The ease of which the papers were accepted by the club is understandable. Once you have a reputation it’s easy to be accepted or rejected by your peers. McIntyre’s latest attempt at publication of the same math Santer used on the same datasets is a good example of the opposite effect. The name and opposite conclusion preventing publication of the exact same calculations.

On one occasion, after a trip to Constance, Schön showed colleagues in a neighbouring lab at Bell an astonishing graph. It appeared to show the output of an oscillating circuit. He claimed he had built the circuit in Constance using crystals from Bell Labs. The result were fantastic – but not impossible – and admiring colleagues helped Schön prepare a manuscript reporting the new claims.

They sent that manuscript, with more than a dozen others over the next year and a half, to the journal Science. Along with its British competitor Nature, Science was – and is – the most prestigious place to publish research. This was not only because of its circulation (about 130,000, compared with 3,700 for a more niche publication such as Physical Review Letters), but also because, like Nature, it has a well-organised media operation that can catapult the editor’s favourite papers into the pages of national newspapers.

This next paragraph is very important – my bold.

By the middle of 2001, more than a dozen laboratories around the world were trying to replicate Schön’s work on organic crystals, motivated by the prospect of building on the findings in Science and Nature. Yet without access to the details of his methods, which were strangely absent from the published reports, no one was successful

I don’t know how strange it is. In Climatology it’s become standard. After all the consensus doesn’t want to be questioned. Santer won’t release his data, Steig and Comiso have refused to release their code and methods, hell even the NOAA doesn’t release their data sources.

One of the most cherished beliefs of scientists is that their world is “self-correcting” – that bad or fraudulent results will be shown up by other experiments and that the truth will emerge. Yet this system relies, far more than is generally realised, on trust: it was natural for his peers to question the way Schön interpreted his data, but taboo to question his integrity.

Self correction only comes from replication. Without the data required, replication is a nearly impossible task. In the case of the physics world a lot of money was riding on replication, after all a molecule sized transistor can make you a mountain sized pile of money. In Climatology, there is no positive product, no new invention which helps people instead there is only a negative one – taxation and regulation. Therefore there are no labs attempting to disprove, only government funded work to increase the knowledge and funding only goes in one direction.

This is the mechanism which has created a consensus, those who disprove claims are not welcome to the discussion, they are not wanted no matter how reasonable their argument. Consider Steve McIntyre’s latest attempt which uses the same math and techniques as Santer yet is meeting resistance to publication for no reason other than it disproves the accuracy of computer models using the same math and data Santer used to prove the accuracy. The only change is to update the data series, a series Santer et al had to chop off in order to make its conclusions.

The question scientists should be asking is why was the data chopped in the first place and then if the extended data reverses the conclusion was there motive involved. Santer is promoting some highly political ideals in the video linked at CA HERE. In the consensus of the government funded environment, this is not openly questioned.

Schön was, in effect, doing science backwards: working out what his conclusions should be, and then using his computer to produce the appropriate graphs. The samples that littered his workspace were, effectively, props. The data he produced were not only faked, but recycled from previous fakeries (indeed, it was this duplication of favoured graphs that would prove his Achilles’ heel).

This is exactly what Mann does. He knows the conclusion and his papers (the two I’ve replicated) work very hard to reach those conclusions using blatantly flawed math. I fully believe that Mann did this with intent in his 08 paper and am highly suspicious (quietly) of the Antarctic temperature paper. The strength of my words are not lightly considered.

But Schön’s case also suggests a recipe for judging any science headline: any time a major breakthrough is reported without the researcher in question showing details of how they carried out the experiment, it’s time to start asking questions. If something seems too good to be true, it very possibly is.

This last paragraph is a weak end to an otherwise excellent article. The questions have been asked for centuries, it has been time for a while to ask the questions of climate science. So far I’ve read dozens of papers and worked on calculations for several. Of the three which caught my attention in particular two had massive flaws in their calculations leading to obviously erroneous conclusions ( the 08 Hockey Stick and the Antarctic). The third was a paper by John Christy on RSS vs UAH and I reached his same conclusion using GISS data to correct the satellite data before I read the paper.

The realization I came to some time ago, was that CA provides a huge service to the world community in exposing the flaws of climate science. SteveM can’t do it all by himself and it’s important for people to realize the magnitude and type of flaws are critical to the base assumptions of anthropogenic global warming. This has been a substantial part of my own drive to dig into some of these papers. It takes hours of time and a lot of study but replication can and must be done.


16 Responses to “Why Replicate”

  1. Phillip Bratby said

    What you have said about Mann, equally applies to the IPCC. Write the conclusions (SPM) and then shoehorn the science to fit.

  2. David Jay said

    Phillip:

    But without a meta-narrative, how can climatology properly converge all of the science?

    welcome to post-modern science…

  3. Page48 said

    The unscientific behavior described in the article started a long time ago.

    Back in the eighties, when I still took a newspaper, I began to notice an increasing number of scientific press releases containing conditionals in the titles (x, y, or z might cause, could cause, may have found the answer to such and such, etc.).

    I called this type of press release “Grant Money Propaganda.”

    It worked. The only change has been that scientists have figured out that the conditionals are unnecessary. Now, they just make a pronouncement and manipulate (or invent) data to fit the conclusions, and the grant money flows.

    Worse, the scientists have discovered power.

    Keep tearing up those graphs, Jeff (and Steve). This scientific community is never going to police itself on the climate issue. There’s too much money and power at stake, now.

  4. Andy said

    We know peer review does not work.

  5. Terry said

    Great article – thanks Jeff

  6. I thought this might be of interest to the denizens of this “world”!

  7. Page48 said

    If you enjoyed reading about the scientific fraud in this post you will also like reading about the Sokal Affair, a deliberate hoax.

    Sokal stated that the purpose of his hoax was to “publish an article liberally salted with nonsense if (a) it sounded good and (b) it flattered the editors’ ideological preconceptions.” He succeeded.

    Here’s the link:

    http://en.wikipedia.org/wiki/Sokal_affair

  8. Ryan O said

    #4 I am still not quite that jaded on peer review. The purpose of peer review is not to expose fraud because the entering argument was that the research was done in good faith. It is too much to ask peer reviewers to become detectives; that would be stifling. The point of peer review is to find obvious errors and/or flaws in logic.

    Replication is the detective work. Because it is so time-consuming, it usually must be done after the fact. The very existence of the linked article shows that this process works. The fraud was uncovered.

    I think many people want all of this to happen in peer review, and as nice as that might be, I am convinced that it would be impractical. Not only that, it could potentially serve to even more effectively censor views that run counter to the prevailing opinion.

    In my opinion, peer review needs some shoring up (like demanding disclosure of methods and data as used), but is fundamentally a sound idea.

    It is the replication side of things that will catch deceit and fraud, not peer review.

  9. omnologos said

    There are many things that need sorting out in scientific publishing, apart from fraud. One is having journals invariably trust their “peer reviewers”, instead of censoring articles with good peer review but alas not aligned to the journal’s editorial stance (or slant).

    Another is “citation amnesia“, something the Lancet and UCL have just committed in their document on global warming and public health.

  10. Page48 said

    RE: #8

    I don’t think that “peer review,” as it was originally designed to function, is the problem. The process was never intended to completely validate a paper, but scientists, whether they have intended to or not, have convinced the public that “peer review” does just that.

  11. Peer review is the first line of filtering, not the last (which is probably replication). When I’m reviewing, my job is to weed out the obvious rubbish and to give an honest opinion on the remainder. If I recommend a paper for publication, that only means I think the content of the paper is going to be interesting to the wider research community; it doesn’t mean I believe the thesis or the results of the paper cannot be questioned.

    I usually spend more time on papers that I recommend be rejected. In the first place, I have to be sure I have understood the authors. In the second place, I have an obligation to state clearly why I think the work is not suitable for publication. I don’t think I’ve ever used the space on the review forms for private communication with the editors — everything I write in a review should be available to the authors.

    What I would like to see is for journals and conferences to allow rejected papers and their comments to be placed on-line in those cases where the authors are willing and they dispute the reviews. This, at least, would allow the target audience the opportunity to offer their opinion on the review comments when controversial papers are rejected.

  12. Brooks Hurd said

    Ryan O,
    I agree with you that it is too much to expect peer reviewers to take the time to replicate the work they are reviewing. However, I do not believe that it is expecting too much for at least some peer reviewers to ask the author(s) where they archived the data they used for the article.

  13. Willis Eschenbach said

    Ryan O., thanks for a good post, and the outstanding work on the Steig paper. However, unlike you, I’m totally jaded on peer review. Why? Well … it doesn’t work. Two things to help:

    1. Don’t let the reviewers know the name of the author until the paper is either accepted or rejected.

    2. Publish the names of the reviewers, and their reviews, if it is accepted. If they have put their seal on it, we deserve to know who they are. Anonymous accusations went out with the Star Chamber, only “science” clings to this discredited model.

    It is precisely the lack of anonymity on one side, and the anonymity on the other, that makes peer review totally useless. It allows entrenched interests to discriminate against new ideas with no accountability.

    And what is worse, it does not even do what you claim, “find obvious errors and/or flaws in logic”. If it did, we would not be treated to endless stories in Science and Nature that are clearly and blatantly false.

    I do not understand those who, like you, defend a system which has failed so obviously and so badly. I implore you to quit claiming that it is doing the job it was designed to do. In climate science at least, it is NOT DOING THAT JOB.

    The Steig paper in Nature is a perfect example. I looked at the graphic on the cover and though “Bullshit, no way nature is that smooth and uniform.” So no, peer review isn’t working for shit.

    w.

  14. Pete Ridley said

    Since November 2008 I have been trying to engage in open-minded debate about human-made climate change on several blogs, mainly those of Jonathan Porritt and Mark Lynas (both devout environmentalists). I have repeatedly challenged other contributors to identify any flaws in Dr. John Nicols’s paper “Climate Change (a fundamental analysis of the greenhouse effect) which supports the findings of Dr. Heinz Hug and John Barrett but with much more detail. All that I get back is the usual dogma, including “not peer reviewed” with no attempt to challenge what Dr. Nichol says. I understand that Dr. Nicol has unsuccessfully invited climate specialists to review his paper, including Professor Andy Pitman and Professor Michael Ashley. Can anyone point me towards a scientific analysis identifying any flaws in Dr. Nicol’s paper, or are there no such papers because there are no flaws? If there are no flaws then the IPCC position is untenable.
    For anyone interested, the related debates I’m currently engaged in can be seen at http://www.marklynas.org/2009/5/5/climate-change-explained-the-impact-of-temperature-rises and at http://www.marklynas.org/2009/1/23/a-new-green-era-is-already-unfolding.

    Regards, Pete Ridley, Human-made Climate Change Agnopstic

    PS: Philip Bratby please come back and add some of your scientific understanding to the debates. Pete R

  15. Just to add a little more here, another item from The Daily Telegraph.

    http://www.telegraph.co.uk/scienceandtechnology/science/sciencenews/5438844/Scientists-faking-results-and-omitting-unwanted-findings-in-research.html

    “More than two-thirds of researchers said they knew of colleagues who had committed “questionable” practices and one in seven said that included inventing findings.”

    “On average, across the surveys, around two per cent of scientists admitted they had “fabricated” (made up), “falsified” or “altered” data to “improve the outcome” at least once.

    A further 34 per cent admitted to other questionable research practices including “failing to present data that contradict one’s own previous research” and “dropping observations or data points from analyses based on a gut feeling that they were inaccurate.”

    Article published in “Public Library of Science” (A peer-revued journal!)

    http://www.plosone.org/article/info:doi/10.1371/journal.pone.0005738

    (Full text article)

    “Considering that these surveys ask sensitive questions and have other limitations, it appears likely that this is a conservative estimate of the true prevalence of scientific misconduct.”

    “A popular view propagated by the media [5] and by many scientists (e.g. [6]) sees fraudsters as just a “few bad apples” [7]. This pristine image of science is based on the theory that the scientific community is guided by norms including disinterestedness and organized scepticism, which are incompatible with misconduct [8], [9]. Increasing evidence, however, suggests that known frauds are just the “tip of the iceberg”, and that many cases are never discovered.”

    “Scientific results can be distorted in several ways, which can often be very subtle and/or elude researchers’ conscious control. Data, for example, can be “cooked” (a process which mathematician Charles Babbage in 1830 defined as “an art of various forms, the object of which is to give to ordinary observations the appearance and character of those of the highest degree of accuracy”[12]); it can be “mined” to find a statistically significant relationship that is then presented as the original target of the study; it can be selectively published only when it supports one’s expectations;”

  16. Pete Ridley said

    Dear readers, this is the lead item in Australian Senator Steve Fielding’s blog (Note 1) which I think all of our politicians and environmentalists should see. I also invite you to read the comments on Senator Fielding’s blog and on his blog “Assessment of Pennny Wong’s Response .. ” (Note 2)

    QUOTE:
    Climate change is real. Yes that’s right, contrary to the misreporting in the media, I do believe in climate change. That might come as a shock to some of those on the left side of politics, but it’s the truth. The question that concerns me, however, is what is driving it? Is it increasing levels of human made carbon dioxide emissions, variations in solar radiation or something else?

    Around three months ago one of my advisors pulled me aside and asked me what I thought was driving climate change. I smiled and said automatically that it was obviously a result of increasing carbon dioxide emissions. I had never really looked at the science and just assumed what was reported in the media to be true. Well wasn’t I in for an enormous shock. My advisor presented me with data and some comments from a number of scientists which suddenly had me asking many questions. This led me to do some further reading and I ultimately decided to head over to Washington on a self funded trip so I could find out more about the science behind climate change.

    In the US I met with numerous scientists on both sides of the debate. Some media outlets would have you believe that I met only with climate skeptics who they accuse of being paid off by the fossil fuel industry. These claims are wholly inaccurate. Moreover, I strongly believe in giving everyone a fair hearing even if it isn’t the most popular view. I believe it’s my role as a a politician, to wade through all of the spin and come up with my own conclusions after hearing all of the facts. Some of the data led me to question whether the Rudd government had got the science right. I then took some of the information and questions I had to the White House where I met with one of President Obama’s senior climate change advisors. While these discussions were fruitful, I was left at the end with even more questions than when I had started.

    In an effort to try to get to the bottom of the issue I started to talk to a number of scientists based in Australia to get a feel for what their views were on the subject. Amongst the many presentations, one item really stood out. I was presented with a graph based on data that IPCC use which showed carbon dioxide emissions sky rocketing over the last 15 years while global temperatures had remained steady. This graph left me nothing short of flabbergasted. Up until this point I had truly believed that human made carbon dioxide emissions were responsible for climate change. However, this graph basically said otherwise. I was left asking myself how I could vote for a carbon pollution reduction scheme if it appeared as though carbon dioxide emissions were not driving climate change. It is important to point out that the IPCC had predicted in their models that there would be a direct correlation between increasing carbon dioxide emissions and increasing global temperatures. However, if you look at the graph it is obvious to everyone that this correlation simply does not exist.

    Armed with this information I sat down with Minister Wong, the Chief Scientist and Professor Will Steffen of the ANU to hear their explanation. After an hour and a half I left none the wiser. I received a written response to my questions from the Minister a few days later which had me even more uncertain. According to the Minister, air temperature, a measurement relied upon by the IPCC and the Rudd Government to justify its emissions trading scheme was irrelevant. Instead, I was told that I should really be concerned with the variability in ocean temperatures. Not only did this contradict all of the information which the Minister had provided me with only a few days earlier but I was also aware of an IPCC report which stated that the measuring of ocean temperatures was not reliable.

    I went back to the government with this question but was met with a wall of silence. They had clearly decided it was safer not to engage with me because I had legitimate questions which they probably were unable to answer. I was left feeling that the only responsible thing to do was to vote against this legislation. At the end of the day, it would be a betrayal of my duty to the Australian people to put at risk the national economy and many thousands of jobs on what is clearly inconclusive science.

    But then enter Al Gore. Here was a man who had a lot of power and went around the world preaching about climate change. I thought he might have the answer for me, the ones I couldn’t extract from the Rudd government. I briefly met Mr Gore at a breakfast in Melbourne attended by more than a thousand people. He was aware of the important role Family First plays in the senate and was keen to catch up. After a series of phone calls I was met with a stonewall of resistance. I offered to meet Mr Gore at any place at any time but had no luck. Here we had the former Vice President of the United States, a self proclaimed climate change preacher running away from me over a few simple questions. I could hardly believe it. I would have thought if Al Gore was really committed to the cause he would want to meet with all senators who had concerns about the science if it would help ensure that the CPRS legislation would pass. Obviously I was wrong.

    I have written to every senator urging them to look at the graph and ask themselves the key question – what is driving climate change? If they can’t answer that simple question they shouldn’t be voting for a CPRS. This decision is the biggest economic decision in this country’s history, one which is too important to vote along party lines. I call on the government to answer my question with a straight answer. If they’re not prepared to do so, I’m happy to fight the lone battle in the senate until those senators who are honest with themselves break party lines.
    UNQUOTE

    NOTES
    1) see

    http://www.stevefielding.com.au/blog/comments/the_real_reason_ill_fight_in_the_senate_on_climate_change/#comments

    2) see http://www.stevefielding.com.au/blog/comments/assessment_of_penny_wongs_response_to_my_3_questions_on_climate_change/#comments

    Regards, Pete Ridley, Human-made Global Climate Change Agnostic

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 148 other followers

%d bloggers like this: