How to Make a Hockey Stick – Paleoclimatology (What they don’t want you to know)

I just learned something today which has left a sick feeling in my stomach.  I learned how the latest Mann et al hockey stick graph was created.  So you don’t have to read the paper HERE, I’m going to summarize the result.

This time Mann avoided all the tree ring reconstructions from the old hockey stick because of a problem with what is termed Divergence.  Divergence means the data don’t correlate well in recent times (last 130 years).  Scientists know that a high percentage of trees “temperature” data don’t fit well when compared to the warming trend of earth – measured data.   To sort the data, they correlate the data, which is very noisy and doesn’t look like a temperature measurement at all by measuring its statistical significance to the 130 year measured temperature record.  Data which doesn’t match to a minimal extent with current temperatures is rejected.  The remaining data sets are averaged together (often using very strange techniques) to create a final signal.

As an exercise assume you start with 1000 sets of random very noisy set of data which swings up and down by 4 degrees C and you average them.  You should get a relatively flat line with wiggles of a magnitude much smaller than any of the individual peaks.

If you take the same random data, calibrate its endpoint to today’s temperature (offset it so the end matches today’s temperature) and then sort it (throw data out) so that only data which correlate to a temperature rise at the end 5% of the dataset remains. Then you average the remaining data you would get a relatively flat line with an upward spike at the end .  The averaged data would have an end spike which would almost certainly be of greater magnitude than the rest of the curve.

I had no idea that is what these guys are doing.  I am absolutely pissed off at this kind of junk science. This wouldn’t pass the smell test in any field except climatology and it only passes here because there is motive.

Ok Deep breath….

So recently, because so many trees don’t meet the temperature trend, Mann and his buddies took some of the other very very noisy data from sediments, pollen and a dozen other reconstruction techniques and ran the same process.  Again, he has created the “hockey stick” using different data, no surprise really.

So now suppose that you had a noisy temperature signal in the data, you have filtered it all for a recent upward trend which is clear, now what would happen to the noisy but real temperature signal in the past.  It would of course be muted by the average of the noise dampening its actual magnitude!!  This is pretty basic math for a scientist so it’s not to difficult to figure out.

I had to point out this problem to Real Climate where they are currently in a circular celebration of this huge victory where they are saying things like the best climate reconstruction ever, now we have proof, na na na  and the like.

So I wrote my own post.

To Real Climate.org HERE.  Although it was a perfect fit for the thread, it has not been shown. Apparently criticism of the conclusions is unacceptable.  I don’t have the original post but it went something like this. —

I explained that if you took a random data set and averaged it as above it would create a strong correlation to an upward trend with a flat line behind it for the reasons above. I then stated that an average of the entire data sets would be more useful in determining the magnitude of present data as compared to history.  Statistically any deviation from the zero of actual temperature would be reduced by averaging with the rest of the data.

I then explained that I could see no basis for elimination of any of the data sets by correlation to measured temperature, as it would only serve to create an artificially strong spike at the end of otherwise flattened data and that for a better trend analysis, all the data should be used.  I could see no scientific basis for the elimination of data by the single criteria that they don’t match current trends other than the intent to make a temperature increase stand out.

I finished by saying, one thing is certain, we cannot make the irresponsible claim from this reconstruction that past temperatures were lower than present.  Past temperatures would be muted in the signal and therefore a high or low temperature would have to be substantially greater than today to show as an equal value.

Clearly they don’t want people to know.

I re-posted asking for an explanation of the deletion, so far no response.

As an addendum, just so people understand,  noisier data, more random and less accurate data, will produce a better hockey stick than data with an actual signal with this method.  They would almost want bad data to prove global warming was caused by man.  We might as well roll dice and run it through the algorithm.   I think I might just do that to prove the point!

This is much worse than I realized.

49 thoughts on “How to Make a Hockey Stick – Paleoclimatology (What they don’t want you to know)

  1. If you feel so strongly that the methodology used by Mann et al is fundamentally flawed, then I recommend formally authoring a letter to PNAS claiming as such. Science is not a one-way street; you are more than welcome to run your “dice roll” experiment, analyze it, and write it up.

    Furthermore, I wouldn’t be too upset if RC deleted your comment. I’ve had my share of tangential comments purged as well.

  2. Thanks for the reply.

    My comment was directly pertinent to the thread. It was the only critical comment to the thread I noticed and it addressed the obvious flaws in the methodology.

    I have no idea why a ‘scientist’ would scrap any piece of data which doesn’t support your conclusion! Well except for one.

    I may do that analysis, I didn’t really want to become a climatologist though.

  3. Count me in as another who had a perfectly good post not accepted at realclimate – and no, no rudeness of any kind, it was on-topic and thought-out. I never bothered again.

  4. Apparently, you are new to RC. Censure, er, data filtering, has been their M.O. since day-1.

    Scuttlebutt has it that RC was started by Mann, et al, specifically to rebut McIntyre & McKitrick’s thorough deconstruction of the original ’98 hockey stick, but I have no evidence for that.

  5. The above post is beyond silly, a perfect display of Christian fundamentalist anti-scientific style reasoning. You write these posts imagining that you are criticizing the conclusions of science while what you are really doing is only demonstration your own wholesale rejection of science.

    John McCain accepts that Global Warming exist and is a problem. So why are you engaged in this crusade against the conclusions of science, Jeff? What is motivating this most futile effort on your own part?

    Barack Obama, Joe Biden, and John McCain agree about this matter. The only person in the presidential campaign who rejects Global Warming is the young earth creationist from oil-soaked Alaska.

    If RealClimate rejected your post, that’s just too bad. Have you paid any attention to what is happening on the planet? We’re having an extremely active and dangerous hurricane season. The Arctic sea ice is melting down to its 2007 lows. How much more evidence do you need to see in order to convince you of the existence of a real problem?

    Creationists reject evolution so absolutely that no amount of scientific evidence will change their mind. The Global Warming Denialists are very similar in their approach. What would it take to convince you that Global Warming is occurring and a threat to civilization?

  6. Wow David, that’s quite a leap in logic. It’s not surprising, however, since your post goes on to repeat the demonstrably incorrect orthodoxy of the RC crowd. Luckily, time is on the side of rational thinkers. The more chicken-littles like yourself are heard, the sooner most will realize what a sham you have created.

  7. Awesome straw man from David, which also managed to include an ad hominem attack on the host.

    Gee, you must really be walking away with your tail between your legs after such a well-thought out refuting of your argument Jeff.

    DMS

  8. David #6

    What planet do you come from?

    Obama, McCain and Biden are not scientiss. If they have an opinion it is a derivative of the spin that others put before them. This will be true of nearly every part of the platform proposed by both political parties and which is then promoted by the candidates.

    Jeff Id is displaying an enquiring mind, and looking past the propaganda into the underlying factual material. If this leads to a conclusion that is contrary to your own I see no reason why he should be castigated for it by folk like yourself.

    There is nothing unscentific in the post by Jeff. Science is about asking questions, examining data and questioning hypotheses. It is not added to by parroting the opinions of others as you seem to want to do David.

    The original hockeystick hypothsis was thoroughly destroyed by McIntyre and McKitrick becaue they dared to look at the data and methods. The new Michael Mann temp hockeystick already seems destined to go the same way. It will probably happen much more quickly this time because thousands of enquiring minds are now watching Mann’s every step.

    Finally I fail to see what Arctic sea-ice and creationism have to do with the post by Jeff. All-in-all your comments are off topic, irrelevant and lack civility.

  9. “John McCain accepts that Global Warming exist and is a problem. So why are you engaged in this crusade against the conclusions of science, Jeff? What is motivating this most futile effort on your own part?”

    Umm… because he’s doing science? Science involves examining data and arguments critically to see if you can find flaws. All the time.

    Unlike, say, religion, which often involves belief in a stated dogma which is promulgated by a central authority figure. This belief is not to be questioned, but must be accepted as an article of faith. This seems to be your position, so I’d guess you are of the Global Warming faith (praise be to Gore)? Tell me, do you knock on doors in your spare time and ask people if they have ‘Felt the Heat’?

    By the way, why do you place so much faith in American politicians as people to believe unquestioningly? They don’t seen to have done America much good recently…

  10. Thanks for the help. It is appreciated.

    I have been engrossed in trying to decipher the real process behind the statistical processes in Mann’s latest work. At first it seemed reasonably transparent but now it has become very difficult to see what really happened to the data. Can anyone with more experience than me describe the process Mann is using in his EIV approach.

  11. Jeff Id,
    Great work. Cherry-picking data to bias evidence toward the desired conclusion is classic junk science. Welcome to the spike club. Joe Romm spiked my challenge to his hurricane intensity post on climateprogress, which he moderates, but my posts are logged on http://luddhunter.wordpress.com/

    He did post 2 of 4 of my submissions, but refused to address my source which refutes his hurricane theory. Joe is a particularly prolific and high profile Ludd, and your insightful challenge and solid defense of your conclusions on http://gristmill.grist.org/story/2008/9/3/12753/54695
    shows a great combo of gray matter and cojones.

    Keep up the good work, the Ludds will vilify you as a heretic and blasphemer, but keep comin at ’em. Their fervent hyperventilations can’t cover for fraudulent science in the long run.

    Joey Panto
    http://www.joeypanto.com

  12. Please run the whole calculation your way and show us your graph of temperatures for the past 1700 years. Send it to PNAS or other journal and get it published. The entire scientific world is not in cahoots with Mann personally, as I am sure you agree (being apparently a thoughtful guy who understands some of science) but a lot of scientists were involved in getting the paper published. Can you not do the same and thus be arguing from the same platform? As others have said, if you don’t do this you are no better than those you say parrot Mann’s results uncritically. If your paper is rejected, publish the entire process, names and all, and we can discuss that openly. Thanks

  13. Gary, thanks for the comments.

    I am going to start on it this weekend. I don’t expect to publish it anywhere but here though. As I said earlier, I don’t want to be a climatologist. I am currently running a company and it takes a considerable amount of time. Also, the facts I outlined above are so clear and so simple, I know the paleoclimatologists have already been through this. I don’t know how it got to be accepted practice.

    I downloaded the dataset from Mann today, I am trying to decide if I should learn a new programming language like R because it has a nice statistical analysis set or if I should work in C++ which I am familiar with.

    One point I would like to make is that while I did beat up Mann, I have been reviewing a half dozen other paleoclimatology papers and this seems to be standard practice in the industry. I have even mentioned the practice on Climate Audit and seem to get limited response. Some of the guys there even think it’s ok practice. Makes me crazy to think we would throw away our oil economy for this kind of BS.

    I am thinking of doing a simple linear curve fit to the end of the data and accept only curves with a negative slope. I wonder what kind of average that would produce.

    Also, as I get the software built, I am considering fitting a strong medeival warming curve to the data to see if I can extract a high temperature rise.

  14. Hello Jeff,

    “As I said earlier, I don’t want to be a climatologist.”

    You don’t need to worry about tha, Jeff. You aren’t a climatologist and you aren’t a scientist.

    But you do love the gun-toting moose-eating young-earth creationist governor from Alaska, don’t you?

    Do you run a business? What sort of business?

    Perhaps you are a polluter and your crusade against Global Warming serves your financial self interests. Is that the case?

    Or you are just another Christian fundamentalist, treating the Earth like a sewer while you wait for God to rapture you to Heaven! If that is the case, Sarah Palin is your woman.

    “Makes me crazy to think we would throw away our oil economy for this kind of BS.”

    What a tragedy … imagine that, morbidly obese Americans losing their SUVs because of something so trifling as the health of the planet and the future survival of the primate plague which seems to hate the Earth and prefer life somewhere else … such as Heaven or conquering the Universe.

    “Also, as I get the software built, I am considering fitting a strong medeival warming curve to the data to see if I can extract a high temperature rise.”

    You are wasting your time, Jeff. You are not a climatologist, you are not a scientist, and you are in love with the creationist Vice President. As such, anything at all that you say about science is highly suspect and no more trustworthy than the anti-evolution tripe spread by the Discovery Institute and the Institute for Creation Research.

    I know that it is pointless to argue with an anti-science fundamentalist, but I really don’t see any reason why the world should heed Global Warming Denialism from a zealot such as yourself.

  15. Hello Dodgy,

    “This seems to be your position, so I’d guess you are of the Global Warming faith (praise be to Gore)?”

    Funny that you would say this … since there are a number of prominent Global Warming Deniers who are also Young Earth Creationists. For example, the hottie Republican Vice President Sarah Palin. Roy Spencer is another Christian fundamentalist who is a prominent Global Warming Denier (and an actual climatologist, unlike the anti-scientist Jeff) and also on record as denying evolution.

    The religion of Global Warming Denialism is Christian Fundamentalism. It is no coincidence that George W. Bush happens to deny both evolution and Global Warming and he is applauded for doing so by those Christians who remain stuck in the 19th century.

    Jeff ain’t a scientist. I would feel pity for anyone foolish enough to rely upon his “analysis” of Michael Mann or anyone else.

    ***

    Isn’t Sarah Palin pretty?

  16. Dave, I have put up with a lot of your posts. I am getting tired. If you have something to add feel free, but this mindless hate has to stop.

    If someone asks me to delete your posts at this point, I will.

  17. David, it’s these AGWers who are the creationists, believing that the world was always at the same temperature until the late 1970s, conicidentally the time of the PDO shift from negative to positive.

    Did you notice that Mann’s graph is comparing apples to oranges? Earlier temperatures are based on proxies, that red addition at the end is based on actual temperature data. Obviously the correlaton of the proxies to actual temerature must be significantly less than 1, so obviously current actual temperatures, if warmer than they were in the past, will show MUCH warmer than the proxies because of that correlation problem. To be fair, you’ve got to compare current proxies to earlier proxies to see if there is a significant change.

  18. Alan,

    I know you are addressing Dave, but you expressed the problem I have with the science perfectly. How did this become accepted science? Are there really 2000 science guys sitting around the IPCC saying, “yup, this makes sense!”. My curiosity has turned into outrage on this.

    This is a serious flaw and the practice in general needs to stop.

  19. Hello Jeff,

    > “Dave, I have put up with a lot of your posts. I am getting tired. If you have something to add feel free, but this mindless hate has to stop.”

    This hate is not mindless. You are a businessman pretending to be a scientist so that you might make slanderous accusations against the climatologists who are threatening your love … the burning of oil and coal, I suppose.

    Okay, Jeff, if you want to play scientist, go ahead and do so … but will you allow a scientist to operate your business? Will you take business advice from a scientist? If some academic was pretending to be a businessman and thereby writing all sorts of negative things about how you run your business … would you accept such criticism?

    What is motivating you to engage in this futile crusade against climatology, Jeff? Do you have some sort of financial interest in defending pollution against regulation and taxation? Does your business pollute … are you protecting your revenues and income?

    I would love to have a reasonable conversation about science with you, Jeff, but I sense that such a conversation is pointless when a person has an anti-science quasi-fundamentalist attitude. The creationists would accept evolution regardless of thousands of pages of scientific evidence. The Global Warming Deniers will reject the conclusions of climatology regardless of thousands of pages of evidence.

    How, then, can we engage in a reasonable argument about this subject?

  20. Hello Alan,

    > “David, it’s these AGWers who are the creationists, believing that the world was always at the same temperature until the late 1970s, conicidentally the time of the PDO shift from negative to positive.”

    This is, quite literally, a *stupid* thing to say, Alan. The climatologists are worried about Global Warming specifically because they *know* that the Earth has had radically different climates in the past and that our civilization which was built consistent with the relatively stable climate of the last 5000 years won’t survive should the climate change dramatically.

    For example: In the past, the entire state of Florida has been covered by the ocean. Should the ice caps melt, the ocean will cover Florida again. It is very difficult to imagine civilization surviving in Florida when the oceans cover Miami, Jacksonville, Tampa-St. Petersburg, etc.

    As to the *stable climate* argument, Global Warming Deniers such as Roy Spencer are on record as having argued that the Earth’s climate is stable. His argument is not internally consistent, though, because he does acknowledge that the Earth’s climate has changed dramatically in the past.

    > “Did you notice that Mann’s graph is comparing apples to oranges? Earlier temperatures are based on proxies, that red addition at the end is based on actual temperature data.”

    Eh … no shit! I think that the climatologists *know* that actual temperature data does not exist prior to the invention of technology to accurately measure temperatures. So the history of the climate for the 4.5 billion years prior to the scientific era must, of necessity, rely upon proxies.

    But climatology is not the only science burdened by this “flaw”. Biologists have only objectively studied the process of evolution for the last several centuries. They have no direct evidence for the billions of years prior to the modern era. So … maybe the creationists are right?

  21. Hello Jeff,

    > “I know you are addressing Dave, but you expressed the problem I have with the science perfectly. How did this become accepted science? Are there really 2000 science guys sitting around the IPCC saying, “yup, this makes sense!”.”

    This is the sort of statement which reveals that you lack even the most rudimentary high school level science education. If you want to know how Global Warming became accepted science there are thousands of pages devoted to this very topic and they are available online for free.

    The fundamentalists and the creationists make similar ignorant remarks about evolution, the age of the Earth and pretty much every other scientific idea which is too difficult for their own small minds.

    I can see why you love Sarah Palin so much, Jeff … ignorance loves the ignorant.

  22. Hi Jeff

    Your pet troll is hilarious 🙂

    I can hardly believe the New Stick is this easy to debunk, but it seems to be shaping up that way. I had hoped for a nice clear picture of it in the Mann paper, but afaict it only appears as one new wiggle in a spaghetti graph with a bunch of old Sticks, though I admit to not reading the entire paper. Pseudoscience makes me puke.

    If that really is the correct method, then obviously red noise ‘proxies’ would produce hockey sticks galore, and the analysis is so simple, anyone can understand it. No need to explain principal components etc this time around – could be a major blunder by the Davids.

  23. I wonder how far Einstein would get in RC’s and Dave’s world? My guess is that he would be rejected without a thought because he didn’t meet the standard set forth.

    The scientific method is nothing more than the search for the truth and the last time I checked there was no prerequisite required for anyone to honestly go searching for it.

    Jeff, if you are really onto something, and up to this point, I think it’s anyones guess, you should consult some people with a very strong background in statistical analysis (maybe Briggs), if they find validity with your work, then a co-authored response in PNAS would be a necessity.

    My guess is you’re not looking for the spotlight…and neither was McIntyre…but the science demands it! And for the record, McIntyre is not a climatologist either! Good luck, I will be following with great interest.

    Lee

  24. I doubt anything I do will get much notice. Steve McIntyre is quickly becoming a hero of mine. Debunking this hockey stick is only part of the problem though. This process of alleged correlation of data as a method for scrapping good information is apparently standard practice in climatology.

    Some of the guys on climate audit are even writing in to show where Mann’s correlations are accepting too much data and his statistical analysis is flawed. It makes my head hurt to think that these smart people are supporting this junk science even while showing flaws in the Mann paper.

    What has to happen if we will ever get peace from the AGW guys, is that we need to find a way to prove beyond a shadow of a doubt that you can’t cherry pick data using this statistical bullsh…

    Cause we all know bullsh.. is still bullsh.. even when you cover it with pretty science words like statistics.

  25. Hello Jeff,

    “I doubt anything I do will get much notice.”

    No doubt. Based upon merits, too.

    Lee asked:

    “I wonder how far Einstein would get in RC’s and Dave’s world?”

    Eh … Jeff isn’t Einstein. He sounds more that Duane Gish or Henry Morris (of Institute of Creation Research fame).

    Jeff’s not a scientist. He fits in well with the other fundamentalist anti-science nutcases.

  26. Ah, yes my troll has returned. Still not one piece of information, only hate and accusation.

    You are a very angry and confused young man. It’s not healthy,it is mildly entertaining though.

  27. Jeff. just go for it. I did statistics ( hell its 50 years ago) and I’m a little rusty, but it’s easy to follow your reasoning and logic. Also its very apparent that your resident troll hasn’t a clue himself, quite hilarious if it wasn’t so tragic. The more ad hominem attacks he makes the more his ideas become rejected
    Adam

  28. Laughing…as I read down through the thoughtfull dialog out pops David, frothing at the mouth. He does add some contrast.

  29. Jeff,

    “Dave, I have put up with a lot of your posts. I am getting tired. If you have something to add feel free, but this mindless hate has to stop.

    If someone asks me to delete your posts at this point, I will.”

    Please delete David’s posts. You could let him know that I am an environmental chemist and I say he’s wrong.

  30. http://www.technologyreview.com/Energy/13830/?a=f

    “That discovery hit me like a bombshell, and I suspect it is having the same effect on many others. Suddenly the hockey stick, the poster-child of the global warming community, turns out to be an artifact of poor mathematics. ”

    You have answered my question on why other proxy studies have confirmed the original fraudulent HS, thankyou.

  31. There are two parts of this post I don’t understand:

    …then sort it (throw data out) so that only data which correlate to a temperature rise at the end 5% of the dataset remains. Then you average the remaining data you would get a relatively flat line with an upward spike at the end.

    What is the source data being correlated against here? Actual recorded temperature? Is that where the 5% rise comes from?

    So now suppose that you had a noisy temperature signal in the data, you have filtered it all for a recent upward trend which is clear, now what would happen to the noisy but real temperature signal in the past. It would of course be muted by the average of the noise, dampening its actual magnitude!!

    (Comma added to the last sentence – is that the meaning you wanted?)

    Where does the “noisy but real temperature signal in the past” come from? This paragraph says to me that the dampening/smoothing which made recent tree data match observed temperature figures, is extended backwards to equivalent tree data from before records began.

  32. #36, some of your questions are answered in my latest post. The source data is correlated against a lat/lon gridded temperature. The proxies which have downslopes are simply tossed in the garbage.

    The comma is correct.

    The noise is sorted in the blade portion of the hockey stick to only accept upslope noise. In the historic portion both up and down are accepted – cancelling. The result is a strong blade made of upslope noise and due to scaling, a reduced signal in the past.

  33. Jeff – great comments from all. I will again teach World Geography and Climatology at a four year university in the Atlanta area. I will use this site’s collection of information and topic strings for my students to research and report on.

    A general observation in this and public confrontational discourse is that the apples and oranges comparisons scream out at each other and neither side makes sense. Like watching the NFL and the other side is yelling for a Home Run!!!.

    The side yelling that “you do not believe in Global Warming . . . ” is showing statistics that show warming. They look only at recent 150 years of temperatures only it seems and have come up with the now famous “Hockey Stick.

    On the other side are folks who say it (Global Warming)is not anthropogenic.

    The problems I want my students to see include the great misuse of resources (to include revenue) to try to stop the inevitable – against using it to determine what to do for humanity when rice is grown in Nebraska and South Dakota and when the bread baskets of the world are Canada and Russian Siberia (after the north running rivers (OB and Lena)have free flow to the Arctic.

    Anyway – my students will be allover this site so please keep the dialogue professional and thought provoking.

  34. In my very amateur opinion:

    Two points about the hockey stick graph, or more accurately, the long handle portion.

    Surely there has to be more temperature variation than is shown on the handle

    Doesn’t the downward slope of the handle come close to the downtrend of the last 2/3 or so of the Holocene?

    And a suggestion:
    Why not study climate data with a channel or Bollinger Band that includes the full temperature range? Isn’t the temperature range part of the climate as well as the average?

  35. Just stumbled upon this wonderful article.This is extremely exciting.I’ve bookmarked it and will come back routinely.Hope to see more such articles.file_links\xrumer_resource\km_sig.txt,1,L]

  36. Pingback: Dreamhost
  37. What floors me about this whole thing, is that this is the same type of stuff you can imagine that unscrupulous drug manufactueres might use to “prove” the efficacy of, say, their latest cure for cancer.

    Can you imagine the lawsuits they would face as soon as their statistical methods were outed? By anybody?

    Yet, these jokers just keep rolling along, doing the same things over and over again.

    It’s disgusting.

  38. I really like your blog.. very nice colors & theme. Did you
    create this website yourself or did you hire someone to do it for you?
    Plz reply as I’m looking to design my own blog and would like to find out where u got this from. cheers

  39. Where can one read the original paper and most important, where can one have acces to all the historical temperature data.

    To get it over with RC or whomever should share the historical data from which all this statistical trending conclusions where drawn.

Leave a comment