Ok, it’s time to summarize what I have learned from exploration of proxy data. I went back to Climate Audit and found the exact moment that I learned how hockey sticks were made.
You can see my dm little mind working trying to grasp what has become so obvious now. My first post:
You know what I got for an answer, check this out
Curt:
September 3rd, 2008 at 11:57 pm
I tell you what, better advice than that is hard to come by.
My next thought was to question Real Climate about what I had just learned. I pointed out that sorting of data seemed like would cause temp graphs to be distorted by the mathematics. Gavin Schmidt had replied to my questions in the past. Instead of a reply, my post was CUT, CENSORED FROM THE DISCUSSION. A engineer who wanted answers but nothing. I realized I had stepped on a hot button.
Well many of you know what happened next. I had to have answers, it’s my nature. I dug into the data and my blog changed completely. After starting this blog only a couple weeks earlier for entertainment now I was buried in proxies-I had to look up what they exactly were, talk about green! Noconsensus was meant to refer to the concept of group agreement rather than global warming, my intent was originally to vent about politics, global warming, energy and a bunch of other things.
All I could think of was what is the best way to demonstrate that the sorting of data for a trend would create artificial temperature distortions, it seemed so obviously wrong there must be a way. After putting in several hundred hours into this study now only 6 weeks later, I have proven the distortions in the temperature graphs to my satisfaction beyond a shadow of a doubt. Talk about obsessive, wow.
I find it interesting to look back at where I was only 6 weeks ago. A regular poster called bender mentioned correctly that I was messing up the thread, I wasn’t worried about that though.
jeff id:
September 4th, 2008 at 6:00 amI’m sorry to push the thread off topic. When I started reviewing AGW science more seriously last year, I didn’t know anything about paleoclimate temperature reconstruction. I had no preconcieved notion of who was right and flatly refused to make any conclusions. I hoped it would be a bit easier to sort out not wanting to become a climatologist after all, no offense. I have read a couple dozen scientific papers on the topic, this is my first reading of Mann’s hockey stick methodology.
I am surprised to say the least, that this is what you have been fighting with and the same was done with tree ring data. I could barely sleep last night after figuring this out. Instead of resolving anything, I need to delve deeper into the reconstructions. Maybe I’ll need to find the datasets myself and write some of my own code.
Thanks for the help. I’ll leave you alone now.
Bender replied that he didn’t want to put me off, but it was only today that I went back to that thread. I never read his reply because I wasn’t put off at all.
Funny stuff looking back now, I really don’t want to be in the middle of this battle but now I am working towards a publication — strange world for sure. I got some help from CA to download the data and made some initial plots, the first graph is very interesting to me even today.
The blue line is a plot of the data which was accepted by M08 with a simple average 484 proxies used averaged together. The purple/pink line is the rejected data from the same plots. I spent a long time looking at this graph. How can they be mirror images, it doesn’t make sense! The green line is the magnitude of the difference, clearly the difference is greatest in recent years. The green line demonstrates clearly that magnification of local (1850-2000)data has occurred once you understand it.
I will continue this series later, in the meantime ask yourselves why are these graphs nearly mirror images. Does it make sense?
————————–
Added Oct 15,08
I tell you what. Z below, really nailed it but look at the assumption he started with.
He said
“if you need to generate a certain profile from a bunch of data which average out to a basically a straight line then ”
His assumption is my conclusion. Z must be a lot smarter than I am or he must have already decided that the data is purely random from something else.
This was my first big revalation, these proxies are supposed to represent temperature, there should be a fairly strong repeated pattern in them somewhere ….anywhere. But there is nothing, the flatness of the green line from 1850 (end of the calibration range) to 1180 is amazing. Somewhere in that line I expected to see a hump, a point where most of the data agreed and the rejected data took on a bit of shape like the accepted data.
There is nothing. The data mirrors nearly perfectly, so Z above makes the assumption that the data is actually random and trendless and the light bulb goes on – everything makes sense.
You wouldn’t believe how many hours I stared at this graph and the code that made it looking for problems. I kept asking myself “where’s the trend.”
————————–
Added Oct 16,08
Added by request below. Same graph as above with filtering removed from green line.
the Air Vent 
jeff_id (25-28):
You are going through the same questions that I went through when I found this blog over two years ago. (BTW, I came here with no particular viewpoint, and after reading realclimate.org for a while.) I kept saying to myself, “No, it can’t be that bad!” But I kept finding out that it was that bad, and worse. It kind of reminds me of watching the movie “Dangerous Liaisons”, thinking repeatedly that the characters couldn’t keep stooping lower, but they did.
I think you will find it worth your while (as for other newbies) to spend a good amount of time methodically reviewing the archives here. Start with Ross McKitrick’s “What is the Hockey Stick Debate About?” Also read the Wegman Report. What struck me about both papers is how quickly they went from the issue of the (in)correctness of the hockey stick (there are lots of mistakes in science) to the bigger issue of how quickly and uncritically it and similar papers were accepted by the climate science community. I agree with them that this is the much more important issue.
As for the issue of intent, we are discouraged from speculating on motives here. But since you are statistically savvy, try this thought experiment as you go through the archives. Start with the null hypothesis that these are random errors that could go either (any) way in terms of your result. Keep track of which “direction” the errors tweak the data. What p-value do you end up with for your null hypothesis?