Demo of Flawed Hockey Stick Math Using Actual NH Data

I got a bit of a surprise while continuing my calcs using red noise.  This time I used a real reconstruction from the NH data.

Last night I spent about 5 hours on more red noise plus signal analysis.  I found some incredible things regarding the methodology of statistically sorting and calibrating proxy data.  While others have examined and published papers on how hockey sticks can be created from red noise, I am taking this to the next step.  What I have done here is to impart artificial known signals in the red noise and go look for it.

Last night I expanded that work to put in my red noise data the CPS northern hemisphere composite data from Mann08 and then used a similar methodologies to his to go look for it.

If you are not familiar with my other work go here first.

Simple Statistical Evidence Why Hockey Stick Temp Graphs are Bent!!

Below is the one curve M08 provided as a result from his proxies.

This is one continuous curve but I colored the end red to show the 1850-1995 calibration range used below in all calculations.  This curve is actually an average of 10,000 red noise graphs with the signal added in.  It is a near perfect match to the original and can be verified by the quality of the tail from 0 to 200 AD.  There is a 0 signal and the average of the red noise is basically 0 here.

So I have 10,000 series of red noise plus a perfect known signal added to each one.   How can we find the best proxies to recapture the data.   Paleoclimatology uses statistical correlation, but they only have temperature for the red section of the graph above so they correlate to the section they know.  This seems a reasonable assumption but is actually a fatal flaw in the math.  Let’s see what happens just by sorting for the best fit to the red section and averaging the result.

The blue line is the original curve from the first graph.  The purple/pink line is all data with a correlation r> 0.1.  Look what happened, it amplified the signal in recent times just from sorting.  What I find interesting is that historic data (prior to the 1850-1995 calibration range) fits the original data almost perfectly.   There is no amplification.  Well naturally you would say look for better correlation data in the calibration range and you will get a better fit to the original series.  I reran the same test with r>0.8 and I got the yellow graph.

The Yellow much higher correlation line is even more amplified in recent times yet historic data is not magnified.  When we consider how correlation works flat low amplitude but noisy signals are given low r’s when compared to data with a sloped signal.  This natural and correct mathematical process prefers high slope data.  The result is clear above.   Higher correlation to a positive (known calibration data) increases the amplification of a signal inside red noise.

Well this isn’t what paleoclimatology does — completely.  In paleoclimatology, high correlation data are then scaled so the calibration range has a signal that is a perfect match to the known.

In my example here we have the advantage of a perfect known signal and a huge set of proxies to sort so we can see clearly what the scaling does.

I used the following process to scale the calibration data from the two graphs above.

Sort proxies by  r>0.8

Fit a line by least squares to the red section of the top graph

Fit a line by least squares to all remaining passed r proxies

Scale and offset each proxy such that linear least squares after magnification has the same slope and intercept as the linear least squares in the red section of the original graph.

This process is analogous to the EIV and CPS scaling values in M08.  I am not an expert in these methods but since they use linear scaling to make the best fit of proxy data to the red calibration data it is a good approximation.  I believe this simplified method will produce a result so similar it would scare the original authors.  Anyway, it doesn’t matter which method I use for this demo, the result is the same.

Here is the r>0.8 graph in red on top of the original signal in blue.

The red signal in the calibration range from 1850-2000 is shaped almost perfectly like the original data series.  But look what happened to historic values.  For those familiar to my recent posts (link above) we aren’t too surprised by this result.  But for those who haven’t seen it, what you are looking at is a demagnification of historic temperatures based on the basic methodology used in every hockey stick paper.

We know the signal is shown in the blue graph, I put it in the red noise with a + sign.  Yet the statistics process of sorting for the best correlation and magnifying it to fit “known” values creates a substantial demagnification of historic values.

Again if you are familiar with my other work you can see the true zero of the red curve offset by the amount shown in the tail.  Not only is the data demagnified it is offset!

You might have noticed I did the high correlation r>0.8 data first.  Science is kind of fun, this next graph caused me to spend hours on what I thought was going to be an easy post.  What happens with an r>0.1 through the same process.  This is the same data as shown in the purple/pink line of the second graph but each series went through the scaling process to match the calibration range data to the highest degree possible.

Ouch, something went wrong.  I spent hours thinking my software had a problem.  I tried different red noise levels and poured through thousands of lines of code. No problems. What?

It took a while to realize what really happened.   The r correlation process rejects flat (nar zero) slope signals in the calibration range.  A loose r value of 0.1 allowed very flat slope correlations to be accepted.  These individual series are then magnified by a multiplier until they fit the original slope of the calibration resulting in a few series having very high weights compared to the rest.

Think of it like this, assume our calibration data has a slope of 1.  The red noise + signal has slopes varying from below zero to above.  If a proxy shows enough correlation to barely exceed the r>0.1 value some of these proxies will likely have a flat slope say 0.01.  This series would then be magnified by 100 times to match the known signal.  Which it clearly did in the calibration period from 1850-1995 (my last graph above).

This is actually the correct result!  I also find it a stunning revalation and confirmation of how others have shown by looking at the magnification coefficients from actual papers that these hockey stick graphs are primarily a few series in historic pre-calibration times.

Here’s the big revelation for me

The math forces historic values to be comprised of the worst r value series!

The lowest r values get the highest magnification.  Why didn’t I see that before?

I hope my readers find this to be as huge of a result as I do.  I had been guaranteeing myself that the historic signal would be much flatter than recent times but as I demonstrated in my other post (link above), flat sloped proxies cause amplification of the historic signal.  You know science is fun even when you are wrong in your expectations.  I just didn’t expect a few proxies to get so much magnification that they took over the rest.

By the way r>0.1 resulted in 3399 or 34% of my original random series plus signal being accepted.

Comments and criticisms welcome.

Also, from M08 we know they added fake upslopes to the ends of 90% of the 1209 series.   This would have the effect of making sure that not one proxy took over the historic graph.  Think about that for a minute!