the Air Vent

Because the world needs another opinion

Mann’s Statistical Amplification of Local Data

Posted by Jeff Id on September 8, 2008

Ok, now I am getting closer.  After scaling the data in SD units from 1400-1800 and averaging both the 484 series from the accepted Mann paper data and 1209 series from the total set I got this plot.

We can see that the data in the calibration period is clearly amplified, pressing slightly lower at 1850 and substantially higher in 2000 as compared to the average.  I found it interesting that this also magnified the 200-750 year data as well, since there is not as much data here the individual series can make a big difference.  This next graph plots the difference between accepted and total data and rejected and total data.

For those who don’t (and don’t want to ) live in paleoclimatology  (ancient temperature reconstruction), there have been temperature measurements for 140 years.  The data below was sorted out from a huge pile of scrambled noise to produce……..the hockey stick.   In order to make the stick the author Michael Mann needed to do some statistical wizardry.   The most important part of the graphs below

Passed Data minus Total data and failed minus total data

Here we can see a sudden and sharp step in graph 2 corresponding to the calibration period of Mann.  The increased amplification of this portion resulted in an approximate 0.5 -1 sigma amplification of the data in the correlation.  The sharpness of the step in the calibration period (the last 100 years gives it away.  The third graph is all the rejected data and shows an inverse step of reduced magnitude.  This is clearly the result of the calibration.

The third graph represents the rejected data note the opposite sign.


The red and blue lines are from above.   The green line represents the absolute value of the difference with a 60 year square filter.  I did it that way to make sure that the amplification of the local (calibration range) data was ‘hit you in the head’ clear.

What does this mean?

To me it means that the hockey stick paper’s statistical meat grinder has clearly weighted the data in a manner which heavily magnifies instrumental temperature rise (the last 140 years).  If we are to use the Mann curves in any subsequent calculation of temperature the remaining data must be multiplied by the difference created through this statistical sorting.

That small period doesn’t matter because the rest of the data is unaffected.

You might look at this and say it doesn’t matter that they have changed the endpoint (140 years) the previous data was unaffected.  You are right but remember Mann uses the (140 year calibration) period to scale (multiply) his data sets for fit.  Realize this amplification of calibration period data reduces the scale (multiplication) factor of his remaining data by what appears to be more than 2 times by eye. this is a significant error in the statistical method.  I read a wonderfully written article on PCA which talked about teasing the data out of the noise. Well, this is the result of that kind of thinking.  The data in the Mann paper is heavily misrepresented.

I can think of no clearer evidence of preferential amplification created by this flawed statistical sorting practice.  This improper scientific method used so widely in paleoclimatology needs to stop!

12 Responses to “Mann’s Statistical Amplification of Local Data”

  1. hswiseman said

    Nice Cherry Picker Depicter

  2. Jeff Id said

    Thanks for the comment. I have rewritten the article a bit to explain it better and added some more graphs.

  3. LeeW said

    Would it mean anything to change the time period used for scaling? Would it have much of an effect on the shape of the graph? It seems to me that using that time frame may provide a bias especially with respect to the effect on how the instrumental data is displayed.

  4. Jeff Id said

    I have been thinking about that also, I chose the scaling period simply because many people who visit here are from the Climate Audit group and I matches the scaling system used by Steve McIntyre. I think there needs to be a better way though.

    I was considering scaling the proxy data individually to have a temperature variation which corresponds to measured temperature but it kind of stinks because of some of the noise in the proxies. I considered scaling like groups of data i.e. tree-rings, mollusk sphincters or whatever as groups but each proxy is different by region. Eventually I am going to have to pick something (Mann did after all).

    Any suggestions?

  5. LeeW said

    Well…I wonder if Mann had a motive in picking that time frame? After all, it takes in the entire LIA, but leaves out any comparable warm period. I think that may explain why the LIA is virtually non-existent in the graphs plotted above.

    I’m not sure how difficult it would be to rescale…but it sure would be interesting to see how things would alter by using a timeframe that consisted only of a warm period (say 600-1000) and then using a combined scale that includes both a warm and cold period.

  6. LeeW said

    One other thought…

    How difficult would it be to separate the proxies by region (or continent) and run each of those series as an independent analysis to see how much correlation there is (or isn’t) to define a global mean.

    A quick eyeball of Figure 1 from Mann’s paper indicates a majority of the proxies were from Europe and N America. So, obviously the weighting of those proxies compared to other continents could alter the global mean.

    I think that the argument that CO2 is a well-mixed gas may become a little ‘foggier’ when the AIRS group at NASA releases their findings soon. Their last release of atmospheric CO2 in 2005 looked anything but well-mixed. Should their findings continue to backup their initial release, the idea of ‘global’ may lose much of it’s appeal when it comes to warming (or cooling).

  7. Jeff Id said

    It should be reasonably easy to do the analysis you describe. There are two big problems I’m having now one is that I only have the data for the proxies, there is another file I need with regional temperatures. Once I figure out that, combing regions correlating local temperatures to proxies and odd scaling techniques become pretty easy.

    The second problem is that I don’t know how Mann has scaled the data for his paper exactly. I need to review more before I can move forward with that.

    On another note, I am ready to do some least squares regressions of the data to see what correlations I can get. I am considering the shotgun approach of setting up some code (everything is in C++)to power though all the data to look for strong correlations so I can do my own sort.

    Check out my latest post with the raw data separated into accepted and rejected, It’s interesting to me.

  8. WhyNot said

    I have not read the Mann Reports, delved into the data, nor researched any of the underlying principles used in reporting “temperature” variations over the last several thousand yrs. However I would like to state my observation of the data Jeff has produced and commend the people who have responded with intellectual insight.

    The first thing to note is that in most instances averaging of data will filter out high frequency noise. One can design averaging filters that take into account high frequency noise.

    The second thing I noticed right away in graph 3 is the near symmetry about 0 of the passed vs failed data. The key being here is “near” symmetry. If one were to randomly remove data from the data set to minimize calculation time/complexity then one would have symmetry, the error about the “0” line would be equal. Obviously, graph 3 shows hand picked data to justify what Mann wanted his conclusion to be. It would be easy to see exactly how much bias was introduced by calculating the error difference.

    Third, this data, without seeing the actual data sets, appears to have both amplitude and frequency modulation with a bunch of noise. Without a 100k yr history it is tough to tell. I wonder what an FFT on the raw data would tell us? Nothing to discredit Mann, but it might show a periodic structure to use as a basis for a different filtering method.

    Now the WhyNot (After reading most of Jeff’s blogs)

    WhyNot correlate tree ring growth to mammal farts, I’m sure the UN would be impressed. It would give them another reason to stop eating meat. I’m sure I could do it using Mann mathematical principles.

    WhyNot ask all the mental midgets who believe Mann and the UN (don’t eat meat for one day) to stop exhaling CO2 for a day a week??? Crud, this only works once……not a good idea!!!

  9. adminor said

    daily blog ranking report

  10. retired ChE said

    Back in the 1930’s a book was published showing that tree ring growth, as revealed by the width of the rings, was correlated with the 11-year sunspot cycle. The theory was that trees grow faster in periods of more intense radiation from the sun. I don’t know if more recent data have been obtained to look at this aspect, but I don’t see how tree rings can be used to determine past temperatures to 0.1 C.

  11. Jeff Id said

    It gets better ChE check out my post here,

  12. sandrar said

    Hi! I was surfing and found your blog post… nice! I love your blog. 🙂 Cheers! Sandra. R.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: