The All Important Blade of the Stick Uses Less Than 5% of the Data

Many of you saw my graph of the net effect of the extrapolated “projected” data used in Mann 08 in my post titled Mann 08 – Variable Data. What was not clear at the time to me was how this data was created and why.  I think this post clears it up a bit!

The graph below is the last graph from the post, which indicates that a strong positive weighting was added to about 1100 of 1209 series which in reality had no data at these points.  The graph is here.  You can think of it as a summation of the “added in” data divided by the 1083 series that the extrapolation procedure was used on.  This represents all of the data which was never measured but was used in the correlation for the hockey stick.

This clearly demonstrates a strong hockey stick modification to the complete data set.  It was accomplished using a RegEM  method which has been used by others to interpolate between series, extrapolating (projecting the future) is another use entirely!  But how many of these were accepted into the 484 series of the final paper?

After some number crunching I found that 391 of the 484 used data sets included extrapolated data.  That is 80% of the final “passed calibration” data set used extrapolated data.  I need to note here that one proxy (NV037) had a problem in the data import so the actual number is 392 modified proxies but let’s not split hairs.  The European Luterbacher series was accepted by calibration 100% with no extrapolation, it had an additional 71 series which meant that 95% of the data in recent times is between Luterbacher and the projected data.

Of course we have to ask, what about the ones which passed calibration, how much effect did they have.

When I saw this graph I thought there had to be a mistake.  The net effect of the added proxy data was, almost nothing?? The average of the purple line is the same data as the blue line without the infilling.  I expected something bigger at the 1950 end.  What happened?

In the graph below I zoomed in on the endpoint to see the smallest variations of the data.  The purple line indicates the non-extrapolated data which is barely shifted from the entire series average.  The blue line is the average of all 484 series which passed calibration, the yellow line is the extrapolated data and it represents 80% of the data used at this point in the calibration.   Mann had more of a hockey stick without the Yellow line included but he only had the Luterbacher series plus 4.5% of the other non extrapolated data, not enough to make big claims by.

The graph below is the same as the first graph which incorporates 1209 series.  The one below depicts the summation of the extrapolation used in 484 series.  The summation was divided by 391 extrapolated series to show the weight of the effect on the final data.  This shows that a very high percentage of the Blue line in the above graph was created by the extrapolated series in the below graph.  It was enough to put an artificial kink in the blue line above at the 1998-1999 boundary.

Ok, extrapolation is a tough business even if you know for sure that the trend will continue.   For instance if you know that your company grew 10% each year for 3 years, how certain are you that the fourth year will also grow 10%.  In the case of proxies, we don’t even know if they represent temperature, Mann’s theory is that he can extrapolate using an algorithm designed for interpolation to tell us what the tree rings are doing, rather than actually measure them.

So we need to look at the reasonableness of the extrapolations.  This next graph is the very first graph I plotted.  All of the following graphs are values from a single proxies.  The purple line was separated by me to demonstrate the extrapolated portion of the data.  ALL OF THESE FOLLOWING PROXY GRAPHS PASSED THE CALIBRATION SIGNIFICANCE TEST.

The blue line is the actual data and the Pink line is the extrapolated data.  When I do intense math, I like to make sure that the result passes the giggle test.  Mann does not follow the same standard.  I found this somewhat encouraging so I plotted some more.

Again, it isn’t the curve I would have drawn with my pencil.

The above graph is filled in to the same extent as a whole bunch of Schweingruber series.  The projected data jumps downward and completely reverses trend from the blue line.  Pretty fancy math I think.

This one above doesn’t hurt so much except that it just doesn’t match the data where the RegEM method takes over.

Pretty strange looking extension also.  All of these curves have a big jump where RegEm takes over.

Does this purple line above look like it belongs to the blue one.

Really strange!  a negative trend.   Because the summation of the added extrapolated data is 100% positive I must have been pretty lucky to find this one!

How did Mann come up with these projections.

The RegEM algorithm of Schneider (9) was used to estimate missing values for proxy series
terminating before the 1995 calibration interval endpoint, based
on their mutual covariance with the other available proxy data
over the full 1850–1995 calibration interval. No instrumental or
historical (i.e., Luterbacher et al.) data were used in this procedure.

He took a very select series of proxies, and essentially “pasted” them onto the 1083 of the 1209 total proxies.

That means that 89.6 % of the total data sets had extensions, with 71 Luterbacher series excluded from the RegEm calculation there are only  1138 total proxies to use.  Of those 1138 – 1083 =  55 remaining proxies determined the actual end value of the other 1083 proxy series. Don’t forget that Luterbacher was just an example and some of the other series he references were also rejected by his above quote.  So the number is actually less than 55.

Now, my first post was about the 148 proxies which were rejected from the original, original posted dataset which was placed on a server at the NOAA before Mann changed the data to the new origninal dataset of 1209 series.  Since the original series was unmodified by regem, let’s look at the graph of the data again.

Sixty four of these 148 rejected non extrapolated data series had values in 1998.  There were 64 more proxies to work with in his extrapolation, yet they were rejected without mention in the paper.

My thoughts:

Steve McIntyre was looking into the Mann software to determine whether the extrapolated data was used or not in correlation “calibration” to  temperature data.   I haven’t had time yet because I stayed up until 1am trying to make this post.  I believe that it must have been used in calibration and the extrapolated data was required for a few reasons. Either way it will be figured out for certain.

1.  Mann had a better hockey stick without the data extrapolation.  3rd figure in my post.  The added data shortened the HS imperceptibly.

2.  The infilled data only was added to recent times (not projected backwards).  We have measured temperature for recent times so additional data points here can only be useful for correlation/calibration procedure.  There is absolutely no other reason I can think of why this would be done.

3.  Without the now HUGE mistake of posting 148 available proxies which weren’t used, the Mann 08 paper would only need a few hand selected proxies to alter all the remaining datasets to have an upward trend.  How would we ever show that he hand picked a few weird proxy’s to make the case.  If you notice, in figure 4 of this post, all of the additions to the data were positive in nature!!!!

4. Many are aware of divergence which means the actual measured tree rings didn’t follow temperature in recent times. In this case, a few non-diverged series could be used to fill in the endpoints of the proxies rather than have to fight with the “inconvenient” data.  I think this is why the MXD data was eliminated as well, this is from the Mann 08 paper.

Because of the evidence for loss of temperature sensitivity after 1960 (1), MXD data were eliminated for the post-1960 interval.

As I understand it the “lack of sensitivity” resulted in a short blade on the hockey stick and a strong MWP is produced compared to today.  The fact that it is so casual a statement is where the consensus comes into effect.  There should be a lot more involved in the elimination of data but the group think allows this to happen.

Keep in mind that the claim to fame is that 40% of the series passed calibration whereas random data would only pass 10-13%. If a large% of the data are rejected in calibraion, no paper.

My conclusion overall is that the blade of the hockey stick (prior to the temp curve overlay) doesn’t even exist.  It was fabricated by an amazingly low number of proxies (under 55) and painted into the remaining series to claim a high degree of correlation. These 55 series were selected after throwing out an additional 148 proxies 64 of which had data which could have been used for extrapolation. This tossing of the data was done for unexplained reasons!

It now seems possible that if we remove the extrapolated data, a significant portion of the datasets would likely fail calibration due to poor correlation and another Mann paper can be shot down.