To the Drawing Board
Posted by Jeff Id on April 9, 2010
It’s a good day for me. Nic sent a link to an Ammann paper which discussed the suppression of variance in paleoclimate reconstructions- the bane of my sanity. TAV wouldn’t be a climate blog without the horrible statistics in paleoclimate. Ammann (A09) proposes a simple method for correcting the calibrations in paleo reconstructions. The correction is ad-hoc and poorly tested in the paper but it is the beginning of some proper self reflection by the paleo community.
The link to the paper is here.
A couple of cathartic quotes for my id, which has been ranting into an apparent vacuum as far as climatologists go, for over a year now.
Regression-based climate reconstructions scale one or more noisy proxy records
against a (generally) short instrumental data series. Based on that relationship, the
indirect information is then used to estimate that particular measure of climate back
5 in time. A well-calibrated proxy record(s), if stationary in its relationship to the target,
should faithfully preserve the mean amplitude of the climatic variable. However, it is
well established in the statistical literature that traditional regression parameter estimation
can lead to substantial amplitude attenuation if the predictors carry significant
amounts of noise.
Thank god, it only took about a minute for this engineer with Mann08 to figure it out. Now, after almost 2 years of frustration, it’s great to see some of the work recognize the problem. All of my hockey stick posts above address this exact issue. Anyone think Ammann is a skeptic?
Climate proxies derived from tree-rings, ice cores, lake sediments, etc.,
are inherently noisy and thus all regression-based reconstructions could suffer from
ALL REGRESSION BASED RECONSTRUCTIONS.
Yup, that’s what I said.
TLS has received significant attention and new NH reconstructions based on this technique generally exhibit more pronounced amplitude (Hegerl et al., 2006, 2007; Mann et al., 2008; Riedwyl et al., 2009).
They are discussing the amplitude of the calibration period — IOW, how to paste a blade on a hockey stick.reconstruction period and the reduced straightness of the hockey stick.
Now this part of the discussion is for the more technical of the readers. It’s from the conclusions portion of the paper.
One trade-off that has to be accepted in regression-based reconstructions is that the 25 correction for bias comes at the cost of increased variance (see Supplementary Material: http://www.clim-past-discuss.net/5/1645/2009/cpd-5-1645-2009-supplement.pdf). This variance increase is mostly concentrated at the interannual scale, and thus decadal smoothing of the reconstructions results essentially compensates for this.
What an interesting statement in the conclusion. The paper discusses correcting for noise in proxies, they have proposed a method which I’m not sure is new or something done elsewhere (a point brought up by one of the scientists who reviewed). The method comes right out of the blue from my perspective. Anyway, they tested it on some very unique noiseless data – for which it worked well. If you can read the equations presented, it corrects proxy scale multipliers (called slope here) by the variance (read annual noise). It doesn’t seem likely that this method would work well when there is significant autocorrelation and multi-year noise. It relies instead on annual variance to properly re-scale the series. The potential biases were oddly not explored by the authors and that particular point was criticized by several of the technical comments. — correctly. My guess is that the reason it wasn’t explored is that the authors knew what they did was an improvement, yet knew it would fail a more real-world test. It will, but like the recent sea ice wind paper, that in itself doesn’t make it bad.
What makes me so happy tonight is the feeling of a teenie tiny bit of vindication for my comments on Mann08, 09 and several other reconstructions. Again, this is a different problem from the one SteveM dealt with in Mann98. Different math, different problems, same result – coincidence?
Even more interesting than the paper though are the technical review comments in the interactive discussion– a nice format for publication.
But, in the real world we are dealing with proxy records which can (and do) have much more complex noise structures. We cannot then simply assume that we have a higher SNR at low frequencies compared to high frequencies at the individual site records. If there is much noise at low frequencies in the original proxy series, then I would intuitively guess that ACOLS would result in an artifical inflation not only of the high-frequency noise component in the final reconstruction (as shown in Figure S1), but also inflate the low-frequency component of the noise.
I don’t know A. Moburg and don’t feel like looking him up but I don’t particularly agree with his intuition, in real world proxy data, the annual noise is quite dominant and different proxies with different annual autocorrelations will average to uncertain results with this method. A statisticians nightmare. However, his points on the more complex noise structures are well taken.
Zorita had this to say at the beginning of his comment.
Are all climate reconstructions wrong? Well, this manuscript does not imply it, but it is a welcomed warning that reconstruction methods may be more complex than they seem.
For instance, when using a linear regression model to reconstruct past temperature (predictand Y) from a temperature-sensitive proxy (predictor X), the estimation of the regression parameter by ordinary-least-squares (OLS) requires that the predictor is noise-free. This is clearly violated most of the times since time variations of proxy records are due to many other processes than temperature. The blind application of OLS in this context leads to an underestimation of the regression parameter, and thus to an underestimation of past climate variations.
Not a surprise from Zorita, who figured this out in 2004.
Christiansen had this to say:
Strangely the authors fail to cite our recent paper (Christiansen et al. 2009) which showed that variance l oss is a serious problem for 7 different reconstruction methods including both direct and indirect regression methods as well as methods based on CCA regression and TLS. All the methods showed substantial underestimation of trends. low-frequency variability, and the pre-industrial level.
Well I think we’re reaching a consensus here, but wait.
Brohan said this:
The main issue with the paper is that both the derivation of the method, and the pseudoproxy tests, are done with simplified and idealised forms of contaminating noise. A clear implication is that the methods will work similarly well for real proxy data, where the contaminating noise is more complex; this is not likely to be so.
And clarifying with this:
The conventional approach to solving this for B0 and B1 is to choose the values that minimise the RMS of E – it is now well established that this often results in biased values of B1: that it mis-represents the true relationship between the proxy and the climate.
B1 being the slope of individual noisy proxies.
This is a standard assumption in mathematical statistics, and the paper does an admirable job of demonstrating the value of ACOLS where it holds. But for real proxy data this assumption is grossly violated: U will be autocorrelated, correlated with X, non-normal, and sigma_U will vary with time. It’s not reasonable to require a calibration method to cope automatically with all these problems (probably no method does), but the value of the proposed method is not how well it behaves in the idealised case, but how well it will do in the real case.
Gotta love it, my mood is improving by the moment.
Anonymous reviewer 3 is one of my favorites. If you are a serious reader here, consider the bold statements below.
A final point highlighting previous online comments is necessary regarding the representation of the literature, as discussed by Christiansen. I think Christiansen’s argument is a little narrow, but I would agree that Ammann et al. have not done a particularly good job at characterizing the arc of the variance loss and bias discussions within the literature. It is, for instance, surprising to see the Mann et al. (2007, 2008) papers cited as acknowledging the need for attenuation correction. These papers are part of a series (e.g. Rutherford et al. 2005; Mann et al. 2005) dating back to the Mann et al. (1998) publication that have argued vehemently for the ‘low-amplitude’ reconstruction originally reported in that paper. While latter studies test new methods, the thrust of the arguments throughout these papers has been that there is likely no variance loss or biases in their reported results. To imply that the need for variance corrections has been advanced by these studies is therefore a rather serious mischaracterization.
Ok, that comment is correct enough to make me quit blogging on hockey sticks. The boys are getting it, and it’s not 100% politics first. He’s absolutely, flatly stated that the premise that Mann had considered variance loss in any way is crap. Ammann seems to me to be looking for the exit.
But we’re still not done. It’s a good day/night in blogland.
Most of the other reviewers have commented that red noise is both more realistic and more difficult to deal with. In my own emulations of the ACOLS method, red noise (lag 1, 0 < r < 1) can introduce spurious variance at the decadal and multidecadal scale, although the longest-term multicentury or millennial mean may still be captured.
I liked Ammann a lot less after this comment. This means I was right, the new method doesn’t fix the problem in a reasonable case. There is a difference between suspecting something won’t work and having confirmation. You cannot write a paper on a fix for paleo reconstructions, find an apparent solution to the variance problems, and employ the ONLY TYPE of noise which allows your fix to work — BY ACCIDENT. Yup, I’m at it again. This looks like intent to me, if Ammann comes by and explains himself here publicly, he has a chance of being off the Mann list, otherwise, there is no realistic choice. If you get the math, you should agree. However, it’s not a big deal because every SINGLE reviewer picked up on it. 100% honesty.
It is a good time for climate science. The Anchukaitis comment finished with this:
The manuscript would benefit most from additional tests and representative examples and comparisons using realistic red noise.
Now that IS a consensus. A natural consensus borne of the truth of simple math in the face of endless reconstructions presented as confirmation that today is the warmest time in history.
Back to the drawing board boys.
I recommend that anyone interested in the hockey stick plots, takes a moment to read the above links and view the appropriate graphs. It has been an interesting and rewarding night for me. It’s cathartic to see some sanity in this world.