Some Other Area Weighted Reconstrucions

As we’re nearing completion of the improvements to Steig et al. Ryan has been working hard on getting the final details of the best reconstructions possble, Nic has found a couple dozen reference papers and I’m working on the relatively mundane verifications which as a side item include area weighted reconstrucions. In engineering, when you make a calculation and there is no text full of answers to go by, you need something as a backup, some kind of check to insure that you’re not using meters instead of feet or not missing some simple factor in the complexities of a powerful math calculations. R is wonderful in it’s open source nature, however it is hideously bad for verification of highly complex functions.

If you’re a programmer, imagine not being able to stop a complex calculation part way to see the contents of a variable. Imagine the debugging required to insure that things are working when you can’t see the status of the loop. R is a nightmare that way. Where R succeeds is in the same areas as MatLab, vector and array math are fast and easy. For example, you can have two enormous matrices and you can multiply them by typing A %*% B, there are libraries in C but there is also a lot of time. That’s very powerful when working with math, however MatLab still smokes R for its more standard debugging tools, yet MatLab is expensive and R is free. That makes it possible for nearly anyone to verify R code without additional costs or hurdles .

In Ryan’s work, he determined that the appropriate stations to use for the Antarctic were those inside the satellite AVHRR grid so no islands at all. To that end we’ve reduced the list of appropriate manned surface stations to 28 and appropriate ground stations with manned to a total of 63. There were no automatic weather stations (AWS) until after 1982 and much of their data is questionable due to snow burial, instrument failure, and other issues but their influence on the reconstruction is useful. This is similar to my previous area weighted reconstruction which used the closest available surface data to infill the satellite grid points. A Voroni diagram.

Where the again very clever methods of Steig, Mann and company can improve on an area weighted diagram is in the distribution of the information of each station according to measured covariance rather than by closest station. Still the results should be VERY similar.

Without presenting all the detail, Ryan has been successful at producing an Antarctic map using 28 PC’s of antarctic AVHRR data with NO overfitting. He used a method of regularized least squares which is more common than many of the techniques in climatology but better than that it is working. I’ll save the verification stats for later, after all the publishers need something to talk about.

The following was created by me using Ryan’s code and data.

This plot has some of the best verification statistics, one advantage is that it uses a regularized least squares method that takes 1/10th ish of the time of other algorithms to run. The question becomes, does this make any sense as a reasonable answer for the distribution of trends. People will remember tAV Verondi plots from before, however for verification the plot can only use the same stations as the satellite based reconstrucion. Previously there were different stations included so I’ve rewritten the code entirely about 3 times now and this is the most recent result.

For the plot below, the algorithm works by cycling through each of the satellite grid points and locating the #1 nearest temp station.  If no data is available it looks to the next until it finds the closest available data. This insures that all points are infilled through the length of the 5509 series.

It’s amazing but you can really see the RTLS algorithm having basically no spatial information other than from PCA recreating nearly correctly located the trends of the Antarctic. Ryan’s post on it is here although this is an improved version.

Since the above Antarctic plot includes automatic weather stations which are known to be problematic, I reworked the calculations using only manned surface station data below.

It’s surprising to see so little effect on the final trend. The story isn’t finished however, there is another detial whcih isn’t terribly minor that we’ll cover tomorrow. I’m tired now so it will have to wait.