A Calmer Conversation with the Nail
Posted by Jeff Id on February 16, 2011
by – Ryan ODonnell
I had made a post concerning the sensitivity of the S09 reconstruction with respect to changes in trend for the underlying surface stations HERE [insert link]. That post was a bit rushed, and the level of explanation I provided was insufficient. Bishop Hill did a better job of explaining it HERE [insert link], and I will use his piecemeal approach here, as it presents the issue very clearly.
The previous post was also incomplete. It did not show how our reconstruction method responds to similar perturbations. I will now rectify that situation.
The sensitivity tests are relatively simple:
1. Add various trends to the raw Peninsula station data
2. Subtract various trends from the raw Peninsula station data
3. Add various trends to the raw spliced Byrd data and Russkaya data
4. Subtract various trends from the raw spliced Byrd data and Russkaya data
For the S09 case, I simply spliced the Byrd data with no offset. For our reconstructions, I spliced them together using the nominal offset determined by RegEM from the original reconstruction.
The purpose of this test is to determine which reconstruction is most responsive to changes in the underlying data. If the reconstruction responds “poorly” (in other words, does not properly respond), then one can conclude that the method results in a reconstruction that is not a “good” representation of the data. If the reconstruction responds “well”, the one can conclude that – for the cases tested – the reconstruction appears to represent the data “well”. Since “good” and “well” are qualitative, rather than try to assign criteria to them, we will simply compare whether the S09 reconstruction or ours does better.
First, let’s take a look at where these stations are:
In the above graphic, the circles represent the stations used by S09 as predictors. The crosses represent stations used by us as predictors. The triangles represent stations not used as predictors for either study, but were used by us for additional verification. Now that we know where the stations are, let’s take a look at the original S09 reconstruction, and our newly published one:
EXPERIMENT 1: ADDING TREND TO PENINSULA STATIONS
For this experiment, we will add trends to the Peninsula stations, which are the stations in the light purple from the first graphic. Let’s see what happens when we do this:
When we do this, we note an interesting thing: adding a trend to the Peninsula stations using the S09 method results in an increase in reconstruction warming in West Antarctica. In fact, the change in the West Antarctic warming (+0.07) is greater than the change in the Peninsula warming (+0.05), and very nearly equal to the change in the East Antarctic warming (+0.04).
Now what happens to our reconstruction? Well, the change in warming in West Antarctica is only +0.01, there is no change in the East Antarctic trend, and the Peninsula trend increases by +0.12. At the actual station locations, the trend is +0.18 . . . which is quite close to the +0.20 we added to the raw data. It is quite clear that our method captures both the magnitude and location of the change much better than the S09 method.
These are slightly different than the results I posted in the “Steig’s Trick” thread, because earlier I added the trends after infilling the missing station data, and then fed these back into RegEM for the reconstruction. As the reconstruction method technically includes the infilling step, a more accurate sensitivity test is to add the trends prior to infilling, which was what I have done above.
EXPERIMENT 2: SUBTRACTING TREND FROM PENINSULA STATIONS
For this experiment, we will subtract trends to the Peninsula stations. Let’s see what happens when we do this:
When we subtract trend from the Peninsula stations, we can note similar, interesting things. For the S09 reconstruction, the Peninsula and West Antarctic trends hardly change at all – a net of 0 for the Peninsula for the experiment, and a net of -0.03 for West. The biggest change actually occurs in East Antarctica . . . where the trend is nearly halved.
For our reconstruction, however, the trend in both East Antarctica and West Antarctica is unchanged. The only regional trend that changes is the Peninsula trend, by -0.15.
EXPERIMENT 3: ADDING TREND TO BYRD / RUSSKAYA
For this experiment, we will attempt to determine how well the reconstructions respond to changes in the West Antarctic data. We will do this by adding trends to Russkaya and Byrd. Let’s examine what happens:
As before, it is quite clear that the S09 method responds poorly. In this case, adding significant trends to West Antarctic stations results in . . . well, no change at all.
For our reconstructions, however, the change is quite dramatic . . . and highlights the problem with the way the AVHRR spatial structure is ignored in S09. By regressing solely against the temporal information in the satellite data – without accounting for where that information primarily came from geographically, the S09 reconstruction cannot properly locate the trend. In our case, because we explicitly use the satellite spatial structure in the regression, the spatial sensitivity is rather good.
EXPERIMENT 4: SUBTRACTING TREND FROM BYRD / RUSSKAYA
Continuing with our attempt to determine how well the reconstructions respond to changes in the West Antarctic data, let’s look at what happens when we subtract trends from the West Antarctic stations:
In this case, the S09 reconstruction does eventually respond . . . with the West Antarctic trend dropping by 0.01 after a -0.40 trend is added to the West Antarctic station data. However, one would have a hard time qualifying that as a “good” response.
For our reconstruction, the response is quite good. If a -0.40 trend is added to Byrd and Russkaya, the West Antarctic trend drops to 0, and the point estimate at Byrd is about -0.25.
At this point, I will reprint something I said to Gavin at RealClimate here on March 3rd, 2009:
The comment about whether 3 or 4 PCs would have changed anything is irrelevant. The only relevant question to ask is “How many PCs would be required to properly capture the geographic distribution of temperature trends?” with the followup question of, “Does this number of PCs result in inclusion of an undesirable magnitude of mathematical artifacts and noise?”
If the answer to the followup is “yes”, then the conclusion is that the analysis is not powerful enough to geographically discriminate temperature trends on the level claimed in the paper. It doesn’t mean the authors did anything wrong or inappropriate in the analysis (I am NOT saying or implying anything of the sort). It simply means that the conclusion of heretofore unnoticed West Antarctic warming overreaches the analysis.
In answer to your question about how much variation they need to explain, that is entirely dependent on the level of detail they want to find. If they wish to have enough detail to properly discriminate between West Antarctic warming and peninsula warming, then they must use an appropriate number of PCs. If they wish to simply confirm that, on average, the continent appears to be warming without being able to discriminate between which parts are warming, then the number of PCs required is correspondingly less.