the Air Vent

Because the world needs another opinion

Antarctic Warming – The Final Straw

Posted by Jeff Id on June 7, 2009

This is the first post I’ve done which gets to the heart of where the trends in Steig et al. came from. Steve M did a post on TTLS reconstruction TTLS in a Steig Context which makes the point that despite the PCA and truncation the result of RegEM is still a linear recombination of station data. This post is the result of a back calculation of station weights to determine which stations were weighted and by how much to create the final trend of Steig et al.

Before I succeeded in this calculation yesterday, I tried it once before some time ago and it didn’t work. There were a couple of errors which prevented me from getting a solution and I was too lazy to fix it. The Climate Audit post pushed me to try again and this time I got it right. I think you’ll find the result a bit telling.

The satellite reconstruction from Steig et al is based on two halves. The pre-1982 half is entirely surface station data, the post 1982 data is satellite based data. The satellite half is easily replicated from the satellite data while the surface station half is simply a linear weight and sum of the surface stations. If the surface station temperature is SST, and the weights are c the net result of all this complex math prior to 1982 looks like this

T output = (C1 * SST1) + (C2 * SST2) ……. (Cn * SSTn)

That’s it!

So in order to calculate the C’s involved in this equation we can back solve a series of linear equations having the form above. There are 42 SST’s in the reconstruction and 1 Satellite trend. Since the satellite is not used pre-1982 we can ignore that for determining the pre-1982 portion of the reconstruction. So we have 42 SST’s but not all of those have any data before 1982. After removing the stations which don’t have any pre-1982 data only 34 remain. These 34 are the only ones mathematically incorporated in the reconstruction and are shown in Figure 1.

Station Location (34 SST)

Figure 1 - Location of 34 Stations Used in Reconstruction

It’s odd that Steig et al included the extra stations at all. I’m not sure if they understood what they were doing when they included stations which had no data in the pre-1982 timeframe. I need to run RegEM without them to see for sure but they may affect the weightings of the other 34 stations but IMO it isn’t likely to be helpful.

The code to perform the reconstruction and sort the correct 34 stations out is as follows:

#perform RyanO SteveM RegEM reconstruction
clip=form.steig
dat=window(calc.anom(all.avhrr), start=c(1982), end=c(2006, 12))
base=window(parse.gnd.data(all.gnd, form=clip), start=1957, end=c(2006, 12))
base=calc.anom(base)
pcs=get.PCs(dat,3)
dd=ts.union(base,pcs[[1]])
reg3=regem.pttls(dd,maxiter=50,tol=.005,regpar=3,method=”default”, startmethod=”zero”, p.info=”Unspecified Matrix”)
dim(reg3[[35]]$X) #600 45

#extract surfacestations and PC’s
regemSST=ts(reg3[[35]]$X[,1:42],start=1957,deltat=1/12)
dim(regemSST) #600 42
regemPC=ts(reg3[[35]]$X[,43:45],start=1957,deltat=1/12)

#calculate full reconstruction
recon = regemPC %*%t(pcs[[2]])
recontr=ts(rowMeans(recon),start=1957,deltat=1/12)
coef(lsfit(time(recontr),recontr))[2] #0.01190505

##find stations which have data pre 1982
mask=colSums(!is.na(base[1:300,]))!=0
sum(mask) #34
sstd=base[1:300,mask]

reconSST=base[,mask]## these stations are actually used in recon
nam=as.character(all.idx[1:42,1][mask])
lats=all.idx[[2]][1:42][mask]
lons=all.idx[[3]][1:42][mask]

After the 34 stations are sorted the task is to set up a matrix which has the form of the equation above.

c1 * SST1(x) + C2 * SST2 (x) …… = output(x)

Where x is the value of each surface station and RegEM output on that particular date. Since we have 34 unknowns we need 34 independent equations to solve. All the SST data has values infilled for all dates from 1957 – 2007 but the infilled values are combinations of the non-infilled values. This makes the matrix singular and indeterminate (unsolvable). Our task then is to find 1 row (date) for each station for which the station has have at least 1 unique measured value. To do this I used the raw data and looked for independent months which contain at least 1 value for each row. (this is where I got lazy last time)

##backsolve regem weights
##find unique rows which have 1 value for each station

index=array(0,dim=34)
for(i in 1:34)
{
j=1
while( (is.na(sstd[j,i]) == TRUE) | (sum(index==j)!=0) )
{
j=j+1
}
index[i]=j
}

##use index rows to backsolve RegEM: Index =
# [1] 65 1 109 2 135 3 52 26 4 25 5 6 7 8 9 171 10 165 148 11
# [21] 12 13 74 292 50 73 14 240 280 275 15 16 27 17
##setup square matrix a from infilled data

The value index listed in the code above is the row (month) number from jan 1957 = 1 forward for which at least 1 value was measured. You can see the first station on the list has a value of month 65 for the starting value, the second has a value of 1 which means the second station has data for the first month. The fourth station has a value in the first month but we can’t reuse the same value or the matrix would be singular so it found the next open value at month 2. The algorithm continues in this fashon through the 34 stations.

After these values are gathered we can set up the matrix and solve the following equation for c.

a * c = b

I like simple. The code looks like this.

##setup square matrix a from infilled data

a = regemSST[index,mask]
dim(a) #34,34
b=recontr[index]
c= solve(a,b)
aa=regemSST[,mask]
#a%*%c
m = aa %*% c
m=ts(m,start=1957,deltat=1/12)

The matrix aa is multiplied times weights c to create the surface station temperature reconstruction m. Here is the replicated trend by RegEM we’ve seen before, thanks primarily to Ryan O and SteveM code.

Steig Recon Trend Replication

Figure 2

For the first time we can see the Steig et al reconstruction as determined by the surface station temperatures only.

Surface station trend

Figure 3

I was a bit shocked the trend was still so high. After all we know the area weighted surface station trend sits at about 0.04 C/Decade.

Just to make it clear, Figure 4 is the difference between the above plots.

Figure 4

Figure 4

The pre-1982 data is a perfect match up to rounding error the post 1982 difference is the satellite data difference which I have to point out boosts the final recon trend a bit higher than the weighted surface stations. The surface stations and weights “C’s” required to recreate the pre – 1982 Steig reconstruction are in Table 1.

trd weight

Now we get to the fun part. Surface station weights for this reconstruction are shown in Figure 5. The graph is color coded the same as Figure 1 by region. I’ve moved Byrd from Ross Ice shelf to West Antarctica which is the only change from Ryan O’s color coding in his posts.

Bar Plot station weights

Figure 5

You can see the dominant number of (black) surface stations located in the peninsula. The Y axis is normalized to 1 equals 1/34 of the total contribution for 1 in 34 stations. This area is of course known to have high warming trend, however 4 stations have strong negative net weights – an oddity I mentioned in my earlier work on this paper explaining RegEM ignores trend in favor of high frequency correlation. It is of course nonsensical to flip temperature data upside down when averaging but that is exactly what Steig et al does. This alone should call into question the paper’s result.

This isn’t the end of the story however, in Figure 6 I multiplied the individual (infilled by RegEM) station trends times their weights and created another bar plot

Bar Plot station weights trends

Figure 6

Ok, at this point my eyes are widening. Figure 6 represents the contribution of each stations trend to the positive total output trend. Negative values here are acceptable if they come from negative trend, so the 4 black bars and one near zero blue which were negative in Figure 5 are incorrect, and the ones which changed sign for Figure 6 are a result of a truly negative trend in temperature.

Figure 7 is a Pie chart showing the station weights for each region- same as Figure 5 – different colors.

PIE station weights

Figure 7

It’s telling in Figure 7 that station weights for the tiny peninsula region were not contained well spatially in that the sum of the weights adds up to an area equal to the entire East Antarctica. A correct reconstruction would contain this information to a section of the pie reasonably equivalent to the geographic area of coverage.

And finally the graph we’ve all been looking for since this all started, the contribution of each region to the total reconstruction trend.

PIE station weights trends

Figure 8

There it is, we can now say conclusively that the positive trend in the Antarctic reconstruction comes primarily from the well known peninsula warming trend.

If we recall Figure 3 is the actual Steig et al reconstruction using both pre and post 1982 surface station data only and yet the trend is nearly the same as the final RegEM. This trend is quite different from simple methods of determining station weights using methods such as these.

Maximum Triviality Reconstruction

Closest Station Antarctic Reconstruction

My final check was to add up the area contribution to trends as a check. These values created Figure 8. The four values in order are from Peninsula, West Antarctica, Ross Ice Shelf, East Antarctica in degrees C/Decade:

0.0709 + 0.0115 + 0.0028 + 0.0134 = 0.0987

This was in fact an exact match (7 figures) of the trend in Figure 3 above. Demonstrating the correctness of the last equation in Steve McIntyre’s CA post linked above.

It will be interesting to see how well RyanO’s latest holds up to the same analysis – don’t expect any favoritism around here ;).


92 Responses to “Antarctic Warming – The Final Straw”

  1. pyromancer76 said

    Enjoyed reading the analysis of the data. Figure 8 is beautiful. Now we need Steig’s reply!

  2. curious said

    Jeff et al – Look forwards to reading in detail later but in the meantime I think we have another set of coaster graphics for Lucia! 🙂

  3. Re #1, I wouldn’t be holding my breath for that, if I were you.
    The truncation of the “debate” over at Unreal Climate shows how keen they are to discuss this.

  4. Ryan O said

    Looks like I need to fix Byrd in mine, too. 🙂
    .
    I’m interested to see what mine comes out as.
    .
    BTW, I never really explained this well before, but the “model frame” reconstructions are exactly the same as the one you did based on surface stations only. The “model frame” one (which I now call the “unspliced solution”) does not retain any satellite data – only station data.

  5. Fluffy Clouds (Tim L) said

    This is the most damming of every thing done so far… no really it is, and harder to explain out of!!!!!!!

    I am going to E-mail Anthony on this unless you want to wait to see how Ryan’s works out for conformation?

    A link for Jeff to read:
    http://mathforum.org/library/drmath/view/58335.html

  6. timetochooseagain said

    Tim L, Damming is what beavers do. Damning is what is going on here.

    By the way, guys, I’m still hoping for a layman’s explanation of this whole episode when it’s all done. So far this is all going over my head.

  7. neill said

    ditto.

    while my appreciation at the transparency for replication is huge, this is a language I no speak.

    translation, in the “for dummies” dialect?

  8. Bob D said

    Jeff,

    This is fantastic work, thanks very much. The title could just as well have been “The final nail”.

    Could I make a wee suggestion? It would be useful if you included a graphic similar to Figure 1 that shows the actual Antarctic regions you used split up using the same colours you’ve used in your Figs 7 & 8. That’ll be just the thing so people unfamiliar with the Antarctic geography & regions can compare the weightings against the map at a glance.

    I threw an example together of what I mean.
    Antarctica regions
    [IMG]http://i39.tinypic.com/3092c8y.jpg[/IMG]

    All that’ll then be required is an overlay picture of the “known” temperature trends across the regions (not sure which one you could use, but perhaps this one), showing that the warmer areas have been favoured.

    I think that’s something the average reader who hasn’t been following this in detail can understand immediately: actual temperature distribution, regions, weightings.

  9. Bob D said

    Well, that didn’t work. Here are those links again:
    Antarctic regions
    and
    Antarctic temperature trends

  10. Jeff Id said

    #4 I think the higher PC reconstruction will have more appropriate weighting but we’ll still get some negative coefficients. I emailed before about trying to get this out of RegEM but this is why your reconstruction is better.

  11. Jeff Id said

    #5, This was my point before. From your article.

    “I should mention that this is only a ‘rule of thumb’, and it sometimes underestimates the precision of an answer. ”

    Although you may have sent the article for that reason.

  12. Ryan O said

    #10 I put up a post here: http://www.climateaudit.org/?p=6071#comment-344422

    Strong negative weights may indicate that RegEM can’t find a combination of eigenvectors that reproduce the station temperature – not necessarily that the correlation is all high-freq. I agree with you that RegEM gobbles up the high-freq stuff at the expense of the low-freq trend, but there may be more to it than that. I will muse on this one for a while.

  13. Kenneth Fritsch said

    Jeff ID, it makes intuitive sense to me what you have found, but I am not in a position to check your linear algebra. The fact that you all have completed a heavy load of analyses/sensitivity tests which are apparently in good agreement presents a good case for (tentative?) conclusions.

    I would suspect if the comprehensive analyses completed here and at CA hold up it will strongly indicate, at least to me, that the Steig authors were not exactly sure of their methods, but were less inclined to do a similar comprehensive analysis with the conclusion that popped out being so much to their liking.

    It would not surprise me to hear in the not so distance future that the Steig paper suffered more from poor data from the Antarctica than the methods applied and now that the models are on track we can use those results to determine warming trends -sans good observed data.

    Good work Jeff ID. Completing this phase of the analysis must have meant that you were not able to watch the Red Wings. They won handily and did what I suspected they might and that was to “break” the Penguin goalie. Those rushes down the ice remind me of what the Montreal Canadians used to do back in the days.

  14. MikeN said

    It could have been worse. From Table 1, the largest trend areas are being given the lowest weights. Still the products dominate.

  15. MrPete said

    Is this worth writing up for a journal?

  16. wattsupwiththat said

    I’ve tried several times to pose the question of “Why do the peninsula stations weight more heavily than those of the continent”, referring to this article, on the new RC where they ask for fresh input and ideas.

    http://www.realclimate.org/index.php/archives/2009/06/groundhog-day-2/

    After 4 post attempts over as many hours, plus a personal email to Gavin, none of my comments have made it. Perhaps readers here would care to give it a go?

    Its a valid question, and one that deserves their attention. – Anthony

  17. page48 said

    RE: #16, “I’ve tried several times to pose the question of “Why do the peninsula stations weight more heavily than those of the continent”, referring to this article, on the new RC where they ask for fresh input and ideas.”

    Because the peninsula stations are warmer.
    Did I mention that I’m a cynic?

  18. Carrick said

    #15, MrPete:

    Is this worth writing up for a journal?

    As a short letter, yes this almost certainly publishable.

  19. Jeff C. said

    Great work Jeff! That was a long a painful trip. We all knew it, but it sure is nice to see it laid out in plain sight.

    I was trolling back through the archives and found this comment back on February 14, 2009:

    https://noconsensus.wordpress.com/2009/02/13/antarctic-temperature-regem-forensics/#comment-2370

    Jeff-

    You getting this to work has really opened my eyes as to what might be going on here.

    Steig says on page 2 of the paper:

    “Unlike simple distance-weighting or similar calculations, application of RegEM takes into account temporal changes in the spatial covariance pattern, which depend on the relative importance of differing influences on Antarctic temperature at a given time.”

    What he fails to state is that regEM uses *no* distance weighting. RegEm has no information regarding the lat/long of either the 42 manned stations or the 63 AWS or how far apart they are. It can draw conclusions based on the similarity in the temperature trend patterns from site to site, but that is about it.

    This is important because the peninsula is known to be warming, yet only constitutes a small percentage of the overall land mass (less than 5%). Despite this, 15 of the 42 manned stations used in the reconstruction are on the peninsula! RegEm doesn’t know this. All it knows is that a lot of stations have a warming trend and infills accordingly.

    That comment was the rationale behind the original “gridded” AWS reconstruction 113 days ago. It was fun to go back and reread that comment thread.

  20. Jeff C. said

    I just read over the post for the fourth time, it really is damning. Amazing job and some clever work de-embedding the weights. Over the last five months, the word “retraction” has been bandied about over and over, but I’m finally becoming a believer. I can’t see any credible response. An area that is known to be warming and constitutes just a few per-cent of the total land area accounts for 71% of the continental trend. End of story.

    The beauty in this analysis is that it is simple, elegant and easy to understand. No regpar, no PCs, no overfitting, or discussions of cloud masking. This is comparable to Steve Mc showing that red noise made hockey sticks in the Mann-o-matic. They can’t respond by baffling with bullshit.

    I think it is finally time for a formal response.

    Again, nice job.

  21. Jeff Id said

    Thanks Jeff and everyone else, I don’t know how they can respond. Anthony Watts was trying to get a message through but it’s truly beyond me how RC could answer. I’ve tried to leave a comment myself but halfway through writing it, I wonder what to even ask.

    SteveM just emailed me that he had the weights calculated months ago as the B matrix in RegEM. I couldn’t figure out how to get it directly so there was no choice but to backsolve.

    —————
    Can you imagine what TCO would do to this post? hehe.

  22. Layman Lurker said

    Jeff, correct me if I am wrong but my take on the cause of heavy peninsula weighting is as follows:

    1. Peninsula stations are close together and therefore have highly correlated HF fluctuations. Other stations are not as close together and therefore the HF correlations are weaker. The “trend” is but a passive “passenger” – it is the strong correlations of these stations that is the vehicle to deliver the warming trend to the rest of the continent. A very different scenario could have occured had there been a cluster of very close, correlated stations around the cooling south pole (instead of the peninsula) for example – something which a valid model for reconstruction should be insensitive to.

    2. The 3 PC processing opened the door for the trend from the highest correlated stations to be distributed without constraint onto the greater continent. Ryan’s higher order processing squished the peninsula trend into smaller space. However, it does not change the underlying reality that the peninsula stations have the strongest correlations.

    If you imposed external weighting on the peninsula stations (similar to the earlier weighted reconstructions) and plugged it into Ryan’s model, the results might help sort out how much impact comes from the disproportionate peninsula covariances and how much from low order processing.

  23. neill said

    I should have just skipped past all the math, the pie charts tell it all.

  24. BDAABAT said

    Thanks LL for pointing out something that had eluded me until now. I got that the stations are weighted and that the peninsula is more heavily weighted than the rest of the continent. I didn’t realize the reason for their heavy weighting. Sure makes intuitive sense that the stations that are a lot closer together would correlate better. Once those stations are removed (seems one could make a very reasonable argument for excluding those stations since the climate in the peninsula is likely quite different than the climate in the rest of the continent), would bet that the warming disappears completely or is so low as to be meaningless.

    Bruce

  25. Carrick said

    IMO, this was the most damning statement:

    0.0709 + 0.0115 + 0.0028 + 0.0134 = 0.0987

    Take it together with Jeff C’s comments about the problems with RegEM not taking into account distance measurements, and you have a slam dunk.

    I’m sure that RegEM works for the areas it was originally developed, namely to infill dropped pixels in a video camera feed, or in a dense seismic network. But as many of us suspect all along, RegEM should not be applied to infill data for a spare network of sensors.

    All I can say is I’ve about had my fill of “creative results” stemming from misapplication of statistical methods by people who were never formally trained to use those methods properly from the start.

  26. Carrick said

    #24, Bruce I agree with your comments. One should not include the different climate region of the peninsula into temperature history reconstruction of the mainland.

    I also advocate separating the trend pre-1980 and post-1980: 1980 is roughly the breakpoint where the global climate models suggest that the effects from anthropogenic CO2 overwhelmed the cooling effects from anthropogenic sulfate emissions.

    I think what you will find is a slight cooling of the Antarctica mainland post 1980. Whether this is consistent with the attribution of the dominant global warming to anthropogenic activity is for somebody more familiar with the models to say.,,,

  27. BDAABAT said

    Might be helpful to include another graph….. Would bet seeing the area for the most heavily weighted part in this reconstruction (the only part of the Antarctic that extends outside of the Antarctic circle) that is < 800 miles long compared to the rest of the continent (5.5 million square miles!) would be worthwhile.

    Bruce

  28. hswiseman said

    Re 15 Mr. Pete:

    I agree that the antarctic work is journal-worthy, but question the value of yet another climate journal article. How about a statistics journal? This would possibly result in a peer reviewed statistics methodolgy for climate data.

  29. Layman Lurker said

    I think it is not the inclusion of the peninsula per se which smears warming onto the continent, it is the over-representation of peninsula stations vis-a-vis the rest of the continent. If there were 300 surface stations distibuted equitably then the covariances of the peninsula would not displace the rest of the continent.

    A further issue with the above is the assumption that post 1982 covariance is a “constant” which can be used to interpolate from sparse surface station data pre 1982. Simultaneous changes in clouds, ozone, polar vortex, ocean oscilations, sea ice, etc, could affect these covariance relationships over time.

    Even if the covariances are “constant” there are many independant factors which could affect trend that work on low frequency time scales (ocean oscilations for example) and affect distant regions differently. Reduction of ozone could strengthen the polar vortex which is the barrier between the cold polar air and ocean air. At the same time there are oscilations in sea ice and ocean temp which could affect trends at the coast and especially the peninsula oppositely. Sometimes the phases of these long term factors are in sync and sometimes they are out of sync. Do the post 1982 covariances capture these LF fluctuations?

  30. Ryan O said

    Jeff,

    I bet $50 that when you run my reconstruction through the same analysis you will find that the amount of trend weighted by region (second pie chart) is just about the same as Steig’s. 😉

    After looking through this a number of times, I think the interpretation of the second pie chart could use a bit of refinement. Here’s an example to illustrate:

    Take a 10% pie of a circle and give it a trend of 0.5 Deg C/decade. Let the rest of the pie have a zero trend. The trend for the whole pie is then 0.05 Deg C/decade. Let us assume that this is the true trend for the pie.

    Now let’s imagine that we estimate the pie trend by measuring points. For whatever reason, we select half of the points from the high-trend region (let’s call this the “Peninsula”. We do some magic math and end up with a trend of 0.05 Deg C/decade for the whole pie based on our samples.

    If you then compute the trend*weights for each station and plot them in a pie chart to show where the trend comes from, 100% of the pie will be taken up by the Peninsula stations. But that’s okay, since the rest of the pie has no trend. Any deviation from zero by definition must come from the Peninsula.

    I would interpret your pie charts a little differently. It is clear from the second pie chart that the greatest contribution to trend comes from the Peninsula – which is almost Q.E.D. So given that the greatest contribution to trend comes from the Peninsula, then the Steig reconstruction map should show the greatest positive trend in the Peninsula. Of course, we all know that it doesn’t.

    So in the end, what is important – in my opinion – is this:

    1. The first pie chart shows mostly positive weights, approximately equal in magnitude (but with stations that are really close to each other having somewhat smaller weights).

    2. The second pie chart matches the geographic distribution of trends in the reconstruction.

    If this is what you meant, apologies . . . from my read, it seemed like you were interpreting the second pie chart a bit differently.

  31. Jeff Id said

    #30, I’m going to guess about 30% (still high but more correct to reality) instead of 70 – and noway on the 50 bucks. The first pie chart is the real beast where weights should closely correlated to geographic area but the second kind of brings it home. I thought that I was going to do a pie chart of weights/affected area and plot that but I got lazy again.

    However, the point that most of the trend came from the peninsula is valid and the lack of the other graph is directly related to the fact that I wanted to do something else this past weekend. I believe (guessing) your reconstruction will have less of the negative weighting and a much better Fig 7. I also wanted to plot weight at each station on the Antarctic.

    If it has more negative weight thermometers, we’ll need to make some decisions as to what that means at the time.
    ———-

    There are other good comments here that I’ll try to get to later. It’s a crazy Monday.

    Thanks everyone.

  32. John Turner said

    Gavin has briefly responded to this, and it’s priceless:

    The analysis you saw is simply a fishing expedition, an analysis of what the calculation is doing (fair enough), combined with an insinuation that the answer is somehow abnormal or suspicious (not ok). But how is this to be judged? What would be normal? No-one there can say and they would prefer simply to let people jump to conclusions. It’s kinda of typical of their tactics, but not a serious scientific point. – gavin]

    Not a serious scientific point?! How fun.

  33. Ryan O said

    #31 I don’t have $50 to pay you even if I lose. Hahaha. 🙂

    At the moment, I don’t like negative weights. Negative weight*trend is okay, though . . . but if something ends up with a negative weight, my initial thought is that it shouldn’t be included in the first place. Have to think about that one.

    BTW . . . I sent you something yesterday from my yahoo email. Letting you know because I think I forgot to put a subject line on it, so I didn’t want you to think it was spam.

  34. Jeff Id said

    I saw your email and just didn’t have time to reply. I’ll get into it tonight.

  35. Jeff Id said

    Anti-thermometers (negative weight) don’t make sense. If the values are small enough it might not matter. In this case they are large.

  36. Ryan O said

    #36 Anti-thermometers, anti-proxies . . . I wonder if they explode if they touch real thermometers . . .

  37. Jeff Id said

    Check out gavin’s reply in #32.

    I’m a little ticked that he can’t be honest about this. I don’t have time to make the post I want to but when I get a minute its ‘under the bus with ’em’

  38. Ryan O said

    #37 I giggled when I read that. Like I say, I have never had a problem with Steig – he’s always been open and honest during the emails and I think much of his overfitting post is due to not understanding our methods (and not fully understanding his own). But Gavin seems to do this kind of thing on purpose. Maybe I’m wrong . . . but I received the same kind of wishy-washy replies on their original “Antarctic Warming is Robust” topic.

  39. Jeff Id said

    #38, I just left a comment over there. I kept it reasonably calm.

  40. Ryan O said

    #39 Must still be in moderation. Damn the queue!

  41. Carrick said

    There is a certain amount of irony that Gaven seems to feel empowered to issue ad hominem attacks on other people from his comments will banning them for doing the same.

    Seriously, at times I’m wondering if we’re dealing with adults here.

  42. Kenneth Fritsch said

    The analysis you saw is simply a fishing expedition, an analysis of what the calculation is doing (fair enough), combined with an insinuation that the answer is somehow abnormal or suspicious (not ok). But how is this to be judged? What would be normal? No-one there can say and they would prefer simply to let people jump to conclusions. It’s kinda of typical of their tactics, but not a serious scientific point. – gavin]

    This is a serious question: Why do you guys bother wasting your time commenting at RC? I can certainly see reading some of the stuff at RC as it sometimes helps me understand a climate science concept and sometimes better understand a POV of the advocates of AGW and immediate intervention, but I would not expect an advocacy group to discuss these issues as a scientist might.

    Steig chose to discuss a small part of his paper at issue from the friendly confines of RC, where he was better able to perform a professorial lecture and then pick and chose to which of those questions he would reply. That is way out of character in my experiences with scientists who really know their specialties. They cannot be kept from answering questions even when the circumstances may deem otherwise – and they are never hestitant to admit when they do have answers.

  43. Ryan O said

    Jeff, something else I forgot to mention . . .

    I found that including additional stations in the post-1982 timeframe does actually change the results. Basically, the additional stations serve to better constrain the solution. This changes the post-1982 relationship between the stations (as they now have to meet more conditions), and this constraint is naturally carried backwards to 1957.

    So there is a good reason for including as much information as possible, even if it is all post-1982. I can’t think of a reason for, say, excluding the AWS stations for example (except when you exclude them deliberately to use them as a verification target).

    Also, I found more station data not in the READER database. It’s got a bunch of crappy header info, though, so I need to do a separate script to scrape it.

  44. Nic L said

    #19 quotes from Steig’s paper:

    On my understanding of RegEM and from Schneider’s paper, that statement is untrue. RegEM, at least as used by Steig, assumes that the same “spatial” covariance pattern applies at all times in the period being imputed. On that basis, the statement suggests that neither Steig nor any of his co-authors have a very good understanding of RegEM – which is a bit surprising in Mann’s case in particular.

  45. Ryan O said

    Kenneth – Sometimes replying at RC works. Not in the normal way, but in the case of the overfitting topic, our replies ended up preventing that whole discussion from snowballing. Sometimes I just do it to see what kind of answer I will get – like in the Antarctic Warming is Robust thread. When Jeff does it, it’s cool, because if anyone subsequently clicks on his name, it takes them here. 🙂

  46. Ryan O said

    Oh, and Kenneth . . . I still plan on putting together the plots to better explain the post at CA. The words are okay, and kinda-sorta answer your earlier question, but the plots show it more clearly.

  47. Nic L said

    Sorry, I’ll try again

    #19 quotes from Steig’s paper:

    Unlike simple distance-weighting or similar calculations, application of RegEM takes into account temporal changes in the spatial covariance pattern, which depend on the relative importance of differing influences on Antarctic temperature at a given time.

    On my understanding of RegEM and from Schneider’s paper, that statement is untrue. RegEM, at least as used by Steig, assumes that the same “spatial” covariance pattern applies at all times in the period being imputed. On that basis, the statement suggests that neither Steig nor any of his co-authors have a very good understanding of RegEM – which is a bit surprising in Mann’s case in particular.

  48. Ryan O said

    #44 Yep, exactly. Steig’s method (which retains the post-1982 PCs) is only valid if the covariance does not change. If the real covariance changes (and the ground station data proves that it does), then you can’t retain the post-1982 PCs. It’s apples and oranges.

  49. Ryan O said

    #44 Damn. Hit . Stupid human.

    Anyway, what I meant to close with is that Steig’s statement from the paper is actually true. You can see it quite clearly in our 13-PC reconstruction: the PCs use the ground stations as anchors. The reconstruction trends follow the ground stations.

    So RegEM most certainly can adjust for changes in covariance . . . but the way Steig did his reconstruction is only valid if the covariance remains constant.

  50. Ryan O said

    That was supposed to read “Hit RETURN”, but WordPress swallowed it.

    And now I’m spamming Jeff’s thread as badly as TCO . . .

  51. Jeff Id said

    It didn’t go through moderation.

  52. Ryan O said

    It appears, then, that their desire for cooperative learning was short-lived.

  53. Carrick said

    Nic L:

    On that basis, the statement suggests that neither Steig nor any of his co-authors have a very good understanding of RegEM – which is a bit surprising in Mann’s case in particular.

    Based on Michael Mann’s background, how is this surprising? Unless you are referring to the density of his publications rather than the sparsity of his training in statistical methods, that is….

  54. Kenneth Fritsch said

    and they are never hestitant to admit when they do have answers.

    Should have been “when they do not have ansewers” obviously.

    The first part of Gavin’s comment (left out of the above excerpt) given below is telling

    The point of the Steig et al paper was to use spatial correlations in recent data to look at how under-sampled parts of the continent likely changed over longer time periods. Those correlations will necessarily weight different stations differently as based on the physical characteristics.

    That is a political answer that says nothing about correctness of the method or any biases it might introduce. If I may interpret it says:

    “Well, of course, we know that the methods are correct and properly applied and therefore if the weightings appear biased this is only because you were searching for a result to indicate that biases exists and then run with it as innuendo. The correct results, whatever the bias, must then have physical significance.” If that sounds like circular reasoning that is only because you are a dolt.

    Actually, in my view, you need the spatial correlations of the AVHRR data to look at the “under-sampled” parts of the Antarctica with the surface station data being spread to the rest of Antarctica through the AVHRR grids. That is the novel part of this reconstruction that we all seem to be able to appreciate – on its face. Not to mention the validity/invalidity of the AVHRR data, the differences in trends of the AVHRR data with the surface station data and that the AVHRR data are being used to represent the 1982-2006 part of the reconstruction as a splice is telling.

    Ryan O, I can understand why you and others involved in these analyses would want to, at least, make an attempt to show at RC what you have learned. It is that natural urge that good scientists and engineers have to offer their knowledge that I reference above. It also gives you more adversarial feedback than what you might be exposed to at friendlier confines. Here is my question that will probably be answered when you put forth your Ryan O versus Steig comparandum: Do you judge that you were being engaged on a scientific level with Steig and if so can you separate for me the wheat from chaff, i.e. legitimately informed science based replies from hand waving?

  55. Ryan O said

    #54 I deliberately avoided any discussion like that with Steig. Instead, I asked questions related to “how” rather than “why”. It was more important to me to make sure I understand what he had done. The why would come out in the wash. So I don’t have any information about whether he would engage me on a scientific level in the emails.
    .
    At RC, I think the rapid-fire series of responses from everyone helped prevent serious engagement. I do not believe that his answers at RC were engaging me in a debate; they seemed more to be deflecting and stalling. Those are both understandable, given that Jeff, Jeff, Roman, Steve, me . . . and lots of others . . . have built on each other’s knowledge on posts that are spread across no less than 5 blogs. I think it likely that Steig only obtained a partial understanding of what we did. With that being said, perhaps the choice to post on it was not well-conceived, but that’s a separate issue. Anyway, long-winded way to answer “no” to your question. Haha. 🙂

  56. Layman Lurker said

    #55

    Jeff speculated that it may have been Mann rather than Steig who did the modelling and the math. Of course no one really knows, but given the post itself and some of his responses he didn’t really come accross as someone who was imersed in the data and the method.

  57. rephelan said

    OK, not that I miss him that much, but where is TCO and what have you guys done to him (them)?

  58. Fluffy Clouds (Tim L) said

    Layman Lurker said
    June 8, 2009 at 2:02 am

    Jeff, correct me if I am wrong but my take on the cause of heavy peninsula weighting is as follows:

    1. Peninsula stations are close together and therefore have highly correlated HF fluctuations. Other stations are not as close together and therefore the HF correlations are weaker. The “trend” is but a passive “passenger” – it is the strong correlations of these stations that is the vehicle to deliver the warming trend to the rest of the continent.

    Layman Lurker, I say this is, SPOT ON!!!

    it “looks for sameness” when it found it, it spread it around!
    You are good on making this into lowest thinking lol but at a high level!

    Thank you LL

    any one trying to post this at RC is deluded LMAO! there is no way to bounce this off of the shoulder with out looking very bad.
    reminds me of ieee classes( http://en.wikipedia.org/wiki/Ieee ) when they talked about the “KISS” principal! Keep It Simple Stupid!

    57 post (all good ones) Jeff you know your on to it when you brake 40!
    agent 86 would say ” loving every minute of it”
    side walk cricket out.

  59. Fluffy Clouds (Tim L) said

    PSSSt hey they let this pass lol ya my post!!!

    #42 Tim L Says:
    8 June 2009 at 12:53 AM

    Looking forward to this! thank you

    “”the need for our group to have a “functioning” and reasonably “realistic climate model” with which to start the new round of simulations.””

  60. Bob D said

    rephelan said
    June 9, 2009 at 2:12 am
    OK, not that I miss him that much, but where is TCO and what have you guys done to him (them)?

    “Publish, you grasshopper!”

    Good times. Here’s to you TCO, whichever rock you’re lurking under, muttering to yourself.

    We shouldn’t encourage him though – he may come back.

  61. This is what I wrote, inspired, when this article was posted at WUWT:-

    Many ordinary folk have no idea of all the work here. They have no idea that there is a substantial and growing number of scientists who dissent from AGW. They still carry the hockey stick image around in their minds. They have no idea that nothing but nothing strange is happening weather-wise. They have no idea that there are perfectly competent (well, better) alternative scientific explanations for everything but everything put forward as “proof” for AGW. They have no idea that scientists are now obliged to give lip-service to AGW if they want grants, recognition, promotion, etc. They cannot imagine that a science could be co-opted to serve politics in a democracy. They have no idea that many scientists cannot speak up without risk of losing their job, their status, their grant money, their future, even their ability to speak up and be heard. And Science has become so compartmentalized that many scientists who know AGW is phony in one area may still believe the rest. And many scientists cannot believe the peer-review system could have been corrupted. Many would be horrified if they did know all this. Many would find it too much to believe.

    I’d like to see two things. One, an anonymous poll amongst scientists, asking questions about the link between supporting AGW and getting grants, recognition, etc. Two, an FAQ that the whole skeptics’ community can own, that can become widely-known, for all who’ve only heard or believed the AGW science…

    WE NEED TO PUBLISH so that ordinary folks can find it and understand it. The sterling work re Steig et al, done by the two Jeffs, Ryan O, Steve M et al, needs to be published in its own right. But we also need an overall FAQ on skeptical Climate Science that all Climate Skeptics know about, support, and can contribute to (to improve). This means a wiki format with contributors limited to proven skeptics!

    Jeff Alberts, are you listening? I see there is no “contact us” point on your website – it would be helpful to have one – and I’ve lost your email. Are you up to hosting such an FAQ? I realize that the full wiki I dreamed of is too much for me, even an FAQ is too much for me for now – but can you host it? Can you or someone else oversee it?

  62. Page48 said

    RE: #61

    I admire your passion, Lucy, and I agree with you for the most part.

    But, I don’t think a FAQ will convince anyone of anything. It’s worth trying, however.

    None of the public support for AGW measures has anything to do with climate, in my estimation.

    There’s a whole culture of (intentionally cultivated) envy and hatred out there right now. Most who belong probably don’t even know to what extent they have been manipulated. But, more importantly, they have been convinced that some sort of retribution is due for anyone perceived to have more. Most assume that none of the energy measures will affect them. In their minds, they are “getting back” at the perceived evils of our time, Big Oil or Big Energy and/or Big Corporations.

    I am convinced that only when the bills come due to the individuals and the standard of life drops will the faithful start to question what they have allowed.

    Nice post! Thanks

  63. TCO said

    I don’t see why you are calling this the “final straw”. You already SHOWED earlier that a strict (“trivial”) area representation scheme gives a lower trend. Therefore, it’s mathematically required that the Steig work contains a higher proportion of the higher trending stations than their area-based weighting would give.

    You don’t have a stunning “aha” here, grasshopper. Just a more compelling graphic, making the same point as what you had before. And a bunch of breathy enthusiasm like “final straw”.

    Ryan on the other hand has some nice work with his skill scores and raising the number of PCs. You would do better to emulate him, both in thoughtfullness of analysis and in measured tone and avoiding the breathy final straw crap (this is NOT the first post of its sort, where you act like this.)

    When you act like a Rumsfeldian idiot–not only not knowing, but unaware of your degree of not knowing–you make all of us real skeptics look bad.

  64. Layman Lurker said

    The “Chosen One” is back. What have you been up to TCO?

  65. Carrick said

    LOL. Must be fun to be the butt of other people’s jokes, TCO.

  66. Layman Lurker said

    A little gem courtesy Mike Mann at RC this morning (response to Steve Fitzpatrick comment #144):

    “I was encouraged by Dr. Eric Steig’s blog exchange with several people who had analyzed the methods used his Nature paper on temperature trends in Antarctica. By the time Dr. Steig ended the exchange, the tone of the discussion was much more reasonable and constructive than at the beginning, and it appeared that even Dr. Steig agreed that there were some legitimate concerns raised, although he did not agree these concerns brought into question the results shown in the Nature paper.

    [Response: Please don’t misrepresent Eric. You need to read what he wrote more carefully. He did not indicate that there were any “legitimate concerns raised”. Rather, he explained in some detail how the analyzes described on a certain fringe website were rather seriously flawed, e.g. violating the assumption of independence of the statistical cross-validation by adjusting the model to fit the validation data–a major no no, at least to anyone who understands cross-validation. Eric did note that an objective analysis of quality issues with the satellite data would be worthwhile–but that is hardly what was provided in the attempts to attack Steig et al. We closed off the discussion after the post had achieved its end, i.e. when the attackers conceded that indeed they were unable to in any conceivable way ‘falsify’ the Steig et al ‘08 results -mike]”

    Talk about spin. Wow. I think he risks pushing more lurkers away with comments like that. Any “polite” responses from the proprieter of the “fringe website”?

  67. Layman Lurker said

    Sorry, here is the link: http://www.realclimate.org/index.php/archives/2009/06/groundhog-day-2/langswitch_lang/tk#comment-126536

  68. Carrick said

    Michael Mann:

    We closed off the discussion after the post had achieved its end,

    This is where my distaste for Mann comes from. In addition to being somebody who doesn’t know when he doesn’t know something, he is transparently intellectually dishonest.

    I am actually glad that he is on the AGW zealotry side and that they idolize him so much. This undermines their position in ways that no commentary from rational people ever could.

  69. Kenneth Fritsch said

    You already SHOWED earlier that a strict (”trivial”) area representation scheme gives a lower trend. Therefore, it’s mathematically required that the Steig work contains a higher proportion of the higher trending stations than their area-based weighting would give.

    TCO, in an effort to avoid all of your peripheral bull shit and mind games –and responses to it, I have excerpted the pertinent part of your comment here and would like to see you expand on it.

    Here is the essence of what Jeff ID is proposing from the introduction to this thread.

    This post is the result of a back calculation of station weights to determine which stations were weighted and by how much to create the final trend of Steig et al..

    Gavin Schmidt seems to think that physical characteristics are the key, but without identifying what those characteristics might be in the excerpt from RC below. Do you care to expand on the apparent (over) weighting of the Peninsula and then getting the Peninsula trends (admittedly) wrong in the Steig reconstruction?

    Also any ideas on what those physical characteristics might be?

    The point of the Steig et al paper was to use spatial correlations in recent data to look at how under-sampled parts of the continent likely changed over longer time periods. Those correlations will necessarily weight different stations differently as based on the physical characteristics.

  70. TCO said

    Ken,

    Gavin is addressing the same issue as I.

    That it is not A PRIORI determined that the best way to estimate the overall trends and interior trends is by area weighting. It is at least concievable that other predictor schemes (in which case there MUST be some deviation from the trivial area weighting) might give better predictions. I personally have my doubts on this. As I said before, if you are going to open the door to more sophisticated pattern-matching type, multiple component predictors, you ought not to do the 3 PC reduction of the data. (I can’t prove that, but it’s my suspicion/concern/etc and if you read back to the RC original comments has been from the beginning.)

    Note that it is a very different thing to jump up and down on the bed (forgive me) because the approach is different than the trivial (JeffId) as it is to show differences in skill, problems with the non-trivial algorithm (Ryan). One is an approach of dismissing out of hand (including duplications of previous points: pie chart as an aha when it’s the same concept as shown in area weighting earlier). The other is an approach of understanding the aim and just seeing if the sophistication was statistically justified.

    I’m kind of stunned that you’re not at least understanding the terms of discussion. Not to agree with me is fine. But not to even GROCK the concepts? This is what makes things groundhog day and why CA and Heartland are WAY more social phenoms of old war horses than genuine ansalysis and thinking of a Volokh quality.

  71. Jeff Id said

    #70, Welcome back TCO.

  72. TCO said

    I’m not staying back. I need to keep losing weight.

  73. Nic L said

    I think that there may be a problem with the identification of the surface stations in the tables and graphs? Great Wall, KIng_Sejong and Mario_Zucchelli don’t have any pre 1982 data. I think that “base”, from which “mask” is derived in your script, excludes Gough and Marion, whilst “all.idx” (to which you apply [mask]) includes them. I think that all the station names after Faraday are therefore out by one or two stations. I’m not sure if and to what extent this impacts the weighting of the peninsular stations in the trend.

  74. Jeff Id said

    #73 You know what, I think your right. The names are all mixed up, I probably didn’t sort them right. Fortunately Figure 4 checks the calc for me, I’ll need to look at the names tonight.

  75. Nic L said

    #73. I have now made an initial (as yet unchecked) calculation of the effect of the station names mis-identification on the contribution of the peninsular (including island) stations to the total reconstruction trend. I get that it reduces from 71.9% to 49.0%. Still very substantial, but not quite so dominant.

  76. Nic L said

    #74 I think that figure 4 allocates the station weights to regions based on station name, so where the incorrect name is assigned to a station that station weighting (and trend) it may be allocated to the wrong region.

  77. Jeff Id said

    #76 Great, I’ll have to check tonight. How did you calculate a reduced contribution of the peninsula?

  78. Nic L said

    #76 Sorry, I meant figure 5 (and 6 to 8), not figure 4.
    The reconstruction weights by region also change. I think figure 7 is based on absolute weights. I calculate the absolute weights for the peninsular & island stations as reducing from 41.0% to 24.8% when the stations are correctly identified.

  79. Nic L said

    #77 I am sorry to have been the bringer of bad news, after all your hard and excellent work.
    I got a list of the correct names by using nam=as.character(all.idx[c(1:16,18:25,27:44),1][mask]), thereby eliminating unused stations Gough and Marion, and created a 34 long vector e with 1s where the stations in the list were peninsular stations and 0s elsewhere. I then used lm on regemSST, followed by [mask], to get a vector f with the trend for each of the 34 stations, and took sum(f*c*e)/ sum(f*c), using your vector c of the 34 station weights. My script is rather too messy to post, but if you get different answers may I suggest that you email me so that we can resolve the matter.
    There is also another, unrelated, slightly odd result (not directly relevant to this thread) that I am getting using from your script and which (if I cannot resolve it) I would quite like to discuss by email, if you are willing to do so.

  80. Kenneth Fritsch said

    Gavin is addressing the same issue as I.

    That it is not A PRIORI determined that the best way to estimate the overall trends and interior trends is by area weighting. It is at least concievable that other predictor schemes (in which case there MUST be some deviation from the trivial area weighting) might give better predictions.

    TCO, you have avoided answering my questions and offered the above which we all know says absolutely nothing in context of the questions I posed to you.

    When Gavin says “Those correlations will necessarily weight different stations differently as based on the physical characteristics” please explain what, a prior, those physical characteristics are.

    Do you agree that in the Steig reconstruction that the Peninsula trends weight the overall trend far beyond its portion of the overall Antarctica area (less than 5%)? How would explain that weighting other than perhaps a methodology error? Why do you think that the trends in the Steig reconstruction actually get the Peninsula trend wrong as admitted by the authors?

    Please spend more words answering my questions than characterizing your views of the people posting here.

  81. TCO said

    Ken: I’ll guess what Gavin meant after you address the remainder of my most recent post. Tit for that, [snip].

  82. Kenneth Fritsch said

    TCO, for your sake, I hope you are either drunk and/or faint-headed from a lack of food, because you are not making any sense. You make pronouncements without any substance and you apparently cannot answer my questions. You are no better than when you left.

  83. TCO said

    BTW, Ken…I CAN give you a good guess on that particular term of Gavin’s what subconcept he is addressing. (Patterns, etc.) But first you address my main points. You piece of s[edit]

  84. TCO said

    I had a couple sure. But you all f[edit]g make me sick, sober or not. There’s something so hack about you all. Look at Steve touting the Ph.D. stats guys on his blog. And how many mainstream stats articles have they published in the last 4 years on CA concepts? F[edit]g crickets chirping. What a waste. H[edit]

  85. TCO said

    Jeff:

    Sorry for cursing.

  86. Jeff Id said

    TCO, I’m not going to let you dominate threads like the last time you were around. It’s not a matter of too much criticism but the criticism has lost its reasonableness. I noticed two things after the TCO spam stopped.

    1 – There weren’t many people left who could read past the TCO comments. Posts went for days without a single comment.

    2 – People started having reasonable discussions again.

  87. geoff chambers said

    Just to start you off again:
    I mentioned this article on comments to an article by Gavin Schmidt on Guardian Environment. I suggested that it might be nice if the Guardian, instead of leaving discussion to us commenters, organised a real debate between say, Gavin Schmidt and you or Alan Watts. After the usual peerless revue remarks about you and Alan (crap, troofers, creationists etc) I got this, as a reason for not discussing Steig’s results.
    “A mathematical result is not debatable”.

    Then the Guardian (or Gavin) closed comments.

  88. ChrisM said

    The station weights for McMurdo and Scott Base, especially as shown in the station weights times trends graph, show the extent of the data manipulation. These are two bases about 5km apart, that have both been continuously manned there for over 50 years. If these have different trends, as indicated by the data, one cannot legitimately extend a construct “average” of that data to the whole continent.

  89. Jeff Id said

    #88 Some of my station names are wrong due to my switching to RyanO’s code from SteveM which I had been using. Different variable names. The different stations names would result in changes to the bar and pie charts according to NicL above. Still heavy on the peninsula but less so – I haven’t had time to check but he’s sent an email to me which was convincing.

    Sorry for the confusion, it doesn’t affect the weights of the reconstruction or the negative thermometer issue at all. I believe Nic was able to replicate that part to the same result as my own.

  90. curious said

    Hi Jeff – just having a bit of a catch up. I note NicL’s comments and that you acknowledge them in the thread.

    Can I suggest you put an update note up on the lead post pending revised graphics?

  91. MikeN said

    TCO, you seem to think you have come up with some sort of brilliant reasoning, –well of course certain areas are overweighted since a regular average is lower–.

    This is what the Jeff and Ryan said to begin with. Then they deconstructed it, and showed that behind all the math is a different set of weightings, with the Peninsula dominating.

    Suppose I did a temperature of the US, and had Chicago be the dominant factor, and this showed a sharp negative trend over time. Critics will say, that a regular average of temp stations showed a positive trend, or a smaller negative trend, and that maybe I’m overweighting some areas. Then they go thru my math, and said, e wait a minute, this guy is weighting Chicago 75%! Well I could just turn around and say, duh of course some areas get overweighted, the trend is different from a strict average.

  92. I want to start a website where i can put google adds on, i have 10 dollars to spend and i know that isn’t a lot but it is better than those free websites that don’t let you do anything. so how do i get started?.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

 
%d bloggers like this: