## Significance?

Posted by Jeff Id on August 5, 2009

UPDATED BELOW:

An interesting development in the Antarctic. It appears Dr. Hu McCulloch who had such difficulty getting even a comment through moderation at RC was correct when he wrote this post for Climate Audit.

## Steig 2009’s Non-Correction for Serial Correlation

Hu Wrote:

Steig

et al.report a 95% CI of ± 0.07 °C/decade for this trend coefficient. Assuming a t critical value of approximately 2, this means that they are estimating the se to be about 0.035. Using the above data, the OLS standard error (“seOLS”) is 0.0330. Twice this value is within rounding error of 0.07. It indicates a very high level of significance (p = .0004).However, the first order serial correlation of the residuals, r

_{1}, is 0.318, and is highly significant (p = .0000). With this serial correlation, the standard AR(1)-corrected standard error (“seAR”) is 0.0458, i.e. 44% larger. This leaves the slope still highly significant (p = .0103), but not nearly as significant as is implied by the published CI. The simpler approximation presented by Santeret al., which is based on a 1952 formula by Quenouille and therefore identified here as “seQ” (see below for details), gives the same numerical value as seAR, namely 0.0458. Using either seAR or seQ, the corrected 95% CI is ± 1.964(0.0458) = 0.0900, which is clear inconsistent with the 0.07 reported by Steiget al.Table 1 below reports similar figures for all the 1957-2006 TIR reconstruction trends for which Steig

et al.report numerical values. Implicit Steig standard errors, obtained by dividing their reported 95% CI’s by 2, are given in square brackets [ ]. p-values are given in parentheses below the respective standard errors. Slopes and standard errors are all in °C/decade.Table 1: 1957-2006

RegionSteig

slopeSteig

[se]OLS

slopeseOLS

(p)r_{1}seAR

(p)seQAll Ant..12 [.035] .1178 .0330

(.0004).318 .0458

(.0103).0458 Peninsula.11 [.02] .1196 .0198

.0000.262 .0259

(.0000).0259 West Ant..17 [.03] .1839 .0311

(.0000).312 .0429

(.0000).0430 East Ant..10 [.035] .0984 .0374

(.0087).290 .0504

(.0512).0504 In all four cases, Steig

et al.are clearly using seOLS, not seQ or seAR. In the caption to Figure S4 of their online SI, the authors state that

Black lines separate areas of significant vs. insignificant trends (>95% confidence based on two-tailedt-test with number of degrees of freedom adjusted for temporal autocorrelation). (SI p. 4)The reference to adjusting the number of degrees of freedom would imply that they are using the Santer/Quenouille seQ adjustment. However, the confidence intervals they report in fact show no correction for serial correlation, by either method. The article’s Figure 3, showing regions of significant and insignificant trends, is therefore erroneous.

Today at Nature an addendum confirming Dr. McCulloch’s claims that autocorrelation had not been accounted for was released with the following text:

In this Letter, we reported trends on reconstructed temperature histories for different areas of the Antarctic continent. The confidence levels on the trends, as given in the text,

did not take into account the reduced degrees of freedom in the time series due to autocorrelation.We report in Table 1 the corrected values, based on a two-tailedt-test, with the number of degrees of freedom adjusted for autocorrelation, usingN_{effective}=N(1 –r)/(1 +r), in whichNis the sample size andris the lag-1 autocorrelation coefficient of the residuals of the detrended time series. The median ofris 0.27, resulting in a reduction in the degrees of freedom fromN= 600 toN_{effective}= 345 for the monthly time series.We also include results of a further calculation that takes into account both the variance and the uncertainty in the reconstructed temperatures. We performed Monte-Carlo simulations of the reconstructed temperatures using a Gaussian distribution with variance equal to the unresolved variance from the split calibration/verification tests described in the paper. Confidence bounds were obtained by detrending each simulation and obtaining the lag-1 autocorrelation coefficient and variance of the residuals; a random realization of Gaussian noise having the same lag-1 autocorrelation coefficient and variance was then added to the trend, and a new trend was calculated. The 2.5th and 97.5th percentiles of the 10,000 simulated trends give the 95% confidence bounds. For the case of zero unresolved variance, this calculation converges on the same value as the two-tailed

t-test, above. The 95% confidence minimum trend value is given by the 5th percentile values of the simulated trends, last row of Table 1.The corrected confidence levels do not change the assessed significance of trends, nor any of the primary conclusions of the paper. We also note that there is a typographical error in Supplementary Table 1: the correct location of Automatic Weather Station ‘Harry’ is 83.0° S, 238.6° E. The position of this station on the maps in the paper is correct.

What’s interesting however is the following table:

So now we have an antarctic temperature of 0.12 +/- 0.10C/Decade correcting for autocorrelation expanded from 0.12 +/- 0.07 C/Decade which was originally reported and an additional expansion for unresolved variance to 0.12 +/- 0.12C/Decade. No mention of Dr. McCulloch, however, in the report to Nature.

Oddly enough this means the difference between the trend for the Antarctic as determined by area weighted ground stations and improved regression analysis of about 0.05C/Decade no longer outside the significance intervals for the paper. I wonder if it would have made the cover of Nature with a claim of 0.12 C/decade +/- 0.12 C/decade!

Hat tip to P – for showing the article and Realist for the link.

====================================================

UPDATE: The Steig et al. Corregendum results are almost exactly the same as Hu’s.

Using seAR values Hu numbers are on the left Dr. Steig’s revised numbers are on the right:

all Ant = 0.0458 x 1.964 = **0.09** —-Steig’s new AR version = **.10**

peninsula = 0.0259 x 1.964 = **0.05** —- Steig’s new AR version = **.05**

west = 0.0429 x 1.964 = **0.084** Steig’s new AR version = **.09**

east = 0.0504 x 1.964 = **0.10** Steig’s new AR version = **.10**

UPDATE 2

Hu made them aware of the required changes in a personal email in February.

— From a comment at CA —

Although Steig & Co. may not read CA on a regular basis, I did call their attention to this post in the following e-mail, dated 2/28/09:

Date: Sat, 28 Feb 2009 15:51:44 -0500

To: steig, dschneid, srutherford,mann, josefino.c.comiso,Drew.T.Shindell

From: Hu McCulloch

Subject: Comment on serial correlation in Steig et al 2009Dear Dr. Steig and co-authors,

FYI, I have recently posted a comment on your 2009 paper in Nature

on Climate Audit, at http://www.climateaudit.org/?p=5341 .

While I was able to replicate or virtually replicate the 1957-2006 trends you report

on p 460 for the three regions and the continent as a whole, the 95% Confidence

Intervals you report appear to have taken no account of serial correlation

in the regression errors. When this is done, the CI’s are substantially wider

than you report.

Any reactions, by comments there or by e-mail, would be welcome!

— Hu McCullochJ. Huston McCulloch

Economics Dept. voice (614) 292-0382

Ohio State Univ. FAX (614) 292-3906, attn. J.H. McCulloch

1945 N. High St.

Columbus, OH 43210

URL:

BTW, the link provided by CoolChill in #136 gives the complete Corrigendum, so you don’t have to subscribe to see more. The 3-page “letter” they refer to is just their original article, not the full text of the correction.

—–

I removed the email links from Hu’s message for spam.

This issue is also being addressed at Climate Audit

## The Steig Corrigendum

## Jeff Id said

Apparently climatology is closely related to meteorology after all.

0.12 C per decade plus or minus 0.12 C per decade.

Hedging your bets?

## Paul Penrose said

How they can claim that this does not affect any of the conclusions of their paper is beyond me. Effectively what they have admitted is that it is just as likely that the average temperature the continent has risen is 0.00 C per decade as 0.12 C per decade, or any other value in between. This is a NULL result.

## Keith said

Jeff, the chart has even more significance, as the two areas where the supposed signal is greater than the noise variance are Western Antarctica and the Antarctic Peninsula. This would suggest your and Ryan’s point that the temperature trend Steig et al. reported was achieved by spreading the temperature increases in the Penisula and parts of Western Antarctica across the entire continent. This validates the area reconstructions you, Jeff C, and Ryan have been pursuing.

Eastern Antarctica has a larger land mass and area than the other two areas, so the cold there offsets the warming in the other areas. The final margin in the total reconstruction, taking into account the margin of error, is effectively nil. This says to me that the warming effects associated with the ocean and vulcanism are the local weather in the West, and that without them, the overall temperature would be negative.

## John F. Pittman said

I don’t think that it is a null result. However, it nullifies their “claim to fame”:””Here we show that significant warming

extends well beyond the Antarctic Peninsula to cover most of West

Antarctica, an area of warming much larger than previously

reported.”” Since their results now include a null result, IMO their Corrigendum is still incorrect since they need to reduce the strength of the claim to “” we show that possible warming could extend well beyond the Antarctic Peninsula.””

## curious said

4 – how about modifying their “claim to to fame” to “we screwed up on the front cover of Nature”?

Page48 – exhaling now! 🙂

## Terry said

So… Am I remembering the sequence of events correctly?

1) Jeff points out that a link to a reference manual on RegEM does not constitute “the reconstruction code… all of it” by Steig and mockingly told to take Steig’s matlab class.

2) Steve Mc points out the mistake on Harry (on Superbowl Sunday) and Gavin rushes to make the correction without attribution, inventing a mystery source along the way.

3) Jeff / Jeff / Ryan (and others?) create an improved reconstruction, are referred to by Mann as “the attackers who we shut up” or something like that, and the thread about this is abruptly ended.

4) Hu Mc points out that the CI was calculated incorrectly, has posts addressing this moderated out of existence at RC, but creates something that looks curiously similar to…

5) A corrigendum is quietly filed with Nature, without crediting anyone who identified (and worked out) the problems the authors and RC “scientists” read about, mocked, and/or deleted. With the typical rejoinder that “it doesn’t affect any of our results or conclusions.

Pillars of science and integrity, the whole lot of ’em.

[self snip]

## Jeff Id said

#4 How about – through the extreme blending of data due to choice of two few PC’s we’ve been able to blend some of the trend of the peninsula and Ross ice shelf around the rest of the continent.

and,

We figured all this out independently from Dr. McCulloch’s post which we never read or heard about and who deserves no credit.

## Jeff Id said

#6 Sounds about right. The boys shone again.

RC makes it fun don’t they. What would bloggers do without them!

## Matt Y. said

#5 – sounds about right to me. Quibbling over confidence intervals with the holes that have been punched in the Steig reconstruction is purely academic.

## Terry said

#8 – No kidding..

And that “self snip” was for real. After reading it, I decided I didn’t want to drop to the level of a bird photographer, and lower the discourse of discussion on your blog to the level I did. But it was pretty damn cathartic typing it (before deleting it) 🙂

## Hu McCulloch said

Thanks, Jeff! Very interesting!

## Jeff Id said

#11

Nope, Thank you. I keep learning.

I should also point out that they now have determined basically the same values as you did. For the rest, the 1.964 is the two sigma value from Hu’s post. What this means is that Dr. Steig’s group decided to provide an additional correction beyond even Dr. Hu’s work for reasons I don’t yet understand. However, they presented the almost EXACTLY the same results as Dr. McCulloch in their table in ROW 2.

Using seAR values Hu numbers are on the left Dr. Steig’s revised numbers are on the right:

all Ant = 0.0458 x 1.964 =

0.09—-Steig’s new AR version =.10peninsula = 0.0259 x 1.964 =

0.05—- Steig’s new AR version =.05west = 0.0429 x 1.964 =

0.084Steig’s new AR version =.09east = 0.0504 x 1.964 =

0.10Steig’s new AR version =.10## Jeff Id said

No credit? My guess is Dr. McCulloch doesn’t care much but where’s the class?

## John F. Pittman said

I thought that was Dr. Steig’s Matlab class at Washington U.? 😉

## Geoff Sherrington said

It would be beautiful to have a future “Nature” cover showing Jeff’s tiles instead of that original Antarctic map. Anyone interested in forming a group to make a submission? sherro1@optusnet.com.au

## Jeff Id said

#15, I wish I could take credit but I believe you’re referring to Ryan’s choose your own Antarctic. Good stuff though.

## Jeff Id said

Here’s a note by Hu on his own thread at CA.

==========================================

Although Steig & Co. may not read CA on a regular basis, I did call their attention to this post in the following e-mail, dated 2/28/09:

Date: Sat, 28 Feb 2009 15:51:44 -0500

To: steig@ess.washington.edu, dschneid@ucar.edu, srutherford@fox.rwu.edu,mann@psu.edu, josefino.c.comiso@nasa.gov,Drew.T.Shindell@nasa.gov

From: Hu McCulloch

Subject: Comment on serial correlation in Steig et al 2009

Dear Dr. Steig and co-authors,

FYI, I have recently posted a comment on your 2009 paper in Nature

on Climate Audit, at http://www.climateaudit.org/?p=5341 .

While I was able to replicate or virtually replicate the 1957-2006 trends you report

on p 460 for the three regions and the continent as a whole, the 95% Confidence

Intervals you report appear to have taken no account of serial correlation

in the regression errors. When this is done, the CI’s are substantially wider

than you report.

Any reactions, by comments there or by e-mail, would be welcome!

— Hu McCulloch

J. Huston McCulloch mcculloch.2@osu.edu

Economics Dept. voice (614) 292-0382

Ohio State Univ. FAX (614) 292-3906, attn. J.H. McCulloch

1945 N. High St.

Columbus, OH 43210

URL: http://econ.ohio-state.edu/jhm/jhm.html

BTW, the link provided by CoolChill in #136 gives the complete Corrigendum, so you don’t have to subscribe to see more. The 3-page “letter” they refer to is just their original article, not the full text of the correction.

==========================================================

Wow!

## Page48 said

RE: #5, Curious

LOL## Steve Fitzpatrick said

Paul Penrose said

August 5, 2009 at 5:35 pm

“How they can claim that this does not affect any of the conclusions of their paper is beyond me. Effectively what they have admitted is that it is just as likely that the average temperature the continent has risen is 0.00 C per decade as 0.12 C per decade, or any other value in between. This is a NULL result.”

Not null, just not very certain. The best estimate from their analysis remains +0.12, though there is about a 2.5% chance that the effect is 0 or less, and a 2.5% chance that the effect is +0.24 or higher. Of course, that does not address the problems with the whole exercise… blending the warm peninsula with the rest to make all of the continent appear to be warming smartly in recent years. It was a political document, so confidence intervals are not important.

## Ryan O said

Jeff and Nic L,

Will be sending an email soon with the new script and a few thoughts that relate to what happened here.

I’m not terribly happy with what I read in Nature.

## Fluffy Clouds (Tim L) said

Well………

VERY ROBUST LMAO………..oh my god!!!!!!!!!!!!!!

I wet my self!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

what a bunch of fools!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

da dah da thats all folks!

WB’s music and out……..

I feel like eating a carrot humm

VERY ROBUST +/- ALL of the robustness

what a bunch of idiots!

I am sorry… wiping tears from my eyes!!!

Doh…………….

## Fluffy Clouds (Tim L) said

#

Jeff Id said

August 5, 2009 at 5:17 pm

Apparently climatology is closely related to meteorology after all.

0.12 C per decade plus or minus 0.12 C per decade.

Hedging your bets?

NO! covering your Asses!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

AFTER R.O. did to them they had no choice.

and you to!!! 😉

I said damming did’nt I?

## Phillip Bratby said

Jeff you said, #1 August 5, 2009 at 5:17 pm:

“Apparently climatology is closely related to meteorology after all”.

Did you mean astrology!?!

## DJA said

Plagiarism is a very great sin in science. Due acknowledgment needs to be very quickly done.

## Phillip Bratby said

Does it really matter whether the Antarctic is cooling, warming or not changing? Whichever it is, the climate models have always predicted it correctly!

## MrPete said

Steve, you wrote

If I understand CI stats correctly from bender’s patient lessons, this is not quite correct. 0.12 is not the “best” estimate, it is simply the center of the 95% CI. In reality, all we know is there is a 95% probability that the real answer lies somewhere within the 95% CI.

It is no more probable that it sits in the center than anywhere else within the CI.

## Geoff Sherrington said

For Ryan O.

You and the 2 Jeffs and Hu work so well as a team that I was remiss in referring to Jeff’s tiles when they were Ryan’s tiles. I was also remiss in not asking permission to float this idea. One of my other hobbies is graphic art and I just KNOW that the Antarctic tiles would make a stunning cover for Nature. Do you mind if I write them a short letter to sound them out? Sooner the better for relevance to the Steig cover.

## Steve Fitzpatrick said

MrPete said

August 6, 2009 at 6:08 am

“If I understand CI stats correctly from bender’s patient lessons, this is not quite correct. 0.12 is not the “best” estimate, it is simply the center of the 95% CI. In reality, all we know is there is a 95% probability that the real answer lies somewhere within the 95% CI.

It is no more probable that it sits in the center than anywhere else within the CI.”

The true value of a parameter is not known, but can be inferred from a population of measurements of that parameter. The “best estimate” of the true value is the mean of the measurements if the distribution of measurements is normally distributed. The estimated standard deviation for the population (and the confidence interval based on that estimate of standard deviation) is generated from the root mean square of the differences between the mean and the individual measurements. For non-normal distributions it is more complicated, but any case the reported “point value” for a measured parameter is usually considered to be the most probable result, or the “best estimate”. It is not correct to say that any value which lies within the confidence interval is equally probable.

## Jeff Id said

In this case the 95% confidence interval is calculated from a normal distribution. The highest probability for trend is therefore in the center of the range.

## Jeff Id said

#28 did a better job explaining it.

## Dan R. said

Re: #25 Phillip Bratby said

“Does it really matter whether the Antarctic is cooling, warming or not changing? Whichever it is, the climate models have always predicted it correctly!”

BINGO! RC post by Spencer Wearthless – It’s Cold in Antarctica – yeah we knew that.

RC post by Eric Steig, Mathmatician suvant – Oh wait, actually it is warm in Antarctica – just like we thought.

Corrigendum by Eric Steig, plagiarist supreme – Well we think it might be warming faster, as fast, or slower than the rest of the globe – as we expected.

## John said

The irony is TCO will be along any time now telling everyone that you are amateurs, blogs are not science and you should do some work and get published……..like Steig who publishes work from the same blogs under his own name……..ohh my sides are hurting!!!!

## curious said

Re: Irony – I like the fact that whilst all this is going the current lead post on RC is titled: “Still not convincing”:

http://www.realclimate.org/

Riot! 🙂

Geoff – re: the tiles. Love it – hint to Ryan: I still hope for a souvenir coaster set!?