the Air Vent

Because the world needs another opinion

Significance?

Posted by Jeff Id on August 5, 2009

UPDATED BELOW:

An interesting development in the Antarctic. It appears Dr. Hu McCulloch who had such difficulty getting even a comment through moderation at RC was correct when he wrote this post for Climate Audit.

Steig 2009’s Non-Correction for Serial Correlation

Hu Wrote:

Steig et al. report a 95% CI of ± 0.07 °C/decade for this trend coefficient. Assuming a t critical value of approximately 2, this means that they are estimating the se to be about 0.035. Using the above data, the OLS standard error (“seOLS”) is 0.0330. Twice this value is within rounding error of 0.07. It indicates a very high level of significance (p = .0004).

However, the first order serial correlation of the residuals, r1, is 0.318, and is highly significant (p = .0000). With this serial correlation, the standard AR(1)-corrected standard error (“seAR”) is 0.0458, i.e. 44% larger. This leaves the slope still highly significant (p = .0103), but not nearly as significant as is implied by the published CI. The simpler approximation presented by Santer et al., which is based on a 1952 formula by Quenouille and therefore identified here as “seQ” (see below for details), gives the same numerical value as seAR, namely 0.0458. Using either seAR or seQ, the corrected 95% CI is ± 1.964(0.0458) = 0.0900, which is clear inconsistent with the 0.07 reported by Steig et al.

Table 1 below reports similar figures for all the 1957-2006 TIR reconstruction trends for which Steig et al. report numerical values. Implicit Steig standard errors, obtained by dividing their reported 95% CI’s by 2, are given in square brackets [ ]. p-values are given in parentheses below the respective standard errors. Slopes and standard errors are all in °C/decade.

Table 1: 1957-2006
Region Steig
slope
Steig
[se]
OLS
slope
seOLS
(p)
r1 seAR
(p)
seQ
All Ant. .12 [.035] .1178 .0330
(.0004)
.318 .0458
(.0103)
.0458
Peninsula .11 [.02] .1196 .0198
.0000
.262 .0259
(.0000)
.0259
West Ant. .17 [.03] .1839 .0311
(.0000)
.312 .0429
(.0000)
.0430
East Ant. .10 [.035] .0984 .0374
(.0087)
.290 .0504
(.0512)
.0504

In all four cases, Steig et al. are clearly using seOLS, not seQ or seAR. In the caption to Figure S4 of their online SI, the authors state that
Black lines separate areas of significant vs. insignificant trends (>95% confidence based on two-tailed t-test with number of degrees of freedom adjusted for temporal autocorrelation). (SI p. 4)

The reference to adjusting the number of degrees of freedom would imply that they are using the Santer/Quenouille seQ adjustment. However, the confidence intervals they report in fact show no correction for serial correlation, by either method. The article’s Figure 3, showing regions of significant and insignificant trends, is therefore erroneous.

Today at Nature an addendum confirming Dr. McCulloch’s claims that autocorrelation had not been accounted for was released with the following text:

In this Letter, we reported trends on reconstructed temperature histories for different areas of the Antarctic continent. The confidence levels on the trends, as given in the text, did not take into account the reduced degrees of freedom in the time series due to autocorrelation. We report in Table 1 the corrected values, based on a two-tailed t-test, with the number of degrees of freedom adjusted for autocorrelation, using Neffective = N(1 – r)/(1 + r), in which N is the sample size and r is the lag-1 autocorrelation coefficient of the residuals of the detrended time series. The median of r is 0.27, resulting in a reduction in the degrees of freedom from N = 600 to Neffective = 345 for the monthly time series.

We also include results of a further calculation that takes into account both the variance and the uncertainty in the reconstructed temperatures. We performed Monte-Carlo simulations of the reconstructed temperatures using a Gaussian distribution with variance equal to the unresolved variance from the split calibration/verification tests described in the paper. Confidence bounds were obtained by detrending each simulation and obtaining the lag-1 autocorrelation coefficient and variance of the residuals; a random realization of Gaussian noise having the same lag-1 autocorrelation coefficient and variance was then added to the trend, and a new trend was calculated. The 2.5th and 97.5th percentiles of the 10,000 simulated trends give the 95% confidence bounds. For the case of zero unresolved variance, this calculation converges on the same value as the two-tailed t-test, above. The 95% confidence minimum trend value is given by the 5th percentile values of the simulated trends, last row of Table 1.

The corrected confidence levels do not change the assessed significance of trends, nor any of the primary conclusions of the paper. We also note that there is a typographical error in Supplementary Table 1: the correct location of Automatic Weather Station ‘Harry’ is 83.0° S, 238.6° E. The position of this station on the maps in the paper is correct.

What’s interesting however is the following table:

confidence intervals

Link to Steig’s Corrigendum

So now we have an antarctic temperature of 0.12 +/- 0.10C/Decade correcting for autocorrelation expanded from 0.12 +/- 0.07 C/Decade which was originally reported and an additional expansion for unresolved variance to 0.12 +/- 0.12C/Decade. No mention of Dr. McCulloch, however, in the report to Nature.

Oddly enough this means the difference between the trend for the Antarctic as determined by area weighted ground stations and improved regression analysis of about 0.05C/Decade no longer outside the significance intervals for the paper. I wonder if it would have made the cover of Nature with a claim of 0.12 C/decade +/- 0.12 C/decade!

Hat tip to P – for showing the article and Realist for the link.

====================================================

UPDATE: The Steig et al. Corregendum results are almost exactly the same as Hu’s.

Using seAR values Hu numbers are on the left Dr. Steig’s revised numbers are on the right:
all Ant = 0.0458 x 1.964 = 0.09 —-Steig’s new AR version = .10
peninsula = 0.0259 x 1.964 = 0.05 —- Steig’s new AR version = .05
west = 0.0429 x 1.964 = 0.084 Steig’s new AR version = .09
east = 0.0504 x 1.964 = 0.10 Steig’s new AR version = .10

UPDATE 2

Hu made them aware of the required changes in a personal email in February.

— From a comment at CA —

Although Steig & Co. may not read CA on a regular basis, I did call their attention to this post in the following e-mail, dated 2/28/09:

Date: Sat, 28 Feb 2009 15:51:44 -0500
To: steig, dschneid, srutherford,mann, josefino.c.comiso,Drew.T.Shindell
From: Hu McCulloch
Subject: Comment on serial correlation in Steig et al 2009

Dear Dr. Steig and co-authors,
FYI, I have recently posted a comment on your 2009 paper in Nature
on Climate Audit, at http://www.climateaudit.org/?p=5341 .
While I was able to replicate or virtually replicate the 1957-2006 trends you report
on p 460 for the three regions and the continent as a whole, the 95% Confidence
Intervals you report appear to have taken no account of serial correlation
in the regression errors. When this is done, the CI’s are substantially wider
than you report.
Any reactions, by comments there or by e-mail, would be welcome!
— Hu McCulloch

J. Huston McCulloch
Economics Dept. voice (614) 292-0382
Ohio State Univ. FAX (614) 292-3906, attn. J.H. McCulloch
1945 N. High St.
Columbus, OH 43210
URL:

BTW, the link provided by CoolChill in #136 gives the complete Corrigendum, so you don’t have to subscribe to see more. The 3-page “letter” they refer to is just their original article, not the full text of the correction.

—–

I removed the email links from Hu’s message for spam.

This issue is also being addressed at Climate Audit

The Steig Corrigendum


33 Responses to “Significance?”

  1. Jeff Id said

    Apparently climatology is closely related to meteorology after all.

    0.12 C per decade plus or minus 0.12 C per decade.

    Hedging your bets?

  2. Paul Penrose said

    How they can claim that this does not affect any of the conclusions of their paper is beyond me. Effectively what they have admitted is that it is just as likely that the average temperature the continent has risen is 0.00 C per decade as 0.12 C per decade, or any other value in between. This is a NULL result.

  3. Keith said

    Jeff, the chart has even more significance, as the two areas where the supposed signal is greater than the noise variance are Western Antarctica and the Antarctic Peninsula. This would suggest your and Ryan’s point that the temperature trend Steig et al. reported was achieved by spreading the temperature increases in the Penisula and parts of Western Antarctica across the entire continent. This validates the area reconstructions you, Jeff C, and Ryan have been pursuing.

    Eastern Antarctica has a larger land mass and area than the other two areas, so the cold there offsets the warming in the other areas. The final margin in the total reconstruction, taking into account the margin of error, is effectively nil. This says to me that the warming effects associated with the ocean and vulcanism are the local weather in the West, and that without them, the overall temperature would be negative.

  4. John F. Pittman said

    I don’t think that it is a null result. However, it nullifies their “claim to fame”:””Here we show that significant warming
    extends well beyond the Antarctic Peninsula to cover most of West
    Antarctica, an area of warming much larger than previously
    reported.”” Since their results now include a null result, IMO their Corrigendum is still incorrect since they need to reduce the strength of the claim to “” we show that possible warming could extend well beyond the Antarctic Peninsula.””

  5. curious said

    4 – how about modifying their “claim to to fame” to “we screwed up on the front cover of Nature”?

    Page48 – exhaling now! 🙂

  6. Terry said

    So… Am I remembering the sequence of events correctly?

    1) Jeff points out that a link to a reference manual on RegEM does not constitute “the reconstruction code… all of it” by Steig and mockingly told to take Steig’s matlab class.

    2) Steve Mc points out the mistake on Harry (on Superbowl Sunday) and Gavin rushes to make the correction without attribution, inventing a mystery source along the way.

    3) Jeff / Jeff / Ryan (and others?) create an improved reconstruction, are referred to by Mann as “the attackers who we shut up” or something like that, and the thread about this is abruptly ended.

    4) Hu Mc points out that the CI was calculated incorrectly, has posts addressing this moderated out of existence at RC, but creates something that looks curiously similar to…

    5) A corrigendum is quietly filed with Nature, without crediting anyone who identified (and worked out) the problems the authors and RC “scientists” read about, mocked, and/or deleted. With the typical rejoinder that “it doesn’t affect any of our results or conclusions.

    Pillars of science and integrity, the whole lot of ’em.

    [self snip]

  7. Jeff Id said

    #4 How about – through the extreme blending of data due to choice of two few PC’s we’ve been able to blend some of the trend of the peninsula and Ross ice shelf around the rest of the continent.

    and,

    We figured all this out independently from Dr. McCulloch’s post which we never read or heard about and who deserves no credit.

  8. Jeff Id said

    #6 Sounds about right. The boys shone again.

    RC makes it fun don’t they. What would bloggers do without them!

  9. Matt Y. said

    #5 – sounds about right to me. Quibbling over confidence intervals with the holes that have been punched in the Steig reconstruction is purely academic.

  10. Terry said

    #8 – No kidding..

    And that “self snip” was for real. After reading it, I decided I didn’t want to drop to the level of a bird photographer, and lower the discourse of discussion on your blog to the level I did. But it was pretty damn cathartic typing it (before deleting it) 🙂

  11. Thanks, Jeff! Very interesting!

  12. Jeff Id said

    #11

    Nope, Thank you. I keep learning.

    I should also point out that they now have determined basically the same values as you did. For the rest, the 1.964 is the two sigma value from Hu’s post. What this means is that Dr. Steig’s group decided to provide an additional correction beyond even Dr. Hu’s work for reasons I don’t yet understand. However, they presented the almost EXACTLY the same results as Dr. McCulloch in their table in ROW 2.

    Using seAR values Hu numbers are on the left Dr. Steig’s revised numbers are on the right:
    all Ant = 0.0458 x 1.964 = 0.09 —-Steig’s new AR version = .10
    peninsula = 0.0259 x 1.964 = 0.05 —- Steig’s new AR version = .05
    west = 0.0429 x 1.964 = 0.084 Steig’s new AR version = .09
    east = 0.0504 x 1.964 = 0.10 Steig’s new AR version = .10

  13. Jeff Id said

    No credit? My guess is Dr. McCulloch doesn’t care much but where’s the class?

  14. John F. Pittman said

    I thought that was Dr. Steig’s Matlab class at Washington U.? 😉

  15. Geoff Sherrington said

    It would be beautiful to have a future “Nature” cover showing Jeff’s tiles instead of that original Antarctic map. Anyone interested in forming a group to make a submission? sherro1@optusnet.com.au

  16. Jeff Id said

    #15, I wish I could take credit but I believe you’re referring to Ryan’s choose your own Antarctic. Good stuff though.

  17. Jeff Id said

    Here’s a note by Hu on his own thread at CA.

    ==========================================
    Although Steig & Co. may not read CA on a regular basis, I did call their attention to this post in the following e-mail, dated 2/28/09:

    Date: Sat, 28 Feb 2009 15:51:44 -0500
    To: steig@ess.washington.edu, dschneid@ucar.edu, srutherford@fox.rwu.edu,mann@psu.edu, josefino.c.comiso@nasa.gov,Drew.T.Shindell@nasa.gov
    From: Hu McCulloch
    Subject: Comment on serial correlation in Steig et al 2009

    Dear Dr. Steig and co-authors,
    FYI, I have recently posted a comment on your 2009 paper in Nature
    on Climate Audit, at http://www.climateaudit.org/?p=5341 .
    While I was able to replicate or virtually replicate the 1957-2006 trends you report
    on p 460 for the three regions and the continent as a whole, the 95% Confidence
    Intervals you report appear to have taken no account of serial correlation
    in the regression errors. When this is done, the CI’s are substantially wider
    than you report.
    Any reactions, by comments there or by e-mail, would be welcome!
    — Hu McCulloch

    J. Huston McCulloch mcculloch.2@osu.edu
    Economics Dept. voice (614) 292-0382
    Ohio State Univ. FAX (614) 292-3906, attn. J.H. McCulloch
    1945 N. High St.
    Columbus, OH 43210
    URL: http://econ.ohio-state.edu/jhm/jhm.html

    BTW, the link provided by CoolChill in #136 gives the complete Corrigendum, so you don’t have to subscribe to see more. The 3-page “letter” they refer to is just their original article, not the full text of the correction.
    ==========================================================

    Wow!

  18. Page48 said

    RE: #5, Curious

    LOL

  19. Steve Fitzpatrick said

    Paul Penrose said
    August 5, 2009 at 5:35 pm

    “How they can claim that this does not affect any of the conclusions of their paper is beyond me. Effectively what they have admitted is that it is just as likely that the average temperature the continent has risen is 0.00 C per decade as 0.12 C per decade, or any other value in between. This is a NULL result.”

    Not null, just not very certain. The best estimate from their analysis remains +0.12, though there is about a 2.5% chance that the effect is 0 or less, and a 2.5% chance that the effect is +0.24 or higher. Of course, that does not address the problems with the whole exercise… blending the warm peninsula with the rest to make all of the continent appear to be warming smartly in recent years. It was a political document, so confidence intervals are not important.

  20. Ryan O said

    Jeff and Nic L,

    Will be sending an email soon with the new script and a few thoughts that relate to what happened here.

    I’m not terribly happy with what I read in Nature.

  21. Fluffy Clouds (Tim L) said

    Well………
    VERY ROBUST LMAO………..oh my god!!!!!!!!!!!!!!
    I wet my self!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    what a bunch of fools!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    da dah da thats all folks!
    WB’s music and out……..
    I feel like eating a carrot humm

    VERY ROBUST +/- ALL of the robustness

    what a bunch of idiots!

    I am sorry… wiping tears from my eyes!!!

    Doh…………….

  22. Fluffy Clouds (Tim L) said

    #
    Jeff Id said
    August 5, 2009 at 5:17 pm

    Apparently climatology is closely related to meteorology after all.

    0.12 C per decade plus or minus 0.12 C per decade.

    Hedging your bets?

    NO! covering your Asses!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    AFTER R.O. did to them they had no choice.
    and you to!!! 😉
    I said damming did’nt I?

  23. Phillip Bratby said

    Jeff you said, #1 August 5, 2009 at 5:17 pm:

    “Apparently climatology is closely related to meteorology after all”.

    Did you mean astrology!?!

  24. DJA said

    Plagiarism is a very great sin in science. Due acknowledgment needs to be very quickly done.

  25. Phillip Bratby said

    Does it really matter whether the Antarctic is cooling, warming or not changing? Whichever it is, the climate models have always predicted it correctly!

  26. MrPete said

    Steve, you wrote

    The best estimate from their analysis remains +0.12, though there is about a 2.5% chance that the effect is 0 or less, and a 2.5% chance that the effect is +0.24 or higher.

    If I understand CI stats correctly from bender’s patient lessons, this is not quite correct. 0.12 is not the “best” estimate, it is simply the center of the 95% CI. In reality, all we know is there is a 95% probability that the real answer lies somewhere within the 95% CI.

    It is no more probable that it sits in the center than anywhere else within the CI.

  27. Geoff Sherrington said

    For Ryan O.

    You and the 2 Jeffs and Hu work so well as a team that I was remiss in referring to Jeff’s tiles when they were Ryan’s tiles. I was also remiss in not asking permission to float this idea. One of my other hobbies is graphic art and I just KNOW that the Antarctic tiles would make a stunning cover for Nature. Do you mind if I write them a short letter to sound them out? Sooner the better for relevance to the Steig cover.

  28. Steve Fitzpatrick said

    MrPete said
    August 6, 2009 at 6:08 am

    “If I understand CI stats correctly from bender’s patient lessons, this is not quite correct. 0.12 is not the “best” estimate, it is simply the center of the 95% CI. In reality, all we know is there is a 95% probability that the real answer lies somewhere within the 95% CI.

    It is no more probable that it sits in the center than anywhere else within the CI.”

    The true value of a parameter is not known, but can be inferred from a population of measurements of that parameter. The “best estimate” of the true value is the mean of the measurements if the distribution of measurements is normally distributed. The estimated standard deviation for the population (and the confidence interval based on that estimate of standard deviation) is generated from the root mean square of the differences between the mean and the individual measurements. For non-normal distributions it is more complicated, but any case the reported “point value” for a measured parameter is usually considered to be the most probable result, or the “best estimate”. It is not correct to say that any value which lies within the confidence interval is equally probable.

  29. Jeff Id said

    In this case the 95% confidence interval is calculated from a normal distribution. The highest probability for trend is therefore in the center of the range.

  30. Jeff Id said

    #28 did a better job explaining it.

  31. Dan R. said

    Re: #25 Phillip Bratby said

    “Does it really matter whether the Antarctic is cooling, warming or not changing? Whichever it is, the climate models have always predicted it correctly!”

    BINGO! RC post by Spencer Wearthless – It’s Cold in Antarctica – yeah we knew that.

    RC post by Eric Steig, Mathmatician suvant – Oh wait, actually it is warm in Antarctica – just like we thought.

    Corrigendum by Eric Steig, plagiarist supreme – Well we think it might be warming faster, as fast, or slower than the rest of the globe – as we expected.

  32. John said

    The irony is TCO will be along any time now telling everyone that you are amateurs, blogs are not science and you should do some work and get published……..like Steig who publishes work from the same blogs under his own name……..ohh my sides are hurting!!!!

  33. curious said

    Re: Irony – I like the fact that whilst all this is going the current lead post on RC is titled: “Still not convincing”:

    http://www.realclimate.org/

    Riot! 🙂

    Geoff – re: the tiles. Love it – hint to Ryan: I still hope for a souvenir coaster set!?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

 
%d bloggers like this: