Experts or little white lies?

UPDATE, It appears that my post uses data published yet not recommended by Phil Jones.   Perhaps a retraction by Dr. Jones  is in order or is it as Lucia points out in her post, a case of white noise.

Douglas J. Keenan 

in comments below notes:

Here is a quote from that: “global and regional time-series should be calculated using unadjusted data”. Indeed, the authors have criticized others for attempting to use HadCRUT3v for time series. Note that Jones is one of the authors.

Here is the full quote:

Whether variance adjusted or unadjusted data should be used in an analysis depends on what is to be calculated. If it is necessary that  grid-box anomalies have a spatially and temporally consistent variance, then variance adjusted data should be used. Otherwise, better results may be obtained using unadjusted data.  In particular, global and regional time-series should be calculated using unadjusted data.

The barely significant adjusted data should NOT be used for trend analysis per Dr. Jones himself.  Will climate wonders never cease.

————————-

Phil Jones made a few famous quotes around the time of climategate.  One of them was brought up again in a happy correction to the record as reported by the BBC on June 10th.  Lucia covered it here already.  Back a year and a half ago, Phil let slip in an unusually difficult interview for a climate scientist that there was no statistically significant warming since 1995.

B – Do you agree that from 1995 to the present there has been no statistically-significant global warming

Yes, but only just. I also calculated the trend for the period 1995 to 2009. This trend (0.12C per decade) is positive, but not significant at the 95% significance level. The positive trend is quite close to the significance level. Achieving statistical significance in scientific terms is much more likely for longer periods, and much less likely for shorter periods.

This refers to the arbitrary yet regularly used 95% threshold or 1/20 chance that the result happened by accident alone.  Now in a new article Phil reports this:

Climate warming since 1995 is now statistically significant, according to Phil Jones, the UK scientist targeted in the “ClimateGate” affair.

Yay!!  We all knew it was coming, because as the data record grows longer even in a temperature stable world, statistical thresholds are naturally crossed.  The Earth would have to cool quickly in order to prevent this threshold from being crossed.  We discussed the issue in 2009 here, just days prior to climategate.

Lucia took the annual averages and corrected she for AR1 (autocorrelation), she found that the claim by Jones was still invalid.  Well, that caught my interest so I looked more closely at the data and wrote a downloader for the monthly HADCRUT3 and HADCRUT3V data in R.  I then dug up some old algorithms and did some of my own plots.  HADCRUT3 is a simple gridded version of the temperature data where anomalies are combined by grid and averaged.  HADCRUT3V has some sort of variance adjustment as described by Jones 2001, which ‘corrects’ the variance.  I have not read or looked for the paper and so cannnot comment on its veracity.

I made the following plot using the code below.  This uses all the available data, calculates a trend and a confidence interval for that trend based on the Quenouille AR1 degrees of freedom correction.  Nothing too magical about that. The red lines are the +/- confidence interval, the black line is the slope of the data in C/Decade for each month to the most recent month.  In other words, I fit a slope for the data from every month in history starting with 1980 January to present,, 1980 February to present, March etc, all the way until 2008 because a slope for three years of global temperatures is silly.

It looks from this plot that Phil Jones called it right at  the crossing point.  The first moment that the data showed significance for 1995, he held a press conference (or at least a phone call/email) to correct the record and proclaimed, the temperature trend from 1995 to present is significant!!

Look how closely he called it.

The black ‘slope’ line just peaks over the top of the red line for under half of 1995.  Since we are in June of 2011 we can still say 16 years with no significant trend, but the silliness of the thing is what this post is about.  I don’t have the data for  last months ‘variance’ adjusted version and can’t recreate it because of this quote on the HADCRUT temperature page:

All the files on this page (except Absolute) will be updated on a monthly basis to include the latest month within about four weeks of its completion. Updating includes not just data for the last month but the addition of any late reports for up to approximately the last two years. In addition to this the method of variance adjustment (used for CRUTEM3v and HadCRUT3v) works on the anomalous temperatures relative to the underlying trend on an approximate 30-year timescale. Estimating this trend requires estimation of grid-box temperatures for years before the start of each record and after the end. With the addition of subsequent years, the underlying trend will alter slightly, changing the variance-adjusted values. Effects will be greatest on the last year of the record, but an influence can be evident for the last three to four years. Full details of the variance adjustment procedure are given in Jones et al. (2001). Approximately yearly, the optimally averaged values will also be updated to take account of such additional past information

Now there is nothing nefarious about the statement here, but it does leave one curious if this months adjustments are what pushed 1995 over the line.  I also processed the unadjusted gridded data presented on exactly the same page and found a different result.

Woah!! So close, no cigar.

Good try though.  Maybe next year.  I wonder why Phil didn’t mention this dataset hadn’t quite made the line yet?  It is from his webpage after all.

Now from the climategate emials, can you imagine the pressure Phil Jones experienced from his colleagues when he let slip that trends since 1995 weren’t significant?   We already knew that but the admission must have cost him something because look how quickly he came out with the correction, and look how quickly the press covered it.

A college professor from a long time ago said:

An expert is just a drip under pressure.

You wonder why people question the experts.

loc="http://www.cru.uea.ac.uk/cru/data/temperature/hadcrut3gl.txt"
#loc="http://www.cru.uea.ac.uk/cru/data/temperature/hadcrut3vgl.txt"

wd=c(5,7,7,7,7,7,7,7,7,7,7,7,7,7)
hadc=read.fwf(loc,skip=0,widths=wd)
mask=rep(FALSE,324)
mask[seq(1,324, 2)]=TRUE
hh= t(hadc[mask,2:13])
dim(hh)=c(162*12,1)
maskb= hh!=0
hh=hh[maskb]

hadcrut=ts(hh,start=1850,deltat=1/12)

plot(hadcrut)

plot(window(hadcrut,start=1995))
mm=lsfit(time(window(hadcrut,start=1995)),window(hadcrut,start=1995))
abline(mm, col="red")
ssq=function(x) {sum(x^2)}

cal.sig=function(dat=dat, st=0,en=2020)
{
	###  Get trend
	fm=lm(window(dat, st, en)~I(time(window(dat, st, en))))

	###  Initialize variables
	N=length(window(dat, st, en))
	I=seq(1:N)/frequency(dat)
	rmI=ssq(I-mean(I))

	###  Calculate sum of squared errors
	SSE=ssq(fm[[2]])/(N-2)

	###  Calculate OLS standard errors
	seOLS=sqrt(SSE/rmI)*10

	###  Calculate two-tailed 95% CI
	ci95=seOLS*1.964

	###  Get lag-1 ACF
	resids=residuals(lm(window(dat, st, en)~seq(1:N)))
	r1=acf(resids, lag.max=1, plot=FALSE)$acf[[2]]

	###  Calculate CIs with Quenouille (Santer) adjustment for autocorrelation
	Q=(1-r1)/(1+r1)
	ciQ=ci95/sqrt(Q)

	return(c(fm$coef[2]*10,ciQ))
}
tim=time(window(hadcrut,start=1980))
slope=rep(0,length(tim))
ci=rep(0,length(tim))

for (i in 1:(length(tim)-12))
{
	mm=cal.sig(window(hadcrut,start=tim[i]))
	slope[i]=mm[1]
	ci[i]=mm[2]
}
sl=window(ts(slope,start=1980,deltat=1/12),end=2008)
nci=window(ts(-ci,start=1980,deltat=1/12),end=2008)
pci=window(ts(ci,start=1980,deltat=1/12),end=2008)
#savePlot(paste("e:/agw/pics/hadcrutV3GL significance.jpg"),type="jpg")

#plot(sl,main="Hadcrut V3GL Slope Significance Since Year X",xlab="Year",ylab="C/Decade", ylim=c(-.5,.5))
plot(sl,main="Hadcrut 3GL Slope Significance Since Year X",xlab="Year",ylab="C/Decade", ylim=c(-.5,.5))
lines(nci,col="red")
lines(pci,col="red")
grid()
#savePlot(paste("e:/agw/pics/hadcrutV3GL significance.jpg"),type="jpg")
#savePlot(paste("e:/agw/pics/hadcrut3GL significance.jpg"),type="jpg")

#plot(sl,main="Hadcrut V3GL Slope Significance Since X",xlab="Year",ylab="C/Decade", ylim=c(.09,.12),xlim=c(1994,1996))
plot(sl,main="Hadcrut 3GL Slope Significance Since X",xlab="Year",ylab="C/Decade", ylim=c(.09,.12),xlim=c(1994,1996))

lines(nci,col="red")
lines(pci,col="red")
grid()

#savePlot(paste("e:/agw/pics/hadcrutV3GL significance zoomed in.jpg"),type="jpg")
#savePlot(paste("e:/agw/pics/hadcrut3GL significance zoomed in.jpg"),type="jpg")