I keep telling you guys the guest posts are great, check it out.
Guest post by Dr. Craig Loehle
—–
Recent news items have shown that adjustments to climate data seem to create warming trends where none exist in the raw data. The Darwin case evaluated by Willis Eschenbach (http://wattsupwiththat.com/2009/12/08/the-smoking-gun-at-darwin-zero/) shows the recent adjusted data to be tilted upwards. Willis also had another post about Karlen’s comments on IPCC data (http://wattsupwiththat.com/2009/11/29/when-results-go-bad/). An analysis of 88 stations in Siberia shows a cooling trend for the raw data but a big warming after adjustments by Hadley (http://heliogenic.blogspot.com/2009/12/adjusting-siberia.html). Data from New Zealand are adjusted by their own weather service to create strong warming where none exist in the raw data (http://www.climateconversation.wordshine.co.nz/docs/awfw/are-we-feeling-warmer-yet.htm). In none of these cases is a rationale for adjustments publicly available. It seems implausible on its face that all corrections for inhomogeneities should lead to a warming bias, especially when the UHI effect if corrected would cool recent years rather than warm them.
I have come across a paper that may help explain how Hadley, GHCN, and GISS do their adjustments and where the warming trends come from. The paper is:
Peterson, T.C., D.R. Easterling, T.R. Karl, P. Groisman, N. Nicholls, N. Plummer, S. Torok, I. Auer, R. Boehm, D. Gullett, L. Vincent, R. Heino, H. Tuomenvirta, O. Mestre, T. Szentimrey, J. Salinger, E.J. Førland, I. Hanssen-Bauer, H. Alexandersson, P. Jones, and D. Parker. 1998. Homogeneity adjustments of in situ atmospheric climate data: A review. International Journal of Climatology 18:1493‑1517.
http://www3.hi.is/~jonasson/peterson.pdf
I include 2 sections of text. 4.2.2 describes what the GHCN does, and then 3.4.4 describes the method they use for detecting inhomogeneities and adjusting them. There are other methods described in the paper which will have a similar effect, with some methods correcting slopes as well as means.
4.2.2. Global Historical Climatology Network. The Global Historical Climatology Network (GHCN; Peterson and Vose, 1997) includes data sets of both original and homogeneity-adjusted time series of mean monthly maximum, minimum, and mean temperature. Because of the paucity of available station history information for many of the 7280 GHCN temperature stations around the world, no metadata beyond latitude and longitude are used in the GHCN adjustment methodology. First a reference series for each station is made as described in Section 3.2 and Peterson and Easterling (1994) then the candidate reference series is tested using a 2-phase regression technique described in Section 3.4.4 and Easterling and Peterson (1995a,b). This determines the date and the magnitude of the adjustments.
3.4.4. Two–phase regression. Solow (1987) described a technique for detecting a change in the trend of a time series by identifying the change point in a two-phase regression where the regression lines before and after the year being tested were constrained to meet at that point. Since changes in instruments can cause step changes, Easterling and Peterson (1995a,b) developed a variation on the two-phase regression in which the regression lines were not constrained to meet and where a linear regression is fitted to the part of the (candidate_reference) difference series before the year being tested and another after the year being tested. This test is repeated for all years of the time series (with a minimum of 5 years in each section), and the year with the lowest residual sum of the squares is considered the year of a potential discontinuity. A residual sum of the squares from a single regression through the entire time series is also calculated. The significance of the two phase fit is tested with (i) a likelihood ratio statistic using the two residual sums of the squares and (ii) the difference in the means of the difference series before and after the discontinuity as evaluated by the Student’s t-test. If the discontinuity is determined to be significant, the time series is subdivided into two at that year. Each of these smaller sections are similarly tested. This subdividing process continues until no significant discontinuities are found or the time series are too short to test (B10 years). Each of the discontinuities that have been identified are further tested using a multiresponse permutation procedure (MRPP; Mielke, 1991). The MRPP test is non-parametric and compares the Euclidean distances between members within each group with the distances between all members from both groups, to return a probability that two groups more different could occur by random chance alone. The two groups are the 12-year windows on either side of the discontinuity, though the window is truncated at a second potential discontinuity. If the discontinuity is significant at the 95% level (probability (P)_0.05), it is considered a true discontinuity. The adjustment that is applied to all data points prior to the discontinuity is the difference in the means of the (station_reference) difference series’ two windows.
Figure 8. Annual temperature time series for Carlsbad, New Mexico. Solid line is unadjusted data. Long dash is the Jones homogeneity-adjusted data and the short dash is the U.S. HCN adjusted data for the same station. Despite very different methodologies and perhaps some differences in the source data, the long-term trends for the two adjusted time series are in good agreement.
Bold above is my emphasis. Figure 8 shows an application of their method (their caption), with figure 9 being similar. The recent period is unchanged, and the past years are dropped 1.4 Deg. C in a series that arguably shows no trend. There are two things to note: Adjustments are applied to the period prior to the “discontinuity” only, and the algorithm will force any earlier period that is warmer than the reference period to become cooler. If there is a wave in the data, such as a warm mid-century period, for whatever reason, this is likely to be identified as a discontinuity and flattened out. This algorithm might detect true discontinuities, but it is applied in an automated fashion to the global data by GHCN. It would seem from the figures that the Hadley algorithm is nearly identical. In their Figure 9, they identify the breakpoint with a plausible station move for Spokane, WA, but not for Fig. 8, Carlsbad NM.
This raises some questions: Is this still how they do it? How often is this type of adjustment done in the global dataset? Can anyone find it in the CRU code? Also, note that in data sparse regions, even a very few adjustments could have a big impact on regional contributions to global trends.
RomanM has an analysis (http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/) that bears on the adjustment question. He asked a very clever question: how do the mean adjustments in GHCN line up over time? His result showing the mean annual adjustments by year for the global data:
Between 1900 and 1990, there is a linear trend of more and more negative adjustments for older dates. Corrections for UHI would go the other way (more cooling for recent dates or warming of older dates). It is difficult to find any records in which a UHI correction looks like it has been applied. To me it looks like 1990 is a pivot point. Roman’s result above is consistent with applying the breakpoint algorithm beginning in the most recent years as the “valid” period and tipping down earlier data from the pivot point in 1990, which would make older dates progressively cooler. This is also consistent with the analysis of the USHCN database (R.C. Balling and C.D. Idso. 2002. Analysis of adjustments to the United States Historical Climatology Network (USHCN) temperature database. Geophysical Research Letters 10.1029/2002GL014825) which found a linear trend of progressive cooling of older dates (RAW-FILNET) between 1930 and 1995 that is even more pronounced than in Roman’s global analysis. A recent post on TheAirVent (https://noconsensus.wordpress.com/2010/01/19/long-record-ghcn-analysis/ ) shows a similar result to Roman’s using a different way of interogating the data:
Note that this plot starts in 1900 and is a cumulative plot rather than yearly.
A valid test of this adjustment algorithm would be to test selections of data, see what “discontinuities” are identified, and then evaluate the metadata to see how often they correctly identify true staion moves or instrument changes. In one of the citations they use as authority for the method (Vincent, L.A. 1998. A technique for the identification of inhomogeneities in Canadian temperature series. J. Climate 11:1094-1104) simple simulated climate data were used (without any major local trends such as a mid-century warming) and the algorithm found spurious inhomogeneities in 13.6% of cases. One of the reasons for this false-positive problem is, I believe, that the reference series compiled from nearby stations will be more uniform in time (smoother) than the test series, since it is merged data based on series that are mutually highly correlated. This will make it more likely to find false discontinuities. If only this percentage (13.6%) of stations were adjusted as in Fig. 8 above, then the overall cooling of early dates for a global dataset would be 0.19 Deg. C, which closely matches what RomanM found above. For real climate data (not simulated) I would guess that even more false breakpoints would be identified by this algorithm. Ideally, breakpoints found by the algorithm should be compared against metadata, but this is not done for the global analysis, which is automated. The Vincent ms is the only paper I could find so far that tests this approach to doing adjustments for inhomogeneities. Other papers only discuss applying the adjustments. Note that for the non-US sites, the TOBS and other adjustments for known issues can not be applied due to lack of metadata, nor is it likely Hadley even applies these to US sites (though I don’t know this for sure).
The above results point to either a conceptual problem with the automated adjustments, or a software implementation problem. It also appears that the algorithm is adjusting slopes, not just means, as would be appropriate for a station move or instrument change. When falsely identified breakpoints are sufficiently frequent that they can lead to false trends of the order of 0.25 deg C/Century (30% of the purported warming), then an automated adjustment method would seem to be not recommended.
the Air Vent 
Thank you for this post. It seems the real reason that we have global warming is because some scientists are not very good at maths. When their product is analysed by serious good mathematicians, such as McIntyre and Wegman and others, it is found lacking. I suppose years ago before the warming theory was used by politicians to force social change, nobody much cared anyway what a few scientists were saying about weather, so nobody checked the maths. Now however, that this science is being used to cripple the West’s economies, at long last the maths is being checked, and found wanting.
An excellent post, and some great links.
Everyone has noted the dramatic drop in the number of reporting temperature stations (as noted in one of the links above: http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/), but I’ve never seen any material discussion of why this has happened. Anybody have any suggested reading on this issue?
http://justdata.wordpress.com/2009/12/28/step-by-step-debunking-climate-change/
Just came across this. It may be of interset in the current debate over accuracy of temp records.
#2 Joe D’Aleo at Icecap has blog posts about this. The missing data is mostly out there. There was a special grant by US Dept. Energy in the late 1980s to update the climate data and they pulled in many weather stations up through 1990. After that, they seem to have forgotten how to do it. Many of them are available via the country’s web pages or WeatherUnderground, but none of the big 3 seems to know about it.
After all this manipulation all they can do is show an increase of around 0.6 C over the past 100 or so years, which can be easily explained to be due to natural forces. So, given what little we know about the climate, man might even have forestalled a much higher temperature rise. I know, this is pure speculation from my part, but no less so than IPCC’s predictions of a runaway catastrophic global warming. In fact I challenge the IPCC to prove me wrong.
“I have come across a paper that may help explain how Hadley, GHCN, and GISS do their adjustments and where the warming trends come from.”
Dr. Loehle I have gone over the post a few times. Maybe I am missing something, but I still don’t understand how the adjustment methods you referenced could always lead to a warming bias. Even though the adjustment is applied only to data prior to the discontinuity, shouldn’t that adjustment just as likely be up as down?
re #6 The problem is that the most recent period has up upswing (due to natural cycles, UHI, and GHG). If the period of the 1960s-1970s is going down, this will be detected as inhomogeneous with the “underlying trend” and adjusted downward. Then earlier periods are similarly adjusted downward to match the fixed 1960s-1970s. At least that is how the algorithms seem to me to work from reading about 20 of them. But: it would be good if someone who is looking at the CRU or GISS code could compare what it does with this hypothesis.
Dr Loehle or someone. Does anyone know where the CRU or GISS adjustment code is?
How did the baseline adjustment from surface to satellite LT come about? For HadCRUT it is .146, but what does it really mean? Is there a ratio of height to temperature? The LT and surface are apples and oranges, so how can it be said they are in “good agreement”.
In other words, for any given month, suppose the surface temperature is reported at .5 anomaly and the satellite is also at .5. Since the absolute temperatures are obviously very much different, does the baseline adjustment really mean anything? Is it even correct to say they are in “good agreement”?
This is a good place to repeat the observation that prior to circa 1975, according to the climate models, anthropogenic CO2 and sulfates balanced each other, so any warming that occurred before then is thought to be natural. (For GISS Model E See this figure.)
So the point is, human generated global warming didn’t kick in until after 1975, and for that we mostly have overlap from satellite measurements to “keep that straight”.
It does make a difference as to what happened prior to 1980 of course. Anthropogenic global warming advocates will use it to increase the “scare value” of the warming, mostly out of ignorance because, for all of their protestations, they don’t understand what the sciences they supposedly support actually says.
And there are implications if you get as much warming as you did from say 1850-1980, and it was natural, then that tends to lessen the importance of the warming since 1980, because it implies natural fluctuations are more important than the AGW advocates claim.
For people claiming that CO2 isn’t important, having the temperature trend flatten prior to 1980, actually works against your argument, even though that appears to be what you are saying on this thread, because it’s indisputable that we’ve warmed since 1980, and if there wasn’t really any warming before that, then CO2 is playing an even bigger role in climate than the AGW advocates are saying. (Keep in mind you need to look at 30 year trends here, most everybody agrees that for very short periods, natural climate fluctuations dominate.)
#10 It is correct as you say that most of the anthropogenic warming would be at least post-WWII, but the point here is not to make a political point but to figure out what they are doing at GISS, CRU etc. The pattern of cooling of earlier temperatures shown here just seems wrong. In addition, there is sleight-of-hand going on, where the modelers say “post-1980” warming but the 100 year warming of 0.6 to 0.8 deg C is tossed about like it is also caused by humans. In addition, if this can be fixed then maybe the failure to correct for UHI can be fixed, and we will get to a reliable record.
The described new adjustment is a bit insulting because it assumes that hundreds of our forebears lacked the abilty to read a thermometer. There is no way in hindsight to determine if a temperature step was real or in error, unless an explanation was written at the time. Maybe examination of rainfall, or reconstruction of temps from wet and dry bulb/dew point observations would help, even rainfall coincidences.
In most of the approximations to raw data that I have seen, there is is gentle decrease in temperature from about 1850 to (say) the 1950s. However, for logistic reasons, there are time breaks of continuity in the data. As noted before, Australia’s BOM has rejected a lot of early data but it remains available for those who wish to demonstrate that a method of homogenisation not only works, but is balanced, ups versus downs. You can’t prove this with rejected data or indeed any data with uncertain provenance.
Oh, good! Now I have a thread that has much better context for my debate at RC over the GISS polar extrapolations and interpolations. The Hansen charts that I am talking to are here:
I did a little rough quantitative comparison between HadCRUT 2005 and GISS 2005. Here is what I just commented at RC:
I’ve been talking about the qualitative problems that I noticed in the GISS charts in figure 3 above. So I thought that I would try to take a rough shot at quantizing the problem as well. I used the HadCRUT 2005 chart and the GISS 2005 chart to make comparisons. And I wanted to compare the HadCRUT gridcell row that was furthest north to the GISS gridcell row that occupied the same position. First I counted the number of gridcells in a row. There are 72. Then I counted the number of HadCRUT cells that have data in that row. There are 24. This means that the topmost HadCRUT row has 30% coverage. So I added up all of the covered gridcell anomaly values in the row. The total was 43.8. Dividing by 24 I got an average covered gridcell value for the HadCRUT row of 1.85 C. The GISS row obviously had 100% coverage using interpolation and extrapolation. When I added all of the gridcell anomaly values together for the GISS row I came up with 300. Dividing by 72 gave me an average anomaly value of 4.17 C. So the anomaly for the top row of GISS is 2.25 times as large as that of HadCRUT.
It seems to me that this reflect very badly on the GISS interpolation extrapolation algorithm. The other problem is that there are 6 cells in that top row that HadCRUT has negative values for. GISS turns them all to the maximum positive value. The difference is 6.7 C or greater per cell for those 6 cells.
I can only conclude from this that the GISS divergence from the HadCRUT data is an artifact of the GISS processing algorithms and not a reflection of actual temperature variance at the poles.
re #13 these effects are mostly AFTER the adjustments to station data, and occur when they interpolate into gridcells by weighting certain stations and filling in grid cells with no data.
If an analyst analyzed (1) a randomly selected half of the available data, (2) posted the results or trend line, (3) then analyzed ANOTHER randomly selected half of the same available data (sample with replacement, so that there remains an equal chance of getting outliers,etc, in both runs) and finally overlaid the new result trend line over that of step (2) — how much difference would be visible?
Seems to me if the data truly is “global” and the analysis is correct then two half datasets ought to have very similar results. If the results aren’t similar, then either the problem is actually more localized, OR the method is overly sensitive to extremes in the dataset.
Has such a comparison been done?
re #14. I understand that Craig. But if you look at the results of the filling by comparing available HadCRUT gridcells to what the GISS algorithms produced, you get some simply unbelievable extrapolations and interpolations happening at the poles.
Dear Dr. Craig;
If I am understanding correctly, I think you are suggesting that the choices within the particular implementation will have a big effect, depending on the time chosen to be ‘valid’.
Based on those ‘thermometer count’ graphs, I think the most ‘valid’ records are those when we had the most measurements.
What is your sense of what would occur if the 1960-70’s period were chosen as the ‘baseline’ of ‘validity’ from which other data would be ‘adjusted’?
TIA
RR
As an interested observer, I have no faith in any algorithms that adjust historic temps either up or down. Just seems so dodgy.
Craig Loehle:
I agree completely of course true and I think this whole situation where bias has crept into the adjustments over time is a result of the lack of understanding of the theory by many of its proponents. I really don’t think anybody is deliberately manipulating the data, but confirmation bias doesn’t require that.
It’s interesting how often do we see before/after glacier melt pictures when the “after” pictures preceded 1970? Most people haven’t internalized that 1450-1850 was the coldest period since the end of the last major ice age (hence “Little Ice Age”) and that the “rebound” in temperature from then to 1980 or so was just a return to “status quo”.
Nor do AGW proponents really grasp just how gradual the effect of global warming is compared to natural variability. If they had, they wouldn’t have “overplayed their hand” by pointing to the warming in the 1990s as if it were definitive evidence of human generated warming.
The other interesting thing about all this is if the IPCC predicted 4-6 C rise is to occur, the so called adjustments will have to continue but will quickly reach the point where it’s obviously a fraud as no one will be able to explain the exponentially rising and upward bias. So, it won’t happen and the whole AGW hoax will be understood by all. Hopefully then we can see some of these fraudsters in jail.
9-The Satellite data start in 1979 (UAH has December of 78, but that’s it) but HadCru uses the base period for averaging of 1961-90 I believe. Since the sixties were colder, this means that a lower mean would be subtracted from the surface temperatures than could be subtracted from UAH no matter what base line they used. The UAH base line is like 1979-1998 or something.
#10,11 &19
A cooling bias for the first 1/2 of the century also tends to dampen the warm period of the 30’s and 40’s and make the CWP appear clearly “unprecedented”.
#8
Chiefo has been analyzing the GIStemp Code on his site here:
http://chiefio.wordpress.com/gistemp/
He broke it down step by step and is working on explainations for each step when the events of Climategate overtook him.
Hey folks, just finished a post which I believe provides proof (or at least a clear way to prove) the AGW theories are mathematically invalid. Basically I think the entire approach is broken, least of which are the ‘adjustments’.
http://strata-sphere.com/blog/index.php/archives/12246
Comments welcomed!
#16 I agree. unbelievable infilling
Dr. Loehle, do you know if anyone has applied ecosystem regime shift detection analysis such as found at this site (http://www.beringclimate.noaa.gov/regimes/) to the temperature series to find discontinuities?
And even if there was a 0.8°C rise, is that really significant given the daily ranges temperatures can swing?
As you concluded, no Ajstrata. I came to the same conclusions as you did using a different method; common sense. Thanks for your efforts, much appreciated.
Does this suggest that the adjustments are being driven by an assumption that the temperature trend should be a straight line? It ought to be simple to test the impact of this on some simple curves with added noise, and see what it makes of a simple periodic function where only 50% of a cycle is present.
Loehle 13-14,
You seemed to have proven my point about the limits of extrapolation. You prove only 50% of the cells had data and the ‘extrapolation’ is dodgy at best. Now if you could compare those cells differences to sat data measurements you could prove they were invalid.
The first sentence is wrong, or at least confusingly put. The plot shows the cumulative effect of adjustment. The 1900-1990 downslope indicates a roughly uniform rate of a slightly warming bias in the adjustment. The total dip is about 0.2C, which makes about 0.2C per century warming effect from the adjustment. Not huge, and countered by a cooling effect before and after.
The GHCN adjustment is purely based (outside US) on statistical detection of discontinuities. As such, it is not suited to correcting for gradual change such as UHI.
What it does detect are changes due to discrete events such as conversion to MMTS. That probably explains part of the change
in the 1990’s. The other big effect is due to the change detection algorithm outside US, which terminated most adjustment about 1994. You need a good stretch of data on both sides of a suspected change to confirm, There was a big drop in stations in mid 90’s in the adjusted file, abd the survuvors were mostly US, using metadata-based USHCN correction. I should emphasise that the drop was not in stations reporting, but in stations which could be adjusted by the GHCN algorithm.
Your last plot uses an identical method to Roman’s, just applied to a selected subset.
I’ve shown here that the GHCN adjustments are almost as likely to produce cooling as warming. For almost every Darwin, there is a Coonabarabran.
I see two problems with this method: first, i think it may find correction points where there are none, because of the cyclic rather tha linear nature of local climate patterns. Second, it may also judge real step changes as artifacts. There seems to be broad agreement that parts of Europe went through an upwards step change in around 1988. Wouldn’t one expect this method to do a false adjustment of at least some European stations around that time?
Craig,
“Note that for the non-US sites, the TOBS and other adjustments for known issues can not be applied due to lack of metadata, nor is it likely Hadley even applies these to US sites (though I don’t know this for sure).”
No, but the algorithm could be tested by comparing its output for US stations with metadata, by seeing whether the metadata justifies the adjustments for those stations. Here are the graphs for all stations within 1 geographic degree of Spokane, WA. They probably make up the reference series for that station. Notice that three of them (Pullman, Ritzville, and Davenport) are adjusted downward before about 1965 (1985 for Davenport). The other three (Priest River, Sandpoint, and Moscow) are adjusted downward over the entire record.
Notice also the GISS v. the GHCN versions of the record for Spokane. No obvious trend in the GHCN adjusted plot, but a pronounced one in the GISS. GISS’s methodology (especially their interpolation and infilling methods) seems a bigger problem than GHCN’s.
A better method than trying to construct a global temp or anomaly trend, which requires all kinds of dubious statistical manipulation, would be to plot trends in those areas where sound long-term data is available from multiple stations which do not require adjustments, or where adjustments can be justified with metadata. Then consider those to be samples of the global picture. Even a few such samples (say from the US, Australia, NZ, S. Africa, W. Europe and Japan) should be indicative of the global trend.
#26 yes, they used regime-shift or trend shift detection methods as I cite above, but without any way to check for false detections (thinking it is a discontinuity when it isn’t) and making unnecessary adjustments
#28 as I read it, the algorithms do indeed assume that the temperature trend should be a straight line and attempt to “fix” that if it is not. What a crazy idea.
#30 “almost as much cooling as warming” but as Roman showed, not quite equal. It only takes a small difference in net adjustments to make a trend. There are enough differences that when you add them up there is a net effect of cooling the past relative to the 1990s.
#31 Yes, I think the method would detect the climate shift Europe experienced in 1988 (or whenever) as an artifact, though it does this by comparing nearby stations to one another, so maybe not
#32 yes the method COULD be tested, and when tested they show a significant % of false detections. I am submitting a manuscript (on a different topic) where I cite these studies.
#30, Nick Stokes
“I’ve shown here that the GHCN adjustments are almost as likely to produce cooling as warming. For almost every Darwin, there is a Coonabarabran.”
Like you I’ve been following Willis E’s Darwin in some detail and went as far reproducing his analysis. I then went on to test look to see fi there were many other examples of ‘cooling turned into warming’ and balanced that by also looking for ‘warming turned into cooling’ by the GHCN adjustment algorithm. My analysis can be see on a guest thread ‘Physically unjustifiable NOAA GHCN adjustments’ on VJones ‘diggingintheclay’ blog here. There are literally hundreds of ‘Darwin’ and ‘Coonabarabran’ examples. It clear thatthe NOAA GCHN algorithm adjusts the raw data for many stations for no particulat good reason. Im almost all cases the adjustment is physically unjustifable and is the the ‘stepwise’ nature shown by Willis E for Darwin. Even NOAA have now admitted in an email sent to Willis E shortly after Christmas that ther eare problems with with adjustment process. They have agreed to modify their adjustment processes and re-issue their data in Spring 2010.
If you are interested in looking at warming versus cooling trends (presented in the form of colour coded maps) in the GHCN and GISS raw and adjusted datasets (and the relative temporal differences between them, then have a good look at my latest guest thread on ‘diggingintheclay’ entitled ‘Mapping global warming’ here. My analysis shows that ‘global warming’ (as seen in the raw data) is hardly global at all, but rather is primarily Northern Hemisphere (NH) warming and in fact is mainly NH warming during the winter season in the high latitudes. Here are a couple if images from thatthrea dthat tell it all for me really (hoepfully they will show correctly)
GISS 1970 to 2010 raw data DJF mean slopes
and
GISS 1970 to 2010 raw data JJA mean slopes
#34,
Ok it looks like I cant include embedded images so here are some links to display the two seaonal maps
#30, Nick Stokes
“I’ve shown here that the GHCN adjustments are almost as likely to produce cooling as warming. For almost every Darwin, there is a Coonabarabran.”
Like you I’ve been following Willis E’s Darwin in some detail and went as far reproducing his analysis. I then went on to test look to see fi there were many other examples of ‘cooling turned into warming’ and balanced that by also looking for ‘warming turned into cooling’ by the GHCN adjustment algorithm. My analysis can be see on a guest thread ‘Physically unjustifiable NOAA GHCN adjustments’ on VJones ‘diggingintheclay’ blog here. There are literally hundreds of ‘Darwin’ and ‘Coonabarabran’ examples. It clear thatthe NOAA GCHN algorithm adjusts the raw data for many stations for no particulat good reason. Im almost all cases the adjustment is physically unjustifable and is the the ‘stepwise’ nature shown by Willis E for Darwin. Even NOAA have now admitted in an email sent to Willis E shortly after Christmas that ther eare problems with with adjustment process. They have agreed to modify their adjustment processes and re-issue their data in Spring 2010.
If you are interested in looking at warming versus cooling trends (presented in the form of colour coded maps) in the GHCN and GISS raw and adjusted datasets (and the relative temporal differences between them, then have a good look at my latest guest thread on ‘diggingintheclay’ entitled ‘Mapping global warming’ here. My analysis shows that ‘global warming’ (as seen in the raw data) is hardly global at all, but rather is primarily Northern Hemisphere (NH) warming and in fact is mainly NH warming during the winter season in the high latitudes. Here are a couple if images from thatthrea dthat tell it all for me really (hoepfully they will show correctly)
GISS 1970 to 2010 raw data DJF mean slopes
and
GISS 1970 to 2010 raw data JJA mean slopes
#34,
Ok it looks like I cant include embedded images so here are some links to display the two seaonal maps
GISS 1970 to 2010 raw data DJF mean slopes
and
GISS 1970 to 2010 raw data JJA mean slopes
Let me clarify, I don’t mean to say the methods ALWAYS lead to a warming bias. Individual sites could be adjusted the other way. It is a NET warming bias when summed over thousands of sites.
I’ve got free wordpress. If someone knows how to allow embedded images in this version, please send an email or comment on the open thread. Thx.
#34, KevinUK
The interesting about it being primarily the NH warming from 1970-2010 is that the NH is where the temperatures seems to have the most prominent multidecadal fluctuations, e.g. in the Barents Sea.
#35, Craig,
As Giorgio Gilestro (GG), RomanM, RC and several others (including myself) have also shown it is a relatively small positive warming imbalance that amounts to roughly a third of the claimed ‘unprecedented’ 0.6C/century 20th century warming trend. So not an insignifant imbalance (I personally wouldn’t call it a bias as that implies motive).
Vjones and I are just in the process of preparing a new thread on ‘diggingintheclay’ which shows some very interesting ‘scatter plots’ (similar to RomanM’s but showing warming/cooling trends instead of adjustments) for raw versus adjusted trends for the NOAA GHCN and GISS datasets. It’s basically an extension of GG’s analysis that shows the raw/adjusted data warming/cooling trends in much more detail. Its what I asked GG to do, but he declined as he felt (very inscientifically in my opinion) that becaus ehis analysis had achieved the result he had wanted (‘no smoking gun’) that it wasn’t worth his while doing any further analysis. Thankfully there are several of us, who disagreed with him (and Eric Steig) and who have completed his analysis for him.
I’m still not understanding how the baseline adjustment is calculated at .146 (HadCRUT) between satellite at 4.4 km and near surface.
Is the absolute temperature the same at 4.4 km and the surface? It is said the trend is the important metric however it is often stated the two products are in “good agreement”. It would seem the absolute temperature would decrease with height, no?
Also, according to AGW theory, shouldn’t the troposphere be warming at a faster rate than the surface?
Some days I just don’t feel like arguing about AGW.
This is all you need to know:
I can add a couple of points to what I think was a reasonable explanation of the USHCN methods of detecting significant change points from adjacent/correlating station time series differences. The change point method for USHCN was applied with the introduction of the Version 2 data series. I do not know for certain, but I believe that the mnethod is now applied to the GHCN series also.
GISS does not use change point but rather meta data and in some cases subjective eyeballing to make homogeneity adjustments. TOBS is a large adjustment that GISS and USHCN apply in similar fashion and USHCN uses in addition to change point.
I did some considerable analyses of change points for USHCN data on using the change point (breakpoint) algorithm in R on Version 1 and Version 2 data for individual stations. Version 1 and 2 were both found to have different (and many) significant change points for some stations. The change point method as applied by Version 2 does not eliminate change points. The differences in trends (1920-present) for the Version 1 versus Version 2 for individual stations could be large and in either direction. Version 2 resulted in a greater overall warming trend for the lower 48 US states from 1920-present than Version 1 by something greater than 0.1 degree C per century – as I recall.
There are statistically significant differences in some time periods and regions of the globe between GISS and GHCN and since we do not have an absolute with which to compare these series, we have to compare one to the other and assume either one is incorrect and other correct or they might both be incorrect.
I have been searching for information on how CRU makes adjustment to raw temperature data and without much success.
It was an eye opener when I went to individual station differences for the USHCN Version 1 and 2 series and found large differences for not a small number of stations. Particulary this is true when you consider that USHCN made claims for the Version 1 series and used it for many years. The USHCN literature does little in the way of individual station comparisons between series versions. What I think individual station differences between versions does is put a larger uncertainty (variation) on the station temperature and trend estimates then other ways of looking at the data would. That is why I am currently comparing individual station temperature trends within 5 x 5 degree grids.
The initial results of the gridded analysis (US/IL centered)indicated to me that these stations retained similar features that might be climate related or alternatively something the adjustment put in the series. The raw data would have to be analyzed to better point to cause here. Temperature anomaly trends within the grid had large variances over several time periods used with the longest being 1900-2005.
Nick Stokes:
You’ve produced a leptokurtic distribution, which is suggestive in itself that there is a problem.
I believe RomanM’s temporal distribution of the corrections is more diagnostic, and they do suggest there are issues with the adjustments.
#44. Leptokurtic, now that’s a good word to bring up at a party.
Re: Espen (Jan 21 10:22),
What would have been surprising would have been if the temperature in the Barents Sea wasn’t strongly correlated to the AMO. Still, it’s nice to see confirmation. The AMO index correlates to the total Arctic ice extent anomaly with an R^2 of 0.32. I’ll have to try that with just the Barents/Kara Sea data.
#30 Nick Stokes I consulted with RomanM and his plot is annual mean monthly adjustment NOT cumulative. In my opinion the only way to get such smooth results as Roman shows would be a bias toward a type of tipping of earlier periods to match the slope of the most recent period.
vsaluki, you raise an excellent point.
Where is your comment on rc? I can’t see it.
I have just placed a similar query, should appear at around #460.
” PaulM says:
Your comment is awaiting moderation.
21 January 2010 at 12:38 PM
Apologies if this has been answered already (I can’t read all 457 comments, most of which seem to be off-topic!) but how does Hansen justify the dark red area at the North pole in fig 3? And the corresponding spike at 90N that is not in this article but is quite striking in the plot if you go to the GISS web page. Clearly this cannot arise through interpolation. I don’t think he has explained this adequately. ”
Maybe Jeff or someone could do a separate post on this.
OK, I have found vsaluki on RC, he is Tilo Reber, comment #401, and most recently #458. He clearly has gavin flustered. Gavin has no answers, merely accusing him of being beyond silly. Great stuff. Anyone else want to join in?
#42
Kenneth according to Dr. Peterson from NCDC GHCN doesn’t use the USHCN v2 adjustments yet. They are are going back to the raw data and reworking the dataset. They are also suppose to be incorporating new data as well. He stated that the GHCN dataset with the USHCN v2 adjustments should be ready either next month or in Mar. He classified it as GHCN v2.5 but the biggest thing is that they will be releasing the computer programs and intermediate files with the change so people can see how and where in the process each change is being done.
re #6
“The problem is that the most recent period has up upswing (due to natural cycles, UHI, and GHG). If the period of the 1960s-1970s is going down, this will be detected as inhomogeneous with the “underlying trend” and adjusted downward. Then earlier periods are similarly adjusted downward to match the fixed 1960s-1970s. At least that is how the algorithms seem to me to work from reading about 20 of them. But: it would be good if someone who is looking at the CRU or GISS code could compare what it does with this hypothesis.”
Craig, I don’t think this is quite right. As you quote, what Peterson says is that “a linear regression is fitted to the part of the (candidate-reference) difference series”. Why should a general trend in temperatures upwards or downwards, on average, affect one way or the other the difference in temperature between the station being adjusted and the reference series (which is constructed from the records of other stations in the same area)?
It does however seem theoretically possible, as I pointed out here: that where a substantial proportion of the reference stations have had their trends increased by a UHI effect, the adjustment might be to increase the trend of rural stations to match. This was perhaps a more remote possibility under the reference series method described by Peterson (which incorporates an additional test based on a 12 year window) than under the new pairwise comparisons method used by US HCN and (I understand) shortly to be adopted by GHCN.
#51
Sorry, the tags went wrong. Only the word “difference” was meant to be in bold. And the missing hyperlink address was https://noconsensus.wordpress.com/2010/01/19/long-record-ghcn-analysis/#comment-18692
I thought was for inserting a hyperlink and that it would show the actual URL if the title part were omitted, but obviously I was wrong 🙂
#51 There is an assumption that the only thing going on from one station to another is either similar weather or some anomaly that needs adjusting. What the “a linear regression is fitted to the part of the (candidate-reference) difference series” means is that if the difference between the candidate and reference shows a linear trend, they “fix” that to make them similar. So, as soon as some station has a difference from nearby stations (which, as you say, may have a UHI effect) they fix the outlier to make it match. BUT without proof that this does not just reflect the inherent error at and natural differences between nearby stations.
OT.
Dr. Loehle, Paul Brand has commented on open thread #1 (comment #56) the following:
“According to Loehle analysis, the MWP was 0.5C cooler than today, and doesn’t deviate materially from the peer-reviewed reconstructions included in the IPCC report.”
I thought rather than continue my discussion with him on this comment that I would ask you to respond.
Craig #47,
What Roman plotted was simply the difference between the adjusted and unadjusted temperatures in each year, That represents the cumulative effect of the adjustments made between then and now.
Why I responded to your way of putting it is that it creates an impression that adjustments are happening at a greater rate in the past, or at least having a more negative effect. But if you make adjustments at about the same rate over a time, with the same small warming bias, then you’ll see just the linear slope that Roman’s plot shows.
I think one of the issues here is using 1979-2000 as the mean period to calculate anomalies, this make it appear that stations today compare well, but in the past didnt so well, then automated processes are used to pull the older data in line i.e. 1950-1900.
Surely its beeter to use the start for the baseline i.e. 1900-1930 to see how rural and urban stations have diverged over time and then pull the urban stations in line with the rural stations
BTW:
NIWA now refer to a couple papers that apparently describe their methodology for the NZ data mentioned above.
http://www.niwa.co.nz/news-and-publications/news/all/nz-temperature-rise-clear/seven-station-series-temperature-data
**THE ABOVE LINK GIVES ACCESS TO THEIR RAW DATA IN EXCEL**
Worth looking at if you know your stats!
Nick Stokes,
Your comment in #55 only makes sense if you use some definition for ‘adjustment’ to be other than the difference between the adjusted and unadjusted annual temperatures. If you are using a different definition please say so, and be so kind as to define the term. Thanks!
#55 I am not sure what you mean about making adjustments “over a time”. The plot by Roman is NOT the effect of adjustments made in the past, but of the most recent version of the data, in which past data was “adjusted” when they created whatever is the latest version. Over many stations, the net effect was that raw data from the 1930s had more that 0.2 Deg C subtracted from the raw data mean. Why should it be that the mean from the 1930s needs to be dropped down relative to 1990? The UHI effect would say it should be raised up (or 1990 lowered).
Craig Loehle (#58),
I’m speculating that Nick Stokes is thinking in terms of an annual adjustment as the incrementental change from one year to the next in the delta of adjusted and unadjusted temps. Why he thinks that only he can answer. It’s a pointless distinction though, in stating that the derivative of a straight line is a constant.
I take issue with Nick Stokes’ chart reading abilities. The slope from 1920 to 1990 is more like .3 C per century. Any graphical display of temperature using GHCN adjusted data has this manufactured trend showing over the time period of 1920 to 1990. If such a graph does not start at 1845 (approximate start of downward adjustments) then it presents a wrong impression of the adjusted data.
Earle,
The GHCN algorithm proceeds by identifying circumstances where it makes a discontinuous change to the data. That’s what I mean by an adjustment. They are discrete, and usually infrequent.
Craig #59,
The GHCN adjusted data set is not something that is altered regularly. It is v2, released in 1997 and widely distributed on CD. v1 was released in 1992. I believe the adjustments as indicated on that file are just the ones made in 1997.
As I mentioned above, the purpose of the GHCN homogeneity adjustment is to recognise, without metadata, discrete events such as station moves, changes in equipment etc, and adjust accordingly. It isn’t expected to identify UHI or other gradual changes. USHCN does that, but not GHCN.
#62
I should add that of course there is data in the set more recent that 1997. I think those are US stations, where the USHCN adjustments are folded in.
Nick Stokes: you are correct that they make a discrete adjustment to some period of time, like 10 or 20 years worth of data for one station. Note that if they identify, correctly or not, a step down at a certain year, all years prior to that will be adjusted downward to “fix” this bias. Note also, that the same station can get multiple adjustments, which could tend to make earlier periods increasingly likely to be adjusted down if there is a bias, thus giving the results shown above. I am also interested in not just GHCN but also Hadley and GISS which also make adjustments in a similar manner.
Not Nick Stokes! He stunk up the WUWT forums with is bull****.
[Stuck in a hotel near the Detroit airport because the airline messed up so I have the opportunity to make a comment!]
Nick, I don’t understand what point you seem to be making regarding “cumulative” adjustments. An adjustment is not a one-time raise or lower, and then next year it is back to what it was. What I calculated was based on the simple difference between adjusted values and the (presumably) raw data.
To put it into another context, suppose for the moment that all of the stations had adjusted and unadjusted data for all of the years. Now calculate the trend (least squares slopes) for the adjusted – unadjusted values for each station. Average these trends.
Now, calculate the same statistics that I did. Difference (adjusted – unadjusted) for each month for each station, average the differences for each station for each year, then average the yearly averages of all stations for each year. Fit the least squares linear trend to the result. You will get exactly the same trend as in the previous case of averaging station trends.
The graph that I made IS a good approximation of the contribution of the adjustments toward the warming trend. The adjustments on the average induce a noticeable increase in the temperatures. However, what is still somewhat an open question is which stations are being used in the final calculations by the end users and how just much that actual increase is.
With regard to the point of Craig’s post, using an automatic test not based on actual documentation for detecting changes in a collection of records (particular when the series do not behave in a linear fashion) is at best statistically naive and at worst foolish.
To understand this whole topic, it helps to read papers cited above like Easterling and Peterson (1995a,b) . These give mechanisms and some reasons. But they also admit to widescale guesswork for in filling missing values and correcting step changes.
While one can easily comprehend the paper, it is entirely different to work out which of the proposed corrections was eventually applied to what and when and for how long and if it was withdawn and replaced by other corrections.
Then it gets really complicated when an adjuster like Giss has some early data that has not been through the national meat grinder, plus later sausage data that we the poor readers cannot disentangle because of a lack of metadata.
Again I plead, do not assume that you have “raw” data.
44.Carrick said
January 21, 2010 at 12:03 pm
Nick Stokes:
I’ve shown here that the GHCN adjustments are almost as likely to produce cooling as warming. For almost every Darwin, there is a Coonabarabran.
…………………..
Nick, I have already pointed out elsewhere that the early years of Coonabarabran, up to 1957, which you claim were adjusted towards cooling, were disowned by the Bureau of Meteorology on quality grounds and are deleted from recent compilations. Did you not read this, or are you comfortable with the USA adjusters ignoring the deletion of bad data from the record?
http://wattsupwiththat.com/2009/12/20/darwin-zero-before-and-after/ Geoff Sherrington (17:08:48) :21/12/2009
Yours was not a quality post.
#68 Geoff,
I don’t feel a strong need to go into the validity of the treatment of Coona. My point is simply that the GHCN adjustment algorithm can yield cooling as much as warming.
On #67, I don’t think there’s any evidence that GISS or CRU use GHCN adjusted values. In fact, it’s not clear that anyone does.
Roman #67
I said the cumulative effect of adjustments. My point is really that people seem to misinterpret your very simple plot as indicating an adjustment process that gets worse as you go back in time. Or something. I’ve read a lot of people suggesting that we should be alarmed by it, without saying why.
So I’m trying to explain how the GHCN adjustment process should be seen. Craig has it right in #64. The adjusted temp is required to equal the unadjusted at the present time. Then going back, the algorithm may decide at some times that a discrete adjustment is needed. If so, then the whole curve before that date is moved up or down by a discrete amount (often 0.7C). So the difference between the adjusted and unadjusted in any year, for any station, is the cumulative effect of these step changes, between that year and the present.
Actually there is sometimes noisy small apparent adjustment as well. But the main effect are these big discrete changes, which is what Peterson’s paper describes.
That’s why I think your plot has a simple interpretation. The gradient represents the average bias of changes made in any one year. And it is fairly uniform (and warming, and not very large – about 0.0023 C/yr) from 1900 to 1990. Not so surprising.
#44 Carrick, I don’t understand your focus on the kurtosis. It isn’t important that the distribution is normal – the issue is whether it is symmetric. And it is close, the mean being 0.0175 C/decade. That’s the extent of the warming bias.
There’s a simple reason why it is leptokurtic. A disproportionate number of stations are not adjusted at all, often because there isn’t a long enough stretch of continuous data to be satisfied that an adjustment could be warranted. So there’s a spike at zero.
Something else that occurs to me – although I’m not sure I’ve got the detail right. The adjustments are looking for hinge points, as if the thermometer sensitivity to temperature is changing. Does that ever occur? UHI could be postulated to introduce a ramp in temperature, I suppose, and maybe vegitation growth, but how about station moves? surely thay ought to be step changes, and that could be identified by optimising a 12+12 year window to fit a curve. We could then look at how the function reacts to an el-ninio spike, and probably have to come up with another approach (cos I can’t see it working too well). I reject the analysis that says ‘looking at the applied adjustments they are balanced – this is not a good test of the process. A good test takes known data, adds noise and an error, then compares the extracted result and the residuals.
#44 KevinUK,
kevin, I read your latest post, and yes, the dropoff in stations listed is dramatic.
But I think this may misunderstand what GHCN is. It isn’t NOAA, GISS or Hadcrut. It is a historic climatology network. It was actually a 1990’s project, described in this Peterson/Vose paper. It produced V1 (1992) and V2(1007). A lot of its work was the locating and digitisation of old records. The old material is the foundation of other databases, but it’s not itself meant to be comprehensive to the present.
Peterson said it thus in 1997 (Sec 8)
#73 errata – I meant V2(1997) And the smiley was accidental: (Sec 8 ) 8)
There is an internal check on temperatures that is possible if sufficient auxilliary data have been gathered:
Q: How do I calculate dew point when I know the temperature and the relative humidity?
A: Relative humidity gives the ratio of how much moisture the air is holding to how much moisture it could hold at a given temperature.
This can be expressed in terms of vapor pressure and saturation vapor pressure:
RH = 100% x (E/Es)
where, according to an approximation of the Clausius-Clapeyron equation:
E = E0 x exp[(L/Rv) x {(1/T0) – (1/Td)}] and
Es = E0 x exp[(L/Rv) x {(1/T0) – (1/T)}]
where E0 = 0.611 kPa, (L/Rv) = 5423 K (in Kelvin, over a flat surface of water), T0 = 273 K (Kelvin)
and T is temperature (in Kelvin), and Td is dew point temperature (also in Kelvin)
So, if you know the temperature, you can solve for Es, and substitute the equation for E into the expression for relative humidity and solve for Td (dewpoint).
http://iridl.ldeo.columbia.edu/dochelp/QA/Basic/dewpoint.html
Thanks to Michael Bell, author.
Nick, why do you not blush? Are you defending this graph?
http://i260.photobucket.com/albums/ii14/sherro_2008/Darwindifferencespaghetti.jpg?t=1264157721
Series 4 is the Giss Adjusted that ranges over -2 to +2 degrees C compared with the latest online BOM data. Series 5 is Hadley (-4 to +0.5) while series 7 is NOAA unadjusted version 2 (-4 to +0.3 then to -3.)
Yep, those adjustments sure work fine.
#76 Geoff, I can’t defend the graph. I can’t understand it. You’d need to explain.
#73, Nick
Thanks I already have the Peterson/Vose paper and already know the inter-relationship between the GHCN dataset and how NOAA, GISS and CRU use of it.
That’s all besides the point though isn’t it? The point is why hasn’t this important (particularly for the rest of the world outside of the US) dataset been updated? Are you saying that its acceptable for NOAA, GISS and CRU (and so therefore Hadley as well) to continue to use this dataset as their primary source of land surface temperatures for producing all those wonderfully scary gridded anomaly maps? Are you saying that no one was interested in measuring temperature in France, China, most of Africa etc prior to 1950? Is OK to use this truncated data as one of the primary sources for formulating policy that involves trillions of dollars without att least attempting to keep it up to date? It’s Ok to use this GHCN (in combination with USHCNv2 and SCAR) dataset as GISS clearly do (as well as Hadley) to make statements that the first decade of the 21st century has been the warmest decade on record?
Nick Stokes: whether anyone uses GHCN or not, the same type of adjustments are used by all of them. The question, as Geoff put so well, is exactly which ones are in the computer codes and how are they implemented? Whatever they did, raw vs adjusted suggests an artificial trend is introduced (and other analyses elsewhere have done similar things for GISS, Russia etc) Let us say for argument that it is “only” a 0.2 deg C trend effect that results. This is 25% of the 0.8 deg C (upper end estimate) trend over the past 100 yrs, or 30% if you take the 0.6 deg C trend used in some places. If you subtract another bit as due to UHI, you don’t have much left, do you?
Craig Loehle @ #53:
It is my view that the candidate-reference difference series is analyzed looking for statistically significant change points. What you have described would appear to be a trend (significant?) in the difference series without necessarily having change points. That presents an interesting question: What would be adjusted in that case? From what I have read it would not require an adjustment. Or perhaps I misunderstand and you are talking about the segmented trends between change points.
#80 Kenneth There are several methods cited and described in documents. Some just fix a discrete step at a changepoint but others also adjust trends. Which did they use? Don’t know.
boballab @ # 50:
Thanks for the update on GHCN adjustment versions. I needed to know that before I completed my GHCN grid analysis.
Via Boris on Lucia’s Blackborard, there’s a new paper on surface station sitings:
Click to access menne-etal2010.pdf
Carrick @ #83:
I quickly scanned the article and I was disappointed in the lack of depth in the analysis. They grouped CRN ratings 1 and 2 versus 3,4, and 5. When I did the analysis with RomanM on Watts CRN ratings the groupings were CRN123 versus CRN45. RomanM did a much more in depth study then these authors and I was hoping that the Watts team was going to take the initiative for doing something similar when more stations were rated. I have reservations about the owners of these data sets doing these types of analyses. Trust but verify.
Kenneth Fritcsch, hopefully Anthony will release his surface station data at some point, and more complete analysis can be performed. I view this as basically the analysis that JohnV should have written up, not the final word on this story.
KevinUK #78
“Are you saying that its acceptable… “
Yes to all those. GHCN is what it is. A historical data set compiled in 1992-1997, available for all to use. GISS, CRU etc can use whatever is valuable to them, and they have to use their own sources of data post 1997. They’ll probably supplement with their own sources pre-1997, but GHCN provides the bulk. If you think GISS/CRU grid coverage is inadequate, it’s an issue for them, not GHCN.
#86, Nick
I asked
“Is OK to use this truncated data as one of the primary sources for formulating policy that involves trillions of dollars without at least attempting to keep it up to date?”
Your answer
“Yes”
I think this says a lot about you Nick! It certainly explains why you seem to want to defend the indefensible. Witness your complete in ability to accept the fact that the Darwin adjustments carried by NOAA and others are just ludicrous.
As you appear to be someone who is prepared to see trillions spent on what appears to be an insignificant and at the very least certainly an over exaggerated problem, can I ask you just what would it take in your opinion to disprove as the IPCC wants us to believe that CO2 is not the primary cause of the late 20th century warming period? Remember these trillions of dollars that you appear to be not mind spending on mitigating the effects of AGW could be spent instead on some real global problems like protecting people from real threats to their lives like earthquakes for example.
Kevin #87
You’re missing the point. Of course up-to-date information is needed. But it doesn’t have to come through GHCN. They were never the primary source of the original data, and they’ve done what they were supposed to do.
Of course, it would be fine if someone did update GHCN as Jeff is suggesting on a new thread, But that doesn’t invalidate what is there.
#73 KevinUK
Apologies are due here. As Boballab pointed out on another thread, The GHCN raw file has been updated to a much greater extent that I had believed, based on Peterson’s 1997 statement and my knowledge of the adjusted file (which hasn’t). Presumably this has been kept up by NOAA. And GISS says that it is what they primarily use.
So I have to readdress your questions. “Is it OK to use truncated data…?” Well, I guess it depends on whether there is enough of it. They have to truncate somewhere. GISS clearly believes that they have enough, and that the data they do not use carries more risk than benefit.
Something I not clear on – do they calculate adjustments every time new data is added (.e.g. every year)? I am trying to understand the migrating 1935 anomaly (from the GISS emails). It would seem to me that once it had been calculated, it was done.
#90 Kan It is my impression that at least some of the GISS, NOAA, and Hadley sources update frequently, because new incoming data can provide additional record length to evaluate and adjust past data for inhomogeneities. How frequently? Don’t know. but this is not the same as how often GHCN updates–separate issue.
Carrick @ #85:
I am going to dig out my link to RomanM’s analysis of Watts evaluations, but in the meantime I want to point out that John V’s analysis, as he posted it at CA, was incomplete, inadequate and misleading. Do not get me started.
Here is what I have posted on several blogs on the analysis of the Watts team CRN evaluations – and usually in response to someone commenting on some analysis done by John V. I am totally frustrated by the lack of acknowledgement and appreciation for what RomanM did here. The link to RomanM’s CA post with all the analysis results is at the bottom of this post.
http://climateaudit.org/2008/06/08/homogeneity-adjustment-part-i/#comment-150852
Thanks for that post Kenneth. Like you mentioned, I allways heard a lot about John V’s findings. I did not know of Roman’s post (predates my reading of climate blogs) until your comment.
#89
Nick Stokes,
“The GHCN raw file has been updated to a much greater extent that I had believed, based on Peterson’s 1997 statement and my knowledge of the adjusted file (which hasn’t). Presumably this has been kept up by NOAA. And GISS says that it is what they primarily use.
KevinUK and I have downloaded the GHCN v2.mean file from both NOAA and GISS FTP sites and compared them: they are identical. Other than adding SCAR and USHCN this is GISS’ primary source (as you now say).
We have also compared the GHCN adjusted file to the raw file. In terms of numbers of stations in each and the numbers whcih are adjusted – you can see the results, compared down to a WMO regional level here: here
“So I have to readdress your questions. “Is it OK to use truncated data…?” Well, I guess it depends on whether there is enough of it. They have to truncate somewhere. GISS clearly believes that they have enough, and that the data they do not use carries more risk than benefit.
Just look at Figure 4[A] and tell me that 25-100 stations is adequate for reporting temperatures in Africa over a 20 year period. There is a strong baseline period in Africa – 1950-1980 with over 300 stations. GISS is so confident of the data that they adjust less of it than any other region, but suddenly it evaporates and we are left with as few as ~25 stations at one stage to cover l11,608,000 Sq. Miles – that is 20% of the land area of Earth.
Just how does the data they do not use carry more risk than benefit? The African data has less adjustment than any other region, so they must be happy to use it between 1950 and 1990 – so tell me just what would the risks be?
Pielke Sr. has a post up right now regarding discussions / presentations at a meeting in Exeter. Here is a sample of some quotes attributed to Matt Menne, Claude Williams and Jay Lawrimore:
Maybe they read your post here Craig.
http://pielkeclimatesci.wordpress.com/2010/09/20/candid-admissions-on-shortcomings-in-the-land-surface-temperature-data-ghcn-and-ushcn-at-the-september-exeter-meeting/
Menne’s presentation is here:
http://5676430411356704223-a-surfacetemperatures-org-s-sites.googlegroups.com/a/surfacetemperatures.org/home/exeterworkshop2010/7_1Wed_exeter-menne.pdf?attachauth=ANoY7coAIkyVm7RRldu6mKlYQYpN4_b2CXm9Vc0t5-a39SORRXspGFOK0X7DdDPx4bEimYhrG8YIivXv3z74eBmckqRPqsUQ_6TyvvDj95W_oupiQHVkNctD-6xDLdYm09hSjvOZsUSjtHHg2ytOFtJMsODJvBi5i4XQ9fInsFUJocuXKU_gUbNzh54kV5OkIzgTvEezH9BGaz_oIfcHPmlqeZybOQm_2SfAmp_nMkUgjrG0HSq7zj0%3D&attredirects=0
Interesting stuff.
(Jeff, which tag do you use to embed the link into comment text?)
Testing….Menne presentation
Testing… Menne presentation here