Just being Captain Obvious here, but none of the sample data will allow a global temperature even for the USA. Period and end of story. What can be said, you have historical data on several temperature reading sites. One can not use it for global or average temperature at all. Just not correct.

Sadly, the temperature data can say maybe for a city that the temperature has changed over time but that is it. All this was well known back in the 1950’s and 1960’s. That is why the thought was to have Satellites measure the overall surface. But, some snags there as well, one is the aging of Sat’s sensors has not been sufficiently verified as recommended by the engineers. For instance, an uniform swath of land with temperature readings would need to be verified by the Sat data. This comparison would give a confidence number as to the correctness of the Sat equipment. But once again, only done initially and dropped due to budget cuts. And, Sat sensing has its problems due to Ionosphere interference, radar curvature problems, atmospheric interference. Hence, the sampling was never corrected. This was deemed OK since they were just used as a general feel not tenths of a degree accuracy. Maybe plus or minus 5 degrees C but then one could not say if the world was warming or cooling a degree.

Säntis (Googel Earth: Saentis, Switzerland) is a peak at 2500 m altitude in a chain of mountains. The only building (which has been there for decades) is the top station of the cable car and a mountain restaurant. There is no way this station can be contaminated by urban heat, unless it sits on the roof of the restaurant. There is no way to move the station more than a few meters before it falls off. Yet, they have “homogenized” the temperatures.

Take any noisy time series – be it the CRU index, an individual station, or a stock price. Break it up into intervals (your choice). Perform a linear regression for each interval and tack them end-on-end. Then compare that result to the linear trend using all the data.

No worky.

The problem is that in the regression model y=mx+b, you have to keep track of all of the b’s (the y-intercept). In other words, even if you try just to use trends, you still have to keep track of the intercepts, which means you have to know what the offset is from one segment to the next. Now how do you know if a “shift” is due to real no-shit temperature change? Or a station move? Or UHI? Using trends gives you no additional information.

The other issue is with anomaly baselines. When it comes to climate, we only care about the change. That change has to be relative to a baseline. If you have incomplete records, you cannot baseline the records to the same period without figuring out what the offset is from one station to the next. With anomalies, the issue is intractable. You can’t go back and add more stations, so if you want to use information that only has partial coverage temporally, you must adjust.

In other words, the trend thing won’t work.

Encourage , if you will, a volcano (or 2) to go off. They really don’t do to much damage to the immediate surroundings, and would drop the old global T pretty quickly…

What would be the effect of truncating short records at multiples of 365 days length, and fitting the trend to each one as #37 was suggesting?
Whenever a purported discontinuity (change of equipment, location, etc) just separately truncate that segment and then mod365.25 days. Hmm, tricky to do that fractional day thing…

I like the idea of avoiding as many ‘adjustments’ as possible.
I’m not sure of the value of trying to adjust the segments to get rid of shifts…
Talk is cheap; I’m not saying anyone ’should’ do this.
I’m just curious what would be wrong with this approach.
TIA
TL

It’s not hard and the data is available.

Now please think about the quality control claimed by USHCN for its stations and then what the Watts evaluation team discovered and then tell me that you have good faith and trust in the TOB and change of station documentation that is claimed or at least implied by the temperature series owners. Look at the sloppiness revealed of the CRU data set owners in “read me Harry” and in losing raw data. Look also how all this is waved away by the owners and claimed it does not matter. And finally look at how the users of these data and other parties who should interested in accurate data shrug and carry on.

Your post proves a point, but perhaps not the one you wanted to make.

The initial mistake I made was to assume GHCN was adjusting raw data, but at least in the US, it has already been adjusted a number of times by NCDC for the USHCN. Each step introducing more possible smaller error while removing gross discontinuities. It has been an eye opening journey to see how many times the data is adjusted by various algorithms. I’m sure I still don’t have it all completely right, but this is what I have seen so far at least for US data.You may want to check that the raw is actually raw.

My observation is that the accumulation of these uncertainties appears to exceed the range of detected warming signal that is claimed.

Every time someone adjustments the data they also increase the band of uncertainty. This uncertainty builds upon the uncertainty already in the raw data prior to hand off to GHCN. Many of the adjustments appear to have legitimate reasons of trying to remove large discontinuities or false overall trends. However, every time they modify the data with an algorithm that reduces the volatility, they add some uncertainty even while these other problems are fixed.

In the US, the data is first gathered daily from the station.
The data is collected and reported in 1 degree F increments.

DSI-3200 Page 4:
“The accuracy of the maximum-minimum temperature system (MMTS) is +/- 0.5
degrees C, and the temperature is displayed to the nearest 0.1 degree F. The observer records the values to the nearest whole degree F. A Cooperative Program Manager calibrates the MMTS sensor annually against a specially maintained reference instrument.”
http://www1.ncdc.noaa.gov/pub/data/documentlibrary/tddoc/td3200.pdf

So before any adjustments are made, the data has an error range of +/- 1F or .5C !

Then it is adjusted least 3 times before it is handed to GISS
http://www.ncdc.noaa.gov/oa/climate/research/ushcn/

Time of Observation Bias Adjustments (Adjustment #1) (Error range unknown)
“Next, monthly temperature values were adjusted for the time-of-observation bias (Karl, et al. 1986; Vose et al., 2003).

The TOB-adjustment software uses an empirical model to estimate and adjust the monthly temperature values so that they more closely resemble values based on the local midnight summary period.”

Homogeneity Testing and Adjustment Procedures (Adjustment #2) (Error range may be shown by NCDC, see below)
“Following the TOB adjustments, the homogeneity of the TOB-adjusted temperature series is assessed. In previous releases of the U.S. HCN monthly dataset, homogeneity adjustments were performed using the procedure described in Karl and Williams (1987).
Unfortunately, station histories are often incomplete so artificial discontinuities in a data series may occur on dates with no associated record in the metadata archive. Undocumented station changes obviously limit the effectiveness of SHAP. To remedy the problem of incomplete station histories, the version 2 homogenization algorithm addresses both documented and undocumented discontinuities.”
Estimation of Missing Values (Adjustment #3) (Error range unknown)
“Following the homogenization process, estimates for missing data are calculated using a weighted average of values from highly correlated neighboring values. The weights are determined using a procedure similar to the SHAP routine. This program, called FILNET, uses the results from the TOB and homogenization algorithms to obtain a more accurate estimate of the climatological relationship between stations. The FILNET program also estimates data across intervals in a station record where discontinuities occur in a short time interval, which prevents the reliable estimation of appropriate adjustments.
Urbanization Effects (NCDC says this is covered by their Homogenization algorithms)
In the original HCN, the regression-based approach of Karl et al. (1988) was employed to account for urban heat islands. In contrast, no specific urban correction is applied in HCN version 2 because the change-point detection algorithm effectively accounts for any “local” trend at any individual station. In other words, the impact of urbanization and other changes in land use is likely small in HCN version 2.”

Now after starting out with an observation error range of +/- 1F (.5C). Every one of these prior adjustments adds uncertainty to the data.

For example:

1. Raw Data point is 15C +/- .5C That means the range is 14.5C to 15.5C
2. The first adjustment add +/- .25. Now the range is 14.25C to 15.75C
3. Second adjustment +/- .25 Now the range is 14C to 16C
4. Third adjustment is +/- .505 Now the range is 13.5C to 16.5C!

I chose these numbers for 2,3 & 4 as examples, I do not yet know the real numbers. However I chose their net value because the NCDC gives an example in the document that describes their process. Their chart for Reno shows error bars that look to be around 1.8C to 2C range error introduced by their adjustments. This is additive to the +/- .5 C built in to the raw measurement as there is no indication it includes the raw error range. It could actually be worse as it is unclear if the TOB & Missing data interpolation is part of their uncertainty calculation.
http://www.ncdc.noaa.gov/oa/climate/research/ushcn/

Only after these adjustments are doen GISS get the data for merge into the GHCN.
http://data.giss.nasa.gov/gistemp/sources/gistemp.html

Then, incredibly, they make more adjustments!

http://data.giss.nasa.gov/gistemp/sources/gistemp.html
First they remove the Adjustment #3 data from above. So in a sense they remove one source of error. (Adjustment #4)
“The reports were converted from F to C and reformatted; data marked as being filled in using interpolation methods were removed.”

If they only remove part of Adjustment #3, then more uncertainty is introduced.

This indicates GISS is using the NCDC/USHCN data that has been through Adjustment #3 and they take it mostly back to Adjustment #2.

Then, they homogenize the data again! (Adjustment #5)

“The goal of the homogeneization effort is to avoid any impact (warming
or cooling) of the changing environment that some stations experienced
by changing the long term trend of any non-rural station to match the
long term trend of their rural neighbors, while retaining the short term
monthly and annual variations. If no such neighbors exist, the station is
completely dropped, if the rural records are shorter, part of the
non-rural record is dropped.”

The specific stated goal of this adjustment is to take into account Urban Heat Island effect. Yet, NCDC says they already adjusted for this when they homongenized the data! Now the data is twice baked for UHI. (Which I thought the IPCC and Jones, Wang (1990) said was negligible)

A bit concerning is the video data analysis show UHI is still in the data by sampling city/urban pairs around the country. Hmm… are we smarter than a 6th grader..:-) (I need the URL, but it is well known and easily reproduced independently)

Roman M has done an excellent job in his blog showing these GISS adjustments are driving a bias into the data.
http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/

I have sampled numerous individual stations in CA and seen the same bias being introduced by this this process. I described the process for doing this in comments over at WUWT. Others using that method have found the same results in NY, Grand Canyon, Calgary, etc. http://wattsupwiththat.com/ (I need to find the exact links, sorry)

So there are two building concerns about this final step. First, it appears duplicative to (at least in the US) what has already been done. Second it appears to be reshaping the curves to fit the story. No one has any indication if this is just a bad algorithm or deliberate.

Don’t forget, this also introduces uncertainty. NCDC estimated their homogenization introduced up to +/- .9C if I read their example chart correctly.

So right now it appears the inherent cumulative range of error far exceeds the claimed warming signal that has supposedly detected.

This chart shows GISS claiming a warming signal of .6 C detected.
http://data.giss.nasa.gov/gistemp/2008/

Remember, even the raw data was +/- .5 C from the moment it was written down per the NCDC.

The most incredibly generous reading of the Reno example for NCDC (assuming it includes the raw error rate, TOB, Missing Value Estimates included) shows an error band of at best +/- .9 C.

Then there is whatever uncertainty is added by the GISS process that also appears to bend the curve.

I don’t claim to know all the right numbers for the total uncertainty being introduced into the data. However, this error budget must be fully disclosed and understood. It certainly appears that the signal is less than the noise introduced by the original measurement and the up to 5 adjustments. Exactly by how much is critical to prove the point that a warming signal has in fact been detected.

I don’t know what process the raw European data goes through. Until someone describes the path from observation to the end, it is hard to say.
Your post show early 20th century adjustments of +/- 1C. Given my earlier mistake on GISS “raw” that turned out to already have been modified by NCDC, you may want to investigate if your data may be in the same position. Assuming a similar raw observation error of +/- .5C, the cumulative error range is already +/- 1.5C. It is really hard to claim a 1C signal detection in that environment.

Given this is an open review process, I’m sure others will find mistakes in this post, let me know and I will investigate and correct as required.

Whatever one believes about how the planet’s temperature is changing, I do hope we can all agree that these guys are NOT the ones we want in charge of dealing with it.
http://blog.heritage.org/2009/12/15/morning-bell-they-cant-even-run-a-conference-let-alone-the-global-economy/

Finally, regarding the suggestion to go and get our own data if we don’t like what they have. That’s just silly. They are the ones who have been charged with collecting it. They have the obligation to do it right, and they are NOT.
http://static.cbslocal.com/station/wbz/wbz/2009/may/SurfaceStations.pdf

Your post proves a point, but perhaps not the one you wanted to make.

The initial mistake I made was to assume GHCN was adjusting raw data, but at least in the US, it has already been adjusted a number of times by NCDC for the USHCN. Each step introducing more possible smaller error while removing gross discontinuities. It has been an eye opening journey to see how many times the data is adjusted by various algorithms. I’m sure I still don’t have it all completely right, but this is what I have seen so far at least for US data. You need to ensure your raw..is in fact raw, not already cooked.

My observation is that the accumulation of these uncertainties appears to exceed the range of detected warming signal that is claimed.

Every time someone adjustments the data they also increase the band of uncertainty. This uncertainty builds upon the uncertainty already in the raw data prior to hand off to GHCN. Many of the adjustments appear to have legitimate reasons of trying to remove large discontinuities or false overall trends. However, every time they modify the data with an algorithm that reduces the volatility, they add some uncertainty even while these other problems are fixed.

In the US, the data is first gathered daily from the station.
The data is collected and reported in 1 degree F increments.

DSI-3200 Page 4:
“The accuracy of the maximum-minimum temperature system (MMTS) is +/- 0.5
degrees C, and the temperature is displayed to the nearest 0.1 degree F. The observer records the values to the nearest whole degree F. A Cooperative Program Manager calibrates the MMTS sensor annually against a specially maintained reference instrument.”
http://www1.ncdc.noaa.gov/pub/data/documentlibrary/tddoc/td3200.pdf

So before any adjustments are made, the data has an error range of +/- 1F or .5C !

Then it is adjusted least 3 times before it is handed to GISS
http://www.ncdc.noaa.gov/oa/climate/research/ushcn/

Time of Observation Bias Adjustments (Adjustment #1) (Error range unknown)
“Next, monthly temperature values were adjusted for the time-of-observation bias (Karl, et al. 1986; Vose et al., 2003).”

“The TOB-adjustment software uses an empirical model to estimate and adjust the monthly temperature values so that they more closely resemble values based on the local midnight summary period.”

Homogeneity Testing and Adjustment Procedures (Adjustment #2) (Error range may be shown by NCDC, see below)
“Following the TOB adjustments, the homogeneity of the TOB-adjusted temperature series is assessed. In previous releases of the U.S. HCN monthly dataset, homogeneity adjustments were performed using the procedure described in Karl and Williams (1987).
Unfortunately, station histories are often incomplete so artificial discontinuities in a data series may occur on dates with no associated record in the metadata archive. Undocumented station changes obviously limit the effectiveness of SHAP. To remedy the problem of incomplete station histories, the version 2 homogenization algorithm addresses both documented and undocumented discontinuities.
Estimation of Missing Values (Adjustment #3) (Error range unknown)
Following the homogenization process, estimates for missing data are calculated using a weighted average of values from highly correlated neighboring values. The weights are determined using a procedure similar to the SHAP routine. This program, called FILNET, uses the results from the TOB and homogenization algorithms to obtain a more accurate estimate of the climatological relationship between stations. The FILNET program also estimates data across intervals in a station record where discontinuities occur in a short time interval, which prevents the reliable estimation of appropriate adjustments.
Urbanization Effects (NCDC says this is covered by their Homogenization algorithms)
In the original HCN, the regression-based approach of Karl et al. (1988) was employed to account for urban heat islands. In contrast, no specific urban correction is applied in HCN version 2 because the change-point detection algorithm effectively accounts for any “local” trend at any individual station. In other words, the impact of urbanization and other changes in land use is likely small in HCN version 2.”

Now after starting out with an observation error range of +/- 1F (.5C). Every one of these prior adjustments adds uncertainty to the data.

For example:

1. Raw Data point is 15C +/- .5C That means the range is 14.5C to 15.5C
2. The first adjustment add +/- .25. Now the range is 14.25C to 15.75C
3. Second adjustment +/- .25 Now the range is 14C to 16C
4. Third adjustment is +/- .505 Now the range is 13.5C to 16.5C!

I chose these numbers for 2,3 & 4 as examples, I do not yet know the real numbers. However I chose their net value because the NCDC gives an example in the document that describes their process. Their chart for Reno shows error bars that look to be around 1.8C to 2C range error introduced by their adjustments. This is additive to the +/- .5 C built in to the raw measurement as there is no indication it includes the raw error range. It could actually be worse as it is unclear if the TOB & Missing data interpolation is part of their uncertainty calculation.
http://www.ncdc.noaa.gov/oa/climate/research/ushcn/

Only after these adjustments are doen GISS get the data for merge into the GHCN.
http://data.giss.nasa.gov/gistemp/sources/gistemp.html

Then, incredibly, they make more adjustments!

http://data.giss.nasa.gov/gistemp/sources/gistemp.html
First they remove the Adjustment #3 data from above. So in a sense they remove one source of error. (Adjustment #4)
The reports were converted from F to C and reformatted; data marked as being filled in using interpolation methods were removed.If they only remove part of Adjustment #3, then more uncertainty is introduced.

This indicates GISS is using the NCDC/USHCN data that has been through Adjustment #3 and they take it mostly back to Adjustment #2.

Then, they homogenize the data again! (Adjustment #5)

“The goal of the homogeneization effort is to avoid any impact (warming
or cooling) of the changing environment that some stations experienced
by changing the long term trend of any non-rural station to match the
long term trend of their rural neighbors, while retaining the short term
monthly and annual variations. If no such neighbors exist, the station is
completely dropped, if the rural records are shorter, part of the
non-rural record is dropped.”

The specific stated goal of this adjustment is to take into account Urban Heat Island effect. Yet, NCDC says they already adjusted for this when they homongenized the data! Now the data is twice baked for UHI. (Which I thought the IPCC and Jones, Wang (1990) said was negligible)

A bit concerning is the video data analysis show UHI is still in the data by sampling city/urban pairs around the country. Hmm… are we smarter than a 6th grader..:-) (I need the URL, but it is well known and easily reproduced independently)

Roman M has done an excellent job in his blog showing these GISS adjustments are driving a bias into the data.
http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/

I have sampled numerous individual stations in CA and seen the same bias being introduced by this this process. I described the process for doing this in comments over at WUWT. Others using that method have found the same results in NY, Grand Canyon, Calgary, etc. http://wattsupwiththat.com/ (I need to find the exact links, sorry)

So there are some building concerns about this final step. First, it appears duplicative to (at least in the US) what has already been done. Second it appears to be reshaping the curves to fit the story. No one has any indication if this is just a bad algorithm or deliberate.

Don’t forget, this also introduces uncertainty. NCDC estimated their homogenization introduced up to +/- .9C if I read their example chart correctly.

So right now it appears the inherent cumulative range of error far exceeds the claimed warming signal that has supposedly detected.

This chart shows GISS claiming a warming signal of .6 C detected.
http://data.giss.nasa.gov/gistemp/2008/

Remember, even the raw data was +/- .5 C from the moment it was written down per the NCDC.

The most incredibly generous reading of the Reno example for NCDC (assuming it includes the raw error rate, TOB, Missing Value Estimates included) shows an error band of at best +/- .9 C.

Then there is whatever uncertainty is added by the GISS process that also appears to bend the curve.

I don’t claim to know all the right numbers for the total uncertainty being introduced into the data. However, this error budget must be fully disclosed and understood. It certainly appears that the signal is less than the noise introduced by the original measurement and the up to 5 adjustments. Exactly by how much is critical to prove the point that a warming signal has in fact been detected.

I don’t know what process the raw European data goes through. Until someone describes the path from observation to the end, it is hard to say. Your post shows early 20th century adjustments of +/- 1C. Given my earlier mistake on GISS “raw” that turned out to already have been modified by NCDC, you may want to investigate if your data may be in the same position. Assuming a similar raw observation error of +/- .5C, the cumulative error range is already +/- 1.5C. It is really hard to claim a 1C signal detection in that environment.

Given this is an open review process, I’m sure others will find mistakes in this post, let me know and I will investigate and correct as required.