the Air Vent

Because the world needs another opinion

CRU Releases (Some) Data

Posted by Jeff Id on December 23, 2009

My next partially completed Siberia post will have to wait.

This is the beginning of the result of climategate. We owe a big debt to those who released the emails, and an even bigger debt to Steve McIntyre for his never ending efforts to force the revelation of some of this information. Now we can begin to understand the premier temperature dataset in climatology. Does this mean it’s done — nope. This is only the beginning, CA has a post describing the release of more metadata (data about data) and actual records for CRU than ever before.

However without seeing the data, the victory is partially hollow. All the criticisms developed over time still hold unfortunately. It appears that CRU hasn’t learned a damned thing from the experience yet either. Why the grumpiness in the face of what can only be described as progress (climate progress :D ), check out these quotes from the CRU page.

The database consists of the “value added” product that has been quality controlled and adjusted to account for identified non-climatic influences. It is the station subset of this value-added product that we have released. Adjustments were only applied to a subset of the stations so in many cases the data provided are the underlying data minus any obviously erroneous values removed by quality control. The Met Office do not hold information as to adjustments that were applied and so cannot advise as to which stations are underlying data only and which contain adjustments.

Another:

Underlying data are held by the National Meterological Services and other data providers and such data have in many cases been released for research purposes under specific licences that govern their usage and distribution.

Remember this pretty quote from the emails.

The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days? – our does ! The UK works on precedents, so the first request will test it. We also have a data protection act, which I will hide behind. Tom Wigley has sent me a worried email when he heard about it – thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that. IPR should be relevant here, but I can see me getting into an argument with someone at UEA who’ll say we must adhere
to it !

I wonder how Real Climate will spin that as out of context.

This is what they say about data storage.

3. Why is there no comprehensive copy of the underlying data?

The data set of temperatures, which are provided as a gridded product back to 1850 was largely compiled in the 1980s when it was technically difficult and expensive to keep multiple copies of the database.

For IT infrastructure of the time this was an exceedingly large database and multiple copies could not be kept at a reasonable cost. There is no question that anything untoward or unacceptable in terms of best practices at the time occurred.

Since before 1990, the entire thing could have fit on a harddrive quite easily. I wonder what happened to the raw data implemented in 1990? I do suspect that the inital problem was partially data storage but what about the years since? And what about the future? And if Dr. Jones doesn’t mind, we’ll be the ones to determine if something untoward happened. I’m certainly not in the mood to have the towardness assigned by these people at this point.

This next point was very dissapointing.

. How can you be sure that the global temperature record is accurate? The methodology is peer reviewed. There are three independent sets of global temperature that all clearly show the rise in global temperatures over the last 150 years.

Peer review in AGW has just been exposed. I’ve not seen one single independent confirmation, sat/sonde data doesn’t agree very closely. Other datasets use the same information, so independent reads as a false claim. The real surprise in this section was the change in focus.

Furthermore, the strong scientific evidence that climate is changing as a result of human influence is also based on the growing evidence that other aspects of the climate system are changing; these include the atmosphere getting moister , global rainfall patterns changing, reductions in snow cover, glacier volume, and Arctic sea ice, increases in sea level and changes in global scale circulation patterns. There are also numerous changes in phenological records which point towards a general warming and support the veracity of the instrumental record.

Just what the f..k does that have to do with the accuracy of a measurement. I don’t swear often here, but come on!! Think about what these jackasses just wrote!! Have they learned nothing?!?!?!! How does sea level rise prove a temperature record within tenths of a degree!!! @@%!^$^@. If a special ed student made the statement you’d give them an F. How dumb do they think we are??? Wow, just wow.

Ok, – five minutes later, I’m calm again.

Why have you not previously shared the HadCRUT data?

We have always provided the gridded HadCRUT product freely and without restriction for research usage. The data set is available through the Met Office, CRU and BADC and has been used widely in research papers and the media.

New question- why didn’t you bother to answer your own question … #$$%!?

. What about the underpinning observations on which the gridbox averages are based ?

The Met Office is not in a position to release that portion of the underpinning land station data for which we have yet to gain permission from the ultimate rights holders.

This rings hollow when you won’t release a list of the stations we need to get rights for? Maybe they did now, but at this point— who knows.

Ok then I skip down through endless – self questions – and find this beauty.

13. How have you dealt with the FOI requests regarding releasing the underpinning global temperature data?

We take our responsibilities under the Freedom of Information Act very seriously and have, in all cases, handled and responded to requests in accordance with its obligations under the legislation.

It’s impossible to respond without adjective.

Ben,

When the FOI requests began here, the FOI person said we had to abide by the requests. It took a couple of half hour sessions – one at a screen, to convince them otherwise showing them what CA was all about. Once they became aware of the types of people we were dealing with, everyone at UEA (in the registry and in the Environmental Sciences school- the head of school and a few others) became very supportive. I’ve got to know the FOI person quite well and the Chief Librarian – who deals with appeals.

The VC is also aware of what is going on – at least for one of the requests, but probably doesn’t know the number we’re dealing with. We are in double figures. One issue is that these requests aren’t that widely known within the School. So
I don’t know who else at UEA may be getting them. CRU is moving up the ladder of requests at UEA though – we’re way behind computing though. We’re away of requests going to others in the UK – MOHC, Reading, DEFRA and Imperial College. So spelling out all the detail to the LLNL management should be the first thing you do. I hope that Dave is being supportive at PCMDI.
>
[snip rest of irrelevent stuff to this topic]
> Cheers
> Phil

What’s to say. Well let’s put it this way: There were a bunch of like minded pseudo-intellectuals, who’ve never made a payroll, never worried about ‘progressing’ outside of a sheltered system, that decided CA readers have a type. And that particular ‘type’ did not deserve protection under the law.

Ok, I’ve left several points alone as a careful reader knows, but the last point may be the best.

25. How do I use the code that you have linked from this web page to calculate gridded statistics from the data files?

The code is a computer programme that is held in two files, using the scripting language “PERL”.

The first, called “station_gridder.perl”, takes the data files and creates a gridded data set. The second, called “make_global_average_ts_ascii.perl”, calculates the time series of global average temperatures from the gridded files that were created by the first program.

You will need PERL installed on your computer. This is available from www.perl.org.

I’ve not reveiwed the code but it seems that we now, after years of McIntyre and others work, can finally run software with replicates the CRU data. Although, I’m not sure how that works without all the actual data?

I’m just an engineer though.

30 Responses to “CRU Releases (Some) Data”

  1. Greg said

    You’re just an engineer? That means you actually know how things work in the real world?

    I think CRU (and the other climate units) could use a few engineers…

  2. boballab said

    Jeff: John Graham-Cumming already found a bug in the program. It showed up when he compared the Graph this program makes to CRUTEM3’s graph. He fix it the program and you can get the fix from his site:

    http://www.jgc.org/blog/

    This leads him to believe this is not CRU code but code whipped up by the Met office to use with this subset.

    As he put it:

    “That’s a bit odd, but not serious. But it makes me suspect something: I’ll bet a mince pie that this code the Met Office has released is not the code they actually use to create CRUTEM3. I bet they wrote it especially for this release.”

  3. Richard said

    boballab said December 23, 2009 at 4:01 am
    Jeff: John Graham-Cumming already found a bug in the program. It showed up when he compared the Graph this program makes to CRUTEM3’s graph. He fix it the program and you can get the fix from his site: http://www.jgc.org/blog/

    That graph on his site doesnt look like the CRU temp graph. Looks more or less flat (trend) till 1980, then a sharp rise after that.

  4. P Gosselin said

    “the data provided are the underlying data minus any obviously erroneous values removed by quality control. The Met Office do not hold information as to adjustments that were applied and so cannot advise as to which stations are underlying data only and which contain adjustments.”

    This means they don’t know what the quality of the data is. Are they saying they are not responsible for ascertaining the quality?

  5. Chuckles said

    Jeff, Curb Your Enthusiasm.

    It is the UK MET Office releasing the data, not the Univ. of East Anglia Cilmate Research Unit. A glance at the URLs in your article above – they are all metoffice.gov.uk…..
    The HAD (Hadley) bit of HADCRU.

    The data may well be sourced from the CRU, but they have been doing the exact opposite – Until very recently their website gave 404 errors or redirects.
    It has reappeared in the last 24 hrs, but the data page there says –
    ‘Sorry, most of this will not be available until the New Year’
    Bit more ‘Hide the Decline’?

    It is also ‘adjusted’ data, and the supplied ‘code’ appears, from the description, to simply read the ‘adjusted’ data and produce a graph of normalised data which agrees with a Met Office graph.
    Hardly surprising, since it is allegedly the code used to produce said graph.
    Albeit, not without error according to JCG.

    Surely a more fruitful line of research/questioning would be something like –


    Your graph shows some data rising at some rate between 0.6-1.0 degrees, and purports to relate to real temperatures in the real world.

    Looking just at the US portion of the source data, it is collected from a data collection network, where 61% of the stations classed as providing data with >5.0 degrees Celsius error due to siting/location problems; where the measuring system used has +-0.5 degrees Celsius accuracy (although measured temps are reported to hundredths of a degree Farenheit); where these measured temperatures are truncated to whole degrees Farenheit, and are then repeatedly smoothed and averaged and adjusted with other measured temps of similar provenance via an unknown algorithm.

    We must now rejoice and believe that the answer is 0.6? Or 2.0, or 1.2.

  6. Jimmy Haigh said

    “The database consists of the “value added” product that has been quality controlled…”

    We all know what that means…

  7. PaulM said

    The “answers” provided by the Metoffice are just amazing.

    The best has to be the one saying that in the 1980s it was difficult to store data so they had to throw it all away.

  8. Roland Lucas said

    Regarding peer review and this:

    “Furthermore, the strong scientific evidence that climate is changing as a result of human influence is also based on the growing evidence that other aspects of the climate system are changing; these include the atmosphere getting moister , global rainfall patterns changing, reductions in snow cover, glacier volume, and Arctic sea ice, increases in sea level and changes in global scale circulation patterns. There are also numerous changes in phenological records which point towards a general warming and support the veracity of the instrumental record.”

    Apart from the justified expletives on relevance, what happened to causality? Large scale arctic ice mass loss occurred ca 1922 and again ca 1817 so presumably today’s carbon dioxide level travelled back in time to (not) cause these similar events.

    They’re still on the glacier trail when from Svalbard to the Himalayas glaciers are advancing near to the politically correct glaciers that know enough to be retreating – mean temperature is nowhere in that.

    The same principle applies with the rest as there is no causality established and so no unambiguous attribution to human activity. IPCC knew this about global climate data for IPCC SAR but Mr Politicisation and Ms Spin make good copy editors.

  9. stan said

    Jeff,

    Just found this at Intrade. http://www.theclimatebet.com/ Kind of interesting. Like to see someone do a bet about eventual changes in the CRU official records.

  10. Greg F said

    PERL? Did they scrap the old code and do a rewrite?

  11. Mike said

    “It’s impossible to respond without adjective.”

    C’mon, I bet you can find some hearty nouns…

    I love this lingo, “value-added product” instead of “data”. The scientific equivalent of gatorade, with a free funnel for easy application.

  12. boballab said

    @3

    I’m going to be kind and believe you didn’t look at the link posted and I’ll just quote exactely what he said and then you can go look at the pretty picture:

    “Here’s the corrected comparison between the output of my program and the output of the Met Office program (with my bug fix to their code).

    Now that’s a pretty picture.

    There are some very small differences that could probably be looked into, but I’m getting a little tired and this is looking really nice now.”

  13. liamascorcaigh said

    I worked in a bank in my younger days, way back in the ’80s. Following a particularly large delivery of cash, I discovered that the vault was too small to accommodate all the banknotes. When I brought this to the attention of the manager he told me to dispose of the surplus.
    “Dispose of it?” I was confused.
    “Yes!”
    “How?”
    “I don’t care. Shred it. Incinerate it. Just destroy it.”
    “Destroy money.” I was aghast. “But it’s our stock in trade. It’s the reason we’re in business in the first place.”
    “Doesn’t matter. Space is at a premium. Space is expensive. Especially storage space. Everything must yield to the iron exigencies of space. Space, as they say, makes the world go round. Every fool knows that, Carruthers!”
    “But, sir, what will we do when the money is required?”
    “We’ll issue meta-money.”
    “Meta-money?”
    Oh, for heaven’s sake, boy. We’ll scribble various amounts on odd scraps of paper and call it value-added tender.”
    But…but, Mr. Wigley- Jones, people won’t accept such a thing. There’ll be the most fearful row.”
    “Nonsense. We’ll bring it down to the House of Lords and pass it before the Duke of Mann and Earl Briffa. That will suffice.”
    “I’m sorry, sir, but I don’t understand.”
    “Peer review, dear fellow. Peer review. As that Keats chap once remarked ‘That is all ye know on earth, and all ye need to know’. Why, I once had a toilet roll peer-reviewed. Bought a race horse with it. Named it after the Duke of Wellington’s famous war steed. ‘Copenhagen’ I called it. Fine beast. Fell ill from exposure in the end. Had to put it down, you know. A messy business. Yes, very messy. Got covered in gore.”

    This really happened. Really.

  14. Mike said

    #13 priceless.

  15. There’s another possible interpretation of the Met Office slimy statement on their disposition of the underlying raw data sets:

    They still have one complete copy of the records, and they won’t copy it now and they won’t release it.

  16. Simone82 said

    Can anyone help me to run the script? I don’t know the perl language: I installed the perl and activeperl but I can’t execute it. :(

  17. KevinM said

    It is obvious that somebody on the team needs to reverse engineer and release raw-ish data with a reasonable process for arriving at the published result. However that person will have no way of rationalizing the act as other than evil.

  18. Chris S said

    The Met Office do not hold information as to adjustments that were applied and so cannot advise as to which stations are underlying data only and which contain adjustments.

    Translation: This data will be of no use to anyone(or their dog).

  19. mpaul said

    At this point, we really need a clear statement from CRU on (1) what they hold but will not produce, (2) what they hold that they will produce, and (3) what they no longer have. Without this, its a bit of a three card monty game. I think the elements needed are (1) the list of stations, (2) the raw station data, (3) the value added data, and (4) all of the adjustment code and methods.

  20. Peter said

    Looking for advice. I downloaded station data for Yarmouth NS Canada, a place I am familiar with from both CRU and Environment Canada. They agree to 1960-61, and then there is a step change of about 0.33 C from then until 2007 when the adjustment disappears. I know this area. Population stable etc., almost to the point of being a time machine. Don’t know how to post a graph, but the trendline from raw data is indistinguishable from 0, while the CRU trend is well, you know.

  21. Bryan H. said

    @Greg

    Science that builds stuff gets less money than science that makes people rich.

    Science that builds stuff hire engineers.

    Science that makes money hire politicians and lawyers.

    Compare budgets of ITER to climate science… The former could truly reduce our CO2 output to near zero.

  22. RC Saumarez said

    I’m interested to see a file for “Guernsey Airport” starting in 1850. As a guerseyman (channel islands, UK) I am aware that Guernsey airport opened in 1928. There is no record beyond 1900!

    I’m beginning to smell a rat in this data.

  23. Olympus Mons said

    Hi Jeff,
    Just for fun (not really into this stuff)I went for file 085350 at 38 43N and 9 09W, that being in the center of Lisbon Portugal, in the middle of the “Jardim Botanico” (botanic garden) built in 1868 and the station as being in there and very well attended and checked since then. Only thing changing around it is that in the last 20 years traffic around it has increase exponentially.

    Looked at the data with Excel STDEV gave me 0.84 from the beginning of the dataset and the end – Wow. This stuff is like they say it is.

    Then I found at the Portuguese meteo institute official numbers for the period 1961-90 for that station and they have 20.8 for high average and 12.8 for low average. If I average both, it gave me 16.8.
    Ok… But If I check average the HADCRU file for data for the same period it gave me 15.9

    Just for fun if I compare this numbers with the rest of Phils dataset it gives me a rise in average of 0,29C — but from Phil values for same period it’s a rise in temps of 1.19!

    Before I try to find data for the period 90-08 (why the hell is it not public) , would like to know if based on HADCRU file for that period one can arrive at the official station values of 16.8 (instead of 15.9 I got from HAdcru average).

    Would really appreciate if anyone could take the time to take a look at it…

    Secondly: those 1981-1984 values at -99 are really strange (I had to infill – 2 with previous, 2 with next year’s data), because I’m sure the Official Portuguese meteo entities has those figures and sent them!

  24. Dev said

    #13

    You owe me a new keyboard and flat panel.

    Last time I ever drink coffee while reading a post…

    Thanks for the belly laugh!

  25. Kate said

    http://www.pbs.org/newshour/rundown/2009/12/excerpt-obama-on-disappointment-in-copenhagen.html

    Please go to PBS here and post a comment regarding Jim Lehrer’s interview with Obama.

  26. Gary P said

    Here is a simple (not necessarily easy) test for the massaged temperature “data” vs. the raw data.
    There has been a worldwide increase in the Urban Heat Island effect and this is known to effect many temperature stations. There is no evidence of any global trend the would have cooled the thermometers. Therefore any upward trend in the global massaged temperatures must be less than any upward trend in the global raw temperature numbers. If the massaged numbers show a greater trend than the raw numbers, then this is proof that the massaged numbers are phony.

  27. [...] CRU Releases (Some) Data My next partially completed Siberia post will have to wait. This is the beginning of the result of climategate. We owe [...] [...]

  28. Doug in Seattle said

    Now that Met Office has provided a teaser, when do you suppose they will be releasing the raw data and code? Just asking.

  29. [...] Over at The Air Vent, Jeff Id discusses some of the details. [...]

  30. [...] CRU Releases (Some) Data [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 142 other followers

%d bloggers like this: