the Air Vent

Because the world needs another opinion

UEA Files Were on a Single Server

Posted by Jeff Id on February 2, 2010

Since these files first came out, there has been a lot of speculation as to the source of them.  Today we learn that, in fact, my Id may have guessed correctly as to the nature of the breach.  David Leigh of the Guardian broke the story using an oddly rare technique these days called investigation – those Brits are so old fashioned.  I would be remiss to not mention that the UK press have represented the only ‘for profit’ media outlets worth a crap on Climategate.

Yesterday David King was making some wild speculations about international terrorists and behind the scenes, I received an email about how stupid he looked for saying so ‘from a British reporter’.  I wrote a whole post last night about it, but it was just a rant by the time I finished, so it went to the trash.  Anyway the whole stupid event he created, prompted the university of East Anglia to release this little tidbit of information.

In fact, as UEA confirmed today, all the files and emails were archived on a single backup server on the Norwich campus. Once access was gained, it would have been simple to copy all the material.

So being the instant expert in profiling that I am, lets look again at the relevant facts.  We have the childish act of taking over the Real Climate server (likely from shared passwords in the files), posting a link at Climate Audit saying then a miracle occurred.  No international organization would behave that way, it’s obviously a giddy kids prank.  Can you imagine how fun that would have been in college, and for those of you who are older, can you imagine how scared you’d be now?  — and Sir King was too slow witted to figure it out.

But there’s more, the people who released the file don’t read a lot of climate papers.  You can tell that because they said this in the comment left at several blogs.

0939154709.txt * Osborn: we usually stop the series in 1960

While the practice is disingenuous and fraudulent, it is absolutely well known.  This point had been discussed here several times, at Climate Audit many times and is in publication over and over.  This does refer to the ‘hide the decline’ incident although this email had nothing interesting except the data, which I plotted within the first couple of days of seeing the email and never posted.  CA again took care of the postings on this later.

The fact that it is NOW well known doesn’t exonerate the hiders of the decline because of the timing of the events.  We shouldn’t forget that those outside the immediate friends of Briffa or the dank depths of an obscure paleoclimatology paper at the time, had no knowledge of it.  The hockey team took advantage of that fact for the IPCC report. And of course the whole practice is IMHO fraudulent.

In fact, one of my first posts on the breaking of the hide the decline story with respect to Phil Climategate Jones saying he didn’t remember what he meant took this exact position.  My point in reply was to basically call him out as a liar and make the argument that EVERYONE KNOWS ABOUT DIVERGENCE OF THIS SERIES!!  In fact, at Real Climate, that was also their first defense of the hide the decline incident, to claim that everyone knew.  Which they do now, but didn’t back then.  –Tricky little slime balls aren’t they, hiders of hiding the decline.

But the point is, this is NOT NEWS, however it is fraudulent and whomever posted the emails was not familiar with this fact.  It wouldn’t have elicited a shrug from a paleo-climate scientist, but if the scientist was on the team, there would be some long winded completely-bullshit explanation about why it’s ok to remove inconvenient data – in this case.

So with all that said, when David King made his comment yesterday, about an international conspiracy which has now forced out this little bit of information, — the files were actually collected together on a backup server by someone else!  They were left in a single place where someone with access – a student say – could grab them all and publish them.

One of the questions the UK anti-terrorism squad asked me was, do you have a UEA password?

In my opinion, this was done by a couple of  students (they said “we” a couple of times in the comment they left at tAV).  They were probably right there on the premises, looking right at the box.  I hope to god the kids are smart enough to stay quiet and not brag to their friends for the next twenty years because, while the cops aren’t interested in prosecuting the real criminals, the kids have a target on their backs.

So Kudo’s to the Guardian and David Leigh for looking into this, you can read the full article here:

David King admits to speculation over source of climate science emails


103 Responses to “UEA Files Were on a Single Server”

  1. windansea said

    yes plausible theory Jeff, but I would also consider the fact that whomever liberated the file was also quite savy about climate websites, posting it not only well know sites like RC and CA, but also on TAV. You weren’t exactly a household word even in climate sites before this happened.

    This should on no way be construed as a slam on you, I loves me some dumb conservative climate change!! (that was for that sod…sod)

  2. Jeff Id said

    #2, You’re right that they knew the blogs. tAV was featured pretty regularly at CA and WUWT prior to the release.

  3. windansea said

    oops, that should be savvy
    🙂

  4. […] UEA Files Were upon a Single Server « a Air Vent […]

  5. […] single server theory 2 02 2010 Jeff Id at the Air vent writes about the recent UEA/CRU announcement that the Climategate files were all left on a single server. […]

  6. Greg F said

    I disagree on the shared password for RC for a number of reasons.

    1. The password would have had to have been an administrative password. I doubt Gavin would create accounts for CRU staff with administrative privilage.

    2. There were numerous hacks that were known for WP. One was a simple exploit that involved passing an empty array on the login. This exploit would have allowed the disabling of all the user accounts.

    3. If you recall, after the RC breach they went offline for a weekend to update the server software.

  7. RomanM said

    What makes you think that the files were password protected?

  8. RomanM said

    Oops, cross post. My comment was referring to the files on UEA not on RC.

  9. Josualdo said

    What I can’t really get over is the appearance that the files and e-mails were carefully read and chosen. In my inexperience, I doubt a hacker would take that time or know so much.

  10. Jeff Id said

    #8, it probably would be standard practice on a server but besides that the anti-terrorism officer specifically asked me and from what I gather, several others if we had a password.

  11. Jeff Id said

    #6 Greg, that’s a good possibility b/c it’s pretty easy and it was very well known among WordPress users.

  12. nc said

    I have a problem with this kids will be kids idea. It has been stated by others there where no personal emails in the release. If it was just “kids” I do not think they would have gone to the trouble of eliminating personal emails. So either these released emails where being set up for an FOI request or collected to be dumped. I believe someone new they where going to be dumped so released them.

  13. Dave L. said

    I doubt that this incident was a prank; it was designed to serve a calculated purpose. I still contend that EAU and Penn State had better perform proper due diligence — if the entire server contents were copied but only a selection of e-mails/files were posted on the Internet, then the possibility of another bomb of hacked e-mails may be in the offing if the incident is white-washed by the investigating committees. Who knows what other goodies may still be out there in cyberspace, waiting for another opportune moment. Could EAU survive a second bombing attack?

  14. Jeff Id said

    #12, I think you’re right about the FOI collection of the emails and potentially a student who knew they were about to be dumped. I don’t believe the people releasing the emails did the sorting themselves, just from the timeframe from the last email to the release. These were pre-sorted by someone else and I believe for the purpose of the IPCC FOI’s David Holland was making.

  15. mpaul said

    It seems odd that they would retain emails for 14 years on a back-up server. I guess if they had no retention policy, this would be possible. But most sys admins will delete stuff after a few years just to make space. If you think about the size of a disk on a departmental server 14 years ago, it seems a bit unlikely that it was large enough to archive 13 years of emails from a whole bunch of users — considering that this group probably attached lots of large files. Just seems odd to me.

  16. RomanM said

    #10 Jeff

    Interesting. I was not asked if I had a password, but he did inquire if I had ever been on the UEA computers. Since I have on numerous occasions downloaded their public data, it was true that I had visited there.

    If I recall correctly, the data files were openly ftp-able on at least several of those occasions in an unprotected fashion. The possibility of this extending to other directories on the same server is distinctly a possibility.

  17. Smokey said

    I posted this on WUWT prior to Sir David recanting his statement that a foreign intelligence service hacked and posted the East Anglia emails [the “nerd” is Sir David’s word]. This is all speculation on my part. But since everyone else is speculating…

    In the Independent’s article referenced by Delingpole, Sir David is quoted:

    “There are are several bodies of people who could do this sort of work. These are national intelligence agencies and it seems to me that it was the work of such a group of people.”

    Translation: “I don’t have the slightest clue about how it was done.”

    Sir David is engaging in rank speculation, intended to take the spotlight off of the criminal wrongdoing committed by CRU scientists and Michael Mann, Gavin Schmidt and others.

    Why would Russia or China — which stood to gain financially from Copenhagen — derail their chances to cash in as developing countries? And when have those countries ever selectively leaked emails on-line like this? It is completely out of character.

    The same unfounded speculation applies to the “What if…” scenario claiming that a nefarious ‘nerd’ in the skeptic community hacked into a CRU computer. Where is there any evidence of that?

    Obviously, if such a nerd had access to CRU emails going back many years, he would have disclosed thousands more. There are clearly a lot of missing/unanswered emails, which were no doubt deleted by the probable insider who posted them on-line, in order to cover the insider’s tracks.

    Law enforcement could probably find the culprit in short order by taking all the emails, including the large number that were not posted on-line, looking at the pattern, and questioning/investigating everyone who had access.

    But neither law enforcement nor the government wants the details to be revealed, and they most certainly do not want charges brought against the dishonest and corrupt scientists who have traded scientific integrity for money and status. If they were indicted, those same scientists would not hesitate to point straight up, in order to save their own skins.

    The AGW scam has already netted the perpetrators tens of $billions, with billions and possibly trillions of dollars more to come, enticing these scientists and their political superiors to promote the increasingly dubious conjecture that CO2, an entirely beneficial and harmless trace gas making up only 0.00038 of the atmosphere, could possibly cause runaway global warming.

    Sir David is simply engaging in classic misdirection based on zero evidence, intended to protect higher-ups, in order to continue the AGW scare for base pecuniary reasons – at the expense of the taxpaying public they are pretending to protect from a harmless molecule necessary to life on Earth.

    That’s my speculation, and I have no reason to change any of it. It still looks to me like an insider leaked the emails.

  18. Steve McIntyre said

    The emails went well beyond any pending FOI requests. I’ve mulled over the possibility that the FOI department felt that Jones had sandbagged them with untrue excuses and wanted to prepare a dossier to see what time bombs might be in the files. It wouldn’t have been an unreasonable precaution for a prudent manager.

    Against that theory is a point that Mosher emphasized – that the time of document creation had been bleached. Iy is also evidence against it being done by an FOI officer since there would be no need for an authorized person to bother bleaching the times.

  19. Jeff Id said

    #18, I read some of the FOI stuff from David Holland, he was basically requesting all IPCC correspondence from certain people. Jones complained in the emails about requests for emials. Maybe there is a clue in them that someone else had requested his IPCC correspondence.

  20. Wayne said

    In engineering work we are required to keep our emails forever to document our work. Miscellaneous emails get archived and deleted from the server annually. Project emails get archived on a server for the life of the project. Only when projects are complete are they taken off the live server and archived. But they are never deleted. And certain files stay available virtually forever. “Attachments” are often not really attached in the email archive but referenced to the project file where they live to keep the email files small.

    Since the CRU “project” was ongoing and people may have been going back to these files from time to time, it would make sense for them to be available on line.

  21. here are the questions I would ask.

    Do you remember the Mole incident on Climate Audit?

    Did you look through all the files on various guys personal pages?

    Do you recall seeing a file on one guys page that had a password in it?

  22. CarlGullans said

    Jeff, OT (I apologize, but I don’t know where else to ask): Is the Antartica paper in it’s final stages at this point?

  23. windansea said

    My favorite fictional crime detective, Harry (Hieronymus) Bosch, always wrote notes about what he knew from evidence, and asked questions to connect the dots.

    1. emails span 10 years…file already existed
    2. found on single server…already aggregated
    3. focus on FOI, not much personal stuff
    4. first site chosen to post RC…password? hack? revenge?
    5. then posted to CA and Tav in comments…knowledge of sceptic climate sites?
    6. sense of humor…a miracle has happened
    7. integrity…climate change is too important.
    8. naming of sample emails..level of knowledge about issues (see 4,5)

    I think another possibility is someone from David Palmer’s staff or actually him, they had intimate knowledge of all the FOI issues and also were “educated” on sceptical sites by Jones..perhaps they started actually reading sceptical blogs and combined with inside knowledge of the games the scientists were playing to avoid FOI, decided to follow their conscience.

  24. Seconding what Mc said.

    the mails contain a bunch of stuff on YAMAL. by this time steve had yamal data. yamal was never covered by FOIA. The files also contain
    stuff about GCMs and SRES. Files about finances. The SOAP files ( something steve had mentioned once in an old post). Also out of office replies
    and stuff that just looks to be collected by an automated process of sorts. Now it MAYBE the FOIAs office attempt to create a
    file using an automated process, you know rather than have jones and others search their mails the FOIA office just had IT write a mail.
    or maybe the IT guy ( who works for JONES and writes papers with him ) tried his hand at writing a program to make it easy to respond to these requests if they had to.

  25. kwik said

    In a company using linux, the developers themselves might doing a lot of stuff which in other companies, this would be regarded as tools, and therefore developed by consultants. Remember that linux people often wants things for free for themselves. When making tools, you often want to do stuff that requires admin privileies. On a larger company using developer studio and microsoft, they might have an IT department fixing all such things centrally, while developers “gets” the tools from e.g. Visual Sourcesafe (or similar) down to their local PC, and running it there.

  26. windansea said

    Steve and Mosh good points

    The person who created the file does not have to be the same person who leaked it.

  27. Windy.

    9. Wanted to cover up the date on which the work was done.
    10. Stopped collecting emails the day McIntyre’s FOIA Appeal was denied.
    11. Published the mails the day before McIntyre would get his Notice of FOIA denial.
    12. Posted the file at 630AM EST with a hint at CA
    13. Posted the file at more blogs almost 10 hours later.
    14. hacker hasn’t taken credit. ( we love credit!)
    15. Russians claim to know the address the file was upload from.
    16. Fingerpointing at Russians from CRU have stopped
    17. An external hack ( other than stealing a password) would leave forensic evidence ( or so Im told by the guy who worked on the
    china hack of google)

  28. RichieP said

    Forgive me if I ask – is there the remotest chance that these mails were actually the ones that they did *not want foi’d and so were sequestered?

  29. Jeff Id said

    #28, I was just about to post the possibility that these were actually the delete file from Phil Jones. He may have been chucking the data himself.

  30. Layman Lurker said

    #28

    Interesting speculation. Anyone who was a party to this with an attack of conscience would have then had an easy opportunity to leak the information.

  31. na jeff, makes no sense given all the fluff in the file, the second to last mail.

    To be sure since we are humans we will find signal where there is none. monster in the closet.

  32. windansea said

    Harry Mosh🙂

    all your note point to an insider

    9, 10, 11 point to someone familiar with FOI status at UEA

    12,13 point to knowledge of CA and other climate blogs (inclusion of yamal etc points to in depth knowledge of CA)

    14 = insider not hacker

    15 not sure what this means, why would they withhold info? are they lying?

    16 points to UEA knows it is insider, or maybe Russians have shared upload address

    17 self explanatory

  33. Kwik

    As best as I can figure IT looks to be handled in part by an individual who also authors with Jones.

  34. Jeff Id said

    Someone just told me that fox news reported 110 different foi’s to fil climategate jones. People were PISSED off. These could have fallen under the FOI’s depending on the wording. Maybe someone should foi the foi’s to find out. Just kidding.

  35. Richie P.

    Whenever people talk about the mails I need to remind them that there are also DOCUMENTS.

    Go ahead and open a few of those document folders.

    Some of them contain data that USED TO BE on the open FTP. Go back to the Mole incident.
    You should find somewhere in the comments me discussing some tree ring records for england, scotland, etc
    these records Were on the open ftp. Then CRU deleted all that data.

    My sense is that all my focus on figuring out the MAILS may be wrong headed WRT the who dunnit.

    The document folders. how were they put together?

  36. windansea said

    There are 2 things to investigate

    who created the file

    who released it

    as I said above, it might not be the same person

    it could be, but I think the manner in which file was released is better way to discovery of leaker. The file and what it contains has many clues but if the creator of this is also the leaker I think they would have been identified by now. Perhaps UEA already knows how file was created, but not who leaked it.

  37. Windy.

    On the Russian deal, need to go back and find that report.

    basically, I suspect this. I suspect the Russians know that the upload came from the UK. That is the upload to
    the russian server came from the UK, maybe they can track down to a service in Norwich. I can’t imagine anyone with brains
    doing it from inside CRU. The russians wont say anything. But they will not be falsly accused. Plus, they are way smarter than
    to post it to a russian server with a comment from a saudi server. The posting to RC had the kind of prankish quality that everyone has noticed
    ( ex pranksters that is)

  38. ZT said

    I don’t believe the ‘single server theory’!

    -14 years email for the CRU is a large amount of email – probably hundreds of gigabytes (terabytes if they were using Notes)
    -Downloading this email would have taken many days
    -Sorting this email would have been an enormous task
    -Email systems change every few years – so keeping an archive of stored email is impractical – purely from a software point of view
    -Probably this ‘single server’ would have had to have been upgraded 4-5 times in this 14 year period – upgrades almost invariably result in the permanent loss of ‘unimportant’ information such as ancient emails
    -The CRU didn’t even keep an archive of all their raw instrument data
    -One can see in the archive that they did not use a common programming language, or even a revision control system, I don’t see them as having the discipline to use a single email system

    I think that an internal email collection effort is far more likely, conducted by someone who was on staff for a long time, who received many of the emails, and had access at various times to email servers.

  39. Bernie said

    #28 & #29
    But why would deleting individual emails create a file given the range of dates, addresses, subjects, etc. If it does then it would obviously stick out like a sore thumb. Wouldn’t this be pretty visible to those looking at how the files were released?
    Seems to me that this stuff was pulled with a “Jones + ‘X’ or ‘Y’…” type of search criteria. Can you imagine somebody, like Jones, going through 10 years of emails? This, however, does not explian the accompanying commentary files. Presumably the investigators would start by determining how this particular cache of emails was created and by whom.
    Obviously there is sufficient added knowledge and conscious effort to suggest some clear intent/malice to create “climategate”. People do do some strange things when they feel their primary values have been compromised.

  40. RuhRoh said

    Hey, I know all of you smart guys love a whodunnit potboiler mystery, but maybe one of youse will go take a look at this
    unique approach to re-calculating the thermometers.

    He seems to talk about much bigger datasets than I’ve heard about from the usual suspects; GB instead of MB.

    Is this just another repeat of existing analyses with a different program, or is it novel?

    http://www.bestinclass.dk/index.php/2010/01/global-warming/

    I posted this on a more appropriate thread but fear it is lost in the flurry of posts about Who’s on First…

    RR

  41. Earle Williams said

    Windansea #36,

    It occurred to me also that the compiler of the email and file archive may not be the leaker. If so, then the leak could have come from an insider or *gasp* an opportunistic operative from a formerly eastern block country. 😉

    Or from John Q Public stumbling onto an unsecured FTP site.

    I think the only way the Id’s theory holds up is if the compiler is a disgruntled academic. You do know why academic politics are so vicious don’t you? It’s because the stakes are so small.

    Along comes a protege student of disgruntled academic. Student gains access to the compiled archive and decides to share it with the world. Hilarity ensues…

  42. tallbloke said

    Mosher #35

    Seems likely to me that at mole time quick action had to be taken so they dumped it all onto the backup server rather than deleting it because they didn’t know what they might miss later. Some sceptical member of support staff who had access to the backup server then made a copy.

    If they were double careful, they might have fired up another box in UEA with a bootable linux distro cd, set a spurious date, mounted the backup server drive NFS across the network and written the files to a usb key. No forensics would find that on anyones drive later.

  43. windansea said

    “FOIA said: November 17, 2009 at 9:57 pm

    We feel that climate science is, in the current situation, too important to be kept under wraps.
    We hereby release a random selection of correspondence, code, and documents.
    Hopefully it will give some insight into the science and the people behind it.”

    heck maybe they are telling the truth🙂

  44. Steve McIntyre said

    #34. Jeff, prior to the July 2009 requests for confidentiality agreements, there had been a negligible number of FOI requests. The summer confidentiality agreement FOI requests did not cause Jones any particular inconvenience. One of the Climategate Letters has Jones saying that he’ll write a “short document” which will suffice for this. They posted up a webpage on the confidentiality agreements and that was the end of that.

    That there were so many FOIs for confidentiality agreements did illustrate to UEA that many people were frustrated with CRU obstruction. It put the issue on people’s radar in a way that had never happened before.

    Since Climategate, there have been a lot of FOI requests for obvious reasons.

    But up to July 2009, there were hardly any.

  45. Layman Lurker said

    #35 and 39

    I think the speculation that this is a Jones “catch all” sequestration of data, documents, emails which he did not want exposed is intruiging (notwithstanding the IT logitics involved). It is consistent with his stated sensitivity to such info becomming public. Others could have known about it or stumbled upon it. The Seems both simple and plausible.

    Sequestration (over time), rather than deletion, would be the logical way to handle sensitive information in these circumstances. You don’t want to delete unless you are cornered. If you do get cornered then you have already compiled all of the sensitive material and it can be handled quickly.

  46. Earle Williams said

    Ruhroh #40

    I notice that Best In Class is working with data that includes daily daily temperature. In that case thare would be 30 times the number of records compared to monthly averages, which it seems most people work with.

  47. Jeff Id said

    Steve,

    I was just thinking where we would be without your work. I mean, I doubt anyone would have bothered to look so deeply into the paleoclimate science, and it’s almost certain that nobody would have gone after the emails and data like this. A hundred and ten FOI’s just shows the power of blogs and the public’s impression of them.

    Truly amazing – thanks!

  48. Jeff Id said

    #40, I saw your comment on the other thread and took a quick glance. I’m working now but my impression was that the blogger has a lot to learn about what he’s doing. The hockey stick was kind of a funny addition to a temperature analysis. The issues are separate. It’s good to see them looking at the data though.

  49. clivere said

    Bishop Hill posted this snippet which for me is quite plausible and entertaining

    http://bishophill.squarespace.com/blog/2010/1/25/the-other-snippet.html

    A long forgotten email departmental archive progressively building up over time! The archive created by some kind of routine that selected emails based on the id of researchers or some other keywords.

    Mr/Ms FOIA then comes along and spots it, manages to read the content and the rest is history.

    One other observation is that F will have read the majority of the content provided. We know that because they provided a list of the content which would only be achieved by substantial reading which I would expect took place between 13th and 17th November.

  50. eems to me that this stuff was pulled with a “Jones + ‘X’ or ‘Y’…” type of search criteria. Can you imagine somebody, like Jones, going through 10 years of emails? This, however, does not explian the accompanying commentary files.

    Yes:

    basically if you have access to the entire repository of emails you merely do this:

    If ( sender == jones, or briffa, or osborn or wigly or santer)
    or if ( reciever == jones or briffa or blah blah blah)
    or if ( text body contains Yamal or SRES or bristlecone or blah blah )

    then copy.

    that will get you a file that looks interesting to CA readers. It will also contain some fluff.

    Thats my theory and I’m sticking to it, despite any facts to the contrary.

  51. Erik said

    ..But Paul Hudson BBC said he was forwarded the very same emails at the 12th of October:
    —————————————————–
    Paul Hudson | 13:07 UK time, Monday, 23 November 2009

    I was forwarded the chain of e-mails on the 12th October, which are comments from some of the worlds leading climate scientists written as a direct result of my article ‘whatever happened to global warming’. The e-mails released on the internet as a result of CRU being hacked into are identical to the ones I was forwarded and read at the time and so, as far as l can see, they are authentic.
    —————————————————–
    BBC weatherman ‘ignored’ leaked climate row emails:
    http://www.dailymail.co.uk/news/article-1231763/BBC-weatherman-ignored-leaked-climate-row-emails.html

    @Steve McIntyre:
    Thank you so much for fighting for the truth!!!

  52. Here is an appearance of FOIA that not many know about.

    http://www.climate-skeptic.com/2009/11/bummer-i-didnt-make-the-list.html#comment-5931

    How many posts did FOIA do altogether?

  53. Chris S said

    It all points to Phil Jones being the creater of the file, as a backup of stuff he then went on to delete.

    It contains e-mails that would be embarrassing if released as part of an FOI request, but also files that a FOI would not pick up.

    It was created on a Unix system.

    If the originals no longer exist on the CRU server, this may well be why those investigating were able to conclude so decisively that UEA had broken the law. If the files had not been deleted, it would be hard to conclude anything but intent from the e-mails.

    Hopefully it will soon become clear. There must be many outside the “circle of trust” who now now the background to the leak;)

  54. Robert E said

    I disagree with your analysis. There are logs for ip-traffic and file accesses. If it would have been some student at the university they would have been caught by now. Most likely it was done by some hackers that were fed up with lies that knew how to hide the identity. I think the email-boxes and personal file areas was on the same server. They hacked in through some proxy servers or some remotely controlled compromised computer to hide their identity. Then they downloaded the mailboxes from some of the interesting researchers and files from personal fileareas. These mailboxes usually have mail-quotas so that they don’t become to large in size. When their nearing their limit users receive notices to erase mails. Files were zipped beforehand on the server they where on. Therefore it need not take long time to download.

    They also took the time to filter out all personal letters like “On the way home by some milk” and “You were so sexy in…” cause they don’t want to cause unnecessary harm. Do you think some student prankster would have done that. From what I’ve heard they also took the time to redirect pages on RealClimate. Only hackers would do that.

    Then they placed the information on remote servers like in Russia not to reveal their true location.

    Anyway that’s my take on it. But I’m just guessing like you do.

  55. From an IT pro perspective:

    A “back-up” server isn’t necessarily a reserve server (old,limited capacity), more likely it is literally the server responsible for performing tape backups. Therefore, it isn’t at all surprising that (a) this server could recall a data archive from 13 yrs ago (they’re likely storing each year’s backup, each month of the current year, and each day of the current week), and (b) would have access to the archive as the backup account would have root/admin privs on all servers.

    B. Not much should be made of the file creation dates. This is a result of copying files from one file system to another (such as a thumb drive).

    My expectation is that the network admin or the low paid help is the culprit here. Possibly overheard something by the water cooler, and just copied off the directory that the FOI officer was gathering his notes into.

  56. Lucy, ya we know about that one.

    The posts I know about:

    (excluding RC and CA)

    WUWT ( earliest) embargoed
    Airvent
    Climateskeptic
    RomanM

  57. Atomic Hairdryer said

    Wish the CRU and/or police would stop teasing and publish their report. The archive server may also be a red herring. If it was a proper archive server, then the files would likely be compressed in whatever format the archival software used. Whoever grabbed it would need the files, indexes and software to extract and assemble the zip release. If it was a simple file server used for archival then finding the data would be easier.

    The date thing I’m not sure about. Some files look modified, some don’t. Changing the dates would be pretty simple, so touch -t 200901010000 *.* although some ended up as 1899. The mail folders had more processing given the message headers aren’t there.

    Also suspect the hacker wasn’t an oldtimer. When I saw the message, I thought of ‘something wonderful has happened’ which Amiga owners may remember.

  58. Phillip Bratby said

    My money is on someone who had always had doubts about the hockey stick and blew hot and cold (pun not intended) over the reconstructions. He was never totally trusted and came in for a lot of criticism, being the ever-present side-kick. His illness and problems with Yamal gave him lots of time to think about his work and he decided he couldn’t take any more of the defending the indefensible and the hassle on his return to work. Ably assisted by his younger and more sassy “loose cannon” colleague, he was able to assemble the FOIA file and release it to the world.

  59. Kay said

    Didn’t the leaker mention that there’s more to come? Also, wasn’t CRU hacked or leaked last summer?

    Where I work, nothing is ever deleted. It may no longer be on your machine, but it’s sitting on a server, backed up by magnetic tape. If you need something that’s been deleted by you, it takes them less than 15 minutes to retrieve and restore it.

    In the emails, starting around August ’09, the hockey team started to get suspicious–you could see it in the way they wrote their mails. I think they knew one of their own was going to betray them.

    Whoever did this took an awful lot of time collecting this stuff. It may not be sequential, but that really doesn’t matter.

    It has to be someone in CRU or someone who authored with Jones and especially Mann. I think they know who it is. For a while, I couldn’t figure out why that person would post it at RC, but maybe that was just a big Eff You to Schmidt, Mann, Jones, and Hansen.(My money is on Keith Briffa, personally. He’s been under the radar; we haven’t heard from him at all throughout the whole incident. Where is he? And his anger and resentment of Mann jumped off the pages at the end. I got the impression they really didn’t like each other at all.)

  60. Charlie A said

    #34 Jeff Idm ‘ …. reported 110 different foi’s to fil climategate jones. People were PISSED off. ”

    I think that a large percentage of those 110 (other sources say 105) FOI requests were related to Steve McIntyre’s attempt to get the HADCRUT3 data. I submitted 3 FOI requests myself back in July 2009. My 3 FOIAs were simply requesting station data for 5 stations, requesting any confidentiality agreements that apply to those 5 stations/countries, and requesting copies of any policies or guidelines issued by UEA regarding release of confidential information. I suspect that the total number of requests submitted by readers of Climate Audit back in July/August were at least 75 out of the total. So 105 or 110 total is a bit misleading, since a lot were really the same request, with each request asking for the confidentiality agreements for 5 different countries.

  61. Kay said

    #58

    Oops, cross post. But I agree with every word you said.

  62. singularian said

    I agree with Windansea – “random selection” are the key words.

    I also think that, the emails anyway, are more likely to be information that wasn’t to be included in a FOI release.

  63. mpaul said

    #55

    I think the concept that this was on tape somewhere raises an interesting possibility. I doubt that a place like CRU would have a tape robot — too expensive. More likely — tapes are dismounted periodically and put in a file drawer. The physical filing cabinets might not be in a secure location. So anyone with access to the facility might have been able to grab a tape. Then all you need is access to a machine that can read that particular tape format — which you can probably find in any student data center. So maybe this server is not a CRU server, but is somewhere else and was used to pull the info from tape, and sort and package it. This would not require foreign spys or extraterrestrial intelligence to do something like this. An IT guy who wanted to cover his tracks would take the tape off-site.

    Pure speculation.

  64. RE 40.

    Its novel in its approach to getting the data and novel in its processing of the data. It is not however very illuminating.
    BUT, everybody has to start somewhere.

    The problem I see here is the kind of problem you see when people appraoch a large database with a meat grinder.

    You think.. hey here is all the data , let me grind it and see what comes out.

    usually junk.

    This is a task that I would probably approach 1 miserable station at a time.

    Get the data into a decent database and stop effing around

    Pull up each file. Look at its metadata. Check the metadata.
    Look at the temperature curve.
    next fricking station.

    That will give you some sense of all the problems your algorithm is going to face.

    .

  65. #12: So either these released emails where being set up for an FOI request or collected to be dumped. I believe someone new they where going to be dumped so released them.

    What if these “carefully selected” documents were picked by the “source” – someone trying to delete the incriminating stuff (like Jones himself) – and then by-golly deleted! And what if that pile of files was sitting on the server in the “delete” bucket (like my Norton trashcan) ans someone else (our hero) found them ALL in the bucket, grabbed them, zipped them up, and …well, you know the rest.

    That would explain the apparent careful selection by someone who might NOT be a follower of the subject to the degree the selection implies.

  66. jmotivator said

    I’m starting to think that the “perpetrator” of this wonderful “crime” is someone we all know, but who is not a “skeptic”.

    Earlier I thought that it was some IT guy tasked with compiling the FOIA data call… but maybe not. Though it does explain access and knowledge.

    As a data security person by trade I would be SHOCKED if UEA-CRUs data archive servers were forward facing appliances accessible from the internet. Seriously shocked. Such archives as a hard rule are complicated to access back end devices with limited back-end restore locations.

    Archives would be the realm of IT, and in most cases unknown to professors beyond some superficial level that states “if I want it back I call IT and TA DA it reappears”.

    But now I am drifting more towards an insider at CRU. There is a list of scientists in the CRU emails that are good candidates for having no love lost with Jones, Mann and others, but were privvy to the goings on enough to know where to dig.

  67. P Gosselin said

    I posted this about 9 hurs ago, and it comments poignantly on this.
    http://www.independent.co.uk/opinion/commentators/dominic-lawson/dominic-lawson-so-all-these-climate-revelations-were-a-dastardly-foreign-plot-1886149.html

    Forgive me if someone has already linked to this.

  68. Charlie A said

    #66 “There is a list of scientists in the CRU emails that are good candidates for having no love lost with Jones, Mann and others, but were privvy to the goings on enough to know where to dig.”

    It doesn’t have to be a scientist mentioned in the e-mails. There are probably a lot of grad students at UEA/CRU. It doesn’t stretch the imagination very much to hypothesize that one of the grad students didn’t like some sort of deceit or unprofessional behavior that the saw, and then after looking into things a bit deeper was truly shocked.

  69. windansea said

    there was a pretty extensive review of the files from an IT guy at WUWT ahile back. He concludes

    I suggest that it isn’t feasible for the emails in their tightly ordered format to have been kept at the departmental level or on the workstations of the parties. I suggest that the contents of
    ./documents didn’t originate from a single monolithic share, but from a compendium of various sources.

    For the hacker to have collected all of this information s/he would have required extraordinary capabilities. The hacker would have to crack an Administrative file server to get to the emails and crack numerous workstations, desktops, and servers to get the documents. The hacker would have to map the complete UEA network to find out who was at what station and what services that station offered. S/he would have had to develop or implement exploits for each machine and operating system without knowing beforehand whether there was anything good on the machine worth collecting.

    The only reasonable explanation for the archive being in this state is that the FOI Officer at the University was practising due diligence. The UEA was collecting data that couldn’t be sheltered and they created FOIA2009.zip.

    It is most likely that the FOI Officer at the University put it on an anonymous ftp server or that it resided on a shared folder that many people had access to and some curious individual looked at it.

    If as some say, this was a targeted crack, then the cracker would have had to have back-doors and access to every machine at UEA and not just the CRU. It simply isn’t reasonable for the FOI Officer to have kept the collection on a CRU system where CRU people had access, but rather used a UEA system.

    Occam’s razor concludes that “the simplest explanation or strategy tends to be the best one”. The simplest explanation in this case is that someone at UEA found it and released it to the wild and the release of FOIA2009.zip wasn’t because of some hacker, but because of a leak from UEA by a person with scruples.

    http://wattsupwiththat.com/2009/12/07/comprhensive-network-analysis-shows-climategate-likely-to-be-a-leak/

  70. […] in US growing fast, Climategate – single server please, small dead […]

  71. ianl8888 said

    I have posted my view of this on CA some months back:

    1) the email compendium was collected by CRU Administrative design to understand the background and depth to the FOI-denied issue, with an FOI decision pending

    2) the public internet dump of FOI2009.zip was done by someone else entirely, someone who is very, very IT-savvy, since it was done with maximum knowledge of how to avoid being caught. I’ve never met a scientist in the age cohort involved here knows how to do that (I’ve met quite a number of very clever scientists within that cohort, and like me they are clueless in this regard – BTW, I’m not implying that I’m clever, just absolutely clueless about how to dump material on the Internet without a trace, as are all my age-cohort peers)

    In short, the “collector” and the “dumper” are different people with different motivations. I think the timing of the dump was aimed at Copenhagen, while the internal timing of the compendium was determined by the FOI issue. I cannot usefully speculate on how the “dumper” became aware of the existence of FOI2009.zip

  72. Jeff Id said

    #68, It can’t be a scientist for the reason specified in the post. The osborne issue is a non-event, and a scientist would know about it.
    ————-

    The UEA said the emails were all collected in a single file which could be easily downloaded. The collection was therefore probably done for a reason other than the final release. It could have been WRT the 110 FOI’s. When I was in school the science computers were unmonitored completely. I’m pretty sure the release person was different from the collecting person.

  73. Layman Lurker said

    From Osborne’s email:

    “Keith has asked me to send you a timeseries for the IPCC multi-proxy
    reconstruction figure, to replace the one you currently have. The data are
    attached to this e-mail. They go from 1402 to 1995, although we usually
    stop the series in 1960 because of the recent non-temperature signal that
    is superimposed on the tree-ring data that we use.”

    Jeff, the leaker may have been attempting to draw public attention (rather than scientist or knowledgable skeptic) to the IPCC multi proxy “hide the decline” issue.

  74. Carrick said

    RomanM:

    If I recall correctly, the data files were openly ftp-able on at least several of those occasions in an unprotected fashion. The possibility of this extending to other directories on the same server is distinctly a possibility.

    That’s my thoughts too, has been from the beginning.

    Never invoke genius when stupidity is as good an answer.

  75. MikeN said

    Jeff, I’m not sure that the leaker had knowledge of climate sites.
    The total list appears to be TaV, RomanM, ClimateAudit, RealClimate, WUWT, Climate-skeptic, and Warren Meyer.

    At the time, Nov 19, Climate-skeptic had a post by Warren Meyer on the first page. It may have been just clicking through links from WUWT and ClimateAudit to decide the best targets.

  76. Jeff Id said

    #75, it’s a reasonable point. CA does have a short list of climate sites. I would have expected the blackboard to make the list of a regular reader. My grumpy posts may have attracted the attention of a certain type of personality. Roman’s blog is really rare considering he won’t do more than a couple of posts per month. Prod prod.

    I wonder..

  77. Steve Fitzpatrick said

    Hey Jerff,

    Could some enterprising blogger not look over comments made in the pre-climategate period for web addresses in the UK physically near or even at the UEA? Maybe the people who posted the emails had already made comments at The Air Vent.

  78. Ausie Dan said

    The detailed email addresses have been deleted from all the emails.
    That was obviously done to spare the Team from being bombarded with possibly rude or angry emails.
    Was that done before release or before publication on the web?
    Who would want to protect the guilty?
    An insider or a hacker?

  79. Eric said

    Just about everyone here knows the science and the IT better than I do, but I know something about people and behavior, and #58 strikes me as a very sound reconstruction.

  80. rephelan said

    71.ianl8888

    Ian, don’t diss the old guys. Right this minute I couldn’t do what FOIA did, but if I had to I could do it in 24 hours. It’s not rocket science and doesn’t require the sophisticated spy agencies Sir David King, vaunted science advisor to Labour Prime Ministers, was drooling about.

    Couple things to keep in mind:

    1. topicality – despite Steve Mosher’s assertions about fluff (and keep in mind that Steve Mosher literally wrote the book on Climategate) the e-mail selection was pretty much confined to topics of interest in the “denialist” sphere… what some of us may now be considering “fluff” may turn out to have significance… Steve Mc’s postings certainly show that. Ask anyone what is “important” and each will come up with differant answers…. simply point of view. The thrust of the leaked e-mails demonstrates knowledge and understanding of the salient issues. To FOIA’s mind, certain e-mails may have seemed relevant, even if the relevance eludes the rest of us.

    2. Timing – The e-mails end on the 12th – culling the relevant ones would take time – far more time than the five days or so between the final e-mail and the release. Keep in mind that most e-mail clients store the e-mails in files that are not easily read by anything other than the e-mail client…

    3. There is also the issue of the documents file… they most certainly were not stored along with the e-mails. Again, they provide insight not just into the technical aspects (like Harry-Read-Me) but political thinking as well.

    My guess?
    1. Not a hacker; inside job.
    2. Not an FOI accumulation.
    3. Graduate Student or junior faculty
    4. UEA/CRU network security is ****!

  81. Greg F said

    Here are some thoughts related to the network and computers. Having seen the same unlikely scenarios repeated on different blogs I thought a overview might be useful.

    All mail going to UEA’s goes through one of two gateway email servers. From there it is distributed to department mail servers including the CRU’s mail server. From an administrative point of view the only sensible way to archive the email would be at the gateway. There were some headers in replies back to the CRU. From the headers we know that the mail going to the CRU went through a UEA gateway mail server (as I am sure all the university departments email does). Also, MX records only exist for uea.ac.uk. There is no MX record for the CRU as it is just a sub domain at the university.

    The emails used the Unix Epoch time stamp for their file names. If you run the file name through a Epoch time calculator it will be exactly the same time the body indicates the email was sent or received. This strongly indicates that the emails were from an archive. The gateway mail system at the UEA splits all incoming and outgoing mail into 2 files, header and body. It is capable of saving a copy of just the body in text format which explains the lack of headers in the emails. There are a number of reasons they were stored in plain text without the headers. They would be easy to search and consume very little disk space. Even less without the headers. They would also be more immune to a file format becoming obsolete (a common problem with old databases). The little over 1000 emails in the FOIA file only take about 8MB. Based on this, 1 GB of storage would hold over 100 thousand emails. A modest amount of storage for the mid 90’s when email volume would have been much lower. It is also rather trivial to move the archive to larger disks as the archive grows.

    Random thoughts here. The files being on the same server doesn’t mean they were on the same logical drive on that server. There is also no reason to believe the emails were archived on the gateway email server. For security reasons it would make more sense to archive them on a network drive on a file server. On a server it is routine to keep the OS and applications separated from the data on a different logical drives (unlike your typical desktop). This keeps the data independent of server software updates/upgrades. From a IT maintenance point of view this is a no brainer.

    The web facing appliances (web, ftp, mail servers) are without a doubt in a DMZ. A short description of a DMZ is a server with one firewall between it and the Internet and another firewall between the server and the internal network. So the servers are sandwiched between 2 firewalls. The server(s) have 2 network connections connecting to the 2 firewalls. One network adapter would have a internal IP address and the other would have a Internet address.

    The CRU’s computer competence is not a reflection of the IT department at the university. It is my experience that the least competent always think they should have the most access. Since the IT department is subservient to the departments it would make sense that security in the CRU sub domain is less than optimal. The best you can do when in a situation like this is to wall off the ‘free for all’ from the rest of the network to confine any potential disasters. The DMZ is one place where IT can enforce security since it is transparent to the user. This is why I think suggestions that it was hacked through the poorly secured FTP server are unlikely.

    I think there are a number of scenarios that are plausible. Here is mine. I think it is obvious the files were collected by someone intimately familiar with the science. The person that did the bleaching and dissemination would likely be a different person (different skill sets). I work at a 501-C that is very much into the green religion. You tend to be careful who you share your opinions with lest you be labeled a heretic. I can see a situation where a CRU and IT staffer become friends over the years. The CRU staffer feels comfortable venting his frustration to his IT friend. Opportunity knocks and they decide to answer the door.

  82. Eric said

    Last three sentences of #81 make much sense. Thanks.

  83. RomanM said

    #76 Jeff

    Clearly the initial “miracle” comment on CA and choice of file name was a result of the email releaser being aware CA had undertaken the earlier FOI requests. That WUWT was involved could be because of the ongoing relationship between them and CA.

    Your blog had been very active and you were a common presence on CA. The only proximate reason I could think of for posting on mine was that I had done a CA post on “RCS, One Size Fits All” a month earlier and I had also continued to be active in the threads relating to the Briffa recons.

    I know of no specific reasons why the others would have been involved.

    By the way, I noticed a tiny difference in wording between the comment on Climate Skeptic ( “This is a limited time, very unofficial offer”.) and the comment on mine (“This is a limited time offer, download now:”). It wasn’t just a cut-and-paste effort.

  84. claw said

    # 78 As far as I remember the email addresses were complete in the original FOIA text. During dissemination the addresses were removed for legal reasons. I also remember all sorts of warnings of possible malware since this skeptical community was skeptical of this gift. Pardon this mixing of metaphors but this was a case of Don’t look a gift horse in the mouth except of course if it’s a trojan horse … wait for it … of course. Sorry couldn’t help myself

  85. Jeff Id said

    #83, that’s a very interesting observation about the wording. I think our insiders were frustrated b/c there was a second comment at WUWT which asked if anyone was paying attention.

    There is another point too. The bloggers on which the posts occurred were those who were more interested in technical details. Someone knew you would figure it out. That’s why Lucia’s would have been a perfect choice. She’s got a much more reasonable tone than I which might not appeal to the college maverick.

    I’m really convinced that the Osbourne comment proves that they don’t have a close grasp of the papers, yet understand them pretty damned well in general. If I were to make a list of interesting comments this wouldn’t have made the cut – maybe it would say – Briffa data or something.

  86. Joanie Berkwitz said

    The point about the ‘read-me’ and data files being included seems very important… it wasn’t just emails being captured or preserved. Would the Harry-read-me be included in an FOIA request? In what scenario would the data files be compiled along with the emails? And no, the emails weren’t random selections among *all* emails, but possibly among *already sorted* emails, which would imply that there are definitely more hanging over CRU’s head.
    Look at what is included, and what is not included, and that will bring us closer to an answer.

    (Enjoying my hard copy of Climategate, though it is heavy reading, it really goes into intense detail! Well worth it!!!)

  87. My crazy theory that Jones accidentally created FOIA.zip using a faulty eraser program is looking better all the time.

    http://appliedimpossibilies.blogspot.com/2009/12/own-goal-at-cru-hockey-team.html

  88. Layman Lurker said

    #83 RomanM

    I don’t think the file was compiled for FOI. There was too much stuff that was sensitive but not FOI related (like Yamal).

    Sensitive documents were being collected by someone at CRU. To me, Jones fits best as the collector but I’m also open to Greg F’s scenario in #81. The reason I lean toward the Jones scenario is that he was the director and as such had the authority for access. As director he also had the broadest familiarity of the sensitive issues.

    I agree with RomanM, I think that the leaker was a regular reader of the technical, skeptic blogs. The fact that he picked Roman’s blog to post shows that he read enough to recognize the value of Roman’s contributions partucularly at CA and tAV. Also – the casual reader would not likely have known that Roman even had a blog.

    I don’t have a hard time believing that a CRU insider would have regularly kept up with these blogs. Some of these guys likely gained more critical insights into work of colleagues then they ever would have had just reading the journal articles.

  89. E.M.Smith said

    Steven Mosher said I suspect the Russians know that the upload came from the UK. That is the upload to the russian server came from the UK, maybe they can track down to a service in Norwich. I can’t imagine anyone with brains doing it from inside CRU.

    Well, were I doing something like this, I would be very tempted (in the process of covering tracks and leaving false trails) to do the “posting” from an inside machine at CRU. Now I’d be logged into that machine from somewhere else and i’d try to erase that login from the syslogs and I’d try to make the ‘source’ of that login something with no forensics, like a router without disk, via a maintenance login, or a nice BBC (bootable biz card, not the radio station😉 in a random box.

    Knoppix is nice – I handed out one to each student at a forensics class I taught once upon a time😉 that way WHEN the forensics guys start the track back the first fickle finger of fate points back at CRU itself then the trail just evaporates… and if at all possible, I’d be remote and working to my first inside box through a VPN with the contents encrypted so you don’t get a record of the traffic in any network equipment. Yeah, you get “tunnel formed’ from box to box. Then nothing. But it depends on what tools you have to work with.

    In the end, you get the trail to CRU, a bounce to another box (or two…) with some amount of the audit trail deleted (if you can) and then a box with a tunnel to some place with anon ownership (like a library box via bbc or a random box at the college (again w/bbc). At any point where you can fudge the logs, the trail ends. And if it gets all the way back to the coffee house “hot spot” they still know nothing. (There is a nice coffee house near me where I can pick up wireless from the parking lot ;-0 so good luck tracking folks down these days… “War Driving” can be fun😎

    Heck, in all probability there are wireless entry points leaking into the campus parking / park benches all over CRU. Folks are very sloppy about that…

    Since folks are speculating:

    My take on the structure of the file is that it was compiled by someone not the leaker (and probably the FOIA department).

    But given the “backup server” statement, consider this:

    Rarely does the FOIA guy do his own leg work. He asks and IT guy to assemble a (typically over broad) subset to sift through. The IT guy often does this on whatever random “server” he has a priv login for (so it stays under a priv level of control, and not in ‘user space’…)

    So the IT guy casts an even wider net. Some semi automated searches (as described above w/pseudo code) for emails. Some directed extracts from known places. This makes the broad sweep that gets sent to FOIA Guy, who filers more to make the FOIA basket.

    But that ‘first IT sweep’ is still sitting on a ‘backup’ server…

    Now for ‘leaker’ (or hacker) it’s just a matter of finding that one file. I could see any of several folks with priv access and a chip on the shoulder. I could see a server being repurposed (i.e. “backup server” gets pressed into FTP server). That they asked about folks with “logins” or “passwords” implies they know the machine was accessed via login that worked, not crowbar hack. But it’s ‘public facing’ or ‘in the DMZ’ or they would not be asking about folks with NO inside access having logins.

    Add in that they took down their web server after the leak, and I’d even go so far as to implicate it as the ‘backup server’ that was being used to collect the broad raw material for the FOIA request.

    Oh, and the datestamp bleaching is common practice. Whoever did this has some skills. They knew to bleach so no time stamp is given from their copy time. They probably knew how to cover their tracks and leave a misdirection fake trail to a CRU location too. And they knew about open relay servers to park the goods. So I’d point first and most at a PO’d grad student with access, or a sloppy IT guy who left something too open (and it got picked up by an outsider who got inside enough to exploit the ‘too open’ server.)

    And at that point I think the trail runs cold. They have a login / access record coming in from outside. They can’t finger where outside (or they would not be asking randoms about logins) and they don’t know who. That login hit the “backup” server and found the goods. Said goods collected by an insider, probably for FOIA demands. But it is unlikely the FOIA guy would have put stuff there… but highly likely an IT guy would have while making the bucket to hand over to the FOIA guy for preening and publishing.

    (I’ve had requests like that before where Mr. Security says “Give me everything to do with Jones” and I had to make a bundle to give him… and you look around for a box with spare disk, and private access. Often you can’t just put it in your home directory for various security and / or size issues. And no, I didn’t delete it after delivery. Just in case he came back for more or a new copy or ‘the same thing plus last week’… )

  90. Harry the Hacker said

    Re emails and no headers.

    Its very apparent this was a unix system.

    A little known fact that on unix systems it is VERY easy to make a plain-text file which is a transcript of all emails that are sent using the sendmail program.

    The plain-text file could then be filtered fairly easily.

    Such plain-text mail logs are routine for data-archival (aka butt covering) in companies, and often the people using the systems don’t even have any idea the log is there.

    As for server upgrades. Well, yeah. So? Upgrades that use the same OS, or a similar one, or a newer version are done all the time. To minimise the disruption, its fairly straightforward to keep things looking and working the same.

    And systems with many, many GB of disk were very common 14 years ago. Programs were smaller then, and in unix mail there was little/no ability to easily send the huge attachments so beloved of emailers of today (the famous 14 MB smiley face attachments of today were not even possible 14 years ago).

    Summary: fitting a huge email log on the hardware of 14 years ago was routine, and easy to do, and did not really use a huge amount of space.

  91. ZT said

    It is just my personal view – but I don’t think that ‘sendmail’ capturing since 1994 will be the explanation. If there were ~100 people getting ~100MB of email per year for ~15 years, that would be hundreds of gigabytes of email to either sift through, or download from the CRU. Every year they would need to store many gigabytes of old email – in a university environment – I just don’t think that will be happening.

    Back in 1994 disk space was 1,000 times more expensive than it is today, and linux had only just been invented.

    The CRU did not use a revision control system, or even keep their raw instrument data. So, I don’t see them exerting themselves to keep their email indefinitely.

    Despite the importance of what they were doing…

  92. Greg F said

    ZT

    1. Email was plain text and HTML email didn’t emerge until the late 90’s. A typical email back then was well under 10KB and nobody was getting a 100MB per year in email.

    3. The university didn’t use Linux for their servers. They use Unix which was born in 1970.

    4. Archiving of the email isn’t CRU’s responsibility, it is the universities. No exertion from the CRU required.

  93. E.M.Smith said

    Per the email space issue: It’s not an issue.

    Even in 1990 you could get GB scale storage fairly easy. (I bought some early GB tape drives (vhs and 8mm) and a TB scale tape robot about 1990 or a bit earlier.) The bigger issues by far were folks packratting things like the net news archives from forever along with every image on the internet (or elsewhere) they could find.

    The size of email is trivial in comparison to things like “everyone having a personal copy of the Unix sources” or “the rec.myhobby.forever” archive or even the “source code for my project and each revision along with the canonical set of binaries”… And folks would show up with those canonical sets from every project they ever worked on. (Imagine a dozen folks all working on various early GNU or open source efforts, each bringing their collected archives with them…)

    It wasn’t until MS starting having folks email bloated apps attachments to each other that the email sizes and archives started to blow up. I’d guess it was about 1995? By 2000 it was getting painful (one guy hits “reply all” to a dozen folks saying “look at this” with a copy of a spreadsheet, MS Project chart, a Word doc (which keeps revision history in it), 10 images, and a tarball and guess what happens to the email archive sizes… At a couple of places we would regularly train folks to keep docs on the docs server and just point at them rather than email repeated copies…)

    Storing Unix email archives is just not a problem. Oh, and they typically compressed well too.

    (I remember, painfully, one guy who kept filling all user free space. I put him on his own fairly large disk so that he could not hurt others. “quota” was politically unacceptable at that particular employer… so I did a sort of a ‘back door quota’ with a physical disk limit… had to watch permissions on other directories, though, else he would stuff his junk there and put links to it from his home dir. Ah, admin wars…)

  94. ianl8888 said

    80. Rephelan

    1) Not dissing the old guys – making the point that we (yes, I’m an old guy) are clueless here. You maintain you could do it in 24 hours of study … fair enough, but I am sufficiently IT-savvy to disbelieve you, OK ?

    2) Also not suggesting that the dump (as distinct from the collection) was therefore some super-hacker from the fossil fuel industry, or China, or any other such paranoia. Quite the reverse. Obviously I believe that it was internal and from a younger IT-savvy person

    One of Jones emails – 1228922050.txt – mentions this: “I did get an email from the FOI person here early yesterday to tell me I shouldn’t be deleting emails – unless this was ‘normal’ deleting to keep emails manageable!”

    There is the link that informs my view, which remains to me the most credible so far offered. Certainly better than wishful thinking. I have experience of IT people within a group talking to each other about their work, and I believe FOI2009.zip was deliberately collected internally as informed background against pending FOI decisions

  95. ZT said

    Greg F,

    What happened to ‘2’?

    Everything you say makes sense – and may indeed be the case.

    I would just point out that if the email archiving was being done at the University level – the number of emailers would be several hundred – and this would increase the volume of ‘stuff’ that would need to be sifted over the volume required for just the CRU.

    We’ll have to wait for the report/whitewash!

  96. ebonybandera said

    #90 I agree about Unix but for completeness’s sake we shouldn’t ignore cygwin on some form of MSFT platform. You’d have all the important bits of a Unix/Linux toolkit including VPN, ssl, ssh, pretty much the works. I’m not suggesting this is likely, though, merely within the realm of possibility without stretching all that much.

  97. Greg F said

    # 95

    ZT,

    So you want to know where the 2 went. Today is Wednesday and I only use 2’s on Twosday. Actually there was a 2 but I deleted it and forgot to renumber.

    The volume of “stuff” to sort through would not necessarily be as large as you think. If I was setting up the archive, at the very least, each department would have it’s own folder.

    A short history of the CRU is here. There is a link to all the staff that have worked there and incomplete dates of when. Some relevant parts from the history.

    The number of CRU research staff as of the end of July 2007 is 15 (including those fully funded by ENV/UEA).

    .

    Even in 2007 not a large staff. Including the students:

    Since its inception in 1972, over 40 students have been awarded PhD degrees, and the rate of awards has dramatically grown in the last 10 years. Today, CRU has a thriving student community of 11 PhD students (July 2007). CRU also runs a NERC-recognized Master of Science degree programme on Climate Change. For the past 10 years, this degree attracted between 6 and 10 students per year, but the last two academic years (2005/06 to 2006/07) have seen an upsurge to 22 students per year.

    So we average maybe 35 email accounts from the mid 90’s to mid 2000’s. Not a lot.

  98. StuartR said

    The Guardian have 3 Journalists making a hash of replicating what is going on this page – in fact, by the looks of it, they’ve just scraped the contents of this page and many other previous sceptic speculations about the hack/leak😉

    However, I love how they took the time to sprinkle in this comment

    Watts previously had a book published by the right-wing Heartland Institute, financed by ExxonMobil until 2006.

    Climate emails: were they really hacked or just sitting in cyberspace?

  99. […] UEA Files Were on a Single Server Since these files first came out, there has been a lot of speculation as to the source of them.  Today we learn that, […] […]

  100. Check out the informations about how to remove them Trojan Virus
    http://www.bitztoday.com/2010/02/how-to-remove-trojan-virus.html

  101. electric airsoft guns…

    UEA Files Were on a Single Server « the Air Vent…

  102. remove google redirect virus…

    […]UEA Files Were on a Single Server « the Air Vent[…]…

  103. Excellent post. Keep posting such kind of info on your page.
    Im really impressed by it.
    Hey there, You have performed an incredible job. I will certainly digg it and in my opinion recommend to my friends.
    I’m confident they’ll be benefited from this web site.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: