the Air Vent

Because the world needs another opinion

Another IT Theory on Climategate

Posted by Jeff Id on February 18, 2010

Ok, here’s one for the endless IT guys out there.  Frank Bi from the Journal of Inactivism has posted a theory on how the IT files were hacked and compiled.  It’s a graphic post so the rest of us can follow. Reproduced with permission here, click the title below to link to the original.

———–

I’ve drawn a diagram that tries to summarize how the FOI2009.zip — or, rather, FOIA.zip — containing the cracked CRU e-mails and data was created, according to the information given in the posts and comments on this sub-blog and the old IJI blog. (Click for a larger version of the diagram.) It probably needs more work, so let me know what you think.

In the meantime, feel free to spread it around!


35 Responses to “Another IT Theory on Climategate”

  1. Carrick said

    A hacker use the sh shell?

    I think not. Certainly it would bash.

  2. Jeff Id said

    I considered that aspect of this theory myself. If you had direct physical access to the server, it’s not a big step to get it to boot in a linux shell.

  3. frankbi said

    Jeff Id: Thanks for putting up the diagram!🙂

    Carrick: I’m guessing it was bash, but invoked under the name /bin/sh. Other Bourne shell implementations seem to prefer showing up as “$” rather than “sh-3.1$”.

    And it’s probably not a root shell — that’ll show up as “#” or “sh-3.1#”.

    Oh well.

  4. frankbi said

    Carrick:

    To clarify, “sh-3.1$ exit” plus a newline were the exact bytes that were observed to be tacked at the end of five Word documents.

  5. Carrick said

    Jeff ID, the back-up server probably runs Unix, so it wold have bash compiled on it.

    Frankbi:

    To clarify, “sh-3.1$ exit” plus a newline were the exact bytes that were observed to be tacked at the end of five Word documents.

    That is pretty bizarre.

    I’ve seen claims the archive was generated over several weeks. You know anyway to prove/disprove that?

  6. Kondealer said

    I’m not so much worried about how the data got out, but how both CRU and the Information Commissioners Office (ICO) who, by the way are supposed to uphold the F.O.I Act, initially colluded to withhold the data.

    I made an official complaint about this and I copy the ICO’s response below.

    Please find attached a letter in response to your query regarding the possible provision of advice by the ICO to UEA in relation to information requested regarding its Climatic Research Unit.

    Dear XXXXXXX

    I am responding to the enquiry that you made regarding what advice may have been provided by the ICO to the University of East Anglia in relation to its handling of requests for information related to its Climatic Research Unit. This has been looked into and I have outlined below the ICO’s view on this matter.

    One of the emails exchanged between IPCC authors and related parties placed in the public domain contains the following sentence:

    Keith and Tim are still getting Freedom Of Information (FOI) requests, as are the Meteorological Office Hadley Centre and the University of Reading. All our FOI officers have been in discussions and are now using the same exceptions not to respond-advice they got from the Information Commissioner.

    Viewed in isolation, this sentence may have created the false impression that the ICO provided advice to the University of East Anglia encouraging it to withhold information.

    The Commissioner does not accept this view and wants to stress that such action would be in direct conflict with the vision, aims, and values of the ICO and would undermine his role as statutory regulator. The ICO would not, in any circumstances, encourage an authority to avoid compliance with the law. To do so would undermine the Commissioner’s role as an impartial regulator and compromise his duty to support the presumption of disclosure implicit within Freedom of Information (FOI) Act and Environmental Information Regulations (EIR).

    Both FOI and EIR assume a default position of disclosure in response to requests made to public authorities, and this presumption is the default position adopted by the Commissioner in responding to enquiries and considering complaints. It underpins all of the Commissioner’s work in relation to FOI and EIR and his officer’s would not provide advice encouraging an authority to avoid compliance with the legislation.

    It is unclear what the ‘advice’ noted in the email consisted of, or indeed whether the use of exemptions being proposed resulted from contact with the ICO’s staff or interpretation of the ICO’s existing guidance. The Commissioner has a statutory duty to disseminate advice and guidance on the operation of FOI and the EIR. This takes the form of guidance documents, responses to written queries, and telephone contacts (usually through his help line).

    Although the Commissioner’s Officers seek to address enquiries as satisfactorily as possible, they only provide general and impartial advice. When responding to queries the ICO gives high level, non-specific guidance on how an authority might consider approaching a request. This can involve directing them to published good practice guidance or to relevant ICO Decision Notices or the findings of the First-Tier Tribunal. The ICO deliberately provides this advice at a general level to minimise the possibility of being drawn into specific discussions about individual requests, as the ICO may subsequently be required to adjudicate on a related complaint.

    The written queries are recorded on the ICO’s electronic case management system. Telephone enquiries are more numerous, with over 2,000 per week, and given their volume it is not practical to record the content of each (assuming that the caller consented to identify themselves, which they are under no obligation to do). The ICO has checked its records and can trace two examples of written advice provided to UEA which predate the email in question, but these were on unrelated topics with no bearing on the climate-data issue. If the University had sought verbal advice before then, the ICO would only have provided general advice, and certainly would not have explicitly supported or endorsed the use of a particular exemption or exception.

    I hope that this goes someway to explaining the ICO’s position and provides some reassurance on this matter.

    I hope to be able to provide you with a response to your other query regarding time limits for criminal prosecutions under the Freedom of Information Act shortly.

    Yours sincerely

    XXXXX
    Senior Complaints Officer
    FOI Team 1

    I think I need a Lawyer!

  7. mrpkw said

    OK, for us dummys, does that mean it was “leaked” or “hacked”??

  8. Ruhroh said

    Rightio,
    I’m also missing the implication here.

    Does the diagram version of the story suggest inside vs. outside job?

    TIA
    RR

  9. TerryS said

    Re: frankbi

    To clarify, “sh-3.1$ exit” plus a newline were the exact bytes that were observed to be tacked at the end of five Word documents.

    Hmm. Your have remote access to a server but you can not ftp/ssh or do anything other than list directories, dump files or perform a few basic shell commands. How do you get a file from the remote server?

    Here’s how.

    Open an xterm and remotely login to the target server.
    Now xterm has a logging feature that will log all output to a local file. This can be activated by echo’ing a command sequence to the terminal. Unfortunately I cant remember what the sequence is so lets call it COMSEQ.
    Run the following command:

    echo ‘COMSEQ\c’;cat TARGET_FILE

    When this completes type “exit”.

    What you now have saved in XtermLog.XXXX is the TARGET_FILE with the command prompt, exit and newline tagged onto the end of the file.

  10. Steve McIntyre said

    The inclusion of Hulme is something odd that no one’s discussed. The Guardian’s spin was that the four of them were on radar screens- but Hulme wasn’t. So why he is grouped with Jones, Briffa and Osborn in the preparation of the dossier?

  11. Duke C. said

    CRU has their own server. OS during the time span the data was taken:

    139.222.104.250 Linux Apache/2.2.3 Scientific Linux 25-Oct-2009

    Source:
    Netcraft look up- cru.uea.ac.uk

    They were running Linux, not Unix. Would this explain the BASH command anomaly?

  12. Greg F said

    There is so much wrong here I don’t know where to start.

    1. There is no MX record for the CRU as it is a sub domain of the university. (MX records are the address records for mail servers on the internet).

    2. All the CRU mail goes through a gateway mail server (Unix) as can be seen from the headers in some of the emails.

    3. It would be insane to archive email from backups of client machines when it could be done right at the gateway.

    4. Any emails received and deleted between backups would never get archived. The legal begals would not like that at all.

    5. At one point the CRU had a MS Exchange mail server.

    The university has 15,000 students and 2,500 staff. There is no way you could manage a system as fragmented as the diagram would indicate.

  13. Dave McK said

    not root shell, eh?

  14. Kondealer,

    have a look over at Lucia’s site. Perhaps you can ask the ICO office if they did a finding on weather the law was broken. If not why not.
    did they decide to not issue a finding because of the SOL

  15. attachments.

    Not all the attachments on the mails are present in the final files.

    See the last mail. See the attachment. That attachment is not in the files. I know, CRU just sent it to me. I love FOIA.

  16. frankbi said

    Mrpkw, Ruhroh:

    OK, for us dummys, does that mean it was “leaked” or “hacked”??

    It was cracked. Or, it was leaked and then made to look like a crack.

    * * *

    TerryS:

    This?🙂

    ESC ] 4 6 ; name BEL   Change log file to name (normally disabled by a compile-time option)

    This is in the console_codes(4) manual page I found on a system.

    Anyway, I tried that in a separate xterm session, but it doesn’t work.😐 I think this logging feature has been disabled in default compilations, because it can be a security problem.

  17. TerryS said

    Re: Frankbi

    I think this logging feature has been disabled in default compilations, because it can be a security problem.

    Logging is disabled by default but they could either have been using an older version of xterm or compiled there own with logging enabled.

  18. frankbi said

    Greg F:

    3. It would be insane to archive email from backups of client machines when it could be done right at the gateway.

    4. Any emails received and deleted between backups would never get archived. The legal begals would not like that at all.

    There are signs that the e-mails in the .zip came from Eudora mboxes, as can be seen from the occurrences of “c:\documents and settings\tim osborn\eudora\attach\” and “From ???@???”. So while you may be right, it’ll suggest that

    either (1) the cracker ripped the mails and attachments directly from the scientists’ Windows terminals (but it’ll be hard to explain the “sh-3.1$ exit”);

    or (2) there was a system for backing up e-mails from the Windows machines in addition to a separate backup system at the mail gateways.

    * * *

    TerryS:

    Logging is disabled by default but they could either have been using an older version of xterm or compiled there own with logging enabled.

    That may be true. Then I guess it’ll have the same effect as using script(1), except it’s more troublesome.😐

    * * *

    Carrick:

    I’ve seen claims the archive was generated over several weeks. You know anyway to prove/disprove that?

    One of the first things I did after getting the .zip file was to write a program to dump the metadata for the archive members, including the file modification times and file access times. (File access times don’t show up under normal .zip tools.) Here‘s a graph of file modification times plotted against file access times. The access times, excluding those which read 1 Jan of some year, start from some time in Sep 2009.

  19. TerryS said

    Re: Frankbi

    That may be true. Then I guess it’ll have the same effect as using script(1), except it’s more troublesome.😐

    Never thought of script but using script would put junk at the beginning of the file (navigating to the document, cat’ing it) which (s)he would then have to remove (perhaps using tail). As always with linux/unix there is more than one way to skin that cat.

  20. Kondealer said

    Thanks Steve, I’ll pop over to Lucia’s and have a gander.

  21. Atomic Hairdryer said

    I’m not entirely convinced by the filter step. I ran a frequency analysis on the email folder and results are as follows, excluding (most) common words, and stopping at <1000 hits-

    mann 3400
    climate 3367
    jones 2740
    phil 2255
    briffa 2180
    keith 2012
    science 1520
    model 1411
    ipcc 1291
    ucar 1281
    mike 1265
    tom 1228
    tim 1161
    psu 1145
    arizona 1115
    osborn 1098
    virginia 1002

    If those had been used in a simple keyword filter, then I think they'd be too broad and would still have taken manual filtering to strip a lot of chatter out.

  22. clivere said

    I still remain of the view that this post from Bishop Hill provides a plausible pointer at what may have been going on.

    http://bishophill.squarespace.com/blog/2010/1/25/the-other-snippet.html

    “He said that the current understanding in the ICO’s office was that the archive was not an official data repository, but was set up by an individual within CRU for their own use.”

    This has come from someone who is more informed than we are. It was a bit of a throw away comment and was more concerned with the FOI activity rather than the mechanism of hacking/leaking.

    This would suggest a repository requested by a very senior member of the CRU with co-operation from their IT support. I dont believe a more junior member of the department would be allowed to create such a repository to collect emails from collegues.

    We know the repository has been building up progressively over time which would probably be enabled most easily by a script to intercept messages from within the email system. That script may be as simple as selecting certain individuals or key words.

    If the person who originally requested it then left/retired the repository would continue to grow and would not be noticed unless it consumed capacity. The mechanism by which it was created may even continue through upgrades of the email system because IT staff would just migrate existing scripts across unless there was a good reason to investigate them.

  23. frankbi said

    Clivere:

    a script to intercept messages from within the email system.

    Doesn’t look like it. The mails and attachments look like they were ripped from Eudora mailboxes, or backups of those. (Besides, even if what you say is true, it’ll still be illegal wiretapping, and there’s just no excuse for it.)

  24. frankbi said

    TerryS:

    junk at the beginning of the file (navigating to the document, cat’ing it) which (s)he would then have to remove (perhaps using tail).

    Ah — indeed. Using an xterm with logging enabled, I can switch to a different log file in the middle of an xterm session, which’ll avoid the need to remove any junk at the start of the session.

  25. re 21.

    the filter would be a filter of key words from climate audit: GCM SRES bristlecone yamal tree ring isotope like so

  26. jimchip said

    I might as well muddy the waters a bit. Osborn is quoted by Mann as saying:

    Hi Tom,
    In Phil’s absence I was just now looked at his PC because I needed some files/emails for a separate matter, and I noticed that you had emailed Phil/Ray/Mike concurring with Ray’s concerns. Until I saw that, I hadn’t realised that anyone else had commented on Yang et al.
    http://eastangliaemails.com/emails.php?eid=330&filename=1056477710.txt

    I bet all sorts of people could look at PC’s, get passwords, and even do the searches from Phil’s desktop…He traveled a lot and it really seems they were loose about a lot of things. Not just data and records, security, too.

  27. Duke C. said

    Re: frankbi (Feb 18 12:29),

    To clarify, “sh-3.1$ exit” plus a newline were the exact bytes that were observed to be tacked at the end of five Word documents.

    Just for fun, I looked at the five .doc files in Notepad, which makes the file look somewhat like output to a text-only terminal screen. The “sh-3.1$ exit” prompt is neatly center-tabbed under the last line of metadata.

    Doesn’t seem like it would be difficult to write a simple terminal text editor shell script that allows the user to open a .doc file from a command prompt. When the file is saved, the last prompt is included, either by design, or just plain oversight. Doesn’t matter since it’s hidden in the metadata. Won’t show up in the doc when it’s opened in Word.

    From what I understand sh-3.1 returns a null character if the script finishes with no errors. Possibly a new line feed?

  28. […] een nieuwe theorie over hoe de CRU-emails en bestanden zijn verzameld in die FOIA.zip folder (link). Feit is toch wel dat wat er gebeurde met het vrijgeven van deze folder op 19-11 met de woorden: […]

  29. frankbi said

    Duke C.:

    From what I understand sh-3.1 returns a null character if the script finishes with no errors. Possibly a new line feed?

    bash will stuff the return status of the last executed command in $?. The value of $? won’t show up on default, but it can be queried by using keywords like “if”, “elif”, “||”, etc., or by directly asking for “$?”:

    $ uname
    Linux
    $ echo $?
    0

    I think the newlines at the end of the 5 files are just newlines.

  30. Borepatch said

    Greg F makes excellent points in his comment, to which I’ll add:

    6. It wasn’t just email. This theory fails to explain how files like the HARRY_READ_ME.TXT got in the archive. It’s unlikely but possible that this went through the email servers; the idea that a source tree did is not.

    7. Occam’s Razor says that the simplest explanation is best. The simplest explanation is that someone at CRU collected data over the course of months. You don’t need to hack the email server, just be on the right mailing lists (or have emails forwarded to you from people who are on the lists, if you’re not). There’s no requirement that this is comprehensive list of emails on this subject: it’s perfectly possible that this is simply the ones that were easily obtained via internal distribution.

    8. Remember, there have been rumors of a “mole” at CRU for the better part of a year. While it’s very likely that UEA’s/CRU’s computer security is swiss cheese, you don’t need a rococo theory of outside attackers to explain this.

    ObDisclaimer: I work in Internet Security.

  31. Duke C. said

    #30

    Frank Bi has done some good forensic analysis. However, his approach is from the standpoint that it was a sophisticated attack orchestrated by skilled hackers. I don’t see anything that would eliminate someone inside CRU using an admin level password (legit or stolen).

  32. frankbi said

    his approach is from the standpoint that it was a sophisticated attack orchestrated by skilled hackers.

    No, I found that it was probably a crack, or at least a leak made to look like a crack. The things that were done to the e-mails and Word documents don’t really make much sense for someone with actual access rights.

  33. At least he didn’t get arrested in Arizona. They have some of the most severe penalties in the US. Read about the penalties enforced under Arizona DUI law, not too pretty.

  34. I not to mention my pals were found to be reviewing the excellent helpful tips on your web page and suddenly came up with a horrible suspicion I never thanked the site owner for those tips.

  35. Hello my friend! I want to say that this article is amazing,
    great written and come with approximately all important infos.
    I would like to peer extra posts like this .

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: