Ok, here’s one for the endless IT guys out there. Frank Bi from the Journal of Inactivism has posted a theory on how the IT files were hacked and compiled. It’s a graphic post so the rest of us can follow. Reproduced with permission here, click the title below to link to the original.
———–
I’ve drawn a diagram that tries to summarize how the 
FOI2009.zip — or, rather, FOIA.zip — containing the cracked CRU e-mails and data was created, according to the information given in the posts and comments on this sub-blog and the old IJI blog. (Click for a larger version of the diagram.) It probably needs more work, so let me know what you think.
In the meantime, feel free to spread it around!
the Air Vent 
A hacker use the sh shell?
I think not. Certainly it would bash.
I considered that aspect of this theory myself. If you had direct physical access to the server, it’s not a big step to get it to boot in a linux shell.
Jeff Id: Thanks for putting up the diagram! 🙂
Carrick: I’m guessing it was bash, but invoked under the name /bin/sh. Other Bourne shell implementations seem to prefer showing up as “$” rather than “sh-3.1$”.
And it’s probably not a root shell — that’ll show up as “#” or “sh-3.1#”.
Oh well.
Carrick:
To clarify, “sh-3.1$ exit” plus a newline were the exact bytes that were observed to be tacked at the end of five Word documents.
Jeff ID, the back-up server probably runs Unix, so it wold have bash compiled on it.
Frankbi:
That is pretty bizarre.
I’ve seen claims the archive was generated over several weeks. You know anyway to prove/disprove that?
I’m not so much worried about how the data got out, but how both CRU and the Information Commissioners Office (ICO) who, by the way are supposed to uphold the F.O.I Act, initially colluded to withhold the data.
I made an official complaint about this and I copy the ICO’s response below.
Please find attached a letter in response to your query regarding the possible provision of advice by the ICO to UEA in relation to information requested regarding its Climatic Research Unit.
Dear XXXXXXX
I am responding to the enquiry that you made regarding what advice may have been provided by the ICO to the University of East Anglia in relation to its handling of requests for information related to its Climatic Research Unit. This has been looked into and I have outlined below the ICO’s view on this matter.
One of the emails exchanged between IPCC authors and related parties placed in the public domain contains the following sentence:
Keith and Tim are still getting Freedom Of Information (FOI) requests, as are the Meteorological Office Hadley Centre and the University of Reading. All our FOI officers have been in discussions and are now using the same exceptions not to respond-advice they got from the Information Commissioner.
Viewed in isolation, this sentence may have created the false impression that the ICO provided advice to the University of East Anglia encouraging it to withhold information.
The Commissioner does not accept this view and wants to stress that such action would be in direct conflict with the vision, aims, and values of the ICO and would undermine his role as statutory regulator. The ICO would not, in any circumstances, encourage an authority to avoid compliance with the law. To do so would undermine the Commissioner’s role as an impartial regulator and compromise his duty to support the presumption of disclosure implicit within Freedom of Information (FOI) Act and Environmental Information Regulations (EIR).
Both FOI and EIR assume a default position of disclosure in response to requests made to public authorities, and this presumption is the default position adopted by the Commissioner in responding to enquiries and considering complaints. It underpins all of the Commissioner’s work in relation to FOI and EIR and his officer’s would not provide advice encouraging an authority to avoid compliance with the legislation.
It is unclear what the ‘advice’ noted in the email consisted of, or indeed whether the use of exemptions being proposed resulted from contact with the ICO’s staff or interpretation of the ICO’s existing guidance. The Commissioner has a statutory duty to disseminate advice and guidance on the operation of FOI and the EIR. This takes the form of guidance documents, responses to written queries, and telephone contacts (usually through his help line).
Although the Commissioner’s Officers seek to address enquiries as satisfactorily as possible, they only provide general and impartial advice. When responding to queries the ICO gives high level, non-specific guidance on how an authority might consider approaching a request. This can involve directing them to published good practice guidance or to relevant ICO Decision Notices or the findings of the First-Tier Tribunal. The ICO deliberately provides this advice at a general level to minimise the possibility of being drawn into specific discussions about individual requests, as the ICO may subsequently be required to adjudicate on a related complaint.
The written queries are recorded on the ICO’s electronic case management system. Telephone enquiries are more numerous, with over 2,000 per week, and given their volume it is not practical to record the content of each (assuming that the caller consented to identify themselves, which they are under no obligation to do). The ICO has checked its records and can trace two examples of written advice provided to UEA which predate the email in question, but these were on unrelated topics with no bearing on the climate-data issue. If the University had sought verbal advice before then, the ICO would only have provided general advice, and certainly would not have explicitly supported or endorsed the use of a particular exemption or exception.
I hope that this goes someway to explaining the ICO’s position and provides some reassurance on this matter.
I hope to be able to provide you with a response to your other query regarding time limits for criminal prosecutions under the Freedom of Information Act shortly.
Yours sincerely
XXXXX
Senior Complaints Officer
FOI Team 1
I think I need a Lawyer!
OK, for us dummys, does that mean it was “leaked” or “hacked”??
Rightio,
I’m also missing the implication here.
Does the diagram version of the story suggest inside vs. outside job?
TIA
RR
Re: frankbi
Hmm. Your have remote access to a server but you can not ftp/ssh or do anything other than list directories, dump files or perform a few basic shell commands. How do you get a file from the remote server?
Here’s how.
Open an xterm and remotely login to the target server.
Now xterm has a logging feature that will log all output to a local file. This can be activated by echo’ing a command sequence to the terminal. Unfortunately I cant remember what the sequence is so lets call it COMSEQ.
Run the following command:
echo ‘COMSEQ\c’;cat TARGET_FILE
When this completes type “exit”.
What you now have saved in XtermLog.XXXX is the TARGET_FILE with the command prompt, exit and newline tagged onto the end of the file.
The inclusion of Hulme is something odd that no one’s discussed. The Guardian’s spin was that the four of them were on radar screens- but Hulme wasn’t. So why he is grouped with Jones, Briffa and Osborn in the preparation of the dossier?
CRU has their own server. OS during the time span the data was taken:
139.222.104.250 Linux Apache/2.2.3 Scientific Linux 25-Oct-2009
Source:
Netcraft look up- cru.uea.ac.uk
They were running Linux, not Unix. Would this explain the BASH command anomaly?
There is so much wrong here I don’t know where to start.
1. There is no MX record for the CRU as it is a sub domain of the university. (MX records are the address records for mail servers on the internet).
2. All the CRU mail goes through a gateway mail server (Unix) as can be seen from the headers in some of the emails.
3. It would be insane to archive email from backups of client machines when it could be done right at the gateway.
4. Any emails received and deleted between backups would never get archived. The legal begals would not like that at all.
5. At one point the CRU had a MS Exchange mail server.
The university has 15,000 students and 2,500 staff. There is no way you could manage a system as fragmented as the diagram would indicate.
not root shell, eh?
Kondealer,
have a look over at Lucia’s site. Perhaps you can ask the ICO office if they did a finding on weather the law was broken. If not why not.
did they decide to not issue a finding because of the SOL
attachments.
Not all the attachments on the mails are present in the final files.
See the last mail. See the attachment. That attachment is not in the files. I know, CRU just sent it to me. I love FOIA.
Mrpkw, Ruhroh:
It was cracked. Or, it was leaked and then made to look like a crack.
* * *
TerryS:
This? 🙂
This is in the console_codes(4) manual page I found on a system.
Anyway, I tried that in a separate xterm session, but it doesn’t work. 😐 I think this logging feature has been disabled in default compilations, because it can be a security problem.
Re: Frankbi
Logging is disabled by default but they could either have been using an older version of xterm or compiled there own with logging enabled.
Greg F:
There are signs that the e-mails in the .zip came from Eudora mboxes, as can be seen from the occurrences of “c:\documents and settings\tim osborn\eudora\attach\” and “From ???@???”. So while you may be right, it’ll suggest that
either (1) the cracker ripped the mails and attachments directly from the scientists’ Windows terminals (but it’ll be hard to explain the “sh-3.1$ exit”);
or (2) there was a system for backing up e-mails from the Windows machines in addition to a separate backup system at the mail gateways.
* * *
TerryS:
That may be true. Then I guess it’ll have the same effect as using script(1), except it’s more troublesome. 😐
* * *
Carrick:
One of the first things I did after getting the .zip file was to write a program to dump the metadata for the archive members, including the file modification times and file access times. (File access times don’t show up under normal .zip tools.) Here‘s a graph of file modification times plotted against file access times. The access times, excluding those which read 1 Jan of some year, start from some time in Sep 2009.
Re: Frankbi
Never thought of script but using script would put junk at the beginning of the file (navigating to the document, cat’ing it) which (s)he would then have to remove (perhaps using tail). As always with linux/unix there is more than one way to skin that cat.
Thanks Steve, I’ll pop over to Lucia’s and have a gander.
I’m not entirely convinced by the filter step. I ran a frequency analysis on the email folder and results are as follows, excluding (most) common words, and stopping at <1000 hits-
mann 3400
climate 3367
jones 2740
phil 2255
briffa 2180
keith 2012
science 1520
model 1411
ipcc 1291
ucar 1281
mike 1265
tom 1228
tim 1161
psu 1145
arizona 1115
osborn 1098
virginia 1002
If those had been used in a simple keyword filter, then I think they'd be too broad and would still have taken manual filtering to strip a lot of chatter out.
I still remain of the view that this post from Bishop Hill provides a plausible pointer at what may have been going on.
http://bishophill.squarespace.com/blog/2010/1/25/the-other-snippet.html
“He said that the current understanding in the ICO’s office was that the archive was not an official data repository, but was set up by an individual within CRU for their own use.”
This has come from someone who is more informed than we are. It was a bit of a throw away comment and was more concerned with the FOI activity rather than the mechanism of hacking/leaking.
This would suggest a repository requested by a very senior member of the CRU with co-operation from their IT support. I dont believe a more junior member of the department would be allowed to create such a repository to collect emails from collegues.
We know the repository has been building up progressively over time which would probably be enabled most easily by a script to intercept messages from within the email system. That script may be as simple as selecting certain individuals or key words.
If the person who originally requested it then left/retired the repository would continue to grow and would not be noticed unless it consumed capacity. The mechanism by which it was created may even continue through upgrades of the email system because IT staff would just migrate existing scripts across unless there was a good reason to investigate them.
Clivere:
Doesn’t look like it. The mails and attachments look like they were ripped from Eudora mailboxes, or backups of those. (Besides, even if what you say is true, it’ll still be illegal wiretapping, and there’s just no excuse for it.)
TerryS:
Ah — indeed. Using an xterm with logging enabled, I can switch to a different log file in the middle of an xterm session, which’ll avoid the need to remove any junk at the start of the session.
re 21.
the filter would be a filter of key words from climate audit: GCM SRES bristlecone yamal tree ring isotope like so
I might as well muddy the waters a bit. Osborn is quoted by Mann as saying:
Hi Tom,
In Phil’s absence I was just now looked at his PC because I needed some files/emails for a separate matter, and I noticed that you had emailed Phil/Ray/Mike concurring with Ray’s concerns. Until I saw that, I hadn’t realised that anyone else had commented on Yang et al.
http://eastangliaemails.com/emails.php?eid=330&filename=1056477710.txt
I bet all sorts of people could look at PC’s, get passwords, and even do the searches from Phil’s desktop…He traveled a lot and it really seems they were loose about a lot of things. Not just data and records, security, too.
Re: frankbi (Feb 18 12:29),
Just for fun, I looked at the five .doc files in Notepad, which makes the file look somewhat like output to a text-only terminal screen. The “sh-3.1$ exit” prompt is neatly center-tabbed under the last line of metadata.
Doesn’t seem like it would be difficult to write a simple terminal text editor shell script that allows the user to open a .doc file from a command prompt. When the file is saved, the last prompt is included, either by design, or just plain oversight. Doesn’t matter since it’s hidden in the metadata. Won’t show up in the doc when it’s opened in Word.
From what I understand sh-3.1 returns a null character if the script finishes with no errors. Possibly a new line feed?
Duke C.:
bash will stuff the return status of the last executed command in $?. The value of $? won’t show up on default, but it can be queried by using keywords like “if”, “elif”, “||”, etc., or by directly asking for “$?”:
I think the newlines at the end of the 5 files are just newlines.
Greg F makes excellent points in his comment, to which I’ll add:
6. It wasn’t just email. This theory fails to explain how files like the HARRY_READ_ME.TXT got in the archive. It’s unlikely but possible that this went through the email servers; the idea that a source tree did is not.
7. Occam’s Razor says that the simplest explanation is best. The simplest explanation is that someone at CRU collected data over the course of months. You don’t need to hack the email server, just be on the right mailing lists (or have emails forwarded to you from people who are on the lists, if you’re not). There’s no requirement that this is comprehensive list of emails on this subject: it’s perfectly possible that this is simply the ones that were easily obtained via internal distribution.
8. Remember, there have been rumors of a “mole” at CRU for the better part of a year. While it’s very likely that UEA’s/CRU’s computer security is swiss cheese, you don’t need a rococo theory of outside attackers to explain this.
ObDisclaimer: I work in Internet Security.
#30
Frank Bi has done some good forensic analysis. However, his approach is from the standpoint that it was a sophisticated attack orchestrated by skilled hackers. I don’t see anything that would eliminate someone inside CRU using an admin level password (legit or stolen).
No, I found that it was probably a crack, or at least a leak made to look like a crack. The things that were done to the e-mails and Word documents don’t really make much sense for someone with actual access rights.
At least he didn’t get arrested in Arizona. They have some of the most severe penalties in the US. Read about the penalties enforced under Arizona DUI law, not too pretty.
I not to mention my pals were found to be reviewing the excellent helpful tips on your web page and suddenly came up with a horrible suspicion I never thanked the site owner for those tips.
Hello my friend! I want to say that this article is amazing,
great written and come with approximately all important infos.
I would like to peer extra posts like this .