I won’ t abuse you with my venting today. Mostly pictures. Pretty interesting ones I think.
I used the CPS method today with actual proxies from M08. Instead of the infilled data with the fake hockey stick glued on the end I used the original original 1357 series before processing. I also chopped off the 95 Luterbacher series which aren’t really proxies, they are temperature. It didn’t make any big difference to the shape of graphs in my results but I don’t like em.
There are more pre-deletion Luterbacher series in this data set because they were re-scrambled from 95 into 71 series for M08 most likely using RegEm according to their locations. I previously did a software pattern match for the original 89 Luter to the final 71 and found no match, yet the average was the same.
Anyway, to the fun. I used M08 CPS and r correlation to scale and sort data according to different trends. The point of this article is to show you can produce any trend you want from this method. ANYTHING!
The red line is the temperature trend I am claiming in each graph was true. The blue is the CPS algorithm temperature reconstruction. Since the big claim in M08 was the percentage of series which passed correlation 484 of 1209 take a look at the Percent used in the graphs below.
200 year Hockey stick
200 year Negative HS
100 year Hockey stick
100 year negative hockey stick — NOTE THE HIGH CORRELATION PERCENTAGE
Low r hockey 100 year hockey stick
Low r negative 100 year hockey stick. Note the amazing correlation percentage of 39.43!
Little Ice age
Little warm age
Remember how temps went up then down then up then down.
or was it down then up?
I can’t remember these things didn’t it go up and down 4 times.
Nope, wrong again. It was down then up 4 times.
What happens with an r >0.1. Wow, 495 of the proxies used!
Oh, now I remember it was seven times down and up. That was it.
Alright, you get the idea.
I don’t think Mann had to worry if he would be able to produce a hockey stick, do you?
The strongest correlation I got was for a negative trend in the most recent 100 years with r=0.1 of 39.43 percent of 1357 series (graph 6). Almost as good as Mann08 got with 484 of 1209 series for a value of 40.03%. But wait, I didn’t use luterbacher’s 89 series because they aren’t really proxies so really I had 535 of 1262 series or 42.2 percent. It is better than that though, of Mann’s 484 series 71 were Luterbacher and 90% of the remaining series were infilled with a fake hockey stick. The Schweingruber MXD series represented 105 of the total group but 95 of those passed correlation after 38 YEARS of hockey stick data was pasted on the end TO REPLACE KNOWN DIVERGENCE (down slope). So at best Mann’s percentage is 484 accepted series – 71 Luterbacher – 95 Shweingruber (briffa) series or 318 of 1209 series passed his correlation for 26.3 percent. It’s not fair to give Mann08 1209 because I deleted some for obvious reasons so 1209-71-95 = 1043 starting series. 318/1043 = 30.5% correlation.
Summary — Id percent of proxies correlation to a temp drop r>0.1 – 42.2% Mann08 30.5 temp rise!!
So without manufacturing any data as M08 did, I have achieved a correlation for a drop in temperature in the last 100 years that exceeds the best correlation Mann08 had to offer for a rise in temperature by 42.2/30.5 or 40 percent!!!!
Say what you want about this last calculation but remember I found an equal correlation without the required corrections. I also need to point out that I didn’t spend any time trying to optimize my result, it was without searching around that I achieved this correlation! Note the second to last graph with 36%!!!!
the Air Vent 
pacer521
Snip! No spam please.
The pernicious part is the elimination of the LIA and MWP though.
I like the visual of just reversing the time series before feeding it into the Hockey Stick Finder.
If you chop it off partway through the MWP, when you swap it -> Voila. Instant strong MWP.
Cool!
or is it warm? 😉
Amazing.
It is worth noting that the shaft of the hockey stick (the long flat bit) is more likely to appear in these graphs if there is a large percentage (say 20% to 40%) of the data used. This is obvious in graph 6, which uses 39% of the data.
For graphs which use little data (i.e. few proxies), the averaging of random noise (aka “proxies”) does not average out very well, and so tends to produce larger swings in temperature.
Anyway, great work Jeff. Although I’d still like to see you do redo what Mann did (i.e. correlating to recent temperature records) with say the last 100 years chopped-off the proxies (so that it thinks 1900 is 2000 for the proxies), and see if we still get a similar hockey stick.
#5 Chris H Good point. I bet he will get a similar hockey stick too. 🙂
Jeff- I just found your website recently. This is one of the best (striking, shocking, clearly described, damning) postings I have seen in the past year. It is reminiscent of the detailed posts from M&M when they first started to unravel the steaming pile known as MBH98.
My favorite fit is the four-cycle sinusoid!
My personal favourite is the 7 cycle sinewave.
Now, using this method, I guess I should go out there, find a big bunch of data series from the real world (data about anything really) and use them as proxies.
Then I should use the same method on those data to confirm the results in Mann08. I mean, with separate individual reconstructions getting the same results, the reconstruction must be right, must it not?
wow…. :)) (stil chuckling on my way out)
Like other posters I am truly impressed by all this work. Very, very telling.
How is it possible to get this sort of analysis through to the people who sit in influential places? Surely there is something or someone that/who could induce the believers (a faith thing, I’m sure) to read a bit outside their blinkered surroundings.
Something I would really like is to obtain the 95 Luterbacher series – that is proper temperature measurements (I presume) – to try them or assemblies of them with my techniques. Is there a separate URL, or do they have to be picked up from the huge array of files containing all the values? I could do with a bit of help here.
Cheers, Robin
Robin,
Actually, nobody McIntyre knows has the original series. The 89 series from the accidentally posted data set were not the same as the 71 used Luter data in M08 but the average was the same????
If you want them, I can write a script, save it as a tabbed text file and send it to you as xls, txt or whatever. Your email is visible to me only so just ask.
For Chris H,
I did download giss temp data, import it and run the math. Same result as above. I am going to have some fun this afternoon and post it tomorrow.
To everyone,
Thanks for the support. It takes hours to do this stuff and your comments and understanding keep me going. I do need to get this out across the web more. When you find yourselves arguing with a global warming group, don’t be afraid to reference this.
In the meantime, if the 500 or so people who read this today don’t mind. Paste a couple of links around, it brings traffic. My online experiment with a hockey stick got picked up by a couple of international organizatons and over 5000 read it, not bad for an 8 week old blog. Still this post is much more important and people need to realize the problems with this math.
Jeff,
Excellent work. Please take up the offers and get this published asap.
Happy B-Day to me, Happy B-Day to me…. Presentations from an engineering perspective. What a perfect present.
I have to stick up for the engineer here. Not being a mathematician, yet never getting less than an A in any math class I have ever taken, I love the engineer’s approach.
Great work. Calcs and methods provided, not obscured.
So, you and Mr. McIntyre are pretty much on the same page here?
Thanks again
EJ,
I think he is trying to replicate what I am doing. I am redoing all this in R so people can do it themselves. THX
I hope I am not correct in my thinking here, that this error could be deliberate. I hope that these people are simply mistaken, and that they did not deliberately do this.
Is this method, apparently used by authorities, the standard? If so, then this discussion should trigger immediate changes. If not, then my take is that this is not about science, but only the ‘aura of science’.
The public can’t differentiate the difference.
It wouldn’t surprise me at this point, as the more I educate myself about ‘climate science’, the more shenanigans I uncover from the ‘IPCC Team’ and thus the necessary misunderstandings by those truly trying educate themselves. So many explanations seem to be written in doublespeak, especially the IPCC summaries.
I am rather tempered, in that most of this is water under the bridge, as defined by public consensus. It no longer matters if it is true. The discussion is now based on beliefs, just like politics. Do you believe in the climate? Do you believe in Civil Rights? Thus science has been reduced to a vote by the scientifically illiterate.
The state of Kansas, through their representatives, have recently banned a planned and designed coal power plant based on this nonsense.
Back to the engineers. Hand our engineers the problem now. Not economists, not climate ‘scientists’, and certainly no social scientists, let alone any biologists.
The Problem: No more manmade CO2 is allowed, ever. What do we do? This deserves a few billion, no?
Preliminary studies indicate that manmade CO2 elimination can only be achieved by the elimination of X billion people.
Next question?
Jeff, I sent a note the webmaster at co2sceptics.com. They posted this article in News Headlines.
The direct URL is http://co2sceptics.com/news.php?id=1938
Thom,
Thanks much. This kind of thing really helps.
Jeff
Ej,
Many of the papers use this technique. I am not the smartest person in the world and I found it quite easy to show the flaws in the methods. My other posts show the magnification of recent years and demagnification of historic year trends based on red noise using the same methods.
As an engineer, I think you will like this post.
https://noconsensus.wordpress.com/2008/09/29/simple-statistical-evidence-why-hockey-stick-temp-graphs-are-bent/
I am so certain that this was deliberate, but I don’t have any evidence other than the simplicity of the manipulation. Mann’s group has come up with at least 3 techniques which produce the same or similar results, CPS, EIV and de-centered PCA. How could his group screw up the to the same result 3 times?
Hi Jeff,
The Wegman Report is the gift that just keeps on giving. Figure 4.7 of the WR shows a white noise pseudo proxy sim with the IPCC 1900 climate reconstruction buried therein. Wegman uses CPS rather than PCA in this example to mine the 1990 shape from the data. It is almost as if Mann 2008 used Fig. 4.7 as the instruction manual to find new hockey sticks.
“The point being made with Figures 4.6 and 4.7 is that if there are hockey sticks in the
underlying data and if they are decentered, then the CFR methodology will selectively
emphasize them. Similarly, if there are ‘hockey sticks’ in the data series and the
remainder of the data are uncorrelated noise, then the CPS method will also emphasize
the ‘hockey stick’ shape. However, if the data contain other shapes and these methods are
applied to data containing these other shapes, then these methods will selectively pick out
those shapes. In Figure 4.6, by decentering the 1990 profile, we inflate its effective
variance so that PCA will preferentially pick it as the first principal component. In Figure
4.7, the independent white noise will be incoherent and thus tend to cancel out while the
‘signal’ is the same in every proxy and will thus tend to be additive. The point here is that
if each (or even a large percentage) of the proxies is selected with the hockey stick shape,
then the incoherent noise will cancel and the coherent ‘hockey stick’ shape will emerge.
Thus even discussions of ‘independent replications’ of the hockey stick results by
different methods may not be what they superficially appear to be.” Wegman Report, p.37
Mann 2008 attempts to be an answer to the critics. The data and code are archived and published (sort of, a vast improvement in any event).The REGM trick is out of Mann 2005 which used REGM to infill temperatures post 1971, thereby responding to the M&M criticisms to the trunction of the instrumental period in MBM98-99. The use of both low and high frequency proxy data in Mann 2008 is directly responsive to Wegman as well. Mann 2008 uses the r sort process to decenter the results in Mann 2008. Would cherry picking by any other name be a sweet?
Wegman’s comment in his report appears to remain true.
“We note that there is no evidence that Dr. Mann or any of the other authors in paleoclimatology studies have had significant interactions with mainstream statisticians.”
Wegman’s point is that neither PCA nor CPS are appropriate to study proxy data. He does make a suggestion and perhaps it might be a fertile path for further efforts on your part.
“In reality, temperature records and hence data derived from proxies are not modeled
accurately by a trend with superimposed noise that is either red or white. There are
complex feedback mechanisms and nonlinear effects that almost certainly cannot be
modeled in any detail by a simple trend plus noise. These underlying process structures
appear to have not been seriously investigated in the paleoclimate temperature
reconstruction literature. Cohn and Lin (2005) make the case that much of natural time
series, in their case hydrological time series, might be modeled more accurately by a long
memory process. Long memory processes are stationary processes, but the corresponding
time series often make extended sojourns away from the stationary mean value and,
hence, mimic trends such as the perceived hockey stick phenomena.
One type of such long memory processes is a process driven by fractional Gaussian noise
(fractional Brownian motion). An object with self-similarity is exactly or approximately
similar to a part of itself. For example, many coastlines in the real world are self-similar
since parts of them show the same properties at many scales. Self-similarity is a common
property of many fractals, as is the case with fractional Brownian motion. A serious effort
to model even the present instrumented temperature record with sophisticated process
models does not appear to have taken place.”
Best,
Howard W.
Howard,
Thanks much for the post. I really have a great audience.
I am fairly new to climate science so it is helpful when people point out these things. I read the MM response to the 08 paper where it complained about missing data and I wondered if that caused the infilling. Thanks for clearing that up.
I pulled this quote below from Wikipedia, it describes how to model Brownian motion with red noise (an integral of white noise). This is the method I have used in my other posts which demonstrate statistical deamplification of historic data compared to recent times. Because I had a lot of red noise proxies, I was able to actually plot the distorted temperature scale on top of the extracted hockey stick signal.
“In science, Brownian noise (Sample (help·info)), also known as Brown noise or red noise, is the kind of signal noise produced by Brownian motion.”
I have been studying statistics on the side so I can properly match Brownian noise to the proxies. Tamino at open mind has several good posts on the subject. I am working on using ARMA (auto regressive moving average) to match the different types of proxies in M08 (red noise). ARMA is what I used in my other posts, I just didn’t match it to proxies, I tried basically everything I could think of. So far it looks like R does ARMA and ARIMA matching pretty automatically for you.
My thought is to match the red noise, then apply more and less standard deviation just to show it is always a real and strong effect. I am not a statistician so it takes a bit of study. Fortunately the subject isn’t that complex.
There is actually a constructive discussion at RC on the Cohn & Lin paper. Rasmus and others at RC argue for non-stationary treatment. This field is also linked to the area of the Hurst coefficient which as far as I understand it, (not too far), measures the level of uncertainty in a stationary time series. I think the math gets alot harder if you treat a series as stationary, which might explain the frequent non-stationary approach (if you treat it as stationery, it is easily folded and filed however).
The elephant in the room that no one really wants to look at is that the data sets (proxies, observations, anecdotals, etc.) we have are inadequate (too short, too inconsistent, too adjusted and tweaked) for the analytical tasks we are using them for. Using long memory techniques immediately runs into the issues of duration (how long is “long”).
Hweismaan,
I don’t believe any of the tree ring data can be treated as temp- MXD or otherwise. There are too many other influencing factors. Certainly there are nowhere near enough proxies but adding more bad ones won’t help. Sorry trees.
I’ll check out the link at RC later.
BTW:
I feel that the discussions about the intricacies of the red noise used are simply methods of obfuscation of the problem. Steve McI had some criticism that his red noise imparted an artificial signal. This argument is completely unreasonable to me as the burden of proof on a new mathematical sorting technique belongs on the author.
When even I can so easily demonstrate a hideous flaw in the math, every paper in which it is employed should be reviewed and likely re-written to correct for the effect (as I hope to do)or scrapped. I am astounded to find this effect is well known in the field and yet reviewers continue to consider accepting these papers at all.
If I can generate a correction method, it would be a positive style paper rather than a critique!! The tree ring guys can continue cutting tree stumps and everything will go on normally until they realize the large re-amplification of historic data creates a huge MWP. Trees still make lousy thermometers.
The criticism was that Steve McI “trained” the red noise to impart (replicate?) the signal. I also think a contention like this has to be modelled and proven. Not just stated, as though it meant something. When you read the Wegman Report, it is obvious that they found MM criticisms to be correct.
In your post Jeff Id Says:
October 13, 2008 at 2:23 am did you mean “MM response to the 08(?) paper”? Did I miss a paper?
You got me on that John — Should be 98.
Perhaps I was seeing the future!
2nTLBg5MmEzN4
Jeff,
SM directed me here from CA following my comment that Propensity to Hockeystickiness looked like a function of the r value used to screen the data. It looks like that’s what you’re showing here. I’m still reading your stuff, in the meantime do you have a link to the Mann (2008) paper?
(2008)…that’s an 8 in case it does it again.
#26 I sent it by email.
Jeff,
In your earlier posts you saw a significant dip prior to the calibration period. Others have mentioned autocorrelation. I’m not sure whether this feature was what they had in mind but it seems to me that this is the explanation. In the calibration period you select series which are increasing with time, over the calibration period. Due to autocorrelation these series will tend to also be increasing with time; that is, continue to decrease as you go backward in time. What concerns me is the time scale of the persistance of this effect. This seems to be more than 200 yers which sems much too long for the likely physical processes involved.
This dip isn’t obvious in your latest study. Does that mean no (or little) autocorrelation here?
Davidc
The level of noise in this number of series can cause small features to change. The best way to learn this is to write a bit of code to see the process work yourself.
IMO, the only physical process we’re looking at has to do with the physics of the math.
Did you find the part where the Briffa (Shweingruber) data was chopped off due to lack of upslope in the calibration range and replaced with a regEm upslope version and 90% of these make it through correlation screening?
Jeff,
Well, it’s not small it’s actually a dominant feature. No, I didn’t notice the Briffa data. So what about P0.1 in the spreadsheet?
I’d like to join others in suggesting that you get this into publishable form. If you want to do that and feel daunted by the formalities I’d be willing to help (my final PhD student will finish this month – I’ve got lots of experience in guiding people through the publishing process; not climate science, but that might be a virtue of sorts).
A suggestion on presentation:
1. The original (fake) proxy data (spaghetti graph?)
2. Instrumental record and proxies over calibration period.
3. Proxy data after screening.
4. Rejected proxy data.
5. Average (or something) to give hockeystick
All different graphs.
Sorry, didn’t post correctly. P less than 0.1 in paper.
David,
Thanks for the offer. I’m not sure it will do any good to publish. I’ve had several offers from qualified people such as yourself but at this point I’m a bit discouraged with the state of the science. It seems so unbelievably simple that I think in this case it’s being done on purpose. Still I need to consider opportunities like this carefully, my company is taking a great deal of time now. Do you mind if I wait on it for a while?
Jeff,
Of course, that’s fine. It just seems that you’ve done most of the work already. As for why bother, it certainly wouldn’t turn things around overnight. But peer review still matters. And a publication opens up new avenues of communication. For example, most universities will have sessions devoted to discussing the literature, which means published papers not blogs. Sometimes papers are put up precisely because someone thinks they’re bad (“this says Mann is wrong, let’s have fun ripping it to shreds”); there’s possible progress when they actually find it a bit harder to destroy than they expected. Another audience would be scientists outside the field of climate science. One of the “rules” in academia is to not criticise other specialities (largely because of the risk of being made a fool of) but it is alright (and generally safe) to ask a question like “so what exactly is wrong in the paper by Id claiming Mann is wrong”. And if the reply is “it’s total rubbish” it’s OK to follow up with “of course, but just in case someone asks me, where is the fatal flaw”. Won’t happen with a blog.
Hello, Jeff,
Sorry about the prolonged absence, but now at last I write re Posting #10. I would dearly like to have the data set you said you could provide. Ideally, a .txt file would be my preference. It doesn’t matter what the delimiter is.
I looked at Mann’s data for the ’98 paper a very long time ago. Steve provided it for me. I was able to show to my satisfaction that that the HS was a myth, and also that there was a good resemblance between many of the proxies and the “real” temperature data, in that they tend to produce the same “signature” when plotted appropriately. Assemblies of the proxies were also interesting. I have recently re-visited the data and done a careful comparison of the data between 1820 and 1971 (I think!) for which all 112 data columns are complete. The picture is the same as for longer periods with some missing data. I’d be happy to email you some plots which I think you will find interesting/convincing!
If it’s now too much trouble to assemble the data please don’t bother. I have plenty of avenues waiting to be explored.
Best wishes, Robin Bromsgrove UK
I’ll work on it today. I thought I sent it before.
Cute looking graphs. But do you have some actual specific assertion wrt Mann’s work?
#36 Click the link hockey stick temperature distortion posts. Besides some of the explorations of correlation based sorting, I went through all Mann 08 data using Mann08 methods and was able to create any pattern I could want. I even provided turnkey R code so you can do it yourself.
The argument of the paper was that a large percentage of the data was retained by correlation so it must be right. I was able to retain a similar percentage of the data with a downslope in this post, demonstrating one of hundreds of falsehoods. The math is simple enough that a determined high school student can figure it out which is why I believe it is intentional.
Write a real paper, make a real assertion.
The modus opperandus of the skeptics is to draw some graphs, say “aha” and then chortle with the choir.
It pisses me off almost as much as RINOs helping Goldman Sachs rape the country.
Jeff: If things are as dramatic as you think, then you should have at least a critical comment worth. The thing is…in general I find that you chortle too early. Heck, even Roman who is on your side has had to caution you.
I mean…I’m at a fatigue level. Why even read this thing? Why consider it? IN the past I have and have dissected your posts, bit by bit. It’s just not worth it though. If you really have something STRIP IT DOWN and publish. Just as an engineering drawing should be clean and well laid out and carefully proofed…SO SHOULD A SCIENCE COMMUNICATION BE!
It’s not JUST sympathy for the reader…although there should be much, much, more of that…but that the process of making definite assertions on the record in a clean written style…forces you to think better yourself…AND makes it easier for others to note your errors.
Whatever.
Moi aussi.
SM directed me here from CA following my comment that Propensity to Hockeystickiness looked like a function of the r value used to screen the data. It looks like that’s what you’re showing here. I’m still reading your stuff, in the meantime do you have a link to the Mann (2008) paper?