Ok, so last night I had written a very nice reply to Dr. Schmidt’s comment on my last post. I was fairly ticked with the aloofness of his comment (which I would have never seen except for help from John Turner, thanks ). I ended up deleting it though in favor of this post which has a little different message.
First here’s gavins reply to the only one of many questions at RC.
# James Martin Says:
7 June 2009 at 10:01 PM
I read today a claim that in the paper published recently by Dr Steig et al. in Nature regarding the Antarctic warming trend, there is a weighting problem. They claim that most of the weighting comes from the peninsula stations, which represents a relatively small part of the continent.
I was wondering if this is in fact the case? It doesn’t seem likely, but could you comment on this at all? If these assertions are left unchecked, before you know it they’ll be taken as fact.
[Response: The point of the Steig et al paper was to use spatial correlations in recent data to look at how under-sampled parts of the continent likely changed over longer time periods. Those correlations will necessarily weight different stations differently as based on the physical characteristics. The analysis you saw is simply a fishing expedition, an analysis of what the calculation is doing (fair enough), combined with an insinuation that the answer is somehow abnormal or suspicious (not ok). But how is this to be judged? What would be normal? No-one there can say and they would prefer simply to let people jump to conclusions. It’s kinda of typical of their tactics, but not a serious scientific point. – gavin]
I didn’t expect a reply from RC at all and was fairly irritated with the point that it’s not ok to question the validity of this paper. Like somehow I’m not qualified to make a comment on the linear weighting of thermometers. I pretty well shredded this last night which was therapeutic but it served no purpose than my own entertainment – so before finishing I sent the post to the either.
Gavin says it’s not a serious scientific point.
So let’s talk about what is a serious scientific point and is the primary problem discovered last weekend. First, the pie charts were actually a surprise to me. Over the many posts on this paper, dozens of people wrote to say that the peninsula warming was spread across the continent. I wrote several times that I didn’t believe that was the entire reason for the trend. You can get a feel for the math when you are working with it enough and I bet several of my more regular readers will recall a few of my comments that I thought the peninsula warming increased the trend but wasn’t responsible for that much of the trend.
– Now we know I was wrong again, it could have been a good title for the post.
The pie chart wasn’t the aha of the last post, although it was the best visual evidence. Ryan, some commentors and others picked up on this right away. The big deal of the last post was that 5 of 34 – 15% of the manned temperature stations were inverted. Completely flipped upside down. What’s more is that they came with heavy weights and trends. This is the #1 problem discovered this weekend. Even if the peninsula weighting was zero it didn’t matter, the real issue is the mystery of the upside down thermometers.
Claims that it is somehow ok to flip thermometers and average them together because the paper get’s the desired result are advocacy – not science. For Gavin to claim otherwise is nothing more than a pathetic and disingenuous attempt to defend his friends. I wonder how prominently this paper will be featured in the upcoming socialist one-world climate summit the apolitical gavin is so excited about.
Again, this doesn’t pin Gavins response on Dr. Steig or his coauthors. I’m certain Steig et al didn’t check the weightings this way so they didn’t know. Weightings should be a component of the reconstruction verification just as they should be in the hockey stick reconstructions. People need to see the weights to evaluate the quality of the result yet this is not a standard in climate papers. Lack of weight verification has resulted in a number of bad reconstructions taken apart at CA and I would be surprised if there aren’t a dozen new papers coming out with the same problems.
Of course any RegEM has the possibility of a flipped trend, this is a potential flaw in the method which can only be addressed through verification. In situations where the data is sparse and noisy the potential for the flip increases. If the weight of the flip is very low and a fractional percentage of the reconstruction, you can say it wasn’t important. However, in this case 4 of the 5 flipped stations had large negative weights and high trends. Oddly enough these stations were in the peninsula so overall they had primarily positive trends and flipping them likely reduced the overall weighted trend.
Anyway, I’m tired of the attitude on RC. I’m tired of working on this broken paper too but the work is not done yet. In the meantime, please don’t start reading your thermometers upside down. I’m just an engineer and not a PhD climatologist but from my own experience there is a 95 percent certainty that they really don’t work well that way.
————
Update,H/T Layman Lurker. — Ryan’s famous now, Michael Mann my favorite hockey stick maker and repeat data crime offender replied to this question:
“I was encouraged by Dr. Eric Steig’s blog exchange with several people who had analyzed the methods used his Nature paper on temperature trends in Antarctica. By the time Dr. Steig ended the exchange, the tone of the discussion was much more reasonable and constructive than at the beginning, and it appeared that even Dr. Steig agreed that there were some legitimate concerns raised, although he did not agree these concerns brought into question the results shown in the Nature paper.
[Response: Please don’t misrepresent Eric. You need to read what he wrote more carefully. He did not indicate that there were any “legitimate concerns raised”. Rather, he explained in some detail how the analyzes described on a certain fringe website were rather seriously flawed, e.g. violating the assumption of independence of the statistical cross-validation by adjusting the model to fit the validation data–a major no no, at least to anyone who understands cross-validation. Eric did note that an objective analysis of quality issues with the satellite data would be worthwhile–but that is hardly what was provided in the attempts to attack Steig et al. We closed off the discussion after the post had achieved its end, i.e. when the attackers conceded that indeed they were unable to in any conceivable way ‘falsify’ the Steig et al ‘08 results -mike]”
So now we’re all wrong too and Mike lives in a fantasy world. Perhaps Ryan will reply to Mikes misunderstanding of what is Ryan’s obviously superior yet still admittedly potentially flawed reconstruction.
the Air Vent 
1. You don’t RATE being tired. You haven’t even written a paper. You make yourself tired because you get all breathy with the “final straw claims”, which tend to be jumps to conclusion and indulgence of us-versus-them emotionalism.
2. Maybe* the recon is better with some inverted stations. You just get yourself all jacked up, because the answer DIFFERS from the trivial one (not area weighted, contains inversions) rather than thinking through and demonstrating that the trivial recon is more skilled than the non-trivial one, that the non-trivial one fails criteria for added complexity (lowered DoF), etc.
*Pre-emptive caveat: I’m not “betting” that it is. Just saying you have not proved the contrary and that from the beginning the hope of Steig et al was that higher sophisticated methods would drive a more likely answer. They could have gotten it wrong, but that’s not a priori…it depends on the quality of their algorithm. Certainly we could create scenarios for certain types of data/algorithm where a more searching signal processing type of correlation study gets a more likey answer than simple sampling/normalizing/averaging. (Have Ryan explain this to you.)
We closed off the discussion after the post had achieved its end, i.e. when the attackers conceded that indeed they were unable to in any conceivable way ‘falsify’ the Steig et al ‘08 results -mike]”
This is what you are up against Jeff. You are one of the barbarians at the gate that must be beat back at all costs. You must know by now that there is nothing you could post nor any analysis you could perform that would be conceded to be anything but nonsense by the team. They are well past the point of looking at numbers or caring about data. The cool weather must be making them irritable.
However, there is a very very good upside to all this. By not allowing any sort of real discussion of the numbers on their site, they have in effect ceded a large part of the playing field and are writing themselves out of the game. I noted that one of the more thoughtful posts there asked a neutral question as to why RC snipped posts that were not in any way derogatory or defamatory or anything else but simply questioned mathematically what was going on with the Steig paper. The answer was the usual flippancy and for sure another intelligent enquirer was left wondering what RC was trying to hide.
If somewhere along the way you find the time I’d like to try my hand at an Antarctic reconstruction and would be grateful if you could email the temperature and satellite sets or maybe point me to a link where I might download them.
thx
david
#2 the datasets are very large and can be downloaded through the R scripts. I’ve been learning Ryan’s format, SteveM’s is what I’m used to. They are different enough that it takes some work.
If you look at Ryan’s recent posts, the code contains URL’s to all the various data sets – there are quite a few.
I tried to post the following comment a while back at RC (now gone from moderation):
JeffId:
Think about market activity. Basically almost every single sector is down. So predicting general economic activity based on sectors would use some weighted sampling of them (maybe auto after-market is less sensitive than OEMs). But there are some sectors (Walmart, cobblers) that are up this recession (and typically up in recessions).
TCO, applying your logic example to Jeff’s post means that if you were trying to impute “Walmart, Cobblers” sector performance using RegEM, the positive sales would be flipped and shown to be a net contributor to the overall economic trend.
Jeff – why on Earth would you keep Gavin’s and Mann’s (and Steig’s) opinions in such a high regard? Unless your goal is to be accepted among them as a peer, the fact that they dismiss your work with “it’s not a serious scientific point” is of little interest (apart than as a base to develop pop-psychology innuendos on their _need_ to present themselves as superior human beings)
Where is this moderation list of posts people keep talking about?
#8
Mike if you are talking about my attempted post at RC, when you submit a post it will tell you that the post is “awaiting moderation”. Once Gavin or MM or whoever has read the post they decide to post it through to the thread or not. My post stayed “awaiting moderation” (visible only to me) for about an hour and then it disappeared. Usually that means that the comment is gone for good, however sometimes they retrieve the comments and/or post a response.
#5, It’s analogous to saying, Wall Mart is down so flip the data over. It’s like the commercial where the monkey’s in the board room rotate the graph to get an uptrend and all cheer together.
It cannot under any form of reality be acceptable to invert a temperature anomaly, the only response is for them to say – it didn’t have a noticeable effect.
It did have a noticeable effect though so there we are – As they say, stick a fork in it b/c it looks done.
Jeff:
Sorry I am slow, but I am unable to locate Ryan’s posts which contain the R script for downloading the data sets. If you would be so kind to point me in the right direction….
#11, See the bottom of this post.
https://noconsensus.wordpress.com/2009/05/28/verification-of-the-improved-high-pc-reconstruction/
From Gavin’s RC reply to James Martin :
[Response: The point of the Steig et al paper was to use spatial correlations
in recent data to look at how under-sampled parts of the continent likely
changed over longer time periods. Those correlations (snip)…]
Jeff, didn’t you show early on that RegEM didn’t take into account the
physical location of the station? So isn’t this just in the arm-waving category?
#13 – That response is worthy of a politician during a debate. Translation: “Yeah, the peninsula got most of the weight. But if I inject a bunch of garbage about spatial correlations and then insult those who are questioning the method, you will probably find the approach entirely reasonable without actually understanding, in the end, whether or not the assertion is true or valid. In fact, if you’re posting here, you’re probably a moron with no life, so we’ll just treat you like the worthless fodder you are.”
I stopped reading RC about two years ago. Thanks to JJ&R I’ve had to start perusing the site once again. Keep up the good work; as a humble EE I truly appreciate the detailed analysis you do.
#14, It was actually an insult to his readers, who of course ate it up. I should try that on you guys, my ass would be under the bus tires so fast …
#13, Gavin is just restating what RegEM is intended to do. It’s total arm waving, you shake the red flag in front of you so the stupid bull focuses on that and step out of the way as it runs through 😉
RegEM is attempting to weight according to covariance with the high density low quality sat data. The trend is weighted according to its ability to match the sat data so if everything was low level noise the spatial weighting could potentially work. That’s what makes it interesting as a method. They just didn’t do enough verification and are defensive as hell about tougher questions.
Jeff,
I really enjoy the technical analysis you have done and think statistical expertise would be a valuable assistance to any practicing scientist, Much as when it happens at RC, the other “bullshit” is uninteresting and not valuable to any good discussion, and it happens at both places.
While I don’t agree with much of what you say outside of the statistical box, I’ll keep reading and filter out the stuff I don’t like (as if you care). I recommend you and your readers do the same thing at RC.
Statisticians and climate scientists need to be corroborating but your approach as well as Steve’s is not going to make it happen. Ryan O has made the best approach so far and I hope he keeps it up. The work you guys have done deserves to be written up (as a comment or letter response to Steig et al and I hope you do it). Not because I think it doesn’t matter if it isn’t peer-reviewed just that it was really good work and should be published.
Okay, back to lurking until the Pens win tonight….
OT: What can justify the tone of the RC authors day after day, post after posts, issues after issues?
After all, these are researchers who virtually have every tool and credits at their disposal. They publish books, papers virtually unopposed. Every of their words are magnified by a sympathetic media. Internationally, they are not isolated but can find solace in every country where even their blog is considered a “scientific file” link like at the IPSL in France the domain of the ubicuistre Jouzel. They even received the highest political award through the Nobel IPCC. The UN bureaucracy, the gold standard if there has to be one in this domain, eats in their hand the new results of their research. Rich, influent, powerful peoples are helping them setting up blogs, managing PR campaigns. Their careers have taken off like few scientists could expect in their lifetime. They are today’s scientific establishment and have the means to comfort their position of power through political support no smaller than during the atom bomb development. Every conference, and there are many, offers tribunes to their unprecedented results.
Yet, their tone is quite unforgiving and exhibits an acute arrogance, they operate as a predatory pack and they hardly exhibit the generous and higher warm qualities of those wonderful gifted human beings whose superior knowledge is never compromised by an inferior moral compass. They simply, ruthlessly cannot be wrong.
Why is it so? Immoderate hubris or deep seeded fear that Nature will retake with time what some men gave them at once? Is this desire to crush any dissent the supreme way to look for and expect to be absolved of any responsibility, “we ALL thought we were right, no single exception” when their models will drift from the reality, when humanity will finally wonder what was this all about and those Robespierres of the Climate will meet their scientific fate? Time will tell and so far time is their enemy: past time, history and its record -that they needed to alter- and future time, when natural variability will start to explain the dichotomy between predictions and reality…
Charles Krauthammer once wrote that the difference between Republicans and Democrats is that Republicans think Democrats are wrong and Democrats think Republicans are evil. Any rational observer of politics over the last 4 decades will recognize the truth of that statement. AGW is a lot more about politics than it is about science. Keep that in mind when wondering about the “tone” over at RealClimate.
Michael Mann has published some truly mind-boggling rants which give a good insight into the way he thinks. He has ascribed political motivations to those who critiize his work. The attitudes over at RC are those of people who view themselves as leaders in a moral crusade against evil.
“He has ascribed political motivations to those who critiize his work.”
This is the phenomenon of “projection”-a clearly politically motivated person or group is accusing the opposition of political motivation. The left projects their own evils onto the right all the time. They even project projection. It’s funny if you are smart enough to see what they are doing…
Don’t worry Jeff, the tone of your blog stands in contrast to RC and of course your resident itch TCO who at least appears to have learnt how to use a spellchecker. Or for him “Leearnt to us a spollchecker.”
Just remember…in every scientific field but one, scientists are constantly seeking debate (instead of cutting it off), are not scared of debating non-scientists holding different views, and win those debates hands down (think Velikovsky, think UFOs, think evolution).
Why would things be different in that one scientific field, is anybody’s guess. Including the possibility that it’s not that “scientific” after all
#22, That’s right. If their position was stronger they would discuss it openly. Nearly all of my polite on topic posts have been cut there in favor of the public circular celebrations of their following. The majority of the comments I left were quite innocuous, requests for data and such.
Obviously requests for data are innocuous in themselves but do you seriously think the off-science comments and topics are covered any better here than any other blog dedicated to one thing or another? You may come up with a couple examples of stifled debate but do you seriously think that impugns the entire field? And you wonder why we can’t have an intelligent debate? It seems that you are not interested in one either.
And spare me the Krauthammer quote. Yeah republicans are the reasonable ones. There are plenty of loons on both sides and he is one of them. The greenies have their stake in this as well as the right and neither can see the forest through the trees.
I’m done. I’ll just go back to RC where I belong. Can’t say I didn’t try.
“Beam Scotty up…” Jean Luc Picard/Gavin… LOL
I believe the rest of that quote is “No intelligent life here” LOL
6. I thought about clarifying this more (dealing with you maroons is tedious). Sure…if you have PERFECT sampling of all sites (temp) or all businessess (GDP), all you do is average them. However in a LIMITED DATA situation like ANTARCTICA, you need to do a correlation study to show what weight to put on the predictor.
Imagine you only had a single market predictor and that was “cobblers”. You would invert it. Capisce??
#17,24- Scott
1) Pens may win tonight but then they will lose next game. Wings will go easy tonight (City/burbs of Detroit need the revenue of a game 7). 😉
2) I think your #17 post was great and made some good points and is a well-deserved “atta-boy” to the bloggers. Your #24 post
illustrates the need for thick skin by all who participate in these blogs. Science is a lot of hard work, by both sides, and when politics/policy is in the mix, emotions definitely run high. It’s great to see debate. go wings!
Scott,
I know I pay a price for venting my opinions. I’ve stated it many times and even put it in the title of the friggin’ blog. It is a place where I and others let off a bit of steam about the insane world we live in where we can debate whether a thermometer should be read upside right or upside down. Which part of that doesn’t sound like politics rather than science?
BTW Scott, which debate are you trying to have? I can still say you didn’t try if you don’t try.
And for the record, the Republicrats are not and have not been conservatives and this is why they don’t have the public support. Our entire government is f’d up so don’t expect me to back them up either. All I want is less of it.
#27 – That’s insane TCO a thermometer is a thermometer and it measures thermo not anti-thermo. It makes me think you are not an honest blogger but a dishonest troll. I had hoped you would be a reasonable critic.
10 and 29.
A. Wrong. Read up on multple regression.
B. Same basic point you’ve been making from a while ago. Where’s the aha? Where’s anything new?
———-
[snip] Groundhog Day.
Definitely OT, but Omnologos, wrote: “and win those debates hands down (. . . think evolution).” That’s a good one — very rich. If you pause to contemplate the similarities in the debates (actually, don’t just pause, go ahead and make a list) it is quite insightful.
Thanks for the response. I thought I had made it clear which debate I am trying to have but will try again. As a meteorologist who in the very near future will be publishing a paper that has significant statistical analysis involved (positve matrix factorization for source apportionment)I very much believe that a collaboration is needed between statisticians and any scientists using said stats to make conclusions. Because my paper will not be about climate change it will not be noticed but I am sure if a well-trained statistician examined it he could find faults in what we did.
I am not completely untrained in statistics (2 semesters) but realize this falls significantly short of any expertise and relied heavily on people with more experience. I would welcome any questions of the work and don’t know of any people I have met in the field that wouldn’t. I can’t and won’t speak for others but when people (not people, comment #22) say that the field I am currently involved in is fraudulent and not really science it’s hard not to get a little bent, and you seem to agree in comment #23 that that is the case.
I guess my hope is to keep an open dialogue with people involved in this issue because the advantages of the collaboration should be obvious to everyone. I came over to this site because of the Steig paper and the response and comments by Ryan O. The debate I think that is necessary is what is the best strategy for seeing that this collaboration takes place.
I think we should all try to be sensitive to the knee jerk reactions people have to political stuff they don’t agree with. Let’s all pretend we have no political views to make Scott more comfortable. After all, he might not be able to accept any analysis we do solely because the politics makes him uncomfortable. That would be…unfortunate. Of course, if it’s us over at RC, we need to tune it out. I just want everyone to be clear on the ground rules.
No need to make me comfortable. I won’t be staying unfortunately.
Hopefully not because of anything I said.
Scott,
Yesterday I left a post at RC whose exact content I can’t reproduce at the moment because I am at another computer but the gist of it was as follows:
Another poster (ray) opined that he would like to see more detail about what the models predicted and how the models evolved over time as new information became available. I agreed wholeheartedly with the idea and chimed in that it would be great if the predictions were quite specific and as detailed as possible because then people would be able to get a real handle on whether or not they were actually making successful predictions. I went on to say that it was reasonable for people to want such detail as they were being asked to take short term hardships for some unknown future payoff. The IPCC has a large enough spread in the model outcomes so that on one end even if it were come to pass it might be less expensive to let the warming occur than to ‘fix’ with present dollars.
I went on to say that I thought that a market based approach, specifically a futures contract on an ocean heat content index could shed some light on the consensus as to what people really feel about the science because what people are willing to carry as an opinion is often different from what they will ascribe to with cold hard cash. Needless to say this post did not make it through moderation.
So my question to you as someone who feels comfortable with RC, is if something as innocuous as this can not make it to the light of day what really is the point of RC other than as a place to pontificate and preach to the choir. Don’t you find it offensive that the people who are supposed to be in the vanguard of the science have so little respect for the opinions of those not in their inner circle that they can’t even stand the slightest hint of questioning or dissent?
You mentioned above that you would look at Jeff’s statistics and leave alone the rest of his political views. If you are conversant with statistics and math just a little bit then for sure you know that the Steig reconstruction is horribly flawed and virtually meaningless. You also know that if this was not the case then Steig or Mann could calmly and patiently point out the error of Jeff et al.’s ways complete with the math and climate physics to back it up. If I were them and had the facts on my side I would slowly and carefully rebut the criticism and leave the comments open for as long as anyone was willing to read. But they don’t cause they can’t and in the end that is all that matters.
Like many of the people who frequent Jeff’s blog and CA I am someone who is not at all connected with climate science but have enough math, physics and programming background to understand what’s going on. Why is it that people such as myself who come to the debate with no preconceived ideas and no agenda find such hostility in the climate ‘establishment’ at least as it is represented at RC? My only brush with academic science was a year I spent as a research assistant (i was responsible for automating some experiments) at a solid state physics lab some 30 years ago. I found the people that I worked with extraordinarily open and willing to patiently explain anything at all to me regardless of the silliness of my ideas or my obvious lack of expertise with the material. With this as background I am astounded at what I see in the climate arena.
Hope you would be someone who would stay here and further the discussion.
Then indeed RC is your planet!
But seriously, please let us know when your paper is published as I’d be interested in reading it for my own info.
#31 Scott,
Actually your approach is reasonable except that you keep throwing the blog out the door. I welcome different views and would enjoy learning your method. There is a good deal of open science in climatology, I’m a fan of the NSIDC as a department for their openness and transparency.
There is a lesson I learned when I was younger that many and perhaps even most scientists haven’t figured out. Science has a very harsh way of being right or wrong, there isn’t much middle ground. Often when you are certain you find out later you are not. I started with the antarctic paper thinking there was a potential issue but hoping there wasn’t. Why start that way? Other scientists have concluded a lower trend for a long time and sea ice is growing not melting. Why hope there wasn’t a problem? Because there would be something which I could point to that would demonstrate an open non-advocacy.
Why the title of the blog? – noconsensus. Think about my previous paragraph. People read it as anti-agw which is not correct but fine and I don’t care. Actually, I’m anti-consensus -proudly.
This crowd could use a good dose of reasonable climatology at this point. Not this upside down thermometer stuff – which is asinine. So I would like to offer you the opportunity to make a post here freely whatever you would like to discuss. Perhaps your upcoming paper. I’ll actually moderate the post too, and chop any ad-hom attacks – something I haven’t done in the past.
Code and data are a requirement if they are used but if I’m right, most people will actually contribute constructively and ask good questions rather than hack away. Give it a thought anyway, it might be fun but beware, in science – you might be wrong.
Scott Robertson #17
Screwed up the blockquote closure after “happen”. Sorry
With cut RC posts, I posted one on ‘Science vs advocacy’ about confirmation bias, saying that if the hockey stick paper had come up with a different result showing that the current temperatures are cooler than in the past, then scientists would have found the math errors. Instead because they liked the results, they didn’t check as hard. Of course this didn’t make it.
I am curious if the hockey stick results could be reproduced, like Ryan did with the Antarctic warming, to produce different shapes?
Scott:
“I would welcome any questions of the work and don’t know of any people I have met in the field that wouldn’t.”
Would you agree that the RC team is not entertaining questions in regard to Steig et al?
“I can’t and won’t speak for others but when people (not people, comment #22) say that the field I am currently involved in is fraudulent and not really science it’s hard not to get a little bent, and you seem to agree in comment #23 that that is the case.”
If questions are not entertained, and data and code for replication is not shared freely, there is a reason for that. What is it?
“I guess my hope is to keep an open dialogue with people involved in this issue because the advantages of the collaboration should be obvious to everyone. I came over to this site because of the Steig paper and the response and comments by Ryan O. The debate I think that is necessary is what is the best strategy for seeing that this collaboration takes place.”
Then why must you leave?
Re #39 “Nobody bit your head off so you seem to be a bit thin-skinned.” Guilty as charged probably the main reason I don’t have a blog.
Re #38 A gracious offer Jeff, but it case I wasn’t clear earlier my paper doesn’t have anything to do with climatology it is about long-range transport of saharan dust to the eastern US. We analyzed data collected by the IMPROVE project using positive matrix factorization (a recent article about PMF – Paatero, P., and Hopke, P. K.: Rotational Tools for Factor Analytic Models, J. Chemom., 23, 91-100, 10.1002/cem.1197, 2009) for source apportionment. My adviser had previously used PCA but felt PMF was superior for source apportionment and it is my understanding that the EPA will be using it also. I don’t think I could provide the code for the program as it is not freely available and requires a license for use. If someone would look it over I would be curious if PMF would have an application to the Antarctic paper. It seems to me it would not.
The only discussion I wanted to pursue was cross-discipline cooperation and I doubt that is worthy of a whole post, but appreciate the offer.
I apologize for any fiestiness in my comments.
Scott:
“The only discussion I wanted to pursue was cross-discipline cooperation and I doubt that is worthy of a whole post, but appreciate the offer.”
cross-discipline cooperation.
am I missing something?
Sheesh. Ask and yee shall receive. TCO, I was kind of worried that there was only one of you and something bad had happened… like being knifed in a bar fight…. or disappearing from the blogosphere at the same time Gavin was busy elsewhere…..
I do agree that this should either be published or assembled into a final PDF that I can send to the idiots who are currently representing me in Congress…
Scott,
“The only discussion I wanted to pursue was cross-discipline cooperation and I doubt that is worthy of a whole post, but appreciate the offer.”
cross-discipline cooperation.
are you talking about me with my IEEE background?( http://en.wikipedia.org/wiki/Ieee )
or the years of math and computer programing from here ( http://lssu.edu or here http://www.NMC.edu )
If you have anything science to post THIS IS the place!!!
be aware how every most of us posting comments are highly edu-mac-ated!
one thing that gets my notice is the lack of regard to scientific principals
this is one that is blatant (0.119 et al steig) http://en.wikipedia.org/wiki/Accuracy_and_precision
please read the above and note that there is no way that you can measure temperature to a 1000th of a degree in the wilds, in a lab with a ten foot thermometer maybe but don’t walk up to it to read it because your body’s infrared will give you a misreading, or you might breath on it and your CO2 breath will heat it up. ( yes the co2 bit is an adhom attack! lol )
every time I see a 1000th of a degree I laugh out loud!
If just one person can convince you of the shenanigans going on would help the debate.
but only the highly educated can catch this type of thing. how many have taken calculus?
or chemistry? I would have been failed and kicked out ( NMC chemistry) if i said I could measure to a thousandth of a degree!
BTW jeff why not give us the computer answer of 0.0987654321 or go out to one million millionth place? why stop at ten thousandth? hope to get you to see this and soon!
cricket out
#27
TCO, what do you think the pie charts would look like if there was a cluster of highly correlated, “pattern matched” stations around the cooling south pole and only a single station on the peninsula with unrelated (or negative) covariance? Your argument implies that nothing would change. Indeed, a valid reconstruction should be insensitive to to any hypothetical change in geographical configuration of the predictor stations. If it is not insensitive, then the negative weighting of Amundson-Scott is an artifact.
For the same reasons, a vaild reconstruction method should also be insensitive to grid weighting but it is not: https://noconsensus.wordpress.com/2009/02/20/satellite-temperature-trend-regridded/
For the same reason, a re-trended reconstruction with a constant trend at every data point should not show the same (relative) spatial distribution of trend as Steig but it does: https://noconsensus.wordpress.com/2009/04/20/retrended-regem-recon/
For the same reason, a valid reconstruction method should be insensitive to dropping the peninsula, but it is not: https://noconsensus.wordpress.com/2009/04/17/no-peninsula-regem/
The covariance matrix should not be dominated by a tiny region within the reconstruction area. This is a recipe for artifacts and spurious correlations.
Scott Robertson said
June 9, 2009 at 11:09 pm
Scott, I think your paper is worth a whole post and may have more to do with climatology than you think. Please, engage us. At the very worst, if you can survive this crowd, the peer-reviewers will be candy.
#39
“beware, in science – you might be wrong.”
I had a prof who once told me: “In science, failure is just as important as success.”
Scott, glad you dropped by. Please take Jeff up on his offer and share your project with us. Maybe you could start with a post giving us an overview of the project and the hypothesis.
I’ve commented before on the presence of negative correlations in other types of atmospheric fluctuation data before besides these climate data, and like many of us I’ve been scratching my head on what they mean for climate data.
Generally a negative correlation is evidence for wave-like behavior, and since large-scale atmospheric convection exhibits that behavior (e.g., Rossby Waves, Kelvin waves, gravity waves, etc) this in itself isn’t that surprising.
As with the data of my colleagues and myself in boundary layer measurements, the expectation here is that one should see a different correlation function down-wind versus cross-wind. And that has an important ramification that the the covariance matrix is not likely to be stationary over time, since large-scale steering currents don’t remain stationary overtime.
The closest I’ve come to an explanation for why that might not matter is from Ryan O, who observed that the seasonal average of the covariance matrix may remain relatively stationary even if a seasonally-averaged covariance matrix fluctuates wildly from winter to summer.
Unfortunately I stay busy enough with my own research that I don’t have the time to properly analyze this data set. I doubt I would use RegEM on Antarctica data , because as I understand it, it was meant originally for a dense network of stations with few missing data points, not a sparse network of stations with many missing data points.
Shorter version of my comments: Negative correlations aren’t always bad for reconstructions (thogh the absence of correlation is), but they may signal that additional care needs to be take to include e.g. steering winds into the model so that you end up with a robust result.
I think the correlational structure of atmospheric fluctuations could be related to underlying causal behavior, but the approach should be to first map out how these two are causally linked before trying to do this sort of reconstruction.
Ideally, one would use a model that includes temperature, pressure and horizontal wind speed fields. I don’t know enough about what data is/isn’t available for Antarctica, but certainly these data are available in the United States, and if one wanted to verify the skill of a particular method, that would be the place to start, not a sparse network of stations in a remote corner of the Earth.
In a real sense, though, the Steig method should be seen as progress because it is pushing the community in the proper direction. (For those of us at least who view examination and iteration upon published manuscripts as other than a “fishing expedition. /snark)
Scott Robertson,
If you think that Charles Krauthammer is a loon, your view of politics and what constitutes cogent, rational analysis is so far removed from the mainstream that you can’t see it from where you are. Charles Krauthammer a loon?!
You got your panties twisted because I pointed out that AGW is more politics than science. If you had actually read Michael Mann’s outrageous slanders of Lawrence Solomon and his statements about McIntyre and McKitrick, you would have realized that my characterization of him in my previous comment was too mild. If you were aware of the AGW alarmists who openly lie to the public and defend their lies as appropriate, you couldn’t find the point about politics offputting. If you understood how the IPCC operates — how its scientists refuse to abide by the rules, its bureaucrats fold, spindle, and mutilate the science in their reports, and its authors routinely ignore all the science which disagrees with their preconceived notions — you’d have no choice but to acknowledge that AGW is a political beast. Don’t take my word for it. Take that of the dozens of scientists, many of whom not skeptical of man’s role in influencing climate, who have made that point.
Perhaps I’m a little cranky, but that tends to happen when I watch a corrupt news media crank out orgasmic propaganda for their messiah in the White House while he steamrolls over the rule of law, spends my children into massive debt, sets himself up as ruler over the financial, auto, and health care industries, while pushing to dominate all other business activity in the nation by adding trillions in taxes and enacting AGW regulation. I’m sorry, but trifles like that tend to affect my mood a bit.
When Krauthammer wrote his piece about Democrats thinking their opponents evil, perhaps he was thinking of the number of times that conservatives have been slandered as mean-spirited, hate-filled, racist, sexist, homophobes who want to exploit workers and despoil the environment. If he was, I have no doubt that Mann’s hit job on Solomon would have been precisely of the type of evidence upon which he relied.
Scott, I’m sorry I have no patience for you, but I’m tired of scientists who have abandoned the scientific method, slander anyone who disagrees with them, demand that opponents be imprisoned, lie to the public and openly advocate that it is appropriate for other scientists to lie to the public as well. That also makes me a little cranky.
The Steig paper is only the latest in a long line of sloppy messes which are passed off as climate science. When one of the reputed leaders in the field refuses to share data on the grounds that someone might try to find something wrong with his work, you know all you need to know about climate science. I find it repugnant that any responsible person would advocate for measures that will deny billions of the world’s poorest any hope of a better life on the basis of some of these “studies”, especially when no one replicates them or even bothers to audit them. Science is supposed to be self-correcting, but climate scientists don’t adher to that concept. They find the notion that someone might check their work offensive. I find their behavior morally bankrupt.
Go on back to RC and wallow with them in their moral and ethical sewer, untroubled by different viewpoints and smug in the confines of their intellectual cocoon.
Re: #53, Stan: Wow!! Onya Stan. Well said. I recall you made some other pithy posts as well. We should be collecting them!
The quotes from Mann et al would be more annoying if snarky, supercilious replies were in any way abnormal for the Team.
The same problems that have always dogged their work persist: (a) they try to do their own math in-house despite demonstrable inabilities to make proper unbiased use of higher statistical methods and (b) they have a clubby in-house peer review process that simply circles the wagons on ideological grounds.
Can anybody find anything this summary from the 2006 Wegman report that has changed or improved in the 5 years since its publication:
test
Jeff, keep up the good work. In fact you should be encouraged: the complete failure of RC to answer the question from “James Martin”, Steig’s rapid closing of the PC thread and Gavin’s inability to answer Ryan on their ‘robust’ thread just shows that they have absolutely no answers to all of this.
As one of the “dozens of people wrote to say that the peninsula warming was spread across the continent”, I’d like to say it was just a gut feeling and it is great to see you have really confirmed it, and the effect is larger than I had thought.
Jeff,
Gavin’s comment made me giggle, Mann’s made me snarl. His is simply patently false. I admitted no such thing, and neither did you.
This is why I have no problem throwing Mann under the bus.
What utter shit.
Anyway, negative thermometers indicate a big problem during the “calibration” phase of their reconstruction. It’s the same problem that the hockey stick has with inverting proxies. A negative thermometer makes no physical sense. None. Period. TCO aside.
Small negatives are okay, because they basically indicate zero contribution plus some sampling error. But the big negatives are not.
Layman (one thing at a time):
“#27
TCO, what do you think the pie charts would look like if there was a cluster of highly correlated, “pattern matched” stations around the cooling south pole and only a single station on the peninsula with unrelated (or negative) covariance? Your argument implies that nothing would change. Indeed, a valid reconstruction should be insensitive to to any hypothetical change in geographical configuration of the predictor stations. If it is not insensitive, then the negative weighting of Amundson-Scott is an artifact.
1. Pie chart: I’m not sure whether you mean the specific pie charts from running the Steig algorithm (including the restriction to 3PCs) or ANY searching algorithm of the sort (as opposed to trivial gridding). I have been trying to differentiate the two concepts. As before, Id misses the subtlety, Ryan gets it and I don’t know about you. I can’t comment on the details of what Steig’s algorithm would do (both from interest and ability), so will comment on the latter.
I think that a good algorithm would allocate weight to the predictors based on their demonstrated correlation during the overlap period. The multiplicity of sensors at the South Pole would make them get incorporated a little more, but that effect would soon die off, so that essentially what you sample is an average of those sensors. Think about election poll sampling: if I poll 100 people on the East Coast and one person on the West Coast, I would probably not completely normalize the weightings of the two groups versus population, but this effect drops off fast.
I would expect that the temperature gradient across Antarctica is not linear (as a trivial area matching routine assumes…think about…respond and let me know you get my point….). Would expect that the ocean moderates temperature near the coast, but that that effect drops off fast. Maybe a parabola, or even a step curve would be a good approximation. IOW, the slope of temp versus lattitude gets lower as you approach the pole. Note, I don’t KNOW this to be the case. I’m speculating. But certainly there will be some deviation from linearity. So independant from the multiplicity issue, I would expect the South Pole sensor group to have a larger representation than the coastal one even if the South Post sensor group were a single sensor.
2. Please clarify what you mean by “pattern matched” (not being argumentative, honestly don’t know what you mean).
3. Unrelated (or negative) covariance at peninsular station:
A. Zero and negative are very different things. Significant negatives could be very helpful and thus justify weighting of that predictor (and with a negative coefficient). Rememeber the cobblers. Remember that there are stocks that have negative beta (well pretty few, but certainly there are financial instruments that do.) I’ll explain the beta thing if that sounds like I’m blowing smoke. Figured you have an MBA and know CAPM. No offense if not.
B. Realize that the approach in discussion creates a predictor for multiple spatial points. So presumably, the peninsula might be a good predictor for near to it and a worse one for further away and perhaps even negative if there are “wave” like patterns (ala jet stream, El Nino, etc.) I think you need to clarify which geographic region the peninsula is negative to, not to the overall average.
C. I tend to use the terms correlation and covariance interchangeably. I realize there is a difference, so let me know if there is anything important here or if the basic insight remains for the purpose of this discussion.
D. “Nothing would change”: In an over-sampling situation and with a good method (perhaps not the Steig one, but one that is non-trivial), the end result should be robust to changed inputs. Of course, we are far from an over-sampling condition. The situation is more one of making the best use of the limited info we have. And certainly having multiple sensors in a tight group at the South Pole along with one in the peninsula would not be maintaining an over-sampling condition.
4. If…then…negative weighting artifact: I’m straining to actually understand what you are asserting. Please break it down more explicitly, so I can respond. My quick response is that I don’t think that you have proven that negative weights are an artifact. It could be. But I don’t think it has to be.
It is possible for instance that there is a weather pattern which cools the US mostly, but which creates local hot spots. I’m making this up, but imagine that Santa Ana winds tend to occcur more often when the trans-Rocky mountain region is cooler in September. In that case, high temps at the beach in San Diego would correlate to high winds from the mountains, would correlate to cold temps in the East, would correlate to overall US lower temps.
TCO-it is one thing to speculate that there maybe be a negative teleconnection, quite another to assume you’ve proven it and exploit that. BTW if you actually find such a connection as your example I will nominate you for a Nobel Prize for meteorology-right after I get the Swedes to create such a prize…
#52 Carrick
What you say makes sense. Since the covariance/correlations reflect high frequency patterns, then a negative relationship means that since it is cooler at point “a” it is likely to be warmer at point “b”. Over time temps would see-saw back and forth and be offsetting. In the long run there may be a “tendancy” favouring “a” or “b” and there would obviously be underlying causes for such a tendancy. What I think has been demonstrated, is that true the relationship between “a” and “b” has been corrupted not just by improper processing parameters, but also biased predictor data. Is it a reasonable inference that a 50 year cooling trend of the south pole is strongly indicative of net warming on the rest of the continent?
Aside from the issue of whether spatial correlations have been properly modeled, one also (as you suggested) has to question that these spatial patterns are stable over time.
I also seriously doubt that the even properly modeled correlation patterns would properly reconsruct a phase shift in a low frequency climate factor such as ozone changes or ocean oscilations. IOW, factors which cause long term cooling of the south pole which are independant (not indicative) of warming in western Antarctica or the coast.
Don’t be fooled by TCO, there is no rational for flipping the trend of 5 of 34 thermometers although in the mush of his rambling in #59 he has expressed exactly the reply I expect to receive.
I’ll save it for when the time comes.
# GTG TCO, I will respond later today or tonight. In the meantime my response to Carrick might tell you a bit where I am comming from.
TCO, What has happened is that the noise level is high enough that oscillations in the weather pattern have taken over the low PC reconstruction and improperly (negatively)weighted temp stations rather than localizing them.
This may have been pointed out upthread, but negative weights are not invalid in a regression if the input variables are dependent. Think of the simplest case, Y = X, and you measure X three times using three different measurements (thermometers) X1, X2 and X3. Then any solution Y = aX1 + bX2 + cX3 is valid provided a + b + c = 1 and a^2 + b^2 + c^2 = 1 (assuming unit variance for the Xs and Y). For the lazy (ie me), Wolfram Alpha gives the solution.
One solution is a=1/2, b = (1-sqrt(5)) / 4 and c = (1 + sqrt(5)) / 4, which has b negative.
The problem is not with the regression itself but in interpreting the weights on the individual predictors Xi.
One solution would be to constrain all weights to be non-negative, which would give you a Quadratic Programming problem. But even then you will still have problems interpreting the weights because the solution is (almost) underconstrained.
Another possibility is to condition the problem using Ridge Regression, rather than OLS. It would be interesting to try that.
Gavin Schmidt is a complete fraud who should be fired and facing charges for his activities that not only include this kind of demogogery, but outright theft of discovery as proven by Steve McIntyre on CA.
REPLY: Please try to refrain from accusations like this. Gavin deserves some scorn but I want a slightly better tone than RC has in their thread.
Layman Lurker, one of the things that we need to keep in mind when we see negative correlations is that the scale-length for where these negative correlations becomes dominant depends on the group velocity of the wave across the continent. This is turn is generally (but not always) proportional to the mean wind speed velocity of the steering winds (which is a different thing that surface winds).
Jeff ID:
I don’t think he’s as far off as you suggest on this, however. he is not familiar with multidimensional problems of this sort so his instincts are probably off about what is doable and not doable with negative correlations in climate data.
I don’t think calling them “negative thermometers” is a fair characterization in any case. Rather they are “just” thermal fluctuations that are approximately 180° out of sync with thermal fluctuations at remote sites.
I am pretty sure the negative correlations are associated with wave-like phenomena, and as such should be model-able using an improved optimization function that takes into account the additional independent variables needed to characterize these phenomena: namely horizontal steering (“geostrophic”) wind speeds and pressure as well as temperature. It is this 4-dimensional field that needs to be constructed, not just one variable in it, and probably the “best” approach would be to solve the problem using a mesoscale weather model (‘back casting”).
If you have the needed data, this isn’t nearly as hard as it sounds, as I understand it most of the heavy lifting has already been done, and public domain codes are available for solving this sort of problem. The one I am most familiar with is MM5,and really it runs fine on any decent desktop computer.
Anyway, ahd the negative correlations not existed, I think the Steig method would have had a decent shot of reconstructing the temperature, even given the sparsity of the network and the large blocks of missing data, My instinct is that the existence of negative correlations is a type of smoking gun indicating that their approach is inadequate for this problem and I think that is where your “alarm” is going off too.
WoodNFish:
Schmidt is not a fraud and is very good at what he does. I don’t think he’s a particularly effective advocate fort his side, in fact I think he completely sucks at it, because his approach is so polarizing. Even somebody like myself, who is more of a “moderate global warmer” than a “skeptic”, find him to be extremely off-putting.
For myself, I would rather we not descend to the level of name calling seen on RealClimate. There’s nothing wrong with setting the standard for a higher level of discourse, no matter how frustrated we get with the tactics of “the other side”.
#69, I asked for the same thing just before I read your comment.
#68, This is the reply I actually expect from the RC crowd. There are a couple of points to your statement.
First, if the AVHRR data was more clear we would expect that a weather pattern based variation from a distant area to be reduced in relationship to the actual area the thermometer was placed. However, when the pattern is broken into 3 pc’s there is a forced oscillation effect which results in a strong negative correlation to the PC data rather than to the actual temperature. This is the reason that I believe Ryan’s higher PC reconstruction may have fixed the problem.
The forced oscillation is created by PCA and is clearly visible in some of the video work I did earlier.
https://noconsensus.wordpress.com/2009/03/29/satellite-anomaly-video-know-your-data-pt-3/
In any case at all, the thermometers long term trend is not correlated to covariance of weather patterns. You simply cannot invert a thermometer based on high frequency covariance with a distant location.
58. Just saying it strongly doesn’t make it so, Ryan.
60. I agree. I’m just reacting to the comments that assume the contrary was proved. Capisce?
62. Value-less post.
63. No sweat. Will look at it. I still haven’t gotten to your links.
64. You’ve said that before. Saying it now, is no new aha. And also you’ve done a crappy job of proving your points previously. You tend to get “tired” and angry and breathless.
You’ve made your point TCO, it’s a foolish one and an untenable position.
I’m unaware of any proof that the thermal variation was used upside down prior to this, I did have the suspicion that it could happen. This was the proof that it did happen and it happened a lot. I wish you understood but in the meantime, you’ve said your piece.
Jeff ID, #71, thanks for the comments and the link.
Is there a “raw” version of this movie that I can download? (I like to use a viewer where I can rock sections of the video back and forth using arrow keys.)
Anyway, I’m not yet sold on this statement:
Are negative correlations are already present in the covariance matrix of the raw data set?
If so, then they are an indication of real physics.
Understand that a different question that: whether negative correlations being created or enhanced by the Steig methodology. I get the point about artificial negative correlations being induced by “forced correlations”.
If this statement is correct, it needs to be made more forcefully because clearly that is a major flaw in the method.
Thanks,
Carrick
Sorry wrote that in a hurry, I mean to say “I get the point about artificial negative correlations being induced by ‘forced oscillations‘”.
#74 75
Carrick,
Consider the physical meaning of what you are saying. If you have a weather pattern which naturally (assume the PC issue isn’t present) oscillates back and forth and you have thermometers distributed around that region, how do distant inverted correlation thermometers correct the actual thermometers at a particular location. If a weatherman in the west measures -66 deg and one measures -56 in the east, what temperature is it in the West? — The answer is -66 deg and clearly distant stations cannot help unless you’re planning a trip. It’s really just that simple.
Also as another thought problem, consider a situation where the whole continent had warmed equally at every station. Isn’t it possible that high frequency oscillations can still exist even though warming has occurred equally. This method would still invert thermometers and average the data together and you would get an incorrect average temp trend. – I demonstrated this effect in a post some time ago.
You’re making a point that it might possibly be ok to flip temperature upside down to help an algorithm which has no connection to climate whatsoever match the high freq covariance of a distant location and adjust its value. The answer is clearly — of course not. It is an artifact of the method, nothing more.
There is absolutely unequivocally no possible rational by which measured thermal variance can be flipped upside down under any circumstances. The temperature is the temperature as measured by thermometers.
I’m sorry to word it so strongly but it’s a real world we live in, not an expectation maximized one.
==========
Also I unfortunately don’t have a link to the video file directly. WordPress free version doesn’t let me host one.
Jeff, here;s something that’s worth experimenting with. A procedure that’s often used in principal components is “varimax rotation” – this applies an orthogonal transformation (“rotation”) to a truncated subset (say first 3) of eigenvectors (weights) to try to group the weights as much as possible. (“varimax” is a means of doing so). A varimax rotation doesn’t affect regression properties – I did a post on this on CA a while ago.
I looked at this method in connection with a Rob Wilson tree ring study – varimax rotation is big in dendro world – and the impact was to load weights into distinct regions.
My surmise is that it would do the same thing in Antarctica with the Peninsula emerging strongly from the varimax rotation.
My guess is that there may be a fairly pretty result: the varimax rotation might isolate 3 regions, which would then be assigned weights and thereby establish “effective” areas for each area in the Continental average.
Once we figure those out, we could then do a sort of Antarctica pseudo-map dilating the areas according to the effective weights, which would nicely illustrate the result. Such a pseudo-map would probably show the Peninsula as very large and east Antarctica as very small.
Scott Robertson said
June 9, 2009 at 8:54 pm:
Scott, reasons for openess in science include that fact that if you listen carefully to critics and their concerns, you may receive essential information to solidify or correct your arguments. Employing methods that effectively isolate one from collegial review is a hazard to both one’s work and reputation. Relying on “closed source” software is a problem throughout many disciplines. Not only can you not “show the code” to your critics, you cannot audit that code yourself. For instance, we all know what the variance is defined as and the basic equation for calculating one. The code that a piece of software employs may or may not do that reliably. For instance, the June 2008 issue of Computational Statistics and Data Analysis carried an entire series of articles that deal with the remarkable number flaws, errors and inaccuracies that excel and other spreadsheets can throw, if you use the built-in analytical packages. Actually, I quit using excel for statistics years ago, the first time it gave me a negative variance. The entire issue in science comes down to “trust but verify.” Trust your colleagues but doubt their conclusions.
Nearly the entire complaint department in rational blogs like TAV and CA is taken up with the issue that many times the words “trust me” are implicit in AGW arguments, but rarely is heard, “here’s the data, the algorithm and the code we used. Get back to us if you see issues.”
This whole trust/verify issue is also at the root of the critical failure of peer review in the last few decades. When reviewers and authors “trust” each other, they often fail to take the critical step and verify. This is an embarrassment and can’t be handled other than by stonewalling, or by blushing, biting the bullet and admitting the omission.
I suspect that many critics of AGW have actually assumed conspiracy where only slack verification practices are to blame. The problem for others is that public policy, guided by slack practice is a threat of far more immediate potential than the worst AGW scenario. Also, his problem is far from limited to AGW issues. I see it in many, many policy issues ranging from ecology and biology to public health. It is a consequence of poor scientific practice, uneducated (and educable) politicians, and a public that seeks to be thrilled by the media (an a media amenable to anything that boosts readership and “watcher-ship”), and is perhaps so tired of the status quo that they hope for a catastrophe to enliven their days.
Steve McI:
I’m also trying to modify the code to do ICA and projection pursuit for that same reason. Varimax won’t separate all mixed modes.
The idea of an ‘upside down’ thermometer is not totally worthless. In the case of temperatures over the course of a year, northern hemisphere temperatures will be roughly anti-correlated with southern hemisphere temperatures. But in that case there is a clear physical reason why that should be so. In TCO’s example, it is clear that during times of economic stress people are less willing to spend money carelessly and hence Walmart will do relatively well compared to a more upscale competitor.
If there is no independent theory about what the mechanism leading to the anti-correlation is then using such a correlation is pure data mining (in the pejorative sense) or better yet, augury.
#79 In ICA you’ll end up with non-orthogonal axes. When you attempt to reconstruct, wouldn’t it over and underweight certain information based on covariance with essentially the same data?
Perhaps the repeated information just gets reweighed back during reconstruction as the original ICA axes do but I’m not certain. I’m also uncertain it helps to have ICA axes over PCA. It may reveal a more natural oscillatory nature in the pattern but does that really help much?
I need to reread varimax stuff to comment but ICA seems like a natural(some pun intended) fit for the dendros.
#80, of course an upside down thermometer is worthless. I don’t for the life of me understand how people get a bit of anti-covariance mixed up with temperature. If you flip a northern hemisphere thermometer, with an overall warming you get an anti-correlated thermometer showing global cooling — totally and completely without meaning!
Jeff ID:
If you have a west-to-east Rossby-wave flow, and you are looking at two locations separated 1/2 a wavelength out of phase from each other, in general you will have negative correlations between the temperature fluctuations of the two sites.
Further if you have a measurement of temperature fluctuation from its mean at one site, it will allow you to predict the temperature fluctuation at the other site, regardless of whether the two sites are correlated or anti-correlated.
If you trust covariance matrix methods to infill missing data points, the sign of the correction has no impact (by itself) on whether the operation is meaningful or not.
Temperature and “fluctuation from mean” of temperature are different things. Flipping the sign of absolute temperature gives a meaningless physical result. Flipping the sign of a fluctuation is both possible and observed in practice.
I can put together a figure of some of my own data showing there is a wave-like structure in the temperature fluctuation field that includes regions that are negatively correlated with neighboring points, if that would help you accept the reality of such a phenomenon.
I’m not saying that it is appropriate to use a method that has a negative correlation in temperature fluctuations at large ranges in the case of RegEM, just that there is nothing nonsensical about negative correlations in the temperature fluctuations between any two stations.
We aren’t multiplying temperature by a minus sign, but temperature fluctuations about a mean. That is entirely a physically relevant concept.
I’d suggesting checking out one of the free file hosting sites, such as http://box.net/lite
#82 Exactly. I had mentioned that I needed to do some thinking about this, but after reading the B&C references from Steve, the meaning of negative weights is really clear. The negative weight means that during the calibration period, the surface station shows an INVERSE relationship to the satellite temperature (as approximated by the PCs). However, this is physically impossible. It is exactly the same concept as the upside-down-Tiljander in M08.
#81 I really don’t think orthogonality matters. Once you take the first SVD in RegEM, none of the PCs are orthogonal anymore anyway. They never again become orthogonal. While I think it is important to prevent them from interacting, this can be accomplished via scaling. The PCs are just representations of temperature. There’s no physical requirement that they be orthogonal; that’s just a mathematical constraint in order to calculate the SVD.
Carrick, you are misunderstanding what the weights mean. If they meant what you are saying, then that would be fine. However, they do not. The negative weight means they physically act as anti-thermometers during the calibration.
Re 82:
Temperature, in general, is just another observable and can in principle be positively or negatively correlated with another observable, including another temperature. Here’s an example. The temperature of the compressor motor of a refrigerator can vary inversely with the temperature of the interior if you design the rest of the setup and measurement system properly.
I can’t think of any such mechanism for Antarctica but then I am not proposing to use any inverse thermometers. That would be a job for the Steig09 or Mann08 authors.
RyanO: The negative weight means they physically act as anti-thermometers during the calibration.
This is a crucial point of course, and one that I missed.
You can’t have negative temperatures any more than (in real world terms) negative stock values.
How sure are you that the weights really invert temperature rather than fluctuation of temperature (i.e., there is no way to rearrange the terms so this is what is happening)?
Thanks Ryan, I didn’t understand what was missing from my remarks to Carrick who I rarely disagree with.
The weights are like this.
Output= C1* T1 + C2*T2 …..
Five of the C’s are negative.
Updated: I messed up the equation the first time.
#88 The weights are the output of the calibration period. In the calibration period, you have the 1982-2006 station temperatures and the PCs. The PCs are calibrated (regressed) against the station temperatures. The weights are the output of that calibration (regression).
Negative weights, then, mean that when Y (station data) goes UP, x (reconstruction temperature) goes DOWN. In other words, the negative weight means that the station is acting as an anti-thermometer.
While I think all of us can understand that due to noise, there may be times when thermometer A might edge a bit higher at the same time thermometer B edges a bit lower . . . all of us would recognize that it is the result of noise over a short sampling period, not that thermometer A actually reads temperature inversely to thermometer B. But if you end up with negative weights following calibration, then that’s how the thermometer will be used – regardless of whether it makes sense. Imagine setting that as your calibration and then extrapolating it for 30 years (or 2000, as the case may be).
yeah…and if it is noise it will average out over the whole set. God you make me sick, snipe.
#91, I thought that Jeff and Ryan indicated that the negative weights magnitude is such that your noise statement is just that, noise. As in, you ought to disaggregate from your misconceptions, per an occasional poster who sometimes makes a decent comment.
John Pittman, negative weights don’t imply noise. Zero weights imply noise.
Negative weights (it appears) indicates a failed algorithm.
#93 Yes, Ryan stated that previously.The magnitudes of the negative weights separate them from noise. As to noise over the whole set, I do not think that this is what either Steig or Ryan found. So, no matter how I read TCO’s comment, it seems incorrect to what has been stated by Ryan and Jeff.
49: “For the same reason, a re-trended reconstruction with a constant trend at every data point should not show the same (relative) spatial distribution of trend as Steig but it does: https://noconsensus.wordpress.com/2009/04/20/retrended-regem-recon/”
Again, I think you are assuming that I am defending Steig or that pointing out a specific issue with Steig validates general blithe other statements by Ryan or Id. Also, not my comments in that thread on the substance of that post itself.
49: “For the same reason, a valid reconstruction method should be insensitive to dropping the peninsula, but it is not: https://noconsensus.wordpress.com/2009/04/17/no-peninsula-regem/”
Again, you seem to think that either I am defending Steig or that pointing out a specific issue with Steig proves a general point. It does not.
“The covariance matrix should not be dominated by a tiny region within the reconstruction area. This is a recipe for artifacts and spurious correlations.”
I agree that the high weighting of the peninsula is a concern. I was actually well in front of the crowd with my comments on RC expressing concern about the low number of PCs creating smearing or over-representation. But just because the high peninsula representation is a concern and likely a valid one DOES NOT prove the from-hip fired comments that ANY deviation from perfect area weighting is wrong. It is certainly possible to have examples where a small area is a good enough general predictor that it’s weight should be higher valued.
Lurker: Jeff Id has cut one of my comments on your linked messages. He probably thought it was a repeat, but actually I went from link to link and replied to all 3. He cut one. Also, he cut a long funny example of how the neglected middle gets confused by you lot. Hope he enjoys hanging out with Bush and the other RINO maggots who have destroyed the Republican party and abondoned any defense of the free market and invited all the Demo bailouts going on now.
REPLY: I did think it was a duplicate. I hate cutting posts TCO. There was too much cussing in the one you thought was funny – it wasn’t very funny. Bush’s real legacy will be the horror of opening the vault and I’d still take him back over the leftist we’ve got now.
jeff or ryan is there a way to run just the unversed stations and then run this hole mess by leaving out these same ones?
I am just dieing to see the out put in 3pc x 3
would the numbers drop or go up?
just asking.
TCO:
Hm… let me guess.
Sniffing glue again?
#59
1. I’m talking about the pie charts in Jeff’s “Final Straw” post showing station weights, and weight times trend.
2. Just a term I threw out there – borrowing one of your phrases. Just another way of saying “correlated”.
3. A. I should have just said negative as this would be consistent with Steig.
B. Negative to the South pole. IOW, underlying temp realities don’t change from Steig – only
the configuration of the predictor stations.
C. I am guilty of using the terms interchangably myself. PCA uses covariance and TTLS
correlations. This is an interesting case as AVHRR has been reduced to 3 PC’s prior to
RegEm. Even though the reconstruction is done with RegEm, the product is dependant on
AVHRR covariance.
D. Not quite sure of your point. My point was that the reconstructed trend would likely
change if the configuration of predictor stations changed putting the cluster at the south
pole instead of the peninsula. The posts I cited tease out various angles of sensitivity
where there should be none. IOW the correlations/covariances are not right. If the
correlations are not right then the nonsensical inference that a 50 year cooling south
pole is strongly indicative of a 50 year warming Antarctica is very likely an artifact.
4. See 3 A
I don’t think you are defending Steig. I think you are missing the point. If covariance is properly modelled – ie: not forced to stretch and oscilate due to PC truncation, or be dominated by tiny ares of the continent – then we would not likely get station and geographical weightings that make reasonable people go “huh”?
100. I have already criticized low PC count, long long ago. When you criticize reduction of the data, I agree with you. What I DON’T agree with are blithe comments that say negative weights can never apply to a predictor in a regression or that the best guess MUST have absolutely area-weighted configuration. And I also protest the muddling of specific with general that Id is STILL doing here even after I left and came back.
Reply: I’m tired of this TCO. I don’t care that you figured out too few PC’s doesn’t work very well. There have been dozens of points you’ve completely missed always to the same FALSE refrain. It is obvious you use words like regression and predictor with an incomplete knowledge of the meaning. You are assigning several false comments to myself and absolutes well beyond what was ever stated. In the meantime you are thick-headedly missing the single biggest problem with the reconstruction – upside down temperature data are nonsensical.
This does NOT mean that you cannot have a negative weight in a regression, it means you cannot have a strong negative weight in a weighted average of thermometer data – Jeeez!.
————————–
Just in case there is still some confusion for others, here’s an example: You have two thermometers, on the south pole and one on the North pole and global warming +0.2C/Decade is occurring equally at both poles so both thermometers show equal temperature rise.
If you do a weighted average you get:
output= C1*T1 + C2*T2
output= (0.5) Tsouth + (0.5) Tnorth :half the weight to each
And the trend is 0.2C, equal to the average of both stations.
After some time you run RegEM on the data against a noisy satellite set which accidentally favors the north pole. The weights you get are equal but opposite (to keep it simple)
output = (-0.6) Tsouth + (0.6) Tnorth
The output has a zero trend with slightly amplified thermometer readings from your algorithm trying to expectation match the satellite noise.
So, is global warming is solved? Or does it really make sense to read the south pole thermometer upside down?
———
The point of this is that inverting any thermometer makes zero sense in a physical world.
#59
“1. I’m talking about the pie charts in Jeff’s “Final Straw” post showing station weights, and weight times trend.”
I know which pie chart you are talking about. Reread my comment. The question is which algorithm in your hypothetical example, the Steig one, or the class of all correlation matching schemes?
“2. Just a term I threw out there – borrowing one of your phrases. Just another way of saying “correlated”.”
Did you attach a meaning to the term or were you literally throwing the term out there as a fill word? I want to know what you mean, since my own usage of the term was in a different context and inexact itself. I’m not gigging terminology…just to communicate, what do you mean? For instance correlated to what? And by pattern, do you include for instance wave patterns?
“3. A. I should have just said negative as this would be consistent with Steig.”
Thanks for disaggregating and clarifying. In response, on substance, negative predictors are fine in terms of efficacy. I know of no stats method (maybe you can point me to the right chapter in the 1979 Box Hunter Hunter edition?) which says to on top of a given algorithm apply a filter to toss the negative predictors. You run a risk this way (and Steve McI has fallen prey to it in other contexts, for instance the Hansen adjustments) of biasing the output. Note that this is the sort of thing we might gig Mann or the like for if it helped their cause.
“B. Negative to the South pole. IOW, underlying temp realities don’t change from Steig – only
the configuration of the predictor stations.”
But the Steig method (which I admit to learning about here) predicts the entire contintent by using a calibration during the overlap of sattelite and stations. Do you not understand that? The key would be what regions in Antarctica correlate to the South Pole, what to the peninsula, and what to which mixture. So if the two (SP and Penin) have some differing behaviour than it is beneficial (and actually neither strict correlation nor inverse correlation of the two will help, since that would be a degeneracy and add no more predictors for use…they need some independance to have use.)
“C. I am guilty of using the terms interchangably myself. PCA uses covariance and TTLS
correlations. This is an interesting case as AVHRR has been reduced to 3 PC’s prior to
RegEm. Even though the reconstruction is done with RegEm, the product is dependant on
AVHRR covariance.”
No sweat. Just as long as we understand that we are using the terms colloquially and that we basically mean how x follows y.
“D. Not quite sure of your point. My point was that the reconstructed trend would likely
change if the configuration of predictor stations changed putting the cluster at the south
pole instead of the peninsula. The posts I cited tease out various angles of sensitivity
where there should be none. IOW the correlations/covariances are not right. If the
correlations are not right then the nonsensical inference that a 50 year cooling south
pole is strongly indicative of a 50 year warming Antarctica is very likely an artifact.”
A good algorithm should not change markedly for these kinds of issues. I get where you are coming from. I think if you simplify the problem too much, it’s impossible for the algorithm to give the same answer, for instance if you literally only had two sites, since there is nothing else for the algorithm to fasten on to. The point about multiplictiy was more of a second order effect, but I understand your first order point.
“4. See 3 A”
Not adequate. You have not proven a thing, just asserted it. In addition, even what you assert is not well stated as a definitive statement. Please do so. (I’m NOT trying to tie you up by the way. I really do want you to clarify your thoughts, even for yourself.)
“I don’t think you are defending Steig. I think you are missing the point. If covariance is properly modelled – ie: not forced to stretch and oscilate due to PC truncation, or be dominated by tiny ares of the continent – then we would not likely get station and geographical weightings that make reasonable people go “huh”?”
We’re actually in agreement and have been for a while. See my old, old comments on RC about low PC numbers and smearing concerns. However, when I hear the GENERAL comments from Id (he’s worse than Ryan) on how any deviation from area-based must be wrong, than I react to it. It hasn’t been proven. And particular failings of the Steig algorithm do not prove a general case. Really there are different issues to examine, when you say, I can never use a negative stock as a predictor of the economy (have a negative weighting on a station in Antarctica) versus when you say that Steig did not have enough PCs to do what they wanted to do properly.
RyanO #90:
Negative weights mean that when Y (station data) goes UP, x (reconstruction temperature) goes DOWN assuming all other station data remains the same. But all other station data does not generally remain the same. The stations are correlated, so a negative weight on one station may in practice be offset by a positive weight on a correlated station elsewhere.
Once you put a thermometer Y into a network of thermometers Y1, Y2, … via regression you need to look at the joint distribution of all thermometer temperatures to understand how the regression behaves, not just the marginal distribution of Y’s temperature.
The negative weights are only crazy if, in practice, the regression predicts a fall in output temperature when in fact the true average surface temperature has increased.
As I mentioned upthread, you could resolve this by constraining the weights to be non-negative. Then the regression problem would become a quadratic programming problem. You’d probably get an ever-so-slightly worse fit, but the overall behavior of the regression would not change much.
As Australians would say, this negative weight thing is a bit of a “Furphy”.
Jeff O and Jeff ID, I made the following query earlier:
It turns out for your formula you can:
Tout = c1 T1 + c2 T2 + c3 T3 + … + cn Tn
Taking the average over the period of observation (and using to denote this operation):
= c1 + c2 + c3 + … + cn
Subtracting the second equation from the first gives:
Tout = + c1 (T1 – ) + c2 (T2 – ) + c3 (T3 – ) + … + cn (Tn – )
so unless I’ve made a glaringly stupid mistake in between period of intense analysis of my own data, it does appear that “there is a way to rearrange the terms so that the weights invert temperature fluctuations rather than absolute temperature”.
Well, crap. Forgot about the html again, so here goes a second try (I was using < T > to denote the average):
——–
It turns out for your formula you can:
Tout = c1 T1 + c2 T2 + c3 T3 + … + cn Tn
Taking the average over the period of observation (and using E[T} to denote this operation on T):
E[Tout] = c1 E[T1] + c2 E[T2] + c3 E[T3] + … + cn E[Tn]
Subtracting the second equation from the first gives:
Tout = E[Tout] + c1 (T1 – E[Tout]) + c2 (T2 – E[T2] ) + c3 (T3 – E[T3] ) + … + cn (Tn – E[Tn] )
so unless I’ve made a glaringly stupid mistake in between period of intense analysis of my own data, it does appear that “there is a way to rearrange the terms so that the weights invert temperature fluctuations rather than absolute temperature”.
#104, I can’t follow your nomenclature. Cn is a vector of constants Tn is a matrix.
Are you saying that if all temperatures are inverted you could invert all the C’s and get a positive output?
re 101:
I don’t want to put myself into the same category as TCO by banging about my pet idea but I think that when there is correlation there is some possibility of a real effect being present.
Your position is correct from the standpoint of measuring temperature across a homogeneous landscape and/or with a large number of sensors but that is not the case in Antarctica and if your work gets some notice you will have to defend it against criticisms like this one.
You didn’t like my refrigerator example so I’ll contrive a meteorological one.
Let’s say you have a thermometer near the shore where there is a current that sometimes brings warm water in and sometimes the current meanders off and cold water predominates. Also, suppose that there is another thermometer further inland and behind a ridge but above some deep valleys filled with very cold air. I can imagine a configuration of water and land such that when the water is warm the shore thermometer warms up, since it is close to the water. And because the warm water will heat the air above it, it will rise and there will be a wind created. If the wind passes across the ridge it will create a low pressure behind the ridge which will in turn cause colder air from the valleys to rise up and cool off the thermometer.
Ok, this is a very contrived example but the world is a complicated place and you can’t say that there would never be any possibility of something similar happening somewhere. Of course, because such a scenario is so unlikely it would be the responsibility of people who like inverse thermometers to justify their beliefs.
I think this discussion about negative coefficients detracts from the more important point that the peninsula had disproportionate weighting.
Hm?
c1, c2, c3,…, cn are a sequence of numbers, right? (Given by Table I of “the final straw” post).
In my notation,
T1, T2, T3, …, Tn
referred to the values at stations 1, 2, 3,…, n at some given time “t”, and Tout is the “outputted” value associated with that particular time “t”.
Here is hopefully a somewhat cleaner exposition with new notation:
Write k = (t – tstart)/DT +1, where tstart is the start time, DT is sampling interval, and k = 1, …, K with K = (tend-start)/DT.
Then,
Tout(k) = c1 T1(k) + c2 T2(k) + c3 T3(k) + … + cn Tn(k).
Making the definition
Mean(T) = (T(1) + T(2) + T(3) + … T(K))/K
we have
Mean(Tout) = c1 Mean(T1) + c2 Mean(T2) + c3 Mean(T3) + … + cn Mean(Tn)
and it follows,
T(k) = Mean(Tout) + c1 (T1(k) – Mean(T1)) + c2 (T2(k) – Mean(T2)) + c3 (T3(k) – Mean(T3)) + … + cn (Tn(k) – Mean(Tn))
[In a least-square fit formulation, my predilection would be to leave the optimization function in this form, since it reduces numerical noise when you have a non-positive definition series, and also to treat Mean(Tout) as an additional parameter of the fit.]
Cast this way, the c1, c2,… cn can be thought of as more or less usual correlation coefficients, and if you have one region which gets warmer when another gets colder, that gets reflected in the sign.
So…
1) Does this derivation make sense to you?
2) Do you agree that there is nothing unphysical about multiplying a fluctuation in temperature by a negative coefficient?
Jonathan Baxter #102
The confusion on this issue is absolutely baffling to me. I don’t understand how people can complicate the averaging of temperature data.
The first point I need to make is that you are assuming that yes we’ve created one problem but somewhere else this problem will be magically fixed by math. If you are making the assumption that there is a magic correction based on correlation it needs to be proven that correlation and temperature trend are magically related linearly to the correct extent, – they are not from my previous work but nobody has made an effort to prove that in Steig et al.
The regression ignores the unphysicality of a negative temperature in favor of tiny wiggle matching. This is a false result.
I have a problem with this statement also, if the regression predicts a ‘different than true’ temperature it has also failed. Flipped thermometers guarantee a different than true answer.
I agree with this, it’s similar to a lens design iteration we do in my work.
It is way way beyond me how people can think in any physical world that it might be even slightly reasonable to invert a thermometers variance from a seasonal mean in an effort to calculate temperature of an area.
In the comments above (other than Carrick), the justification for this insanity is simply due to the better wiggle match you receive by regression. Those who are confused are conflating the result of a mathematical expectation maximization algorithm with the physical reality of temperature.
The discussion is so nuts that I can counter it by saying,
Your argument fails to recognize that temperature is positively related to temperature with a multiplier of about 1.
It’s really nuts! Claiming a magic correction for an obvious error through unspecified covariance is equally off the wall but it is exactly what I expect Steig et al will claim with the same kind of hand waving.
==============
I can design a perfect non-imaging, refractive or reflective lens with a deterministic matrix solution, as long as I have negative light. Unfortunately, negative light (in a non-phase optic) is a physical non-reality. RegEM would not recognize that and would actually converge to the deterministic solution resulting in a non-physical lens.
Jonathon Baxter:
They are separate issues and both need to be explored.
Even if it isn’t unphysical to have negative coefficients, that doesn’t mean there aren’t problems when this occurs, and that this is something that needs to be explored to develop a fully robust temperature reconstruction method.
Jeff ID:
Let’s not get frustrated over this.
This is an important but, it seems, subtle issue.
The devil hides in the details in statistical methods.
For all those who think that negative weights are physically possible (or are of minimal importance), let’s consider an example.
Let’s assume that you are trying to develop a stock market index. You have a period of complete data (1982-2006) and a period where, for the entire market, you have data on only 42 stocks that is approximately 40% complete.
You approximate the complete period by 3 functions. We’ll call them PCs. Each function is a time series of coefficients and an index that contains weights for individual stocks. To get a particular stock price at time t, you take the time t coefficients for each PC, multiply it by the appropriate weight for each PC, and sum them together.
To backtrack to 1957, you regress the 3 functions on your 42 predictor stocks. Let’s say that while the stock market was going up, 5 of the predictors stocks just happened to have short periods where they went down. The regression will give those stocks negative weights.
Now extrapolate back to 1957. Anyone think that those negatively weighted stocks will give you an accurate extrapolation given that the index by definition goes up whenever any stock goes up? Anyone think that this negative relationship holds for all times and is usable for prediction? What sense does it make that when a particular stock goes down the index goes up?
The answer is that it doesn’t. Whenever a stock goes up, it contributes a positive number to the index. It is by definition incorrect for the opposite relationship to occur.
Now how does this happen? A couple of ways. First, the stock with a negative weight may have few points. This leads to a spurious correlation (or, if you’re a climate scientist, a “teleconnection”) because you have few samples. Yep, during that period the stock may have gone down while the rest of the market went up, but this is hardly a good predictor of market performance years in the past or years in the future. But because people don’t understand that correlation does not imply causation, people make bad investments based on these kinds of correlations all the time.
The other way it could happen is that you approximated the market with too few PCs. This forces negative relationships because the number of PCs you chose does not have enough resolution to properly model the market. So to make the numbers work in the period where the data is complete, the math forces individual stocks to have unphysical behavior with respect to the market. Even if the overall market trend matches, when you multiply the PC coefficients by the weights for a particular stock, you get the wrong answer for the price of that stock. The math compensates by giving a wrong answer to some other stock . . . and so on. While this may work in the period where your data is complete, using these wrong answers to extrapolate behavior back in time by 30 years is extremely unlikely to give you right right answer for the index.
So yep – in the period where you have complete data – negative weights may be viewed as compensating for some other mistake. Carrick, Jonathan, et al., you could look at it that way during that period. However, the meaning of the negative weights changes when you do the extrapolation. You don’t have all of those other stocks to compensate for the error. All you have are the wrong answers from the first phase – which is the calibration period. During the reconstruction period, there aren’t any other stocks available to compensate. If there were, you wouldn’t have to do the reconstruction in the first place. So those errors, rather than getting compensated for by other errors, are used as fact to reconstruct the market.
The negative weights are, in my opinion, one of (if not the) most important diagnostic for the reconstruction. Strongly negative weights mean at least one of the two effects described above has occurred. You then need to look at the particular stock to see if the negative weight is due to spurious correlation (from having few data points) or, if it has a long history of points, then the answer is likely due to your choice of PCs to approximate the market.
I do not hesitate for a moment in saying that if my reconstruction has any strongly negative weights for long record-length stations then my reconstruction is crap. If it has strongly negative weights for short record-length stations, then those stations should be removed from the analysis.
Jeff’s weights are, in my opinion, the most important single diagnostic for a reconstruction.
#108, The equation seems fine, but the mean’s are zero’s because the anomaly data is centered.
Cast this way, the c1, c2,… cn can be thought of as more or less usual correlation coefficients, and if you have one region which gets warmer when another gets colder, that gets reflected in the sign.
I don’t agree that the weighting changes to negative with temperature trend. The weighting will change according to covariance in a regression. The trend in this case is a very minor factor, not that my answer will change if it wasn’t.
If you have 2 thermometers in a room one by the inner wall T1 and another on the opposite side by the window T2. You measure every second from noon to midnight to 0.00001 noiseless degrees. The window thermometer will register a stronger cooling trend than the wall. The best average for the anomaly in room temp is (first subtract the mean) Taverage= 0.5 T1 + 0.5 T2 –
To state that it might possibly be 0.5 T1 – 0.1 T2 is non-physical. You’ve then flipped the thermometer trend upside down creating a false warming trend by the window and averaged it against T1. In our imagined regression, the claim is that we have another temperature T3 which will somehow draw it all together. If T3 is negative trend and is measured at the middle of the room, a multivariate regression of T2 and T1 can reveal a negative coefficient for one or the other. However, negative coefficients are still non-physical. A plot would show warming on one side of the room and cooling on the other.
My guess though is that both thermometers in this example would naturally receive positive coefficients.
The result of this simple example is that unfettered multivariate regression may not be the best method for temperature distribution in a high noise situation. In cases where there are negatives that you know should be positive, your spatial distribution has been distorted.
—
Now in the case at hand we have very high noise data reduced to 3 pc’s. This data doesn’t match trends of 0.1C/decade at all, it matches high frequency covariance. It’s important to note that inverse covariance is nearly guaranteed by PCA with 3 PC’s. This will create a flip the thermometer tendency – equivalent to flip the thermometer data at the window and add it too the thermometer at the wall to figure out the center of the room.
–not good even if your regression attempts to modify the wall temp to compensate, the shape of the daily curve will change and the regression is no longer a physical result.
RE Jeff Id #109:
The regression will always predict something other than the truth because it is an approximation.
Think of it this way. The definition of average Antarctic temperature is the average (integral) over the continent of the temperature at each point. An (ideal) thermometer only measures the temperature inside the instrument. Not the temperature outside, not 5 meters nor 5 miles away. So the only truly kosher way to measure the average temperature of Antarctica is to have a thermometer at every point and average the readings with a weight of 1 on each. Even then, two adjacent and perfectly correlated thermometers could have weights of +2 and -1 respectively without affecting the average.
Unfortunately we don’t live in an ideal world, so we try and come up with a “best approximation” to the true average Antarctic temperature from the few thermometers we have by averaging them together with nonuniform weights. Given that even in the ideal case a thermometer could get a negative weight without hurting the average, it should be even less surprising that it happens in the approximation.
Now it may or may not be the case that the negative weights are unphysical. I can imagine (as can other commenters) weird weather phenomena in which a particular thermometer in practice is usually negatively correlated with average temperature. In that case a negative weight would be perfectly physical. Of course, if the temperature only changed at the location of that thermometer and nowhere else, the negative weight would result in that thermometer making an incorrect contribution to the average. But if such a situation occurred often enough in the data, the regression would not set the weight to be negative in the first place.
So (to me), the correct interpretation of a (physically reasonable) negative thermometer weight is not “this thermometer locally measures negative temperature”, but “the temperature local to this thermometer is negatively correlated with the average”.
The other situation in which negative weights arise is if the solution is underdetermined, as in the ideal case with +2 and -1 weights for adjacent perfectly correlated thermometers. In that case, if negative weights offend you, you can use quadratic programming to constrain them to be all positive without hurting the reconstruction skill.
That would actually be an interesting exercise. If constraining all the weights to be positive does not damage the reconstruction skill dramatically, that suggests the negative weights we are seeing are simply mathematical artefacts of an underconstrained problem. OTOH, if such constraints do hurt reconstruction skill then those negative weights represent real physical phenomena (or at least real problems in the data).
#113 The weighting definitely doesn’t change to negative with a negative temperature trend. If the weighting is negative, then a positive trend for the predictor equals a negative trend for the reconstruction, and vice-versa. You’re 100% right that these are regression coefficients.
This is not correct, there are many ways to average temperatures. Kriging (least squares polynomial surface fit) and closest station area average are good examples of improved methods. Another method is RegEM Ryan style, the thing that makes it interesting is that the satellite data can determine the best area of influence based on covariance with the data.
A negative area of influence is absolute nonsense, even if it helps color in some short term variation.
This is not a long drawn out complex or subtle thing, it’s hit you in the head with a brick simple. You are confusing the subtleties of regression with the reality of a thermometer.
#114 Please read my example. What you are saying has validity only in the calibration period because in the calibration period you have complete data and you can compensate for that error. It does not have validity in the reconstruction period, because there is no data with which to compensate.
Everyone needs to keep in mind that for each thermometer, when that thermometer goes UP, it contributes POSITIVELY to the trend. You may be able to find periods where the average goes down while a particular thermometer goes up, but that does not mean you can use that to predict temperatures.
This idea of benchmarking the thermometers vs. an average is exactly analogous to using too few PCs. An average has very little resolution, right? It’s just an average. It would be like averaging the pixel brightness on your computer screen. Yep, that’s the average, but the resolution is crap. And yes, some real pixels will display an inverse behavior with respect to the average if your sampling is over short enough periods. That doesn’t mean the negative relationship is therefore justified. What it means is that your choice to describe the computer screen in terms of the average doesn’t have enough resolution to describe a meaningful relationship for the individual pixels. That makes a prediction of the future “average” of your computer screen using individual pixels extremely suspect.
No, because the points are not independent of each other. Antarctica shows a good degree of spatial and temporal autocorrelation, meaning that the points are definitely related to each other. Because of that, you can use partial information (assuming you have enough partial information to capture each of the important modes of behavior) to predict the entire continent.
If the points were truly independent (i.e., no autocorrelation), then your statement is actually correct.
#117,
That doesn’t mean the negative relationship is therefore justified. What it means is that your choice to describe the computer screen in terms of the average doesn’t have enough resolution to describe a meaningful relationship for the individual pixels.
This is also an important point. The trend in the sample range is very small and has no real effect on the calibration. If we had a hundred years of data, the currently flipped stations would probably be dominated by long duration trends and be flipped positive. (Assuming we don’t create strong oscillations with too few PC’s)
RE #118, #116: I meant only that in the absence of extra assumptions (like spatial correlation), the only way to measure the average temperature is with a thermometer at each point. Of course there is spatial correlation so you can make do with a lot fewer thermometers. But even in the case of a thermometer at each point it is ok to have negative weights if all you are using the thermometers for is predicting the average temperature.
Ok, I think I am starting to see where you are coming from. Are you saying the problem is not that the regression produces negative weights, but how those negative weights contribute to the trend calculation? If so, can you tell me how is the trend calculated from the regression coefficients?
#120,
The regression is getting a better fit by using a negative weight and is working as expected to calculate weights Cn
The trend is calculated:
Tout(t) = c1 * T1 (t) + c2 * T2 (t) …..
Where T1 is the measured data at each (1 – n)temperature station and Tout is the final result.
Trend is least squares fit.
—
How is it possible to take an accurate average temperature of an area when you subtract the measured temperature from a thermometer in that area? All frequencies in the data get flipped simultaneously with the reading. It is no longer temperature information, but systematic noise in the EM regression.
On a slighlty different note.
If the thermometer noise is anti correlated with a region, the thermometer isn’t related to that region (not nearby) and may have the same long term trend as the distant anti-correlated region – before it’s flipped. Nothing constrains the EM in steig et al case except high freq. covariance.
It’s just an extension from complaining about negative weightings as unphysical to complain about any weightings other than normalizing within sample (area weighting for climate, demographic normalization for a political poll or market research) as unphysical. This is because in the limit of oversampling, of having a thermometer on every spot, that you don’t need any weighting scheme at all. However, in low sampling regimes, correlation-weighting based on calibration period makes sense. The basic point is that if you hate “unphysical negative weights”, you might as well hate all weights, you might as well hate multiple regression.
Jonathan:
If a thermometer goes up, then the contribution of that thermometer to the average is POSITIVE. It is never ever ever ever negative. Even if the average goes down, that particular thermometer still has a POSITIVE contribution.
The concept of negative weights means that when the thermometer goes UP, its contribution to the average temperature is NEGATIVE.
Please note that there is a huge mathematical difference between a negative weight – which inverts the contribution of the thermometer – and the ability for the average to move in the opposite direction of any individual thermometer. You cannot equate the two concepts, because they are not the same thing at all.
If the math behind the reconstruction were correct, then all stations would have a positive or near-zero weight. Those weights may be different, which means that different thermometers may dominate at particular times, but the weights must all be positive. This doesn’t dictate that the trend is positive. If a thermometer with a positive weight shows a negative trend, then it will contribute a negative trend to the average – and the average can trend down.
But the idea of a negative weight means that when region A increases in temperature, it contributes a NEGATIVE temperature to the average.
This is physically false.
126 (Ryan):
A. Realize that Baxter is the real deal. A man with more stats expertise than you.
B. You still have not addressed the trivial example of degeneracy where we get the same end answer when giving one of a pair of sensors a negative one factor and the other a positive two factor. Please try to stop and address this stuff.
TCO, first, you have no idea how much stats expertise I have. Second, this isn’t even a statistics question. This is a calibration question. Third, I have addressed (B) many times. In fact, (B) is the only thing I have addressed today.
Ryan:
I don’t know. I’m guessing. Based on what I see from you and what I see from him, plus his background. Best estimate with limited data.
I missed your engagement on the degeneracy thing. I read a lot of other stuff from you, but never saw that. Are you really sure that you did address it? Perhaps you think you did, but didn’t. If you did, please send me to the best post (cite the tread, post number and paragraph. Thanks.) Perhaps, you really did discuss that and I missed it. But I suspect instead that you just made some other arguments while not addressing this. For one thing, Baxter, who is a [snip], repeated his comments as well. Not just “stupid TCO”.
One last attempt to answer the analogies to oscillatory behavior.
Remember that the reconstruction, mathematically, is the sum of each PC multiplied by its respective shape. The shape describes the covariance between points. If oscillatory behavior exists, that behavior is already captured by the shape. The sum of all of the shapes times their coefficients (the PCs) yields the temperatures at each grid cell.
The negative weights are not related to oscillatory behavior. The negative weights mean either that the record length is short enough to cause spurious correlation during the calibration or that an insufficient number of PCs were included, meaning that the area where the negative weighting occurs is incompletely resolved.
There are no negative thermometers in Antarctica. A negative weight physically means a negative thermometer. It means you have done something wrong, and in order to compensate during the calibration period, the math inverts one or more of your thermometers. You cannot then take that erroneous result and perform a reconstruction.
Ryan O:
Suppose you have two thermometers, one on the West coast of Antarctica and one on the East coast (actually, looking from the South pole the whole coast is the North coast but you know what I mean). Suppose the West coast thermometer represents 1/3 of the continent and the East coast thermometer represents 2/3. And suppose that by some strange quirk of Antarctic weather whenever the the West cools the East warms by the same amount, and vice-versa (assume we have already subtracted the average (PC0 if you like) so the thermometers are reading differences from the mean).
Now suppose I tell you that you have to approximate the average temperature over Antarctica using only the West-coast thermometer. So T = c * TW + K where c is some weight. TW is the temperature reading from the West coast thermometer, and K is the average Antarctic temperature (averaged over time). Given the strange quirk of Antarctic weather I described, c = -1/3 is the correct value, ie negative.
From this I suspect your counter to my example would be something like “But you only added PC0. PC1 would capture the weird anticorrelation between the West and East coasts so including that would make all coefficients positive”. Am I right?
Jonathan,
Your assumption is that the west side of Antarctica is the perfect opposite of the east side of Antarctica. If this were the case I would be forced to concede that negative temperature by an odd twist of fate from god to smite my rational understanding of thermometers, has merit and global warming in the antarctic would be a physical impossibility as the long term trend invalidates the assumption.
However, as an engineer in the real world any oscillatory nature is based on weather patterns. We need to understand that inverting the temperature is not a simple explanatory improvement but rather a noise based non-sequitur.
Jeff Id, my example was deliberately simplified to illustrate the point, which is that if you have correlation between sensors, interpreting their weights in a linear regression is problematic. Two sensors, perfectly (anti) correlated is the simplest example of correlation, but you can imagine more complex correlations amongst 30 or 40 Antarctic stations might yield similar interpretational difficulties. No smiting from the almighty required.
(I actually deal with this issue in my day job; sometimes we add add a new “feature” (sensor) to our system certain that it will be positively correlated with some category we’re trying to predict, only to find that the algorithm sets its coefficient to a negative value. Invariably in turns out that some other existing feature is highly correlated with the new feature and has already slurped up all the weight, so that in the context of the existing features, the new feature is actually a mild negative predictor).
128 – Sorry – probably a dumb question as I’ve not followed the whole thread – but what weight would you assign to TE?
Jonathon, I’m looking at your example and I’m not seeing that it is the same as the problem being discussed herein. Maybe I’m just confused at what you are doing, but it seems like you have a priori knowledge of the average temperature K in your example, which is not true in this problem, and what is T?
Thanks,
Mark
#130, I fully expected you to come back with this answer, but I try to be honest. I think the rest of my comment gave it a better context.
Upside down temperature is still a nonsensical match to data. Thermometers have a known relationship to temperature which cannot be ignored or inverted. Imperfect inverse covariance does not prove an inverse relationship to temperature and therefore is not a rational for a negative weight and in this case (Steig et al) guarantee a faulty result.
Imagine a situation where a west thermometer and an east thermometer have perfect inverse covariance on a short term scale and both have an equal positive trend. In your example above you would still get a negative C weighting on one yet the average trend would be incorrect. – this is a much closer situation to reality and it would require an act of god to change the nature of thermometers.
This is the reason inverted temperature is not a physical value.
You don’t have that data, i.e., his problem is “estimate K with only TW” which is why the way the problem is worded is confusing to me. Assume somebody broke the thermometer use in the east and you hired some cleaning company to come in an clean up the mercury.
I have more to say/ask, but I’ll wait for an answer to my previous question.
Mark
Jonathan,
The behavior you describe is already accounted for in the shape of the EOF. The shape has both positive and negative coefficients. That’s where the inverse behavior is captured. You don’t get to re-invert that by giving a negative weight to the thermometer.
In the context of the reconstruction, a negative weight means that the temperature anomaly at the location of the thermometer is INVERSE to what the thermometer actually reads at that point. So if you have a thermometer at the south pole and it is given a negative weight in the reconstruction, then when that thermometer at that location goes UP, the reconstruction will have the temperature at the same point go DOWN. The average temperature across the continent may (and I emphasize may) still be accurate, but the temperature at the location is inverted.
On your final paragraph – that this means there is some other mode that was not captured that would cause the reconstruction temperature to go the right way – you’re exactly right.
Jeff ID:
I’m still struggling to understand the argument. Admittedly my brain is split about a dozenty different directions ATM, and probably this is contributing to my confusion….
It seems to me there are really two contributions to the covariance matrix, a long-time scale covariance associated with warming/cooling of the continent and a short-time scale covariance associated with weather fronts etc.
It may be what is happening is that the short-time-scale met fluctuates is overwhelming the long-term temperature trends.
It would be interesting to see what happens if you e.g. low-pass filter the signal (and appropriately decimate it to remove the correlations you have introduced by doing so, e.g., a 1/month low-pass filter would need to retain at least 2-points per month according to the Nyquist Sampling Theorem), and then fit this data with your TTLSF algorithm.
Would the data retain the negative coefficients when you did this?
Thanks for you and Ryon O’s patience on this. Us data geeks love to beat data until it stops twitching before moving on.
Think of the problem this way, Jonathan.
You have a fleet of 5509 cars. Each gets a certain MPG. You want to use certain ones to predict MPG during periods where you have incomplete information about speed and miles driven.
Let’s say that you do PCA on the fleet. You select 3 EOFs to retain. You then regress the EOFs versus the individual cars, so that you can predict the MPG based on incomplete information. The outcome of the regression gives negative weights to some of the cars – i.e., they display a negative MPG. You then select a few of these negatively weighted cars, along with some positively weighted ones, to perform a reconstruction.
Do you think your answer will be right?
Or do you think the negative MPGs are a mathematical artifact based on not including enough information (EOFs) such that the regression will yield physically valid results?
Now during the calibration period – where you do the regression – you can certainly argue that the negative MPG cars will be counterbalanced by excessively positive MPG cars. This may be true. But when you do your reconstruction, do you know that the cars you are using for your prediction (because you only have 42 of them) display the same balance shown out of the field of 5509?
134 – Sorry Mark – lack of clarity on my part. I should have phrased the question “Now assume the other case ie. we now only have TE – what weight would you assign to that?”
Nice example Ryan,
For those who’ve missed my point the argument that they WILL be counterbalanced misses the fact that the 5509 series data in our case is ungodly noisy while the trend signal is very very small and my opinion is that we don’t have a hope in heck for RegEM to pull off a counterbalance. Ryan’s MAY is correct. Ryan’s satellite data correction had a strong influence on AVHRR trend for which you would expect to see a difference in weightings however the net change was very small.
The trend is almost ignored in this noise level – until the final result.
^Curious: Gotcha.
^Carrick: Mine, too.
I think, btw, without any more information than I have at the moment, Jonathon and Ryan/Jeff are arguing different points.
Oh, one quick question for either Jeff or Ryan: are the temperature weightings for the anomalies, or for the temperatures themselves?
Mark
RE #132: RE Mark T #132: T is the average temperature of Antarctica at some point in time. K is thrown in just so that we can assume that the thermometers are measuring deviation from the mean (ie temperature anomaly).
RE #138: +1/3 (East is the opposite predictor to West)
RE #137:
Your argument applies regardless of whether the coefficients on the 42 are negative or not. That is, even if all the coefficients are positive, how do you know the selected cars model the 5509 well? I agree that forcing physical constraints (positive MPG) can help reduce the degrees of freedom and potential for overfitting, but not by that much. Seems to me that if your 42 are not representative of the 5509 allowing full freedom on the coefficients, they are still not going to be representative if those coefficients are restricted to the positive orthant.
In fact, as I understand your argument, you are claiming the negative coefficients are a result of underfitting rather than overfitting, so restricting the coefficients to the positive orthant will only make things worse.
I can buy that negative coefficients are an indicator of underfitting in this case (although it would help me if you could explain it in more detail). What I am trying to argue is that negative coefficients can also be nothing more than a result of dependency amongst the input variables, which is quite benign.
140 – 🙂
141 – thanks re: +1/3. If your example had started with the second case of having TE rather than TW, what value would you have assigned to c?
+1/3
Ryan (129):
I’m not sure if this post was a response to me, but if so all I asked you to do was to point back to your previous lucid explanation of how degeneracy is handled (the whole 2×1-x2=x1 if x1=x2). You were exasperated that I had not noticed your great explanation before, so all I asked for was a direct citation to the silver bullet. You don’t give it and instead give a new rephrasing. Furthermore, given that Baxter (who is quite good) is still disagreeing, you sure have not stated your case well or definitively. Let’s dig into this thing a little more. Let’s nuke it out, baby.
On the content:
You do a poor job with comments like multiplying a shape times a PC. Also, none of it addresses the quite CLEAR example of degeneracy that Baxter, Carrick and I have brought up. Also, you don’t show why your description (regardless of its poor explication) forces all station contributions to be positive. That we can’t have some negative station weights coming through the PCA itself and the weighting of the 3 PCs. I mean if you can ACCEPT that you might have less or more weighting than a perfect equality, why is it so hard to understand some factors being negative? Heck it’s as easy as saying the penin (or some other region) has a certain amount of influence…and that the math made a few of the stations too high and compensated with the others low (to the extent of negativeness). And this was the optimum predictor solution.
143 – Thanks – sorry if this is completely dim, but please can you explain for this second case why +1/3? and whether or not there are any other values that could have been chosen and what their sign would have to be?
Jonathon, the way I read what Ryan and Jeff are saying is that the negative coefficients are a warning sign for the methodolgy being applied and that restricting a flawed method to a positive orthant would not be a reasonable remedy.
Ryan, gave a helpful and instructive example with MPG which would appear to be applicable to the temperatures. Perhaps you could give a counter example to show how the benign dependency of variables will give negative coefficients in a valid method.
RE #145: Ryan’s MPG example essentially says the following: if you represent 5509 cars with just 42, how can you be confident that the 5509 are well modeled by the 42? I believe Ryan is arguing that if some of those 42 have negative coeffcients then you know the model is poor, because negative coefficients are, in a sense, unphysical. My claim is that’s not generally the case. That is, negative (unphysical) coefficients by themselves do not necessarily mean that the model is poor (you need more: you need to show that in your sample of 42 there are insufficient counterbalancing excessive positive coefficients).
NRyan also seems to be making a further argument that negative coefficients indicate underfitting in the Antarctic case. I don’t fully understand that argument so I have asked for clarification.
RE #144:
TW represents 1/3 of the continent, TE represents 2/3, so the average temperature anomaly T = 1/3 TW + 2/3 TE. TW is anti-correlated with TE, so if TW increases by a degree, TE decreases by a degree and vice versa. Thus TW = -TE and so (replacing TE by -TW) we have T = 1/3 TW – 2/3 TW = -1/3 TW which gives c = -1/3. Alternatively, replacing TW by -TE we get T = -1/3 TE + 2/3 TE = +1/3 TE, or c=+1/3.
They are the only solutions writing T as a function of TE or TW alone.
I apparently can’t make these points enough.
First, I’m surprised at the confusion. There is a big difference between a negative covariance and flipping a thermometer for averaging.
Second, Thermometers cannot ever be allowed to contribute negatively to the reconstruction. Negative trends are fine, but if a thermometer measures 6 degrees anomaly it cannot be changed to minus 6. Negative covariance is not a good excuse. The fact that the math iterates to a better solution is a non-physical result. Nothing more. The hypothesis that it somehow will be balanced by other stations misses the reality that station trends and covariance are independent. You are changing trend according to an property 99% independent of trend.
The stations must be constrained positive or minimally weighted when the calc is done. The fit would be worse but the result will be more accurate.
The example presented in #128 defines the negative covariance as a negative of temperature in the assumption. This allows a reasonable flip of the second station but is non-physical and negates the possibility of a net rise or fall of temperature average for the two stations.
The whole argument conflates the accuracy of expectation maximization with the meaning of a thermometer. The reconstruction is simply a weighted average of temperature stations, nothing more.
I never expected there would have to be a debate about this.
RE #148: You want to interpret the coefficients on the stations as indicating something about the contribution to the average of the region local to the station, which is fine, but constraining the coefficients to be positive won’t solve your problem. If the stations are correlated you could still end up with (eg) a pair of coefficients, one close to zero and the other large when in fact both regions they represent make an equal contribution.
This is why I say – modulo this underfitting thing – negative coefficients are not the achilles heel you are making them out to be. The huge weighting of the peninsula in the overall trend is more significant – the PAC-man plots.
Baxter (Ryan, Jeff) different question:
Does anything in the Steig algorithm allow for higher order pattern finding like “interactions”: i.e. not just some of Xi and some of Xj, but some XiXj or even Xi^2? I figure the answer is no, but wasn’t sure if the multiple steps (composition of the PCs and then weighting of them, would allow for that. Also, do you think there should be? I realize that it reduces doF, but I wonder if it enables more sophisticated pattern finding that helps generate a more likely answer.
152: I agree. But it;s scary that Jeff and Ryan did not get this and instead responded to crits on the peninsula weighting by pointing to the negative weighings as the more interesting result. And then when they don’t get the basic point of degenracy it just scares me. I worry that they are not only ignorant but unable to assess their own ignorance (I know I’m ignorant at least). That means we are FAR from a final straw. It means that the principles here still have some basic things to learn to even be able to dissect Steig properly.
#150, I agree with you up to the last line.
You are right that one being zero and a very similar one getting the weight is also bad and I’ve seen iterative algorithms do that fairly often. My company uses iterative methods to desing non-imaging optics. This can is also be a problem and from an email NicL sent me, RegEM may not be converging yet, Steig et al may have stopped it really short from an actual minima.
If a zero weight and a high positive weight are adjacent it isn’t very good either, but at least then we haven’t flipped 15% of the data before averaging.
151: If you protest any negative station as unphysical, you mnight as well protest any stations in the same region that don’t have equal value as unphysical. But the end result on the overall trend is the same either way if they’re degenrate. Think about orbitals, Jeff. First mode is ++++. Second mode is ++–. Third mode is +-+-. and you can do linear combinations.
This is a physics problem not an unconstrained math problem. The idea behind the reconstruction is to build a surface of energy distribution across the continent by using the sat data which is not sparse to form the surface. A better technique might have been to build a set of surfaces and then try to figure out which surface to fit to for a particular time in the historic reconstruction. Whether creating one surface or many it makes no sense from a physical standpoint to have a part of the surface be negative or even close to zero. The temperature of those points are positive and real by assigning negative values to them you create a nonphysical energy distribution as in if you look at the reconstructed temperatures (not anomolies) they will be negative which might work for virtual particle in a vacuum but surely has nothing to do with a realistic energy distribution across the Antarctic.
The second big problem with the steig method is how the surface is formed in the first place. If the goal of the reconstruction is to develop trends for the unsampled areas then the long term trends in the calibration period would be far better as a predictor from any physical standpoint rather than the high frequency high variance which was used.
Jonathon, I argued with Jeff on this point as well a while back and he eventually got through to me. I don’t have any problem with the concept of negative covariance (assuming the relationships are modelled properly) . Since the Antarctic covariance is dominated by HF signal, the negative covariance would seem to indicate oscilating HF modes or some other physical phenomenon. Even TCO’s example of Walmart (up) vs the economy (down) – another example of a negative HF relationship – may well be valid. However the 50 year trend of the economy and Walmart is positive for both. IOW, negative covariance between Walmart and the economy may be a good predictor of HF fluctuations of the economy, but has no value at all for predicting a 50 year trend. The “it will all come out in the wash” counter-argument seems like arm waving to me.
Jeff posted on this a while back where he showed a case example of two Antarctic stations with negative covariance but had postive 50 year trends:
https://noconsensus.wordpress.com/2009/04/04/whats-wrong-with-regem/
He followed up with this post to demonstrate the disconnect between trend and covariance:
https://noconsensus.wordpress.com/2009/04/14/warming-the-raw-sat-data/
BTW, y’all have different implicit assumptions in your arguments. Jonathon implies perfectly modelled covariance. Jeff/Ryan implies imperfect modelling is the cause of unphysical negative thermometer as an artifact – and hold this out as evidence of a flawed model.
I feel the real issues of the value of covariance as it pertains to the reconstruction are:
1. Are the post 1982 covariances stable for extrapolating? 2. What good is HF covariance if there is a low frequency phase shift going from pre to post 1982?
Layman Lurker:
If this is true (as I think is probable), I would suggest that the data need to be low-pass filtered and appropriately decimated before being fitted.
Jeff,
Do you have an email address I can contact you with?
Mark
148 – Thanks, that helps.
In 128 you said “So T = c * TW + K where c is some weight.”
In 148 you say “T = -1/3 TE + 2/3 TE = +1/3 TE” – ie. T=1/3 TE
What happened to “K” and what was its definition?
Lurker: The “it will all come out in the wash” counter-argument seems like arm waving to me.
Me: Actually it is Jeff and Ryan who are hand waving and sayting its just wrong to have negative thermometrs and the like. The wash arguers have math on their side. The negative thermometer types are just Mannian “shouting louder”. Listen to Baxter. He’s the real deal.
P.s. Seeing this scares me in terms of Jeff and Ryan’s insights. Doesn’t mean that Stieg is valid. But means that Jeff and Ryan prone to bumbling.
154: If you protest any negative station as unphysical, you might as well protest any stations in the same region that don’t have equal value as unphysical.
Thats correct. They better be highly correlated from a multi-year trend perspective unless the penguins are building cities and you have unaccounted for UHI effects.
An open offer to anyone (RC folks welcome) who thinks the Steig reconstruction is better than Ryan’s 13 PC version or a simple kriging method or some other such area weighted reconstruction. There is no way we can go back in time to definitively determine what the temps in the Antarctic were however we are perfectly free to use the Steig model going forwards to predict both temperatures and trends. Anyone who would like to take the Steig model going forward and make some bets vs. the 13 PC recon, or a simple kriging model will find a happy bookmaker over here.
Like I said upthread, there are a lot of beliefs that people will ascribe to for purposes of argument but in the end won’t pony up to the betting window cause they know the arguments are just for the sake of arguing.
jeff and or ryan:
I copied the file recon.doc, turned it into a text file, uncomented the line which causes the original data to be downloaded off the web. The first four files download happily but then I get the following error.
Error in paste(“http://www.antarctica.ac.uk/met/READER/”, type, “/”, reader.names[i], :
object “reader.names” not found
It looks like reader.names is declared before it is used.
Any thoughts?
thx
david
I suspect that we might be losing site of what Jeff ID was pointing to in the introduction to this thread and how it relates to statements in the the Steig et al. (2009) paper. The title of the thread is reference to Jeff ID presenting evidence (counter to his initial intuitions on the matter) for why the Peninsula warming was spread to the West Antarctica, and the Antarctica as a whole, by way of his discovery of flipped thermometers and flipped thermometers mainly in the Peninsula. Jeff says:
From the excerpted statements from Steig below we see an interest early in the paper of showing that the Peninsula warming has spread to the Antartica as a whole. Otherwise one could, I suspect, assume that the Peninsula is a relatively very small part of the Antarctica with a climate separate from the Antarctica as a whole.
However, later in the paper the authors make an admission that the methods applied do in fact fail to capture the variance (warming) of the Peninsula and arm wave it off by saying we already knew it was warming rapidly. The problem they point to is in an early truncation of PCs.
Therefore, I asked: how could such an unphysical flipping of thermometers be benign when it apparently takes warming from the Peninsula to the remainder of the Antarctica – unless of course it can be shown that all the compensation for the inverted thermometers occurred in the Peninsula in contradiction to the Steig statement above?
I further conjecture that Steig et al., on obtaining a “preferred result” were not all that interested in doing further and extensive sensitivity testing of the nature that Jeff ID did here.
RE #160: T = +1/3 TE + K. I dropped the K since I was showing only the dependence on TE and TW.
TCO
Does the same argument (that flipping is OK) also apply to the Tiljander series used by Mann?
RE #161: TCO:
FWIW, I think what Ryan O and Jeff have done is very impressive. I would not be scared 🙂 Reproducing these kinds of studies from the limited available information is a herculean task. Their follow-on analysis is very relevant. My only quibble is the emphasis on the negative weights: they could well be an issue in this case, but in general they are not a problem per se.
165 – Noted, thanks. Please can you also comment on the definition of T and K as you envisaged them in your example of 128?
#161 TCO
Before you suggested that these negative weighted covariance relationships could all be valid. So now you are suggesting symmetry of HF covariance false positives and false negatives could contain the correct 50 year linear trend signal for weighting surface stations?
So here is a question for you TCO. Is this “symmetry” a necessary condition for negative thermometers to be acceptable? This is where the arm waving comes in. Before any unphysical factors can be legitimized, the “it all comes out in the wash” hypothesis must be proven. This is also where the theoretical and the reality of Steig part company. We already know that Steig covariance is not modelled properlly. In order even just to arm wave this argument, is it not necessary for covariance to be properlly modelled? I suggest that if the necessary conditions for symmetry do not exist, then there is asymmetry (of false positives and negatives) and therefore the station weightings are spurious.
168. I agree both that the reconstruction work has been impressive and that there general vice specific comments are wrong. I further add that I see a general flaw in their ability to do issue analysis, in their muddling of general and specific, etc. I’m not saying they are crap people with nothing intersting in progress. Just that they are not done. And that they don’t understand this stuff as well as they think they do. Nothing wrong with that…it’s tricky stuff. Further, I’m not claiming to understand the stuff better than them (I understand technical issues less) but that does not prevent me from seeing some of their flaws.
Layman:
“Before you suggested that these negative weighted covariance relationships could all be valid.”
I suggested that having negative weightings is not in and of itself wrong for the CLASS of all such problem solutions. For instance regardless of the unphysicality of negative temp or MPG, this may be compensating for other higher positive nmbers elsewhere. I suspect there are SPECIFIC things wrong with Steig. But I react against the GENERAL statement (not referenced to a text and disagreed with by a well credentialled expert, Baxter) that having negative weightings for temps in and of itself must be wrong.
“So now you are suggesting”
Please don’t put words in my mouth. For one thing, your words in general are so ambiguous, I don’t even know if you are properly describing my position! 😉
“symmetry of HF covariance false positives and false negatives could contain the correct 50 year linear trend signal for weighting surface stations?”
I don’t know what you mean by symmetry or HF in this context. I will say though that the general kvetching about HF and stuff needs to be backed up more by Jeff. He says things and then has no citation to explain it. Further, lack of HF and reliance on LF matching is something whcih denialists typically crticize Teamsters for in dendroclimatology! In any case, the burden of proof is on Jeff. He is making a broad statment, but can’t back it up.
“So here is a question for you TCO. Is this “symmetry” a necessary condition for negative thermometers to be acceptable?”
There can be reasons of degenracy (essentially uinconstrained) OR physical reasons for the negative weightings. Furthermore, ask yourself this, how does an algorithm make any of t weightings differ from each other? Really think about it.
“This is where the arm waving comes in. Before any unphysical factors can be legitimized, the “it all comes out in the wash” hypothesis must be proven. This is also where the theoretical and the reality of Steig part company. We already know that Steig covariance is not modelled properlly. In order even just to arm wave this argument, is it not necessary for covariance to be properlly modelled? I suggest that if the necessary conditions for symmetry do not exist, then there is asymmetry (of false positives and negatives) and therefore the station weightings are spurious.”
Jeff and Ryan are making GENERAL statements. They are the ones handwaving. They have not cited a refernce. They just argue louder how they don’t like things. Heck, you even have Watts drawing pictures of backwards thermomoeter, yuck yuck.
167. Yes.
166:
I think they argue loudly because they can’t understand why some folks don’t get that absolute zero is a constraint on temperature. But hey, maybe steig et al have discovered some new physics and can get themselves a Nobel prize like the esteemed mr gore
“”170.TCO said June 12, 2009 at 11:45 am
168. I agree both that the reconstruction work has been impressive and that there general vice specific comments are wrong. I further add that I see a general flaw in their ability to do issue analysis, in their muddling of general and specific, etc. I’m not saying they are crap people with nothing intersting in progress. Just that they are not done. And that they don’t understand this stuff as well as they think they do. Nothing wrong with that…it’s tricky stuff. Further, I’m not claiming to understand the stuff better than them (I understand technical issues less) but that does not prevent me from seeing some of their flaws.””
Jeff this statement should have said:
1. Steig et al claimed the 3 PC’s captured, as they say, “”Assessments of Antarctic temperature change have emphasized the contrast between strong warming of the Antarctic Peninsula and slight cooling of the Antarctic continental interior in recent decades1. This pattern of temperature change has been attributed to the increased strength of the circumpolar westerlies, largely in response to changes in stratospheric ozone2. This picture, however, is substantially incomplete owing to the sparseness and short duration of the observations. Here we show that significant warming extends well beyond the Antarctic Peninsula to cover most of West Antarctica, an area of warming much larger than previously reported. “”
2. Steig have claimed a physical relationship that explains that the cooling of the interior is slight and that the slight cooling with warming indicated limited to the penisula is flawed by being incomplete per “”This picture, however, is substantially incomplete owing to the sparseness and short duration of the observations. Here we show that significant warming extends well beyond the Antarctic Peninsula to cover most of West Antarctica, an area of warming much larger than previously reported. “” Our analysis shows that the methodology Steig developed resulted in a different realtionship than claimed.
3. We show that the rigorous and more complete analysis show that the limitation to 3 PC’s imparts a non-physical explanation for claiming “that significant warming exextends well beyond the Antarctic Peninsula to cover most of West Antarctica, an area of warming much larger than previously reported.”” Instead the method employed by Steig “smears” the positive temperature anomoly of the penisula across the continent due to restricting the reconstruction to 3 PC’s, and is indicated the non-physical temperature reconstruction.
In this paper, we show first that the weights show a non-physical basis that directly challenges the claim and assumption that 3 PC’s adequately capture “”This pattern of temperature change”” that “”has been attributed to the increased strength of the circumpolar westerlies, largely in response to changes in stratospheric ozone2.”” Our analysis shows that the restriction of 3 PC’s gives Chalndi (sp) patterns, and can be demonstrated by the negative weights of temperture reconstructions, and the 1982 splice, as an inappropraite and unsupported extrapolation of the non-physical.
4. Further by assigning weights to areas (pac man piechart)we show that the methodology of Steig et al, results in non-physical correlations of high frequency noise substituting for low frequency signals while the stated relationship by Steig et al is a low frequency signal. This is sunstantiated by a more rigorous and better defined selection criteria for determining retained PC’s with better CI’s, R^2, and limted (noise level) non-physical correlations which are found to be restricted to the noise level of our methodology.
Jeff, and Ryan stick to your guns “”Correlation is not causation””. The admittance of non-physical to a correlation can be shown to mean “”correlation is causation”” WRT NOISE in a POORLY defined system. The strength of your work is that you have a better defined system and have reduced the noise (CI, if for no other reason than your work is open and transparent 😉 . Your work shows what is noise and what is not. This is the Achilles heel of Steig. It is supposed to be showing that everybody else got it wrong. A BOLD claim. Your work shows that their claim of “”This picture, however, is substantially incomplete owing to the sparseness and short duration of the observations. Here we show that significant warming extends well beyond the Antarctic Peninsula to cover most of West Antarctica, an area of warming much larger than previously reported “” is not supported by the MATH and actual PHYSICAL relationships (No, negative thermometers allowed on Rura Pente, Star Trek VI).
Your work shows 1. “”an area of warming much larger than previously reported”” is a non-physical artifact. 2. That “”This picture, however, is substantially incomplete owing to the sparseness and short duration of the observations”” may be true, but Steig et al methology is shown to be substantially incomplete owing to the spareness of the method (PC’s) to replicate physical reality; and instead implements imaginary relationships between high frequency noise and low frequency noise that can be visually seen by the 1982 splice. Further, extrapolation of a non-physical relationship of a physical measurement, that may be acceptable in a noisy interpolation of correlations,cannot be supported a priori for a reconstruction; and this non-physical relationship of a physical measurement has not been proven by Steig et al to be post facto. (If I remember the terminology correcty, big assumption)
Longwinded post to say TCO could have done much better than I just did, but did not; since I have only tried to flesh out several of TCO’s good criticisms. THERFORE, I designate a new (I assume) term for TCO. He is not a troll. He is a Gremlin. A troll just jumps up and eats your blog by waylaying innocent posters. A Gremlin engages in “”creative”” destruction, such that the innocent posters think that the Gremlin posts are legitimate and that you cannot respond because you are incapable of responding in a constructive manner, not realizing he is engaged in “”creative”” destruction. Rather than going away for awhile until the troll is gone, they stay away forever because they think you are incompetent. When this does not work, he (TCO)pretends to be a troll because a half-assed victory over you is better than a loss.
Excellent summation, John Pittman. Thank you. And thanks Jeff, Jeff and Ryan for all the hard work. Very interesting blog.
Jeff – please can you let me have an email contact point?
I added my email contact in the top right.
Thanks
John Pittman, I think you are making essentially the same point as I was in my previous post. What I think is important is staying on point and hoping that the questions raised will be addressed by those who might be inclined to approach the issues in less detail.
I consider the analyses of Jeff ID and Ryan O (and others here and at CA) to be sensitivity analyses/tests of the methods and data used in Steig. Each of us reading and commenting about these analysis/tests are required to do some digging on our own and finally making our own judgments of the merits of the cases being presented.
It is much, much too simplistic an approach to assume that a peer reviewed paper invariably has the better chance of getting it right and therefore should put more of the onus of claims on those doing the analyses and sensitivity testing in a forum other than a peer reviewed process.
It is also a simplistic notion that a peer reviewed paper can only be refuted in any manner or form with another peer reviewed paper. A further simplistic notion (by simplistic here I mean taking a consensus or popular view and applying it generally without personally testing those POVs) is that the internet can be judged in a general context as opinionated and that it cannot do better in any instances than the MSM or peer reviewed papers. While there is plenty more noise on the internet than in peer reviewed publications and the MSM, for that matter, it also presents an (and sometimes better) opportunity for some thoughtful and beneficial investigations that go against the more conservatively oriented MSM and peer review product.
I think that the Steig paper, like some other climate science papers are presented in a fashion that cries out for a further look into the paper’s claims and the methods used. Questions that arise to the thoughtful analyst are the whats and whys of paper’s claims and are there statements in the paper that actually point to the paper’s weaknesses but in a tone and manner that appears to be minimizing them. Why amongst the many methods available would the authors chose the one used? What do sensitivity tests show about the methods results compared to others that could have been selected and particularly the simplists ones. Are the authors using/applying methods in ways these methods were not applied previously? Can the basic data used be arbitrarily adjusted/manipulated/selected and do those choices prove to be sensitive to the end results (as indicated by sensitivity testing)?
Sometimes these papers fail to point to issues that even the less informed realizes as deficient, as in Steig paper, the sensitivty of Antarctica temperature trends (using their own data – and residual AR1 corrections that they note but do not publish) to starting dates and in Santer in not doing all the difference trends between observed and modeled surface to troposphere temperature trends.
I find that the protestations of those simplistic thinkers from above who appear to have general prejudices against all internet investigations make those protestations online, as opposed to in published form in the MSM or peer reviewed publications, and do it in the noisy fashion that they see as the internet weakness. And just like the oft quoted movie comment I say to them “You can’t handle the internet (truth)”.
Nice post John. Thanks Ken, this thread has been the longest on the air vent by far and it generated more disagreement than I could ever have imagined.
In TCO’s case I assumed it was a gremlin type post – excellent definition. However, the others leave me really scratching my head.
My opinion on this hasn’t changed in the slightest bit. Inverted signals can have meaning in regression, inverted temperatures are meaninless. The claim that they somehow will explain some opposite variance in the weather pattern is completely without merit. I understand mathematically that the final trend may regress to a similar value by overweighting other areas but this paper was about more than the final trend. It was also about temperature trend distribution in the antarctic and matching to models which show a continuous warming.
If there is an anticipated anti-correlation (on all frequencies of the inverted signal) it must be explicitly described and explained by the authors and it wasn’t. It isn’t possible to have a truly anti-correlated temperature flip. The assumption that it exists misses the point that temperatures from multiple stations cannot rise or fall together for long term trends. This would give positive correlation which would also be inverted. This makes the result completely non-physical and obviously spurious. Correlation is not causation.
The inverted thermometers represent a failure in the math to represent the physical world – nothing more. I believe now I understand what happened to create the problem and I believe Ryan’s extra PC’s will fix it — future post. It is also my opinion that Ryan has fixed a great deal more of the reconstruction by including only surface station information in the final reconstruction. If Ryan and I are right about higher PC’s fixing the problem, it will be an advancement of the RegEM technique in this type of application.
Jeff: You’re continuing to make very broad claims about the wrongness of negative weighting. You should read the literature and see if you can find anything to back you up. It seems like a basic concept. If you’re right, it should be there. You continue to just protest the same points over and over again.
Of course the negative weightings are just a math thing and should not worry you. Heck, when you have two positive weightings and one is several times the other, does that bother you as thinking that one is several times warmer than the other? Maybe in addition, to Watts silliness of negative thermometer pictures, you can also show scales that are several mulitples of each other? Realize that all you have is a weighted average of several temps. If there is low correlation, it goes towards zero. If there is inverse correlation, it goes to negative. And the best predictor, may include some negative weighting counteracting other positive weighting.
If you can’t even resolve such a basic point with some one like Baxter, what makes you think you are ready for final straws? Or conversely, if you really do have a new and intersting insight, that is so fundamental, drop the Steig stuff and publish your “no negatives” in the top stats journal so every field can use it. Seriously, both you and Ryan are overdue for backing yourselves up with something more than asking me to prove the contrary or Watts’s pictures or Ryan’s “shit” labeling. Just pull out a textbook and cite the point in your favor.
I’m really not that worried about you having to eat crow on some basic point, like Watts and his Goddard buddy on some basics of college gas thermo. I’m actually worried you’ll never get there. Neither will we ever have a resolution. Just you all sitting in your little Internet gardens playing croquet with each other.
You are absolutely nuts to think you can flip a temperature anomaly upside down to do a weighted average. I’m tired of these posts. People will be confused by your wording because it sounds nearly reasonable, yet you are flat wrong. Upside down thermometers are so obviously shit I shouldn’t have to say it. All trends at all frequencies flipped over to match some similar wiggles?? — it’s stupid.
Johnathan Baxter’s point, as I understand it, is that other areas of the regression can still compensate. However, the compensation is also unnatural and even the discussion of compensation recognizes the problem we pointed out. So please don’t worry your pretty little head about our ability to follow a point and try to understand, you are the one who again missed the boat.
If you read RomanM, carefully, you will see that while he is being kind to you, he realizes that you don’t understand some basic things…
Jeff ID:
I suppose you meant to say “multiple stations must rise together, not “cannot”?
I think part of the problem here is just nomenclature. Mainly what is meant by “negative thermometers”.
I went through a fairly detailed derivation showing (I think) that the coefficients c1, c2, … cn can be interpreted as more or less normal correlation coefficients multiplying fluctuations at individual stations about their mean.
In this sense there is nothing unphysical about any of the coefficients being negative, even if it is possible to write the expression to look like it is really inverting temperature. It is easy to write down models in fact where one can get negative correlations in temperature fluctuations between two separate measurement locations. Take for example this:
T(x,t) = T0 cos(2*pi*f * t – 2*pi*x/lambda),
which describes the temperature fluctuations associated with a propagating acoustic wave along the positive x axis with f its frequency of oscillation and lambda its wavelength. If you take the separation to be lambda/2 you’ll find a normalized correlation coefficient of exactly -1 between the two sites.
A second example is the cross-correlation coefficient associated with temperature fluctuations in the boundary layer. What we find there is:
rho(x) = exp(-k0 |x|) cos(k1 x)
(x is distance to second thermometer from first).
Here there are regions of positive and negative correlation, and for large enough x, rho(x) → 0.
I think the real problem here is that we have two types of variations, long-term secular variations that you are trying to fit to a linear trend, and “high frequency fluctuations” associated with meteorology.
The basic problem (I think) is that the HF fluctuations are large enough they are “leaking” into the linear trends that you are trying to fit for. This is the opposite problem to the one I normally have, namely I usually am trying trying to extract the amplitude of a small HF fluctuation riding on top of a large secular variation. This is a problem I solve by detrending the data before attempting to analyze it.
ONe solution in the case of the Antarctic data would be to prefilter the data to remove the high frequency fluctuations (i.e., you could use annual averages) before trying to apply the TTLFS algorithm to obtain a linear trend.
Anyway, the confusion comes in that you have always implicitly been talking about secular temperature variations associated with global warming. (There are decade-level variations that allows one climate zone to warm while forcing another to cool for example. Obviously this is not what you are trying to glean on to.)
Carrick – please can I check my understanding here?:
“Anyway, the confusion comes in that you have always implicitly been talking about secular temperature variations associated with global warming.”
I think you have perhaps written “…with global warming” when “…within a global system” may be more accurate? Or are you merely saying the “secular temperature variations” would be different for the global cooling case?
Also, sorry if you have already posted on this but do you have any further info. to support this statement:
“There are decade-level variations that allows one climate zone to warm while forcing another to cool for example.”
Are you talking about this in the context of sub zones within Antarctica? elsewhere? What is the mechanism of the forced cooling?
And please can you elaborate on the physics of the acoustic wave and temperature example you give:
“Take for example this:
T(x,t) = T0 cos(2*pi*f * t – 2*pi*x/lambda),
which describes the temperature fluctuations associated with a propagating acoustic wave along the positive x axis with f its frequency of oscillation and lambda its wavelength.”
Apologies if this has been covered and I’ve missed it. Thanks
Curious:
The reason I said “global warming” was I wanted to distinguish between global trends and regional trends such as those caused by decadal-period climate fluctuations, as opposed to trends caused by the hypothesized long-term warming. I should adopt language like “global trend”, which is much less ambiguous….
The causes of this are way outside my specialization. But there are decadal scale atmospheric oscillations such as the ENSO that cause one climate zone to warm while simultaneously cooling another. There is in fact something called the Antarctica Oscillation (E.g., discussed here) that may be of relevance here.
Again I’m just starting to read about the Antarctica climate, so I hardly claim anything more than an amateur on this….
If, as I suspect, the Antartica peninsula is in a different climate region than the Antartica interior, in addition to a global trend in temperature you may have decadal-duration trends in temperature that anticorrelate with each other, that will show up if you use the relatively short calibration periods of a few decades. I’ve had in the back of my mind for a while that seeing negative decadal interval correlation coefficients between different loations may be evidence for trying to correlate data across climate zones. At the moment, I favor the hypothesis of the negative correlations being associated with spectral splatter from the HF weather-driven temperature fluctuations.
Any process, such as the propagation of an acoustic wave that involves pressure and particle motion, requires a corresponding temperature fluctuation in order for the process to be adiabatic (in acoustic terms this translates into “no net transfer of heat”).
For any adiabatic process we have in general:
P^(gamma-1) = constant * T^gamma
and gamma = c_v/c_p is the ratio of specific heats of air and where P is pressure and T is absolute temperature.
The Helmholtz equation describing acoustic propagation in the atmosphere (neglecting gravity, assuming constant density etc) can be written as:
grad_r^2 P(r, t) – 1/cair^2 d^P(r,t)/dt^2 = sources
where grad_r^2 is the Laplacian in vec r and cair^2 = gamma R T and R (=287.04 in mks units) is the gas constant for air.
Anyway for a planar wave propagating in the +x direction this has the solution,
P(x,t) = P0 cos(2*pi*f * t – 2*pi*x/lambda)
which combined with the relation between P and T for adiabatic processes, necessitates the temperature fluctuation equation I wrote down.
At really high frequencies, the assumption of adiabaticity breaks down, leading to net heat flow. Interestingly this also happens at very low frequencies, where gravitational buoyancy and nonlinear effects cannot be neglected, leading to a wintertime heating of the stratosphere above the Rocky Mountains…. That’s getting way off-topic though!
Ken #179, Yes, I agree you did. I was trying to show that TCO’s post could be constructive. Rather than “simplistic”, I believe that the term that should be used is “naive”. “Naive” meaning it explains a little but is not powerful in explaining.
Jeff 180, Steig has made several specific and IMO bold claims. The work done by all of you is most interesting. The long post was my outline for what I think is most important. It is not just your work, but the claims of Steig, especially the bold ones, that your work challenges. I think that this should be the underlying theme of the paper if you publish. By appearing on the cover of Nature, and the claims made, Steig’s work has ascended to the pinnacle for explaining a physical observation: that the Antarctic is warming appreciably. If the claims can be shown to be false or most unlikely, and a better and more yours a rigorous methodology giving different results, you will have accomplished a real contribution and advancement.
Big if’s though. However, remember that Steig has to prove his position. I think your group’s work has already reached the point you can readily show that it did not prove the bold claims made.
I noticed that the link disappeared to the paper on the Antarctica oscillation so I’ve uploaded it here.
Carrick – many thanks for clarifying, I wondered if you were referring to ENSO. I’ll take a look at the Antarctica Oscillation paper link.
As far as negative correlation issues for the Antarctic goes I think there was a comment elsewhere by Geoff Sherrington discussing the possible influence of circulatory weather patterns being detectable in the coastal AWS data – this seemed reasonable to me given the largely coastal nature of the stations but I don’t recall it being picked up and expanded on.
Thanks also re: the acoustic wave – temperature relationship: Do you have any numbers which give a feel for the scale of these type of effects and where they may be at play in the Antarctic? One of the things that concerns me is the level of precision that is sometimes quoted is not supportable by the quality of the instrumentation and I suspect these type of acoustic – temperature effects would fall into the unmeasurable by the systems in place/available in the Antarctic?
Curious:
The accuracy of a given temperature measurement is usually specified at about 0.1C and a precision of 0.04C. In my opintion, hey are plenty good enough to resolve the types of long-term trends we are describing here. (Places like the US where we have changing patterns of human activity are more problematic, but that has nothing to do with the inherent limits of temperature measuring devices.)
Pure acoustic temperature effects of course are pretty small, because the pressure fluctuations associated with them are tiny. A large sound (94 dB SPL) corresponds to just a 1-Pa amplitude. Given that atmospheric pressure is of order 10^5 Pa, this implies pressure and temperature fluctuations of order 10^-5 of their ambient values.
Obviously ordinary thermometers can’t resolve this, but high frequency thermal sensors can.
carrick:
Although Jeff is more than capable of speaking for himself, I don’t believe he is arguing that it is impossible to have negative correlation between thermometers that are affected by regional or micro climatic forcings. However, if you had such a system and you were looking to model the temperature say at a point say halfway between two measurement sites a realistic model would not have the point be say 2.5 times the temp at the first point -1.5 times the temp at the second point. A proper physical model would take some part of one plus some part of the other.
Atmospheric temperature fields even in the short term are gradients. Sure you can have processes that change the slope of the gradient so that for some phenomena where there is another energetic component such as ENSO temperature at one point goes up as temperature at another go down. However the temperature at each point in question whether they go up and down synchronously or out of phase is still positive. So modeling their rise and fall is not properly accomplished by weighting one negatively. IF you work in a de-trended space then of course you can do this but that is NOT what the weights are in the Steig paper. They are weights of the thermometers directly. This is purely an artifact of unconstrained regression and has nothing to do with what has or hasn’t been found in the data set. As such it is wrong, period.
I think that Jeff’s intuition is correct in that Steig and co. did not realize that this had happened and now that it has been pointed out they wish to wave it away as not a big deal.
Forgetting the complexity of the math think for a bit what is the actual problem which is trying to be solved. You have a set of data which are spatially dense (the sat data) which cover the whole continent for 30 years. You have another set of data which is spatially sparse which goes back another 20 years. You are trying to use the 30 years of sat data to possibly improve on an interpolation using just the sparse data during its period. If you were to just use the sparse data during the reconstruction period you could go from the simplest reconstruction (Jeff’s area weighted one) to more complex ones (trying to change the area weighting using some kind of correlation analysis — kriging).
One way to use the sat data would be to create a single energy surface which best fit the composite set. It would be the surface that minimized the error among all the surfaces (one for each timeslice) for the sat period. The surface selected would then contain information on how to interpolate between the known temperature points in the reconstruction period. Other more complicated ways could use different surfaces for different times depending on say some measurable regime shift so in a warming period one surface might dominate and in a cooling period another might. In any event it surely would be good to start from the simplest and work from there, rather than to do as Steig did and start with something that is mathematically obscure and call it a day.
DavidA: I think we are close to being on the same page. Yes, I agree that a degenerate situation (essentially unconstrained), it seems simpler to use smaller numbers versus 2.5 of one minus 1.5 of the other. (This example assumes the two stations are identical.) However, the IMPACT IS THE SAME!
For your example of a situation with anticorrelated sensors (anomoly anticorrelated of course), imagine that you TAKE AWAY one of the two stations and then want to know the temp at that point using the other station. The answer is simple. Use the negative of the station remaining.
(Assume we are using anomoly data, variance from average here.)
Jeff,
1. Peace man. We are eaching starting to repeat comments. I will try to engage more with the other commenters and not with you so much.
2. I agree with Baxter. In addition, I think that both he and I can construct situations where collinearity is the reason for the negative and also physical examples (at least theoretical ones). And in no case, does this require that temperature is physically negative. It is just a math thing to have a predictor that is negative anomaly.
3. How can we resolve this dispute? Would it help to have some established authority like Jollife or the like make a ruling and explain the error to whichever one of us is confused (since there must be a right answer and you and I persist in differences and since the issue seems simple…so that one of us must have a gap that needs to get fixed).
4 Would you please reply to my comments with new posts at the end of the thread? I have a long wait for my stuff to show up and it is hard to know when it has posted and then to see your replies. Also, it looks too much like the voice of God from RC, where they set themselves above the commenters and where they can’t STAND the thought of an adverse comment that is not immediately rebutted in the same block. You don’t want to be like Mike and Gavin, right?
hooly moolys 191 posts!
Layman Lurker said
June 12, 2009 at 11:18 am
Nice one Mr. Lurker!
simple point but here it is ,,, if you invert enough temps that are negative leading then you make a positive trend!
ta da!!!!!
search my name here at Tav and see how many times I said “looks like an inversion”
go ahead and DO IT, btw you will find TCO comment that “I was out there”
David, Carrick
First things first. Jeff needs to present station weightings for Ryan’s reconstruction to see if there are still unphysical artifacts.
Even if Ryan’s work cleans up the negative thermometers, this does not necessarily mean that these correlations can be confindently incorporated into a reconstruction. Beta coefficients of stocks are not constant, they can vary with time. Econometricians attempt to model the factors to explain changes in the coefficients over time (in other words: “correlation is not causation”!). In addition, as we have been discussing here on HF vs. LF, the usefulness of covariance in stocks tends to deteriorate with progressively longer time scales – as the factors associated with low frequencies tend to better explain long term performance. Just as with HF, as the causal factors change, so do the LF correlations.
Layman: You mouth these meandering comments of seeming meaning. I’m starting to wonder what is wrong with your thought processes. It’s not as bad as TimL, who seems like a flake. But it’s still not good.
1. First things first. Jeff needs to present station weightings for Ryan’s reconstruction to see if there are still unphysical artifacts.
Huh? Wah? That’s not the first thing. And regardless, we have more than one issue to examine.
2. Even if Ryan’s work cleans up the negative thermometers,
you’re asserting they need cleaning up just by assuming it. That’s circular.
3. this does not necessarily mean that these correlations can be confindently incorporated into a reconstruction.
No duh. You’re just making the point that there are multiple issues to consider.
4. Beta coefficients of stocks are not constant, they can vary with time. Econometricians attempt to model the factors to explain changes in the coefficients over time (in other words: “correlation is not causation”!).
Where the hell did that come from? How does it tie to the rest of your paragraph? And what the heck does the very last part have to do with the rest of it?
5. In addition, as we have been discussing here on HF vs. LF,
We have not had a good discussion of frequency variation. Jeff just swings for the fences without referencing literature, without theoretical justification from detailed math and without considering the issues of lack of DoF for LF and also how his stance is basically 180 from the anti-bcp stance (where matching a single period trend, but no higher freqs is criticized).
6. the usefulness of covariance in stocks tends to deteriorate with progressively longer time scales – as the factors associated with low frequencies tend to better explain long term performance.
What are the stocks doing here again? And you have not well explained your points here. Certainly, I see no refernces to Brealey and Myers chapters.
7. Just as with HF, as the causal factors change, so do the LF correlations.
You just seem to be babbling here.
———————————–
Seriously Jeff, y’all are scaring me lately. TimL is a flake. Layman is senile or something. You and Ryan can’t state your points well and can’t prove them, but get pissed that you’re challenged on them, by me, Carrick, RomanM and Jonathan. I’m really worried that with so much still up in the air, that you hop up and down and expostulate about final straws. If you can’t even judge your own level of knowledge, how can you judge stats algorithms?
here is one https://noconsensus.wordpress.com/2009/03/06/flow-chart-of-regem-satellite-reconstruction/
Tim L said
March 6, 2009 at 2:15 am
Cloud masking?…….Cloud masking?……….Cloud masking?
three trash bins? If the temp is too high let’s toss it out!!!!!!!
I don’t see inversion in the flow chart but it is there some where……
Thank you jeff 1 and jeff 2, at lest there is some honesty here.
I will examine this closer, may want more on subroutine (steig Cloud masking)
TCO said
March 6, 2009 at 10:29 am
1. It’s a nice approach.
2. Tim, there’s no separate inversion “step”. When you put coefficients on the series, you can use positive or negative numbers. It’s just like a regression. If you had a variable negative, you just get a negative “k” and the result is same.
see how his skin changes color from tree to tree!
I was the last post.
#17 Tim L said
March 10, 2009 at 7:59 pm
TCO said
March 6, 2009 at 10:29 am
It is there … look at Mr. ID’s movie and those first graphs all there.
No one can see neg. input and see pos. out put like me, AND THEY refuse to give up the inputed data!!!
how do u know it is not turned upside down??????(or any one). give us the GD data!!!!!!!
wheeew! calm down now…
sorry Jeff, just a little rant! LOL
here is one movie I think I refer to, look at the blue and red small squares next to each other, they flash by real fast.
http://tinypic.com/usermedia.php?uo=JPmY%2BsPmqjxePHKZylu9FQ%3D%3D
reference 2nd time in reversed order
https://noconsensus.wordpress.com/2009/02/28/a-little-bit-of-magic/
Tim L said
March 1, 2009 at 3:54 am
It looks like what I said before…” some kind of inversion going on ”
((x-y)-(y-x))
ya this works but is extremely simple…lol
Nice work Jeff!!!!
are your eyes crossed yet?
sat type inverting, look at red and blue side by side
look at the EPCs 1-9
we start at blue then red then blue an so forth it oscillates from red to blue.
https://noconsensus.wordpress.com/2009/03/27/preliminary-pca-of-steig-data/
https://noconsensus.wordpress.com/2009/03/31/the-blendinator/
Fluffy Clouds (Tim L) said
April 1, 2009 at 12:06 am
#1 + mirny (((( LOOK I GOT THIS ONE! LOL ))))
#3 – campbell ((( at zero for weights)))
#2 – esperanza ((( at zero for weights))
by looks only
don’t know if + or minus has any meaning
but there could be that inversion I keep saying, good bad or ugly I don’t care.
DANG I was close miss this one when I looked… https://noconsensus.files.wordpress.com/2009/03/corplots-2.jpg
next:
at 1995 I see a phase error.
next# and TCO claims I am high!!! but I am RIGHT!!! lol
de-correlation….. or de weighting?
https://noconsensus.wordpress.com/2009/04/04/whats-wrong-with-regem/
Fluffy Clouds (Tim L) said
April 6, 2009 at 4:21 am
The freq. keying is fascinating!
The algorithm is not that smart. Actually it simply reverses the trend upside down according to the magnitude of the de-correlation …….. Here we are with the inversion I talked about!
Great work here Jeff!!!!
TCO putout too. LOL
#
TCO said
April 6, 2009 at 4:40 am
You’re a freq. Do you post high?
http://www.ncdc.noaa.gov/oa/climate/research/cag3/mi.html
#190 David_a: What I am suggesting is there is a potential corruption of the trend analysis by HF meteorology. This is easy enough to control by simply low-pass filtering the data and decimating appropriately (again, you need to remove the artificial correlations in the data introduced by the LP filter), then fitting the data.
If the coefficient values persist after this operation, then we can discuss the significance of having large negative values in some stations. You are suggesting that this is a red flag that the method has failed.
However, I would suggest we need a bit more formal test to understand why and how the method failed. When you can do this, you really can say you’ve “nailed it down”.
In this case, the most likely cause of the failure (in my opinion) is due to having too sparse a network of stations. Easiest enough to fix: For the calibration period, just introduce a denser network of stations and see what happens to the coefficients!
If for a sufficiently dense network, they remain positive, that is clear evidence that the method is becoming unstable as you start reducing the number of stations in the model you are fitting too.
If you get positive and negative coefficients over different regions, the method isn’t necessarily off the hook. There’s still the problem of having most of the stations in the peninsula, and the artificially high weighting they are given because of that. (We have the curious problem of simultaneously a poorly sampled continent spatially, and a relatively large oversampling of a small section of the total continent, one that seems to be heating anomalously relative to the rest of the continent.)
I’m pretty sure seismologists deal with this problem all of the time, I’d be interested in seeing how they approach it.
Carrick – thanks for follow up in 189 re: numbers and temp. measures. The accuracies and precision you mention seem optimistic to me in the Antarctic environment. I think Comiso was saying about 3degC (please check – from memory) error on AVHHR data and (some of) the AWSs have intermittent data and resiting issues. Also with the AWS data I haven’t seen a description of how the BAS averages are arrived at in the READER database. IMO this could have effects greater than the precision you mention. I haven’t bottomed this view out so please check – it may not be relevant to the current discussion.
Also just an idea following on from above – what do you think of the point re: coastal station correlation? Do you think there is any benefit in taking a relatively small subset of the coastal stations with potential for correlations and (assuming correlation exists) extending this one station at a time to see how large a range (arc of coastline) the correlation holds over?
TCO: For your example of a situation with anticorrelated sensors (anomoly anticorrelated of course), imagine that you TAKE AWAY one of the two stations and then want to know the temp at that point using the other station. The answer is simple. Use the negative of the station remaining.
YES!!!! if you want to know the ANOMALY at that point. But if you want to know the TEMPERATURE, then you have to add back in the average which you have previously subtracted out to get the anomaly. As per Jeff and Ryan’s analysis the Steig paper DOES NOT DO THIS. The paper reconstructs a TEMPERATURE field where the TEMPERATURES (NOT THE ANOMALIES) are NEGATIVE. This is the same idiocy as used with other upside down proxies in the newer MBH paper.
sorry about the bold. just wanted to highlight comment TCO and i messed up the tags.
TCO, the Antarctica isn’t particular “harsh” with respect to temperature.
Sure it’s cold by our standards (though a thermoresistor doesn’t know that), but its very dry, and the temperature doesn’t fluctuate all that much. There are no changes in land usage or other auxiliary conditions that need to be taken into consideration here.
As long as the thermometer’s are regularly calibrated, really 0.1°C is very doable.
Carrick:
I completely agree with you as to the incorrectness of imputing a long term temperature field by using short term correlations. If you are going to look for long term trends and you want to look at correlations then use the long term correlations but still be cognizant of the physics.
Once I get a little bit better with R I’d like to plot some of the simplest things. Like what is the average temperature over the grid for the satellite period. Not the trends but just the simple average temperature. Is it like the rest of the planet where temperature goes down as the absolute value of latitude goes up? If so, what does the gradient look like? How stable is it? Is it different in winter when there is no daytime, than in summer when there is daylight 24/7? Once you had a decent handle on some of the basic continent wide things you could start understanding some of the regional effects and so on. Then maybe you could start making a decent reconstruction of the pre-satellite period that would have some basis in the physical reality as opposed to just playing games with matrix algebra.
As far as verifying whatever model you end up with, the very best way in my practical financial world is to make predictions going forward and then take a look at your P&L. Or bringing sports metaphors into the discussion (my condolences Jeff) you are what you’re record says you are……
David:
1. I don’t think anomoly versus absolute in the recon weighting is critical. For instance, imagine a case with 3 sensors:
A: Our experimental point, exists in the calibration period, not in the earlier period.
B. A point that is anticorrelated* (in anomoly) to A.
C. A point midway between A and B, whose temp is equal to the average of A and B: C=(A+B)/2.
Now how do we calculate A’s temp when that sensor is missing? The answer is simple: A=2C-B
proof:
A=2C-B
C=(A+B)/2
A=2((A+B)/2)-B
A=A+B-B
A=A
QED
*Note that this actually works regardless of A and B being perfectly anticorrelated.
1. First things first. Jeff needs to present station weightings for Ryan’s reconstruction to see if there are still unphysical artifacts.
Huh? Wah? That’s not the first thing. And regardless, we have more than one issue to examine.
2. Even if Ryan’s work cleans up the negative thermometers,
you’re asserting they need cleaning up just by assuming it. That’s circular.
3. this does not necessarily mean that these correlations can be confindently incorporated into a reconstruction.
No duh. You’re just making the point that there are multiple issues to consider.
4. Beta coefficients of stocks are not constant, they can vary with time. Econometricians attempt to model the factors to explain changes in the coefficients over time (in other words: “correlation is not causation”!).
Where the hell did that come from? How does it tie to the rest of your paragraph? And what the heck does the very last part have to do with the rest of it?
5. In addition, as we have been discussing here on HF vs. LF,
We have not had a good discussion of frequency variation. Jeff just swings for the fences without referencing literature, without theoretical justification from detailed math and without considering the issues of lack of DoF for LF and also how his stance is basically 180 from the anti-bcp stance (where matching a single period trend, but no higher freqs is criticized).
6. the usefulness of covariance in stocks tends to deteriorate with progressively longer time scales – as the factors associated with low frequencies tend to better explain long term performance.
What are the stocks doing here again? And you have not well explained your points here. Certainly, I see no refernces to Brealey and Myers chapters.
#196 TCO
1. ?
2. Any model which depends on unphysical relationships shows the model is wrong.
3. & 4. You can slice and dice AVHRR temp relationships any way you want but the question needs to be asked if these covariance relationships will hold in another time period.
5. Speaking of meandering comments… 😉
TCO at Post #196: If you would cease and desist with the nonsensical and transparent mind games and generalizations, perhaps you could hear better the likes of a Carrick at Post #198 put forth some useful suggestions and detailed observations of your own.
I believe Carrick is not that far from the level of concerns that Jeff ID has noted in his thread introduction, but simply would like to see a more detailed analysis showing and linking more directly the negative thermometers to reconstruction biases. Carefully read his post and this excerpt – which is essentially what Jeff’s thread is all about:
The Peninsula issue is at the heart of the paper and essentially its spreading warming to the rest of the Antarctica its reason for being on the cover of Nature. It can be iconic, like the hockey stick has become, and valuable in its marketing value for policies for immediate AGW mitigations.
TCO, I do not expect a reply.
Ken: You’ll get a reply.
The high weighting of the peninsula is a legitimate concern. Even with that, though, you have to say is the method artificially giving over-much weight to that region (perhaps because of low PC resolution or the like) or is that the correct best guess (iow that the overall continent follows the peninsula disproportionately to other stations. Note, I’m not advocating that. Just saying what question you have to answer EVEN AFTER you see the high weighting.
However, Jeff has already said that high peninsula weighting WASN’T the main finding and that the negative stations was the main finding. Also, they are two different things. And if Ryan and Jeff persist in making GENERAL statements that “you can’t have negative temps”, then I, RomanM, Carick, Jonathan Baxter, and the like will continue to say, “not so fast on that one”.
196: No mind games intended on the detailed editing. I took the vagueness apart. You should be happy. Even if it’s someone “on your side” vagueness and non-logical argument (just words thrown out) are bad. You can’t even engage. Can’t further the science. It’s bad when warmers do it. It’s bad when colders do it. It’s just bad. Feynman weeps.
At the end, I do slap Layman and TimL around a little, but it’s more in the manner of hoping to knock some sense in to them. It’s not that I’m scared of their arguments, so I need to make fun of them. It’s that they honestly don’t make sense. TimL talks as if he’s high. Layman blathers as if there’s a point in there, but you can’t tell what the heck it is. And then when he replies to me, he still doesn’t address the meaning of what he said, but just comes out with new things to say. It’s like trying to wrestle with a ghost. It’s not science. It’s not logic. It’s not math. Euclid weeps.
Ken, you’ve got your faults, but are clearly way better than Layman or DEFINITELY than TimL. You deserve a reply and I’m happy to give you one. You could do better, but that’s another story.
tco:
your arithmetic is correct but it is not what is happening.
the problem is that when the negative weighted station is used to reconstruct itself it gives a nonsensical answer both in terms of trend (which is upside down) and temperature (which is non physical).
TCO:
I agree with TCO.
The equations can be equally cast as temperature deviations from their local mean as well as absolute temperature. I have formally shown this to be true above. In fact, if you use my form, you may end up with a more numerically stable answer because you aren’t subtracting temperatures when you are adding terms with alternating signs, but deviations from their means….
You can argue 1) possibility of corruption of the trend estimates by HF noise, 2) under sampling spatially (Nyquist limit issues when you have an oscillatory function that you are undersampling), 3) irregular sampling that is putting too high a weight on a small geographical region that may not even be in the same climate zone or 4) an algorithm that has simply failed numerically (e.g.,if you can show it shouldn’t be obtaining negative correlation coefficients but it is, that is a “fail”).
But I simply don’t think the “inverted thermometer as nonphysical” works as an argument here. Just because it has a form that “looks unphysical” doesn’t mean that it is, as long as it can be written in a form where there is a physical interpretation to the quantities.
There are still issues that need to be addressed. I think TCO is putting the cart before the horse here. It isn’t Ryan’s responsibility to clean up the problems with Steig et al’s paper. It is technically the responsibility of the authors to defend their paper, and if the criticism is that there are negative weights at some stations they need to either demonstrate why this is a nonissue, amend the paper with a corrigendum or agree to with draw the paper.
Playing “rope-a-dope” with critics and claiming they do not have a responsibility to meet reasonable criticisms that have been raised simply because they aren’t peer reviewed and published (yet) is not a viable alternative.
Carick:
But Ryan and Jeff are doing more than saying, “it’s a concern, Steig. Defend yourself.”* They’ve actually made the “non-physical argument”.
*Not that I think this is a very tenable position, either. Ryan and Jeff need to advance things further. Not just think that Steig have to respond to blog postings and indefinite concerns. If Jeff or Ryan have a point to make, they can make it.
Jeff I think Carrick’s point of under sampling spatially problems wrt Nyquist limit issues when you have an oscillatory function that you are undersampling needs to be incorporated. Though we do not think of such a cold region have oscillating temperatures; it does. Also, as several have noted, the times that data is missing or assumed compromised tend to occur at the same time of the year, in which case an artificially (instrument) introduced oscillation with undersampling at the time of introduction could also be a possibility.
Such bold claims with sparse data and limiting the method to 3 PC’s would appear to be ripe for Nyquist limit issues, and the common underfitting problem as well.
TCO, there is plenty that Steig et al could address right now, instead of circling the wagons as they have chosen to do. That is irrelevant to how Ryan & Jeff have chosen to frame their criticisms.
These are all pretty definite issues:
Stop perambulating about how Jeff and Ryan should or shouldn’t approach this. Address the real issues here, which don’t have to do with questions of whether you agree with somebody else’s style.
I think that there is a lot that Steig could do. I agree.
That does not change my general impression that Jeff and Ryan swing for the fences more than they should and that Jeff in particular over-maginfies the import of what they’ve done so far.
Basically, I think they’ve brought up some interesting questions. Let’s not try to make it more or less than that.
However, I will continue to hammer Jeff and Ryan when they make broad, unsupported, wrong statements on negative thermometers being aphysical and the like. This is a science discussion board. And it’s my right to put down obvious stuff like that. And it’s actually helpful, since it pares back the Jeff/Ryan questions to more key issues.
TCO, let me be as clear as I can about why I think you waste a lot of bandwidth with your mind games and your constant efforts (unsuccessful as far as I can observe) to play people off against each other. There are 5 paragraphs in your last reply and only two deal with the issue at hand.
You say:
But I interpret what Jeff ID stated in the following excerpt as saying that negative temperatures do lead to the Peninsula “problem” and are thus not separable.
It appears to me that you are attempting to stir up a conflicts where they do not really exist. No one in this discussion says that you cannot have a method give, in a reconstruction like Steigs, a negative temperature – because it did. No one is saying that in some circumstances that negative entities cannot have some legitimate meaning and interpretation in those methods (mis)used by Steig. No is saying that a such a negative entity cannot be unphysical and a strong indicator that the methods used are not correct or its applications are incorrect.
The point of contention, if there is one, is whether in this particular application the negative thermometers are unphysical and furthermore an indicator that the methods were either wrong or incorrectly applied.
Ryan O made the analogy of the MPG where one could readily see that a negative MPG is not physical and having a negative value for it pop out of Steig like methods applications would be a red flag – without some explanation for it and showing that it would not upset the distributional results of the methods.
I really doubt that the methods used by Steig et al have been studied on a case by case basis as applied to other situations and, as a result, that there is probably a dearth of literature sources to reference. I have asked for literature references that would show a negative thermometer analogous result as legitimate in other studies and have not seen any presented on this blog.
I do judge that Jeff ID and Ryan O have the best feels for the data and what it means practically, but I would not mind seeing RomanM weigh in here with some theoretical and/or empirical insights on the matter.
This matter can be resolved for me, at least as best it can be resolved, without any replies from the Steig authors – even assuming they want to reply.
Moving on to some more useful analysis it appear that the antarctic is somewhat like the rest of the planet in that temperature goes down as absolute value of latitude goes up.
Just using the raw sat data from 1982 onwards here are some quick numbers.
lat > 80, lat > 70 and lat 60 and lat < 70
Number of grid cells
1492,3238,77
Average temp over 25 years
235.541901, 238.883743, 252.695465
So if you are going to reconstruct surface temps in the antarctic it would appear that at a minimum they should get colder as you move towards the pole. Time to try my hand at plotting..
TCO, I was in the same situation with you on another blog, arguing about prob abilities of temp increases. They kept disagreeing with me, and said I was wrong for various reasons, which I consider spurious. I don’t brag that I am deconstructing their arguments on a different blog. Instead, I decided it was up to me to prove what I was saying, and am getting model data to address their points.
average temp’s using raw sat data by latitude.
A better table:
highlat = -90.000000 lowlat = -87.000000 numcells = 128 avg temp = 227.010254
highlat = -87.000000 lowlat = -84.000000 numcells = 398 avg temp = 234.060944
highlat = -84.000000 lowlat = -81.000000 numcells = 682 avg temp = 237.148621
highlat = -81.000000 lowlat = -78.000000 numcells = 920 avg temp = 237.376175
highlat = -78.000000 lowlat = -75.000000 numcells = 992 avg temp = 236.109741
highlat = -75.000000 lowlat = -72.000000 numcells = 989 avg temp = 238.971191
highlat = -72.000000 lowlat = -69.000000 numcells = 878 avg temp = 245.751740
highlat = -69.000000 lowlat = -66.000000 numcells = 483 avg temp = 254.320343
highlat = -66.000000 lowlat = -63.000000 numcells = 39 avg temp = 263.558136
Total Cells = 5509.000000
I guess the theory of reconstruction is, that values at any location must have SOME correllation to expected values elsewhere…so, as I understand this, if the universe of data shrinks to the anti-predictive weather stations only (those with negative coefficients), then predicted temperature every where else (or I guess including those stations) must be ANTI-CORRELATED to reality. So it must be that this is not like (for example)a Fourier transform, where negative coefficients would be allowed…It MUST be a sign of an algorithm failure. (Unless I am totally misunderstanding the arguments.)
Ken:
1. Your quoted reference does not say that the two issues are dependant. Show me how you get that from the text. It boggles the mind.
2. Jeff and Ryan (and that silly billy Watts) have made GENERAL statements that negative temps are wrong, wrong, wrong. I could quote them if you want…just ask baby…
3. It’s not just me who’s calling this out. It’s RomanM, Carick, and Jon Baxter.
4. How ’bout we ask Jolliffe to come by and adjuticate? On the general issue? Work for you? For Ryan? For Jeff? Maybe you can tell him “negative temp is shit, shit, shit” or draw pictures of negative thermomoters like Watts did, when Jolliffe schools you!
http://www.climateaudit.org/?p=424
TCO, When you first came by I thought you were wrong and honest. Now I just think you’re wrong.
Tell me I’m wrong or go to the bucket.
Caspice?
I guess my “last attempt” wasn’t really the last attempt.
TCO: Stay out of this, please. You do not understand the math and you are repeating irrelevant points. If you don’t understand what the shape of an EOF represents, then you are in over your head. You do not understand what you are talking about, so quit repeating yourself. This is not something you can just brute force your way through. If you want to understand quit wasting my time and do the math. Otherwise, be quiet.
Jonathan, Carrick: You are confusing the “station weights” with coefficients. There definitely are negative coefficients, and these are certainly physical. These coefficients all appear in the EOF shapes (or the eigenvectors, if you prefer). The shapes are a map of coefficients. Nothing more, nothing less. If there is oscillatory behavior, or an east-west phase relationship, or anything similar, these are captured in the shape.
For the east-west example, one half of the shape (eigenvector) will have positive coefficients; the other half will have negatives. This stuff is already captured by the shape.
The station weights, on the other hand, must be positive or near-zero. The weights describe how important a particular station’s temperature is to the average. Whenever temperature at a specific location goes UP, that location’s contribution to the average is by definition positive. These are not abstract quantities. These are thermometers. They do not measure inverse temperature.
For a particular EOF, if a station is located in a region where the shape coefficients are negative, then yes, for that particular EOF, the contribution can be inverse. This is because each individual EOF doesn’t represent temperature. It represents an abstract partial temperature. It does not have a physical definition – only a mathematical one. It is the sum of the EOFs that represents something physical: temperature. That is why the station weights for temperature must be positive.
So Carrick and Jonathan, please try to disambiguate covariance from station weights. The covariance information – which describes your Rossby waves and any other out-of-phase behavior – is already contained in the shape (or eigenvector) for the EOF. The mathematical definition of the EOF shape is the covariance for that mode.
Station weights, on the other hand, are the contribution of station temperature to Antarctic temperature. The station weights contain exactly zero covariance information. They do not describe Rossby waves. All of that information is contained in the EOFs.
The explanatory factors are the EOFs – not the station temperatures. The station temperatures are what the EOFs are calibrated against. A non-physical relationship (inverse temperature) does not indicate anything about the covariance. What it does indicate is a failed calibration.
1. I sent Jolliffe a message asking him to adjuticate. Let’s see who he sides with, Jeff.
2. Don’t be so all or nothing. Just because I (and Roman, Carick, and Baxter) have schooled you on negative thermometers doesn’t mean you don’t have any intersitng insights.
3. Keep plugging away. You’re far from a final straw, but maybe you’ll get some initial straws soon.
#223,
Best troll ever TCO. – No question. I am surprised that you would waste Dr. Jolliffes time with weather it’s ok to read a thermometer upside down. —
Do the math yourself, then show us the truth. I will give you comment free space any time you want to try. Don’t waste Dr. Jolliffes time with this kind of thing until you’ve developed the ability to try it yourself.
Do you understand that you cannot even ask a question that won’t be answered in a standard text?
If Dr, Jolliffe chooses to answer, you can reply.
Besides, TCO, you don’t understand what Roman said. You haven’t schooled anybody on anything.
I.E.: Indicates a failed calibration. As I have said.
Click to access chap3.pdf
The solution of the collinear situation is indeterminate–agreed. But the output answer does not change regardless of the akward coefficients. And the jumping up and down about negative thermometers as violating physics is silly. Especially since the output does not change and since the zero coefficients are also aphysical. In addition, there are physical examples that one can construct where use of negative weightings is best: Carick myself and Baxter have all given examples of this.
Thanks, Ryan O for very clearly making the distinction between thermometers and EOFs and please consider my literature references as references to EOFs and not thermometers.
Will TCO discuss this issue with us now or will he be saying sometime in the future that “Dr Jolliffe and I took that negative thermometer issue apart piece by piece and really showed those skeptics not to celebrate prematurely.” I was there with the John V issue that TCO keeps repeating and I know he got that one wrong, or, if I wanted to be more sympathetic, fails to give the important details.
>Lately I’ve been having fun at The Air Vent going round and round on the “no negative thermometers silliness). I even have some other skeptics who are good at stats backing me up. Ryan and Jeff have their heels dug in, but will have to say uncle eventually. Sad, tho, since I really think their degective work on Steig, at least on the method, has been fascinating…
TCO @ Tamino
#227 No. Your examples all apply to covariance, which is already (by definition) captured by the shape of the EOF. The weighting is the result of the regression, which should always be positive assuming (1) a long enough record length such that you do not have spurious correlations; and, (2) enough EOFs are included to properly resolve the region in which the station is located.
That sounds like the John V story all over again. There appears to be a common thread that runs through TCO’s spinning of the facts.
Ryan O, thanks for the comments. I’m going to wait until I have a chance to work through the math myself before making any more comments.
TCO, did you really say that on tamino’s blog? That’s really pathetic.
I said it. It’s the way I feel. What’s surprising about it? Go pathetic yourself, dude.
I said it and stand by it.
TCO: I said it. It’s the way I feel. What’s surprising about it? Go pathetic yourself, dude.
LOL.
It’s pathetic as hell, you loser.
…I mean seriously?
You’re bragging on Tamino’s blog about your acuteness with statistics.
How would they know?
Carick:
It’s even more arrogant than that! I think I’m capable of picking at things, even when I’m NOT an expert. Capisce?
Jeff was wiser than I. He had a sarcastic reply to the revelation that I was making comments on Tamino. But he dissapeared it. (Which I’m fine with either way.)
#237 You’re missing the simple stuff this time TCO.
#238 I’m not sure what I disappeared. It seems like every thing’s going through. Try a different name or something.
You cut your own comment, Jeff. Last night. I watched it go up and come down. You had a comment at 230 last night, repying to Mike. And then disappeared it. No worries though.
REPLY, I am still the most snipped record holder on the air vent. I’ve chopped more of my own stuff than you can believe. I was just grumpy again so I cut my comments.
TCO, not to grind on you to hard (you seem to like that anyways so why am I apologizing???), but…
1) If I am not a skeptic, then I am not a scientist.
2) And it is the clinging to ideologically held beliefs that makes one stupid, not the willingness to question.
I find your constant verbal baiting and trash talk annoying and a waste of bandwidth, but I welcome people with backgrounds like yourself in general who come from other areas of expertise because you have insights from your discipline that can inform on the discussions we have here. Science is democratic that way, credentials don’t elevate the stature of ones ideas.
All I’m gonna say on this ’cause this is a waste of valuable time that could be better spent picking stray pieces of lint off my jeans.
I love you Carrick.
By the way, who is Tamino? Is he an expert in the field, on the level of Gavin and Mike, or what?
#243, Grant Foster – I believe. He coauthored some papers with them a while ago. If you search his name you will find stuff a hundred times more extreme than anything I’ve written. Tamino is nothing but a communist pretending to be a non-advocate scientist.
#244 Tamino’s smart, and he does do some good analysis. I actually read his blog quite a bit. I never post there, though. But he’s definitely an advocate and waaaay left politically for me. In my opinion, he doesn’t try very hard to pretend he’s not an advocate. Either that, or he’s not nearly as good at hiding his advocacy as he is at math. Haha. 🙂
Carrick: I don’t mind that TCO posted that, personally. He’s definitely entitled to his opinions. And TCO does make good points at times. By the way, your math you presented on the negative weights gave me an idea. Rather than doing this problem as a principal component regression problem, I think it could be much more cleanly done (without the use of RegEM for the PCs) by regressing the individual gridcells of the PCs against the corresponding surface station location. After each regression, you find the plane that minimizes the overall error for each eigenvector – and voila! Reconstruction complete. I’m still on vacation, so it might take me a couple of weeks to get to that, but I think it’s a much cleaner, easier-to-understand method that minimizes the number of “black box” algorithms needed to do the reconstruction. Plus, it should avoid the thermometer inversion problem.
TCO: Sometimes you do actually annoy me. Most of the time you don’t. The only times you do annoy me is when you insist on something without understanding even the meaning of the math. Carrick and Baxter presented mathematical reasons for their interpretations, which is helpful even if I may feel they are off base. The mathematical reasons provide a framework for their arguments, which makes the discussion so much more productive. Plus, both of them are smarter than me, so even if I may disagree on this specific interpretation, I always learn from them.
I think you would be much more effective if you did spend some time with the math. That way, you would be able to frame your criticisms more effectively. Plus, you’re a smart guy, and if you learn some math, you would find yourself making additional insights that you wouldn’t be able to do otherwise. 🙂
#245, I say some extreme things (especially when I get wound up) but Tamino- posting under different names says some really far left stuff that makes Obama look like Limbaugh. I used to have a bookmark collection of a few of his comments but the laptop caught a bug and zip — all gone.
Ryan, I lubs you too.
Ryan O, I agree essentially with your assessment of Tamino. He tried to give a primer on PCA at his blog one time and then get the decentering wrong. I give him credit for doing that primer, and others on various subjects of interest in climate science, but he had a very difficult time backing away from the decentering issue and even when Jolliffe came on his blog. I also recall that Jeff ID went way up in my esteem when he noted some of Jolliffe’s comments and related them back to a reviewer of a paper on the decentering of a PCA in a Mann reconstruction and was able to encourage Jolliffe to out himself as that reviewer. I thought that Jolliffe made some misleading comments at Tamino’s blog about his recollection of the review in order to keep his identity covered. He appeared to me to be re-tracking his comments after the fact. He also seemed almost apologetic in his critiques of the decentering issue and had to early on note that he was with the consensus on AGW. The whole episode was enlightening (thanks to Jeff) and at the same time sad.
What I see wasteful and distasteful about TCO’s comments are when he goes off on a continual tangent interjecting his personal assessments of people who post here and at other blogs. I find that not only rude but also discouraging of a higher level of discussion and understanding of the issues at hand. Note the difference between the intercourse between Ryan and Jeff with Carrick as opposed to TCO.
If TCO were a guest at my house and insulted my guests I would ask him to desist and if he did not, I would kick his ass out and never look back.
#248, That’s how I got kicked out of open mind, you forget these things after a while. Thx, good stuff.
Ryan O:
I agree he’s a smart guy, but IMO he makes too many basic errors, then refuses to back off them when they are pointed out to him.
He’s a candidate poster child for how ideology makes smart poeple stupid too.
Carrick,
Yah. I’m guilty of that sometimes, too. I have to catch myself. While I’d like to think I’m open-minded, I’ve caught myself doing things incorrectly because it gives me the “right” answer. It’s not always easy to police yourself. Though . . . I do think I do a better job than Tamino. 🙂
Ryan, thanks for your tolerance. I agree that I am mathematically lazy and a troll (you left out long-winded, although compared to Kennie-boy…haha).
Now back on the math:
Let’s get this fundamental issue squared away. I have Baxter, Carrick, and RomanM on my side. You got JeffId, Wazzup and Coyoteblogboy. I think I got ya outgunned on experts.
Jolliffe did not respond to my emails, that slacker. Maybe he forgets that I am that one he said he “loved” in the Tamino kerfuffle. Anyhoo, I’ve moved on to questions to Wegman, Peter Shaw, and Richard Kramer. If that doesn’t work, I’ll move on to phone calls to major universities and look at their staff bios and find a good expert to consult with.
Sorry to get all Hotelling on your ass, but we need to settle these theoretical statistical issues.
#252 I don’t like your claims of who’s on your side TCO, it may make you feel good but from what I read it doesn’t look very truthful. Keep it realistic.
Can you all point to a
1. Steig article pdf?
2. “the actual raw PCs and
corresponding eigenvalues. Is there a link where those data are available?”
Found the pdf on Mann’s site.
I just have to point out TCO that you criticize the people who have replicated the work in this paper, you’re writing emails to Dr. Jolliffe and YOU HAVEN’T EVEN READ IT!!!
Jeezus that ticks me off. Why do I listen to you?
Don’t forget to read the SI … @#^%@!
You haven’t followed along well enough to know STEIG WON’T PROVIDE THE PC’S AND EIGENVALUES YOU GET TO CALCULATE THEM YOURESLF. I’VE LEFT A FEW HUNDRED R SCRIPTS AROUND SO IT SHOULDN’T BE TOO HARD.
1. Calm down, friend. I figured that was the answer, but was double checking.
2. I read the paper, but wanted a pdf.
3. The more fundamental issues is the general “anti negative thermometers” claim that you and Ryan have made. This is what RomanM, Carrick, Baxter, and I have all disagreed with. This is a general statement irrespective of the Steig paper.
🙂
I’m going to calm down later. Now I’ll say it again, Roman referred to the flipping of sign in a regression being normal, this is the same as Carrick, Ryan, Myself and Baxter, this is a separate issue from the one Ryan and I have tried to explain to you.
In the meantime, the paper is only available in PDF form, the eigenvalues and pc’s must be calculated and I’m still pissed.
TCO, I did NOT disagree with Jeff on anything.
What I mentioned was that in a situation where a bunch of regression predictors are highly correlated (collinear), some of the regression coefficients could flip signs (i.e. a predictor which had a positive correlation with the response variable being predicted could end up with contradictory negative coefficients) and/or become unduly large. The cause is usually minor random fluctuations in the highly correlated data. It often goes along with large error bounds for the coefficients. This is not a good thing – it indicates that the regression is BAD and that the results could be spurious, particularly if used outside the range of the data from which the regression calculated. I actually testified about the effects of such a regression in a court case(!) in 1979 indicating why the use of it to predict a property value was unjustifed.
One solution to the problem is to do PCA on the predictors and to use several of these (which account for a good part of the variability) in the regression instead of the original variables.
There were some earlier comments about this possibly reflecting subtleties of changes in the data (e.g. Jonathan Baxter and Carrick back around comments 100 ff.), but I would think that that sort of cosideration would only apply to a data set with complete, excellent quality data. Thinking that you might be looking at such subtle natural climate features in data which has been highly manipulated (e.g. cloud masking and stripping out “extreme values”) and in which around 50% of the data is being created by infilling (you don’t suppose that collinearity might be related to this?) is dreaming in technicolor.
#259, Ok, now I settled down. I’m very tired today, if you get the data that’s a big step for you.
If you find something, write it up and you can make a post here.
259.
A. In examples of collinearity, the output result TREND does not change. You have an unresolved problem in determining the correct coefficients (multiple solutions to the equation), but the TREND is still the same.
B. The issue is not one of the Steig situation or Antarctica ONLY but of Id and Ryan saying EXPLICITLY and generally, that we may never have the negative weightings (except near zero). So your second para is not responsive to the general problem. Ryan says negative thermomoeters are shit, point blank. Not only in the context of this problem.
Also, Roman, it was NOT only Baxter and Carrick who brought up physical examples which would cause negative weighting. I did as well.
I can’t remember if it was Baxter or Carrick who said essentially that we can construct examples which would fit with negative weightings. That doesn’t mean that this IS the case. Nor does it even mean that the Steig algorithm doesn’t improperly created those negative weightings It just means that some sort of frumpy “shit-calling” and statement that negative weightings MUST be wrong are not mathematically true…like Euclid or something.
Furthermore, Ryan has been persisting in some really silly examples (for instance the mpg example) where Baxter correctly pointed out that what Ryan has a problem with is paucity of data in general (not having perfect sampling), but that the best recon must be made with what we have.
Oh…and btw, Ryan and Jeff haven’t even acknowledged the collinearity case. They just respond with shit calling, rather than acknowledging that in a collinear case, we can construct cases where there are negative contributions, but the trend is unchanged.
If the weightings are near zero, the solution is also likely unnatural but it will have minimal effect. It’s just a fancy method to determine a weighted average.
oh…and why does it blow us away that there might be actual collinearity in this exact situation. Consider the peninsula, where there is a general similar weather pattern. And consider the multiplicity of sensors, and consider that we are talking about actual thermomoters here (not tree-mometers, which have a lot of variance even in same region from non-temp factors and noise). I mean crap, is it REALLY so surprising that one sensor went negative and another went high positive in this sort of situation?
And it’s amazing that these guys just RECENTLY clued into the fact that this phenomon limits the contribution of the peninsula as a whole. Give ’em a little while and they’ll clue into the fact that collinearity means the trend is unchanged and the peninsula contribution unchanged with multiple different solutions of the weighting of individual within-peninsula stations.
Gotta go lift arms at YMCA. See y’all in a couple hours.
P.s. Interesting that Ryan has gone all radio silent.
P.s.s. I’m gonna keep dropping…eh…sonobouys on top of him. I know he can’t get away from two helos with dipping sonar, regardless.
TCO, the trend is not necessarily correct because the solution of the regression is unstable. The X-transpose-X matrix can have a determinant close to zero so the elements of the inverse may have large calculational errors. The cefficients can be completely out of whack.
The “best recon must be made with what we have” may not be worth having if it has no reliability.
I suggest that you do your own analysis instead of throwing crap at Jeff…
I still rebel against the “negative is wrong…like as if it were a rule from Euclid”. I think you are not quite addressing the real issue, RomanM when you say “not necessarily correct” when Ryan has been saying nescessarily wrong. For a crew that hangs with SteveM and for all his kerfuffle on “nescessary but not sufficient” and other fine gradations of logic, you all should do better.
For instance, we can construct physical examples where negative weighting makes sense. Imagine TCOland, which is divided into 10% Downity province and 90% Uppity land. Perfectly anticorrelated. And say that we have satts during one period to tell us that. And then in another period, we must reconstruct using a single station, Baxteria from “downity land”. Obviously, you can see that we will be able to reconstruct the entire land mass and that the trend for the overall continent will equal a constant -80% times Baxteria. (Hope I got that algebra right, but you get the drift.
People are always telling me to do my own recons, but if Id or McI or whoever are going to post their stuff and have discussions on it, I have no problem engaging and criticizing it. Plus I know it’s lazy and arrogant, but I fix problems all the time in fields I’m unfamiliar with, often enraging the people who have to watch me bumble and extract basics from them…but still often seeing very basic logic errors for fixing.
TCO, I am just letting you know that me, Aristotle, Socrates, and Einstein say you are throwing out contrary generalities without a willingness to learn or make the effort to read the material and if you were at our houses we would throw your sorry ass out for wasting everybody’s time.
Ken: I heard you the first time.
Kenneth,
You are right but I hate blogs that decide who’s right. TCO has been to my eyes dishonest in his criticism lately, if he’s dishonest I’ll toss him so fast he’ll think he’s Joe Romm.
Right now he has the dubious honor of being the first one ever blocked here looking him right in the eyes. Not for criticizing – if you can’t tell, but for being dishonest in criticizing. Dishonest criticism is propaganda[snip – venting]
Jeff:
A. You lack the perspective to tell when I’m being dishonest. Just as you lack sufficient expertise or thoughtfullness to know what is a “final straw”…or even to differentiate different criticisms (since here, several months later, many of y’all’s critiques still just devolve to being unhappy that simple distance weighting was not used. Something that even if it’s wrong, is not earth-shattering and is dealt with in the first para (or so) of the Introduction to the Steig paper. Remember that Rumsfeld was capable of considering he might be wrong. He even had a list of 20 things that might go wrong on Iraq and one was no WMD being found and one was a persistent civil war and one was an insurrection. Contrast this to your average ditto-head or even a Feith upper-middle-brow who drank the Bush (who’s really a pussy compared to Reagan) kool-aid.
The alternative to dishonest is not good TCO. There were two items in the final straw post, one which you agreed with? I am a Rumsfeld and Cheny fan — sorry to the rest of you guys. Rumsfeld didn’t take any crap.
B. If I decided to be dishonest, I’d really start kicking your ass. 🙂
no you’ll continue be snipped and corrected
C. You’re still not even doing a good job at disaggregating debates on fundamental properties of statistics (something that a Hotelling type would write a proof on) versus the specific issues and likely issues with Steig.
???
D. Hold the banhammer. I’ve gotten an expert to start considering some of the arguments. (He thinks it a little tiresome to read your blog threads, though, so we’re still early…) Who knows, maybe he’ll back you up. Or get otherwise interested in this whole kerfuffle.
I apologize to your expert. Did you explain to him that you so far refuse to try any math yourself or even read the paper?
P.s. The beast has been exercised. Shoulders and lats feel like they are growing.
This sounds like great news. I may even understand what you are talking about this time. I hope to find a new reinvigorated CO.
Ryan fixed Steig et al – until someone proves otherwise. The final straw was the actual little straw in the corner.
Cheney was cool, but Rumsfeld was an idiot.
Radio silence because there’s not much more to say on this. I stand by my negative thermometers are shit. 🙂
Dude,
A. Rumsfeld had a clue even if he was not perfect. Cheney is a stuffed shirt. And do you think it gives me pause, who you support politically?
C. Jeezuz Christ. Figure out my point, man.
D. I got a second expert now…
E. Seriously, stop with the VoG replies. It’s hard to keep a thread of replies. And take me out of cialis-land so we can do this real time. Put me back in there if I start cursing.
Dear North Korea,
If you fire your missiles at us and kill some of us … we’ll be mad. Really mad and we’ll take a time out.
So mad in fact, we’ll give you less free money to feed your army!
And we mean it.
Maybe.
Dearest love,
USA
NK is a little misbehaving brat. We should stop subsidizing them, but allow trade to happen with them. We should pull our troops from SK, the south has twice the people and way more money. Time to stop relying on Uncle Sugar to protect them.
Ryan (responding to your comment on Climate Audit):
1. Glad to see that you are still working on this, if even to come to grips with how to explain this to Carrick, Baxter and lil ole me. I did a bit of expert consulting as well. I’m almost ready to break down and get a book from the library* or something. Anyhow, even though you are very firm in thinking that you are right and we are wrong and we the reverse, I think that if we persevere in looking at this one side or the other will realize its error. It could even be that in the work of trying to convince the other side, one realizes the other side is correct.
2. (Sorry if repetitive) My main issue is not with your concerns on the negative weightings in this specific situation. It is certainly possible that these are a result of something wrong in the Steig algorithm. Or that the specific situation in Antarctica is not amenable to having these large contrasts.** But what I react against (and I think my reluctant cohorts as well) are the statements that it MUST be wrong.
3. Consider the insights in this thread:
http://en.wikipedia.org/wiki/Talk:Principal_component_analysis#Can_you_have_negative_weightings_.28even_if_unphysical.29.3F
(In particular, Jheald’s first couple of comments.)
4. We’ve already brought up the example of collinearity***, But consider a more nuanced version.
Think about a continent, imagine it is sitting in front of you and is divided into two sides (East and West). You are God.**** You have two forcing “knobs”, you may twist. One knob is the overall temp knob. It makes the whole table go up the same amount. Another knob is the contrast knob. It makes the East get hotter than the West (or more simply it makes the difference between the two of them increase).
Now you as God, twist these knobs arbitrarily over time. We observe the output via sattelite (or tablelite), but don’t know the knobs and you, and do PCA on the results. Obviously the two PCs will correspond to an average and to a difference. (And of course the difference contains a negative weighting.)
Now imagine that we are trying to reconstruct individual areas in the past, based on select stations, no table-lite. Imagine also that we have two stations E1 and E2 and W1 and W2. They are identical (or let’s say identical minus some noise or lower level effects). Since we’ve already shown how collinearity can lead to arbitrary decisions of negative and postive, it should be obvious that we might have E1+W1 and E2-W2 as the two PCs.
But consider that North to South Hemisphere is an example of a situation where a contrast exists physically. So looking at the temp data, we might see a general trend of both hemispheres warming (from solar, AGW, less volcanoes, etc.). Ignore the issue of different responses based on water as a higher order consideration. But obviously, there will also be a very strong contrast between the two hemispheres in the same month. So these could be different PCs. Heck even the different responsiveness of the two hemis might give a further down PC from the forcer itself.
*Have you had any luck finding book or academic references advising to check for negative weightings as a trouble sign, to throw out recons that have them?
** Your comment on “mixing” seems to go in this direction. But let’s stay clear on the difference between a GENERAL, never have a negative thermometer, draw pictures on WAZZUP to mock TCO attitude from a more particular statement that mixing in THIS CASE makes negatives unlikely.
**Which the more and more I think about it is very possible for temp sensors in close location or with a region with mimimal differences (sheet of ice).
***If you don’t like the sacrilige, imagine instead you’re twisting knobs on a stereo.
My (other) expert does not want his info posted on here. He thinks that the negative weighting is a strong matter of concern, but is not axiomatically wrong. He’s now more interested in just the paper in general and other areas of concern (low PCs and thus 40% variance excluded).
VOG: Without the context of the Q and A this post has no meaning TCO
Just wondering – Can knobs on a stereo give negative sound?
If God did as TCO said in that incredibly, long stupid post #277, HIS decision would make the physics real.
The problem and only problem, but unsurmountable, is that Steg claimed to be explaining physical phenomena in the Antarctic. No amount of negative weightings discussion being unphysical can change that Steig used a PHYSICAL defition for only 3 PC’s. If it contains an unphysical result it may be underfitted, it may be overfitted, or it could be just plain wrong. However, the stated reason was physical. Allowing unpysical means that he should have included more PC’s to retain more information. He chose a limit, that according to him, captured the physical. He can’t have it both ways. If he had claimed some other reasons perhaps it would be correct. However, he did not.
#277 Your example relates to PCA, not EOF analysis. While the two terms are often used interchangeably in climatology, they are subtly different.
Traditional PCA for signal analysis does not use the eigenvector. (See here: http://en.wikipedia.org/wiki/Principal_component_analysis ) The matrix that contains the covariance information (W* in the Wiki example) is not used. If you do not use the covariance information, it is likely that you will have negative weights because you essentially determine what the weight is at the station location by regressing each individual station against the loadings matrix (VE in the Wiki example) rather than using the covariance information. In other words, in traditional PCA, the station weight contains the covariance information.
In terms of your example (and the Wiki postings), this is exactly what is happening – and it’s okay. You definitely will get negative weights this way, at least with respect to an individual PC. Keep in mind, though, that the “weights” that Jeff and I are talking about are NOT the results of the regression against an individual PC. The weights are the sum of all the weights for each PC at each station location. Given that the subset of PCs you choose for the analysis are supposed to represent the same thing as temperature anomalies at ground stations (i.e., deviation from an average temperature), this overall weight should still be positive.
EOF analysis (which is what Steig did and is what Jeff and I are doing), unlike signal analysis, makes explicit use of the covariance information (W*). This means that the station weight and the covariance information are split. The covariance information – which indeed can legitimately include a negatively weighted area – is already present in W*. So whatever divergent or oscillatory behavior that you might think exists is already captured.
In the EOF case, once you get to the end of the analysis, you multiply each imputed PC by the associated eigenvector (covariance) and then sum to get the final result. This explicitly captures the covariance – not only for the station but for all areas of the grid. Again, as in the signal analysis case, if the subset of PCs you chose truly does represent the same thing as ground temperature anomalies, the overall weight should still be positive. Negative weights indicate the same problem as in the traditional PCA case.
In other words, I agree entirely with the answers by Jheald on the WikiTalk – but you are not interpreting them correctly. Jheald is talking about negative weights with respect to a particular PC. In the traditional case, that is indeed captured by the regression results. In our case, it is captured by the eigenvector. In both cases, as long as the subset of PCs chosen does actually represent temperature anomalies, the sum of those individual weights should still be positive or near-zero.
#277 TCO
From his comment at the link you referred to:
It is pretty clear that your “expert”, Jheald, does not understand what the situation being dealt with here is so frankly his “insight” is off base. This is not surprising given how long it took the rest of us “amateurs” just to wade through the convoluted methodology of Steig et al. to figure out what was being done.
His answer assumes all we are dealing with is simple PCA: take a bunch of input variables and calculate PC1 (or some other PC). The PC is a simple linear combination of the original variables and, yes, some of the weights in that linear combination may be negative and some positive. There is no argument there.
What is actually happening here is much more complicated:
-Take the satellite data for the time period when it is available. Condense the information from the entire continent to three “PCs” thereby capturing all of the nuances of that data!
-Take highly incomplete temperature data from the stations – many of whom have little real data prior to the satellite era.
-Mix together using the blackbox RegEM with TTLS (using new PCs calculated from the original PCs, the incomplete station data and large quantities of imputed station data which is being treated as equal in quality to the original sparse observed data and ALL of the condensed satellite data) to “reconstruct” three sequences which are interpreted as extensions of the three satellite PCs to the pre-satellite period.
-Take the original three PCs, append the recons, and recombine linearly to form an “average temperature” sequence for the entire time period.
Now examine the impact of each station on the final “average” product. Would you expect that there would be substantial negative correlation between some of the stations and the “average temperature”? Do you really believe that at the end of all the ad hoc manipulation in the entire process, we can actually see subtle regional interactions in the behaviour of the Antarctic climate?
Don’t you find it strange that there are no calculated negative trends in the “average” reconstruction in over 5500 grid cells – even in areas where some stations show some cooling ? Given that the original observed satellite temperatures do not strongly agree with the observed station temperatures at many locations, might you not think that an explanation of the negative correlations is necessary and not to be accepted as “well, it could happen…”?
If it looks unbelievable, it probably is. Steig made the claims and Steig needs to explain and justify the unrealistic aspects.
…(other) expert… Don’t be so pathetic in your comments. Go through the Steig reconstruction yourself to understand it better, THEN post your comments and maybe they will have some value and content
Ryan: If you re-read my example, you’ll see how it’s possible for a sensor to only be negative within a difference type PC. OBviously then the overall weighting for that sensor will be negative.
REPLY: TCO you are not understanding. The negative weight has nothing to do with PC’s whatsoever. It is after the pc’s have been reconstructed back into a reduced information temperature field that the negative coefficients show up. It is an inverted negative thermometer.
282:
I’m aware of the long algorithm. No need to repeat it. You made your point in the first para (to prove, I’m aware of the long para, be aware that I cited the “flow chart”.) I actually agree that he does not understand the overall Steig situation, but I DON’T agree with Ryan’s statements that negative weightings are unphysical as some AXIOMATIC statement. Imagine if we had some Arctic thermomoters mixed into the group. It’s completely reasonable to have some of the pointed negative. Similarly patterns could occur on this continent. Given a sufficient plurality of sensors and then some unequal sampling, it’s NOT HARD to imagine negative weightings.
“It’s completely reasonable to have some of the pointed negative.”
They are not pointed negative. The data is upside down, warming is cooling and cooling is warming.
“it’s NOT HARD to imagine negative weightings”
One can imagine what one likes but it doesn’t mean it is of any value.
TCO, I’m sorry, but it is you who is not understanding. The weights Jeff calculated are not with respect to individual PCs but with respect to the overall reconstruction. No one is arguing with you that a station can be negatively correlated to a PC – they can and they will be. All of your examples relate to weighting to an individual PC, not overall weights for the reconstruction.
I didn’t see Roman and Ryan’s answers before.
This is the problem TCO, it’s a little cold to say but the truth is you haven’t followed the analysis well enough to ask the proper questions of your ‘experts’. They are likely answering them very correctly yet your interpretation is incomplete. It reminds me of a student that doesn’t study first and runs to the professor for the answer to a question they aren’t yet qualified to ask.
— This is different than intelligence, you can do it I believe but so far you won’t. Do your brain pushups, don’t take the easy road b/c it goes nowhere and takes too much of my time.
Ryan is right and Roman is right and your expert is even right. The problem lies in the question you posed, and the answer does not conflict with anything done here that I’m aware of.
TCO, also, your examples relate to PCA without use of the covariance information (W*). If you use the covariance information explicitly, the “patterns” you talk about are already accounted for.
RomanM:
I think the Steig algorithm suffers from not keeping enough PCs. That is a DIFFERENT ISSUE from saying axiomatically negative weights in a recon are wrong in every conceivable situation.
We skeptics need to do a better job of disaggregating issues. Because I call Ryan on something like this where he makes a general statement, does not mean I’m supporting Steig. Because there are things wrong with Steig, does not make Ryan’s general statements true.
#284 TCO
You just don’t get it. It is irrelevant whether you consider the criticism as “axiomatic” or not. The observed effects need to be explained and justified.
We all know about negative correlation. Imagine anything you want, but we DON’T have any straw man Arctic “thermomoters” mixed in the Antarctic group (unless the temperature database is worse than we thought 😉 ) so that is misleading nonsense.
Either way, whether it is due to a flawed methodology or subtleties of the regional climate, it is a problem which needs to be addressed and the ball is in the court Steig et al. making the exaggerated claims for their reconstruction. Given everything else in the paper, my money is squarely on RegEM methodology.
(Jeff, take me out of Purgatory so we can talk real-time.)
283 and 287. I’m well aware that the negative weighting happens within the overall reconstruction, after going all the way through the reconstruction. That we are not just talking about an individual component of an individual PC. I’ve said so, well before this conversation.
Read my comments on the wiki thread for instance. Or read the examples that I, Carick and Baxter have ginned up. We can come up with situations where proper predictions are made for the overall average trend while retaining negative weightings of individual stations.
It’s you all who have the block here. In this case, even a block to understand that I understand! I GET IT that the overall trend line includes some negative weighted stations. I GET IT. This is what we are talking about.
My comments on PCA and on collinearity clearly showed how a difference PC might use a negative station and because of collinearity, that we could make up for it with another station.
285. You’re losing me, Jeff. Make your point again. Are you saying that an inverted function won’t have reversed slope?
286. Jeff, you think I’m wasting bandwidth and you let these kind of little ankle-biter no-content posts get through from your hoi polloi?
288. You’re repeating yourself.
289:
But my examples show how a predictor equation can be composed with “negative thermometers” for the overall TREND. So maybe all y’all’s pictures of negative thermometers and claims of unphysicallity are a little silly.
291.
A. You continue to confuse a discussion of an axiomatic issue with a defense of Steig. Steig is not right, because Ryan and Jeff have made overclaims about “physicallity” and postured about negative thermometers.
B. I agree with your money bet. And I agree that we should find out what the deal is with Steig. Note, this position is consistent with A.
C. But in the case of Arctic thermometers, Roman, you would still NOT have a “negative temperature”. You would have anti-correlation. The response is an extreme example versus Ryan’s AXIOMATIC assertions. Also, it is possible to have components of anti-correlation within Antarctical. Consider my God’s table/sound-speaker example. Consider Baxter’s very simple (1/3, 2/3) example.
——————————-
Maybe this will make it easier to understand. Imagine we are going through and investigating a shuttle crash and some formula that the computer generated that had all kinds of issues. In the course of that investigation, Ryan and Jeff say, “Hey! There’s a problem here since the second derivate of the function is zero and the program did not register that point as an inflection point.”
I intervene and clarify and say, no…look not every example with second derivative equal to zero is an inflection point. In general, yes, sure, that’s right. But I can construct an example (y=x^4) where the second derivative is zero, but the point is not an inflection point.
Now, when I do this, I’m NOT saying that this point that Ryan found was not an inflection point. And I’m NOT saying that NASA wasn’t all screwed up or even that they don’t have the burden of proof to figure out how their code works. What I’m doing is correcting an A Bridge too Far axiomatic remark that is incorrect. This is valuable for two reasons:
1. We don’t want people thinking the wrong thing about some general axiomatic issue which is not so, and going around thinking that every second derivative at zero is an inflection.
2. Ryan can’t cite the second derivative at zero as proving that point was an inflection. He needs to do some other test. (And maybe it really is an inflection. But his reasoning is not sufficient.)
TCO, you don’t get it. I never claimed that the negative thermometers can’t happen legitimately in any random reconstruction.
Steig is not claiming that the PCs are some abstract representation of temperature that can show a negative correlation to real temperature. Steig is claiming that the PCs ARE temperature. He does this both in words and mathematically – his 1982-2006 portion is entirely PCs. He claims that the PCs ARE temperature.
If they ARE temperature, and their covariance represents the actual temperature covariance, then negative thermometers ARE unphysical.
If the PCs are NOT temperature – but some abstract representation of temperature – then the negative thermometers aren’t really negative thermometers in the first place – the negative weights are simply calibrating the PCs to actual temperatures.
But if it is the latter case, then you CANNOT simply tack the PCs onto the end of the 1957-1982 reconstruction as Steig did because the PCs are not temperature.
I have no idea how a flipped temp measurement can be used in what is in this case a weighted average of a non-flipped reconstruction.
I think TCO is trying to ‘make’ this into a confusing point. I can’t even understand #294 as a post. If you want the best reconstruction, you don’t flip your data over.
#295 Ryan O
you said “”you CANNOT simply tack the PCs onto the end of the 1957-1982 reconstruction as Steig did.”” This was just scalar addition at each i, correct? IIRC, there was a complaint on an earlier thread concerning the problems (inappropriate) with no overlap, etc.
#297 Not quite. In Steig’s reconstruction, he took the raw PCs from the AVHRR data and used them entirely for the 1982-2006 portion. The 1982-2006 portion is not a reconstruction at all – it’s simply the AVHRR data, uncalibrated to ground data, as approximated by 3 PCs.
#298 When you say it like that, now it’s really bad.
Just kidding, I already knew. I like to word it as a reduced information version of the satellite data which happens to replace any recent cooling trend with a strong warming trend.
#299 It kind of makes you wonder why they used TLS at all. What sense does it make to use a regression method that assumes errors in both the ground data and the PCs if you throw away the post-1982 solution and simply assume the AVHRR PCs are perfect?
298: I thought he used the actual sattelite data after 1982? He used a PCA of it? Why not just use the actuals?
#301 Given that it’s never calibrated to the ground stations, my only answer is that I have no idea.
Jeff,
Can I make a couple of observations re this long thread. I sort of agree with TCO’s mate that sifting the wheat from the chaff can be ‘tiresome’. You know, I know, most observers know, that much of the to-and-fro is due to TCO’s approach. What I think we need is a synopsis of the discussion, perhaps as a new post, that brings out the issues that have been traversed in this thread, and sets them down as a neat discussion that summarises the different viewpoints. I would offer to do it. However, I am confident that I am not across the issues nearly as much as you, RyanO, Ken, RomanM, SM etc are, and I would surely make a mess of it, just creating more problems.
Also, re TCO, I appreciate your patience in tolerating him, even though sometimes it is clear that he frustrates you enormously. Personally, having watched TCO over several years now, at several sites, it is pretty clear that there is more than one TCO posting. There is a daytime TCO that is rational, incisive, and makes (perhaps) good points. There is another TCO who perhaps should not post. I learned long ago never to post if I have had a glass or two of red wine. Perhaps TCO needs to learn the same lesson. I reckon if you just set your moderation policy to reject all TCO posts between say 7:00 pm and 11:00 am you might find the discussion runs at a higher level.
It is also clear that TCO is primarily a stirrer, as he himself acknowledges. He does seem to have good critical thinking skills, but as you point out, on some issues seems not to have done the reading/work to actually be in the same space as the other experts, thus embarrassing himself (is that possible?) and frustrating the other players.
#TCO 301
From your comment, it is pretty clear that you didn’t bother reading the explanation that was written specifically for you in #282. I clearly state the fact about the three PCs replacing the raw satellite data in the writeup but you sluffed it off without reading and understanding the information.
If the raw data were to be used, you would be using RegEM to reconstruct 5509 sequences (instead of only 3) using only the highly incomplete manned station data from the pre-satellite era. Calculationally it wouldn’t be tractable for a variety of reasons: huge matrices from which PCs and eigenvalues needed to be calculated and the likelihood that convergence of RegEM results would not occur or, given that they did converge, results would be spurious (even more so than in the reduced problem).
Instead, they chose to use a starting point of only three PCs automatically limiting their ability to properly capture the complexity of the antarctic climate system. Once the information is not there, it is NOT there! This effect is exactly what Ryan and Jeff have been trying to evaluate.
304:
I been seeing signs of this and asking questions for a while. No offense intended. You are one of the class of this forum, so appreciate your responses to me.
Digging into the content.
A. What’s to stop them using satts post 82 and the PCA stuff pre-82? Even pre-82, with 3 PCs (3 series), my impression was that they still had some grid mask, where they said how much the PCs contributed at each given grid square. So even though the true doF are less in the pre82 version, it still has the same grid resolution. No?
B. Is there some better alogorithm (than RegEM) which would allow for more PCs? Or do a better signal extraction usage of the multiple stations and their correlation to sattelite temps (iow the basic idea of the recon, but with a different engine)?
Ryan: You had some nice comments on the TTLS thread. Glad to see you having better insights than Steve. Also, glad that you are not area, area, area like JeffId is.
TCO, maybe you have finally figured out the negative thermometer thing. If so, then good. If not go back and read your post #294 and Ryan’s response #295. I think this has been the crux of your misunderstanding for this whole thread. No? Surely you are not going to continue arguing now that you see your point in #294 was based on a false premise.
Ok RomanM: I will engage with that post:
“TCO, you don’t get it. I never claimed that the negative thermometers can’t happen legitimately in any random reconstruction.”
You said they were shit, shit, shit. Also if they are “unphysical”, surely they are unphysical in general. If you want, I will tediously go back and quote all the general, flat statements you made about how negative weighted stations were wrong.
REPLY: I believe you are referring to Ryan’s statement about flipping of thermometers not Roman.
“Steig is not claiming that the PCs are some abstract representation of temperature that can show a negative correlation to real temperature. Steig is claiming that the PCs ARE temperature.”
A. I think what he claims is that the grid-square temps are temps and they are composed of a summation of PCs.
REPLY:Yes
B. Does he have any negative grid squares?
REPLY: It doesn’t have anything to do with the issue at hand
C. Does he have actual negative temperature? He’s probably using centered data, so a “negative” is not negative, but a negative deviation from a reference value. So much for the negative thermometer pictures of Watts.
REPLY:Negative is not the issue, FLIPPED OVER is the issue
D. Also, the picture from Jeff shows negative weighted STATIONS, which are PARTS of PCs, Ryan. Go back and look at Jeff’s figure, if you doubt me.
REPLY:The negative weights apply to the individual temperature station data – not pc’s
E. The PCs are not temperature, Ryan. Nor is Steig claiming that. The temp is a feature of the grid square. The “shape” of the PCs is not a characteristic of the PC, Ryan, but a characteristic of the region. (When you do the mathematical PCA itself all you do is maximize variance of the inputs, and geo data is not a part of that process.) You are getting yourself confused thinking about the match of the surface (PC weighting) as part of the PC. If you are saying a PC itself (the actual series) may never take a negative value, when it’s centered data, that’s just bizarre.
REPLY:I didn’t like the way Ryan worded that one, he’s still right but the wording is confusing. Your statement in A is correct
“He does this both in words and mathematically – his 1982-2006 portion is entirely PCs. He claims that the PCs ARE temperature.”
It’s an independant issue from the negative thermometer kerfuffle, but I’m (still) currious (admitting ignorance) as to what Steig uses for the post 82 recon. I thought you all had said earlier that the actual RECON uses the sattelite data, post 82.
REPLY:Steig uses 3 pc version of the uncalibrated satellite data for post 1982
“If they ARE temperature, and their covariance represents the actual temperature covariance, then negative thermometers ARE unphysical.”
See all before, but in particular the issue of station weighting being what Jeff graphed.
“If the PCs are NOT temperature – but some abstract representation of temperature – then the negative thermometers aren’t really negative thermometers in the first place – the negative weights are simply calibrating the PCs to actual temperatures.”
No duh.
“But if it is the latter case, then you CANNOT simply tack the PCs onto the end of the 1957-1982 reconstruction as Steig did because the PCs are not temperature.”
They are what they are, Ryan. If Steig did a separate sin in his splice, then that’s a different issue.
REPLY:My back calculation is a determination of how you add the thermometers together to get the final output from RegEM. The addition of the thermometers included some strong negative values. It’s saying- I have ten thermometers in the room so the average temperature in the room is plus some and minus others. Clearly if you have ten thermometers in the room and you want to know temperature you don’t subtract 3. —EVER!
308:
A. Saying RomanM, was a mistake. What I meant to say was, “Layman, I will respond to Ryan’s post, as you requested.
B. It’s a separate issue from the negative thermometers debate, but I HATE the idea of using the grid of 3PC impact for the post 1982 time. I mean you have the data. Use that. Just splice the damn thing with duct tape. But use the satt coverage where you have it, for what it’s worth. I mean heck, if you had satts all the way back to 1957, would you PC that? Should UAH and RSS global temp series be based on some PC version or the actual satt data?
C. (On the reply, you’re just repeating yourself. I can (and Carrick and Baxter have) come up with examples where predictions of the overall temp anti-correlate to a small area. If you then combine that phenomenon with non-equal sampling (and this is very conceiveable, that we have limited set of sensors), then an overall prediction might use some of that small area in inverted form.
I don’t agree that Carrick or Baxter’s examples do what you say.
TCO, I disagree that post 1982 use of 3 pc is a separate issue. It is a necessary part of the method that produced the negative thermometers. Ryan’s argument has never been broad based axiomatic on this matter. He has argued all along that the negative thermometers are a symptom of a faulty method and that any other work using similar method flaws would produce the “shit, shit, shit”. Perhaps most of the method defect lies in reduction of AVHRR to 3 PC’s which forces higher order complex patterns into low order oscilating modes.
Now that you understand what post 1982 is represented by, do you not see how this ties in with negative thermometers? Jeff has already shown that the post 1982 (not just pre 1982) temp trends have been distorted both spatially and temporally.
Layman Lurker for TCO to admit he learned anything positive here would assume he was here to learn and not to merely hold the line that these skeptic blogs are not sufficiently serious in their analyses -and that by him, simply and nonthinkingly, remaining contrary that their weaknesses will pop out.
Both Jeff and Ryan have made point blank statements against the negative weightings as being simply and always wrong. Jeff even has a comment up thread a few posts, where he says he would NEVER hhave an inverted sensor component when measuring a room.
REPLY: TCO you are not being an honest critic. You seem to get the point that the ‘sensor’ is inverted now, where just a few comments before you didn’t and before that you did. Now you neglect to mention that the ‘sensor’ is a thermometer and you are using it to measure temperature.
I don’t like dishonest TCO.
Jeff, I agree that your comments were on thermometers. no special import attached to thermomoter versus sensor. Don’t read anything into it one way or another.
HOWEVER, you can have a reconstruction IN THAT ROOM OF YOURS that weights a thermometer negative, Jeff. For instance if there is a spot in the room, that is anticorrelated with the average and it is your only sensor, you will end up having it negative weighted (centered data of course). This is the EXACT same thing Baxter and I have been telling you for a while. And when you do that it DOESN’T mean temps were negative. Doesn’t mean Watts needs to make pictures of negative thermometers. It’s a completely understandable rationale for having that sensor (thermometer) weighted negative in the RECON.
GAAAAAAH!!!
REPLY: Inverted temperature is non-physical and spurious TCO as is your non-physical example. A ‘perfectly’ anti-correlated temeprature does not allow for net warming or cooling and is non-physical. It’s not my fault TCO, temperatures don’t work the way you want. While John Baxters point that regression can compensate for the errors is correct, a proper reconstruction will not have spuriously inverted temperatures.
This is what I mean when I tell people, physics is a harsh mistress. We don’t get to argue with it.
So Blame god, not me.
TCO, you have no idea what you are talking about. There have been several points where you’ve stated that the stations are somehow “part” of the PCs:
WTF? Stations are “parts” of PCs? You obviously do not know what a PC is or how the station data relates to the PCs. This statement is entirely nonsensical.
Again . . . WTF??? This makes no sense whatsoever. The stations and PCs are entirely separate entities.
Holy WTF, Batman. Can your ignorance of the mathematics get any deeper? Did you even bother to read any reference about EOF analysis? Do you have any idea what the mathematical definition of the shape is? If you’re not going to take even a couple of minutes to even try to understand what it is you purport to criticize, I am not going to take any time whatsoever to explain it to you.
And you don’t understand EOFs:
Grid mask??? You mean the left singular vector, or shape. This is the W* matrix in the Wiki link I gave you, which you have obviously refused to read. I’m not going to waste my time explaining EOF analysis anymore if you are unwilling to even read the links I provide.
And you do not understand the purpose of the PC decomposition and RegEM, as evidenced by this statement which completely conflates the two steps:
And the real winner:
Try having even a basic understanding of what is going on before asking others to “disaggregate” issues that you cannot even correctly identify.
In short, TCO, I’m done. I have given you the benefit of the doubt on every occasion, but you have chosen not to even bother reading anything about the mathematics behind the reconstruction. You have no competence to question whether negative weights are valid or invalid because you do not understand what a negative weight means. You conflate weights with covariance. You conflate station data with EOF shapes. You don’t even know what an EOF is. You don’t understand the difference between the PC decomposition of the AVHRR data and RegEM.
Instead, you simply parrot what you think other, more competent people (like Baxter and Carrick) are saying. After this last exchange, you have proven that you do not even understand what they were saying because you have shown you do not understand the mathematical context.
Criticize at will, TCO, but do not expect any more responses from me. I will save my responses for others.
Ryan: You may be quite right, that I am wrong and staying stupid things about the PCs being weighted samplings of the stations. That’s cool. That you’re right on that. It is a nonconsequential point, though, since what we are talking about is the recon itself which consists at the end of the day as a weighted sampling of the stations. The issue is that you and Jeff have been saying it is impossible to have a recon that includes negative stations. We have been giving examples of how they can. If you want to obsess on my boner and ignore the real issue of debate, fine.
Looks like the Wiki expert is still backing me up as well “not a red light”:
http://en.wikipedia.org/w/index.php?title=Talk%3APrincipal_component_analysis&diff=297879993&oldid=297772344
#316
As Yoda said, “And that is why you fail.”
By not understanding what the parts are, and by not understanding how they interact during calibration and reconstruction (and the interaction is different between the two periods), you fail to understand how your examples are not valid.
If you pay attention carefully, you will note that I have said that negative weights during the calibration period – where you have complete data and you can compensate one negative weight with an excessively positive weight – can, indeed, give the right answer. This is in both the stock price and MPG examples.
The problem comes when you extrapolate using only a subset of the data. Unless both the negative weight station and the positive weight station exist at all times during the extrapolation and both have stationary means, the resulting reconstruction will yield invalid results. See also the comment at CA about this.
Rather than me simply telling you, would you like to guess as to whether the station means are stationary?
Understanding the math is absolutely critical, TCO. Because you do not understand how the reconstruction is done and what the components are, you do not understand the differences between the calibration period and the reconstruction period. I have said before, your examples (once adjusted to appropriately reflect what PCs and EOF analysis actually is) may be – but are not necessarily – valid during calibration. However, they are not valid during reconstruction.
By not understanding the issues you criticize, it is you who is avoiding the real issue of the debate.
Ryan, I’m tired of this point with TCO as you are. I’ve had to make two entries now to Wiki, which I don’t know or like or whatever because TCO has confused some guy who TCO presents as the expert. His questions are off of course and the result is a confused expert – followed by an indignant and incorrect TCO.
Yah, I think I’m done.
The math is easy to justify for a paper, regardless.
BTW, the going is slow for the post on using the eigenvector weights. While the results look good, my math was wrong the first time through. Based on the original AVHRR data being simply the sum of each PC*eigenvector, I’m almost certain I have to subtract each imputed PC*eigenvector from the appropriate stations before imputing the next one. I posed a question for Roman on CA to make sure I’m not being stupid. So I’m somewhat stalled while I modify the script to do that and verify that I’ve done it properly.
#320, I wonder if you’re working your way to a true regression, removing every uncouth interaction one at a time engineering style. Of course, I don’t understand fully what you are doing. Without the code, it’s obscure.
That’s another reason I can’t understand TCO. He hasn’t run the code.
A. Release 17.
B. The wikii guy still backs me up. Negative weight is not a “red light” even after the clarification.
C. Baxter and Carick and I have given several examples where the predictor would be negative weighted in the recon. The simplest conceiveable example is a small area that is anti-correlated with the average, but is the only sensor availble for the recon. I’ve also given more elaborate examples.
#321 I had the same thought about the regression. The RegEM part of the PC imputation is getting damn close to being unnecessary.
I don’t see a way around the RegEM part for the ground station matrix, though.
#312 Kenneth Fritsch
Alas, you are probably right Kenneth, but it sure is fun trying to convince him. 😉
Call me twisted, but I have enjoyed the thread. I know it has been frustrating for Ryan and Jeff dealing with TCO, but for me any discussion helps me think the logic chain through. Because I am not working the math through myself this is important (still have “R” on my list of things to do Jeff 🙂 ).
0. Sorry to be a latecomer to this thread, but I can’t help but feel that some of the more extreme rhetoric about “negative thermometers” is rather overstated.
While I’m not saying the negative weights here are necessarily right, there are situations where they can indeed make perfect sense. It is not correct to see negative weights as necessarily in themselves a “slam dunk” against the paper.
1. As a simplified example, suppose you were interested in a 1D system, say a metal bar perhaps, which you were modelling as having a constant temperature gradient; and suppose you want the average temperature between x=0 and x=5. What estimate should you give for that average, if you have two thermometers, located at x=0 and x=1 ?
According to the model, the temperature y(x) is being taken to go as
: y = m x + c
so the temperature integrated from x=0 to x=5 is
: int y dx = (25/2) m + 5 c
so the average temperature is
: (1/5) int y dx = (5/2) m + c
substituting now our readings y0 from the thermometer at x=0, and y1 from the thermometer at x=1, gives
: = (5/2) (y1 – y0) + y0
: = (5/2) y1 – (3/2) y0
So in this case, even though thermometers measure real positive temperatures, etc., etc., etc., nevertheless this is a case where it can make perfect sense to give one of the thermometers a negative weighting for calculating the average.
2. What is happening in the paper is not so qualitatively dissimilar to what is happening above. Above we’re using two basis functions to model the temperature distribution – a constant term, and a linear ramp; that’s not so different from the paper, which is using three basis functions.
I don’t find it beyond the bounds of possibility that one of the basis functions correlates with the difference between some stations and some other stations, so that those stations would have negative weights towards the overall strength given to that basis function; nor is it impossible that, when extended across the whole of the rest of Antarctica, that basis function may make more of a contribution to the continent-wide average temperature than a basis function for which the stations had a postitive weighting; so I don’t see it as *impossible* that some of the thermometers could end up with a negative weighting.
Saying simply then just that such a negative weighting is “illogical” or “unphysical” therefore seems to me not to cut it. Such rhetoric simply ain’t so.
That’s why I wrote elsewhere that such negative weightings shouldn’t be seen in themelves as an automatic red “stop” sign. There are cases when they could be right.
3. But it may be appropriate to see them as an amber “caution” light.
The main thing going on in this paper, as far as I have picked up, is basically the following regression:
Down-projected satellite series (300 x 3) = Surface series (300 x 24) . Unknown Projectors (24 x 3)
which is essentially of the standard regression form y = M beta
Having estimated the appropriate projectors, the equation is then used in reverse to project from the first 25 years of the surface series to create 25 years of synthetic reduced satellite series.
It’s made slightly trickier because the surface series is dirty (with missing data) and noisy, but that is essentially what is going on.
Now in any regression, the big underlying risk to be on guard against is “overfit”. Essentially, we only expect the regression variables ‘beta’ to be able to account for some of the signal in the obervations ‘y’. Beyond that, there will be some signal ‘y’ which is not accounted for by the regression variables. As far as the regression is concerned, this unaccounted-for part of y is “noise”. Equally in the regression there is some inevitable statistical uncertainty in the value of beta. The danger is that, in general, some of this uncertainty can be used to fit a bit more of ‘y’ than one really should have. This is not a true fit, instead the extra that has been fitted is noise, and it has been fitted by getting the values of beta slightly wrong. This is what is known as “overfit”. It has two serious consequences. Firstly, the values of beta are wrong, so the fit on any new data (not seen in the regression step) is actually made worse. Secondly, because the predictor has apparently done better than it actually should have done, the amount of unfitted signal in y (“noise”) is underestimated, which in turn means that the true amount of uncertainty in subsequent out-of-sample prediction may also be underestimated.
These directions of greatest uncertainty in beta correspond to the directions associated with the smallest singular values in an SVD of the matrix M. Corresponding small changes in y get magnified by the reciprocal of this singular value; so if overfit is allowed to go unchecked, small amounts of overfitting of y can produce very big swings in beta parallel to these ‘least significant’ singular vector directions. That can lead to regression coefficients substantially offset – positively and negatively – from what one might expect. Typically instead of regression coefficients all quite small, one therefore sees regression coefficients unduly strongly positive, or unduly strongly negative.
The big question is: has that happened here?
And that is where the presence of some comparatively large positive and large negative weights at least deserves an amber “caution” light.
4. Rather than so much shouting that the negative thermometer weights *must* in their very nature be wrong (which isn’t true), more credible therefore would be the quieter suggestion that the negative weights *might* be wrong, accompanied by some auditing to investigate further.
The first question should be, what sort of condition numbers were found for the matrix M (and was anything done about them?)
If the singular values are all quite large (as they might be, since M is a much taller matrix than it is wide), then you have to fit a lot of y to earn a small change in beta, and overfit is less likely.
But if the last few singular values get much smaller, then large changes in beta are being caused by small (possibly quite spurious) “improvements” in the fitting of y; and if that is happening, it is better to ignore these singular directions completely.
So how small is too small? There is no certain answer. But one approach, which could be easily checked, is to use cross-validation: run the regression using only 200 members of the timeseries, and then increase the value of the cut-off parameter for the smallest acceptable singular value, until the remaining 100 instances in the timeseries are being fitted no worse than the 200 being regressed against. Having got the right value of the cut-off, a “production” version of the regression can then be run, using all 300 members of the time series but with the now identified correct setting of the cut-off.
This cut-off doesn’t have to be done in an all-or-nothing way; “soft” versions also exist, which leave the effect of the largest singular values almost completely unchanged, but scale back the effects of smaller ones sharply as they fall below a critical threshold. This is called “ridge regression”, and can be shown to be equivalent to putting a weak prior distribution on what the values of beta ‘should’ be. This has little effect on directions strongly determined by the data, but prevents beta running off after the data in directions only weakly determined by the data (ie ones which correspond to the smallest singular values of M). As an alternative to cross-validation approach set out above, there are also Bayesian methods to estimate the most appropriate strength of the ridge parameter; and it can be useful to compare the results of the two approaches.
This is all something concrete and quantifiable which could be done, to check whether the regression presented in the paper does actually look right or not.
5. The final thing I would wonder, is does the paper present a sufficiently detailed audit trail of how the uncertainties (ie error bars) in the slope of its final trend line were assessed?
In particular, were the uncertainties estimated in the projection matrix “beta”, and in the signal variance left unfitted by the regression both calculated, and both used to propagated forwards appropriately assessed uncertainties for the “early years” synthetic reduced satellite time-series?
How much different could alternative reconstructed “early years” series be, yet still remain within the bounds compatible with the regression?
My worry is that done properly, with a proper eye to prevent overfit, the regression may be able to capture only some of reduced satellite signal, and therefore only be able to synthesise part of it, leaving a substantial part unexplained and later unsynthesised. It also may not be correct to assume this missing signal power, unfitted and therefore not correspondingly put into the synthesis, is necessarily temporally uncorrelated and independent.
6. I have to confess though, that I haven’t read all the posts and all the threads, so perhaps some of these points have already been addressed. If that is the case, would anybody like to give a summary to bring me up to speed?
JH: I appreciate your settling of this minor over-statement by Jeff and Ryan. I do think that there may be other issues/concerns with the Steig paper. My major worry is the PCA, where a significant amount of variance is dropped from the examination, by condensation to 3PCs. I also had a worry that the true uncertainty might not being taken all the way through the process. Would welcome your attention to the Steig paper itself on that subject, since it’s too technical for me, to really have more than gut feelings.
I think it would be rather hard to go through the whole set of threads on the Steig paper. Probably best to just read the paper itself, come up with your own opinion, and perhaps sample some of the threads and comments here. Those by Ryan are stronger than those by Jeff Id (not a slight, Jeff. I figure you would agree.)
I think the stuff here is more head scratching and thought starting, than definitive. So would view the threads in that manner. Unfortunately Jeff gets a little carried away at times and thinks he has the smoking gun, when it’s a much more ambiguous clue. But please look past that.
JH:
Okay. Now assume your metal bar heats up over time (i.e., the mean is nonstationary). Predict the future temperature using only y0.
Then you tell me if predictors with negative weights give the correct answer.
Ryan: assuming that you keep the gradient, but just over time change the temp, your average temp at any given time, will just be the same combination of y0 and y1 (at any given time) and the trend line of average temp casn be constructed from that.
If you only have y0, that changes the problem and now you would use y0 as a positive weighted component.
I’m not sure why you mixed the two different issues at the same time, since the first pointis trivial and the second one is really what you are looking for.
Also, Ryan, if you are now admitting that the yo and y1 situation gives a negative weighting and that the y0 only gives a positive weighting, it is tantamount to saying that sometimes you have all positive and sometimes you have some negative weights. Which is tantamount to saying you can’t just aximoatically say negative weightings are always shit. Maybe you are in a yo/y1 situation.
JH, I will answer in the morning.
Well, as long as you’re upa nd moving on to other threads, howzabout apporving my posts? You can answer them too in the morning. But don’t be like Real Climate and not let something show until you have responses ready…
TCO you are a dishonest troll. Before I gave the benefit of the doubt, honest trolls are welcome to post…any time.
I don’t want to read your posts on SM… he’s not here. Read the detail and tell me the error, lately you’ve been proven wrong over and over and yet you still don’t admit your error. You’ve attacked my posts before even reading them. What is the intent of that?? I’m serious, you give me honesty or you get the guillotine. I’m sick to death of the government, idiots and liars. Only the honest can post here, anything less and I’m wasting my time!
I’m sorry you’re upset. Please let the posts through. They are honest arguments and contain content.
Jeff: Shake your manly hand.
Actually the more I think about it, JH’s example is just degeneracy, since y0 and y1 are related and therefore average temp can be done with many different combinations of weightings. (We’ve shown this to Ryan before several times.) I do wonfer if there are more complex problems, addition of noise, non-perfect linear relationships, multiple dimensions, etc. where the prescence of negative weightings is required, rather than just possible.
TCO:
#329 You don’t get to change the weighting of y0. The weight is determined during the calibration period. You by definition do not have any information to change it during the reconstruction period (else you wouldn’t need to do the reconstruction). You don’t know what the average temperature is – that’s what you’re trying to reconstruct! And if you only have the negative weighted component (and, by definition, no ex ante knowledge that the mean is nonstationary), then you have no information you can use to change the sign.
Your answer proves my point: in order to have a valid prediction using just y0, it must have a positive weight. The fact that you admit that the weight of y0 must be positive for the prediction to be valid means you agree that I am correct.
If the mean is nonstationary – i.e., the average temperature increases – then negative weights will yield incorrect predictions unless the system is truly divergent (and air temperature is definitely not).
This is exactly the problem in Steig’s reconstruction. During time A (calibration), he has y1 and y0. He uses that period of overlap to determine a weight of 5/2 for y1 and -3/2 for y0; and, as a corollary, a relationship of y1 = -3/5 y0. Then, during time B (reconstruction), he only has y0 and the only available relationships are those determined during calibration. It is trivial to show that the negatively weighted y0 will predict an incorrect average temperature and an incorrect temperature for y1 if the mean is nonstationary.
Steig’s situation is more complex because we don’t even know which y1 goes with which y0, so we can’t even estimate the potential error due to the combination of negatively weighted predictors and nonstationary means because we don’t know which y1 provided the offset for which y0 during calibration. All we do know that there are significant periods where many of them are not present.
336.
A. If y0 and y1 correspond to stations used in the recon, certainly he does know which ones exist in the earlier period. He writes a lot about just using the continually manned stations.
B. I agree that if you only have y0, it will be positive. However, if you have both, it may not be. Since the y0 and y1 correspond to stations, then certainly any situation corresponding to a y0/y1 situation may include negative weights. That is why your flat statement that it MAY NOT, is APHYSICAL is wrong. All it takes is a single counter-example to disprove a thereom.
C. If you did, for some bizarre reason, form a general algorithm that weights various stations for the recon y0, y1, y2…y40, ANd THEN you may not have all those stations, sayin a future projection, OF COURSE your average temp may then be dramatically off! you can’t train on one thing and then just take stations away and no retrain.
————————————–
Ryan (and Jeff), you’ve had multiple people disagree with you and supply cogent examples. Good people like Baxter and Carrick and now JH. We just keep drafting the same example after example and you all just keep fighting and saying “negative thermometer”. Show me the textbook or review article or what have you that supports your position, that you may never have negative weightings, that the 5 stations swung significantly negative MUST be a sign of something wrong.
Jeff: Take me out of the penalty box. I’ve been proved right on this tiny issue, haven’t cursed in ages, and can’t keep up with defiant failure to grock without the ability to reply more real time.
Nope, not yet. I still have strong doubts on your honesty. You criticize work before even reading it and I still can’t make sense out of your mathematical discussions.
JH, Thanks for the reply. It’s an interesting example you present which I cannot disagree with. I don’t disagree that there are situations where average temperature could be calculated using a negative weight and you are right that some of my statements are too strong without the context of this paper. — how’s that for an answer.
I think it was Carrick that presented a similar example to yours where each half of the antarctic was perfectly anti-correlated somewhere in this ungodly long thread.
In your example you have assumed a correct ‘locked in’ relationship of the signal between temperatures at two points on a bar for both short and long term temperature which results in the two stations being perfectly related and a predictor which at the opposite (distant) end of the bar is perfectly anti-correlated that has a perfect anti-covariance. The negative coefficient comes from having neither thermometer past the halfway point of the bar to measure the anti-covariance.
This is different from the situation in the Antarctic in which satellite grid points are not originally pre-PCA locked together by an inverse linear relationship but can vary together and against each other and that’s part of where my problem with the solution lies. Since they are free to measure different temperatures and since strong anti-covariance with no thermometers nearby is not the true situation in the Antarctic the result is non-physical.
But the problem is worsened by PCA, here’s what I wrote in an earlier comment on this thread:
In centered PCA PC1 takes the trend out of the data, PC2 and 3 end up splitting the data in halves representing the perfect anti-covariance of continental halves in this case. I don’t know if you’ve seen the Chiladni patterns on CA, but it’s worth reading. The fact that we’re using PC data rather than actual creates a non-natural, non-physical, anti-covariance situation causing some stations to be applied negatively to the reconstruction weight.
And just to make it more confusing and more different from the bar example, the inverted peninsula stations are right next to strong positive stations so they can be reasonably canceled out in the final run and their presence may represent a case of overfitting due to lack of covariance information in the PC data.
Anti-covariance from negative stations must be explained by a physical effect – this was not done in the paper. The fact that the stations came from one region with very high density of measurements means that the result is more likely due to overfitting and is a failure of the algorithm to isolate these stations information to their appropriate locations.
Jeff:
Shake hand again.
I think your picking apart at the Steig algorithm and at algorithms in general is good stuff. Keep it up. I hope you really run it all down, figure it all out, however it shakes out.
One thing I’m wondering is what are the general shapes and weightings of PCs themselves? IOW, don’t the PCs consist of weightings of the individual stations (Ryan feel free to pounce). So is it possible that some of these PCs correspond roughly to differencing regions? If so, does that require negative weightings or is negative weighting still a sign of degeneracy?
Also, is it reasonable to expect that we WILL and DO have some degeneracy? I mean we are talking thermometers, not tree-mometers and some are quite close. Are those effectively duplicates?
I think this discussion has devolved at the margins into what others view as Jeff and Ryan’s stand on negative coefficients in general and whether Jeff and Ryan have shown that in the Steig case that “negative thermometers” are unphysical and therefore an indicator of a problem with the method. When a poster comes on here without reading or understanding the details of the Steig methods and the Jeff and Ryan analysis and comments on a more general (and hypothetical case) we are getting further away from real issues at hand.
Why not simply end the discussion until someone wants to offer a reasonable explanation for the negative thermometers in the Steig reconstruction and not in some general hypothetical case.
#339 I don’t understand your interpretation of the word degeneracy and I don’t want you to mistake my post for changing my mind.
I think there are multiple problems with the reconstruction including the use of inverted thermometers. Which as I stated before (not even sure where it was) that this point and the peninsula overweighting are minor points like dustballs in the corners of the reconstruction. This is just one more sign that Steig et al didn’t perform as advertised. Ryan’s repair of the work is the primary solution, which I trust slightly less than an area weighted version for trend accuracy. We still need to check on Ryan’s trend weightings, maybe he has some strong negative ones also!
It’s pretty obvious to me that these particular inverted thermometers are non-physical rubbish and are artifacts of failing math rather than an unexplained anti-covariance.
Jeff, you’ll have to tell me whether I’m getting this right, but there seem to be two different possible current key assertions in the case against Steig:
(1) There’s a problem caused by truncating at too few PCs.
It would be different if PC1 represented the average temperature over the whole continent, and all the other PCs made zero net contribution to the average temperature. But that isn’t how the basis functions have been constructed. Instead PC2..PCn make non-zero contributions to the average temperature at a given instant in time; and in fact several of PC4..PCn show a distinct cooling trend with time, that has been lost by truncating at PC3.
(2) You don’t believe the regression coefficients for each PC, calculated from comparing the time-series, used to project from the land stations to the satellite PCs.
— for example, they don’t correspond to the geographical highs and lows of the PCs.
(Yes, I know that your argument immediately above was actually against the coefficients of the *total*, rather than the weights used to project a coefficient for each PC. But I think the latter is more interesting to ask whether or not you think there is another possible error in play here, over and above the throwing away of some potentially important basis functions? Or on the other hand does it seem that Steig’s projection coefficients are actually correct, and it really is only that he hasn’t considered enough PCs ? (as some of the posts about this paper seem content to take as their starting point)
Unfortunately, (at least at the level of PCs) I’m not sure it’s enough to say that Steig’s weights, plotted geographically, don’t match the PC plotted geographically.
The problem here is that the PCs aren’t orthogonalised over the station locations; so we wouldn’t be able to just transpose the matrix of PC values at the station locations, retaining their geographical coherence. Instead there we have to invert the matrix (or more accurately, to find its pseudo-inverse)
If we want
y (24 x 1) to more-or-less match G (24 x n) . a (n x 1)
where G_ij is the value of basis function #j at thermometer #i, then the weightings we have to apply to the data vector y are given by G*, the Moore-Penrose pseudo-inverse of G, not G itself; i.e.
G* (n x 24) y (24 x 1) = a (n x 1)
The weights represented by each row of G* would be interesting, and I’d be interested to know how much geographical coherence they show; but they won’t be exactly the same (nor show the same amount of geographical coherence) as the elements of G.
Thus the observation that Steig’s station weights for each PC don’t look like the PC values at each station isn’t quite enough to clinch it. What do the rows of G* look like, and how closely do they compare to Steig’s weights for each PC?
Apart from that, is there any other evidence to suggest whether Steig may or may not have over-fitted his time series regression? I believe you’ve managed to replicate the RegEM algorithm. So what are the properties of the cleaned-up M matrix? Is it well-conditioned, or does it have some singular values that start to tail off? And what are the cross-validation properties of the regression? How well does RegEM run for 200 points of satellite PC data manage to predict the next 100?
These are important questions, I think.
But I’m keen to know your working hypothesis at this stage. Do you think RegEM is producing accurate regression coefficients (just not for enough PCs)? Or do you think it is overfitting?
JH, You have a lot of questions, I think the best method would be to skim through these posts which contain some of the key posts on RegEM in Steig et al.
First is an area weighted version of the reconstruction by myself.
https://noconsensus.wordpress.com/2009/04/12/closest-station-antarctic-reconstruction/
Then we began doing other reconstructions. – this was the original replication.
https://noconsensus.wordpress.com/2009/04/05/steig-avhrr-reconstruction-from-satellite-data/
SteveM noticed spatial blending of the data in the satellite form.
https://noconsensus.wordpress.com/2009/03/01/2547/
Ryan used SteveM’s code rather than RegEM with surface data only. Satellite data overlays the surface data.
https://noconsensus.wordpress.com/2009/05/01/steig-et-al-a-successful-reproduction/
Ryan delivers a more accurate reconstruciton.
https://noconsensus.wordpress.com/2009/05/20/antarctic-coup-de-grace/
Ryan verifies that the calibration coefficients are substantially better than Steig et al.
https://noconsensus.wordpress.com/2009/05/28/verification-of-the-improved-high-pc-reconstruction/
My own post back calculating weights. The pie charts and station names are incorrect but the weights themselves are close. The station name change cuts the peninsula contribution by 1/3 according to Nic L. It’s still more than what I expected.
https://noconsensus.wordpress.com/2009/06/07/antarctic-warming-the-final-straw/
My own post on improving the weight calculation.
https://noconsensus.wordpress.com/2009/06/15/improved-weight-calculation/
I have a new post by Nic L coming tonight which uses a unique brute force approach to getting the trend contribution of each station.
341. Jeff:
A. When I use the word degenerate, I mean that two sensors are either identical or indentical with some linear scaling. Yn=kYm. Is this usage incorrect?
B. I wasn’t trying to lure you into admitting anything. Honest, boss. Just canoodling on the general (and specific for Steig) aspects of weightings. I think this can turn and shed some light on the discussion as to the negative weightings being a trouble factor. But that wasn’t my immediate interest. More just thinking through things in general. HONEST.
TCO: were you just canoodling when you compared my Mann08 calculations to cold fusion. As though you might have a clue…. You don’t get it, yet you criticize. This is not honest behavior.
You use words you’ve heard in the threads like mush, read the definition of covariance, variance, correlation, standard deviation and learn to work with it.
#337
TCO, that is exactly what is happening. The station data is missing in large chunks. You only have periods of overlap to do the training, and the period of overlap between stations that show enough correlation for RegEM to make a decent prediction may be short or nonexistent. This is the training.
Outside of the training is the reconstruction period. The weights that were assigned in training (“calibration”) are used throughout the reconstruction period. They are not modified in the reconstruction period. So if the mean is nonstationary in the reconstruction period, the negative predictors will give unphysical results. You yourself have admitted this.
For the life of me, I cannot fathom how you believe you’ve won on this point when you’ve agreed in successive posts that, without a recalibration (which you by definition don’t have the option of doing), negative weights will yield unphysical results if the mean is nonstationary.
JH,
You may also want to look at this, which explains an even more fundamental mathematical problem with Steig. It’s a letter I sent to Dr. Beckers asking for his comments on our infilling method. In the letter are 3 major problems with the infilling and PC regression done by Steig, as well as our proposed solution.
Click to access Dr_Beckers.pdf
Ryan:
I really hope you are not devolving to an “I never said that argument” Here are just a few of the general statements made against negative weighting of temperature by you or Jeff:
“It is of course nonsensical to flip temperature data upside down when averaging but that is exactly what Steig et al does. This ALONE [emphasis added] should call into question the paper’s result.”
“At the moment, I don’t like negative weights. Negative weight*trend is okay, though . . . but if something ends up with a negative weight, my initial thought is that it shouldn’t be included in the first place. Have to think about that one.”
“Anti-thermometers (negative weight) don’t make sense. If the values are small enough it might not matter. In this case they are large.”
“The pie chart wasn’t the aha of the last post, although it was the best visual evidence. Ryan, some commentors and others picked up on this right away. The big deal of the last post was that 5 of 34 – 15% of the manned temperature stations were inverted. Completely flipped upside down. What’s more is that they came with heavy weights and trends. This is the #1 problem discovered this weekend. Even if the peninsula weighting was zero it didn’t matter, the real issue is the mystery of the upside down thermometers.
Claims that it is somehow ok to flip thermometers and average them together because the paper get’s the desired result are advocacy – not science.”
“It cannot under any form of reality be acceptable to invert a temperature anomaly, the only response is for them to say – it didn’t have a noticeable effect. ”
” a thermometer is a thermometer and it measures thermo not anti-thermo.”
“A negative thermometer makes no physical sense. None. Period.”
“There is absolutely unequivocally no possible rational by which measured thermal variance can be flipped upside down under any circumstances. The temperature is the temperature as measured by thermometers.”
” you are thick-headedly missing the single biggest problem with the reconstruction – upside down temperature data are nonsensical.”
“This does NOT mean that you cannot have a negative weight in a regression, it means you cannot have a strong negative weight in a weighted average of thermometer data”
“inverting any thermometer makes zero sense in a physical world.”
——————————-
This only takes me up to post 112 in this thread. I can go get more, if you want Ryan.
Ryan, I read your pdf for Dr. Beckers and as a layperson in this matter was very appreciative of the effort to which you went to explain your proposed method. I judge that these discussions and analyses have been significantly elevated since we have had clearly articulated and patiently explained posts/presentations like the one you composed for Dr. Beckers. I hope he replies and we are all able to read the exchange at some point.
In my past life I had exposure to people who delivered techinal reports on their work for consumption at all levels. I do not mean to pry but does your work involve that kind of communication.
Ryan:
1. I thought Jeff’s comment “I don’t disagree that there are situations where average temperature could be calculated using a negative weight and you are right that some of my statements are too strong without the context of this paper” was much more forthright than any of yours so far.
2. Please refer to this statement of mine (repeated in essence several other times also) where I discuss the difference between general statements made versus Steig-specific concerns:
“And particular failings of the Steig algorithm do not prove a general case. Really there are different issues to examine, when you say, I can never use a negative stock as a predictor of the economy (have a negative weighting on a station in Antarctica) versus when you say that Steig did not have enough PCs to do what they wanted to do properly.”
3. ON THE CONTENT: The use of different stations in the training and in the prediction is a DIFFERENT issue than negative weighting, Ryan. It would still be a significant concern if you had this occuring even if the weightings were all positive.
Jeff:
1. I was going to try to be all gracious, Jeff (of manly hand fame). But Ryan has pissed me off.
2. Please either let me post real-time or I will not post anymore, here.
Ok TCO, no live posting. You have had your say on this thread for a bit. Let the boys talk.
Goodbye.
That’s okay, TCO. Of all your quotes in #347, only 2 were mine:
Promises, promises.
I’m still hoping to get this hifi working properly!? ;0
Jeff, what is the point in continuing a conversation with TCO if the intent is for these discussions to enlighten and inform. TCO’s intentions are rather transparent, in my view, and they have much more to do with TCO’s image of TCO and his opinion of skeptics criticism of climate science than about learning or informing.
I think you and Ryan have been respectful and patient with TCO and I suspect he has enjoyed the attention he has received from both of you and the reinforcement, in his mind anyway, that he can and has contributed to these analyses and doing so by basically not knowing the material and methods in any good detail, but rather by his native abilities to reason.
I cannot speak for the other bystanders here, but for me, continuing this discussion with TCO is a major waste of time consisting of arguing minutia on the margins.
#356 Well his ultimatum at the same time I’m pointing out his dishonesty wasn’t the best approach to win me over. I suspect he’s claiming victory over at open mind or somewhere.
Whatever, the internet is a big place maybe he’ll find a home.