the Air Vent

Because the world needs another opinion

Mullagain

Posted by Jeff Id on March 29, 2009

This is the second time in two day’s I’ve had to replace a post and frankly I’m not happy. There’s some detail involved in converting the lat and lon to Cartesian that was botched. Again, I’m going to snip myself in the meantime before I lay out a bunch of @#$! and just present the fixed and many times checked plots. Before you pass judgment, what did you do with your Saturday?

Per TCO request and my oversight, the following graphs represent surface station data on the Y axis and the closest Steig AVHRR gridcell on the X axis.

Here are the correlations.

steig-surface-raw-correlations-13


steig-surface-raw-correlations-24

steig-surface-raw-correlations-antarctic3

I left the islands in the last plot as they are correlated to the nearest surface station cell. Dr. Comiso did an excellent job removing cloud cover from this data and IMO deserves our congratulations. He’s done some good work not that I’ve ever even implied otherwise, it’s just a fantastic match to surface stations.

After this latest problem, I’m too grumpy to work on blogging today.

Scripts are below.

conv= pi/180
cdist=abs(-90-surcoord[,2])
scx=cdist*cos(conv*surcoord[,1])
scy=cdist*sin(conv*surcoord[,1])

sadist=abs(-90-sat_coord[,2])
sacx=sadist*cos(conv*sat_coord[,1])
sacy=sadist*sin(conv*sat_coord[,1])

for( i in 1: length(surcoord[,1]))
{

dist = ((scx[i]-sacx)^2 + (scy[i]-sacy)^2)^.5
satindex[i]=which(dist==min(dist),arr.ind=TRUE)
print(min(dist))

}

The print min dist returns.

[1] 0.4056697
[1] 0.264035
[1] 1.065037
[1] 0.2897361
[1] 0.1774862
[1] 0.07663658
[1] 1.134680
[1] 0.1873764
[1] 18.34042
[1] 0.3635363
[1] 0.1290426
[1] 0.8768884
[1] 0.3156276
[1] 0.2121938
[1] 0.3557787
[1] 1.141575
[1] 1.157662
[1] 14.44150
[1] 0.1004181
[1] 1.075487
[1] 1.093500
[1] 0.2477843
[1] 14.53145
[1] 0.391306
[1] 0.2080031
[1] 0.9612466
[1] 0.3361359
[1] 0.05271883
[1] 0.2564120
[1] 0.2173724
[1] 0.1332896
[1] 0.2641893
[1] 0.1093217
[1] 6.73377
[1] 0.09877636
[1] 0.3231953
[1] 0.1298149
[1] 0.05271883
[1] 6.343382
[1] 0.1468818
[1] 0.1140925
[1] 0.08862605

You can see there are several values greater than 1.5 degree (distance is in degrees). These represent points in my above plot which have been correlated to satellite data fairly remote from the surface station.

Tomorrow when I’m less grumpy, I’ll look at what happens when I eliminate these high distance values. In the meantime, I’m going to think hard about not posting as often. It looks bad to post short term results which can have errors. My style is to let it go and people will realize the quality of work I do over time but it may not be the best decision.

Whatever, I’ll think about it.


17 Responses to “Mullagain”

  1. rephelan said

    Jeff, you’re doing all this in public and if we don’t hear it from you first you’re second or third. Just keep up the work and be honest. I can’t ask for anything more. Keep it up.

    Kinda OT… no, it’s really OT but I don’t know where to put it… definitely not on CA… but if you look at this stuff, have a stiff drink first!

    http://www.uncommonwisdomdaily.com/

    http://www.uncommonwisdomdaily.com/?p=470

  2. rephelan said

    By the way, does this mean we’ve been too mean too Dr. Steig? I don’t mean you, you’ve ALWAYS said the math is the math, I guess I meant me.

  3. Jeff Id said

    #2 I don’t think so. Steig could have released the code and data before. I’ve said many times, if the antarctic is warming that’s exactly what the Air Vent will say.

    We do know there is a problem in spatial correlation from the satellite data. It was caused by the 3 PC approximation. I’ll keep digging until I find out what we can, it just means my posts might be slower.

  4. Jeff C. said

    Yes, it is an impressive match. I’m not sure it indicates outstanding cloud masking as the satellite cells could have been forced to match the surface values through some sort of interpolation algorithm. For now, I’ll give the good doctor the benefit of the doubt. Dr. Comiso has some impressive papers out there and he usually includes quite a bit of detail in his methods explanation.

    My tinge of skepticm is from how I spent my Saturday. I’ve been downloading, collating and plotting the per-cent cloud cover numbers from the monthly NSIDC AVHRR data. I was very surprised to see the continent averages up to 80% cloud cover in the winter, around 50% in the summer. Regardless of how good the masking is, it seems like it would be extremely difficult to pull good data out of the low percentage of clear sky days. I’m putting some plots together and hope to have more in the next day or so.

    Jeff, after being burned yesterday, I know how you feel. It happens, but I think most of the readers understand your depth of knowledge on this subject and realize we all make mistakes. Also consider that most normal people are out having fun on a Saturday instead pouring over this stuff, so no one doubts your commitment to figuring it out. But if it happens again, expect a 10% pay cut. ;)

  5. Jeff C. said

    I forgot to mention, if you figure it out, your pay gets doubled!

  6. Jeff Id said

    Ten percent, jesus I’d have expected 3 or 5 –harsh stuff.

    Funny thing about running a blog is that you need new posts every day. It pushes you to go fast. People like SteveM have a lot of time to work on posts compared to me and I suppose you. I think there’s not much choice but to slow down a bit. As I’m sure you know being an engineer requires it, if you screw up big $$ go down the hole and when you’re in charge it happens very quickly.

    I do a good job with accuracy at my 70 hours per week job as opposed to the 20-30 I’ve been putting into this. It also helps that it is a field I’m familiar with. Hell 8 months ago I didn’t know R was a software program.

    In the end,these correlations are fantastic across the board. If they used good math and good algorithms I would like to see them because the result is great. We can pick out bad surface stations directly from the sat data!

  7. rephelan said

    Jeff & Jeff!

    Pay cuts? Bonuses? You guys in the pay of BIG ______ (fill in the blank with your own sugar daddy)? Speaking of which, someone ought to make a copy of Dr. Mann’s C.V. where he documents the monetary amounts he controls…

    I’ve always been suspicious of advanced statistics, prefering to look at merely descriptive numbers (totals, percents, ratios, rates..)- if they were good enough for Emile Durkheim, they’re good enough for me (or as a colleague puts it: “Durkheim? Weber? Marx? They’d never be published today! Not a Pearson’s R amongst them!”)

    I can’t do what you guys are doing but I CAN follow and am inspired. Keep it up. One of Phelan’s Law’s (I think I’ve elevated it to number 7) is that “Life has no end to lessons of humility”. You’re doing all this in public, so the lessons may come more often, but you’re doing it right. This may be the paradigm for 21st century science, the science the late Michael Crichton was trying to resurrect: Public Science.

    I do offer one caveat however: whether your analysis finally supports Dr. Steig or not, all the number crunching is essentially a proxy and not a direct, empirical observation. In the long run, we won’t be able to say that the interior of Antarctica really did warm, cool, or stay consistent, but rather we have reason to suspoect that it did…

    (Oh! and if I can be forgiven for one idiosyncrasy: James Hansen, Gavin Schmidt, Michael Mann and Eric Steig all have titles that they have earned, even if many of us are upset with them. Courtesy costs us nothing.) >

  8. Fluffy Clouds (Tim L) said

    In the end,these correlations are fantastic across the board. If they used good math and good algorithms I would like to see them because the result is great.

    We can pick out bad surface stations directly from the sat data!

    THIS CAN Verify A. Watts work??????????????

    (snip), I almost wet my self… If you can do it millions of peps will owe you BIGG!!!

  9. Fluffy Clouds (Tim L) said

    what did you do with your Saturday?
    Chopped 3″ x 2′ x 50′ ice out of drive way. now can drive car up to property.
    new storm to dump 6 to 8 inches of heavy snow in the next 12 hours.
    late storm according to NWS.

    now can drive car up to property,,, been seance november 20th car has been on lot proper :)

  10. Fluffy Clouds (Tim L) said

    opps since

  11. rephelan said

    Naw, I’m not one of those high-falutin’ Doctors, yah understand, but in my old age I’ve started getting a bit persnickity about the honorific “perfesser”….

  12. pyromancer76 said

    IMO mistakes make for interesting reading. Like a detective novel: who dunnit and how? And what you and your readers are attempting! (You deserve a 25% pay raise.) Anyway, it all begins with so-called scientists not reporting their data and methods. They should be publicly shamed. I think you and others like you should be immensely proud of all your efforts. I read a little too much humble pie yesterday on CA for my taste.

  13. Neil Fisher said

    I agree with several of the posters above – let it all hang out, Jeff! Showing what sounds reasonable but doesn’t work – and why – is important. That people will make mistakes is expected – there is no shame in making a mistake! That you acknowledge them, correct them, and continue looking for them is what really matters. And you do. You not only learn from these mistakes, but others do as well. Which, IMO, is what makes the difference between places like the Air ent and, say, RC – you acknowledge your mistakes; you acknowledge that you are a human being trying hard to understand a complex topic; you’re not afraid to ask “dumb” questions; and you’re not afraid to dive right in the deep end and get your hands dirty (if I can be allowed to mix metaphors like that! ;-) )

    You are somewhat of an inspiration Jeff, so just keep doing what comes naturally.

  14. TCO said

    *Is the data: monthly, daily, yearly?

    (It’s important given your frequency noodling.)

    *Is the data anomoly from average?

  15. Jeff Id said

    Monthly.

  16. TCO said

    anamoly from average (or e.g. difference from previous month)?

  17. Jeff Id said

    It is anomaly data.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 142 other followers

%d bloggers like this: