the Air Vent

Because the world needs another opinion

The Blend…Inator!!!

Posted by Jeff Id on March 31, 2009

I ve been watching Phineus and Ferb cartoons with my 3 year old lately. The villan makes one machine after another with -inator on the end it cracks me up. Actually this post explores the blending of trends across the Antarctic from RegEm processing of he surface station and satellite data. It’s the Bass-o-matic for temp data.

This post uses a modified mapping script courtesy of RomanM which shows the correlation of satellite gridded data relative to a single time series across the antarctic. There are three columns of graphs below comparing surface station data with three varieties of the satellite data. I left all 42 surface stations in just as a reference, only 38 were used in the reconstruction.

1 – The first column is the raw Comiso AVHRR data as presented by Steig09. The data exists from 1982 to 2007.

2 – The second column is from the Steig09 reconstruction data, this data is also post 1982.

3- The third column is also from the reconstruction data, this is a comparison of the surface station data to the pre-1982. The satellite data didn’t exist during this period so it is entirely RegEM.

Remember the shape of the 3 pcs as plotted across the antarctic by CA HERE and the fact that there are only the first two effective PC’s as shown by this plot below.

three-main-pcs2

In these plots below, data with less than 5 values for correlation in the time period was not plotted.

corplots-37

corplots-1corplots-2corplots-3corplots-4corplots-5corplots-6corplots-7corplots-8corplots-9corplots-10corplots-11corplots-12corplots-13corplots-14corplots-151corplots-16corplots-18corplots-19corplots-201corplots-17corplots-21corplots-22corplots-23corplots-24corplots-25corplots-26corplots-27corplots-28corplots-29corplots-30corplots-31corplots-32corplots-33corplots-34corplots-35corplots-36

corplots-38corplots-39corplots-40corplots-41corplots-42

First look at the Faraday/Vernadsky plot. Column 2 has a clear vertical separation corresponding to a strong PC3 componenet yet in the pre 1982 data of column 3 the pattern shows a PC2 shape (If you are missing my point see the orange graphs in the right column at the CA link above). I just thought I’d point this out as the reduction in PC’s has an effect.

Another thing I noticed is that the correlations are much more pronounced in the pre-1982 reconstruction data as compared to the real data in the left column Leningradskaja is a good example as well as vostok.

Now many of us assumed that the peninsula trends must be blended across the antarctic in order to make the final trend happen. As far as station correlation is concerned, that doesn’t seem to be the case. Correlation is a funny thing though. Here’s a plot of 1982 correlation vs distance.

antarctic-correlation-vs-distance-aws-pre-19801

You can see that correlations as high as 1 exist over thousands of kilometers for the pre 1980 data. It appears now that the reason for this is a result of the 2 pc’s remaining in the RegEM portion of the satellite data – Column 3 above. It’s actually pretty easy to see the effect when you consider the multiple plots of large red high correlation areas in column 3 above where two thirds of the antarctic is dark red.

In my opinion, this doesn’t look very good as far as localizing stations but it’s not as bad as I had expected. We have only two areas of the antarctic in column 3 for determining when a station correlates well with gridded satellite data, so it’s just not possible to localize and properly area weight stations. It’s not like the peninsula has zero effect on 4000 km away either, in nearly every instance above there is a powerful negative correlation across the antarctic.

This post was a lot of work again but it really answers questions for me about the blending process. I think the next step is to try higher order PC’s and see what happens to the trends.

18 Responses to “The Blend…Inator!!!”

  1. Fluffy Clouds (Tim L) said

    I see the testing, but gets way over the average head.

    it seams that the higher orders flat lined fast.

    we need dead nailed but to many “IFs”

    humm…………..

  2. Fluffy Clouds (Tim L) said

    The Blend…Inator!!!
    LOL on DR. Becker had that too.

    CloudInator out!

  3. AS a summary of Mannian methods, don’t forget: http://www.youtube.com/watch?v=vvmyxTm6hkg

  4. Jeff Id said

    #1, I should have made it more clear but by the time you get done putting something like this together you get tired.

    The right column is more consistently correlated than the real data in the left column. The trends are spread across the continent from the eastern stations while the peninsula stations on the right actually have a negative correlation (trend) spread across the eastern continent. Everyone kept saying the peninsula warming was spread to the east. It appears now that this may not be the case. From left to right we have low medium and high spread.

    The columns also represent left to right 288 pc’s (full raw anomaly data), 3 pc’s and 2 pc’s.

    Correlation in this paper means short term high frequency shifts in the temperatures, I believe long term trends should also follow but in a low pc breakdown I think there can be some serious issues there as well — that will be the key.

    Imagine if the long term trends from a high correlation station were not properly distributed according to the high frequency correlation. The peninsula trends could still be a problem but demonstrating that would be difficult.

  5. TCO said

    Style: Your explication is hard to follow. With effort, I can get there. But it is still a pain.

    Content: Why not show the difference of 288, 3 and 2 PC’s over same time period? (I realize 288 and 3 are). How much does reduction in PC’s change end results?

  6. RomanM said

    I saw these earlier jeff. Good stuff! These guys smoothed the temperatures out with a broad brush using methods only known to the literati. I have been trying for two days to reproduce the “ordinary 3 PC reconstruction” with little success. Since what they said they did isn’t defined in any meaningful fashion, as you already know everything is a crap shoot.

    I see you have some sidewalk engineer crickets hanging out here giving advice on what you and everyone else should do without much contributed effort of their own to add to the mix.

  7. Fluffy Clouds (Tim L) said

    #1 + mirny

    #3 – campbell

    #2 – esperanza

    by looks only
    don’t know if + or minus has any meaning
    but there could be that inversion I keep saying, good bad or ugly I don’t care.

    sidewalk engineer cricket, happy to be on the side walk this time. lol

  8. Jeff Id said

    SteveM’s John Denver link is hilarious and it might be right.

    I’m still hoping we can get to a reasonable conclusion about this paper.

  9. Jeff Id said

    #6 Thank’s Roman,

    I’ve been trying to get TCO to do some correlation calcs for bristlecones but no luck yet. I told him he could post it here. Unfortunately TCO is so famous in the global warming blog world that he could say the sky is blue at this point and he would have a hundred people telling him he’s nuts.

  10. TCO said

    I’m an internet ninja.

  11. rephelan said

    Jeff, sorry, but I’m pretty sure that TCO is neither a he nor a she nor an internet ninja. We’re looking at an internet consortium. TCO is a “they”. quit feeding the trolls.

  12. Kenneth Fritsch said

    I see you have some sidewalk engineer crickets hanging out here giving advice on what you and everyone else should do without much contributed effort of their own to add to the mix.

    RomanM, I liked your reference excerpted above and will not be chirping any advice, but instead offering my view of what Jeff ID has shown in an attempt to determine how confused I am.

    What I see is some rather subdued (color-wise), but not necessarily unimportant differences (since we are looking at small changes here), between the correlation differences between raw and reconstructed data depending on where the station is located. I may well be seeing what I want to see, but, on going from raw to reconstructed data, I see Peninsula stations blending the high correlations within the Peninsula to lower intra Peninsula correlations to increaed correaltions with West Anartica. It gets a little confusing with stations outside the Peninsula, but I see some where there appears to be little change in correlations between the raw and reconstructed data and others where on going from the raw to the reconstructed data seems to restore the intra Peninsula correlations and makes them more negative with respect to the station in question (that is outside the Peninsula).

    Confused?

  13. Jeff Id said

    Kenneth,

    It’s all explained here.

  14. Kenneth Fritsch said

    Jeff ID at #13:

    I remember that skit, but not the young and slim guy doing it. He did remind me of a little of an older and much stouter man I saw dropping a ceremonial puck the other night at an NHL game.

    Actually the skit does make a good point: blending can make things much easier to prepare, consume and digest – providing one can keep the end result at its final destination.

  15. Geoff Sherrington said

    For Jeff Id,

    There are adjusted versions of station data for Antarctica. To assist in your lovely correlation work, can I please present a graph for Mawson Base temperature anomalies in an early form, before it got to USA. I have infilled about 12 days by assuming the value of the preceding day, which will have negligible impact on the graph. The data go back to 1958-64, then there is a break 1964-1970. Daily data are available should they be useful.

    I have latitude and longitude -67.6014, 62.8731.

    http://i260.photobucket.com/albums/ii14/sherro_2008/Mawsonannualtempanomaly.jpg?t=1238759354

  16. Jeff Id said

    Geoff,

    It’s too bad there are only 12 years pre 1982. I’m interested in the uncorrected surface data for the Antarctic. I haven’t even spent two minutes looking at the corrections yet so I don’t even know how the data was adjusted but bulk GISS correction wouldn’t make sense on so small a station count in an unpopulated area.

    One thing which keeps rolling around my little brain is how to explain the details of correlation in this paper. The low freq trend will have no real effect on how the surface data is spread around the antarctic. This means the whole reconstruction can consist of surface station data spread according to correlation weighting on the satellite data grid. If we do that correctly we might actually get a more reasonable trend from a small number of stations although the stations probably still need to be gridded prior to RegEM.

    So the best version of this might be to have the raw uncorrected data from the surface stations, the AVHRR data from Comiso and a good regridding technique prior to a high order expectation maximization.

    Long winded but can I ask where you found your data?

  17. Geoff Sherrington said

    For Jeff Id,

    The Australian Government makes a claim to sovereignty over more than 40% of the Antarctic land area amd so has to do work to maintain its claim. Departments like the Bureau of Meteorology were drivers of the International Geophysical Year 1957, which is when a lot of data recording starts. There have been several Australian bases intermittently over the years since then and weather records have been kept, sometimes with gaps. The meta data sheets, if useful for the Antarctic, are on sale at somethig like $US10 a base but I have not bought any. The daily temperatures (max & min) are on a CD available for about $Aust 120 (say $us85). I could go broke buying one for every deserving scientist. There are about 1,000 Australian weather stations covered to various degree.

    The data are described as “raw” by the BOM, but there have been homogeneity adjustments made. I do not know the extent of these. Scribal errors seem to have been corrected but for the Antarctic I do not know if much more has been done.

    Therefore, I thought I important that if you were doing correlation matrices, you would benefit from use of the least manipulated data, which is what I outlined above. If you send me an email address that will get to you, I can send attach some data that is for scientific research and within the fair use provisions of copyright. You might need to select the items of reasearch interest from below. I do not wish to infringe copyright.

    These are the bases covered for the periods indicated in the header which is not so user friendly. They open easily in Excel but I have been a bad boy and not gone the R road yet.

    \dc,Station Number,Year,Month,Day,Maximum temperature in 24 hours after 9am (local time) in Degrees C,Quality of maximum temperature in 24 hours after 9am (local time),Days of accumulation of maximum temperature,Minimum temperature in 24 hours before 9am (local time) in Degrees C,Quality of minimum temperature in 24 hours before 9am (local time),Days of accumulation of minimum temperature,#

    st,300000,300 ,DAVIS ,01/1957, ,-68.5772, 77.9725, ,ANT, 18.0, 23.2,89571,1957,2007, 91, 97, 3, 0, *, 0,#

    st,300001,300 ,MAWSON ,01/1954, ,-67.6014, 62.8731, ,ANT, 9.9, 16.0,89564,1954,2007, 99, 98, 2, 0, 0, 0,#

    st,300002,300 ,TAYLOR ROOKERY ,01/1957,12/1958,-67.4500, 60.8667, ,ANT, 3.0, , ,1957,1957, 49,100, 0, 0, 0, 0,#

    st,300003,300 ,WILKES ,01/1959,12/1969,-66.2500, 110.5833, ,ANT, 12.0, 13.0, ,1960,1969, 97,100, 0, 0, 0, 0,#

    st,300004,300 ,MACQUARIE ISLAND ,01/1948, ,-54.4994, 158.9369,SURVEY ,ANT, 6.0, 8.3,94998,1948,2007, 98, 98, 2, 0, *, 0,#

    st,300005,300 ,HEARD ISLAND (ATLAS COVE) ,01/1948, ,-53.0190, 73.3918,GPS ,ANT, 3.0, 3.5,95997,1948,1987, 18,100, 0, 0, 0, 0,#

    st,300006,300 ,CASEY (THE TUNNEL) ,01/1969,01/1989,-66.2833, 110.5333, ,ANT, 12.0, 15.0, ,1969,1990, 99,100, 0, 0, 0, 0,#

    st,300007,300 ,MOUNT CRESSWELL ,01/1972,02/2003,-72.7333, 64.3833, ,ANT,1161.3,1300.0, ,1971,1974, 25,100, 0, 0, 0, 0,#

    st,300008,300 ,MAWSON (MOORE PYRAMID) ,01/1972,12/1984,-70.3000, 65.1000, ,ANT,1460.0, , ,1972,1974, 8,100, 0, 0, 0, 0,#

    st,300009,300 ,MAWSON (KNUCKEY PEAKS) ,01/1975,12/1984,-67.8000, 53.5000, ,ANT, , , ,1974,1975, 88,100, 0, 0, 0, 0,#

    st,300010,300 ,MAWSON (MOUNT KING) ,01/1975,12/1984,-67.1000, 52.5000, ,ANT, 112.5, , ,1979,1980, 91,100, 0, 0, 0, 0,#

    st,300011,300 ,CASEY (LANYON JUNCTION) ,01/1983,12/1987,-66.3000, 110.8667, ,ANT, 470.0, , ,1984,1985, 19,100, 0, 0, 0, 0,#

    st,300015,300 ,LAW BASE ,01/1987,02/1992,-69.4167, 76.5000, ,ANT, 77.0, , ,1987,1988, 19,100, 0, 0, 0, 0,#

    st,300016,300 ,PRINCE CHARLES MTNS (DOVERS) ,01/1988,12/1992,-70.2333, 65.8500, ,ANT,1058.0,1059.0, ,1988,1989,100,100, 0, 0, 0, 0,#

    st,300017,300 ,CASEY ,02/1989, ,-66.2792, 110.5356, ,ANT, 40.0, 42.3,89611,1989,2007, 99, 98, 2, 0, *, 0,#

    st,300028,300 ,HEARD ISLAND (THE SPIT) ,03/1992, ,-53.1082, 73.7225,GPS ,ANT, 12.0, 12.5,94997,1992,1993,100,100, 0, 0, 0, 0,#

  18. Geoff Sherrington said

    Ooops, mistake time. The data I sent in the graph marked Mawson on April 3rd at 11.51 was in fact Davis station, lats and longs given in the later post above for Aust station no 300000.

    Next, the annual average for year 2002 was wrongly calculated and should be -1.94 deg C.

    Working too late at night. I am just compiling years 2007, 2008 for Davis so the corrected graph is below.

    I do not have GISS or HADCRU data for comparison.

    Davis Antarctica

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 142 other followers

%d bloggers like this: