the Air Vent

Because the world needs another opinion

Measuring the Clouds

Posted by Jeff Id on March 31, 2009

Guest post by Jeff C



Jeff C has located a possible problem with cloud contamination of the surface dataset. This is an ongoing investigation but there are some apparent differences in result from the most recent NOAA 16 satellite as compared to the rest. This is an important issue as the methods section of Steig’s Antarctic paper cites clouds as the single greatest source of error.

Cloud masking is probably the largest source of error in the retrieval of TIR data from raw satellite spectral information. Wehave updated the data throughout 2006, using an enhanced cloud-masking technique to give better fidelity with existing occupied and automatic weather station data. We make use of the cloud masking in ref. 8 but impose an additional restriction that requires that daily anomalies be within a threshold of 610 uC of climatology, a conservative technique that will tend to damp extreme values and, hence, minimize trends29. Values that fall outside the threshold are removed.

As you might have guessed, there is not much information about the ‘enhanced’ masking technique given so the question becomes, what about the data.


Jeff C

In the previous post HERE , I plotted cloud cover percentages over time using the “cldtype” (cloudtype) parameter of the U Wisconsin AVHRR dataset. Cloud type is a bit of a misnomer, as the data is actually something known as “cloud fraction”. Cloud fraction is a value from 0 to 1 that expresses the frequency of cloud cover. It is very similar to the cloud mask, but not identical. Here are the cloud types that would be displayed if we were looking at the cloud cover for a single day/time:
0 – clear
1 – partly clear
2 – warm water cloud
3 – super cooled water cloud
4 – opaque ice cloud
5 – non overlapped cirrus cloud
6 – overlapped cirrus cloud
However, we are looking at monthly means, not single day/time data. Since cloud cover changes from day to day, the monthly means can’t list a single cloud type. Instead it lists the cloud fraction; the percentage of time the cloud cover was any of #1 through #6 (i.e. not clear sky). The cloudtype algorithm is not contained in the released version of CASPR (Cloud and Surface Parameter Retrieval algorithm version 4.03) and is not described in any of the documentation supplied in the released version. Cloudtype is included in the latest unreleased version CASPR (version 5.0), which was used in generating the UWisc dataset I have been reviewing.
Since the cldtype parameter is new, poorly documented, and giving funny results, I decided to generate new plots from the “cmask” (cloud mask) dataset from the UWisc server. These are different files from those containing the temperature and the cloud type information; I just recently noticed them on the server and downloaded them last night. Below is the same plot as in the previous post, but with cloud mask (black) and cloud type (red) overlayed.

image2The traces are quite different, and seemingly out of phase with each other. The large discontinuities have also shrunk with the exception of the step at the 2000 to 2001 transition. Clearly there is something wrong with the “cldtype” data. After checking with some knowledgeable sources, it appears that the cloud type data doesn’t have a correction applied for the diurnal drift that occurs over the life of the spacecraft. This may be by design, or it could be a bug in the cloudtype algorithm of CASPR version 5.0. It doesn’t seem like the data is very useful without the correction applied.

Here is the same cloudmask plot (the set that does appear to have the correction applied) with the spacecraft transition from NOAA-14 to NOAA-16 shown. It has a large discontinuity that did not disappear when using the cloudmask data instead of cloud type data.

image4I was gratified to see that the cloudmask data is much cleaner than cloudtype as now we may be able to actually make something of it. First off, here is a cloud-cover map from 1982 to 2001 for Antarctica. This covers the era up to the large discontinuity in the plot above (the transition from NOAA-14 to NOAA-16).


Note there is no blue on the plot as cloud cover doesn’t drop much below 50%. If you plot cloud cover during a shorter period of this same time frame, you get virtually the exact same result, the pattern is very consistent. Here is 1990 to 2000.

image6Do these patterns look at all familiar? They certainly did to me. The Comiso data (also known as the Steig “raw” data) has a virtually identical temperature anomaly trend pattern over the same period. Here is the Comiso trend plot accompanied by a similar 1982-1999 trend plot from Steig’s SI.

image7Note how the areas of heavy clouds correspond very close to Comiso’s areas of warming. I’m not sure why this would be, but it seems that it is probably more than coincidental.
There is also another oddity. The cloud pattern over NOAA-16 (2001 to 2004 in this data set) has a distinctly different cloud pattern than that seen previously. This is not a huge surprise as the plot over time shows a noticeably different signature during this period. Clearly there is a major difference in the cloud detection/masking of data from the NOAA-16 satellite.
image8Interestingly, Dr. Comiso’s trend plot over this period also shows a dramatic change. It is not as close a match to the cloud patterns as those seen previously, but it is close enough that it makes me wonder if the Comiso dataset is cloud contaminated.


The abrupt change from 2001 on is also troubling as Comiso’s dataset has a net continental warming trend over this period; the UWisc dataset has pronounced cooling.

13 Responses to “Measuring the Clouds”

  1. Michael Lenaghan said

    Why is it that clouds are assumed to “contaminate” temperature data? Dr. Spencer believes that models assume clouds have a positive feedback while in fact they may (and from his perspective more likely do) have a negative feedback. If that’s true, wouldn’t the effect of clouds be too important to toss out? And isn’t it a given that the result of tossing out cloud-contaminated data would be warmer temperatures?

  2. TCO said

    I think the cloud coverage matches up more with a certain kind of climate regime, rather than with a certain trend in change of climate. For instance if you looked at North America, you would have less clouds over Arizona than over Georgia. And this would be much more significant than correlation to whether Georgia or Arizona is warming more.

  3. Michael Lenaghan said

    So let’s follow up with your Arizona example. Let’s say the average temp over the previous decade was x. Now let’s say that cloud coverage increases substantially over the next decade and (for the sake of argument) let’s say that clouds have a negative feedback effect. Temperature over the next decade is now x (“long term average”) minus y (“negative feedback from increased clouds”) but we “adjust” the data to eliminate the effect of clouds…? Doesn’t that prevent us from seeing that temperature went down *because* of clouds? (In fact it doesn’t matter whether you believe clouds have a negative or positive effect; either way you’re trying to mask out the effect as if it were noise when in fact it may be a key driver of the very thing you’re trying to measure.)

  4. Jeff Id said

    #3 What they’re trying to do is measure clear sky temps by looking between the clouds. This means TIR ideally is only clear sky temperatures. If more cloud temperatures make it through at one time or other it could create a slope bias.

    You are right though, and it’s a very good point. If we assume there are more clouds and warmer weather, only measuring clear sky temperatures could really hide that effect.

    I have receieved a private email from an expert in this area who said something in an email like — Why only measure clear sky temperatures, doesn’t temperature happen on cloudy days also?

    When you think about it, what if cloud cover went up dramatically in 30 years and average ground surface temperatures also go up, but from AVHRR clear sky data you’d only measure the clear sky days and the temp might appear unchanged.

  5. Michael Lenaghan said

    > When you think about it, what if cloud cover went up dramatically in
    > 30 years and average ground surface temperatures also go up, but from
    > AVHRR clear sky data you’d only measure the clear sky days and the temp
    > might appear unchanged.

    Wasn’t that my point? 🙂

    But take it one step further: what if clouds are actually the key *driver* of temperature? Because that is, in fact, the core of Dr. Svensmark’s theory: clouds follow cosmic rays and temperature follows clouds.

    > Note there is no blue on the plot as cloud cover doesn’t drop much
    > below 50%.

    Maybe it’s worth seeing what kind of difference clouds make to the results. (Btw, Svensmark predicts that low-level clouds have a net cooling effect everywhere other than Antarctica.)

  6. TCO said

    I think all of these points are interesting/plausible. I was only making a much more limited comment to someone who was equating trend with cloudiness. But a key realization in thinking about this problem should be delta climate (2000-1950) at any spot (Western Antarctic, Eastern Antarctic, Arizona, Georgia) is likely to be MUCH less than delta climate from Arizona to Georgia (or West to East Antarctica).


  7. Jeff Id said

    #5 The point of the post is to show that the clouds are being sorted differently probably due to diurnal shift or instrument change or something.

    The other point is what if the cloud levels change naturally which I mentioned you have a good point. Considering that AVHRR after cloud sorting is only attempting to measure clear sky temps, we might completely miss the real trend even with perfect data. The NSIDC literature isn’t clear about the effect because clouds measure both cooler and warmer at times.

    Good point for sure.

  8. Fluffy Clouds (Tim L) said

    Here we are again!
    “there could be that inversion I keep saying”, good bad or ugly I don’t care.
    and at 1995 I see a phase error.
    If the data were miss handled it could mess up the output and frankly I just don’t trust anything that comes from hanson/nasa/mann ECT.

    Michael Lenaghan said
    March 31, 2009 at 10:24 pm

    (Btw, Svensmark predicts that low-level clouds have a net cooling effect everywhere other than Antarctica.)

    This is in error. examples are the desert and water body’s.
    except fog of course!

  9. Kenneth Fritsch said

    I find the introduction to the thread by Jeff ID and Jeff C (in deference to my identical twin granddaughters, not the two Jeffs) well organized and easy to follow.

    Any time one finds a change in a variable, or an adjustment to that variable, that could effect or explain at least part of the main variable under consideration (temperature anomaly trend in this case), I judge it to be an important find and one that needs further analysis and discussion.

    If indeed the cloud cover changed and in step with a temperature change, both spatially and temporally, then one must consider whether (1) that change is an artifact of the adjustment procedure and in turn could artificially affect the final adjusted temperature trend result, or (2) the change in cloud cover is real and should that change be considered contributing to a temperature change or a result of a temperature change.

  10. Jeff Id said

    #9 Kenneth,

    Thanks, you’re right about changes or adjustments. Especially when the trend is so sensitive to the methods. There is one interesting link which describes an improvement in sensor packages for NOAA-16.

    Channel 3A on the NOAA 16 AVHRR/3 instrument allows improved discrimination between snow and clouds by using the 1.6 µm wavelength. At 1.6 µm, snow has very low reflectance, while the reflectance of clouds remains high. Refer to Figure 4. Therefore, both cirrus and optically thick clouds can be directly classified and distinguished from snow at the 1.6 µm wavelength (Warren 1982).

    Jeff C who did all the work in this post doesn’t feel they are related I’m not sure what it means so if he stops by perhaps he would explain why.

  11. Jeff C. said

    Thanks all for the comments. Regarding #9 and #10

    The page you linked above ( is a great resource for trying to sort out this data. There seems to be three major differences from NOAA-14 to NOAA-16.

    1) The improved cloud detection instrumentation you mentioned (described in section 4 figure 4 of my link above).

    2) A problem with channel to channel data contamination that occurs on specific days on NOAA-16 (section 3 table 12 of the link)

    3) A significant change in the diurnal drift pattern for NOAA-16 compared to the others. NOAA-16 appears to have much less drift over time (section 4 figure 5 of the link)

    The difference could be caused by any of these or something else that I’m not aware of. I suspect it is #3 for several reasons. I would expect improved cloud masking (#1) to refine the cloud pattern, not completely change it. If improving the cloud detection completely changes the cloud pattern that had been seen for the last 20 years, all data prior to NOAA-16 is pretty much worthless. Perhaps this is the case, but I have not read anything suggesting this.

    I don’t think it is #2 as in some years this problem only occurred on a few days. Yet the discrepancy seems to affect the entire NOAA-16 period.

    The main reason I think it is #3 is that I processed the “cloud type” data (the cloud data without the diurnal correction) for the NOAA-16 period and the cloud pattern reverted to that seen in the earlier satellites. I don’t think it’s a coincidence.

  12. Jeff C. said

    Ryan and Jeff – great job and thank for keeping the heat on after all these months. I’ve been following with interest and was happy to see Ryan’s post at WUWT. I think Anthony’s headline might have been a little strong and that might have twisted Dr. Steig’s tail a bit and clouded his response. I hope he takes the opportunity to reread the article and follow some of the links and doesn’t get hung up the satellite transition stuff. It’s an important point, but Ryan was clear it didn’t significantly influence the results.

    The amount of work dedicated to reconstructing his methodology is truly staggering (Jeff Id, Ryan O, Roman M, Steve Mc and lots of good comments), although the most praise should go to the wives for putting up with this for the last five months.

    In a comment above, Jeff linked to a post I did regarding satellite transitions. In a later post I described a better data set for determining cloud cover that I found at the University of Wisconsin website. In this data set, there are still troubling discontinuities in the satellite record, particularly from NOAA-14 to NOAA-16. I corresponded with those responsible for the data set at UWisc and they told me they were aware of the problem but did not know how to fix it. Their candor was admirable.

    That post shows that Dr. Comiso’s data set has a discontinuity at the same point. There are definitely real problems with the satellite transitions that should have been fixed or at a minimum disclosed. However, the real issue here is the reduction to 3 PCs for no good reason. The reasons given (spatial similarity to known atmospheric phenomena) are specious as shown by Steve M in the Chlandi posts. The sidetracking to the satellite transitions is a bit of a smokescreen thrown out by the team and its fans.

    I’ve been missing from the debate for the past six weeks as my five-year old son has been dealing with some long-term medical issues. He is now doing fantastic and has made an almost complete recovery from a neurological disorder. Jeff offered to let me post on this and I’ll probably right something up in the future. In the meantime, I look forward to getting back into the AGW discussion.

  13. Jeff Id said

    #12 Welcome back Jeff, Glad to hear the good news, I was getting worried.

    I think we could use some help with the cloud data if you get the time. Ryan has done some interesting corrections, however if a pre-determined basis was used for correction which brought trends in line it would be better. No idea if it can be done, after all Dr. Comiso did pretty well with what he had to work with.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: