the Air Vent

Because the world needs another opinion

Reading Between the Clouds

Posted by Jeff Id on March 29, 2009

Jeff C left this comment over on Climate Audit. It touches on a very important subject regarding the current data, the instrument quality. Satellite’s experience harsh environments in space. Temperatures fluxuate by hundreds of degrees, mircrodebris hurtle around at speeds which make bullets look like they’re standing still, there are high radiation levels and consequently the instruments decay over time. During the satellite record of the antarctic there were 5 separate satellites used to collect the data. All of these factors need to be accounted for in understanding the data quality.

This is especially important when trying to nail down a very subtle (less than 1 degree) shift in data that varies by 60 degrees C being measured from an environment (space) that shifts every hour by hundreds of degrees C. Clouds in the Antarctic are known to be both warmer and cooler than ground temps depending on a number of factors. These clouds must be masked out which is an extremely difficult process especially when they can’t be seen at night. It’s no surprise then that clouds are cited by the NSIDC as one of the sources of noise in the surface temperature data.

Well anyway, Jeff took a look at the cloud levels as presented by the NSIDC and check out what he found.


Jeff C

I have been reviewing the cloud cover percentages from the UWisc data set. These values can be extracted from the UWisc data set using Steve’s script in the UWisc thread. Change “surft” to “cldtype” to retrieve the cloud type parameter. For the monthly datasets, cloudtype is a value from 0 to 99 that gives the per-cent of time a cell was considered cloud covered during the given month.

I have all the individual values, but calculated some monthly means to give a quick read on how often the continent is cloud covered. I was surprised by the high frequency, but also found some other interesting items that make me question the effectiveness of the cloud detection and masking process.

Here are plots of the continental monthly means (average of all 5509 cells) for the 0200/1400 average, the 0200 alone, and the 1400 alone. I have added the time periods of the various spacecrafts to the plots.

There seems to be a clear step at the transitions from one spacecraft to another. In the 0200 set, there is a downward trend over time of the cloud percentage over the life of each spacecraft. This isn’t readily apparent in the 1400 set.

The 1400 set has a clear seasonal pattern, that isn’t as clear in the 0200 set. This may be caused by problems with cloud detection at night as the optical channels aren’t much help in the dark. This may explain some of the seasonal difference in the 1400 set.

The continental cloud covered percentage exceeds 80% fairly often. It makes me wonder how high-quality temperature values can be extracted from these periods with so much of the continental temperature measurements cloud contaminated.

Here is an average over time for each of the continental cells. This is over the entire UWisc record of 1982 to 2004. White is 50%, blue is less than 50%. More than half of the continent is cloud covered 50% or more of the time.

As you might expect from the average over time plots, the continental plot looks considerably different during each of the different spacecraft durations. NOAA-16 is particularly striking as it looks completely different from the others.

It bears a strong resemblance to Dr. Comiso’s trend plot shown in the SI. It might be a coincidence; I need to dig into this more. I’m going to write Dr. Key at UWisc to see if he can help answer some of the questions.

I’m happy to forward the code and data files if anyone wants to look into this more.

21 Responses to “Reading Between the Clouds”

  1. TCO said

    The 0200 plot is striking. Would like to see a lit search to see if this is already known (or some sort of knowledge of trend or other deterioration that is behind the plot). If not, would be interesting to document it and see what impact it has on things

  2. Jeff Id said

    #1 For sure it’s interesting, it could have a lot of implications. Jeff did a great job and has spent a lot of hours looking into this. These trends correspond to actual cloud areas identified by the instruments. If the masking is inconsistent the long term trend could easily get tweaked.

  3. Jeff C. said

    #1 and #2. Thanks for the comments. I would like to add a word of caution that we are still puzzling though this dataset and aren’t quite sure what to make of it. This was provided by Dr. Key at the University of Wisconsin-Madison using the CASPR program (Cloud and Surface Parameter Retrieval) using one of its cloud masking algorithms. It is not the Comiso dataset and we don’t know if Dr. Comiso used CASPR or something of his own. There may be some spacecraft to spacecraft calibration that Dr. Key didn’t apply that Dr. Comiso did apply.

    There are a number of other interesting cloud parameters in the dataset that I hope to unravel over the next few days. This is the “cldtype” parameter, “cldphase” also looks like it might be interesting.

  4. Layman Lurker said

    Jeff C. Was there anything in your research that explained what corrections or adjustments take place before the time series data product is posted? The paper I linked to a couple of days ago talked about many potential sources of measurement error. Particularly in the winter months with temp inversion and stratospheric clouds.

  5. Fluffy Clouds (Tim L) said

    Let my handle do the talking LOL!
    Thank you J&J

  6. IanH said


    Is it possible to devise a metric which gives us some measure of how many days per month for a cell or small group of cells we could have a clear(ish) view of the ground 0200 & 1400 and hence how much of this data can be used for its stated purpose

  7. George Tobin said

    Doesn’t this finding tend to confirm something along the lines of Lindzen’s “iris effect” such that when it warms (as it did on average for the period in the graph) cloudy-moist regions contract such that satellites would tend to report a net loss in cloud cover?

  8. rephelan said

    The things one learns. I hadn’t realized that the operational life of those satellites was so short. Your charts seem to indicate that their useful life is actually a lot shorter – is there any kind of over-lap data to compare e.g. when NOAA-14 was replaced by NOAA-16 was there a period of over-lap when they were both active and measuring the same thing? I’m sure someone over at NOAA had to be asking the same question.

  9. Jeff Id said

    Here’s what the methods section in the paper says.

    Cloud masking is probably the largest source of error in the
    retrieval of TIR data from raw satellite spectral information. Wehave updated the
    data throughout 2006, using an enhanced cloud-masking technique to give
    better fidelity with existing occupied and automatic weather station data. We
    make use of the cloud masking in ref. 8 but impose an additional restriction that
    requires that daily anomalies be within a threshold of 610 uC of climatology, a
    conservative technique that will tend to damp extreme values and, hence, minimize
    trends29. Values that fall outside the threshold are removed.

  10. Jeff Id said


    I don’t believe there is any overlap for these instruments simply because of the huge step and decay pattern. For sea ice from the same group an overlap was used for calibration.

  11. timetochooseagain said

    It looks like there are some clear satellite cross-calibration problems-the data are clearly often discontinuous between satellites.

  12. rephelan said

    Ouch! Is this the same satellite data set that Dr. Steig was using? Is it too early to comment on the effect it had on his analysis?

  13. rephelan said

    What would happen if you took the first 12-18 months of each satellite’s records and then used something like REG-em to infill the remaining observations?

  14. Jeff Id said

    #12, The data came from the same instruments. Who knows how or if Comiso corrected for it.

  15. Jeff C. said

    #12 This is NOT the same set as Steig’s as it was post-processed using different techniques by different individuals. We don’t have this level of detail for the Steig/Comiso set so I am using this much more complete set to try and spot trends and shortfalls. See comment #3. These both originate from the same raw data.

    I’ve got some new info I’ll put up soon regarding the cloud cover percentage using a different parameter from the U Wisconsin data set. It looks better than those shown above, but still has noticeable discontinuities at the spacecraft transitions. I think that the CASPR software “cldtype” parameter may have a bug that exaggerates the discontinuities, so use caution in drawing conclusions from the data until I have a chance to post more. I’m going to email Dr. Key to see if he can help explain what we are seeing.

    One thing that is becoming clear is that the cloud cover pattern has a very distinct similarity to Dr. Comiso’s trend plots. This could mean the clouds are correlated to warming or possibly that the Comiso set is cloud-contaminated. Again, too early to draw conclusions but it certainly is interesting.

    Jeff – I’ll forward the latest later today.

  16. Jeff Id said

    #15 Ship it over when you get time. This should be pretty interesting.

  17. JAE said

    “This is especially important when trying to nail down a very subtle (less than 1 degree) shift in data that varies by 60 degrees C being measured from an environment (space) that shifts every hour by hundreds of degrees C. ”

    ?? HUNDREDS of degrees C?? Aren’t we exaggerating a little here?

  18. Jeff Id said

    #17 Nope, the temperature shift from day to night in space is that large. I didn’t look up a reference but I bet Jeff C could nail it down pretty close.

  19. Layman Lurker said

    From J. Cosimo, “Variability and Trends in Antarctic Surface Temperatures from In Situ and Satellite Infrared Measurements”
    JOC, May 2000:

    “The monthly averages derived from the infrared data are not true monthly averages since they are just averages of surface values during cloud-free conditions. The magnitude of the associated error has been studied by taking the differences between the true monthly averages, using station data, and the monthly averages of cloud-free data (as identified by the AVHRR cloud mask) from the same stations. The results show that the cloud-free only monthly average is colder than the true monthly average by about 0.3°C with a standard deviation of about 0.6°C during summer and 0.5°C with a standard deviation of 1.5°C during the winter.”

    Is this perhaps an indication of how Dr. Comiso infilled cloud masked grids? If he used the above or something like it to calibrate cloud infilling with surface stations, then that might explain why the Steig AVHRR data fits well with corresponding AVHRR grids.

    Thinking out loud here so let me know if this does not make sense:

    Is Dr. Comiso saying above that the amount of warming chaulked up to cloud contamination is determined by calibration to surface station monthly means? If you throw insturmental bias into this formula then there is obvious potential for over or underestimating this warming factor. So the warming factor is thrown off AND there is biased estimation of % cloud contamination.

  20. Layman Lurker said



    “then that might explain why the Steig AVHRR data fits well with corresponding AVHRR grids.”

    should read:

    “then that might explain why the surface station data fits well with corresponding AVHRR grids.”

  21. Jeff C. said

    #19 I think you might be on to something here. I have read that section of the paper several times and I never noticed the mention of the surface data in that sentence. It does seem to suggest there is some sort of calibration of the AVHRR data to the surface stations.

    I’ve suspected this for some time, but had never found a mention of it. I’ll go reread the paper you linked.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: