I took some flack on how I missed such an obvious problem in the data. Well, things are rarely as clear as they first seem. Let’s look at the facts, below is a plot of the uncorrected bootstrap data.
See the offset at 1987, no how about the same graph zoomed in.
There it is, that little scribble after 1987.5. That’s the entire difference between my first and second posts. But it’s worse than just this, here is some text from the actual pages.
link here- http://nsidc.org/data/smmr_ssmi_ancillary/area_extent.html
In computing the total ice-covered area and ice extent, pixels must have an ice concentration of 15 percent or greater to be included; thus, total ice-covered area is defined as the area of each pixel with at least 15 percent ice concentration multiplied by the ice fraction in the pixel (0.15-1.00). Total ice extent is computed by summing the number of pixels with at least 15 percent ice concentration multiplied by the area per pixel. Sea ice concentrations are assumed to be 100 percent around a circular sector centered over the Northern Hemisphere pole (known as the pole hole) which is never measured due to orbit inclination. The Southern Hemisphere also has a pole hole; however, it does not affect this sea ice data set; since only land is under this hole. For SMMR, the hole is 611 km in radius and is located poleward of 84.5 degrees north. For SSM/I, the hole is 310 km in radius and is located poleward of 87 degrees north.
As I read this paragraph the data at the north pole is simply filled in for both “area and extent” and would have little or no offset due to the sensor area change. This is in regards to the data I used.
It gets worse.
http://nsidc.org/data/nsidc-0079.html
Important Note: The most recent revision (Version 2, released September 2007) has tie-points adjusted to be consistent with the AMSR-E Bootstrap algorithm. This involved a complete reprocessing of the entire SMMR-SSM/I time series. Because Version 2 is inter-calibrated with AMSR-E, it is NOT consistent or compatible with the original version of this data set (Version 1). Users should acquire the entire Version 2 data set in order to update their time series.
This is in regards to how the olde SMMR data is aligned to the SMM/I data. The AMSR-E is the new style data. The paragraph clearly states that the correction made the data consistent but you must have V2 data. Which I did. There’s also some info on how the data was aligned here.
Think were done yet. No way.
http://nsidc.org/data/docs/daac/nasateam/index.html#tie
The selection of the SSM/I F8 tie points (table 1, p. 40) was based on an analysis of SSM/I TBs, PR-GR distributions, histograms of sea ice concentrations, and on comparisons with near simultaneous measurements from the Nimbus-7 SMMR during July and August 1987. The two sets of SSM/I tie points (one for the Northern Hemisphere and one for the Southern Hemisphere) represent a global set designed for mapping global ice concentrations. While this global set of tie points provides a uniform measure of sea ice concentration on the large scale, improved accuracy is obtained with the use of regionally selected tie points (Steffen and Schweiger 1991).
A very interesting paragraph. The data’s alignment is being discussed including histograms of sea ice concentrations. The tie point for the two datasets was calculated using near real time data from the two systems.
Not yet, there’s more. Remember my anomaly data, well I happened to check it on the site at this link.
This looks a lot like my original anomaly plot (below) on a slightly different scale.
If this anomaly plot is correct then my entire original post is correct but wait there’s more.
Here’s the IUIC cryosphere graph which looks impressively similar to my original data. Click for the large version.
Below was part of tamino’s reply from a completely different site (link here )that points to a dataset which I didn’t use. Still if you look into the source of the data (SMMR and SMM/I) they are the same and I did find the step in the second graph above and admitted as far and widely as I could my screw up.
Here’s the text from the link.
Important Note: The “extent” column includes the area near the pole not
imaged by the sensor. It is assumed to be entirely ice covered with at
least 15% concentration. However, the “area” column excludes the area not
imaged by the sensor. This area is 1.19 million square kilometers for SMMR
(from the beginning of the series through June 1987) and 0.31 million
square kilometers for SSM/I (from July 1987 to present). Therefore, there
is a discontinuity in the “area” data values in this file at the June/July
1987 boundary.
As you see this paragraph is quite different from the info I was reading and it represents a different dataset. Another point is that the little jig in the second graph above occurs at the July/August boundary not at the Jun/July boundary as stated above.
Well, I think there is still some small question as to whether adding the area back in is appropriate. Were the tie points discussed above the result of a rescaling operation that offset the data to a point where a trend analysis can be done? I doubt it now but why would they introduce the data as a continuous time series with a simple correction left out.
I emailed the National Snow and Ice Data Center for clarification.
It should be interesting.
the Air Vent 
Jeff, don’t beat yourself up. You made an honest mistake, you came out and admitted it and you corrected it. You are open and honest with the data which is more than I can say for the alarmist side…
Thanks Charles,
I just wanted people to know that there is no mention of the step in the data on the documentation. There is in fact a detailed description of how the data was aligned so well. I had no way to tell this was there other than discovering it myself or having friends in the ice center.
I probably would still have eventually found it from following some of the comments.
AMSR and AMSR-E (Aqua) are the most recent satellites and have higher resolution than SSM/I and SSM/R. JAXA and Uni-Hamburg among others use date from AMSR-E. The starting date for this data is June, 2002.
I’m convinced that CT is using version 1 data for their seasonal and annual extent data before 2007 and version 2 starting in 2007. That’s the only way I can explain the step change down in 2007.
Thanks #3
I made the update. I wonder if you’re right about 07 the down step was pretty severe for other reasons to apply. It’s worth looking into, that would be pretty interesting news wouldn’t it.
re: Jeff’s post
“why would they introduce the data as a continuous time series with a simple correction left out” (?)
Why indeed. And if they did leave the correction out, why would there be nothing to draw attention to needed corrections? Is it possible such a notation has been overlooked? After all, this is data being made available for public use.
I think it’s either overlooked or is so known in the industry they ignored it. It isn’t documented on the site anywhere I can find and I spent a long time there.
The one other ‘unlikely’ possibility is that the data prior to the discontinuity in this case already contains the corrections in it and they just let the endpoint not fit perfectly. Why else would they provide the same anomaly graph I calculated? I wrote to the NSIDC to ask how they would recommend handling the issue.
Maybe I am not considering something, but based on the comparisons of the two anomaly graphs it would seem that either a)NSIDC made the same mistake you did (unthinkable) or b) The NSIDC data has corrected for the 1987 step as you imply in your post. This is interesting. I will be anxious for NSIDC to provide clarity on this when they respond. Just for fun, could you run a “corrected” version of your anomaly graph?
The problem I have with the correction is that summer doesn’t always 100% fill the sensor areas. Any correction I make therefore has too high a summer slope. At least that’s what I think now.
It is also interesting to me how similar the IUIC cryosphere plot looks so similar to my original result.
BTW: The upward step in the NSIDC anomaly at around 1987 has more than one point so most of the step is real. I only mentioni it because it looks so severe in the plot.
Hello,
I’m new here, but intrigued by the discussion.
Just for openers, let me ask what your take on the IPCC ice core data charts is. I’ve seen Gore’s smoothed version in his book, “An Inconv Truth”, and am trying to understand the spikes of temp/CO2, when snow andice are deposited on the poles, and what role human activity plays in climate cycles.
Thank you.
There is a lot of info to be considered in ice core analyis. My opinion is that if you want answers you should look at the data yourself and not accept what people tell you. There is a lot of politics involved in the interpretation.
My own view is that the ice data shows a great deal of filtering due to diffusion of the gasses in the core. The o18 isotope ratio is used as evidence of warming whereas it really has a strong wind signal involved. This means that we really don’t know for sure if the temp graph is temp.
If we accept that it is, diffusion caused by permeability of snow and eventually ice as the snow piles up causes heavy diffusion of the gasses. Peaks are therefore removed/filtered. The effect is noticeable as older ice data looks more rounded.
If we accept the temp and CO2 data:
The Co2 graph is very well known to lag the temperature graph by about 200 to 1000 years. CO2 is therefore not the cause of the variation in temp but rather is the result. This is explainable by the solubility of CO2 in salt water. It is backwards from what you expect from high school chem, where CO2 is actually more soluble in cold water than hot.
I have heard anecdotal reports that Al’s movie shifted the CO2 plot by the infenetesmally small amount required to align the graphs. I have not confirmed this. If it is true that they would shift the graph by a microscopic 500 years (on that huge 700kyr scale). If this is true that means to me they recognized the problem and decided to cover it.
This is a scientifically well known inconvenient truth.
Regarding human activity. I’m not sure we do anything measurable to climate yet. I understand the CO2 greenhouse effect well but the models are not trustworthy in my opinion. I have not studied them thoroughly enough to debunk any I just have seen some of the output and they vary widely depending on assumption.
We may be warming the earth but even my adjusted post shows very little ice melt. I doubt that the effect is anywhere near the doom scenarios we hear. Much of the hype reads like government issued propaganda to me.
Jeff,
The smoothing at depth is not a result of diffusion, it’s a problem with dating. The deeper in the core, the smaller the difference in spacing between years. There’s a practical limit on the physical spacing between samples, so high frequencies are effectively filtered out by the Nyquist criterion.
Also there is very little diffusion even in the firn layer before the pores close completely. There is exchange with the atmosphere very close to the surface, but that only affects a time range of less than a decade. Besides, diffusion requires a concentration gradient. The differences in concentration per year are so small that there is insufficient driving force for diffusion. Gases do not diffuse out of the bubbles into the ice because the ice is already in equilibrium with the bubbles when it forms.
Re: #4,
Yes, there’s a drop in extent in 2007, but the drop is much greater for CT than for other sources of extent data like JAXA and Uni-Hamburg. For Summer (July, August and September) average extent, the delta in extent between CT and JAXA goes from +973,000 km2 in 2006 to -233,000 km2 in 2007. This is way larger than the year to year differences in 2002 to 2006. The average difference for 2002 to 2006 is 976,906 km2 and the average difference for 2007 and 2008 is -219,239 km2. The graph of the data is here.
Jeff Id Says:
December 18, 2008 at 4:36 am
If we accept the temp and CO2 data:
The Co2 graph is very well known to lag the temperature graph by about 200 to 1000 years. CO2 is therefore not the cause of the variation in temp but rather is the result.
– This has been so roundly debunked it appears in the popular lit. CO2 fills both a positive feedback role and an initiating role depending on the specific event one checks.
http://www.newscientist.com/article/dn11659
http://www.skepticalscience.com/co2-lags-temperature.htm
And lastly, from hated RealClimate, a site which everyone likes to complain about but no one seems to disprove:
http://www.realclimate.org/index.php/archives/2007/04/the-lag-between-temp-and-co2/
This is explainable by the solubility of CO2 in salt water. It is backwards from what you expect from high school chem, where CO2 is actually more soluble in cold water than hot.
– If true, this would seem to be an explanation for the lag which would not negatively affect AGW theory. If the new CO2 lags in the record because it is initially being absorbed by the oceans, then perhaps it is already doing warming work from day 1. =)
cmb Says:
December 18, 2008 at 6:42 pm,
I have no problem in principle with the argument that a lag in CO2 concentration at either end of glacial cycle does not disprove that CO2 acts as a positive feedback, but the articles you link to are merely hand waving. Can you provide a link to an article that actually demonstrates, as opposed to asserts, how this works? Boris couldn’t or wouldn’t when he was arguing the same point with lucia at The Blackboard.
Cmb,
You apparently have not been to The Blackboard. Lucia Lundgren is having a field day with RealClimate.
#11
Gases do not diffuse out of the bubbles into the ice because the ice is already in equilibrium with the bubbles when it forms.
I understand your points about dating and diffusion, however suppose that CO2 spiked to 2 times concentration because of some major natural event. I believe that the layers as they press close would have enough gradient that the equilibrium would not exist.
Your point about measurement dating is likely the majority effect, I should have mentioned it. I have read about dating of ice cores also. I get a bit frustrated with the assumption by the scientists that diffusion doesn’t occur. So far the supporting evidence I have read for this assumption is weak, not that better evidence doesn’t exist. I’ve done no calcs myself but it is worth looking into.
Incidentally, thanks for the add on.
cmb Says:
December 18, 2008 at 6:42 pm
“This has been so roundly debunked…”
Have you read “CO2 Acquittal”? URL: http://www.rocketscientistsjournal.com/2006/10/co2_acquittal.html
DT
Hi Jeff,
I forgot the step in the data, too. But I am not a scientist and I do not want to prove a conspirancy. But I think, you claimed there is some data manipulation going on. (For example: …I will request the data for this plot from the IUIC guys again but it clearly is another adjustment in favor of AGW…) Did you apologize to the people at NASA und UIUC directly? I think, you should because they do their jobs honestly.
The CONSPIRACY THEORIES of “skeptics” are the most annoying things in the climate debate. I like the rest quite a lot, it is interesting and i learn a lot. Again, stop claiming frauds, please.
Cheers.
#18
I don’t expect to issue an apology as this is another adjustment in favor of AGW. I simply meant that the ice was melting more than originally the data showed, nothing else. From the tone of some of my other posts I can see how you would take it wrong.
In defense of my other posts–
Consider that the AGW science is led by huge propaganda engines, it’s not the only science with an advertising budget but I bet it’s the biggest budget. Also consider the paper Mann08 which I believe has been intentionally manipulated to produce a hockey stick. No scientist can be so perfectly stupid to reproduce hockey stick using various math techniques at least four times on accident. I therefore can’t prove intent but anyone who has worked with data knows this work is incredibly false.
I wish you would also consider the structure of the IPCC before you get too skeptical that this science is being manipulated. Their power, funding and growth depends on disaster. At least consider the point that only the more extreme conclusions are highlighted. Papers which are moderate or negative to global warming are regularly swept under the rug. The IPCC reports are used for global policy. The moderate papers in fact may be the correct results, wouldn’t you rather work from the correct data when making global policy. I would.
As examples check out Dr Craig Loehles paper on tree ring non-linearity. This article demands an answer from paleoclimatology at least, yet none is forthcoming. You can read it here by the 800 lb gorrilla in the hockey stick locker room post. If you understand why this must be addressed then ask yourself, why isn’t it addressed. It is a newer paper so there hasn’t been much time yet but it’s not just his paper but many others referenced by his paper which also describe the problems. Yet still people use tree rings without consideration or reply.
Also consider that there are mathematically reasonable temp reconstructions which show temps 1000 years ago higher than today. Why aren’t those graphs referenced multiple times by the IPCC. Why would the IPCC only choose the Mann hockey stick?
There are frauds in this science there are also mistakes, I didn’t claim any fraud here. If I find it though, I’ll let you and everyone else know.
Hi Jeff,
okay, I understand. I think the climate policy and also the climate change and global warming may have a big impact to our life. So, skepticism and control is very good and necessary. I think, it is good, there are skeptics like you and others. It is also good, that scientist are blogging, too and sharing their thoughts. It is fantastic that much of the data is online and everybody can use it (one has to be careful). But sometimes I think, it can be counterproductive because there is too much personal stuff going on and no real discussion, too much blaming. For example, the GISTEMP data. Maybe, there are quality problems, but there is no fraud (I believe) as written on many blogs and blog comments.
I do not follow the Mann-McIntyre-Discussion because I think there are other works which show similar results, i.e., it’s pretty warm nowadays. So, I cannot comment about this. BTW, some climatologists (no skeptics ;)) thought the concentration on the hockey stick was wrong , and I think in the new IPCC reports of 2007 there is no such focus. Maybe, i will read something about it again. I do not know.
I do not agree with you about the IPCC. I think the report is pretty good and the summaries are relatively moderate and cautious. I like it as a good summary of the current state of the research.
I do like sites like this, I have to recheck my beliefs (i am not an expert, I have to believe at a certain point). But I still think climate changes will be a big problem in the future.
Anyway,
Merry Christmas :).
Jeff, any word from NSIDC yet?
I’ve been in contact with the NSIDC regularly. I’ll have an update soon.
Jeff