the Air Vent

Because the world needs another opinion

Dr. Weinstein’s Experience With Peer Review

Posted by Jeff Id on May 31, 2009

I always prefer to put up articles which show actual experience rather than a hearsay report. Dr. Weinstein has taken the time to describe his own first hand experiences with peer review and confirmation bias in a far less politicized field. It is IMO one of the primary issues in play in AGW, as the Santer issue discussed at Climate Audit demonstrates. I definitely plan to discuss that one more, after SteveM’s correct and reasonable rebuttal paper is accepted by some journal.

Dr. Weinstein’s discussion has to do with the reduction of airfoil drag by manipulations of the boundary layer. – Something I also have experience in ;)

—————————————————————————–

Some Limitations of the Peer Review Process and Its Effect on the AGW Issue

Leonard Weinstein, ScD

May 29, 2009

Monographs, books, technical papers, white papers, and other forms of published material that are to be widely disseminated should ideally be examined by independent reviewers (generally called peer reviewers) with reasonable knowledge of the subject area. The reviewer may or may not be as expert in the details of the subject as the author, but should be at least generally expert in a broader overlapping field. Frequently papers have material that covers more than one subject area and people within a narrow field may do poorly reviewing the broader subject. It is frequently best to have some reviewers that are knowledgeable people but not as close to the narrow field as the author, for a more independent and broader coverage.

The reviews, possibly resulting in suggested changes, do not assure the final material is correct. The lack of a review also does not mean the results are not correct and clear. It just increases the chance for errors to slip through.


If the paper is very complex or has numerous references, it may not be possible to do as good a review as desired in a reasonable time unless the reviewer is intimately familiar with the work. These reviews are often done within a small group of experts, who may even have collaborated on each others papers. The members of the overly narrow review group may have evolved toward a common understanding on the subject, and if the understanding is not correct, almost all of the limited number of experts in that field may make common mistakes that outside reviewers may not make. However, a reviewer outside that group may not be expert enough to be fully on top of the material. This is a problem without a clear solution, especially in fields with a limited number of experts. There are also occasions when errors in assumptions used in developing equations, or in interpretation of instrumentation result in erroneous conclusions that are repeated in many papers until the error is caught and corrected. For this reason all papers must be considered suspect until history vindicates or falsifies them. A paradigm shift on a subject may result from these types of errors being corrected at later times.

An example of the problems that may arise with peer-reviewed papers occurred on a project that I worked on for several years. The branch I was working with was examining possible methods to reduce turbulent skin friction over surfaces, with the goal of reducing aircraft and ship drag, and thus save fuel. We had developed several successful concepts that gave small but useful reductions in drag. These included “longitudinal microgrooves”, “large eddy breakup devices”, and for water flow, “air bubbles in longitudinal grooves”. An additional concept we worked on was a “compliant wall”. This last method appeared to be particularly promising as a method to damp out near wall turbulence by absorbing wall pressure fluctuations. Unlike the other methods, this last technique required a soft and movable wall, so it was more difficult to measure the drag of a test object. Several other groups all over the world also started work on this concept (bandwagon effect), and our group (and many of the other groups) obtained what we thought were favorable results in several tests. Dozens of peer-reviewed papers were published in main line journals (e.g., Journal of Fluid Physics, AIAA journal, etc.). The main experimental technique used to measure the drag for this type of model was a hot-wire Reynolds stress probe. Test results seemed to indicate a large reduction in drag. In order to independently verify the favorable result obtained with this technique, I designed and had built a large sensitive drag balance to directly measure the drag on a panel model. The direct measurement did not show the reduction in drag indicated by the hot wire probe. After much analysis, I found the cause of the problem. Reynolds stress drag measurements are proxy drag measurements. They are related to the drag under some special assumptions. In this case the critical assumption was that all of the vertical motion energy in the boundary layer was due to turbulent flow, and stayed in the boundary layer. It turned out that with a compliant wall, the flexible wall motion induced a vertical motion pressure wave that propagated as sound out of the boundary layer, so the required assumption was not valid.

Many of the other favorable tests had used the same type of probe with the same false positive result. Those that did not use the hot wire often gave results that were confusing, but many still passed peer review. It appears that since some of the other papers seemed to be getting the correct answers, they assumed they had it also and selectively chose the more favorable parts of their results for publication. The idea was they knew what the answer had to be so they just had to be selective in choosing data so they would be with the successful group.

When my paper was published showing the cause of the errors in the results, almost all of the compliant wall drag reduction efforts ended.

It appears to me that the AGW supporters are on a similar bandwagon. Initially the observation that the temperature had significantly increased (1 degree F) over the last 150 years, and the atmospheric CO2 content had increased significantly (over 30%) in the same period suggested a relationship. Previous analysis had indicated that CO2 is a greenhouse gas, and could cause some temperature increase. The effort to show that the temperature rise rate and level was unusual was pushed by several scientists, and initially it looked as if this might be true. After many other scientists jumped on the bandwagon, because they assumed the hypothesis was valid, they then had a personal stake (funding, prestige) to support the claims. Later data and analysis seemed to show the temperature increase was a local period of global warming (GW), but did not have a dominant component of AGW. This threatened the whole structure of the AGW position, and as is typical when a paradigm shift threatens to occur, was met with considerable resistance. This was further exasperated by the political and social involvement that had started a major drive to cut man made greenhouse gas production, and at a very high cost to society.

Recent efforts by even well respected authors to publish papers that falsify claims of AGW have had great difficulty passing peer review, apparently because they went against the current paradigm, not because they were shown to be in error. The misuse of claims of support for AGW by “peer reviewed publications”, and rejection of analysis because it is not peer reviewed is often used to try to discredit the opposition. This is not how science is done. The facts should speak for themselves.

New technology is changing some of the older paradigms of communication, and use of electronic media now allows very rapid and sometimes real time interaction discussing material and presenting results. Paul Coppin stated in a response to a recent blog “Technical blogs are rapidly replacing the “letters” sections of formal journals as the place where a public airing of a journal article or topic takes place. The very nature of blogs also provides a means for the lay populace to look inside both the science and the process of the science. In the past this has been generally closed to common scrutiny. In particular, science writers have access to information and opinion they had to previous dig out or interview for”. This crossover of discussion will surely increase. Since the purpose of science is (supposed) to advance knowledge, the real time open interaction may actually do a better job than the normal peer review process for much (but not all) material. Formal publications still should be made when possible for unique data or concepts, but discussions of the consequences are probably better done in the electronic medium. One major purpose of the formal publication is getting the author credit for the material. There is not presently an equivalent way to archive and give credit for a publication on the web. There needs to be such a structure made available. The draft of a publication may be put on a web site and left open for review and comments. This might actually be a more effective peer review process for some types of papers since a wider audience has access to look for flaws. This may be done in conjunction with some formal peer reviewing. The corrected final paper then may be archived, but with continuous access to later catch overlooked problems.

16 Responses to “Dr. Weinstein’s Experience With Peer Review”

  1. Jeff Id said

    Thanks for this, the topic of boundary layer drag hit home a bit. As an Aeronautical undergrad, I worked with a master student on their thesis. The premise was to vibrate an airfoil skin in different modes to create a drag reduction, we used hotwire as well as a force mount for the aluminum skinned airfoil at different frequencies and amplitudes.

    Unfortunately, the noise level was too great and we couldn’t discern the signal from the turbulence in our low quality wind tunnel system. My part was to develop circuitry which allowed the viewing of modes of vibration by pulse width modulating an acustoptic modulator in sync with the airfoil vibration. Very cool project.

    I bet if we had found an improved drag signal greater than the input energy, we would have fallen for the same problem you describe.

  2. Page48 said

    The draft of a publication may be put on a web site and left open for review and comments. This might actually be a more effective peer review process for some types of papers since a wider audience has access to look for flaws.

    This is a great suggestion, but I suspect Hell will freeze before scientists go for it.

  3. Alex said

    Journal of Fluid Physics doesn’t exist. Likewise probably the “author” which never did anything in this field (find his publications, please). It is obvious that the Reynolds stress cannot be used as a drag measurement quantity. In addition, hot-wire probe measures velocity, not the Reynolds stresses.

    Apparently, this is another way of ‘misusing’ the popular subject of ‘peer-review, yes or not’. Definitely, NOT, due to all kind of “ScD” that freak out the public with the near-scientific works.

  4. Alex said

    well, somebody by this name does exist, sorry:

    http://www.aiaa.org/content.cfm?pageid=406&gTable=japaperimportPre97&gID=60485

    Reply: For some reason WP put this in the spam bucket.

  5. Jeff Id said

    #3 Alex,

    I’m interested in seeing Dr. Weinstein’s reply. Certainly the implication that he’s lied is a bit distasteful. Especially from your description.

    As you obviously know Alex, hotwire probes are single axis devices. Sometimes multiples are used for directional velocity but as far as I’m aware it’s not often done in boundary flow studies where proximity to a surface with minimal disturbance of flow is required and flow direction is known/assumed a priori. When you include the assumption of viscous laminar flow at the boundary layer at static pressure Reynolds stress is a natural simplification of average velocity.

  6. PaulM said

    Alex, have you heard of Google Scholar? You can find his publications in two minutes by typing in “L M Weinstein”.

    Weinstein’s only crime seems to be the muddling up of two journal names (Journal of Fluid Mechanics and Physics of Fluids).

    It is true that there are ‘bandwagons’ in science as this seems to be an increasing problem these days. Research that follows the bandwagon is waved through and published, regardless of whether it is correct or adds to knowledge, while anything that tries to go against the bandwagon is trashed. Of course AGW is the best example – the people involved are trying to outdo each other in terms of scary predictions in order to hit the headlines and secure further funding. Meanwhile those who raise valid questions and doubts cannot get their work published (because it is trashed in peer review by those on the bandwagon) and so have to resort to blogs or writing books to get their views across.

    The latest McIntyre & McKitrick draft paper has been put on a website

    http://arxiv.org/abs/0905.0445

    although that site does not have the facility for comments.

  7. lweinstein said

    Alex:
    It is true that I misremembered the journal of Fluid mechanics name (I wrote the present note from memory & that work was long ago). Hot wire Reynolds stress measurements are made frequently. Either an X probe (two crossed wires, or a single slanted wire that is then rotated does the job. My doctoral dissertation was the main publication showing the effect of the error in assumption. Many of the papers published on the topic were published in several other journals. As to my publications, go to: http://docs.google.com/Doc?id=dnc49xz_21g6h6q6f2&hl=en

  8. lweinstein said

    Alex:

    A Google search using: Compliant wall drag reduction will get some of the papers you think are not there.

  9. lweinstein said

    Jeff,

    Small X-wire hot wire probes are used near the wall to measure Reynolds Stress as well as fluctuating and mean velocity. They are less disturbance than pitot probes due to their small diameter. probes can be made very small. If the wall does not move, the Reynolds stress can be related to local skin friction.

  10. Jeff Id said

    #9 I’ve used them many times. I’ve never seen a pitot used for boundray layer analysis – not that it couldn’t be done with some kind of microtube.

    We typically used airfoil shaped single probes on our experiments assuming no movement in other directions. I understood every point you made in your article as being reasonable because it is the same problem we grappled with. Fluid energies with a moving surface sound pretty ugly to work out.

  11. Jeff Id said

    I should also add that I was pretty comfortable in telling Alex about his error because the story made too much sense from a science, math and experimental standpoint – including the generalization of Reynolds stress and drag. You would have to be a savant to make this up if you didn’t work in the field.

  12. woodNfish said

    So Alex, will you offer Dr. Weinstein an apology for accusing him of lying without giving him an opportunity to explain his simple error?

    It seems that peer review is held up to a higher standard by the general public who mistakingly believe that it means a paper’s research has been proven. This incorrect belief has been promoted by AGW alarmists such as the Team at Real Climate and by an opportunistic Press looking to hype a non-crisis.

  13. lweinstein said

    I now see where I messed up on the “Journal of Fluid Physics” comment. There is a “Journal of Fluid Mechanics”, and a “Journal of Physics of Fluids”, and I just mushed them together.

  14. pyromancer76 said

    Nothing should be considered “peer-reviewed” if the data and methods for the research are not posted in full. It does not matter how complicated the paper or how many authors are on the paper, data and methods must be posted. If they are not, then no one really knows if each of the authors stands fully behind the research to which they have given their name. And, one can suspect that the primary authors might be hiding something about their work. And, one can wonder at the lack of due diligence of the so-called peer reviewers. And, finally, the reputation of the so-called (premier)science journal that did not require such posting should a suffer serious decline — degradation might be an even better word.

  15. Jack said

    “The draft of a publication may be put on a web site and left open for review and comments. This might actually be a more effective peer review process for some types of papers since a wider audience has access to look for flaws.”

    “This is a great suggestion, but I suspect Hell will freeze before scientists go for it.”

    This has already been in place for many years with Atmospheric Chemistry and Physics.

    http://www.atmospheric-chemistry-and-physics.net/

    Yes, there are many climate-related articles published there.

  16. ThomasL said

    “The draft of a publication may be put on a web site and left open for review and comments.”

    Conceptually, this isn’t much different than standards’ organizations do. For example, see the RFC (Request For Comment) submission process: http://www.rfc-editor.org/pubprocess.html

    To be 100% clear though, I am not advocating that a paper once reviewed should become an international standard. :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 148 other followers

%d bloggers like this: