9 - The Immense Edifice That Wasn't Many people believe the IPCC goes to the trouble of verifying the research on which it bases its conclusions. An oft-repeated quote from President Barack Obama's science advisor, John Holdren, is a marvelous example of this. Holdren says the IPCC is the source of "the most important conclusions" about climate change, and that these conclusions rest on:...an immense edifice of painstaking studies published in the world's leading peer-reviewed scientific journals. They have been vetted and documented in excruciating detail by the largest, longest, costliest, most international, most interdisciplinary, and most thorough formal review of a scientific topic ever conducted.
Here's a similar quote, from climate modeler Richard Rood:The scientists who write the IPCC reports use exquisite rigor...the result is a document which is based on the facts...which have been scrutinized to the highest level possible.But as we have discovered, the IPCC takes research findings at face value. It doesn't double-check that the raw data actually shows what a researcher claims it does. It feels no need to look under the hood - and discourages its expert reviewers from doing so.
... Holdren and Rood are therefore mistaken. The IPCC does not scrutinize the facts on which it relies. It performs no vetting whatsoever - never mind the sort that could be described as excruciatingly detailed.
When IPCC insiders were asked for their thoughts about quality assurance, their questionnaire answers confirmed this. Here are some of their verbatim remarks:As far as I can tell, there is no data quality assurance associated with what the IPCC is doing… (p. 99) Since the IPCC is a review body, it does not do data assurance or quality control in a systematic fashion.(p. 52) Quality assurance and error identification is not existent… (p. 384) Data quality assurance, per se, is beyond the scope of the work of the IPCC… (p. 203)
- - -7. What is your view of how IPCC handles data quality assurance and quality control and identification and rectification of errors, including those discovered after publication? Data quality assurance, per se, is beyond the scope of the work of the IPCC since its job is to assess the science.That said, it does have the task of assessing the scientific methods that are used to develop a given data product so as to determine whether we should have confidence in a given estimate, of say, the global mean temperature anomaly. Thus it must inter-compare results that are obtained using different data sources and procedures, and must also assess the reliability of different data sources. But it cannot undertake data quality assurance itself.- - -Many of these individuals said the IPCC should not be held responsible for the accuracy of statements that appear in research papers it cites since "that is an issue for the journals concerned." In the words of someone else, "it is expected that a paper published in an important journal" has already received a quality assurance check.
Other IPCC insiders, however, recognize the shortcomings of this approach. There are thousands of journals out there, but no accreditation process to ensure their quality.
How smart is it, therefore, to blindly assume that a published paper is an accurate paper? As one person observed, some research merely makes an interesting contribution to the 'intellectual conversation' (p. 332). That standard is surely far too low to justify an IPCC conclusion.
Once we understand a few other relevant facts, Holdren's claimed edifice crumbles entirely. Academic journals make use of a quality-control mechanism called peer review. The general idea is that, when a paper is submitted to a journal in the hope that it will be published, it gets assigned to an employee of the journal called an editor. The editor sends copies of the paper to reviewers presumed to be knowledgeable about the topic under discussion.Generally speaking, there are three reviewers whose identities remain anonymous even after the paper is published. Sometimes these reviewers are called referees. Although the reviewers look over the paper it is important to appreciate that, in many cases, only the most cursory of assessments takes place. In the words of one senior scientist:A reviewer is normally not paid for his work. With the best will in the world, he is able to spend no more than a few hours examining any particular manuscript. He is able to do little more than see that the story being told is superficially coherent and makes no obvious errors of fact.
If the reviewers have concerns, they tell the editor about them - who then asks the paper's author for a response. Sometimes a paper will undergo a major re-write before the editor is satisfied it is fit for publication. But this does not mean its conclusions are correct. Far from it. Richard Horton, the editor of The Lancet medical journal, argues this point forcefully:Peer review does not prove that a piece of research is true. The best it can do is say that, on the basis of a written account of what was done and some interrogation of the authors, the research seems on the face of it to be acceptable for publication…Experience shows, for example, that peer review is an extremely unreliable way to detect research misconduct.A recent commentary titled The Peer Review Fetish [http://blogs.cfr.org/levi/2011/07/08/the-peer-review-fetish/] makes a similar point:A couple reviewers, of course, are a poor substitute for mass scrutiny. Sometimes reviewers are chosen poorly; other times they're lazy. ...
Conflating peer review with scientific soundness impoverishes our appreciation of the scientific process. Peer review should be one criterion that people use in assessing the strength of any given piece of research – nothing more, nothing less.What this all adds up to is that the only time research findings can be considered valid is if someone else, working entirely independently, follows the same procedures as those described in the paper and arrives at the same result.
There used to be perfect clarity in the scientific community that unless a piece of research had passed that kind of test, it should be viewed with caution. Based on McIntyre's experience with the two unpublished papers discussed above, it appears the IPCC now regards research as reliable long before it has even appeared in print.
When one remembers that a great deal of climate research involves computer modeling (employing millions of lines of computer code), there's another reason for concern. As geophysics professor Jon Claerbout points out:An article about computational science in a scientific publication isn't the scholarship itself, it's merely advertising...The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.Peer-reviewers don't get within a mile of climate modeling supercomputers and their software. Which means they have no realistic way of evaluating entire categories of research papers that are central to the IPCC's analysis. All a peer-reviewer can do is assess the advertising - the portions of the story the climate modeler chooses to discuss in his or her paper.
Phil Jones, one of the world's most prominent climatologists, has published in the most prestigious journals. When he testified before a UK parliamentary committee early in 2010, he was asked how often peer reviewers had sought to examine his raw data and computer codes. "They've never asked," he replied.While we're on the subject of quality assurance, IPCC insiders who answered the questionnaire identified another weak link. A great deal of climate research involves huge collections of data - such as temperature records from thousands of locations stretching back scores of years. But the accuracy of these numbers has never been verified by independent personnel. As one IPCC insider observed, academic journals may consider unverified data good enough - but quality control mechanisms surely need to be in place before the IPCC relies on such data to make real-world decisions.
Let us return to that quote from presidential advisor John Holdren. He says the IPCC's conclusions are the result of the most thorough formal review of a scientific topic ever conducted. How can this be the case when the IPCC hasn't bothered to verify the temperature data on which so much of climate science rests?Would an auditor approve a company's financial statements before confirming the accuracy of the underlying numbers?
- (i) Provide inputs to improve inventories;
- (ii) Build confidence in emissions estimates and trends;
- (iii) Help to improve scientific understanding related to emissions inventories.