A Scientific Referee's Guild

In this essay I'd like to talk a bit about my own experiences, not in scientific research but in science funding. I don't want to appear to complain; I have no complaints as I have been able to pursue many different interesting fields of research and the fiscal support, while dicey at times, was always there at some level. But integrity impels one to write about the process the scientific community calls "refereeing" in which other scientists judge one's work as worthy of publication, for example, or--the topic I'm talking about--worthy of funding (I have not had many problems getting papers published in the scientific literature.)

Since this is not a complaint essay, I shall not name the one or two funding agencies that I have submitted proposals for funding to, with not just disappointing results, but disgraceful refereeing. But I can give bone fide examples which I feel on these occasions would argue for a Referee's Guild; a body of experts in no way connected with the funding of the projects being refereed, and (ideally) high above reproach. And, unlike today's referees, would be paid by the publication or program using their services.

The need for such a guild from my experience has stemmed from mistakes in how referees are picked. For example, in one agency program the referees were almost exclusively picked from the pool of those that had also submitted proposals. The director of this program argued that this did not produce poor refereeing. I thought it did, and I never got funding from this program even after about two dozen proposals. (However, I must say that the local programs of this agency almost always funded me with what is known as "seed" money.) But as to the quality of the refereeing from this program, you judge for yourself. I got the following responses from proposing, in the early 1990s, to search for extrasolar planets using the transit method--again, not one of these proposals received funding for the following reasons given.

"Astrometric interferometry is the only way to detect planets." This one-sentence rejection was given before the first extrasolar planets were discovered using pulsar timing and then radial velocity measurements.

Another reason for rejection: "This project can only detect brown dwarfs, not Jupiters." A whole committee let this get through. We were proposing the transit method, which measures the drop in light of a star as a planet moves in front of it. Since brown dwarfs are smaller in size than Jupiter-sized planets, and transits measure the drop in light produced by the area of the transiting object, it was absurd to say that a smaller object could be measured and not a larger one.

Another: "This work is already being done by the TEP (Transit of Extrasolar Planets) Network." This was true, but I had to point out to the agency that I was the Director of the TEP Network and so was well aware of the work being done, and could we please obtain some funding to do it? (We were all volunteers at the time-I was making my living teaching at a small college, and observing in the summer.) The reply was basically that it therefore looked like the work was already being done without funding, so why should they fund it? A wry sense of humor is needed for this kind of response.

Anyway, you get the idea. There are many other examples, some less, some more ridiculous. OK, if you insist, I shall share my favorite. I got a referee's report once with the first reason for rejection being, "It's probably going to rain at the observatories." Very wry sense of humor! There were three observatories involved in the proposal, so my colleague immediately calculated (assuming that "probably" meant at least 51%) that it would have to rain about 80% of the time at each observatory, and rightly concluded that no one would build an observatory in such a place. But so much for logic.

I decided that I just had to track this one down. When I finally found the scientist who had written the original referee's report to the agency, he told me he had recommended the project for funding. He had written, "This is one of the best proposals I have read; the only way it can fail is if it rains at the observatory." So, in this case, it was clearly not the scientist's refereeing that had denied us funding but the grasping for negative reasons for rejection by the program itself. Human nature, endlessly amusing, no?

Hopefully, then, we have gotten to this part of the essay without your thinking that I am holding any grudges against any science programs--certainly not. While disappointing at the time, I also just had to shake my head and sometimes even laugh out loud. Ah, human nature...bless its heart. It is also confidence-assuring that these proposals, almost universally, are now funded programs--not to me but to various other people and institutions. So at least we know we were in the right scientific ballpark. My institute has called this effect, "being ahead of the funding curve." But I'd like to think that scientific referees could, and would, tell good science when they see it. After all, science is supposed to be subject to objective judgment as to its correctness or not.

At the moment, as mentioned, papers for journals, as well as proposals for funding, are refereed by scientific peers. It is understood that this is something one must do voluntarily, and many times (at least in my experience) one has to put in a significant amount of time to do a good job of it--sometimes even a significant fraction of the time it took to produce the original paper itself (as I have also heard colleagues mention). A good start would also be that the names of the proposal authors would be withheld, while the name of the referees would have to be specified, so they could stand by their reputations.

So, although I don't know how practical such a suggestion may be, I would like to nevertheless suggest here that a Scientific Referee's Guild be established, consisting of professional (i.e., paid) referees with the highest credentials. Perhaps they might only be called upon in disputed cases, but it might be good to know such a body exists out there to get an impartial scientific judgment in such cases. They could hear cases (like the brown dwarf versus Jupiter-like planet detection mentioned above) and intervene to point out that a scientifically valid reason would have to be given to justify rejection of funding, for example. The whole goal of such a guild would be to insure that intellectual integrity and mutual respect be maintained in the scientific community.

Do I see such a guild being established very soon? Perhaps not. But perhaps just bringing up the subject has helped a bit to focus thought on this issue, encourage those scientists whose funding experience might be going through a similar phase, and strike a note for scientific integrity over funding considerations when push comes to shove. I know that this last point may have caused some to gasp, but I remind those scientists out there that they certainly did not go into science primarily to make money. And to remember why they went into science in the first place is to remember the ideal that they must have had in the beginning--that life is primarily an adventure of discovery and that science itself must be the final arbiter of truth.