Redish Lab on Nostr: Unpacking: "What we need review for is to provide context for people who are not in ...

Unpacking: "What we need review for is to provide context for people who are not in the field."

Within a field, we can do our own "peer review". As a hippocampal expert, I can read a paper about hippocampus and do my own review. I can read between the lines and determine whether they have done the right controls. I know the literature and can tell if there are other alternative explanations that are more compatible with the rest of the literature than the speculation they've chosen. I can decide whether to build on their work or to ignore it. This is much of what we do in my lab's journal club. I assume that other experts do this to our papers in their journal clubs.

But, outside of my field, we cannot do our own "peer review". I can't read a paper on neanderthal burial rituals and determine if they've done a good job with the work. I need an expert to provide that context, to tell me if they've taken into account all the other data that exists in that literature, if they've done the controls that are expected from that field of science, if they've made any of the mistakes that often happen when using that technology.

This is particularly important when we start using science outside of the confines of its field. If we are going to decide whether to trust a vaccine or to eat horse-dewormer-paste or to change addiction policy, we need to have evaluation (review) by an appropriate scientific community of experts (peers). That process can be gate-keeping by journals or through post-evaluation discussion, but this review cannot just be some random volunteer, it needs to be an expert in the field who knows what to look for. That means it is serious work that can only be done by a very limited set of people and that takes serious time. Thus it requires some reason to take the action (usually an editor cajoling a scientist into "doing it for the community").

Peer review has also been used to help the authors make the paper better. While it is absolutely true that every paper I have ever published is better for the peer review that was done for it, it is very rare that the peer review has truly changed the meaning of the paper or the implications of the results. And there are other pathways that also provide this kind of feedback (discussions at conferences, questions at talks, post-publication commentary, papers written in response, etc.) Honestly, I think peer review does this better than any of other systems, and so I am loathe to lose it, but the cost of peer review is high and the other systems do work acceptably, especially if you are willing to take the long view on science.