Science Fictions by Stuart Ritchie is a thorough review of the so-called 'replication crisis' in science and research. Building out from the well-documented issues in psychology, Ritchie's discipline, the book is an excellent survey of the problems of fraud, bias, negligence and hype in research. While much of the material and argument are familiar to anyone who has been following these debates, drawing the material together into a well-written narrative is a valuable contribution.
Overall, I enjoyed reading the book and agree with much of Ritchie's diagnosis of the problem and proposed solutions. In particular, his analysis of the issues with the publication system is spot on. More openness in research publication, the use of preprints, and reducing the dominance of journal brand are all part of the solution. In some aspects, I was less convinced by the argument of the book and had two particular reflections.
Despite taking popular science books to task for hype and oversimplification, in some areas, this book has precisely those problems. For example, in discussing the practice of paying researchers cash bonuses for publication in specific journals, Ritchie states that it occurs in "certain universities in other countries, including […] the UK". The claim turns out to refer to one department in one UK university, so limited evidence for a widespread issue. Similarly, Ritchie tells us that "many scientists quit the profession out of frustration", but the evidence to support this statement isn't at all compelling. There are other places where a rather black-and-white argument is presented, hiding considerable nuance. The story isn't quite as simple as Ritchie would have us believe.
The solutions to the problems with science that Ritchie presents in the book's closing chapter also merit further scrutiny and consideration. First, as has been pointed out in a review of the book, there is a tendency to refer back to a 'golden age' of scientific research whose existence is not necessarily supported by the evidence. While there is no doubt that improvements to the extrinsic incentives on researchers are needed, there will still be intrinsic motivations that might encourage bad practice. We also need to pay attention to the balance between false positives and false negatives in the research record. Science Fictions is rightly concerned that there are too many false positives, but false negatives are problematic too, with the potential to prematurely close off promising lines of inquiry. An effective research system balances false-positive and false-negative risk, acknowledging that it is impossible to eliminate either.
