Tuesday 25 September 2012

Why I will no longer review or publish for the journal Neuropsychologia

 
A quick post.

I recently had an interesting experience with the journal Neuropsychologia, which led to a personal decision that some of my colleagues will probably think is a bit rash (To which my answer is: hey, it's me, what do you expect?!)

We submitted a manuscript that related pre-existing biases in spatial perception to the effects of transcranial magnetic stimulation (TMS) on spatial perception performance. The results are interesting (we think), even though there are some 'weaknesses' in the data: one of the significant effects is reliable in itself but doesn't dissociate significantly from another condition that is non-significant. For this reason we were careful about the interpretation, and the study was reasonably well powered compared to other studies in the field.

The paper was eventually rejected after going through two rounds of review. Once the initial downer of being rejected had passed, I realised that the reasons for the rejection were simple: it wasn't that our methodology was flawed or incomplete, it was that the data didn't meet the journal's standard of perfection.

This obsession with data perfection is one of the main reasons why we face a crisis of replicability and dodgy practices in psychology and cognitive neuroscience.

So after some consideration, I wrote to the action editor and the editor-in-chief and officially severed my relationship with the journal. The email is below. I'm a bit sad to do this because I've published with Neuropsychologia before, and reviewed for them many times -- and they have published some good work.

However my gripe isn't with either of the editors personally, or even with the reviewers of our specific paper (on the contrary, I am extremely grateful for the time and effort everyone invested). My problem is with the culture of perfection itself. 

For that reason I'm leaving Neuropsychologia behind and I urge you to do the same.
_________

Dear Jennifer and Mick,

I wanted to write to you concerning our rejected Neuropsychologia manuscript: NSY-D-12-00279R "The predictive nature of pseudoneglect for visual neglect: evidence from parietal theta burst stimulation".

Let me say at the outset that I am not seeking to challenge the decision. The reviewers make some excellent points and I'm very grateful for their considered assessment of the paper. I'm also grateful that you sought an additional review for us when the decision seemed to be a clear 'reject' based on the second review alone. That said, I would like to make a couple of comments.

First, the expectations of reviewers 2 and 3 about what a TMS study can achieve are fundamentally unrealistic. Indeed, it is precisely such unrealistic expectations for 'perfect' data that have created the file drawer problem and replicability crisis in psychology and cognitive neuroscience. It is also this pressure that encourages bad practices such as significance chasing, flexible data analyses, and cherry picking. All of the reviewers commented that our study was well designed, and it is manifestly well powered with 24 participants. If we had simply added another 10 subjects and shown 'cleaner' results, I wonder how many of the reviewers would have spotted the fatal flaw in doing so without correcting for data peeking. I suspect none.

Second, a number of the comments by the reviewers are misplaced. For instance, in commenting on the fact that we found a reliable effect of right AG TMS but not left AG TMS on line bisection performance, Reviewer 3 notes that "One cannot state that two effects are statistically different if one is significant and the other is not. A direct comparison is necessary." This is true but is also a straw man: we never state (or require) that the effects are statistically different between left AG and right AG. Our predictions were relative to the Sham condition and we focus our interpretation on those reliable significant effects. Similarly, Reviewer 2 challenges the categorisation of our participants into left and right deviants, noting the variable performance in the initial baseline condition. But this variation is expected, and we show with extra analyses that it cannot explain our results. Reviewer 2 simply disagrees, and this disagreement is sufficient grounds for rejection.

Overall, however, my main concern isn't with our specific paper (I am confident we will publish it elsewhere, for instance in PLoS One where 'perfect' data is not expected). My real problem is that by rejecting papers based on imperfect results, Neuropsychologia reinforces bad scientific practice and promotes false discoveries. It worries me how many other papers for Neuropsychologia get rejected for similar reasons. As Uri Simonsohn and colleagues note in their recent Psych Science paper on 'false positive psychology', "Reviewers should be more tolerant of imperfections in results. One reason researchers exploit researcher degrees of freedom is the unreasonable expectation we often impose as reviewers for every data pattern to be (significantly) as predicted. Underpowered studies with perfect results are the ones that should invite extra scrutiny."  (Simonsohn et al., Psychol Sci. 2011 Nov;22(11):1359-66.)

Based on my previous experiences as both an author and reviewer for Neuropsychologia, I have long suspected that a culture of 'data perfection' dominates at the journal. In fact, I have to admit that - for me - the current submission served as a useful experiment to test whether this culture would prevail for a study that is robust in design but with 'imperfect' (albeit statistically significant) results.

For this reason, my main purpose in writing is to inform you that I will no longer be submitting manuscripts to Neuropsychologia or reviewing them. I will be encouraging my colleagues similarly. Please note that this is in no way a criticism of you personally, but rather a personal decision to oppose what I see as a culture that needs active reform. I felt I owed you the courtesy of letting you know.

best wishes, Chris


12 comments:

  1. Agree 100% with the sentiment. But I fear you may soon run out of journals. I've just agreed to review for Brain. Below is from their instructions for reviewers. Haven't read the paper yet but I intend to disagree with the editorial policy when I submit my review.

    INFORMATION FOR REVIEWERS: Brain is receiving a steadily increasing number of manuscripts, and is thus under considerable pressure for acceptance. We are currently able to publish only 21% of submitted manuscripts. When making a recommendation, please highlight originality, the balance between a speculative versus definitive study, importance in the field and the extent to which the manuscript may appeal only to the specialist and not our general readership. Thus, a manuscript that is scientifically sound but represents only a small increment in knowledge, or merely consolidates an existing story would rarely merit publication.

    ReplyDelete
    Replies
    1. Thanks Jon. It's true that many journals act the same, and the reason they do is because we (the authors, reviewers and editors) allow them to. That's why I'm voting with my feet. Ultimately Neuropsychologia needs me more than I need it.

      The editorial policy for Brain is worrying but also common. Such premier journals prize novelty and importance above reliability or replicability. One thing's for sure: in psychology and neuroscience we get the science we deserve.

      Delete
  2. Interesting post,

    However, can you really blame them? A journal, in its traditional form as print, gets delivered to scientists every month. Given time constraints that scientists have, they may resort to reading only 'high-impact' journals to keep up to date and only want to read studies with new, significant findings.

    I do agree that its unfair that you went through many rounds of revisions before they rejected you. Perhaps they should have a better screening process to avoid wasting people's time.

    ReplyDelete
    Replies
    1. I think you're correct, but for partially the wrong reason.

      I don't think many scientists read print editions of journals anymore, and I doubt publishers make much money from individual subscriptions to journals these days. Most of the profit in journals come from institutional subscriptions by university libraries and the like. Given the constraints that always apply to higher education funding, these institutions are constantly having to reassess that value for money of these subscriptions and will regularly cut journals when there is assessed to be limited demand for them within the institution. Hence, commercial publishers will want to fill their journals with 'sexy' papers that will be of interest to as general readership as possible so that institutions can continue to justify paying for access.

      Unfortunately, the end result of this process is that commercial imperatives trump scientific imperatives, in that perfectly good (and sometimes quite important) studies get shunted into more obscure journals simply for not being 'sexy' enough.

      Delete
  3. Just to set the record straight, as the Action Editor of this manuscript, there weren't "many rounds of revisions". The reviewers asked for Major Revisions, the authors revised their manuscript, which was then sent back to the same reviewers to see whether they were satisfied with the revisions (standard practice for Neuropsychologia and many other journals). The reviewers had found merit in the design and execution of the study but still had serious concerns about results and interpretation. Given the points of merit, I did not want to reject the article outright but thought it best to seek the advice of a new expert reviewer. I appreciate Dr Chambers gratitude for this course of action. Unfortunately for Dr Chambers and his colleagues, this third reviewer also felt that the article was unpublishable in its current state, leading to a Reject decision. However, Neuropsychologia does allow the possibility for resubmission of manuscripts, with the Action Editor's approval.

    I would also like to point out that we have an active screening process in operation - personally, I triage around 50% of the submissions I receive (i.e. they don't go out for review). This policy is primarily to save author's and reviewer's time. To guard against personal editorial bias, papers are only triaged when another of the editors agrees that it should be. Of the manuscripts I receive that do go out for review, around 40% are rejected and around 60% are accepted. I would like to point out that these numbers reflect my own personal stats, which may vary from one section editor to another.

    I have nothing against Dr Chambers position - I just wanted to set the record straight from an editorial point of view.

    ReplyDelete
    Replies
    1. Thank you Jennifer, I appreciate you taking the time to leave a comment. Those statistics are interesting to know.

      Just to make clear for other readers, the comment about "many rounds of revision" was made by a previous commenter rather than me. I just pointed out that we went through two rounds of review, which is correct.

      My main concern with this case is that it represents an all too typical scenario in the traditional journal model: papers that are rejected purely on the basis of the results rather than the methodology. This creates publication bias and is precisely why psychological science is in such trouble.

      That said, in Jennifer's defence I want to reiterate that I have no beef with the way our paper was handled by her or the reviewers. Neuropsychologia is no worse than many other journals in enforcing publication bias. But is also no less guilty.

      I have no problem with how the rules were applied to our manuscript. My problem is with the game itself. I am becoming increasingly convinced that this form of publishing is doing great harm to science.

      Delete
    2. Just another quick point, if you're interested in the stats - of the 40% that were rejected, the vast majority (35%) were rejected after the first round of revisions and only a handful (5%) were rejected after having been revised. Unfortunately, your study fell into the latter category.

      Delete
  4. sorry, that should read "(35%) were rejected after the first round of REVIEWS"

    ReplyDelete
  5. Agh I wrote a comment and lost it. Something like I too am alarmed at the culture of perfection and impact/novelty that seems to have trickled to the specialist journals. It used to the Glamours that sought perfection and rejected without even review but now the journals that we are supposed to take our "imperfect" papers are also behaving this way. Something's got to change.

    The other thing is, reviewer culture. In many cases, even at non-Glamour journals the reviews are so demanding sometimes. As you say I think this reflects something not just about this journal, these editors and these reviewers but the general culture we're in. Every paper must tell a simple, clean story. Where do imperfect papers go?

    ReplyDelete
    Replies
    1. Hi Ayse, thanks for commenting.

      I think the problem is that we've placed such a premium on *quantity* of publications that it's created a flood of submissions, such that even standard specialist journals now face an impossible pressure to triage. Jennifer's comment is telling: even at Neuropsychologia, a whopping 50% of submissions are binned without review. 50%!

      In a perfect world increasing pressure for journal space would lead to the best science being published, but in fact all we are doing is adding pressure to an already biased system, enforcing the rule that publication requires 'clean' results that could be entirely false and unreplicable. But at least they are clean and look nice. Tick.

      This in turn encourages (even mandates) dodgy practices such as cherry picking and data mining. And it means that the imperfect papers go one of two places: either PLoS One or the file drawer.

      To fix this system we need to change how we behave as editors and reviewers. And we need to start valuing replication and quality of science above quantity. And all of us need to submit more to PLoS One. It needs us and we need its philosophy more than ever.

      Journals that reinforce publication bias are a cancer.

      I have to say that what worries me most is that nobody, even Jennifer Coull above, disagrees with any of this. Yet still we do absolutely nothing about it. We truly are slaves to our own irrational groupthink.

      Delete
    2. Chris,

      Yeah that's what I mean. 50% not reviewed at Neuropsychologia is amazing to me! I do understand it's happening because there are so many papers. That ties in to our assessment systems. This is all true.

      I do support Plos One. I am an editor there and contrary to popular belief we reject a lot (at least I do) but not for reasons like impact or clean story. I There is a parallel discussion going on about publication fees/open access and why academics put up with it. It seems everyone is stuck at "academics should do something, put their feet down" but few are willing to leave the game.

      In high school, I'd tell everyone if we all left there would be no class. People would be "yeaa! lets go" but within minutes there'd be fears and excuses and people would drop out. Many times I left alone and had great adventures (and finished school too).

      I wonder what else might be done to change our own behaviors. I honestly think if we did put a strong front, the publication culture and models can change. But the inertia I experienced in high school will be a problem.

      Delete
  6. Nobody might disagree with the problem of the current system setting incentives for dodgy practices. But I'm not sure I'd embrace the alternative you propose: an entirely PLOS One-ian world. Neither am I sure about whether there is any out-of-the-box alternative yet.

    Data-based review (as opposed to data-neglect, entirely methodology-based review) has a meaningful function. It filters stuff for potential interest. I like the fact that reading Nature I can be sure to come across some exciting (sometimes even important) findings. If high impact publishing was based on methodolicagl grounds *only*, I think it would miss the point of science. Of course being competent and rigid in applying our methods is part of the job description. But ultimatively (imho) society funds us in the hope to find out interesting (and/or relevant) stuff about the world. And experiments and scientists seem to do that to varying degrees. (Read: most results are rather dull if we're honest - tough luck of being a scientist). I think it's perfectly ok for the wider readership to prefer reading about the few interesting results. There should be a place for the not-so-interesting stuff as well, of course. And there is. But I disagree with the notion the system shouldn't differentiate between the two (read: within the vast range of the spectrum).

    What about the wrong incentives then? Not sure, that's not an easy one. If (some) scientists behave 'rational' in a sense that stirs their ethics in the direction of maximising career benefits - maybe we need to increase the risk for fraudulent behaviour and the incentives for attempts to replicate. Aren't we going in that direction right now? Both, fraud and false positive chasing seem to become quite popular at the moment...

    ReplyDelete