Another point to consider is that writing a wall of text as the review encourages reflection and reconsideration whereas the system you propose encourages impulsive comments and nitpicking. Given that reviewers donate their time, this increases the risk that reviewers just add 4-5 superficial comments here and there and call it a day rather than reflecting on the paper as a whole. So I see the danger that review quality would actually go down with this new system and it would be important to study if that’s the case or not.

@MikeMorrisonPhD

Ай бұрын

That's a great point on the nitpicking. I had actually figured on that effect happening for good: People will conform to the feedback that the tool encourages, so if the tools focus on concrete suggestions that could be good, but hadn't thought of the narrowing effect that could create.

🚀

All the journals I review for have basic HTML features if you want to write in the box. Every journal also has a file upload where you can use Word to do the review for even better features. Of course, nobody writes in the box, we all use word.

@MikeMorrisonPhD

8 ай бұрын

Sounds better than a plain text box, at least! Are the authors sent multiple word files with different feedback, or one with all the feedback? From the author's perspective, how many word files are circulating by the end of two rounds of review? Also, as a reviewer, what frustrates you about Word's comment system? If you could design your own feature for it, what would you build first?

Hm, I think you have exaggerated the time it takes to process a review a lot in favor of your point to redesign the process. Never haver I seen me or my colleagues taking two month to organize their thoughts on the reviews. Typically we have a fair idea after the first read and a detailed understanding once we prepared the draft of the response letter the next day with the review broken up into pieces followed by answers. Also, in many important fields, peer review does not take 2 years. More like 2 months (+ whatever time you need for changes). Another point you have exaggerated is pointing to locations in papers. Line numbers that are automatically added to the PDF and page numbers work just fine. Many journals have already started using a system similar to what you suggested for the final proofreading. It’s nice to have but the time it saves is marginal compared to the time spend on reading. Similar arguments apply to the reviews: The majority of the time is spend on reading and understanding the paper and thinking about the science or the thoughts of the reviewers. While I am a big fan of your thoughts on posters, this one felt overblown.

@MikeMorrisonPhD

Ай бұрын

First, thanks for all this feedback. Building better peer review tools is a considerably more complex challenge than posters, and part of that is needing to hear from a LOT of reviewers, so your comments here will really help me develop version 2 of these concepts. In this case I based those delay times on an actual public peer review. But, only one. I've also waited months or even a year on paper feedback before. But to your point, that was the intake phase not actually once reviewers were assigned. With journal shopping and rejections, papers in my field at least can easily take 2 years to get published. Did you mean that papers in your field typically go from submission-to-publication in a few months?! If so that's great! Also, even though I didn't exaggerate in this case, I'm totally not above exaggeration so it was also a fair criticism haha. Also, what's your thoughts on receiving a todo-formatted review as an author, versus thinking as a reviewer?

@R-ok3cl

Ай бұрын

@@MikeMorrisonPhD Thanks for considering my comment and for your feedback: Yeah, in my field, chemistry, 2 months (+whatever time you need to fix things) from submission to publication is on the faster end but quite common. I am just working on the minor revisions for one of these papers that we submitted May first and it will probably be published in late June. For another work, even after initial rejection in a high impact Journal, another rejection after peer review in another Journal, substantial additions and an appeal against this decision, the paper was out about 6-7 months after the first submission. Both your and my experience are anecdotal, but I am sure somebody has looked at this more rigorously. While it initially seems like a wall of text, most paragraphs can be easily broken down into simple todos. So this feature would be a nice-to-have convenience feature that would save me maybe 5 minutes per paper.

@MikeMorrisonPhD

Ай бұрын

@@R-ok3cl Yeah maybe a sensible next step for me is to go through one of those big datasets of peer reviews so I can rely less on anecdotal experience. And also, the PR system I’m working on is for computational articles written in Myst Markdown or Quarto, which are way easier to build reviewer tools for than word files, so it allows for improvements that couldn’t be made before. And for me, even saving 5 minutes per paper is a win. If I can save you a single click I’ll put weeks of effort into it. One last question: where do you think the biggest waste is in the peer review process, if you had to choose and even if it’s small?