Home › Forums › An open forum to discuss the future of peer review › Can we use base article-level research evaluation on open reviews and how?
October 13, 2015 at 10:57 pm #2477
I have been thinking a lot about this initial question and come up with a proposal that I posted here on The Self Journal of Science: http://www.sjscience.org/article?id=401
The essence of this proposal is that I believe we are many who would like to see scientific research evaluated on its own merits and not by the reputation of the journal in which it is published. In order to start changing this, I believe we may need to provide some brief “score” or “metric” that can be used to express the quality/impact/importance etc. of each individual published article. This can hopefully be used in place of relying on journal impact factors etc. while being based solidly on more thorough evaluation of the individual article. I propose to create a scoring system borrowing (but also modifying) some of the principles behind the StackExchange network.
This proposal is at the same time also an experiment in testing (and trying to promote) SJS by using its review and discussion features to review the idea. So far this has gone well. The proposal got a few good comments relatively quickly and in that respect has been a modest success in my opinion so far.
Some of the important points that have come up are:
- Konrad Hinsen has clarified that one of the open issues of my proposal – how to calculate an aggregate metric from individual evaluations – is quite important to how the system treats different kinds of articles.
- He has also pointed out that it is very important for any proposed such system to realise what kind of behaviour its scoring mechanisms encourage in its users. For example, it may affect what kind of articles authors try to write.
October 13, 2015 at 11:21 pm #2478
I also discussed this proposal briefly on Twitter with @michael_nielsen and @david_colquhoun. Michael Nielsen pointed out that I should relate my proposal better to the history of open peer review – previous efforts: his tweets. He has a good point and I think a good starting point may be this paper that he proposed: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1000242.
David Colquhoun is somewhat sceptical – as I expected – but that was the whole idea of reaching out to him anyway. If this idea is any good I had better expose it to as much constructive criticism as possible to test and improve it. He has a number of good comments. I think all of them can be found here. One of the important comments he made is that in order to get a reliable assessment of an article, we probably need a substantial number of reviews of it. How would we attract enough reviewers? That is indeed one of the hardest questions.
You must be logged in to reply to this topic.