Aleksandr Biryukov, Dzhozef Veykling, Klaudio Bartolini, Fabio Kazati, Mauritsio Markeze, Katsyaryna Mirylenka, Nardin Osman, Adzurra Ragone, Karlos Sierra, Aalam VassefPublished in:
Scholarly Communication Review
– Volume 5, Number 1 (January-February, 2015)



Keywords: bidding, LiquidPub, metrics, opinions, peer-review, research evaluation, UCount


Share:

Abstract:

In this paper we review several novel approaches for research evaluation. We start with a brief overview of the peer review, its controversies, and metrics for assessing efficiency and overall quality of the peer review. We then discuss five approaches, including reputation-based ones, that come out of the research carried out by the LiquidPub project and research groups collaborated with LiquidPub. Those approaches are alternative or complementary to traditional peer review. We discuss pros and cons of the proposed approaches and conclude with a vision for the future of the research evaluation, arguing that no single system can suit all stakeholders in various communities. The article was 1st time published here:Birukou A, Wakeling JR, Bartolini C, Casati F, Marchese M, Mirylenka K, Osman N, Ragone A, Sierra C and Wassef A (2011) Alternatives to peer review: novel approaches for research evaluation. Front. Comput. Neurosci. 5:56. doi: 10.3389/fncom.2011.00056

---

References:
Adomavicius, G., and Tuzhilin, A. (2005). Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17, 734–749.

Akst, J. (2010). I hate your paper. Scientist 24, 36.

Best, M. L. (2004). Can the internet be a human right? Hum. Rights Hum. Welf. 4, 23–31.

Black, N., van Rooyen, S., Godlee, F., Smith, R., and Evans, S. (1998). What makes a good reviewer and a good review for a general medical journal? J. Am. Med. Assoc. 280, 231–233.

Bornmann, L. (2007). Bias cut. women, it seems, often get a raw deal in science – so how can discrimination be tackled? Nature 445, 566.

Bornmann, L., and Daniel, H.-D. (2005a). Committee peer review at an international research foundation: predictive validity and fairness of selection decisions on post-graduate fellowship applications. Res. Eval. 14, 15–20.

Bornmann, L., and Daniel, H.-D. (2005b). Selection of research fellowship recipients by committee peer review. reliability, fairness and predictive validity of board of trustees’ decisions. Scientometrics 63, 297–320.

Bornmann, L., and Daniel, H.-D. (2010a). The validity of staff editors’ initial evaluations of manuscripts: a case study of Angewandte Chemie International Edition. Scientometrics 85, 681–687.

Bornmann, L., and Daniel, H.-D. (2010b). The usefulness of peer review for selecting manuscripts for publication: a utility analysis taking as an example a high-impact journal. PLoS ONE 5, e11344. doi:10.1371/journal.pone.0011344

Bornmann, L., Wallon, G., and Ledin, A. (2008). Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European molecular biology organization programmes.PLoS ONE 3, e3480. doi:10.1371/journal.pone.0003480

Brody, T., Harnad, S., and Carr, L. (2006). Earlier web usage statistics as predictors of later citation impact. JASIST58, 1060–1072.

Burnham, J. C. (1990). The evolution of editorial peer review. J. Am. Med. Assoc. 263, 1323–1329.

Casati, F., Marchese, M., Mirylenka, K., and Ragone, A. (2010). Reviewing Peer Review: A Quantitative Analysis of Peer Review. Technical Report 1813. University of Trento. Available at:http://eprints.biblio.unitn.it/archive/00001813/

Ceci, S. J., and Peters, D. P. (1982). Peer review: a study of reliability. Change 14, 44–48.

Ceci, S. J., and Williams, W. M. (2011). Understanding current causes of women’s underrepresentation in science.Proc. Natl. Acad. Sci. U.S.A. 108, 3157–3162.

Cho, M. K., Justice, A. C., Winker, M. A., Berlin, J. A., Waeckerle, J. F., Callaham, M. L., and Rennie, D. (1998). Masking author identity in peer review: what factors influence masking success? PEER Investigators. JAMA 280, 243–245.

de Solla Price, D. (1976). A general theory of bibliometric and other cumulative advantage processes. J. Am. Soc. Inf. Sci. 27, 292–306.

Evans, J. A. (2008). Electronic publication and the narrowing of science and scholarship. Science 321, 395–399.

Fisher, M., Friedman, S. B., and Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. J. Am. Med. Assoc. 272, 143–146.

Gichora, N. N., Fatumo, S. A., Ngara, M. V., Chelbat, N., Ramdayal, K., Opap, K. B., Siwo, G. H., Adebiyi, M. O., El Gonnouni, A., Zofou, D., Maurady, A. A. M., Adebiyi, E. F., de Villiers, E. P., Masiga, D. K., Bizzaro, J. W., Suravajhala, P., Ommeh, S. C., and Hide, W. (2010). Ten simple rules for organizing a virtual conference – anywhere. PLoS Comput. Biol. 6, e1000650. doi:10.1371/journal.pcbi.1000650

Ginsparg, P. (1994). First steps towards electronic research communication. Comput. Phys. 8, 390–396.

Godlee, F. (2002). Making reviewers visible: openness, accountability, and credit. JAMA 287, 2762–2765.

Godlee, F., Gale, C. R., and Martyn, C. N. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports a randomized controlled trial. JAMA 280, 237–240.

Goodman, S. N., Berlin, J., Fletcher, S. W., and Fletcher, R. H. (1994). Manuscript quality before and after peer review and editing at annals of internal medicine. Ann. Intern. Med. 121, 11–21.

Greaves, S., Scott, J., Clarke, M., Miller, L., Hannay, T., Thomas, A., and Campbell, P. (2006). Overview: Nature’s peer review trial. Nature. doi: 10.1038/nature05535.

Harnad, S. (1990). Scholarly skywriting and the prepublication continuum of scientific enquiry. Psychol. Sci. 1, 342–344.

Hu, N., Pavlou, P. A., and Zhang, J. (2009). Overcoming the J-shaped distribution of product reviews. Commun. ACM 52, 144–147.

Ingelfinger, F. J. (1974). Peer review in biomedical publication. Am. J. Med. 56, 686–692.

Jefferson, T., Rudin, M., Folse, S. B., and Davidoff, F. (2007). Editorial peer review for improving the quality of reports of biomedical studies. Cochrane 41, MR000016.

Jefferson, T., Wager, E., and Davidoff, F. (2002a). Measuring the quality of editorial peer review. JAMA 287, 2786–2790.

Jefferson, T., Alderson, P., Wager, E., and Davidoff, F. (2002b). Effects of editorial peer review: a systematic review.JAMA 287, 2784–2786.

Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., Rennie, D., and PEER Investigators. (1998). Does masking author identity improve peer review quality? A randomized controlled trial. JAMA 280, 240–242.

Kassirer, J. P., and Campion, E. W. (1994). Peer review: crude and understudied, but indispensable. J. Am. Med. Assoc. 272, 96–97.

Katz, D. S., Proto, A. V., and Olmsted, W. W. (2002). Incidence and nature of unblinding by authors: our experience at two radiology journals with double-blinded peer review policies. Am. J. Roentgenol. 179, 1415–1417.

Kronick, D. A. (1990). Peer review in 18th-century scientific journalism. JAMA 263, 1321–1322.

Lawrence, P. A. (2003). The politics of publication. Nature 422, 259–261.

Lee, K., Boyd, E., Holroyd-Leduc, J., Bacchetti, P., and Bero, L. (2006). Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med. J. Aust. 184, 621.

Link, A. M. (1998). Us and non-US submissions: an analysis of reviewer bias. JAMA 280, 246–247.

Lock, S. (1994). Does editorial peer review work? Ann. Intern. Med. 121, 60–61.

Lynch, J. R., Cunningham, M. R., Warme, W. J., Schaad, D. C., Wolf, F. M., and Leopold, S. S. (2007). Commercially funded and united states-based research is more likely to be published; good-quality studies with negative outcomes are not. J. Bone Joint Surg. Am. 89, 1010–1018.

Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., and O’Mara, A. (2009). Gender effects in the peer reviews of grant proposals: a comprehensive meta-analysis comparing traditional and multilevel approaches. Rev. Educ. Res.79, 1290–1326.

Masum, H., and Zhang, Y.-C. (2004). Manifesto for the reputation society. First Monday 9 [Online].

McCook, A. (2006). Is peer review broken? Scientist 20, 26.

McNutt, R. A., Evans, A. T., Fletcher, R. H., and Fletcher, S. W. (1990). The effects of blinding on the quality of peer review: a randomized trial. JAMA 263, 1371–1376.

Medo, M., Cimini, G., and Gualdi, S. (2011). Temporal effects in the growth of networks. Available at:http://arxiv.org/abs/1109.5560

Medo, M., and Wakeling, J. R. (2010). The effect of discrete vs. continuous-valued ratings on reputation and ranking systems. Europhys. Lett. 91, 48004.

Odlyzko, A. M. (1995). Tragic loss or good riddance? The impending demise of traditional scholarly journals. Int. J. Hum. Comput. Sci. 42, 71–122.

Olson, C. M., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J. W., Zhu, Q., Reiling, J., and Pace, B. (2002). Publication bias in editorial decision making. JAMA 287, 2825–2828.

Opthof, T., Coronel, R., and Janse, M. J. (2002). The significance of the peer review process against the background of bias: priority ratings of reviewers and editors and the prediction of citation, the role of geographical bias.Cardiovasc. Res. 56, 339–346.

Osman, N., Sabater-Mir, J., and Sierra, C. (2011). “Simulating research behaviour,” in 12th International Workshop on Multi-Agent-Based Simulation (MABS’11), Taipei.

Osman, N., Sabater-Mir, J., Sierra, C., de Pinninck Bas, A. P., Imran, M., Marchese, M., and Ragone, A. (2010a).Credit attribution for liquid publications. Deliverable D4.1, Liquid Publications Project. Available at:https://dev.liquidpub.org/svn/liquidpub/papers/deliverables/LP_D4.1.pdf

Osman, N., Sierra, C., and Sabater-Mir, J. (2010b). “Propagation of opinions in structural graphs,” in ECAI 2010: Proceedings of the 19th European Conference on Artificial Intelligence, Vol. 215 of Frontiers in Artificial Intelligence and Applications, eds H. Coelho, R. Studer, and M. Wooldridge (Lisbon: IOS Press), 595–600.

Parra, C., Birukou, A., Casati, F., Saint-Paul, R., Wakeling, J. R., and Chlamtac, I. (2011). “UCount: a community-driven approach for measuring scientific reputation,” in Proceedings of Altmetrics11: Tracking Scholarly Impact on the Social Web, Koblenz.

Purcell, G. P., Donovan, S. L., and Davidoff, F. (1998). Changes to manuscripts during the editorial process: characterizing the evolution of a clinical paper. J. Am. Med. Assoc. 280, 227–228.

Ragone, A., Mirylenka, K., Casati, F., and Marchese, M. (2011). “A quantitative analysis of peer review,” in 13th International Society of Scientometrics and Informetrics Conference, Durban.

Ragone, A., Mirylenka, K., Casati, F., and Marchese, M. (2013). On peer review in computer science: analysis of its effectiveness and suggestions for improvement.Scientometrics 97 (2), 317–356. doi: 10.1007/s11192-013-1002-z

Reinhart, M. (2009). Peer review of grant applications in biology and medicine. reliability, fairness, and validity.Scientometrics 81, 789–809.

Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., Hachinski, V. C., Gibbons, R. J., Gardner, T. J., and Krumholz, H. M. (2006). Effect of blinded peer review on abstract acceptance. J. Am. Med. Assoc. 295, 1675–1680.

Sense About Science. (2009). Peer Review Survey: Preliminary Results. Available at:http://www.senseaboutscience.org.uk/index.php/site/project/29/

Smith, R. (2006). Peer review: a flawed process at the heart of science and journals. J. R. Soc. Med. 99, 178–182.

Spier, R. (2002). The history of the peer-review process. Trends Biotechnol. 20, 357–358.

Swan, A. (2007). Open access and the progress of science. Am. Sci. 95, 198–200.

Underwood, A. J. (2004). It would be better to create and maintain quality rather than worrying about its measurement. Mar. Ecol. Prog. Ser. 270, 283–286.

van Rooyen, S., Black, N., and Godlee, F. (1999). Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J. Clin. Epidemiol. 52, 625–629.

Walsh, E., Rooney, M., Appleby, L., and Wilkinson, G. (2000). Open peer review: a randomised controlled trial. Br. J. Psychiatry 176, 47–51.

Ware, M., and Monkman, M. (2008). Peer Review in Scholarly Journals: Perspective of the Scholarly Community – An International Study. Survey Commissioned by the Publishing Research Consortium. Available at:http://www.publishingresearch.net/PeerReview.htm

Wenneras, C., and Wold, A. (1997). Nepotism and sexism in peer-review. Nature 387, 341–343.

Zhou, T., Kuscsik, Z., Liu, J.-G., Medo, M., Wakeling, J. R., and Zhang, Y.-C. (2010). Solving the apparent diversity-accuracy dilemma of recommender systems. Proc. Natl. Acad. Sci. U.S.A. 107, 4511–4515.