Weighting peer reviewers

Our scientific community faces a sort of paradox. A large bulk of work has been done on data-oriented techniques devised to improve peer reputation and knowledge extraction from data, so as to improve trustworthiness of digital services involving coordination and cooperation among heterogeneous peers. But, perhaps surprisingly, to the best of our knowledge, such techniques have rarely been applied to the (for our own community, crucial) process of reducing noise in the process of peer reviewing our own papers. Goal of this work is to provide initial insights on the applicability of methodologies and tools from inferential statistical to the field of peer review quality control. Our contribution is threefold. First, we propose a statistical model where each technical program committee member (reviewer) is characterized as random noise added to the “actual” value of the paper. Second, we provide an iterative data-oriented approach based on Expectation-Maximization devised to estimate mean value and variance of the noise added by each reviewer; our approach uses only the ratings provided by the reviewers themselves and does not rely on any additional source of a-priori knowledge. Third, we make use of the estimated mean values and variances to improve the accuracy of paper's evaluation and ranking.