We analyze the predictiveness and accuracy of evaluators’ assessments of research proposals. Using a two-stage estimation that considers both funding decision and subsequent outcomes -measured by publications, citations, field-citation ratios, and public attention (Altmetric)- we find that the predictive validity of evaluations is contingent on the evaluators’ expertise. Specifically, assessments by domain experts, defined as those with specialized knowledge in the proposal’s area, are predictive of scientific outcomes after funding, unlike those by evaluators with average or minimal domain expertise. Experts’ assessments also carry more weight in funding decisions, indicating that their abilities are factored into the decision-making process. However, experts are not particularly accurate in assessing the actual outcome levels. While their unfavorable opinions are generally correct, their favorable opinions are prone to significant errors. Experts are also better at predicting non-scientific impacts, although the latter can also be predicted, to a lesser extent, by non-experts. Our findings suggest that, although domain expertise is critical for funding excellent scientific research, domain experts also favor proposals that underperform in their area of expertise.

Expert Predictions and Errors in Research Funding Decisions

Chiara Franzoni;Massimiliano Guerini;Andola Stanaj
2025-01-01

Abstract

We analyze the predictiveness and accuracy of evaluators’ assessments of research proposals. Using a two-stage estimation that considers both funding decision and subsequent outcomes -measured by publications, citations, field-citation ratios, and public attention (Altmetric)- we find that the predictive validity of evaluations is contingent on the evaluators’ expertise. Specifically, assessments by domain experts, defined as those with specialized knowledge in the proposal’s area, are predictive of scientific outcomes after funding, unlike those by evaluators with average or minimal domain expertise. Experts’ assessments also carry more weight in funding decisions, indicating that their abilities are factored into the decision-making process. However, experts are not particularly accurate in assessing the actual outcome levels. While their unfavorable opinions are generally correct, their favorable opinions are prone to significant errors. Experts are also better at predicting non-scientific impacts, although the latter can also be predicted, to a lesser extent, by non-experts. Our findings suggest that, although domain expertise is critical for funding excellent scientific research, domain experts also favor proposals that underperform in their area of expertise.
2025
85th Annual Meeting of the Academy of Management, AOM 2025
File in questo prodotto:
File Dimensione Formato  
11186.pdf

Accesso riservato

Dimensione 613.04 kB
Formato Adobe PDF
613.04 kB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11311/1298569
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
  • OpenAlex ND
social impact