Cognitive Distances between Evaluators and Evaluees in Research Evaluation: A Comparison between Three Informetric Methods at the Journal and Subject Category Aggregation Level

摘要

This article compares six informetric approaches to determine cognitive distances between the publications of panel members and those of research groups in discipline-specific research evaluation. We used data collected in the framework of six completed research evaluations from the period 2009-2014 at the University of Antwerp as a test case. We distinguish between two levels of aggregation – Web of Science subject categories and journals – and three methods: while the barycenter method (2-dimensional) is based on global maps of science, the similarity-adapted publication vector (SAPV) method and weighted cosine similarity (WCS) method (both in higher dimensions) use a full similarity matrix. In total, this leads to six different approaches, all of which are based on the publication profile of research groups and panel members. We use Euclidean distances between barycenters and SAPVs, as well as values of WCS between panel members and research groups as indicators of cognitive distance. We systematically compare how these six approaches are related. The results show that the level of aggregation has minor influence on determining cognitive distances, but dimensionality (two versus a high number of dimensions) has a greater influence. The SAPV and WCS methods agree in most cases at both levels of aggregation on which panel member has the closest cognitive distance to the group to be evaluated, whereas the barycenter approaches often differ. Comparing the results of the methods to the main assessor that was assigned to each research group, we find that the barycenter method usually scores better. However, the barycenter method is less discriminatory and suggests more potential evaluators, whereas SAPV and WCS are more precise.