Metric studies in the artificial intelligence (AI) ecosystem

Abstract

The purpose of scientific journals is to serve as a communication channel that contributes to solving social problems through the lens of science and technology;therefore, articles based on metric analyses must present quantitative research on academic output, collaboration networks, journal impact, or thematic trends using valid indicators—such as citations, the h-index, and altmetrics, among others (González, 2025).

These studies are distinguished by the rigor of their methodological implementation regarding data selection—drawing from prestigious databases like Scopus, WoS, and Dimensions—and by originality, manifested in the efficient analysis of knowledge domains or niches, as well as issues underrepresented in the literature that serve as a foundation for new research aimed at future academic application or science policy across various regional contexts (López et al., 2021).

Like any human endeavor, the results of metric studies are susceptible to errors at various stages of information analysis that can compromise their credibility.

These include data collection errors—such as database bias, where authors focus on a single source (e.g., WoS or Scopus) while excluding regional journals, preprints, or less-represented disciplines like the social sciences or humanities; poorly defined search criteria involving keywords that are either too broad or too narrow;

inadequate time filters that fail to account for research remaining relevant more than five years after publication; and record duplication, where a report includes multiple versions of the same work without first consolidating formats such as preprints, final versions, translations, and conference papers, among others. We must not overlook the indiscriminate use of the Journal Impact Factor (JIF)—a metric designed for journals—to evaluate individual researchers; other errors include ignoring disciplinary context by comparing h-indices or citation counts across fields with distinct dynamics; the over-reliance on traditional metrics while disregarding alternatives like Altmetric; the misinterpretation of percentiles and unchecked self-citations; incorrect normalization—such as comparing raw citation counts without adjusting for document age or field size; confusing correlation with causation; making improper generalizations by extrapolating findings from a subfield to an entire discipline; and ethical breaches—such as manipulating inconvenient data, inflating metrics through coordinated self-citation, or publishing in so-called predatory journals—among other malpractices coming to light as AI tools are adopted by international journal editorial and scientific committees (González, 2025).

 Current procedures to reduce manuscript rejection rates involve comprehensive documentation, data triangulation using various databases, and proper contextualization (López et al., 2021). Similarly, the increasing adoption of AI tools requires the scientific community to implement procedures for analyzing the information generated by these powerful technologies. For the benefit of the community, a list of free tools for metric studies is recommended; however, one must consider the dynamics of the information market, where applications go through various stages of promotion and commercialization. Consequently, some tools require registration or have limitations in their free versions. Furthermore, regarding access to advanced data from major platforms like Web of Science or Scopus, a growing number of academic institutions provide subscription-based access to tools that are not entirely free—a fact that highlights the importance of designing and implementing international scientific collaboration projects:

• VOSviewer: Designed to construct and visualize bibliometric networks (co-citation, co-authorship, bibliographic coupling, etc.).It includes data mining capabilities for analyzing key terms within scientific and academic literature. • Bibliometrix: An R package that allows for the import of data from Scopus, Web of Science, PubMed, and other databases.It is open-source software.
• BibExcel: A tool for analyzing bibliographic data and generating Excel-compatible files. It is highly useful for studying citations and collaboration among authors and institutions.
• CitNetExplorer: Enables the visualization and analysis of scientific citations, direct data import from Web of Science, and the exploration of publication clusters.
• Google Scholar Citations: A free service that tracks citations of articles, theses, and books.

It is ideal for analyzing individual author profiles.

 Among the free altmetric tools, we can mention:

• ImpactStory: aggregates alternative metrics (social media, blog mentions, etc.) closely linked to researcher profiles via ORCID and DOI; these features are free for registered users.
• Altmetric.com (basic version): offers a multicolored "donut" visualization showing the impact of the articles under study across social media and mass media.

Note that the API and bookmarklet are free, although the full version is a paid service.

• PlumX Metrics: integrated into Scopus;

although owned by Elsevier, it allows access to certain metrics, such as social media mentions.

• Crossref Event Data: provides researchers with raw data on altmetric events.

It is worth noting that, according to experts (López et al., 2021; Calle et al., 2025; Amézquita et al., 2026), AI is revolutionizing metric studies and contributing to systematic improvements across all implementations. These include processing vast volumes of data (millions of scientific papers processed in seconds) to identify patterns impossible to detect via traditional methods;

applying alternative techniques to improve the assessment of scientific impact; detecting existing and potential new collaboration networks; predicting trends using neural networks; automating processes related to systematic reviews; and, above all, detecting potential fraud and bias. The latter presents a risk—and a significant ongoing challenge—stemming from the fact that some

AI models function as "black boxes." This makes it difficult for authors to understand and subsequently describe in their manuscripts how these models reach decision-making conclusions, thereby complicating the reproducibility of research results and potentially undermining their credibility within the scientific community (López et al., 2021; Calle et al., 2025; Amézquita et al., 2026).

In summary, the rise of AI highlights the need to emphasize practices from the recent past associated with scientific collaboration networks—such as comparing results against expert criteria and conducting independent audits (involving third parties to analyze the chosen model and identify biases)—in order to reduce uncertainty and enhance credibility.

Enviar comentarios