Structural Similarities between Research Impact Indicators: Factor Analysis of Citation and Altmetric Indicators

Document Type : Research َ Article

Authors

1 Associate Professor, Knowledge and Information Science, Shiraz University

2 MA, Knowledge and Information Science, Shiraz University

3 PhD Candidate, Knowledge and Information Science, Shiraz University

4 Assistant Professor, Computer Engineering, Shiraz University

Abstract

Purpose: By investigating the structural similarities between altmetric and citation indicators through factor analysis, the present study attempts to identify similar indicators and experimentally group them based on their impact dimensions.
Methodolgoy: Applying a citation analysis method, it concentrates on a purposive sample consisted of papers published in PLoS journals during 2010-2012.
Findings: The results of the factor analysis led to identification of a model consisted of 3 factors explaining 53 percent of the variance of the latent variable, i.e. “the impact of research outputs”. The experimental model seems to adhere to the theoretical model proposed by Junping & Houqiang (2015). According to the yielded model, the factors differ in terms of the kind, extent and depth of the impacts they explain. The first factor, called “impact at perception level”, implies the widest, though the most superficial, impact. The “perception level” can be considered as the “consumption” level that does not necessarily lead to “usage”. At another point of the impact continuum, there exists “impact at social media level” that is relatively higher in terms of the interaction between users and texts. The level is relatively more limited in terms of the quantity of users, but deeper in terms of the impact the related indicators may have. At the other extreme, one may find “the impact at usage level”, that is achieved after reading the texts, deeply interacting with and selecting them purposely.
Conclusion: The finding indicates the power of altmetrics in measuring different types of impact with different extents and depths. It helps understanding the nature of the impacts the indicators may represent and thereby help recognizing the association between the indicators and overcoming their multiplicity and chaos.  

Keywords

Main Subjects

References

جمالی مهموئی، حمیدرضا (1390). ارزیابی پژوهش: رویکردها، شیوه‌ها و چالش­ها. رهیافت، 21 (49)، 39-52.
حری، عباس (1373). بررسی چگونگی مصرف و تولید اطلاعات علمی در میان متخصصان کشور. روان­شناسی و علوم تربیتی (دانشگاه تهران)، (1101) 1، 41-55.
ستوده، هاجر (1389). گذاری بر ضریب تأثیر مجله‌ها و دلایل ناکارامدی آن در ارزیابی پژوهش در رشته‌های مختلف. رهیافت، 20 (47)، 33-44.
قدیمی، آفتاب؛ ستوده، هاجر (1393). سنجش ارزش استنادی مقالات شیمی ایران با استفاده از شاخص سهم استناد متنی. پردازش و مدیریت اطلاعات، 30 (2)، 357-372.
کیم، جی-آن؛ مولر، چارلز (1378). کاربرد تحلیل عاملی در پژوهش­ اجتماعی همراه با دستورهای نرم‌افزار SPSS (مسعود کوثری، مترجم). تهران: سلمان (نشر اثر اصلی 1978).
موئد، هنک (1387). تحلیلاستنادیدرارزیابیپژوهش (عباس میرزایی و حیدر مختاری، مترجمان). تهران: چاپار (نشر اثر اصلی 2005).
میرز، لاورنس اس.؛ گامست، گلن؛ و گارینو، ا.جی. (1391). پژوهش چند متغیری کاربردی: طرح و تفسیر (حسن­پاشا شریفی و همکاران، مترجمان). تهران: رشد (نشر اثر اصلی 2006).
Bar-Ilan, J., Haustein, S., Peters, I., Priem, J., Shema, H., & Terliesner, J. (2012). Beyond citations: Scholars' visibility on the social Web. arXiv, 1205.5611. Retrieved February 20, 2019, from https://arxiv.org/ftp/arxiv/papers/1205/1205.5611.pdf
Bollen, J., Van de Sompel, H., Hagberg, A., & Chute, R. (2009). A principal component analysis of 39 scientific impact measures. PloS one, 4 (6), e6022. Retrieved February 20, 2019, from https://doi.org/10.1371/journal.pone.0006022
Bornmann, L. (2015). Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics. Scientometrics, 103 (3), 1123-1144.
Brody, T., Harnad, S., & Carr, L. (2006). Earlier web usage statistics as predictors of later citation impact. Journal of the Association for Information Science and Technology, 57 (8), 1060-1072.
Colledge, L. (2014). Snowball metrics recipe book. Amsterdam: Snowball Metrics Program Partners, 110.
Retrieved February 20, 2019, from https://www.snowballmetrics.com/wp-content/uploads/snowball-recipe-book_HR.pdf
Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66 (10), 2003-2019.
De Winter, J. C. (2015). The relationship between tweets, citations, and article views for PLoS ONE articles. Scientometrics, 102 (2), 1773-1779.
Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. Journal of medical Internet research, 13 (4), e123. Retrieved February 20, 2019, from https://www.ncbi.nlm.nih.gov/pubmed/22173204
Galligan, F., & Dyas-Correia, S. (2013). Altmetrics: Rethinking the way we measure. Serials Review, 39 (1), 56-61.
Haque, A. U., & Ginsparg, P. (2009). Positional effects on citation and readership in arXiv. Journal of the Association for Information Science and Technology, 60 (11), 2203-2218.
Haustein, S. (2012). Multidimensional journal evaluation: Analyzing scientific periodicals beyond the impact factor. Berlin: De Gruyter Saur.
Haustein, S., Peters, I., Sugimoto, C. R., Thelwall, M., & Larivière, V. (2014). Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Science and Technology, 65 (4), 656-669.
Head, A. J., & Eisenberg, M. B. (2010). How today's college students use Wikipedia for course-related research. First Monday, 15 (3). Retrieved February 20, 2019, from https://firstmonday.org/ojs/index.php/fm/article/view/2830/2476
Holmberg, K., (2015). Classifying altmetrics by level of impact. Retrieved February 20, 2019, from https://pdfs.semanticscholar.org/49df/91ccd2579b4b345c135a6ff751bf39f7ee86.pdf
Holmberg, K., Bowman, T. D., & Didegah, F. (2015, October). The meaning of impact in altmetrics. Paper presented at the 2015 Altmetrics Workshop, Amsterdam. Retrieved February 20, 2019, from https://altmetrics.org/altmetrics15/holmberg/
Junping, Q., & Houqiang, Y. (2015). Stratifying altmetrics indicators based on impact generation model. In Proceedings of ISSI 2015 Istanbul: 15th International Society of Scientometrics and Informetrics Conference, 29 June - 3 July, (pp. 115-116). Retrieved February 20, 2019, from https://pdfs.semanticscholar.org/8fd1/fc122678c8afe725cbeba67319239f12d93c.pdf
King, D. W., & Tenopir, C. (2004). An evidence-based assessment of the “author-pays” model. Retrieved February 20, 2019, from https://www.nature.com/nature/focus/accessdebate/26.html
Kurtz, M. J., & Bollen, J. (2010). Usage bibliometrics. Annual Review of Information Science and Technology, 44 (1), 1-64.
Lin, J., & Fenner, M. (2013). Altmetrics in evolution: Defining and redefining the ontology of article-level metrics. Information standards quarterly, 25 (2), 20-26.
Liu, C. L., Xu, Y. Q., Wu, H., Chen, S. S., & Guo, J. J. (2013). Correlation and interaction visualization of altmetric indicators extracted from scholarly social network activities: Dimensions and structure. Journal of medical Internet research, 15 (11), e259. Retrieved February 20, 2019, from https://doi.org/10.2196/jmir.2707
MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: a critical review. Journal of the American Society for information Science, 40 (5), 342-349.
Maflahi, N., & Thelwall, M. (2016). When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals. Journal of the Association for Information Science and Technology, 67 (1), 191-199.
Mohammadi, E., Thelwall, M., Haustein, S., & Larivière, V. (2015). Who reads research articles? an altmetrics analysis of Mendeley user categories. Journal of the Association for Information Science and Technology, 66 (9), 1832-1846.
Neylon, C., & Wu, S. (2009). Article-Level metrics and the evolution of scientific impact. PLoS biology, 7 (11), e1000242. Retrieved February 20, 2019, from https://doi.org/10.1371/journal.pbio.1000242
Peters, I., Jobmann, A., Eppelin, A., Hoffmann, C. P., Künne, S., & Wollnik-Korn, G. (2014). Altmetrics for large, multidisciplinary research groups: a case study of the Leibniz Association. Libraries in the Digital Age (LIDA) Proceedings, 13, 245-254. Retrieved February 20, 2019, from https://bib.irb.hr/datoteka/762790.LIDA2014_Proceedings.pdf
Priem, J., Piwowar, H. A., & Hemminger, B. M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. arXiv, 1203.4745. Retrieved February 20, 2019, from http://jasonpriem.org/self-archived/PLoS-altmetrics-sigmetrics11-abstract.pdf
Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Altmetrics: a manifesto. Retrieved February 20, 2019, from http://altmetrics.org/manifesto
Rowlands, I., & Nicholas, D. (2007). The missing link: Journal usage metrics. Aslib Proceedings, 59 (3), 222-228.
Schweitzer, N. J. (2008). Wikipedia and psychology: Coverage of concepts and its use by undergraduate students. Teaching of Psychology, 35 (2), 81-85.
Shema, H., Bar‐Ilan, J., & Thelwall, M. (2014). Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics. Journal of the Association for Information Science and Technology, 65 (5), 1018-1027.
Shuai, X., Jiang, Z., Liu, X., & Bollen, J., (2013). A comparative study of academic and Wikipedia ranking. In Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries, July 22-26, (pp. 25-28). New York: ACM.
Shuai, X., Pepe, A., & Bollen, J. (2012). How the scientific community reacts to newly submitted preprints: Article downloads, twitter mentions, and citations. PloS one, 7 (11), e47523. Retrieved February 20, 2019, from https://doi.org/10.1371/journal.pone.0047523
Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98 (2), 1131-1143.
Thelwall, M. (2017a). Why do papers have many Mendeley readers but few Scopus-indexed citations and vice versa? Journal of Librarianship and Information Science, 49 (2), 144-151.
Thelwall, M. (2017b). Three practical field normalised alternative indicator formulae for research evaluation. Journal of Informetrics, 11 (1), 128-151.
Thelwall, M., & Wilson, P. (2015). Mendeley readership altmetrics for medical articles: an analysis of 45 fields. Journal of the Association for Information Science and Technology, 67 (8), 1962-1972.
Wang, X., Liu, C., Fang, Z., & Mao, W. (2014). From attention to citation, what and how does altmetrics work? arXiv, 1409.4269. Retrieved March 13, 2019, from https://arxiv.org/ftp/arxiv/papers/1409/1409.4269.pdf
Wouters, P., & Costas, R. (2012). Users, narcissism and control: Tracking the impact of scholarly publications in the 21st century. Utrecht: SURFfoundation.
Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics, 101 (2), 1491-1513.
Send comment about this article
CAPTCHA Image
Librarianship and Information Organization Studies
Volume 29, Issue 4 - Serial Number 116
January 2019
Pages 171-192

Files

History

  • Receive Date: 07 July 2018
  • Revise Date: 06 October 2018
  • Accept Date: 24 October 2018

Share

How to cite

Statistics