Ranking and Relevance in Noormags and RICEST Databases

Document Type : Research َ Article

Authors

1 university of Isfahan

2 Associate professor, Knowledge and Information Science, University of Isfahan, Isfahan, Iran

Abstract

Purpose: The main purpose of information retrieval systems is to retrieve relevant information for users. This means that the results of the search must answer the questions provided to the system. Therefore, the evaluation of relevance is very important in such systems. In addition to relevance, the order and placement of articles are also important to the user. The retrieval systems should put more relevant articles at the top of the retrieval list. Evaluating the quality of ranking performance is a key activity in the field of information retrieval. This article assesses relevance and ranking of two databases.
Methodology: The sample includes 390 Persian articles retrieved in each of the Noormags and RICeST databases. For each topic inquired were carried out in both databases in two phases within the span of one month. The first 10 articles retrieved from each database were recorded based on the system ranking. Relevance score was given by 3 subject specialists within the range of zero to ten. Spearman correlation test was used to compare the ranking of the system with the ranking of the user. Data analysis was performed using descriptive and inferential statistics using SPSS software. The distance precision formula carried out to check the accuracy of the retrieval precision of related documents in the two databases, and the expected Reciprocal Rank was used to evaluate the quality of the ranking of articles.
Results: Users were far less familiar with RICeST database. Significant, consistent, and moderate correlation was found between system rankings and user rankings at the Noormags database in the first phase, i. e., ranking by users increases or decreases as the system rank increases or decreases. We found significant, consistent, and strong correlation between system’s ranking and user ranking in Noormags in the second phase. However, there was no correlation between system ranking and user ranking in RICeST database in both the first and second phases. Therefore, Noormags database ranking was found closer to the users’ ranking. Ranking quality by Noormags was relatively better than that of RICeST. Also, accuracy of the relevance precision of Noormags articles was higher than RICeST. From the users' point of view, Noormags database retrieved more relevant documents.
Conclusion: Noormags' new algorithms and capabilities have increased the relevance and ranking of its output. The findings could help database administrators to upgrade their databases by taking advantage of technologies to make semantic retrieval possible.

Keywords

Main Subjects

References

اخوتی، مریم (1383). مفهوم ربط در نظام‌های بازیابی اطلاعات، مروری بر نظریه‌ها و ادبیات موجود. اطلاع‌شناسی، 2 (1) : 23-45.
امینی مقدم، مهدی (1392). قابلیت­های جدید پایگاه مجلات تخصصی نور در یک نگاه. فصلنامه ره­آورد نور،44: 57-58.
بیزاییتس، ریکاردو؛ ریبرونتو، برتیه (1385). قلمروهای نو در بازیابی اطلاعات (ج1). ترجمه سیروس آزادی، علی جوامع و علی حسین قاسمی. تهران: چاپار، دبیزش.
پائو، میراندا لی (1380). مفاهیم بازیابی اطلاعات. ترجمه اسدالله آزاد و رحمت‌الله فتاحی. مشهد: دانشگاه فردوسی.
حری، عباس (1383). زنجیره داوری ربط در فرایند انتقال اطلاعات. اطلاع‌شناسی،2 (1) : 177-193.
داورپناه، محمدرضا؛ رمضانی، عباس‌علی (1385). بررسی معیارهای قضاوت ربط در فضای الکترونیکی. مطالعات تربیتی و روانشناسی، 25: 5-30. https://doi.org/10.22067/fe.v7i1.1829
ریاحی‌نیا، نصرت؛ رحیمی، فروغ؛ لطیفی، معصومه، و الله بخشیان، لیلی. (1394). بررسی میزان انطباق ربط سیستمی و ربط کاربرمدارانه در پایگاه‌های اطلاعاتی Google Scholar- ISC- SID. تعامل انسان و اطلاعات، 1 (4) : 1-11.
غلامی، تکتم (1386). سنجش میزان ربط در بازیابی اطلاعات در پایگاه‌های اطلاعاتی Ebsco، Scopus،Science Direct  از دیدگاه دانشجویان کارشناسی ارشد علوم تربیتی و روانشناسی دانشگاه الزهرا (س). پایان‌نامه کارشناسی ارشد، گروه کتابداری و اطلاع‌رسانی. دانشکده علوم تربیتی و روانشناسی. دانشگاه الزهرا (س).
کیانی، محمدرضا (1391). رویکردهای ارزیابی نظام‌های بازیابی اطلاعات: پس‎زمینه و چشم‎انداز پیش‎رو. کتابداری و اطلاع‌رسانی، 15 (2) : 243-258.
لنکستر، ویلفرید (1379). نظام‌های بازیابی اطلاعات (ویژگی‌ها، آزمون، و ارزیابی). ترجمه جعفر مهرادلنکستر، . شیراز: نوید شیراز.
میدو، چارلز تی؛ بویس، برت آر؛ کرافت، دونالداچ؛ و باری، کارول (1390). نظام‌های بازیابی اطلاعات متنی. ترجمه نجلا حریری. تهران: چاپار.
نادی راوندی، سمیه؛ حریری، نجلا (1395). نظام­های بازیابی اطلاعات. تهران: کتابدار.
 
Chapelle, O., Metlzer, D., Zhang, Y., & Grinspan, P. (2009). Expected reciprocal rank for graded relevance. Proceeding of the 18th ACM Conference on Information and Knowledge Management, 621-630. https://doi.org/10.1145/1645953.1646033
Chu, H., & Rosenthal, M. (1996). Search engines for the World Wide Web: A comparative study and evaluation methodology. The Annual Meeting-American Society for Information Science, 33, 127-135.
Clarke, S. J., & Willett, P. (1997). Estimating the recall performance of Web search engines. In Aslib proceedings, 49 (7), 184-189. https://doi.org/10.1108/eb051463
Cooper, W. S. (1968). Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. Journal of American Society of Information Science, 19 (1,, 30-41. https://doi.org/10.1002/asi.5090190108
Ding, W, & Marchionini, G. (1996). A comparative study of web search service performance. In: ASIS 1996 Annual Conference Proceedings, Baltimore, MD, Oct 19-24, 136-142. https://www.learntechlib.org/p/83946/
Johnson, F. C., Griffiths, J. R., & Hartley, R. J. (2001). DEVISE: a framework for the evaluation of Internet search engines. CERLIM (Centre for Research in Library and Information Management) , Manchester Metropolitan University.
Nowak, S., Lukashevich, H., Dunker, P., & Rüger, S. (2010). Performance measures for multilabel evaluation: a case study in the area of image classification. In Proceedings of the international conference on Multimedia information retrieval, Philadelphia, Pennsylvania, USA, 35-44. https://doi.org/10.1145/1743384.1743398
Powell, R. R., & Connaway, L. S., (2010). Basic research methods for librarians. London: Libraries Unlimited.
Rees, A. M. (1966). The relevance of relevance to the testing and evaluation of document retrieval systems. In Aslib Proceedings, 18 (11): 316-324. https://doi.org/10.1108/eb050068
Reitz, M. J. (2006). Dictionary of library and information. London: Libraries unlimited.
Saracevic, T. (2007) Relevance: a review of the literature and a framework for thinking on the notion in information Science. Part II. Journal of the American Society for Information Science, 58 (13) 1915-1933.https://doi.org/10.1002/asi.20681
Su, L. T., Chen, H. L., & Dong, X. (1998). Evaluation of Web-Based Search Engines from the End-User's Perspective: A Pilot Study. In Proceedings of the ASIS Annual Meeting, 35, 348-61.
Sawade, C., Bickel, S., Von Oertzen, T., Scheffer, T., & Landwehr, N. (2013). Active evaluation of ranking functions based on graded relevance. Machine learning, 92 (1), 41-64.
DOI 10.1007/s10994-013-5372-5
Tang, M. C., & Sun, Y. (2003). Evaluation of web-based search engines using user-effort measures. Library and Information Science Research Electronic Journal, 13 (2).
Tomaiuolo, N. G. and Packer, J. G. (1996). An analysis of Internet search engines: assessment of over 200 search queries. Computers in Libraries, 16 (6), 58-62.
Urhan, T. K., Rempel, H. G. Meunier‐Goddik, L, &. Penner, M. H. (2019). Information Retrieval in Food Science Research II: Accounting for Relevance When Evaluating Database Performance. Journal of food science, 84 (10), 2729-2735.
https://doi.org/10.1111/1750-3841.14769
Vaughan, L. (2004). New measurements for search engine evaluation proposed and tested. Information Processing & Management, 40 (4), 677-691.
 https://doi.org/10.1016/S0306-4573(03)00043-8
Send comment about this article
CAPTCHA Image

Files

History

  • Receive Date: 17 March 2020
  • Revise Date: 24 May 2020
  • Accept Date: 02 June 2020

Share

How to cite

Statistics