Document Readability Level and Objective Complexity of Work Tasks: Any Impact on Relevance Judgment and Ranking?

Document Type : Research َ Article

Authors

1 PhD Candidate, Knowledge and Information Science Department, School of Education & Psychology, Shiraz University, Shiraz, Iran

2 PhD in Knowledge & Information Science, Assistant Professor, School of Education & Psychology, Shiraz University, Shiraz, Iran

3 PhD in Department of Clinical Psychology, Professor, School of Education & Psychology, Shiraz University, Shiraz, Iran

Abstract

Purpose: Aims to identify the role of document readability level and the objective complexity of the work task on students' relevance judgment and document ranking.
Methods: Quantitative content analysis was utilized to determine the readability level of texts. The statistical population of the research included 2825 graduate students of Shiraz University of Humanities. To determine the sample size, due to time-consuming process of relevance judgment, students’ time constraint, and sample size of similar studies, a non-probability sampling method of voluntary type was used. Two work tasks were designed based on Borland's (2000) Repository of Assigned Search Tasks and experts' opinions. After confirming the validity of the work tasks by the experts, searching through the Persian article databases, including SID, Magiran, and Noormags, four relevant, four partially relevant, and two irrelevant articles were selected by five experts The articles were divided into difficult and very difficult levels by determining their readability levels using Flash-Diani readability formula. In the next step, ten articles related to two work tasks were scored by users in a six-level spectrum (from entirely irrelevant to entirely relevant) then ranked based on the relevance degree (from 1 to 10). Data was analyzed using social science statistical software version 23 (SPSS), and T-tests and Mann-Whitney U tests were conducted.
Findings: The results showed significant difference between students' relevance judgment with different readability levels. Students rated documents with difficult readability levels more relevant than documents with very difficult readability levels. In addition, considering the role of readability level on the ranking, there was a significant difference between the rankings of relevance in the documents with different readability levels according to the students. Documents with difficult readability levels were ranked lower in terms of relevance. Therefore, students performed finer on rating documents with difficult readability levels.  Findings about the role of complexity level of work showed that objective complexity of the work task affected the students' relevance judgment. The retrieved documents related to the simple work task were found to be more relevant than the complex work task. Regarding the role of complexity level of work tasks on ranking, findings  revealed that the level of objective complexity of the work task affected the ranking of documents, and the simple work task ranked lower in terms of relevance ranking. Therefore, students performed finer on the simple work task than on the complex work task regarding relevance rating.
Conclusion: The role of level of readability of documents and objective complexity of the task affected relevance, judgment, and ranking. Further research on the effect of these indicators on relevance judgment  can improve the design of information retrieval systems and increase the quality of the relevance and ranking algorithms.

Keywords

Main Subjects

References

ارسطوپور، شعله (1391الف). بررسی امکان رتبه‌بندی مجدد نتایج جستجو بر مبنای درجه دشواری متن به صورت خودکار: مورد کاوی متون تخصصی حوزه علوم رایانه و فناوری اطلاعات. فصلنامه کتابداری و اطلاع‌رسانی، 15(3)، 58-79.
ارسطوپور، شعله (1391ب). امکان‌سنجی تعیین مفهوم پایه با هدف بهبود بازیابی اطلاعات در حوزه‌های تخصصی در زبان فارسی: موردکاوی علوم رایانه. شیراز: تخت جمشید.
افشار اردکانی، زهره، یزدان‌پناهی، فریبا، زیرراهی، مریم، و افشار، زهرا (1399). تحلیل محتوای فارسی ١ پایه دهم براساس سطح شاخص‌های گانیک (مورد مطالعه: درس شانزدهم: خسرو). تحقیقات جدید در علوم انسانی، 3(28)، 13-22.
انتظاریان، ناهید و فتاحی، رحمت‌الله (1389). مبانی طراحی رابط کاربر مبتنی بر شناخت ویژگی‌ها، ادراک و رفتار کاربران. فصلنامه کتابداری و اطلاع‌رسانی، 13(2)، 9-29.
‎بحرینی، نعیمه خاتون، میرزابیگی، مهدیه و ستوده، هاجر (1394). بررسی تأثیر سطح پیچیدگی وظایف کاری بر عملکرد جستجوی مروری کاربران در تعامل با وب. کتابداری و اطلاع‌رسانی، 18(4)، 99-122.
دیانی، محمدحسین (1366). سه فرمول برای تشخیص سطح خوانایی نوشته‌های ویژه نوسوادان. مجله روان‌شناسی و علوم تربیتی، 39، 59-80.
سروی، اعظم، طالب‌نیا، قدرت‌الله، پورزمانی، زهرا، و جهان‌شاد، آزیتا (1397). ارزیابی خوانایی و قابلیت فهم استانداردهای حسابداری از منظر حسابداران و حسابرسان با استفاده از شاخص‌های فلش و کلوز. مجله پژوهش‌های کاربردی در گزارش‌گری مالی، 7(13)، 241-274.
سعیدی‌زاده، مرضیه، طاهری، ابوالفضل و صنعت‌جو، اعظم (1395). معیارهای تعامل در فرایند جستجوی اطلاعات مبتنی بر وظایف کاری (نقش پیچیدگی عینی و نوع محصول). پژوهش‌نامه پردازش و مدیریت اطلاعات، 31(4)، 931-952.
عبداللهی گیلانی، محمد (1401). شاخص خوانایی و پیچیدگی متنی در کتاب‌‌های زبان انگلیسی دبیرستان. پژوهش‌های زبانشناختی در زبان‌های خارجی، 12(2)، 37-59.
غلامی احمدی، معصومه، نوکاریزی، محسن و صنعت‌جو، اعظم (1395). تعیین سطح خوانایی مقاله‌های دانشنامه رشد با هدف شناسایی گروه مخاطب با استفاده از شاخص‌های گانینگ/دیانی، فلش/دیانی و فرای/دیانی. پژوهش‌نامه کتابداری و اطلاع‌رسانی، 6(1)، 341-358.
فرهودی، فائزه و حریری، نجلا (1392). تأثیر ویژگی‌های شخصیتی کاربران بر قضاوت ربط. پژوهش‌نامه پردازش و مدیریت اطلاعات، 29(2)، 3۳۱-3۱۷.
کامران طرقی، فرشته، صنعت‌جو، اعظم و تجعفری، معصومه (1400). هفت وجه تجربه سواد اطلاعاتی براس و پیچیدگی وظیفة کاری: بررسی مروری. تعامل انسان و اطلاعات، 8(1)، 20-41.
کندری، فائزه (1399). سنجش و مقایسه سطح خوانایی کتاب فارسی خوانداری پایه سوم ابتدایی براساس تکنیک‌های خوانایی. پویش در آموزش علوم تربیتی و مشاوره، 6(12)، 22-41.
مفتون، پرویز و دقیق، مهتاب (1380). معیار تعیین میزان خوانایی ترجمه‌های فارسی متون انگلیسی. پژوهش‌نامه علوم انسانی، 29، 61-80.
واعظی، ملک‌رضوان، کوکبی، مرتضی و ابراهیمی، مختار (1395). پیشنهاد اصلاحاتی در فرمول‌های فلش/ دیانی و گانینگ/ دیانی. مجله مطالعات کتابداری و علم اطلاعات، 8(1)، 1-18.
References
Abdollahi-Guilani, M. (2022). Readability Index and Reading Complexity in high school FL ‎Textbooks. Journal of Foreign Language Research. 12(2), 37-59. Doi: 10.22059/jflr.2022.336999.928. [In Persian]
Afshar Ardekani, Z., Yazdanpanahi, F., Zirrahi, M., & Afshar, Z. (2020). Content Analysis of Farsi (1) of the 10th grade Based on Guning Indexes (Case study: 16th lesson: Khosrow). New research in the humanities, 3(28), 13-22.
Allen, B. L., & Kim, K. S. (2001). Person and context in information seeking: interactions between cognitive and task variables. The New Review of Information Behaviour Research: Studies of Information Seeking in Context, 2, 1–16.
Arastoopoor, Sh. (2018). Domain-specific readability measures to improve information retrieval in the Persian language. The Electronic Library, 36(3), 430-444. https://doi.org/10.1108/EL-01-2017-0007. [In Persian]
Arastoopoor, Sh. (2012a). Investigating the possibility of re-ranking search results based on the degree of text difficulty automatically: the case of specialized text mining in the field of computer science and information technology. Library and Information Sciences, 15(3), 58-79. [In Persian] 
Arastoopoor, Sh. (2012b). The feasibility of determining the basic concept with the aim of improving information retrieval in specialized fields in Persian language: a case study of computer science. Shiraz: Takht Jamshid. [In Persian]
Afshar Ardekany, Z., Yazdanpanahi, F., Zirrahi, M., & Afshar, Z. (2020). Content analysis of Persian 1 tenth grade using the Gunning Fog Index (case study: Lesson 16: Khosro). New Research in the Humanities, 3(28). 13-22.
Arnost, W.; Lull, E.; Schueder, J.; & Engler, J. (2021). Reading Level Identification Using Natural Language Processing Techniques. SMU Data Science Review: 5(3), 1-18
Bahreini, N., Mirzabeigi, M., & Sotoudeh, H. (2016). Investigating the effect of the level of complexity of work tasks on browsing search performance of users in interaction with the web. Library and Information Sciences, 18(4), 99-122. [In Persian] 
Bilal, D. (2014). Comparing google's readability of search results to the flesch readability formulae: A preliminary analysis on children's search queries. In Proceedings of the 76th ASIS&T Annual Meeting: Beyond the Cloud: Rethinking Information Boundaries (p. 30). American Society for Information Science. 50(1), 1-9.
Bora, A. (2006). Alternative Architectures for Improving Document Readability. Arlington, Graduate School of the University of Texas at Arlington.
Borgman, C.L. (1986). The Users Mental Model of an Information Retrieval System: an Experiment on a Prototype Online Catalog. International Journal of Man-Machine Studies, 24(l), 47-64.
Borlund, P. (2000). Experimental components for the evaluation of interactive information retrieval systems. Journal of Documentation, 56(1), 71-90.
Byström, K., & Järvelin. K. (1995). Task complexity affects information seeking and use. Information Processing and Management, 31: 191-213.
Campbell, D. J. (1988). Task complexity: A review and analysis. Academy of Management Review. 13(1): 40–52.
Chi, E., Jabbour, N., & Aaronson, N. L. (2017). Quality and readability of websites for patient information on tonsillectomy and sleep apnea. International journal of pediatric otorhinolaryngology. 98, 1-3. From http://www.sciencedirect.com/science/article/pii/S0165587617301726
Choi, B., Ward, A.R., Li, Y., Arguello, J., & Capra, R.G. (2019). The Effects of Task Complexity on the Use of Different Types of Information in a Search Assistance Tool. ACM Transactions on Information Systems (TOIS), 38, 1 - 28.
Crossley, S. A., Skalicky, S., & Dascalu, M. (2019). Moving beyond classic readability formulas: new methods and new models. Journal of Research in Reading, 42(3-4), 541-561.
Cuadra, C. A., & R. V. Katter. (1967). The relevance of relevance assessment. Proceedings of the American Documentation Institute. American Documentation Institute, Washington, DC4: 95-99.
Cutrell, E., & Guan, Z. (2007, May). What are you looking for? An eye-tracking study of information usage in Web search. Proceedings of the Conference on Human Factors in Computing Systems, USA, California, DOI: 10.1145/1240624.1240690
Dayani, M. H. (1987). Three formulas to determine the readability level of writings for beginners.
Psychology and Educational Sciences, 39. [In Persian]
Dekker, S. W. A., Nyce, J. M., Winsen, R. V., & Henriqson, E. (2010). Epistemological self confidence in human factors research. Journal of Cognitive Engineering and Decision Making, 4(1), 27-38.
Dillon, A. (2003). User Interface Design. MacMillan Encyclopedia of Cognitive Science,4, 453-458
Eleyan, D., Othman, A., & Eleyan, A. (2020). Enhancing Software Comments Readability Using Flesch Reading Ease Score. Information, 11(430), 1-25.
Entezarian, N., & Fattahi, R. A. (2010). Basics of user interface design based on knowing the characteristics, perception and behavior of users. Library and Information Sciences, 13(2), 9-29. [In Persian]
Falkenjack, J., & Jonsson, A. (2016). Implicit readability ranking using the latent variable of a Bayesian Probit model. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC) (pp. 104-112. (
Farhoodi, F., & Hariri, N. (2014). The Effect of User’s Personality Characteristics on Relevance Judgment. Iranian Journal of information processing management, 29(2), 317-331. [In Persian]
Gholami Ahmadi, M., Nowkarizi, M., sanatjoo, A. (2016). Readability Assessment of Roshd Encyclopedia Articles with the Aim of Identifying the Group of Readers Using Guning/Dayani, Flesch/Dayani and Fry/Dayani Indexes.  Library and Information Science Research, 6(1), 341-358. Doi: 10.22067/riis.v6i1.43011. [In Persian]
Hu, X., & Kando, N. (2017). Task complexity and difficulty in music information retrieval. Journal of the Association for Information Science and Technology, 68(7), 1711-1723.
Huang, K., Chen, J., Liu, C., & Zhang, L. (2020). A Comparative Study of the Relationship between the Subjective Difficulty, Objective Difficulty of Search Tasks and Search Behaviors. Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020.
Ingwersen, P. (1992). Information retrieval interaction (Vol. 246). London: Taylor Graham.‏
Jameel, S., & Qian, X. (2012, October). An unsupervised technical readability ranking model by building a conceptual terrain in LSI. In 2012 Eighth International Conference on Semantics, Knowledge and Grids (pp. 39-46). IEEE.‏
Jung, J. (2018). Effects of task complexity and working memory capacity on L2 reading comprehension. System, 74(11), 21-37, DOI:10.1016/j.system.2018.02.005
Kamran Toroghy, F., Sanatjoo, A., & Tajafari, M. (2021). Seven aspects of the information literacy experience and the Complexity of the task: A review. Human Information Interaction, 8(1), 20-41. [In Persian]
Kim, K. S. (1999). Searching the Web: effects of problem-solving style on information- seeking behavior. In Proceedings of ED-MEDIA 99: world conference on educational multimedia, hypermedia and telecommunications. 1541–1542.
Kim, K. S. (2008). Effects of emotion control and task on Web searching behavior. Information Processing and Management, 44(1), 373–385.
Kim, K. S., & Allen, B. (2002). Cognitive and task influences on Web searching behavior. Journal of American Society for Information Science and Technology, 53(2), 109–119.
Kim, J. Y., Collins-Thompson, K., Bennett, P. N., & Dumais, S. T. (2012). Characterizing web content, user interests, and search behavior by reading level and topic. In Proceedings of the fifth ACM international conference on Web search and data mining (pp. 213-222). ACM.
Kinley, K., Tjondronegoro, D., Partridge, H., & Edwards, S. (2012). Relationship between the nature of the search task types and query reformulation behaviour. In Sitbon, L, Trotman, A, & Cunningham, S (Eds.) Proceedings of the Seventeenth Australasian Document Computing Symposium. Association for Computing Machinery, United States, pp. 39-46.
Kondori, F. (2020). Assessing and comparing the readability level of Farsi reading books in the third grade of elementary school based on reading techniques. Pouyesh in Education and Consultation, 6(12), 22-41. [In Persian]
Lei, P. L., Sun, C. T., Lin, S. S., & Huang, T. K. (2015). Effect of metacognitive strategies and verbal-imagery cognitive style on biology-based video search and learning performance. Computers & Education, 87(326-339). Available at: www.elsevier.com/locate/compedu
Li, Y. (2010). An Exploration of the Relationships between Work Tasks and Users' Interaction Performance. Proceedings of the American Society for Information Science and Technology, 47(1), 1-9.
Li, Y., & Belkin, N. J. (2010). An exploration of the relationships between work task and interactive information search behavior. Journal of the American Society for information Science and Technology, 61(9), 1771-1789.‏
Liu, J. (2021). Deconstructing search tasks in interactive information retrieval: A systematic review of task dimensions and predictors. Information Processing and Management, 58(3), 102522.
Liu, M., Liu, Y., Mao, J., Luo, C., Zhang, M., & Ma, S. (2018, April). Satisfaction with Failure" or "Unsatisfied Success": Investigating the Relationship between Search Success and User Satisfaction.  Proceedings of The Web Conference. Lyon, France. 1533-1542. https://doi.org/10.1145/3178876.3186065
Lorigo, L., Haridasa, M., Brynjarsdóttir, H., Xia, L., Joachims, T., Gey, G., Granka, L., Pellacini, F., & Pan, B. (2008). Eye tracking and online search: Lessons learned and challenges ahead. Journal of the American Society for Information Science and Technology, 59(7). 1041-1052.
Maftoun, P., & Daghigh, M. (2001). The criterion for determining the readability of Persian translations of English texts. Research Journal of Human Sciences, 29, 61-80. [In Persian]
Martinc, M., Pollak, S., & Robnik-Šikonja, M. (2021). Supervised and Unsupervised Neural Approaches to Text Readability. Computational Linguistics, 47(1), 141–179.
Muresan, G., Liu, L., Cole, M., Smith, C. L., & Belkin, N. J. (2006). The effect of document readability on perceived familiarity and relevance. Proceedings of the American Society for Information Science and Technology, 42(1).  https://doi.org/10.1002/meet.14504201282
Navarro-Prieto, R., Scaife, M., & Rogers, Y. (1999). Cognitive strategies in Web searching. Retrieved June 6, 2020, from <http://zing.ncsl.nist.gov/hfweb/proceedings/navarro-prieto/index.html>.
O'Brien, H. L., Arguello, J. & Capra, R. (2020). An empirical study of interest, task complexity, and search behaviour on user engagement. Information Processing & Management, 57(3), 102226.
Ojha, P. K., Ismail, A., & Kuppusamy, K. S. (2021). Perusal of Readability with Focus on Web Content Understandability. Journal of King Saud University-Computer and Information Sciences, 33(1), 1-10.
Palotti, J., Goeuriot, L., Zuccon, G., & Hanbury, A. (2016). Ranking health web pages with relevance and understandability. In Proceedings of the 39th international ACM SIGIR conference on Research and development in information retrieval. (pp. 965-968). ACM.
Parsaei, E., Mirzabeigi, M., & Sotudeh, H., (2022). Identifying the factors affecting query formulation: a Systematic review. Librarianship and Information Organization Studies, 33(2): 37-53.
Doi: 10.30484/NASTINFO.2022.3135.2131
Phakiti, A. (2006). Modeling cognitive and metacognitive strategies and their relationships to EFL reading test performance. Melbourne papers in language testing, 1(1), 53-96.
Ruthven, J. (2003). Incorporating user search behavior into relevance feedback. Journal of the American Society for Information Science and Technology, 54(6): 529-549
Saeedizadeh, M., Taheri, A., & Sanatjoo, A. (2016). Information Interaction Criteria among Students in Process of Task-Based Information Searching (Role of Objective Complexity and Type of Product). Information Sciences & Technology, 31(4), 931-952. [In Persian]
Sánchez, L. & Kalamakis, S. Z. (2023). The interplay of task complexity (± Here-and-Now) and proficiency in shaping written narratives in Spanish as a foreign language. System, 112, 102949. https://doi.org/10.1016/j.system.2022.102949
Sarvi, A., Talebnia, G., Pourzamani, Z., & Jahanshad, A. (2019). Assessment Readability and Understandability of Accounting Standards by Accountants and Auditors Using Flesch and Cloze Indexes. Appleid Research in Financial Reporting, 7(13), 241-274. [In Persian]
Sasaki, Y., Komatsuda, T., Keyaki, A., & Miyazaki, J. (2016). A new readability measure for web documents and its evaluation on an effective web search engine. In Proceedings of the 18th International Conference on Information Integration and Web-based Applications and Services. (pp. 355-362). ACM.
Soodmand Afshar, H., & Tofighi, S. (2021). The Role of Task Complexity and Working Memory Capacity in L2. Journal of English language Teaching and Learning, 13(27). 389-416. DOI: 10.22034/elt.2021.44767.2353.
Tran, B. N. N., Singh, M., Singhal, D., Rudd, R., & Lee, B. T. (2017). Readability, complexity, and suitability of online resources for mastectomy and lumpectomy. Journal of surgical research, 212, 214-221.
Vaezi, M. R., Kokabi, M., & Ebrahimi, M. (2017). Suggesting amendments in Flech/Dayyani formula and Gunning/Dayyani formula. Journal of Studies in Library and Information Science, 8(17), 1-18. [In Persian]
Vakkari, P. (1999). Task complexity, problem structure and information actions: Integrating studies on information seeking and retrieval. Information Processing & Management, 35(6), 819-837.
https://doi.org/10.1016/S0306-4573(99)00028-X
Yan, X., Song, D., & Li, X. (2006). Concept-based document readability in domain specific information retrieval. In Proceedings of the 15th ACM international conference on Information and knowledge management. (pp. 540-549). ACM.
Yaneva, V., Temnikova, I., & Mitkov, R. (2016). Evaluating the readability of text simplification output for readers with cognitive disabilities. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) (pp. 293-299).‏
Zhang, Y. (2012). The Impact of Task Complexity on People's Mental Models of MedlinePlus. Information Processing & Management, 48(1), 107.
 
 
 
Send comment about this article
CAPTCHA Image

Files

History

  • Receive Date: 25 December 2022
  • Revise Date: 26 January 2023
  • Accept Date: 30 January 2023

Share

How to cite

Statistics