Study on the Ability of ChatGPT to Address User Queries, based on Rothstein's Reference Service Theory

Purpose: The breathtaking evolution of information and communication technology, driven largely by the emergence of artificial intelligence, has ushered in a transformative era, reshaping how we produce, disseminate, access, and utilize information. These profound changes have significantly influenced libraries, empowering them to meet users' increasingly intricate and diverse information needs with remarkable speed and efficiency. However, one of the daunting challenges libraries face lies in effectively evaluating the capabilities of artificial intelligence tools, such as chatbots, in responding to users' reference inquiries. As libraries seek to remain relevant in this digital age, it is crucial to explore the strengths and weaknesses of these tools to better serve patrons.
Method: This study aimed to assess the responsiveness and information services provided by ChatGPT in addressing users' reference questions based on Rothstein's theory of reference service levels. Adopting a decisive and applied approach, the research engaged a survey method paired with a meticulous descriptive-analytical lens. Our focus was on librarians from the reference departments of academic libraries and specialists in the vast realm of information science. To facilitate this exploration, we crafted a comprehensive questionnaire comprising 20 meticulously designed questions, inspired by Katz's categorization of reference queries. The first phase involved administering this carefully constructed questionnaire through ChatGPT, followed by rigorous evaluations from experts based on Rothstein's criteria, which delineates responses into maximum, middle, and minimum levels of comprehensiveness. This multifaceted method ensured a thorough analysis of ChatGPT's capabilities in providing accurate and useful information.
Findings: The findings illuminated ChatGPT's impressive ability to provide well-structured answers to specific types of inquiries, particularly those involving specialized searches, earning enthusiastic positive feedback from participants. However, the chatbot's performance faced notable challenges when confronted with complex research questions, revealing significant areas for improvement and refinement. Respondents pointed to various limitations, including ChatGPT's struggle to grasp emotional subtleties, its tendency to deliver incomplete or inaccurate answers, and the inadequate coverage of databases in certain languages and geographical regions. These results compellingly suggest that, while ChatGPT possesses the potential to elevate reference services and offer quick solutions, substantial enhancements are urgently needed to build trust and assurance among users. By identifying these limitations, we can carve a path toward the evolution and integration of such technologies in library settings, ensuring they complement traditional reference services effectively.
Conclusion: In summary, this research conducted a thorough examination of ChatGPT's aptitude for addressing library reference questions, drawing upon respected theories within the realms of librarianship and information science. The insightful feedback from librarians and experts confirmed that ChatGPT excelled in many scenarios, aligning with established research findings that underscore the growing role of AI in information retrieval. It became increasingly clear that as questions grew in complexity and specialization, ChatGPT's performance displayed marked improvement, showcasing its adaptability in a rapidly changing information landscape. Additionally, the language and cultural context of the inquiries played a pivotal role in shaping the quality of responses, highlighting the critical need for ongoing advancements and training in AI-driven reference services. As we advance into the future, fostering collaboration between human librarians and AI technologies may lead to a more enriched and effective reference service framework in libraries, ultimately enhancing the user experience and meeting the dynamic needs of our communities.