Soaring into the Future of Chat Reference: Assessing for Quality in Cooperative Chat Reference

Over the summer of 2021, Drexel University Libraries was preparing to join a reference cooperative to provide chat reference when local librarians were not available. Librarians expressed concerns that other librarians would not provide an acceptable level of assistance. To ensure that cooperative chat was meeting the needs of our patrons we decided to assess chat transcripts from the cooperative chat service over one academic term. Our main questions were:

• Are librarians in the cooperative chat service able to meet a patron’s need when local librarians are not available? 
• Are librarians in the cooperative chat cooperative chat service providing our patrons with a quality service when we are not available? 

We evaluated the librarian behavior in chat transcripts for:

• Questioning/Listening: did librarians ask questions to understand a need and “listen” to patrons?
• Communication/Contact: did librarians pick up chats promptly and stay in communication with patrons during the chat?
• Accuracy: did librarians provide complete and accurate information?
• Follow up: did librarians confirm that a patron had what they needed and/or tell them they could return?

See separate handouts for rubric and annotated bibliography of related literature.

A team of 7 librarians worked together to created a rubric that was created using RUSA's Guidelines for Behavioral Performance of Reference and Information Service Providers and local concerns as a guide. The rubric was tested using a batch of sample transcripts. Over one 10 week term, over 120 transcripts were collected from chat reference between Drexel University patrons and outside librarians who staffed the cooperative chat service. A library worker outside of the team collected the transcripts de-identified them and numbered them. Each week, librarians rated transcripts from the previous week using the rubric as a guide. The responses were collected in a secure form. From there, the ratings from librarians could be downloaded and evaluated. Each of the 120 transcripts that were analyzed were rated by between two and five separate librarians whose subject matter expertise represented the disciplines of Engineering, Public Health, Nursing, English, Humanities, and the Social Sciences.