Customizing Large Language Models for Automated Academic Advising at Universities

Kiszla, Matthew; Butziger, Kathryn; Filizzola Ortiz, Jose Fabrizio

Student Work

Customizing Large Language Models for Automated Academic Advising at Universities

Public Deposited

In this report, we explore the customization of Large Language Models (LLMs) for automated academic advising at universities, with a focus on streamlining access to the constantly evolving academic information at Worcester Polytechnic Institute (WPI). This includes course details, room bookings, and updates to tracking sheets. To overcome LLM knowledge gaps, we built customized LLM solutions through three distinct pipelines: one utilizing the BGE embedding model with Pinecone vector database for context mapping, one employing Pinecone’s Retrieval-Augmented Generation (RAG) model for efficient data retrieval, and one integrating OpenAI’s Assistants API for precise query responses. In an experiment with 150 WPI academic advising questions, each of these pipelines achieved promising results compared to almost all incorrect answers from ChatGPT without context. Through our research, we not only built a tailored Q&A system that tackles the knowledge gap at WPI but also showcased the potential for LLMs to be customized for similar challenges in other institutions.

This report represents the work of one or more WPI undergraduate students submitted to the faculty as evidence of completion of a degree requirement. WPI routinely publishes these reports on its website without editorial or peer review.

Creator

Publisher