2025 ASEE Annual Conference & Exposition

Given the rapid proliferation of artificial intelligence (AI) tools in academic environments, critical questions about AI and its role in economizing the research process continue to emerge. While AI tools have the potential to foster limitless productivity and frictionless collaboration, there remain overarching concerns about AI's impact on information ethics, privacy, and equitable access. However, determining the effectiveness and alignment of AI tools within academic research contexts is a challenge for stakeholders across the academic ecosystem, including librarians, faculty, administrators, graduate students, and undergraduates. To address this absence of uniform guidance in the face of unrelenting technological disruption, this proposal describes the tripartite framework developed by the Brown University Library’s Critical AI Learning Community to enable systematic evaluation of AI tools across the landscape of institutional stakeholders in higher education.
As such, this proposal describes the group’s motivations, intentions, and process for creating a three-tiered evaluative methodology, which synthesizes and grounds traditional system usability frameworks, usage-driven assessments, and critical approaches developed within the fields of Science and Technology Studies (STS) into one model. In addition, this project seeks to provide insights from the experience of building our framework within a non-hierarchical and cross-disciplinary space like a learning community. Finally, the work seeks to present our typology and discuss several case studies that demonstrate how this framework can be applied to evaluate AI tools in authentic, real-world academic settings.