Publisher
University of Tennessee at Chattanooga
Place of Publication
Chattanooga (Tenn.)
Abstract
Underemployment remains a persistent challenge for psychology graduates, often making it difficult for students to identify and communicate workforce-relevant competencies gained through coursework. To address this issue, the Eugene platform was developed to help students connect completed courses to knowledge, skills, and abilities (KSAs) and generate potential career pathways. A central component of Eugene is a faculty-led reconciliation process in which instructors evaluate the extent to which course activities develop specific work-related competencies. However, this process is resource-intensive and difficult to scale. This in-progress study examined whether an artificial intelligence (AI) agent could generate work activity (WA) ratings from course materials that align with faculty evaluations. It was hypothesized that AI-generated ratings would demonstrate substantial agreement with faculty ratings, supporting the use of AI as a supplement to the existing reconciliation process. Course materials were collected from four undergraduate psychology courses and consisted of syllabi and one grade distribution. For one course, the syllabus, grade distribution, and both combined were rated separately, yielding 84 AI-faculty WA rating comparisons where faculty ratings were available. Using Microsoft Azure, the AI agent evaluated course materials and assigned ratings to a sample of 15 O*NET-derived work activities on a 1-7 scale, where higher scores reflected greater competency development. The AI generated both numerical ratings and written justifications, while faculty ratings served as the comparison standard. To evaluate the correspondence between AI-generated and faculty ratings, analyses examined exact match rates and the proportion of scores falling within a two-point threshold. Preliminary results indicated that 32% (n = 27) of AI-generated ratings exactly matched faculty ratings, and an additional 63% (n = 53) fell within one to two scale points. Only 5% (n = 4) differed by more than two scale points. Overall, 95% of AI-generated ratings met the predefined agreement criterion, suggesting substantial consistency between AI and faculty judgments. These findings provide preliminary evidence that AI can serve as a promising adjunct to faculty evaluation within the KSA reconciliation process. From a research perspective, the study contributes to emerging work on human-AI collaboration in competency assessment. From a practical perspective, integrating AI into Eugene may reduce faculty workload, improve process efficiency, and enhance the scalability of translating coursework into career-relevant competencies. Future research should examine AI-faculty agreement across additional courses and larger datasets to further evaluate the effectiveness of this approach.
Date
November 2026
Subject
Industrial and organizational psychology
Document Type
posters
Language
English
Rights
http://rightsstatements.org/vocab/InC/1.0/
License
http://creativecommons.org/licenses/by/4.0/
Poster
Included in
Reimagining Career Pathways: Bridging Education and Employment Through AI-Assisted KSA Mapping
Underemployment remains a persistent challenge for psychology graduates, often making it difficult for students to identify and communicate workforce-relevant competencies gained through coursework. To address this issue, the Eugene platform was developed to help students connect completed courses to knowledge, skills, and abilities (KSAs) and generate potential career pathways. A central component of Eugene is a faculty-led reconciliation process in which instructors evaluate the extent to which course activities develop specific work-related competencies. However, this process is resource-intensive and difficult to scale. This in-progress study examined whether an artificial intelligence (AI) agent could generate work activity (WA) ratings from course materials that align with faculty evaluations. It was hypothesized that AI-generated ratings would demonstrate substantial agreement with faculty ratings, supporting the use of AI as a supplement to the existing reconciliation process. Course materials were collected from four undergraduate psychology courses and consisted of syllabi and one grade distribution. For one course, the syllabus, grade distribution, and both combined were rated separately, yielding 84 AI-faculty WA rating comparisons where faculty ratings were available. Using Microsoft Azure, the AI agent evaluated course materials and assigned ratings to a sample of 15 O*NET-derived work activities on a 1-7 scale, where higher scores reflected greater competency development. The AI generated both numerical ratings and written justifications, while faculty ratings served as the comparison standard. To evaluate the correspondence between AI-generated and faculty ratings, analyses examined exact match rates and the proportion of scores falling within a two-point threshold. Preliminary results indicated that 32% (n = 27) of AI-generated ratings exactly matched faculty ratings, and an additional 63% (n = 53) fell within one to two scale points. Only 5% (n = 4) differed by more than two scale points. Overall, 95% of AI-generated ratings met the predefined agreement criterion, suggesting substantial consistency between AI and faculty judgments. These findings provide preliminary evidence that AI can serve as a promising adjunct to faculty evaluation within the KSA reconciliation process. From a research perspective, the study contributes to emerging work on human-AI collaboration in competency assessment. From a practical perspective, integrating AI into Eugene may reduce faculty workload, improve process efficiency, and enhance the scalability of translating coursework into career-relevant competencies. Future research should examine AI-faculty agreement across additional courses and larger datasets to further evaluate the effectiveness of this approach.
Department
University of Tennessee at Chattanooga. Dept. of Psychology