Publisher
University of Tennessee at Chattanooga
Place of Publication
Chattanooga (Tenn.)
Abstract
Vocational interests did not receive much attention in I/O research though recent research has shown that they are strong predictors for both life and job-related outcomes. Measures of vocational interests still rely heavily on traditional self-report method which is subject to several weaknesses. The present research explores the plausibility of measuring vocational interests through an Artificial Intelligence (AI) chatbot. We intend to examine the psychometric properties of the AI chatbot in measuring vocational interests, namely its (1) internal consistency, (2) split-half reliability, (3) generalizability across samples, (4) factorial validity, (5) convergent validity and discriminant validity, (6) generalizability of convergent and discriminant validity across samples, (7) difference of convergent and discriminant validity on manifest vs. latent variable levels, and (8) criterion-related and incremental validity. 1200 participants will be recruited through a subject pool operated by the Psychology Department at a Southeastern public university. For cross-validation, the whole sample will be split into a training sample (n = 1000) and a test sample (n = 200). All data collection will be completed online. For the training sample, participants will first interact with an AI chatbot provided by an AI firm for approximately 60 minutes. Participants will be asked several questions organized around a series of general topics and will type their responses into the chat box. Participants will then complete two online vocational interest measures on Qualtrics. Demographic information will be collected in this questionnaire. For the test sample, participants will first go through the same study procedure as was used in the training sample. After that, participants will be asked to provide their college GPA, ACT or/and SAT scores, and complete surveys that measure the five criterion variables: Major satisfaction, withdrawal intentions, absenteeism, health complaints, and subjective physical health. To build the predictive models, we will extract textual features of the chat scripts of the training sample and conduct regression analyses. The predictive models will be applied to predict vocational interest scores of the training and test samples by conducting correlation and regression analyses. Psychometric properties will be examined by conducting correlations and regression analyses. We expect the machine-inferred vocational scores to demonstrate good psychometric properties. The proposed research will be the first study using a machine learning approach in predicting vocational interests by developing a cognitive, AI-based chatbot approach, and validating it through comprehensive examinations of its psychometric properties.
Date
October 2022
Subject
Industrial and organizational psychology
Document Type
posters
Language
English
Rights
http://rightsstatements.org/vocab/InC/1.0/
License
http://creativecommons.org/licenses/by-nc/4.0/
Included in
Predicting vocational interests through an AI-based chatbot
Vocational interests did not receive much attention in I/O research though recent research has shown that they are strong predictors for both life and job-related outcomes. Measures of vocational interests still rely heavily on traditional self-report method which is subject to several weaknesses. The present research explores the plausibility of measuring vocational interests through an Artificial Intelligence (AI) chatbot. We intend to examine the psychometric properties of the AI chatbot in measuring vocational interests, namely its (1) internal consistency, (2) split-half reliability, (3) generalizability across samples, (4) factorial validity, (5) convergent validity and discriminant validity, (6) generalizability of convergent and discriminant validity across samples, (7) difference of convergent and discriminant validity on manifest vs. latent variable levels, and (8) criterion-related and incremental validity. 1200 participants will be recruited through a subject pool operated by the Psychology Department at a Southeastern public university. For cross-validation, the whole sample will be split into a training sample (n = 1000) and a test sample (n = 200). All data collection will be completed online. For the training sample, participants will first interact with an AI chatbot provided by an AI firm for approximately 60 minutes. Participants will be asked several questions organized around a series of general topics and will type their responses into the chat box. Participants will then complete two online vocational interest measures on Qualtrics. Demographic information will be collected in this questionnaire. For the test sample, participants will first go through the same study procedure as was used in the training sample. After that, participants will be asked to provide their college GPA, ACT or/and SAT scores, and complete surveys that measure the five criterion variables: Major satisfaction, withdrawal intentions, absenteeism, health complaints, and subjective physical health. To build the predictive models, we will extract textual features of the chat scripts of the training sample and conduct regression analyses. The predictive models will be applied to predict vocational interest scores of the training and test samples by conducting correlation and regression analyses. Psychometric properties will be examined by conducting correlations and regression analyses. We expect the machine-inferred vocational scores to demonstrate good psychometric properties. The proposed research will be the first study using a machine learning approach in predicting vocational interests by developing a cognitive, AI-based chatbot approach, and validating it through comprehensive examinations of its psychometric properties.
Department
University of Tennessee at Chattanooga. Dept. of Psychology