A mixed-methods study comparing human-led and ChatGPT-driven qualitative analysis in medical education research

Qualitative research, used to analyse non-numerical data including interview texts, is crucial in understanding medical education processes. However, it is often complex and time-consuming, leading to an interest in technology for streamlining the analysis. This study investigated the applicability of ChatGPT, a large language model, in thematic analysis for medical qualitative research. Previous research has used ChatGPT to explore the deductive process as a qualitative study. This study evaluated thematic analysis including the inductive process by ChatGPT with reference to human qualitative analysis. A convergent design mixed-methods study was used. Using a thematic analysis approach, ChatGPT (model: GPT-4) analysed some interview data from a previously published medical research article. The assessors evaluated the qualitative analysis of ChatGPT using human qualitative analysis as a benchmark. Three assessors compared the human-conducted and ChatGPT-driven qualitative analyses. ChatGPT scored higher in most aspects but showed variable transferability and mixed depth scores. In the integrated analysis including qualitative data, six themes were identified: superficial similarity of results with human analysis, good first impression, explicit association with data and process, contamination by directions in prompts, deficiency of thick descriptions based on context and research questions, and lack of theoretical derivation. ChatGPT excels at extracting key data points and summarising information; however, it is prone to prompt contamination, which necessitates careful scrutiny. To achieve deeper analysis, it is essential to supplement the research context with human input and explore the theoretical framework.