AI in Qualitative Analysis: The Hype, the Hope, and the Reality Check

This week in The Campaign for Real Qual, we explore one of the most actively discussed areas in qualitative research today – using AI for analysis. The attraction is clear – analysis time can be significantly shortened. But how does AI analysis, even when guided by a human, compare to traditional methods? And how do specialised Qual analysis platforms compare to general LLMs such as ChatGPT? Should either have a place in our toolkit?

To address these questions, at FUEL Asia, we did our own pilot in October / November 2024. We selected a study that we felt suitable for AI assisted analysis - a project where the business issue and the topic of discussion were straightforward.

Of course, 4 months is a long time in the world of AI and things change fast – so we’d encourage you to try your own experiments to see the possibilities for your own unique workflows.

Experiment Overview: Three Approaches to Qualitative Analysis

This case study explores three approaches to qualitative analysis on a dataset of 10 online in-depth interviews of 30 minutes each, conducted in Thailand.

1. Human-Only Analysis: As our baseline, we utilised the skills of an experienced qualitative researcher to interpret the data manually. This traditional human-led approach remains the gold standard, providing nuanced insights and setting a clear benchmark for quality.

2. Commercial Qualitative Research Tech Platforms: These specialised platforms are specifically designed to handle qualitative data at scale. In this case study, we tested 3 platforms. They offer user-friendly interfaces and often claim compatibility with diverse languages, allowing for streamlined analysis, quick deployment, and transparency in sourcing findings.

3. ‘DIY AI’: For our third approach, we adopted a more flexible ‘DIY AI’ strategy, combining AWS for transcription* and ChatGPT 4o for analysis.

* Since this experiment, Zoom has introduced free transcription for online meetings, which automatically delineates speakers through their individual logins. However, it struggles with face-to-face sessions where a single audio channel is used, in which case dedicated transcription services offering speaker delineation are preferable.

Human-Only Analysis

Our researcher watched the interviews live, capturing subtle cues and contextual nuances in detailed notes. Using a blend of thematic and narrative analysis, the researcher systematically identified recurring themes while also weaving in the narrative flow of each interview. This approach set the bar for quality, providing a rich, detailed understanding of the data and serving as a valuable benchmark for comparing the AI-driven methods.

Commercial Qualitative Research Tech Platforms

Commercial qualitative research platforms are designed specifically for streamlined, efficient analysis. We only tried 3 in our experiment, but a quick search on Insight Platforms reveals over 100 platforms offering AI assisted analysis. As an aside, the sheer volume of providers can feel overwhelming to potential buyers, especially when many appear to offer the same solutions. I assume as the market matures, the differentiation between providers will become clearer.

Again, a reminder – this experiment took place a few months ago and the platforms we tested will no doubt have received numerous upgrades. This was our assessment at the time:

• User-Friendly Design: These platforms are made for quick deployment and typically require little to no training. They’re intuitive and designed to guide users smoothly through analysis, making them appealing where speed matters, a quick topline read is needed, or when the user isn’t an experienced qualitative analyst.

• Transparency and Segment Analysis: Unlike some DIY approaches, commercial platforms allow users to trace findings back to their sources, increasing transparency and giving users confidence in their analysis. They also simplify segment-specific analysis and streamline the production of supporting media, such as video reels, which of course can be incredibly useful during reporting.

• Language Development Gaps: However, when it comes to the Thai language, these platforms revealed a notable development lag. Different platforms presented different challenges; one platform could not recognise Thai speech in Zoom audio files, despite claiming language support. Another platform required us to split the analysis into two parts (due to token limits of the platform), which meant we had to create two reports and later combine them manually (using ChatGPT as a workaround).

Every AI tool is of course now a ‘work in progress’. I suspect the language specific issues we faced may have already been addressed.

‘DIY AI’ Approach: Flexibility, Customisation, and Price

The flexibility of a DIY AI approach quickly became clear. Using AWS and ChatGPT, we tailored workflows specifically to our project’s needs and our approach to qualitative research. These workflows can be easily adapted to different projects or client needs by developing custom GPTs or using the Projects function within ChatGPT - both straightforward options even if you haven’t experimented yet.

Here were some ways we were able to customise our workflows:

1. We ran different types of qualitative analysis (content analysis, thematic analysis, narrative analysis, grounded theory) – credit to Ray Poynter for suggesting this in his training. We found this helped our thinking.

2. Similarly, the ability to use well known frameworks (e.g. Maslow’s hierarchy or Jobs to be Done) to frame the findings. Again, experiments that can be run quickly and with the potential to quickly look at an issue through a different lens.

Compared to specialised platforms, setting up ‘DIY AI’ requires initial adjustments to workflows and prompts, along with a lot of trial and error to get the best results. However, this flexibility can be a major advantage, allowing us to quickly adapt analysis and reporting to meet the unique needs of each project. This approach is inexpensive – a Chat GPT Plus subscription at USD30 per month and about USD15 of transcription costs.

When using LLMs, it’s critical to have robust QC mechanisms in place to identify potential inaccuracies, hallucinations or questionable interpretations. In practice these means asking for evidence from your LLM – and if necessary, asking exactly which transcripts they used to report a specific finding. LLMs also tend to make inferences that a human would not necessarily make, based on its training with other data (even when you instruct it to only work with the information provided in the project). We previously referred to AI chatbot ‘moderators’ acting like a novice moderator, the same can be said at times with analysis, especially with some of the inferences being made. The experienced qualitative practitioner’s ability to frame a learning in a business context still feels a very human endeavour. Even when AI analysis appears accurate on the surface, its lack of contextual grounding means it may miss the deeper, more strategic interpretations that clients value most.

Of course, language models can analyse language quite well but understand meaning less well. The experienced qualitative practitioner, as we highlighted in The Essence of Good Qual, can understand someone’s context – the often hidden forces shaping their responses, spot the contradictions, the tone, the body language etc to uncover what they mean, not just what they say. As qualitative practitioners, we know never to take what someone says purely at face value. While language models continually improve, the subtlety required to interpret nuanced cultural and situational contexts remains a distinctly human skill.

Conclusion

In summary, AI assisted analysis, in our experience, can’t come close to that of a skilled human – because it analyses language and not meaning. That’s not to say it’s not useful - we currently see four use cases:

• Analysing standalone textual data, such as social media content or customer reviews, where text alone represents the entire dataset

• Quickly summarising conversational surveys or AI chatbot-led interviews

• Supporting standardised analysis and reporting on international projects, especially where transcripts are used in a non-native language

• As a supportive analysis assistant, to do tightly defined analyses, under the guidance of an experienced qualitative researcher (as used in our pilot)

We all differ in how we approach analysis. Self-experimentation is necessary to understand the models well enough to create your own use cases and workflows.

One area we’re keen to explore further is AI’s potential in analysing visual materials collected through pre-tasks or online communities, or perhaps even supporting semiotic analysis of categories or visual social media data. We’re seeing other agencies experiment with this, and it’s an area we plan to explore.

As always, I'd love to hear about your experiences - have you tried AI analysis, and in which scenarios have you found it truly useful or perhaps limited? Next week we'll examine AI's impact on project deliverables, an area where I believe AI is opening exciting new possibilities for qualitative practitioners.

See you next week.