Collaborative Intelligence: 3 Use Cases and 3 Watch-Outs for Using AI in Qualitative Market Research
By Esther Park, Insight Analyst and Jack Fadely, Senior Insight Strategist at RealityCheck
Imagine having an assistant that could sift through vast amounts of unstructured market research data, provide a succinct summary of the information, help identify themes and even help you craft questions for your next discussion guide or questionnaire. With language models like ChatGPT and other AI tools, this is not just a possibility—it’s a reality. At RealityCheck, we’ve been exploring how to best use AI for qualitative market research for a while now and would like to share some thoughts from our experiences. As with all powerful tools, knowing when and how to use them is key.
For us, the challenge is incorporating new technologies into our work to gain efficiency while still delivering the deep, human insights qualitative research is known for.
So, let’s delve into the practical, exciting, and at times challenging world of using AI for qualitative research. In this article, we’ll share three use cases and three watch-outs for using the latest AI tools in qualitative market research. Along the way, we’ll discuss how to leverage AI as a tool and point out the limitations and ethical considerations that come with this.
AI Use Cases for Qualitative Market Research
1. Summarizing Data
One of the most straightforward ways to use ChatGPT and other AI technologies is for summarization. Summarization is a major strength of language models because of their expansive training that enables them to recognize patterns and identify key points. Survey open-ends and interview transcripts are no exception. Whether it is a long response from a respondent or a set of responses from multiple respondents, ChatGPT can help distill a large amount of text into a digestible summary. This isn’t just about saving time, although it can reduce the manual effort involved in qualitative analysis. It also provides a fresh, unbiased perspective and can eliminate the risk of human bias that might overlook some points or give more weight to others.
This positions AI as a great place to start analysis. Becoming more familiar with high-level takeaways can serve as a springboard for deeper analysis. Once you have a handle on the overall themes and insights, you can dive back into the data with a more focused lens, exploring points in greater detail. In this way, ChatGPT acts as a guide, helping you navigate a large amount of data and highlighting points of interest for exploration.
2. Thematic/Sentiment Analysis
Going beyond summarization, AI tools are well suited to perform two common qualitative analyses: Thematic and Sentiment.
Thematic Analysis: We can take AI a step beyond summarizing text to develop a first pass of themes for creating a codebook for human coding. When adopting this approach, it’s crucial to involve a subject matter expert, such as a moderator, in the workflow to suggest hypotheses of themes that may be important and should be represented in the data. Analysts can begin the process by inputting the necessary context or data into the AI interface and asking questions like:
- Summarize the key topics discussed in these responses
- Identify common themes and patterns in this data
After receiving the first output from ChatGPT, we can refine the instructions and run the analysis again. At RealityCheck, analysts, strategists, and moderators work together to discuss emerging findings, often leading to refinement of the initial codebook. By the end of the process, the result is highly humanized, and while traces of AI are left behind, every code that makes it to this point serves a purpose and has passed the test of scrutiny. With a carefully curated set of codes, the analyst proceeds with the rigorous process of human coding, going through the transcripts line-by-line and assigning each verbatim to its respective code.
AI also can provide a frequency analysis of a final list of codes. If given the data to consider and the list of coded themes, AI tools can tell you how many times respondent comments fall into each of your coded buckets. This can be helpful in quantifying qualitative data, especially with larger samples.
Sentiment Analysis: AI does a good job of categorizing sentiment when the positives and negatives expressed in responses are explicit. AI can identify and group the clear expression of joy, satisfaction, frustration or disappointment, for example. As such, AI can be an invaluable tool when you’re trying to get a broad sense of sentiment.
However, it’s essential to pair this AI-enabled sentiment analysis with human oversight and interpretation. AI technologies struggle with categorizing ambiguous, neutral or nuanced responses. It can misinterpret or miss altogether sentiments that are expressed with subtlety.
Discussion Guide Development
Finally, AI can help when it comes to interview and discussion guide development. We’ve seen its value in two areas:
- Generating insightful questions
- Reviewing a completed guide
AI is not going to write a high-quality interview guide, discussion guide, screener or questionnaire on its own. We’ve seen it write guides and questionnaires very poorly on its own. Again, it is best to think of it as a tool or an assistant. With that caveat, let’s say that a researcher is just beginning to write a discussion guide for a study and wants to brainstorm with ChatGPT. They might write something like this:
- Researcher Input: You are another market researcher, collaborating with me to put together a questionnaire to meet the business objectives below. Please note that this product is not yet available and will not be known to consumers. You want to ensure to include a section on reactions to a product packaging concept:
- Specific objectives include:
- Quantify interest in [The Brand/Product] among buyers in general and among specific segments ([Competitive Purchaser 1], [Competitive Purchaser 2], etc.)
- Explore the specific aspects of the [Brand/Product] proposition and brand that drive its appeal among [Audience].
- Understand how to position [The Brand] to minimize cannibalization with [Other Brand Offered by Company]
This will give the AI a specific role to take on and the context required to begin writing questions. You can take the useful parts of ChatGPT’s response and scrap the rest. ChatGPT builds on previous messages. To aid this process, it’s useful to inform ChatGPT about which of its suggestions are being used before transitioning to discuss any gaps in the brainstorming. These interactions are never as simple as copying and pasting into the final discussion guide but can speed up the process and help consider approaches that may not have been top of mind.
Once the draft is completed, you can use ChatGPT to pressure test the guide to ensure it covers all study objectives. To do this, the researcher provides ChatGPT with the background, objectives and any other context for the study, along with the guide, and then ask questions like:
- Take a look through the objectives/content I provided and then the questionnaire and let me know how well this should be expected to answer those key questions from the client. What are the strengths and weak spots?
- Please map key questions from the questionnaire to each of the objectives below: [provide study objectives]
Weaknesses and Watch-Outs
While AI offers a lot of advantages in assisting with qualitative research, it is crucial to recognize its limitations. AI is not the “easy button” that many perceive it to be.
1. Creative/Critical Thinking
The historian David McCullough once said, “Writing is thinking. To write well is to think clearly. That’s why it’s so hard.”
Writing is central to our analysis process at RealityCheck. The act of writing is how our analysts and strategists think through their analysis. It it how conceptual relationships in the data are revealed, pressure-tested and refined into a coherent narrative.
AI can’t do the hard work of thinking-through-writing for you!
What’s more, AI models like ChatGPT are trained on existing data and, thus, are incapable of generating brand-new ideas. These models only remix or reinterpret what they’ve been trained on. As a result, if a discussion guide, questionnaire or report leans too much on AI, it is guaranteed to lack innovation and novelty. For us, it’s critical to have humans do the creative and critical thinking. An analyst and strategist can question, weigh competing perspectives and make the nuanced judgments that are hallmarks of good qualitative research.
While AI is a valuable aid in processing and analyzing large amounts of data, it is not a substitute for the deep, creative, and critical thinking that human researchers bring to the table.
2. Word Count Limit
Another limitation of ChatGPT is the word count limit. Qualitative research often results in a huge amount of textual data that makes data input tedious. Using ChatGPT as we’ve discussed above, requires a workaround. Let’s say you’d like to develop a codebook for a transcript that is six pages long. ChatGPT is only able to receive one page as an input at a time. To avoid a long-winded response to each page of the transcript, you can tell the AI model upfront that you have a large amount of data that you intend to share and that you will send each page individually. In the same message, tell the AI to “only reply by saying ‘acknowledged.’ until otherwise noted.”
ChatGPT remembers the content of previous messages and can take that data into consideration if asked. After you have provided the six pages of input, you can ask questions about the data which the AI has “acknowledged” receiving.
Fortunately, to avoid these challenges, a few qualitative data collection and analysis platforms have integrated GPT technology. These platforms overcome the problem of data limitations and also offer a more customized way to incorporate AI into your workflow. More on this below.
3. Ethical Considerations (Confidentiality)
Finally, and perhaps most importantly, is the need to ensure that confidential information (i.e., client-owned data, PII) remains confidential. It is unclear how data provided to platforms like ChatGPT is used. Therefore, it’s safest to assume that all data shared with the AI will be stored and analyzed in ways that are outside the control of the user. A ChatGPT user can access data controls and turn off chat history/training. When turned on, this ensures that any data provided is permanently deleted and isn’t used to train future models.
Again, using other platforms with AI capabilities may be the best option as they can be more secure and private than more widely used platforms like ChatGPT. For instance, we partner with aha insights technology for online qualitative research. The aha platform has robust, built-in AI capabilities. The aha intelligence tool uses an AI engine that is not shared openly through a third-party system. No proprietary client data is mixed in with general AI knowledge systems.
AI tools have a lot of potential to improve and streamline qualitative research workflows but are no silver bullet. As we’ve found with many new technologies, the power lies in combining human intelligence and artificial intelligence to gain efficiencies in analysis without losing the deep, human insights that qualitative research is known for. We hope that this helps spark ideas on how to incorporate AI into your own market research practices. Remember, the goal is to use these tools as assistants rather than replacements for human skill and creativity.
And, by the way, this article was written with help from ChatGPT. We used it just like you might in your research—to help us but not replace us. It’s all about using AI to improve day-to-day work and to make our research better.
As always, thanks for reading, and we’d love to hear from you. How, if at all, have you used ChatGPT or other AI platforms for qualitative analysis? What pros and cons do you see in AI for the future of market research?
