What started as a deep-dive in to the grocery shopping habits of male college students at Michigan State University became an opportunity to further validate a technology that Quester, a market research firm in Des Moines, has been using for more than ten years–a software-based moderator, backed by artificial intelligence.
Results-wise, as a whole college-aged males were disengaged in health. Although they were aware of how to be healthy, and, for the most part, thought of themselves as healthy, their stated behavior did not match their claims. When grocery shopping, they are likely to mirror their parents’ eating and shopping habits. At the end of the day, most probably wish their parents were still shopping for them–the sample saw grocery shopping as “annoying” and “a hassle.”
The real story, however, lies in the section that most people skim: the methods. The original study, conducted in 2012 at Michigan State by Dr. Patricia Huddleston and her team, involved several interviewers who conducted face-to-face in-depth interviews with undergraduate students. With the help of Garrett McGuire, VP of Corporate Strategy at Quester and one of Huddleston’s former students, Quester ran a second identical study using their online moderator. The proprietary software, developed in 2001 and debuted in 2004, is built around psychiatric interviewing techniques and computational linguistics. The technology, leveraged by marketing researchers and trained linguists allows companies to conduct hundreds or thousands of one-one in-depth interviews at a time in seven different languages.
To compare the face-to-face interviewers and Quester, the interview discussion guide and original protocol were replicated exactly. Sampling was conducting the same way for each study: drawing students from MSU’s “Angel” system whereby students can be targeted to take certain studies in exchange for class credit. In this case, we targeted college-aged men who grocery shop.
Findings were not shared across Michigan State University and Quester teams until both studies were completed. Once fielding and analysis was completed in the online study, transcripts from Michigan State were provided to Quester for a comparative analysis. Both studies went through the same analytic process leveraging Quester’s proprietary natural language processing software, led by a trained language analyst.
Both approaches garnered highly similar results, both in terms of quality of language received from participants and the ability to add depth of understanding through follow-up questioning. Though small sample sizes, the table below shows frequency of key themes from natural language. Overall, similar ideas and themes came up in responses at comparable frequencies, regardless of method.
| Key Themes & Ideas | Face-to-Face | AI |
|---|---|---|
| sample size | 18 | 43 |
| Feel Good About Myself | 94% | 91% |
| Positive/Carefree | 89% | 79% |
| Energized | 50% | 49% |
| Not Lethargic/Tired | 22% | 23% |
| Full/Satisfied | 22% | 23% |
| Hunger | 0% | 7% |
| Better About Health | 22% | 23% |
| Less Digestion Problems | 6% | 9% |
| Getting Nutrition | 11% | 7% |
| Less Weight Gain | 6% | 5% |
| Diet | 6% | 2% |
| Not Feel Guilty | 0% | 5% |
| Motivation | 33% | 14% |
| Accomplishment | 11% | 9% |
| Think Clearly/Focused | 22% | 5% |
While the overall results showed very high consistency between methods, there was one area in which Quester held an advantage: consistency in questioning. How questions are asked clearly impact the responses collected.
As exemplified in the table below, it is clear that, compared to face-to-face interviewers, Quester was better able to maintain higher levels of reliability through consistent wording and a lack of bias. As a specific example, one question on the interview guide was “What are some examples of a typical breakfast, lunch or dinner?” While the online moderator used the exact wording every time, the face-to-face interviewer varied, with one phrasing the question as “If you just want to let us know how you choose to eat throughout the day; like how you pick your meals and things like that.” Although the wording difference may seem small, this inconsistency and use of the word “meal” or “eat” can lead respondents down different paths. In the face-to-face interviews that include those words, beverages were rarely mentioned. However, when thinking of “a typical breakfast,” the exact wording used by Quester, respondents thought of drinks, like coffee or water as part of that meal.
| Face-to-Face Questions | AI Questions |
|---|---|
| And what are some examples of a typical breakfast, lunch or dinner? | What are some examples of a typical breakfast, lunch or dinner? |
| If you just want to let us know how you choose to eat throughout the day; like how you pick your meals and things like that. | What are some examples of a typical breakfast, lunch or dinner? |
| If you can give us examples of a typical breakfast, lunch & dinner? So you can kind of just take us through your day of what you would eat normally. | What are some examples of a typical breakfast, lunch or dinner? |
| Key Themes & Ideas | Face-to-Face | AI |
|---|---|---|
| sample size | 18 | 43 |
| Protein | 78% | 77% |
| Grains | 83% | 72% |
| Produce | 56% | 58% |
| Meals | 67% | 30% |
| Eating Out | 28% | 14% |
| Skipping Meals | 33% | 12% |
| Preparing Meals | 39% | 7% |
| Crock Pot Meals | 17% | 2% |
| Dairy | 39% | 28% |
| Sugars | 44% | 21% |
| Beverages | 6% | 19% |
| Coffee | 0% | 14% |
| Water | 0% | 9% |
| Sport Drinks | 6% | 2% |
| Juices | 0% | 0% |
Inherently, there are areas that face-to-face interviewers can excel in, and areas that an online moderator can excel in. The table below summarizes the main metrics analyzed in the comparison:
| Face-to-Face | Metric | Quester’s AI |
|---|---|---|
| Yes | Method yields quality language and ideas | Yes |
| Yes | Moderator can follow-up questions for deeper meaning | Yes |
| No | Method can be employed with quantitative samples | Yes |
| No | Moderator always asks questions with same wording | Yes |
| No | Moderator is NOT prone to insert bias | Yes |
| No | Respondent language can be used without transcription | Yes |
| Yes | Moderator can witness respondent non-verbal cues | No |
| Yes | Method yields quality language and ideas | Yes |
| Yes | Moderator can infer meaning based on respondent tone | No |
So, what are the tradeoffs between using face-to-face interviewers vs. a virtual moderator? Are a few wording inconsistencies worth the opportunity to observe body language? Hear voice tone? Or are losing those cues a small price to pay for this level of efficiency and limited bias? As Quester continues to blur the lines between quantitative and qualitative research, these questions will continue to be tried and tested.
Either way, we know to not bank on college guys being passionate about grocery shopping.
For more information about this study, please contact Garrett McGuire at Garrett.McGuire@Quester.com.
