Almost Half of Market Researchers are doing Market Research Wrong! – My Interview with the QRCA (And a Quiet New Trend – Science Based Qualitative).
Two years ago I shared some research on research about how market researchers view Quantitative and Qualitative research. I stated that almost half of researchers don’t understand what good data is. Some ‘Quallies’ tend to rely and work almost exclusively with comment data from extremely small samples (about 25% of market researchers surveyed), conversely there is a large group of ‘Quant Jockey’s’ who while working with larger more representative sample sizes, purposefully avoid any unstructured data such as open ended comments because they don’t want to deal with coding and analyzing it or don’t believe in it’s accuracy and ability to add to the research objectives. In my opinion both researcher groups have it totally wrong, and are doing a tremendous disservice to their companies and clients. Today, I’ll be focusing on just the first group above, those who tend to rely primarily on qualitative research for decisions.
Note that today’s blog post is related to a recent interview, which I was asked to take part in by the QRCA’s (Qualitative Research Consultant’s Association) Views Magazine. When they contacted me I told them that in most cases (with some exceptions), Text Analytics really isn’t a good fit for Qualitative Researchers, and asked if they were sure they wanted to include someone with that opinion in their magazine? I was told that yes, they were ok with sharing different viewpoints.
I’ll share a link to the full interview in the online version of the magazine at the bottom of this post. But before that, a few thoughts to explain my issues with qualitative data and how it’s often applied as well as some of my recent experiences with qualitative researchers licensing our text analytics software, OdinText.
The Problem with Qualitative Research
IF Qual research was really used in the way it’s often positioned, ‘as a way to inform quant research’, that would be ok. The fact of the matter is though, Qual often isn’t being used that way, but instead as an end in and of itself. Let me explain.
First, there is one exception to this rule of only using Qual as pilot feedback for Quant. If you had a product for instance which was specifically made only for US State Governors, then your total population is only N=50. And of course it is highly unlikely that you would ever get all the Governors of each and every US State to participate in any research (which would be a census of all governors), and so if you were fortunate enough to have a group of say 5 Governors whom were willing to give you feedback on your product or service, you would and should obviously hang on to and over analyze every single comment they gave you.
IF however you have even a slightly more common mainstream product, I’ll take a very common product like hamburgers as an example, and you are relying on 5-10 focus groups of n=12 to determine how different parts of the USA (North East, Mid-West, South and West) like their burgers, and rather than feeding directly into some quantitative research instrument with a greater sample, you issue a ‘Report’ that you share with management; well then you’ve probably just wasted a lot of time and money for some extremely inaccurate and dangerous findings. Yet surprisingly, this happens far more often than one would imagine.
Cognitive Dissonance Among Qual Researchers when Using OdinText
How do I know this you may ask? Good Text Analytics software is really about data mining and pattern recognition. When I first launched OdinText we had a lot of inquiries from Qualitative researchers who wanted some way to make their lives easier. After all, they had “a lot” of unstructured/text comment data which was time consuming for them to process, read, organize and analyze. Certainly, software made to “Analyze Text” must therefore be the answer to their problems.
The problem was that the majority of Qual researchers work with tiny projects/sample, interviews and groups between n=1 and n=12. Even if they do a couple of groups like in the hamburger example I gave above, we’re still taking about a total of just around n=100 representing four or more regional groups of interest, and therefore fewer than n=25 per group. It is impossible to get meaningful/statistically comparable findings and identify real patterns between the key groups of interest in this case.
The Little Noticed Trend In Qual (Qual Data is Getting Bigger)
However, slowly across the past couple of years or so, for the first time I’ve seen a movement of some ‘Qualitative’ shops and researchers, toward Quant. They have started working with larger data sets than before. In some cases, it has been because they have been pulled in to manage larger ongoing community/boards, in some cases larger social media projects, and in others, they have started using survey data mixed with qual, or even better, employing qualitative techniques in quant research (think better open-ends in survey research).
For this reason, we now have a small but growing group of ‘former’ Qual researchers using OdinText. These researchers aren’t our typical mixed data or quantitative researchers, but qualitative researchers that are working with larger samples.
And guess what, “Qualitative” has nothing to do with whether data is in text or numeric format, instead it has everything to so with sample size. And so perhaps unknowingly, these ‘Qualitative Researchers’ have taken the step across the line into Quantitative territory, where often for the first time in their career, statistics can actually be used. – And it can be shocking!
My Experience with ‘Qualitative’ Researchers going Quant/using Text Analytics
Let me explain what I mean. Recently several researchers that come from a clear ‘Qual’ background have become users of our software OdinText. The reason is that the amount of data they had was quickly getting “bigger than they were able to handle”. They believe they are still dealing with “Qualitative” data because most of it is text based, but actually because of the volume, they are now Quant researchers whether they know it or not (text or numeric data is irrelevant).
Ironically, for this reason, we also see much smaller data sizes/projects than ever before being uploaded to the OdinText servers. No, not typically single focus groups with n=12 respondents, but still projects that are often right on the line between quant and qual (n=100+).
The discussions we’re having with these researchers as they begin to understand the quantitative implications of what they have been doing for years are interesting.
Let me preface this with the fact that I have a great amount of respect for the ‘Qualitative’ researchers that begin using OdinText. Ironically, the simple fact that we have mutually determined that an OdinText license is appropriate for them means that they are no longer ‘Qualitative’ researchers (as I explained earlier). They are in fact crossing the line into Quant territory, often for the first time in their careers.
The data may be primarily text based, though usually mixed, but there’s no doubt in their mind nor ours, that one of the most valuable aspects of the data is the customer commentary in the text, and this can be a strength
The challenge lies in getting them to quickly accept and come to terms with quantitative/statistical analysis, and thereby also the importance of sample size.
What do you mean my sample is too small?
When you have licensed OdinText you can upload pretty much any data set you have. So even though they may have initially licensed OdinText to analyze some projects with say 3,000+ comments, there’s nothing to stop them from uploading that survey or set of focus groups with just n=150 or so.
Here’s where it sometimes gets interesting. A sample size of n=150 is right on the borderline. It depends on what you are trying to do with it of course. If half of your respondents are doctors (n=75) and half are nurses (n=75), then you may indeed be able to see some meaningful differences between these two groups in your data.
But what if these n=150 respondents are hamburger customers, and your objective was to understand the difference between the 4 US regions in the I referenced earlier? Then you have about n=37 in each subgroup of interest, and you are likely to have very few, IF ANY, meaningful patterns or differences.
Here’s where that cognitive dissonance can happen — and the breakthroughs if we are lucky.
A former ‘Qual Researcher’ who has spent the last 15 years of their career making ‘management level recommendations’ on how to market burgers differently in different regions based on data like this, for the first time is looking at software which says that there are maybe just two to 3 small differences, or even worse, NO MEANINGFUL PATTERNS OR DIFFERENCES WHATSOEVER, in their data, may be in shock!
How can this be? They’ve analyzed data like this many times before, and they were always able to write a good report with lots of rich detailed examples of how North Eastern Hamburger consumers preferred this or that because of this and that. And here we are, looking at the same kind of data, and we realize, there is very little here other than completely subjective thoughts and quotes.
Opportunity for Change
This is where, to their credit, most of our users start to understand the quantitative nature of data analysis. They, unlike the few ‘Quant Only Jockie’s’ I referenced at the beginning of the article already understand that many of the best insights come from text data in free form unaided, non-leading, yet creative questions.
They only need to start thinking about their sample sizes before fielding a project. To understand the quantitative nature of sampling. To think about the handful of structured data points that they perhaps hadn’t thought much about in previous projects and how they can be leveraged together with the unstructured data. They realize they need to start thinking about this first, before the data has all been collected and the project is nearly over and ready for the most important step, the analysis, where rubber hits the road and garbage in really should mean garbage out.
If we’re lucky, they quickly understand, its not about Quant and Qual any more. It’s about Mixed Data, it’s about having the right data, it’s about having enough data to generate robust findings and then superior insights!
Final Thoughts on the Two Meaningless Nearly Terms of ‘Quant and Qual’
As I’ve said many times before here and on the NGMR blog, the terms “Qualitative” and “Quantitative” at least the way they are commonly used in marketing research, is already passé.
The future is Mixed Data. I’ve known this to be true for years, and almost all our patent claims involve this important concept. Our research shows time and time again, that when we use both structured and unstructured data in our analysis, models and predictions, the results are far more accurate.
For this reason we’ve been hard at work developing the first ever truly Mixed Data Analytics Platform, we’ll be officially launching it three months from now, but many of our current customers already have access. [For those who are interested in learning more or would like early access you can inquire here: OdinText.com/Predict-What-Matters].
In the meantime, if you’re wondering whether you have enough data to warrant advanced mixed data and text annalysis, check out the online version of article in QRCA Views magazine here. Robin Wedewer at QRCA really did an excellent job in asking some really pointed questions that forced me too answer more honestly and clearly than I might otherwise have.
I realize not everyone will agree with today’s post nor my interview with QRCA, and I welcome your comments here. I just please ask that you read both the post above, as well as the interview in QRCA before commenting solely based on the title of this post.
Thank you for reading. As always, I welcome questions publicly in post below or privately via LinkedIn or our Inquiry form.
Hi Tom! Thanks for your post. While I understand the premise of what you’re speaking about, when I was on the client side, we used qualitative research for a number of purposes. Should it only be used to inform quant? In my opinion yes, however, there were many times where a “quick” read was needed and so marketers asked for qual to validate their thoughts and assumptions. It is not a proper way of using qual, but it is often used due to time and money constraints. And with qual, you can put the product in front of the consumer and gauge their reactions real time, whereas with quant you have to wait weeks. Hopefully, with agile MR on the rise, we may finally be able to get the hard numbers needed to validate some of these inquiries. Until then, however, qual will be used to make major decisions, although we all know, it shouldn’t be.
Interesting perspective…though I think you’re vantage point of qual (at least in your article/blog) in the context of how its used is very narrow. In prep for our webinar next week (https://www.leresearch.com/resources/webinars/6steps-to-perfect-qual-step1 ), we all discussed how qual is often OVER used in sample, and how, when looking for patterns prior to moving to more scientific measure, engineering scientists have often lauded that more than 10 interviews per segment is usually a waste of time. Hopefully you’ll register/listen in, and where applicable, question the panel on this?
I do agree with your perspective that text analytics is typically not a good tool for quallies…unless they are doing more quant/qual work (which is something we are seeing more of). Specifically, as it relates to buyers moving more to behavioral and less to attitudinal…video that translates to text that translates to NLP…the ability to combine those tools would on the surface hold strong potential.
Noticed you’ll be at the conference in Miami in two weeks (IA)…look forward to meeting you, and hopefully getting an opportunity to discuss further.
Brett
@Courteney, Thanks for your comment. I have no doubt that ‘Qualitative’ will continue to be used in an incorrect way, in fact, this is another trend I’ve ironically also seen increasing as fewer methodologists remain in the field relative to the number of new entrants, some being technologists only who are implementing additional fast, cheap, small data type solutions to get “a quick read”. The question I ask, is why is that trade off needed? Why cant it be cheap, quick and large enough (accurate)?
@Brett, Yes I’ll be in Miami, look forward to catching up with you there 🙂
Traditional qualitatitive is storytelling. Traditional quantitative is storytelling with evidence (which few people look and understand). My intention here is not to disrespect either, but to draw attention to something else.
What I’ve been thinking recently is the success of design thinking/service design/ethnography in certain fields. There, far more advanced conclusions are drawn from far smaller samples, using far less evidence. I’ve read a manifesto written by a successful practitioner claiming that he can get all the necessary insight from observing and interviewing 4-6 people in the target group. At that point he can infer whole group’s motivations and needs. What’s more, the evidence is not in the format that anyone else could decipher or argue with, because it is in the researcher’s head (or in the groupthought illusion developed by the research team together).
Yet this is a successful and fast-growing industry. My conclusions are that:
– They have better stories to tell.
– The design-prototype-test-process they are using does yield additional information, but it also pulls the client persons “into the magic circle” rather than allow them remaining outside as more or less disinterested observers.
– There is a large-enough set of situations where traditional qual/quant/mixed falls short of client needs.
Hi Tom,I’m a former market researcher at Sprint, and have received your emails for some time. Now I have a “mobile direct response” advertising solution that involves a universal mobile speed dial (#250) paired with a Spoken Keyword, which causes a voice call to be routed to an advertiser’s front desk or call center.
But my original concept was an easy customer feedback solution, based on the idea that nobody really wants to answer a bunch of rating scales in a CSAT survey — all they really want to do is leave a comment.
Try this out on your own mobile phone, and let me know if you have any interest in experimenting with our ‘customer access method” that can capture verbatim comments — which you can then transcribe and analyze.
dial #250 and say PORK SURVEY
(this is a real application for a researcher friend of mine).
I’ve used qual as a sample for quant successfully and I’ve used qual as an end in itself. However when I use it as an end in itself, I use no texts, or quant language at all and I’ve had enormous success working this way — it’s customer-focused research in it’s truest form. Using this method, I’ve brought clients results from 60%-1,000% ROI (and I’m referring to launches of Rx products to one person shop marketing).
You always have an interesting perspective Tom, and I agree with most of your points, especially your thoughts on mixed data. However, I think you overreach unintentionally when you say, ““Qualitative” has nothing to do with whether data is in text or numeric format, instead it has everything to so with sample size.” I don’t think that is really the point you are trying to make.
Yes, in context, with adequate quantitative sample sizes, data can be quantitatively analyzed in either text or numeric format. However, it is not true that qualitative has only to do with sample size. Qualitative also very much relates to the structure and biased or unbiased manner of data collection. Assign me a project whose goal is to prove a point, and I can pick a moderator and design them a guide to achieve the desired results. To get truly scientific quantitative results, it is necessary to deploy a technique where the data collection instrument is administered consistently to respondents in an unbiased format.
Thank you @Jeffrey“ Assign me a project whose goal is to prove a point, and I can pick a moderator and design them a guide to achieve the desired results. To get truly scientific quantitative results, it is necessary to deploy a technique where the data collection instrument is administered consistently to respondents in an unbiased format.”
Respectfully I disagree. Whether “Quantitative” or “Qualitative” (I’m continuing to use these terms even though I think they have ever decreasing value other than sample), anyway, either of these ‘studies’ can be designed to create leading questions and bad data and conclusions. However, it is false to assume that ‘Quantitative’ research can’t get unleading questions, or drive into subconscious decision making etc. There are numerous techniques that can be employed even in surveys (though we are not limited to surveys), but there are techniques like thematic apperception tests (think picture story exercises), laddering (with skip logic), fill in the blanks/sentence completion exercises, I could go on and on, there are really few ‘qualitative techniques’ that can’t be employed en masse with good sampling to get you projectable and repeatable results, even in surveys.
I think the power of larger qual sample sizes is a fantastic area of growth in the coming future and the power of system like OdinText enable this. however, I don’t necessarily agree that ‘qualitative’ research has everything to do with sample size. Qual research is sooo much more than that. In fact, the definition of the word ‘qualitative’ in the dictionary is “….Relating to, measuring, or measured by the quality of something rather than its quantity”: Note that it is about the quality of something and NOT its quantity. In the MR case, I don’t mean quality in that qual research is better than quant. No, I love quant and statistics and advocate mixed method research all the time. The definition, though, highlights that qualitative research is about the nature of the data and collection method. Qualitative research is about the moderation, the two-way dialogue, the digging under the skin of something to gain a different perspective. And that, is not about open ends, its much more. that’s not to say that I don’t believe ‘mass qual’ isn’t possible in the future (as technology enables productive collection methods) nor that scaled samples don’t play a part, but Qualitative research is definately not just about small samples. They are often small because of the need for interation and moderation.
Hi Tom,Very interesting post and I love your forward-looking perspective on research methods. The one thing I disagree with is the assertion that qual on its own (without quant numbers to validate) has no place in the researcher’s arsenal. While qual is certainly not sufficient in the example you cite (where the goal is to identify differences across groups), this is not the way in which qual research is generally (or should be) used – at least not in my experience. Qual can be a great option for gaining a much deeper understanding of consumer motivations, pain points and even brand perceptions – all of which can be used to identify new opportunities for innovation. While a follow-up quant study may be a luxury in these cases (for validation), it is often not feasible to conduct two phases of research and we have gotten tremendous value from the qual research in and of itself. I do, however, agree that the ability to do this type of “qual” research at scale would be nirvana!