Qualitative vs. quantitative data for product managers

At its simplest, data can be broken down into two different categories: quantitative data and qualitative data. But what’s the difference between the two? And when should you use them? And how can you use them together?

Types of Data:

1. Quantitative data.

This type of data can be counted, measured, and assigned a numerical value. It's data you can put a number on – like user counts, revenue figures, and frequency of specific user actions.

Here are some primary types of quantitative data with examples:

• Discrete Data: Data that can only take specific, distinct values and cannot be made more precise. Example: Number of students in a class (you can't have half a student).

• Continuous Data: Data that can take any value within a range and can be made increasingly precise. Example: The weight of a person (it could be 70 kg, 70.5 kg, 70.55 kg, and so on).

While there are other types of quantitative data, such as: ration, interval, time series, we will be focusing on these 2 types as the most important.

Scenario: An e-commerce website, "Trendy panda", wants to improve its sales for the upcoming holiday season. To devise an effective strategy, the product manager decides to look at quantitative data from the previous year's holiday sales.

Quantitative Data examples:

• Total Visitors: The number of unique visitors to the website during the holiday season. Data: 500,000 visitors

• Conversion Rate: The percentage of visitors who made a purchase. Data: 5% (25,000 purchases)

• Average Order Value (AOV): The average amount spent by customers per transaction. Data: $50

• Cart Abandonment Rate: The percentage of users who added products to their cart but did not complete the purchase. Data: 60%

• Daily Active Users (DAU): The average number of users active on the website daily during the holiday season.Data: 16,667 users/day

• Top-selling Products: The products with the highest number of sales.Data: "Panda Jacket" with 7,000 units sold, "Holiday Panda Mug" with 6,500 units sold

• Customer Feedback Score: On a scale of 1-10, the average feedback score after making a purchase. Data: 8.5

Advantages:

• Objectivity and Accuracy: Since it's based on numerical data, quantitative research is more objective and can provide precise and accurate results.
• Generalizability: Quantitative research often uses larger sample sizes, allowing for generalizations and conclusions that can be applied to the broader population.
• Replicability: The structured and standardized nature of quantitative research makes it easier to replicate the study for validation purposes.
• Statistical Analysis: The numeric nature of the data allows for a broad range of statistical analyses, making it possible to determine relationships, correlations, and causations.
• Clear Metrics: Quantitative data provides clear, definitive metrics and results that can be easily benchmarked, tracked, and compared over time.

Disadvantages:

• Lack of Depth: Quantitative data focuses on breadth rather than depth, meaning it may not capture the full essence or nuances of a subject or issue.
• Limited Flexibility: Once the research process begins, it can be challenging to make changes to the structured design of a quantitative study.
• Potential for Misinterpretation: Without context or qualitative insights, raw numbers can sometimes be misinterpreted or may not capture the full story. Causation errors also common.
• Detachment from Human Aspect: Being number-centric, quantitative research may miss out on human emotions, feelings, and experiences that can be crucial for understanding certain phenomena.
• Sampling Bias: If not done correctly, there's a risk that the sample chosen for a study might not be representative of the broader population, leading to skewed results.

Quantitative Data Collection methods:

Quantitative data collection methods involve gathering numerical data which can be  statistically analysied later. These methods are designed to collect structured and measurable data from participants. Here are the main quantitative data collection methods:

It is recommended to ensure that sampling technique is free from bias (measurement system error) and random. If tools for continuous data measurement (example length) involved they should have enough resolution to capture difference in measurement (rule of thumb - at least 10 measurement between anticipated range)

• Surveys and Questionnaires: Structured sets of questions designed to gather data from a large number of respondents. They can be administered face-to-face, over the phone or online. Example: Customer feedback, market research, academic research.

• Observational Research: Observing and recording behaviors or events as they occur in a structured manner without interference just recording natural things hapenning in life. Uses: Traffic studies, behavioral patterns in natural environments.

• Structured Interviews: Description: Interviewers use a set list of standardized questions, ensuring consistency across all participants. Uses: Market research, job interviews.

• Censuses: A complete enumeration of a population at a point in time with respect to well-defined characteristics, e.g., age, gender, occupation. Uses: Governmental planning, demographic studies.

• Clickstream Data: Collects user clicks and navigational data as users interact with websites or software. Uses: Website optimization, understanding user navigation patterns.

• Biometric Data Collection for vital parameters: Collects data based on physiological characteristics. Uses: Health monitoring, fitness tracking.

• A/B Testing (Split Testing): Compares two versions of a webpage or app against each other to determine which one performs better in terms of specific metrics. Uses: Website optimization, marketing campaign evaluations.

• Feedback Forms: Description: Forms filled out by users or customers providing feedback on a product, service, or experience. Uses: Product improvement, service quality assessment.

• Counting and Tracking: Counting events, occurrences, parameters or items Uses: Inventory management, monitoring frequency of events.

Quantitative Data Collection common errors:

• Sampling Errors: This error occurs when the sample is not representative of the entire population. It can result from a flawed sampling technique. Mitigation: Use stratified sampling, random sampling, or other techniques to ensure representativeness.

• Measurement Errors: Errors that arise from flawed data collection instruments or techniques. Mitigation: Regularly calibrate instruments and pilot-test questionnaires to ensure reliability and validity. You need to make sure that measurements: Repeatable with the same tool, with same tool and different persons, discrimination is ok and results are not changing due to external factors (weather, humidity, accumulated measurement error)

• Response Errors: Errors that occur when respondents provide inaccurate answers intentionally or unintentionally. Mitigation: Design clear and unbiased questions, and provide assurance of confidentiality to encourage honest responses. Questions can be repeated in longer surveys
• ‍Processing Errors: Mistakes in coding, data entry, or data handling. Mitigation: Implement double data entry, use automated tools to check for errors, create entry rules and make sure survey team aware of them, Poka Yoke principles on data entry.

• Social Desirability Bias:Respondents might answer questions in a way that they believe will be viewed favorably by others. Mitigation: Ensure anonymity and emphasize the importance of truthful responses.

‍

2. Qualitative data.

This revolves around the descriptive and the intangible aspects. It's more about the user's experience, feelings, and reasons behind their actions – it gives context to the numbers.

• Binary Data:  Data that has two distinct categories or states, often represented as 0 or 1. Example: Gender, when simplified, can be binary: Male or Female. Another example is survey responses with "Yes" or "No" choices.

• Nominal Data: Data that represents categories without any kind of order or hierarchy. It's purely naming or labeling data.Example: Colors of cars (Red, Blue, Green), types of fruits (Apple, Banana, Cherry), or brands of a product (Nike, Adidas, Puma). Each category stands alone without implying any kind of ranking or relation to previous groups.

• Ordinal Data: Data that can be categorized and ranked in a specific order. The intervals between the categories are not known, but there's a clear hierarchy or progression. Example: Likert scale survey responses such as "Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree." Another example is educational levels where "High School" precedes "Bachelor's Degree," which in turn precedes "Master’s Degree."

Scenario: Following the quantitative analysis, the "Trendy Panda" e-commerce website's product manager decides to gather qualitative data to understand the reasons behind certain behaviors observed during the previous holiday season, especially the high cart abandonment rate.

Qualitative Data Examples:

• User Feedback via Surveys: Responses: "I love browsing your site, but the checkout process is too long and complicated.""I wish there were more product images from different angles.""Your site doesn’t seem to remember my cart items if I leave and come back later."

• User Reviews and Testimonials: Reviews: "The Panda Jacket is super warm and fits perfectly. However, I wish there was a size guide to help me choose better.""The Panda Mug's design is lovely, but it's smaller than I expected."

• Focus Group Feedback: Insights majority of the participants expressed that they'd like a more interactive platform, where they can see user-generated images of products or video reviews.

• Customer Support Interactions: Common Queries: "How do I use this discount code?""Is there an option for gift wrapping?""Can I change the delivery date for my order?"

Qualitative Data Advantages:

• Depth of Understanding: Qualitative data provides rich, detailed insights which can help in understanding the depth, context, and reasoning behind certain behaviors or phenomena.
• Flexibility: The qualitative approach often allows for more flexibility in the research design, allowing the researcher to adapt and delve deeper into areas of interest as the study progresses.
• Unanticipated Insights: Open-ended, unstructured formats can lead to unexpected discoveries that might not be found through structured quantitative methods.
• Natural Settings: Many qualitative methods observe participants in their natural settings, offering a more authentic and unbiased perspective.

Qualitative Data Disadvantages:

• Subjectivity: Analysis of qualitative data can be influenced by the researcher’s own biases and interpretations.
• Time-Consuming: Collecting, transcribing, and analyzing qualitative data can be labor-intensive.
• Not Easily Generalizable: Due to typically smaller sample sizes and the subjective nature of the data, results might not be generalizable to a larger population.
• Lack of Replicability: The dynamic and flexible nature of qualitative research can make it challenging to replicate.
• Expensive: Data gathering is expensive due to high time consumption during data gathering and processing.  

Qualitative Data Collection Methods:

These qualitative methods offer a deep understanding of human behavior, perceptions, and motivations. However, it's important to select the method that aligns best with the research question and objectives.

• Interviews: One-on-one conversation where the researcher asks open-ended questions. Use: Gain deep insight into personal experiences, feelings, and perceptions.

• Focus Groups: A facilitated group discussion used to gather perceptions, opinions, and feedback about a topic. Use: Understand group dynamics, consensus, or varying viewpoints on a topic.

• Observations: Researcher watches and records behaviors and interactions in a natural setting without intervening. Use: Understand behaviors in their natural context.

• Case Studies: In-depth examination of a specific group, or event. Use: Explore complex issues in detail.

• Document or Content Analysis: Analyzing written, visual, or digital content. Use: Understand themes, patterns, or biases in existing content. Sentinel analysis

Qualitative Data Collection Methods common errors and mitigation strategies:

Avoiding these common errors is essential for collecting high-quality qualitative data that offers meaningful and reliable insights.

• Researcher Bias: The researcher may inadvertently allow personal beliefs, expectations, or values to influence data collection, interpretation, or presentation. Mitigation: Employ reflexivity (self-reflection on one's biases and influence) and ensure that multiple researchers analyze data to provide varied perspectives.

• Leading Questions: Questions that steer participants towards a particular answer or perspective. Mitigation: Craft neutral, open-ended questions. Example 1: Isn't it true that you left the office early yesterday? A non-leading version could be: What time did you leave the office yesterday? Example 2: Wouldn't you agree that the new policy is more effective? A non-leading version could be: What are your thoughts on the new policy's effectiveness?

• Overgeneralization: Assuming that findings from a small or specific group are applicable to broader populations. Mitigation: Clearly define the scope and limitations of the research and avoid making sweeping conclusions. Use purposeful sampling strategies to ensure diverse participant inclusion.

• Participant Bias: Participants may provide answers they believe the researcher wants to hear or that present themselves in a favorable light (social desirability bias). Mitigation: Ensure anonymity and emphasize the importance of truthful and candid responses.

• Inadequate Documentation: Failing to properly document observations, interviews, or focus groups, leading to lost or  data or details. Mitigation: Note-taking, audio recordings (with consent) or video recording.

3. Combine both.

A holistic data analysis requires both qualitative and quantitative insights. They provide a comprehensive view – the numbers give the "what" and the qualitative data explains the "why."

Scenario: The product manager at "Trendy Panda" observes a high cart abandonment rate (quantitative data) and wants to find the root causes and solutions for the same.

Quantitative Data:

1. Cart Abandonment Rate: 60%
2. Average Time Spent on Checkout Page: 8 minutes (should ideally be 3-4 minutes)
3. Drop-off Points: 40% users drop off at the payment gateway page.

Qualitative Data:

1. User Feedback: "The checkout process is too lengthy."
2. User Reviews: "I wasn't sure if my payment went through, the site didn't give a confirmation immediately."
3. Focus Group Feedback: Participants felt the checkout process was not intuitive, and they were unsure about the security of their payment details.

Compare both types:

Which data product managers use for their data analysis:

The question of which data is "better" for data analysis—qualitative or quantitative—depends on the research objectives, questions, and the context in which the analysis is being conducted. Generally product managers use

Lets look at what data analysis we will do as product managers:

One of the foundational analyses PMs undertake is descriptive analysis. This type of analysis provides a snapshot of past behaviors and performances. By looking at metrics such as Daily Active Users or Monthly Active Users, PMs can get a clear view of user engagement over time.

For instance, if a PM wants to understand the immediate impact of a new feature launch, they would turn to descriptive statistics to get an overview of user engagement pre and post-launch. For this type we will use Quantitative data, as well as qualitative data can be used too for categorization. example: MAU users with native language Spanish

Predictive analysis is the window to future possibilities. Leveraging algorithms and machine learning models, PMs forecast future outcomes based on historical data. This is particularly valuable when launching a new feature or assessing the potential impact of a change in strategy. For example, PM wanted to estimate the growth in user numbers for the next quarter, predictive analytics would be the tool of choice. For this type of analysis also use quantitative data for filtering

A/B testing, though not strictly a type of analysis, is an invaluable method for PMs. By presenting two different versions of a feature or interface to users, PMs can directly measure and analyze which version performs better in terms of user engagement, satisfaction, or other relevant metrics. Both types of data can be used.

Prescriptive analysis. This approach recommends specific actions to achieve desired results. If the predictive analysis indicates a potential decline in user numbers, prescriptive analysis could suggest strategies, such as targeted marketing campaigns or feature enhancements, to counteract that decline. For this purpose qualitative data is used.

In addition to these understanding the 'why' behind user behaviors is equally crucial. This is where open-ended surveys or user interviews come into play, offering a richer context and deeper understanding of user motivations and pain points.

Overall: Qualitative data used to understand on more early stages of decision making process, when we need to understand "Why?" and "How to do better?". While quantitive data is more useful when product managers need to confirm impact of work or current performance.

‍

‍

‍

‍

‍