UXR lays the foundation for successful product design and development.
As a UX researcher, you’re always looking for ways to improve products and improve user experiences. By collecting insights about users, teams can design products and services that genuinely meet their needs and expectations. It involves understanding user behavior, preferences, and pain points, enabling product managers and designers to develop solutions that solve real problems and enrich the user experience.
It also highlights potential obstacles in the user journey, allowing teams to address these issues proactively. Additionally, it reduces the risk of developing products that users don’t want or need by testing ideas with real users before launch. Don’t just take our word for it, Forbes reported that on average, every dollar invested in user experience brings 100 in return. That’s an ROI of 9,900%.
But how do you know which research method to use? It’s the classic conundrum of user research. We are here to illuminate the difference between two key research methodologies: generative and evaluative.
Generative vs evaluative research methods are two techniques for conducting user research when managing your product. Both UX research methods are different but provide valuable insights to help your product development process.
By the end of this post, you’ll have a clear grasp of the significance of user research in product development, the methodologies at your disposal, and how to wield them effectively to create user-centered products that thrive in the marketplace.
Evaluative vs generative research
Are you the generative researcher or the evaluator of designs?
Dr Asma A QureshiResearch and Insights Lead Consultant at A2 Online
In a nutshell…
Evaluative user research
Typically uses quantitative methods to assess the success or failure of a specific product/feature and identify necessary improvements.
Generative user research
Typically uses qualitative methods to understand user needs and motivations to inspire design and product development innovation.
TL;DR
This table sums up these two methodologies throughout product development, and how to wield them effectively to create user-centered products that thrive in the marketplace.
Evaluative Research | Generative Research | |
|---|---|---|
Purpose | Measure existing design. | Uncover new insights and opportunities related to user needs and behaviors. |
Focus | Assessment and improvement. | Foster innovation and exploration. |
Design Stage | Later stages (post-design completion, iteration). | Early stages (ideation, concept development). |
Methods | Surveys, Tree testing, Usability testing, A/B testing, Closed card sorting | Ethnographic studies, User interviews, Surveys, Focus groups, Open card sorting |
Data Collection | Quantitative and qualitative data on user interactions and experiences with the existing product. | Qualitative data on user behavior, feedback, and needs. |
Outcome | Improved usability and design validation. | New insights, ideas, and opportunities informing design direction. |
But if you like getting into the details, let’s get into the nitty gritty to understand the differences of each approach, when each approach should be used and examples of each methods.
What is generative research?
If you’re asking 'why is this important?' or 'what’s the point?', generative user research can help you find those answers.
Crystal ChinSenior Digital Interaction Designer at RBC
Often dubbed exploratory or discovery research, generative research main goal is to pinpoint a problem statement, frame the relevant issues to be tackled, and gather enough evidence to confidently move forward with your next steps in product development.
Simply put, generative research is your gateway to identifying and defining opportunities that address real human challenges.
Typically, this type of research thrives in unconstrained environments, where little is known about a particular area. The intention? To introduce fresh solutions or innovations that meet customer needs—sometimes even needs they didn’t know they had! Keeping an open mind is essential during generative research as it’s all about exploring possibilities without getting bogged down by existing solutions or what’s already been built.
Generative UX research methods
You can leverage several effective generative research methods. Here are some common approaches:
Contextual inquiry
Sometimes referred to as ethnographic studies. This observes users in their natural environment as they interact with your product or service. This method sheds light on real-world needs and behaviors that might elude traditional lab-based interviews.
User interviews
User interviews facilitate in-depth conversations that yield valuable qualitative insights. When conducting interviews, it’s essential to ask open-ended questions that encourage participants to share their experiences, preferences, and challenges related to the topic at hand.
Open card sorting
Open card sorting helps you understand users’ mental models by revealing how they perceive relationships between different pieces of information and how they organize and prioritize content. This method is invaluable for informing the information architecture of your product or website.
Focus groups
Focus groups consist of a small, carefully selected group of participants who engage in moderated discussions about a specific topic. Typically made up of five to ten people, participants are chosen based on criteria like demographics, interests, or experiences, making these sessions rich in qualitative data and idea generation.
Surveys
Quantitative measures of attitudes gathered through structured questions. Intercept surveys can trigger during user interactions, while others might be distributed via email or social media. A user research CRM, like Ethnio, can help automate this process in order to recruit your ideal participant.
Leveraging generative research to define design problems
Crystal Chin’s, product designer, experience with the Bridge School’s Product Design Program showcases how generative user research stands out as a critical tool for understanding user needs and identifying areas for product development.
Context of the project
Chin's project involved tackling a design challenge aimed at helping Canadians better manage stress through improved work/life balance. The stakeholder discovery workshop was scheduled weeks after the project began, prompting the team to initiate user research promptly to utilize their time effectively.
Generative research was particularly for this case study because:
The project scope was too broad or undefined.
The problem to be solved was unclear.
There was little existing data supporting the project's rationale.
Conducting effective research
Chin's approach can be distilled into four key steps:
She formulated questions focusing on the causes and cues of stress and burnout, aiming to understand user experiences deeply.
Through five user interviews and a survey, Chin gathered diverse perspectives, ensuring a comprehensive understanding of the issues at hand.
Utilizing tools with automations for segments and tags, like Ethnio, she organized and coded over 600 data points into themes, employing affinity mapping to identify relationships and patterns.
This synthesis revealed critical insights about the impacts of stress on personal relationships, coping mechanisms, and the need for better communication, which were backed by user evidence.
Crystal Chin’s project demonstrates that when project scopes are too broad or unclear, generative research provides valuable insights into user experiences, needs, and behaviors. By employing a systematic approach—defining research questions, conducting qualitative research, synthesizing data, and deriving actionable insights—designers can clarify the challenges users face and identify opportunities for impactful solutions. Implement one or two tips, and you’ll be surprised at how your next study will run smoothly.
What is evaluative research?
Unlike generative research, which focuses on exploring new ideas, evaluative research hones in on refining and validating what has already been developed. Evaluative research is employed to assess the effectiveness of existing products, designs, or concepts. Its primary goal is to validate whether a solution meets its intended objectives and to identify areas for improvement.
Many people think evaluative research = usability testing, but it goes much further than that.
Lidia SambitoLead UX Researcher at Hurtigruten Expeditions
Simply put, it is about assessing how a product/service works when placed in front of a user. In addition to functionality, it is also about findability, efficiency, and the emotions associated with using the product/service. Incorporating evaluative research throughout the product development life cycle is key to delivering solutions that are both effective and competitive.
Evaluative UX research methods
You can leverage several effective evaluative research methods. Here are some common approaches:
Surveys
Surveys gather quantitative data on user preferences, satisfaction levels, and attitudes toward a design or product, helping to uncover trends and patterns across a broader user base. This information enables targeted improvements and informed design decisions.
Usability testing
Usability testing involves observing participants as they complete specific tasks with the product or design, collecting qualitative and quantitative insights into user interactions. This method helps identify usability issues and pain points, enhancing the overall user experience.
Closed card sorting
Closed card sorting allows participants to categorize content into predefined groups, providing insights into how users classify information. This understanding aids in refining design elements and evaluating existing structures for improved usability.
A/B testing
A/B testing compares two or more versions of a design or feature to determine which one performs better in achieving a specific objective. This approach informs design decisions based on user preferences and behaviors, optimizing product performance and user satisfaction.
Leveraging usability testing to enhance product performance
Liz Holland, researcher at Answerlab led research for an extensive 2-part, 3-segment research project for Google Sheets. Her case study exemplifies how usability testing, a key method of evaluative research, serves as a critical tool for assessing user interactions and improving product functionality across diverse user segments.
Context of the project
Holland was tasked with evaluating how different user groups—large enterprise users, small to midsize businesses, and personal users—engaged with both mobile and desktop versions of Google Sheets. The Google Sheets team sought insights into user motivations and behaviors to inform their product roadmap effectively, aiming to ensure all versions of the product addressed user needs.
Conducting effective evaluative research
Holland’s approach can be distilled into three key steps:
She engaged participants in usability testing sessions, where they performed tasks on both versions of Google Sheets while screen sharing. This method allowed her to observe user interactions in real time and probe deeply into their experiences.
Through these sessions, Holland focused on uncovering pain points that users encountered. This real-time feedback provided valuable insights into where users struggled and which features were effective or in need of enhancement.
The findings from the usability testing were synthesized into a comprehensive report that included visuals of usability issues and user journey maps. This report prioritized recommendations for feature improvements, directly informing the Google Sheets team’s design decisions and ensuring that subsequent iterations of the product better aligned with user needs.
This proves no matter how big or small your brand, user-centric design can significantly elevate user satisfaction and product performance. This kind of real-time feedback is invaluable for keeping a product aligned with user needs. Implement these strategies, and you may be surprised at the depth of understanding you gain from your next evaluative research study.
Have you heard of hybrid research?
Hybrid research is a combination of generative research and evaluative research. It helps us simultaneously understand our users, as well as how a product/service is performing.
UX research is a strange thing, being divided into generative and evaluative studies, the former conducted ideally early in product development, the latter further along in the process.
Lindsey Wallace, PhDDirector of Design Research and Strategy, Cisco Security Business
Both are essential. If we skip the evaluative research phase, our solutions will lack refinement. If we skip the generative research phase we’ll end up with a great design for a problem that doesn’t exist.
We can also mix generative and evaluative methods. In some cases, generative and evaluative research can be combined to gain a more comprehensive understanding of users and their needs. For example, a generative research study might identify a new user need, while an evaluative research study could assess the effectiveness of a proposed solution to address that need. We can have a two-phase project with interviews first and a survey next. A one-phase project might start with a contextual inquiry and end with a usability test in the same sitting.
The key is to use the right type of research at the right time. Generative research avoids the build trap by ensuring the right problems are solved before development. Evaluative research keeps us honest and grounded, ensuring our execution aligns with user needs.
Scale your research efforts with better tools
In a world where understanding users is more critical than ever, managing user research can often feel overwhelming. Especially with tight timelines and the need for diverse participant pools. A user research CRM, like Ethnio, empowers teams to concentrate on what truly matters: conducting research that leads to impactful design decisions.
By automating the operational side of research, organizations can streamline participant recruitment, schedule sessions, and engage in both generative and evaluative studies without compromising participant privacy or compliance with regulations like GDPR and CCPA.
With the right enterprise solution in place, this not only reduces the manual tasks associated with user research but also democratizes the research process, empowering product managers and designers to conduct ongoing studies confidently.
As we wrap up…
It’s clear both generative and evaluative research are essential methodologies for UX researchers, product designers and People Who Do Research (PWDRs) alike, enabling teams to understand user needs and refine product offerings. They complement each other, providing a comprehensive understanding of users and guiding the creation of successful, user-centric products. Start with generative research to explore unknowns and lean on evaluative research to refine solutions for optimal outcomes.
As you incorporate these approaches into your workflow, consider the value of having tools in place that support your research goals. As this improves the capacity to scale research and gather insights that drive impactful design decisions, and achieve user-centric excellence.