The UX Research reckoning is here | Judd Antin (Airbnb, Meta)

Judd’s background

• Judd helped build the user research practice at Facebook.
• He was a longtime head of research at Airbnb and his direct reports have gone on to lead research teams at various companies.
• Judd now spends his time consulting and helping companies with various organizational challenges.

Critiques and responses to Judd’s post “The UX Research Reckoning Is Here”

• Judd's post titled "The User Research Reckoning is Here" caused a stir in the research community and adjacent communities.
• The post sparked both positive and critical responses.
• Some critiques focused on the notion that researchers are at fault, not taking responsibility as a research leader, and the idea of being more profit-focused.

The state of user research

• Many lay-offs in user research teams have been observed, leading to a moment to take stock.
• Judd believes that the system is more broken than perceived, and research is not driving the value or impact it should.
• He suggests that the focus on the wrong type of research is a root issue.

Macro, middle-range, and micro research (00:08:53)

• Macro research focuses on big-picture, strategic, and business-focused aspects such as market analysis, long-term product direction, and innovation.
• Micro research includes technical usability testing and focused research on understanding specific aspects such as AB test results.
• Middle-range research deals with intermediate-level user understanding questions but may lack direct impact on the business strategy or goals.

What teams get wrong when it comes to research (00:14:05)

• Research can be less impactful when treated as a service function and brought in at the end of the process.
• Lack of consistent integration of researchers throughout the product development process can lead to suboptimal impact.

The importance of integrating research from the beginning (00:15:46)

• Integrating research from the beginning and maintaining consistent engagement throughout the process can lead to more impactful insights and drive business improvement.

Traits of great researchers (00:17:30)

• Great researchers possess multi-method skills, including formative and generative research, usability testing, survey design, applied statistics, and technical skills such as prompt engineering.

Advice for evaluating user researchers

• Look for a researcher with a diverse set of skills and methods
• A good researcher provides multi-method answers to open-ended questions
• Building a team with researchers who have deep expertise in different areas is important

Balancing business and product focus

• Researchers need to be more business-oriented and consider the overlap between user needs and business objectives
• Researcher advice includes understanding company strategy, OKRs, metrics, and the conversion funnel
• This shift in focus requires collaboration and changes in organizational structure

User-centered performance

• User-centered performance involves symbolic actions that signal customer-centricity without driving better decisions or more impact
• Examples include last-minute user studies to validate assumptions and performative focus groups not focused on learning
• Cognitive biases and ego can lead to user-centered performance and hinder impactful research

The role of intuition in product development

• Intuition is important in product development but can be influenced by biases and blind spots
• Great researchers expand horizons and challenge intuition with evidence, improving product decisions
• Designers and PMs also have performance issues related to user-centricity and intuition

Ultimately, the key is to trust intuition developed from experience but remain open to the limitations and biases, leveraging research to balance and inform product decisions.

Checking your gut instincts

• Engage in slow, methodical thinking to check your gut instincts
• Utilize the wisdom of the crowd by involving diverse sources of information and judgment
• Research teams should have the same set of metrics for success as other team members
• Researchers should aim to be an active participant in the decision-making process

Common tropes about PMs, from researchers

• The relationship between product managers and user researchers is often challenged
• Tropes that researchers have about PMs include the perception that research slows down the process, the belief that researchers are not needed, and an overreliance on AB testing without understanding the underlying reasons for changes

Research timeframes and value

• Research can be fast and yield significant business value
• Micro-level research can drive a huge amount of business value in a short amount of time
• Middle-range research can sometimes slow down processes and needs to be more pointed and valuable

The role of researchers

• Researchers have the expertise to turn low-quality input into valuable insights
• Researchers can provide evidence to address the reasons behind changes shown in AB tests

A/B testing vs. user research

• A/B testing provides conclusive and statistical data
• User research emphasizes understanding how and why things happen
• Both A/B testing and user research are valuable and should work together in partnerships
• Misquoting historical figures or relying on customer opinions can be misleading in research

Hindsight bias and narrative fallacy

• People often have hindsight bias, believing they already knew the outcome
• The narrative fallacy involves constructing simple and convenient stories about the past, which may not be entirely true

Making recommendations based on research

• Research can help build a strong mental model of the user and domain
• Good research doesn't just identify problems but also presents a framework based on evidence for the team to develop a successful product

Advice for teams on how to leverage researchers

• Researchers need to develop diverse research skills and strong business knowledge
• Creating strong relationships and focusing on important projects is crucial for researchers
• Companies should integrate various insights disciplines and create an integrated insights function
• Researchers should be involved in a unified, lean process from beginning to end to extract maximum value and avoid service-oriented roles

How product managers can be better partners to user researchers (00:51:18)

• Product managers should integrate user researchers and insights from beginning to end of the product process.
• They need to partner with researchers on prioritization and participate in user research activities.
• Great product managers take the time to go into the field with researchers, even to travel for research.

Insight from user research at Facebook (00:51:18)

• A story from Facebook reveals the importance of experiencing user research first-hand to gain valuable insights.
• The story emphasizes the need for researchers and product teams to burst their own bubbles and avoid naïve assumptions.

Understanding user behavior (00:51:18)

• The story illustrates the significance of individual user insights in challenging assumptions and developing better solutions.
• Obtaining insights from user behavior can be crucial, even without achieving statistical significance.

The impact of privacy concerns on user behavior (00:51:18)

• The Paris study revealed that hosts were reluctant to connect with guests through Facebook due to privacy concerns.
• The study showed that users were apprehensive about giving Facebook any data, pointing towards privacy-related behavioral impacts.