Master
ThoughtWorks
菜单
关闭
  • 产品及服务
    • 概况
    • "客户体验、产品及设计业务线 "
    • 数据战略、工程及分析业务线
    • 数字化转型及运营业务线
    • 现代化企业、平台及云业务线
  • 合作伙伴
    • 概况
    • 汽车企业
    • 医疗企业
    • 公共服务机构
    • 清洁技术,能源与公用事业
    • 媒体和出版业
    • 零售业和电商
    • 金融和保险企业
    • 非盈利性组织
    • 旅游业和运输业
  • 洞见
    • 概况
    • 特色

      • 技术

        深入探索企业技术与卓越工程管理

      • 商业

        及时了解数字领导者的最新业务和行业见解

      • 文化

        分享职业发展心得,以及我们对社会公正和包容性的见解

    • 数字出版物和工具

      • 技术雷达

        对前沿技术提供意见和指引

      • 视野

        服务数字读者的出版物

      • 数字化流畅度模型

        可以将应对不确定性所需的数字能力进行优先级划分的模型

      • 解码器

        业务主管的A-Z技术指南

    • 所有洞见

      • 文章

        助力商业的专业洞见

      • 博客

        ThoughtWorks 全球员工的洞见及观点

      • 书籍

        浏览更多我们的书籍

      • 播客

        分析商业和技术最新趋势的精彩对话

  • 加入我们
    • 概况
    • 申请流程

      面试准备

    • 毕业生和变换职业者

      正确开启技术生涯

    • 搜索工作

      在您所在的区域寻找正在招聘的岗位

    • 保持联系

      订阅我们的月度新闻简报

  • 关于我们
    • 概况
    • 我们的宗旨
    • 奖项与荣誉
    • 多元与包容
    • 领导层
    • 合作伙伴
    • 辅助功能
    • 新闻
  • 联系我们
China | 中文
  • United States United States
    English
  • China China
    中文 | English
  • India India
    English
  • Canada Canada
    English
  • Singapore Singapore
    English
  • United Kingdom United Kingdom
    English
  • Australia Australia
    English
  • Germany Germany
    English | Deutsch
  • Brazil Brazil
    English | Português
  • Spain Spain
    English | Español
  • Global Global
    English
博客
选择主题
查看所有话题关闭
技术 
敏捷项目管理 云 持续交付 数据科学与工程 捍卫网络自由 演进式架构 体验设计 物联网 语言、工具与框架 遗留资产现代化 Machine Learning & Artificial Intelligence 微服务 平台 安全 软件测试 技术策略 
商业 
金融服务 全球医疗 创新 零售行业 转型 
招聘 
职业心得 多元与融合 社会改变 
博客

话题

选择主题
  • 技术
    技术
  • 技术 概观
  • 敏捷项目管理
  • 云
  • 持续交付
  • 数据科学与工程
  • 捍卫网络自由
  • 演进式架构
  • 体验设计
  • 物联网
  • 语言、工具与框架
  • 遗留资产现代化
  • Machine Learning & Artificial Intelligence
  • 微服务
  • 平台
  • 安全
  • 软件测试
  • 技术策略
  • 商业
    商业
  • 商业 概观
  • 金融服务
  • 全球医疗
  • 创新
  • 零售行业
  • 转型
  • 招聘
    招聘
  • 招聘 概观
  • 职业心得
  • 多元与融合
  • 社会改变
体验设计创新技术商业

User Research Will Destroy Your Product

Ted Nielsen Ted Nielsen

Published: May 25, 2016

Properly used, user research will destroy your product. This is a feature, not a bug.

You have an idea, maybe even a great one. You’ve done your design sprint, your visioning session, your intense few days of white-boarding. You’ve built your draft roadmap, have a few personas and a well-worn copy of the Lean Startup. No plan, though, survives its first contact with the enemy.

Lean methodologies praise the power and impact of user research without actually talking through how to use it. Product managers, particularly in the enterprise, have piles of competitive, quantitative outputs, and analytics reports. Startups have budgets to hand out gift cards to participants in coffee shops, and both think they’re doing it right.

Talking to your customer is hardly a novel idea. Keep in mind, mold is over a billion years old. Penicillin was only released in the 20th century. Just because something exists doesn’t mean it’s useful to you, especially if you’re not using it the right way.

In that vein, get ready for a bitter pill.

The Lies We Tell Ourselves

In too many process implementations, 'value' is a convenient fiction. Agile and Lean teach us to drive to value and to frame everything in the context of the user’s need. This results in backlogs full of stories that start with the dubious 'as a system' and with UI tweaks. Noble ideals have a way of leading us astray if we don’t think critically about them.

Our understanding of value in this context is anemic and cheap.
Tweet this

To reclaim the word, a product is a promise of shared value: lasting commercial and competitive benefit for the provider and ongoing unique fulfillment for the consumer. Like a handshake, it only achieves its purpose in the moment that the two collide.

I’ve already talked about the importance of 'unique' in that definition: Markets crowded with me-too solutions are reduced to bowls of oatmeal — homogenized, boring, and different only in incidentals.

Value, though, is how we crack open 'fulfillment' in the above definition.

Risk and Reward

Pirate metrics (Acquisition - Activation - Retention - Revenue - Referral) were popularized by Dave McClure of 500 Startups as a funnel to describe an engagement model for B2C companies. While each stage is a fundamental change in the nature of the relationship of the user to the product, I want to focus on the middle, Retention. Stickiness and the realization of value in the product lives there.

Hypothesized value is where you test, and the model for measuring the return of user research is iterations of Risk and Reward. Your risk is composed of two costs: the cost of staging the test (whether as ethnographic observation, qualitative interviews, panels, sending out a survey, setting up the clickstream analytics, etc.) and the cost of being wrong (your sunk cost in what you’re testing). Reward is pretty simple; it’s the benefit of being right.

The ratio between risk and reward is not stable over the course of taking a product from concept to customer. Like a functional test smoke suite, the first few tests are highly valuable: Do people need this? Does this product create fulfillment? Have I made what makes my company different and amazing a critical part of this product and how it works?

Over time, your focus narrows as you pass those broader, qualitative tests: Does this feature work? Can a different design or interaction paradigm improve the performance of this KPI? As you move from 'should' to 'how', your questions get smaller, and so your reward gets smaller too. Like all iterative testing frameworks, your process suffers from diminishing returns.

The relationship of risk and reward over the lifecycle of a product is therefore described logarithmically — rapid advancement and value from the early stages of the investigation to stable and shrinking return in the later stages. Because of this, both the beginning and the end of the journey encourage the most dangerous behavior.

Big Bang

The first tests are unquestionably the riskiest; the fate of your idea hangs in the balance. You use them to establish the fitness for need for the idea. These tests establish the boundaries within which the fitness for purpose for the product’s features and capabilities live and develop. Your first tests are the gladiatorial arena for your idea.

In organizations without a core testing principle, the risk borne by the first tests balloons the longer they don’t vet their ideas. This is never worse than when they commit the cardinal sin of a Big Bang release. The sin of the Big Bang is either fueled by fear of being wrong (so why wouldn’t you stop earlier?) or a case of mistaken identity: the idea has been conflated with its implementation.

There’s an inherent amount of risk in the first tests—don’t make it worse by expanding that risk beyond the potential reward value. You end up shouldering higher premiums on that risk for the same information that other organizations are getting on the cheap.

The Narrows

In a psychology paper published last year, Wagenmakers et al. said, “A single experiment cannot overturn a large body of work. […] A strong anvil need not fear the hammer[.]” A healthy research capability develops the 'strong anvil' of your product – you can develop and evolve your product with confidence, and validate your instincts against customer feedback and behavioral analysis. With an established track record of tests and successful reactions to the test results, no single test or line of inquiry is going to break the world.

But as your line of inquiry narrows from idea to implementation, and from implementation to options within an implementation, your potential benefit faces dramatically reduced returns on the costs of the research. There are only so many 'red vs. blue buttons' and 'tabs or hamburger menu' tests that you can put into the field.

There is a minimum cost for any research initiative, whether the developer time to implement new tags for your analytics platform, the time to write a script to walk a customer through the application flow, or the cost of that Starbucks gift card to thank your customer for their time and consideration—you’re paying someone something for that information. You must ensure that the information you’re getting is worth what you’re paying for it.

The Narrows, as the risk-reward loop of research becomes neutral and smaller, are the more dangerous location on our curve – it’s unusual for digital organizations never to have heard of user testing and willfully do big-bang releases (accidentally is another matter entirely). Overusing testing and falling prey to the narrows is all too common, though.

The danger of the Narrows is simple: If you’re spending all your time and money in your research budget fine-tuning a finely tuned instrument, your competitors are already looking for the next opportunity.

The Narrows are just another kind of stagnation, and it’s a sure sign that a major pivot or breakthrough capability set is needed. When dealing with the chilling effects of a stagnated roadmap, a product manager should purposefully introduce risk as a way to unlock potential new reward. Escaping the Narrows requires a step change to leap out of it: modifying your product to break into adjacent markets or evolving from one product into a platform play to leverage network effects.

Your competitors are hungry and trying to out-innovate you.

First Contact

No matter how clever or well-conceived the roadmap, every user test I’ve been a part of has produced insightful, interesting, and actionable data. The produced product always was different from the planned product—and better too. That plan didn’t survive first contact with the user, and the user research—happily—destroyed my product.

The usefulness of any given test is measured against its subject, scope, and the outcomes of all the prior tests.

This means that the user testing framework for any product incurs diminishing returns over its lifecycle. While modeling this return as iterations of risk-reward, there are two dangerous periods in the testing harness: the earliest and the last ones.

Early tests run the risk of having uneven distribution of risk, especially considering more sophisticated solutions that have advanced too far into their implementation. Big bang releases, surprise launches, and other isolationist tactics bear significantly higher risk premiums than collaborative and iterative investigations advocated by standard Lean practices.

Later tests, with narrower scope, run the risk of stagnating the product’s advancement of value by confusing activity (staging tests and analyzing results) with progress (incremental or dramatic production of value). The narrowing risk-reward cycles mean a step-change (pivot, launch, or new market) should be introduced to drive the product to higher levels of lasting value.

Lean on research smartly—answer the big questions for your stakeholders and checkbook-holders (should we build this thing at all?). Connect with your customer in ways that ensure they benefit from all the hard work you and your teams have done (be the penicillin, not the mold). Understand when you have a stagnating product and push it out of the nest; find a new market, a new segment, or re-imagine your role in your customer’s life.

Properly used, user research will break your product. It also will show you the way to pick up the pieces.
Tweet this

相关博客
体验设计

Business, Design and Technology: Joining Forces for a Truly Competitive Advantage

Ben Melbourne
了解更多
创新

Practical Product Innovation

Ben Melbourne
了解更多
体验设计

Conducting User Testing Remotely

Suganth Chellamuthu
了解更多
Master
政策声明 | 现代奴役声明 | 辅助功能
Connect with us
×

WeChat

QR code to ThoughtWorks China WeChat subscription account
© 2021 ThoughtWorks, Inc.