ThoughtWorks
  • Contact
  • Español
  • Português
  • Deutsch
  • 中文
Go to overview
  • Engineering Culture, Delivery Mindset

    Embrace a modern approach to software development and deliver value faster

    Intelligence-Driven Decision Making

    Leverage your data assets to unlock new sources of value

  • Frictionless Operating Model

    Improve your organization's ability to respond to change

    Platform Strategy

    Create adaptable technology platforms that move with your business strategy

  • Experience Design and Product Capability

    Rapidly design, deliver and evolve exceptional products and experiences

    Partnerships

    Leveraging our network of trusted partners to amplify the outcomes we deliver for our clients

Go to overview
  • Automotive
  • Cleantech, Energy and Utilities
  • Financial Services and Insurance
  • Healthcare
  • Media and Publishing
  • Not-for-profit
  • Public Sector
  • Retail and E-commerce
  • Travel and Transport
Go to overview

Featured

  • Technology

    An in-depth exploration of enterprise technology and engineering excellence

  • Business

    Keep up to date with the latest business and industry insights for digital leaders

  • Culture

    The place for career-building content and tips, and our view on social justice and inclusivity

Digital Publications and Tools

  • Technology Radar

    An opinionated guide to technology frontiers

  • Perspectives

    A publication for digital leaders

  • Digital Fluency Model

    A model for prioritizing the digital capabilities needed to navigate uncertainty

  • Decoder

    The business execs' A-Z guide to technology

All Insights

  • Articles

    Expert insights to help your business grow

  • Blogs

    Personal perspectives from ThoughtWorkers around the globe

  • Books

    Explore our extensive library

  • Podcasts

    Captivating conversations on the latest in business and tech

Go to overview
  • Application process

    What to expect as you interview with us

  • Grads and career changers

    Start your tech career on the right foot

  • Search jobs

    Find open positions in your region

  • Stay connected

    Sign up for our monthly newsletter

Go to overview
  • Conferences and Events
  • Diversity and Inclusion
  • News
  • Open Source
  • Our Leaders
  • Social Change
  • Español
  • Português
  • Deutsch
  • 中文
ThoughtWorksMenu
  • Close   ✕
  • What we do
  • Who we work with
  • Insights
  • Careers
  • About
  • Contact
  • Back
  • Close   ✕
  • Go to overview
  • Engineering Culture, Delivery Mindset

    Embrace a modern approach to software development and deliver value faster

  • Experience Design and Product Capability

    Rapidly design, deliver and evolve exceptional products and experiences

  • Frictionless Operating Model

    Improve your organization's ability to respond to change

  • Intelligence-Driven Decision Making

    Leverage your data assets to unlock new sources of value

  • Partnerships

    Leveraging our network of trusted partners to amplify the outcomes we deliver for our clients

  • Platform Strategy

    Create adaptable technology platforms that move with your business strategy

  • Back
  • Close   ✕
  • Go to overview
  • Automotive
  • Cleantech, Energy and Utilities
  • Financial Services and Insurance
  • Healthcare
  • Media and Publishing
  • Not-for-profit
  • Public Sector
  • Retail and E-commerce
  • Travel and Transport
  • Back
  • Close   ✕
  • Go to overview
  • Featured

  • Technology

    An in-depth exploration of enterprise technology and engineering excellence

  • Business

    Keep up to date with the latest business and industry insights for digital leaders

  • Culture

    The place for career-building content and tips, and our view on social justice and inclusivity

  • Digital Publications and Tools

  • Technology Radar

    An opinionated guide to technology frontiers

  • Perspectives

    A publication for digital leaders

  • Digital Fluency Model

    A model for prioritizing the digital capabilities needed to navigate uncertainty

  • Decoder

    The business execs' A-Z guide to technology

  • All Insights

  • Articles

    Expert insights to help your business grow

  • Blogs

    Personal perspectives from ThoughtWorkers around the globe

  • Books

    Explore our extensive library

  • Podcasts

    Captivating conversations on the latest in business and tech

  • Back
  • Close   ✕
  • Go to overview
  • Application process

    What to expect as you interview with us

  • Grads and career changers

    Start your tech career on the right foot

  • Search jobs

    Find open positions in your region

  • Stay connected

    Sign up for our monthly newsletter

  • Back
  • Close   ✕
  • Go to overview
  • Conferences and Events
  • Diversity and Inclusion
  • News
  • Open Source
  • Our Leaders
  • Social Change
Blogs
Select a topic
View all topicsClose
Technology 
Agile Project Management Cloud Continuous Delivery  Data Science & Engineering Defending the Free Internet Evolutionary Architecture Experience Design IoT Languages, Tools & Frameworks Legacy Modernization Machine Learning & Artificial Intelligence Microservices Platforms Security Software Testing Technology Strategy 
Business 
Financial Services Global Health Innovation Retail  Transformation 
Careers 
Career Hacks Diversity & Inclusion Social Change 
Blogs

Topics

Choose a topic
  • Technology
    Technology
  • Technology Overview
  • Agile Project Management
  • Cloud
  • Continuous Delivery
  • Data Science & Engineering
  • Defending the Free Internet
  • Evolutionary Architecture
  • Experience Design
  • IoT
  • Languages, Tools & Frameworks
  • Legacy Modernization
  • Machine Learning & Artificial Intelligence
  • Microservices
  • Platforms
  • Security
  • Software Testing
  • Technology Strategy
  • Business
    Business
  • Business Overview
  • Financial Services
  • Global Health
  • Innovation
  • Retail
  • Transformation
  • Careers
    Careers
  • Careers Overview
  • Career Hacks
  • Diversity & Inclusion
  • Social Change
SecuritySocial ChangeDiversity & InclusionTechnologyCareers

Big Data Reality Causes Privacy Concerns

Mike Mason Mike Mason

Published: Feb 11, 2014

Twice a year, ThoughtWorks publishes the “Technology Radar”—our view on the technology trends that are important in the industry right now, and the trends that will be important in the near future.

It’s a unique perspective from ThoughtWorks and our 2,500 consultants around the world, based on first-hand experiences delivering real software for our clients. Third parties cannot pay to have themselves featured on the Radar and the report is entirely independent in which technologies we include and what we say about them. The latest edition of the Radar was published last week.

One of the large themes we have been tracking over the past couple of years is around Big Data and Analytics. We think the “big” part of Big Data is over-hyped; most of the time you don’t actually need a massive cluster of machines to process your data. But the sheer variety, or “messiness” of all of this data presents new challenges, and there’s a real opportunity to use Advanced Analytics—statistical modeling, machine learning and so on—to gain new insight into your business and into customer behavior. An important trend we note in the Radar is the accessibility of all of these new Analytics techniques. If you do truly have lots of data you can simply go rent a portion of the cloud to process it, with SaaS offerings from Amazon, Google, Rackspace and others. If you want to analyze your data you can do it with point-and-click tools or open-source offerings such as the amazing D3.js JavaScript library.[1] Open-source is a huge democratizing factor here—you no longer need to pay for an expensive “big iron” solution for data processing and analysis.

We’re excited about the increased awareness around data because software systems can use data and analytics to provide significantly better end-user experiences, as well as delivering increased value to businesses. As has already happened with unit-testing, we expect it to become every developer’s job to understand the importance of data and what can be done with it. That’s not to say every developer needs a statistics degree or a PhD, but we’re expecting data engineering and analysis to become a bread-and-butter part of a developer’s job rather than some weird thing “those data science people” do in a corner.

While there’s much to be gained from better retention, analysis and understanding of data, it comes with a darker side. Companies employing advanced analytics have quickly realized that they need to avoid being too accurate with their insights or people feel unnerved, even violated. One way to avoid spooking people is to deliberately include less-relevant offerings and advertisements to a customer, so they don’t feel targeted. The strategy is to get right up to the “spookiness” line but not to cross it.

As we’ve seen over the past few months, any digital trail can potentially be considered an indelible record. Responsible organizations need to look at these revelations, as well as the weekly news of private-sector security breaches, and consider their response. In Europe, many companies are adopting a strategy of Datensparsamkeit[2], a term that roughly translates as “data austerity” or “data parsimony.” The method originates in Germany where data privacy laws are significantly stricter than in the US. Rather than taking an approach of storing and logging every possible scrap of information about a customer and their interactions, Datensparsamkeit advocates only storing the data you absolutely need in order to provide your service to that customer. This way their privacy is maintained even in the unfortunate event of a data breach.

Society is increasingly driven by technology, and changing at an ever increasing pace. As technologists it’s our responsibility not just to consider what we can do with our new tools, but whether it’s the right thing to do. Ethics are not the sole purview of philosophers, lawyers and politicians: we must all do our part.

This blog was originally posted on CitzenTekk.

[1] http://d3js.org/

[2] http://martinfowler.com/bliki/Datensparsamkeit.html

  • What we do
  • Who we work with
  • Insights
  • Careers
  • About
  • Contact

WeChat

×
QR code to ThoughtWorks China WeChat subscription account

Media and analyst relations | Privacy policy | Modern Slavery statement ThoughtWorks| Accessibility | © 2021 ThoughtWorks, Inc.