Master
ThoughtWorks
Menu
Close
  • What we do
    • Go to overview
    • Customer Experience, Product and Design
    • Data Strategy, Engineering and Analytics
    • Digital Transformation and Operations
    • Enterprise Modernization, Platforms and Cloud
  • Who we work with
    • Go to overview
    • Automotive
    • Healthcare
    • Public Sector
    • Cleantech, Energy and Utilities
    • Media and Publishing
    • Retail and E-commerce
    • Financial Services and Insurance
    • Not-for-profit
    • Travel and Transport
  • Insights
    • Go to overview
    • Featured

      • Technology

        An in-depth exploration of enterprise technology and engineering excellence

      • Business

        Keep up to date with the latest business and industry insights for digital leaders

      • Culture

        The place for career-building content and tips, and our view on social justice and inclusivity

    • Digital Publications and Tools

      • Technology Radar

        An opinionated guide to technology frontiers

      • Perspectives

        A publication for digital leaders

      • Digital Fluency Model

        A model for prioritizing the digital capabilities needed to navigate uncertainty

      • Decoder

        The business execs' A-Z guide to technology

    • All Insights

      • Articles

        Expert insights to help your business grow

      • Blogs

        Personal perspectives from ThoughtWorkers around the globe

      • Books

        Explore our extensive library

      • Podcasts

        Captivating conversations on the latest in business and tech

  • Careers
    • Go to overview
    • Application process

      What to expect as you interview with us

    • Grads and career changers

      Start your tech career on the right foot

    • Search jobs

      Find open positions in your region

    • Stay connected

      Sign up for our monthly newsletter

  • About
    • Go to overview
    • Our Purpose
    • Awards and Recognition
    • Diversity and Inclusion
    • Our Leaders
    • Partnerships
    • News
    • Conferences and Events
  • Contact
Global | English
  • United States United States
    English
  • China China
    中文 | English
  • India India
    English
  • Canada Canada
    English
  • Singapore Singapore
    English
  • United Kingdom United Kingdom
    English
  • Australia Australia
    English
  • Germany Germany
    English | Deutsch
  • Brazil Brazil
    English | Português
  • Spain Spain
    English | Español
  • Global Global
    English
Blogs
Select a topic
View all topicsClose
Technology 
Agile Project Management Cloud Continuous Delivery  Data Science & Engineering Defending the Free Internet Evolutionary Architecture Experience Design IoT Languages, Tools & Frameworks Legacy Modernization Machine Learning & Artificial Intelligence Microservices Platforms Security Software Testing Technology Strategy 
Business 
Financial Services Global Health Innovation Retail  Transformation 
Careers 
Career Hacks Diversity & Inclusion Social Change 
Blogs

Topics

Choose a topic
  • Technology
    Technology
  • Technology Overview
  • Agile Project Management
  • Cloud
  • Continuous Delivery
  • Data Science & Engineering
  • Defending the Free Internet
  • Evolutionary Architecture
  • Experience Design
  • IoT
  • Languages, Tools & Frameworks
  • Legacy Modernization
  • Machine Learning & Artificial Intelligence
  • Microservices
  • Platforms
  • Security
  • Software Testing
  • Technology Strategy
  • Business
    Business
  • Business Overview
  • Financial Services
  • Global Health
  • Innovation
  • Retail
  • Transformation
  • Careers
    Careers
  • Careers Overview
  • Career Hacks
  • Diversity & Inclusion
  • Social Change
Data Science & EngineeringTechnology

Agility, Big Data and Analytics

Ken Collier Ken Collier

Published: Dec 16, 2013

Agile and Lean techniques seem to be the best way we currently know to create complex software in the face of risk, uncertainty, and changing requirements. Agile hinges on embracing and adapting to change by enabling rapid feedback cycles and evolutionary development. However, bringing agility into big data (and small data) analytics has been a challenge for many, very bright and talented, data scientists and engineers. In this article we’ll explore what makes analytics uniquely different than application development, and how to adapt agile principles and practices to the nuances of analytics. We’ll also examine how the disciplines of data science and software development complement one another, and how these intersect in an agile project environment.

The data scientist, the software developer, and the data engineer

First let’s look at what differentiates analytics experts from software developers. C.F. Jeff Wu first introduced the term “data science” in 1998 as a discipline that encompasses statistical analysis, science, and advanced computing. The use of analytics by social media companies like LinkedIn, Facebook, and others in recent years has boosted the popularity of “data scientist” such that Harvard Business Review published an October 2012 article entitled “Data Scientist: The Sexiest Job Title of the 21st Century.” Simply put, a data scientist has a unique, and very deep, blend of the skills depicted in Figure 1.

Figure 1: The Disciplines of Data Science, Source: Calvin Andrus, Wikipedia

Data science skills are both complementary to, and overlapping with, software development skills. Data science requires programming, but data scientists are not often trained in modern software engineering practices. Conversely, many developers have skills in data engineering, advanced computing, and statistics, but these are not commonly their areas of deep expertise. Data scientists commonly code in multi-paradigm languages like R and Python, which have powerful statistics libraries and an active research community behind them.

Data engineering is the bridge between data science and software development. A data engineer supports the data scientist in data discovery, harvesting, and preparation. Data engineers support developers in operationalizing analytical models for production deployment, which we will discuss shortly. This role requires expertise in data management technologies (“big data”, NoSQL, and SQL), data modeling, data architectures, and data manipulation languages and techniques. 

Read the rest of Agility, Big Data and Analytics on InfoQ.

Master
Privacy policy | Modern Slavery statement | Accessibility
Connect with us
×

WeChat

QR code to ThoughtWorks China WeChat subscription account
© 2021 ThoughtWorks, Inc.