Agile engineering practices
Pairing, Are You Doing it Wrong?
The Thoughtworks Technology Radar identifies a number of themes in each edition, but tends to be focused on fairly specific recommendations. But the discussions we have as we decide on those recommendations typically hint at larger things happening in the tech industry. So here, I’ll expand on that bigger picture.
Those of us in the software industry are well placed to deal with working from home, but are not immune to the realities of closed schools, unavailable childcare and general pandemic-induced stresses. It’s especially important to be tolerant of individual circumstances as not everyone is lucky enough to have a well-appointed home office or children who are able to take care of themselves.
In the first few weeks there was a scramble for basic infrastructure, access and tooling but the basics are all in place now. Remote collaboration is moving from “can we even do this?” to “what is the right way to do this?” and there’s a lot to be learned here. Many people are realizing that being on video all day is draining, and that translating in-person collaboration techniques directly into the digital world isn’t very effective. Instead, we all must think carefully about when teams should be online, when asynchronous collaboration can be used instead, and which digital facilitation tools work best for our context. We produced the Radar remotely due to the pandemic and learnt a lot in the process; if you’re interested in learning more we recorded a detailed podcast on our experiences.
Thoughtworks is a strong advocate of pair programming, and we’ve blipped “pragmatic remote pairing” in this edition of the Radar.
The pandemic has shown every CIO the cracks in his or her digital infrastructure, whether that’s resiliency and scaling as customers moved online, or the ability to rapidly make changes, add features and get them into production. Most organizations have realized the inconvenient truth: they’re way behind the state of the art and are struggling to do “digital transformation” (whatever that means) to catch up.
As a consultancy, we have a fairly broad view across the industry and some of the changes have been quite positive. Clients that previously said “you must work in the office on our hardware” have relaxed their stance. In-person time has been replaced by virtual face-to-face time. In some cases this is a massive benefit: our clients can tap into a global network of experts and it doesn’t matter where that person is located (modulo time zones, of course). What’s interesting is that although in theory our clients could have accessed these people pre-pandemic, there’s much less of a barrier to doing so right now. There’s zero expectation someone will show up in person, so there’s no reluctance on our part to propose a non-local expert to help a client. I’m hopeful that across the industry we can all do more remote work in future rather than getting on airplanes all the time.
Prof. Tom Malone (MIT) believes that the current crisis has accelerated us forwards a decade in terms of acceptance of remote working, and that there is no going back. Downtown office space will become less valuable as people decide they’d rather work from home or in a small neighborhood office than drive an hour into work. Tech companies expect to have a large proportion of their staff working remotely even after the crisis, maybe even 70% of staff on a permanent basis.
Tech conferences have also gone fully virtual, and are innovating in really interesting ways. One example is “Deserted Island DevOps” a one-day conference hosted entirely inside the game Animal Crossing New Horizons. Presenters and attendees used in-game avatars to visit the conference islands and were able to create some of the smaller-group interactions that we value so much from physical conferences.
One positive aspect of everyone staying home and industry being shuttered is that we have, albeit temporarily, reduced pollution. Some estimates say carbon emissions in 2020 could be reduced by up to 8%. Renewables continue to increase their market share, for the first time surpassing coal for US power generation. We’re driving less, with cities seeing improved traffic flows and reduced accidents. The state of California has halved traffic accidents, saving it $40 million per day or roughly $1 billion since its shelter-in-place order began. It’s possible that the collectivism and trust in science required to fight an invisible virus may help us do more to tackle climate change in future, too.
While the human and economic costs of the pandemic continue to be very serious, I’m hopeful that the tech industry can use it as an opportunity to accelerate, reduce carbon heavy travel, and increase inclusivity and innovation.
A good example of this is “infrastructure as code.” This is quite old advice, originally added to the Radar in 2011. Our colleague Kief Morris published a detailed book on the topic back in 2016, with an updated copy due to be released later this year. Today, we’re seeing many organizations get into trouble with their cloud migrations in part because they aren’t applying enough rigor to cloud infrastructure definitions, management and automation. So rather than putting “hand crafted cloud templates” or “failing to version control cloud infrastructure definitions” on Hold, we’ve taken the opportunity to refresh and re-highlight our advice on the matter. Similarly, we have reblipped and refreshed pipelines-as-code.
Much of this arises because one of the major storylines of the past decade has been the rise of software, the rise of cloud and the rise of “as a service.” All of these enable us to use easily malleable software, or descriptions of a thing, rather than a hard-coded or physical instantiation which is slower to change. As a result, all of the important practices we’ve learnt about how to create software well — modularity, version control, automated testing — can and should be applied in these wider “software powered” contexts.
Firstly, there tend to be what I would call “highly optimistic” cost savings and ROI projections for cloud migration. Yes, owning and operating a data centre is expensive, but cloud hosting isn’t cheap either. It’s unlikely you’ll really save anything on staffing costs. And when the limit on your usage is how fast teams can click the “create an instance” button, rather than IT provisioning a server, your overall environment sprawl can become a massive, costly problem.
Secondly, “lift and shift” is often used as a deliberate migration strategy, rather than a derogatory comment that someone isn’t thinking carefully about their migration. Enterprise IT estates have many hundreds, and usually thousands, of individual systems that need to run somewhere. Re-engineering all of them to properly understand and operate in a cloud environment is obviously cost prohibitive. But cloud isn’t like on-prem — servers fail, workloads are constantly moved around. Getting a traditional workload to run reliably in the cloud usually isn’t a zero-cost venture either. The reality is that the best answer for an organization is highly complex and varies for each application or group of applications, and the more we can re-engineer for cloud, the more we can improve reliability and reduce costs.
Third, and very related, is that there is still much to learn about the right way to build and run systems in the cloud. Many organizations are discovering that “being in the cloud” hasn’t helped them cope with the surge in demand from the pandemic and associated lockdowns and hasn’t helped them release new features faster. Cloud architecture is a difficult new field of expertise and teams need to learn how to deploy, evolve, maintain and improve their cloud-based systems. This is something that takes time and much more thought than “just throw the stuff on Amazon.”
Today’s most meaning-free term is probably “digital transformation” which, depending on who’s using it, can mean any one of the following:
I hope you’ve enjoyed this round up of trends within the tech industry, and that you are staying safe and healthy. I’d like to thank Andy Yates, Fausto de la Torres, Gareth Morgan, Kief Morris and Rebecca Parsons for their suggestions on drafts of this article
Remote-everything, for the foreseeable future
Like me, you’re probably exhausted reading overwrought pandemic-related commentary. But the implications of COVID-19 for the tech industry are large and things won’t ever return to the way they were before this crisis. Here are what I consider the most important facts and likely changes.Those of us in the software industry are well placed to deal with working from home, but are not immune to the realities of closed schools, unavailable childcare and general pandemic-induced stresses. It’s especially important to be tolerant of individual circumstances as not everyone is lucky enough to have a well-appointed home office or children who are able to take care of themselves.
In the first few weeks there was a scramble for basic infrastructure, access and tooling but the basics are all in place now. Remote collaboration is moving from “can we even do this?” to “what is the right way to do this?” and there’s a lot to be learned here. Many people are realizing that being on video all day is draining, and that translating in-person collaboration techniques directly into the digital world isn’t very effective. Instead, we all must think carefully about when teams should be online, when asynchronous collaboration can be used instead, and which digital facilitation tools work best for our context. We produced the Radar remotely due to the pandemic and learnt a lot in the process; if you’re interested in learning more we recorded a detailed podcast on our experiences.
Thoughtworks is a strong advocate of pair programming, and we’ve blipped “pragmatic remote pairing” in this edition of the Radar.
In short, pairing can absolutely be done in a remote-only world but you have to be sensible about tools, bandwidth and time zones.
The pandemic has shown every CIO the cracks in his or her digital infrastructure, whether that’s resiliency and scaling as customers moved online, or the ability to rapidly make changes, add features and get them into production. Most organizations have realized the inconvenient truth: they’re way behind the state of the art and are struggling to do “digital transformation” (whatever that means) to catch up.
As a consultancy, we have a fairly broad view across the industry and some of the changes have been quite positive. Clients that previously said “you must work in the office on our hardware” have relaxed their stance. In-person time has been replaced by virtual face-to-face time. In some cases this is a massive benefit: our clients can tap into a global network of experts and it doesn’t matter where that person is located (modulo time zones, of course). What’s interesting is that although in theory our clients could have accessed these people pre-pandemic, there’s much less of a barrier to doing so right now. There’s zero expectation someone will show up in person, so there’s no reluctance on our part to propose a non-local expert to help a client. I’m hopeful that across the industry we can all do more remote work in future rather than getting on airplanes all the time.
Prof. Tom Malone (MIT) believes that the current crisis has accelerated us forwards a decade in terms of acceptance of remote working, and that there is no going back. Downtown office space will become less valuable as people decide they’d rather work from home or in a small neighborhood office than drive an hour into work. Tech companies expect to have a large proportion of their staff working remotely even after the crisis, maybe even 70% of staff on a permanent basis.
Tech conferences have also gone fully virtual, and are innovating in really interesting ways. One example is “Deserted Island DevOps” a one-day conference hosted entirely inside the game Animal Crossing New Horizons. Presenters and attendees used in-game avatars to visit the conference islands and were able to create some of the smaller-group interactions that we value so much from physical conferences.
One positive aspect of everyone staying home and industry being shuttered is that we have, albeit temporarily, reduced pollution. Some estimates say carbon emissions in 2020 could be reduced by up to 8%. Renewables continue to increase their market share, for the first time surpassing coal for US power generation. We’re driving less, with cities seeing improved traffic flows and reduced accidents. The state of California has halved traffic accidents, saving it $40 million per day or roughly $1 billion since its shelter-in-place order began. It’s possible that the collectivism and trust in science required to fight an invisible virus may help us do more to tackle climate change in future, too.
While the human and economic costs of the pandemic continue to be very serious, I’m hopeful that the tech industry can use it as an opportunity to accelerate, reduce carbon heavy travel, and increase inclusivity and innovation.
X is software too
For me, some of the most interesting (and entertaining!) discussions around the Radar are what to put on ‘Hold.’ Thoughtworkers around the world raise things they think are a problem and if we consider something to be a cross-industry concern, we add it to the Radar in the Hold ring. But as we dug into the reasoning and concerns surrounding several proposed ‘Hold’ blips it became clear to us that some of these problems aren’t new, they’re an older problem arising in a new (or more widespread) context. In many cases our previous advice on the right thing to do — the positive practice to ‘Adopt’ — is still relevant today.A good example of this is “infrastructure as code.” This is quite old advice, originally added to the Radar in 2011. Our colleague Kief Morris published a detailed book on the topic back in 2016, with an updated copy due to be released later this year. Today, we’re seeing many organizations get into trouble with their cloud migrations in part because they aren’t applying enough rigor to cloud infrastructure definitions, management and automation. So rather than putting “hand crafted cloud templates” or “failing to version control cloud infrastructure definitions” on Hold, we’ve taken the opportunity to refresh and re-highlight our advice on the matter. Similarly, we have reblipped and refreshed pipelines-as-code.
Much of this arises because one of the major storylines of the past decade has been the rise of software, the rise of cloud and the rise of “as a service.” All of these enable us to use easily malleable software, or descriptions of a thing, rather than a hard-coded or physical instantiation which is slower to change. As a result, all of the important practices we’ve learnt about how to create software well — modularity, version control, automated testing — can and should be applied in these wider “software powered” contexts.
Cloud is the new legacy ball of mud
Every organization in the world is moving to some kind of cloud hosting, whether private, public or hybrid, and the vast majority are choosing between the “Big 4” — AWS, Azure, GCP and Alicloud. But in their exuberance for getting systems off-prem and onto cloud, many could be creating problems for tomorrow. There are several issues here.Firstly, there tend to be what I would call “highly optimistic” cost savings and ROI projections for cloud migration. Yes, owning and operating a data centre is expensive, but cloud hosting isn’t cheap either. It’s unlikely you’ll really save anything on staffing costs. And when the limit on your usage is how fast teams can click the “create an instance” button, rather than IT provisioning a server, your overall environment sprawl can become a massive, costly problem.
Secondly, “lift and shift” is often used as a deliberate migration strategy, rather than a derogatory comment that someone isn’t thinking carefully about their migration. Enterprise IT estates have many hundreds, and usually thousands, of individual systems that need to run somewhere. Re-engineering all of them to properly understand and operate in a cloud environment is obviously cost prohibitive. But cloud isn’t like on-prem — servers fail, workloads are constantly moved around. Getting a traditional workload to run reliably in the cloud usually isn’t a zero-cost venture either. The reality is that the best answer for an organization is highly complex and varies for each application or group of applications, and the more we can re-engineer for cloud, the more we can improve reliability and reduce costs.
Third, and very related, is that there is still much to learn about the right way to build and run systems in the cloud. Many organizations are discovering that “being in the cloud” hasn’t helped them cope with the surge in demand from the pandemic and associated lockdowns and hasn’t helped them release new features faster. Cloud architecture is a difficult new field of expertise and teams need to learn how to deploy, evolve, maintain and improve their cloud-based systems. This is something that takes time and much more thought than “just throw the stuff on Amazon.”
Semantic diffusion keeps diffusing
Without meaning to sound too much like an old man yelling at clouds, I have to point out that the industry continues to adopt and misuse terms that have specific meanings. I’ve long been annoyed with developers using “refactoring” to mean “fixing something” and today we see more of the same with terms such as “CI/CD” (those are two different things!), “DevOps” (no, please don’t build a “DevOps team”), “microservices” (that’s different from SOA how?) and “as code” (that’s how you would treat your code?).Today’s most meaning-free term is probably “digital transformation” which, depending on who’s using it, can mean any one of the following:
- Improving digital literacy for employees
- Better tools for collaboration, such as Zoom, Teams, Trello and Slack
- Agile IT, or
- Holistic transformation of the business leveraging the power of technology to create more value for customers and respond to rapid changes in the competitive landscape
I hope you’ve enjoyed this round up of trends within the tech industry, and that you are staying safe and healthy. I’d like to thank Andy Yates, Fausto de la Torres, Gareth Morgan, Kief Morris and Rebecca Parsons for their suggestions on drafts of this article
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
