Embracing and scaling effective engineering practices in tough times

1. Intro: The struggle to navigate a perfect storm 

The roaring return of inflation, supply chain pressures and geopolitical uncertainties are sapping momentum on all fronts, from housing markets to plans for business expansion. As companies wrestle with volatility not witnessed for decades, could the ambitious digital transformation plans accelerated by the pandemic be the next casualty? 

Early indicators suggest there’s still a high degree of determination to follow through with planned tech investments. A recent study by research firm ETR found tech executives still plan to boost IT spending by an average of over 6 percent this year, roughly on par with last year, regardless of the deteriorating environment. Similarly, a poll by data center provider Equinix found that, despite economic headwinds, a clear majority of IT decision-makers plan to either increase or maintain spending on data centers and interconnection infrastructure.

Majority of global IT decision-makers aim to boost or maintain data center, connectivity spending in 2022 

Nonetheless, Thoughtworks experts see the outlook for digital transformation changing in several key ways. First is an overall squeeze on capital expenditures as the cost of capital rises and business confidence is challenged. 

“With the current economic climate there won’t be as much access to full-time headcount, and technology projects will have to show clear progress against financial indicators and customer outcomes,” says Tim Cochran, Technical Director, Thoughtworks. “CFOs and investors will require you to tighten your belt – and that doesn’t just apply to startups, even larger public companies will be under scrutiny. That means digital transformation may not be as easy to justify.” 

At the same time, with so many businesses rethinking their strategies during the pandemic, investment in digital has transitioned from a wave into a full-blown flood, with enterprises forced to spend more and more just to keep up.

“Investments in enterprise software have increased dramatically as companies seek to do exponentially better than their competitors,” says Sagar Paul, Thoughtworks Director of Client Services and Strategy. “In turn, more people are needed to write custom software so that push has grown significantly, all while newer technologies are coming in. It used to be that the typical enterprise architecture was made up of 50-100 components; today it’s 1,000.” 

What’s more, any effort to slash spending can quickly backfire, says Rachel Laycock, Global Managing Director of Enterprise Modernization, Platforms, and Cloud at Thoughtworks. “Cost cutting is a race to the bottom and that’s a really critical mindset to keep,” she explains. “When times are tough, of course, be smart and judicious about your operational efficiency and don’t spend money you don’t need to. But focusing entirely on cost-cutting is a very dangerous path, because your competitors may be continuing business and customer value-driven work which drives top line revenue even with limited funds.” 

Rather than shedding resources, a better strategy is to ensure those the company does have are deployed and applied as effectively as possible. The key to that rests with minimizing wasted or misplaced technology investment, and fostering high-performing engineering teams.

“As business grows more digital-dependent, the ability to capture new opportunities and manage risks has shifted from the boardroom to engineering teams,” says Paul. “Engineering efficiency and effectiveness have therefore become even more critical.” 

Unfortunately, businesses have to contend with a shortage of qualified developers that is only expected to worsen. In the Equinix survey, a lack of talent ranked behind only the speed of transformation as the biggest challenge impacting tech skills. 

Pace of change and lack of talent rank as biggest challenges for IT decision-makers

Instead of reducing headcount or investment, then, the priority has to be figuring out how to build and empower the highest-performing, most cost-effective engineering teams, so the enterprise can keep spending in check while remaining competitive and bringing innovations to market faster. 

“Changes in the macroeconomic climate can be a good opportunity to really assess what you’re focusing on,” says Laycock. “Ruthless prioritization, refocusing the resources that you have onto the most valuable things, can help ensure you get top line benefits with the same costs you have currently.” 

2. Putting the development portfolio under scrutiny

In essence, says Laycock, efficiency comes down to helping teams make better decisions – and not shying away from decisive action when required. 

“Your business and product strategy may need to adapt to the macroeconomic climate,” she explains. “It’s important to ask what variables and inputs you based your strategy on, and question whether your original hypotheses and assumptions are still true. Strategy isn’t just assessing how you’re doing on your execution plan, but assessing whether you should be sticking with the plan at all, and whether it’s still valuable. If it’s not, that’s the point you need to pivot, rather than falling into the sunk cost fallacy of continuing only because you started that way.” 

Similarly, with conditions evolving so rapidly, companies have to move away from the traditional delivery mindset where projects proceed in a linear fashion from start to finish – because it may be necessary to change direction midstream. 

“Having a ‘build it and they will come’ strategy is very high risk,” Laycock says. “And we believe strongly in de-risking things, which is why we approach strategy and product thinking in an iterative way, and platforms in a product-centric way. This encourages a laser focus on end value. It’s not about spending forever, building out the perfect infrastructure or product to meet every single customer and every single requirement. It’s about building for what you know and need to achieve today. And then continuously iterating and incrementally building on top of that, based on customer needs.” 

Of course, the definition of value itself may vary – but should always align closely to the enterprise’s business goals, and dictate in turn how resources and investment are structured. 

In challenging circumstances, Paul advises enterprises to make strengthening core products and services a top priority, while ambitious investments should be approached more carefully and incrementally. 

“The basic need is to stay afloat – to make sure whatever risks are being faced by your business, are being addressed,” he says. “Predictability of revenues is crucial. Products that see consistent demand, establish you as a differentiator, deliver consistent profitability – these must be continued in some way or form.” 

This doesn’t equate to standing still. In adverse conditions there’s an even greater need to invest in making products better, and in ensuring they stand out. “Any product development that helps you differentiate is among the most important,” Paul says. “In a recessionary market where the pie is not growing, you must be able to take market share from the competition, and you can only do that with highly differentiated, high-quality products and services.” 

Companies also have little choice but to continue supporting projects that underpin compliance, governance and sustainability goals, he notes. “You have to follow the rules of the land, and your customers will view the organization in a different light if you’re not meeting your promises.” 

Part of the prioritization challenge is that organizations often lack a real sense of the value projects deliver – or how they fit into the enterprise’s broader strategic fabric. 

“Despite a lot of improvements, we still see that companies aren’t tying what they’re building back to business and customer outcomes,” says Cochran. “They’re not necessarily tracking things like customer engagement, or how they’ve moved profitability or revenue. Leading modern digital businesses have experimentation capabilities that allow them to reach out and to figure out what the value is, combined with gathering qualitative information through user research. That means you don't have to do a big program of work before you get value – you can do a small amount, or thin slice, and measure how customers are responding. If you’re not practicing incremental development, and you don't have a way of measuring product metrics, those are the things you need to focus on.” 

A layer of analysis to ensure the business is focusing on the right projects is even more crucial when financial and development resources are under strain – though companies often have more slack or room for maneuver than they may think. 

“Often teams are building services that they could buy or rent from a SaaS or cloud company,” Cochran says. “It may have been the right investment when they started a couple of years ago, but especially with SaaS, products improve all the time. So there may be an opportunity to reassess, but a lot of enterprises are not set up to do that. Often the architecture is so tangled, they can’t easily swap out capabilities.” 

3. A toolkit for effective engineering teams 

As well as misallocating investments, many organizations have yet to tap into the full potential of their development teams. 

“We believe that in any engineering organization, there is at least a 20% bloat sitting there,” says Paul. “The important thing to understand is that when you look at resources, and you look at talent, you’re not looking at one person, you’re looking at teams, and you’re looking at your entire engineering organization.” 

Taking a holistic view of the engineering organization’s makeup and context positions businesses to generate cost savings without starving software innovation of resources. From our experience, there are six key focus areas that enable a strategic approach to engineering transformation. Each of these elements are important pieces of the puzzle, but will have limited impact on the effectiveness of the engineering organization, and its ability to deliver, if tackled alone.

Engineering effectiveness transformation

“When businesses think about modernizing, they often associate that with spending money on new platforms, cloud storage or hiring,” notes Cochran. “But often it’s less about needing more resources than making a process and culture change. Businesses can afford to be more frugal if they better enable their existing talent by changing outdated or inefficient practices.” 

This requires a frank assessment of what needs to be done differently – with the input of senior management.   

“Quite often, organizations don’t know what they are spending money on,” says Cochran. “If you think about a cloud transformation, without decoupled teams and systems, you can’t easily tie the bill your CFO gets every month to projects or products you're building. It’s also difficult to establish which projects are inefficient or not doing too well. Giving leadership the right information at the right time will help organizations start paying attention to these areas, and provide the impetus to improve them.” 

“Before the hyperscalers and having cloud on tap, organizations were used to upfront capital budgeting and being conscientious with spend on servers, that limited innovation but also forced ways of development that were efficient. With IaaS, the new CFO has little control over the infrastructure spend as it bloats due to poorly written code or servers being indiscriminately spun up”, says Paul. There is a clear need for the new CFO and the new CTO to align on this before the bills get out of hand. We are already seeing the adverse impact on operational expenses at several customers.

In addition to being more efficient, organizations must seek to be more effective, which requires a focus on reducing bureaucracy and friction, Cochran explains. Current organizational processes involving product teams should be examined to identify bottlenecks – whether a lack of information sharing across different functions, or developers simply not creating enough technical documentation in the first place.

Examining entrenched practices and zeroing on specific areas of friction can be tricky, especially from the inside. That makes it beneficial for the organization to apply an external lens, in the form of a framework or workflow tool that measures performance against independent benchmarks and provides insights on where resources may be misaligned and dynamism lost.

Thoughtworks, for example, frequently employs the Polaris system, a tool-assisted consulting framework, that creates easy-to-understand diagnostics that enterprises can then use to course correct, scale, and ensure teams don’t revert back to less effective ways of working, post-transformation.

“The goal is to help organizations systematically analyze the work of their entire engineering organization and gain line of sight from business outcomes to engineering success,” says Paul. 

Tracking delivery excellence metrics to identify areas for improvement

Cochran notes teams can also take advantage of a growing number of tools such as Jellyfish to track their performance, or generate and maintain key metrics themselves. Enlisting the capabilities of a product manager can be another game-changer. 

“What we're seeing a lot now are DevOps, developer experience, and platform teams using technical product managers who understand the metrics, and can connect them to business strategy, then create dashboards that track and provide the necessary insights,” he says.

Though metrics are important, Laycock notes organizations should be careful about how they’re wielded. 

“While metrics-driven insights give engineering and product teams indicators as to where they can improve, it’s critical to understand that they shouldn’t be used to measure an individual employee’s performance, but rather to give the team a baseline from where they can progress,” she explains. 

“The use of the tool is more important than the tool itself. In my experience, when you give people a baseline and a target and assure them they won’t be beaten up for not achieving them, they will get excited by the initiative and genuinely want to become better.”

Quantitative metrics also have to be balanced with qualitative steps to listen to and empower engineers, Cochran notes. Companies like Spotify and Etsy conduct regular surveys to gather day-to-day feedback, measure satisfaction and source new ideas, with the results used as another measure of the success of tech investments. 

Organizations adopting a product-centric approach can encounter issues with knowledge fragmentation. This is a byproduct of teams focusing on their own priorities, and a major problem considering cultivating a healthy engineering ecosystem rests not just on platform thinking and tools that drive individual developer efficiency, but also “how developers are collaborating within teams,” notes Paul.

Building developer portals with platforms such as Backstage to create a centralized development environment for storing information and sharing across teams can help reduce waste, and improve consistency and creativity as developers build off prior work and trade ideas. 

“The same person can potentially be much more productive and skillful given the right environment,” says Paul. “It’s not just the skill and talent of the individual that is important, but what makes up the engineering ecosystem and how the ecosystem is supportive and geared towards quality and throughput. Transplanting high-performing teams from other organizations, therefore, fail often.”

4. Achieving, and maintaining, organizational-engineering team alignment

While a product-centric approach democratizes decision-making and facilitates the flow of ideas and innovations, consistent governance and organizational backing is needed to channel those innovations in the right direction. 

“Organizations need to ensure strategic, high-level decisions made in the boardroom still make sense when they are incorporated into development team decisions, and conversely that the decisions made on the ground still make sense and are aligned with the overarching strategy,” notes Laycock.

Ultimately frameworks and process improvements can only take an engineering organization so far – typically a shift in workplace culture is needed for new practices to take shape and be fully adopted. “Business leaders’ willingness to use the information to create an environment where people can be more productive and learn new skills is critical,” Laycock says. 

“A lot of the hard work is more around culture change, which boils down to having good leadership and empowering individuals, making sure everyone is learning, and those learnings are shared across the organization,” agrees Cochran.

Particular attention should be paid to change management because of the tendency to fall back on old ways of doing things after the initial push to adopt new practices dies down, Paul notes. “Effecting change requires organizations to systematically embed new work practices and culture into the day-to-day, support their developer teams through these changes, and have processes in place to ensure their developers refresh their skills and knowledge regularly to stay up-to-date.”

Four key elements for successful and sustainable organizational change

“You can have a great strategy written down, but a great roadmap will not materialize if organizations do not have people with the ability to execute,” Laycock says. “Good leaders are critical to success. Whether you develop them internally or bring them in from other organizations, having a way to identify strong leaders who can lead the charge on the things you must do is critical. And that doesn’t only refer to managers and executives. Anyone can be a leader within the organization.” 

Finally, being transparent about the rationale behind decisions will enable the enterprise to introduce adjustments, even tighten belts, where necessary while maintaining employee morale. 

“If an engineering or product team doesn't understand why something is being cut or changed, it's very hard for them to empathize and adjust their ways of working,” Cochran notes. “On the other hand, if organizations share the context and details behind their strategies and decisions, the same employees are better positioned to direct their talents and reorient their practices towards new business goals.”

5. Making the best of a crisis 

As organizations brace for tougher times ahead, they can take solace in the fact that investments directed at improving their ability to execute and stand out among competitors are bound to bear fruit over time. 

“History has demonstrated again and again that hard times and constraints create the best innovations,” notes Laycock. “Organizations don’t have to look further than the recent pandemic to realize that those that were able to adapt quickly to changes came out stronger.”

Research has shown that when businesses continue to invest in innovation in lean times, they often outpace competitors when the economy stabilizes. A report from McKinsey suggests that through-crisis innovators outperform the market by 10% during a crisis, and 30% plus, post-crisis.

Continued investments during a downturn boost organizations’ post-crisis performance

Normalized market capitalization, index (Q1 2007 = 100)

Cochran notes the reverse is also true. “Technological progress is always ongoing and competitors will continue leveraging new technologies like AI to differentiate themselves,” he says. “Enterprises that don’t continue with such investments will be left behind.”

Though CFOs and investors may be treading cautiously, most businesses remain willing to fund high quality initiatives where they can be justified. “The efficient use of capital is very important and the people managing these funds are figuring out where the deployment will be profitable, and where the risks are,” says Paul. 

An adverse environment can therefore be viewed as a foundation to build adaptive capability and resilience – which isn’t defined by the organization’s financial resources or technology infrastructure, but the much more valuable capacity to quickly alter course, or come up with a new product or approach where required.   

"Regardless of the current state of an organization’s technology, it’s usually not the bottleneck in and of itself,” says Laycock. “The ability to adapt to change is more critical. An effective tech strategy should support the goal of becoming a learning organization; one that can adapt and improve regardless of what comes next.”