Techniques

Although path-to-production mapping has been a near-universal practice at Thoughtworks since codifying Continuous Delivery, we often come across organizations unfamiliar with the practice. The activity is most often done in a workshop with a cross-functional group of people — that includes everyone involved in designing, developing, releasing and operating the software — around a shared whiteboard (or virtual equivalent). First, the steps in the process are listed in order, from the developer workstation all the way to production. Then, a facilitated session is used to capture further information and pain points. The most common technique we see is based on value-stream mapping, although plenty of process map variants are equally valuable. The activity is often eye-opening for many of the participants, as they identify delays, risks and inconsistencies and continue to use the visual representation for the continuous improvement of the build and deploy process. We consider this technique so foundational that we were surprised to discover we hadn't blipped it before.

Team interaction is a key concept when redesigning an organization for business agility and speed. These interactions will be reflected in the software being built (see Conway's Law) and indicate how effectively teams can autonomously deliver value to their customers. Our advice is to be intentional about how teams are designed and how they interact. Because we believe that organizational design and team interactions evolve over time, we think it's particularly important to measure and keep track of the team cognitive load, which indicates how easy or difficult teams find building, testing and maintaining their services. We've been using a template to assess team cognitive load that is based on ideas by the authors of the Team Topologies book.

We continue to be impressed by the positive impact of applying this book's concepts when communicating to clients and redesigning organizations. The authors recommend a simple but powerful approach to organizational design, identifying just four types of teams and three modes of interaction; this helps reduce ambiguity within the organization and provides a common vocabulary for teams, stakeholders and leadership to describe and design a team's work. To implement an org design change, we design the ideal to-be team topologies structure, apply any technical/staffing constraints (i.e., not enough employees) and then end up with the final to-be structure. That allows us to better advise clients and anticipate whether we're indeed improving cognitive load by comparing the as-is/to-be team structures.

We continue to recommend that teams carry out threat modeling — a set of techniques to help you identify and classify potential threats during the development process — but we want to emphasize that this is not a one-off activity only done at the start of projects; teams need to avoid the security sandwich. This is because throughout the lifetime of any software, new threats will emerge and existing ones will continue to evolve thanks to external events and ongoing changes to requirements and architecture. This means that threat modeling needs to be repeated periodically — the frequency of repetition will depend on the circumstances and will need to consider factors such as the cost of running the exercise and the potential risk to the business. When used in conjunction with other techniques, such as establishing cross-functional security requirements to address common risks in the project's technologies and using automated security scanners, threat modeling can be a powerful asset.

Since we last talked about BERT (Bidirectional Encoder Representations from Transformers) in the Radar, our teams have successfully used it in a few natural language processing (NLP) projects. In one of our engagements, we observed significant improvements when we switched from the default BERT tokenizer to a domain-trained word-piece tokenizer for queries that contain nouns like brand names or dimensions. Although NLP has several new transformer models, BERT is well understood with good documentation and a vibrant community, and we continue to find it effective in an enterprise NLP context.

Visual regression testing is a useful and powerful tool to have in your toolbox, but it has a significant cost given it's done for the entire page. With the rise of component-based frameworks such as React and Vue, we've also seen the rise of component visual regression testing. This technique strikes a good balance between value and cost to ensure that no undesired visuals have been added to the application. In our experience, component visual regression testing presents fewer false positives and promotes a good architectural style. By using it with tools such as Vite and the webpack feature Hot Module Replacement (HMR), it could be seen as a paradigm shift for applying test-driven development to front-end development.

When faced with the challenge of using a design system consistently across many form factors and platforms, the team at Salesforce came up with the concept of design tokens. The tokens store values, such as colors and fonts, in one central place. This makes it possible to separate options from decisions, and it significantly improves collaboration between teams. Design tokens are not new, but with the introduction of tools like Tailwind CSS and Style Dictionary, we see design tokens being used more often.

Using test email accounts or entire test SMTP (Single Mail Transfer Protocol) servers remains a common software testing practice. However, using a real server carries the risk that test emails will be sent to real people and often complicates automated integration testing. We've seen success using a fake SMTP server to test mail sending, which records a request to send an email without actually sending it. Multiple open-source tools exist in this space, including fake-smtp-server, which renders emails in a web UI for visual testing, and mountebank, which exposes the sent emails through a REST API for integration testing. We recommend exploring this technique to reduce risk and improve testing efficiency.

We're now seeing client projects that use federated machine learning (ML). Traditionally, ML model training has required data to be placed in a centralized location where the relevant training algorithm can be run. From a privacy point of view, this is problematic, especially when the training data contains sensitive or personally identifiable information; users might be reluctant to share data or local data protection legislation may prevent us from moving data to a central location. Federated ML is a decentralized technique for training on a large and diverse set of data that allows the data to remain remote — for example, on a user's device. Network bandwidth and the computational limitations of devices still present significant technical challenges, but we like the way federated ML leaves users in control of their own personal information.

We've been writing about developer platforms and how to build them in almost every edition of the Radar since 2017. In the meantime, the Team Topologies book has also done a great job of describing the ideal of a platform that supports developers with "self-service APIs, tools, services and knowledge." However, we often see teams shooting for too much of that platform vision too fast. Instead, building an incremental developer platform is key.

Team Topologies recommends to always strive for what they call the "Thinnest Viable Platform" necessary at any given stage, where the first version could even be just a set of documentation on a wiki. The next increment could increase the service level by providing templates or allowing teams to create pull requests. Further increments could then introduce self-service APIs, but only if valuable. In short, even though we've cautioned against fully ticket-driven platform operating models, going from zero to self-service is the other extreme. Pace yourself, treat your platform as a product and build it up incrementally.

Since introducing them in the Radar in 2016, we've seen widespread adoption of micro frontends for web UIs. Recently, however, we've seen projects extend this architectural style to include micro frontends for mobile apps as well. When an app becomes sufficiently large and complex, it becomes necessary to distribute the development over multiple teams. This presents a number of challenges around team autonomy, repository structures and integration frameworks. In the past we've mentioned Atlas and BeeHive, but these frameworks failed to gain traction and are no longer in active development. More recent approaches include Tuist or the Swift Package Manager for integrating the work of multiple teams into a single app. But in our experience, teams often end up implementing their own framework for integration. While we definitely see a need for modularity in scaling up mobile development teams, the case for micro frontends is less certain. This is because while micro frontends imply a direct correspondence between teams and pages or components, this structure could end up blurring responsibilities for business domain contexts, thereby increasing team cognitive load. Our advice is to follow the basics of good, clean application design, embrace modularity when scaling up to multiple teams and adopt a micro frontend architecture only when the modules and the business domain are strongly aligned.

Observability practices have shifted the conversation from monitoring for well-understood problems to helping troubleshoot unknown problems in distributed systems. We've seen success taking that perspective outside of the traditional production environment by applying observability for CI/CD pipelines to help optimize testing and deployment bottlenecks. Complex pipelines create developer friction when they run too slow or suffer from nondeterminism, reducing important feedback loops and hindering developer effectiveness. Additionally, their role as critical deployment infrastructure creates stress points during periods of rapid deployments, as happened to several organizations responding to the recent log4shell vulnerability. The concept of traces translates nicely to pipelines: instead of capturing the cascade of service calls, child spans capture information about each stage of the build. The same waterfall charts used to analyze a call flow in a distributed architecture can also be effective in helping us to identify bottlenecks in pipelines, even complex ones with fan-in and fan-out. This enables far more focused optimization efforts. While the technique should work with any tracing tool, Honeycomb supports a tool called buildevents that helps capture pipeline trace information. An alternative approach of capturing information already exposed by CI/CD platforms, taken by the open-source buildviz (built and maintained by a Thoughtworker), allows for a similar investigation without changing the step configurations themselves.

As software continues to grow in complexity, the threat vector of software dependencies becomes increasingly challenging to guard against. Supply chain Levels for Software Artifacts, or SLSA (pronounced "salsa"), is a consortium-curated set of guidance for organizations to protect against supply chain attacks, evolved from internal guidance Google has been using for years. We appreciate that SLSA doesn't promise a "silver bullet," tools-only approach to securing the supply chain, but it does provide a checklist of concrete threats and practices along a maturity model. The threat model is easy to follow with real-world examples of attacks, and the requirements provide guidance to help organizations prioritize actions based on levels of increasing robustness to improve their supply chain security posture. Since we first mentioned it in the Radar, SLSA has added more detail around software attestations with examples to track concerns like build provenance. Our teams have found SLSA to strike a nice balance between implementation guidance and higher-level awareness around supply chain threats.

With continued pressure to keep systems secure and no reduction in the general threat landscape, a machine-readable Software Bill of Materials (SBOM) may help teams stay on top of security problems in the libraries that they rely on. Since the original Executive Order was published, the industry has gained clarity and understanding of what an SBOM is and how to create one; the National Institute of Standards and Technology (NIST), for example, now has more specific advice on how to comply with the order. We've had production experience using SBOMs on projects ranging from small companies to large multinationals and even government departments, and we're convinced they provide a benefit. More organizations and governments should consider requiring SBOMs for the software they use. The technique will be strengthened by the new tools that continue to emerge, such as the Firebase Android BOM that automatically aligns an application's library dependencies to those listed in the BOM.

Sustainability is a topic that demands the attention of enterprises. In the software development space its importance has increased, and we're now seeing different ways to approach this topic. Looking at the carbon footprint of building software, for example, we recommend assessing carbon efficiency as an architectural characteristic. An architecture that takes into consideration carbon efficiency is one where design and infrastructure choices have been made in order to to minimize energy consumption and therefore carbon emissions. The measurement tooling and advice in this space is maturing, making it feasible for teams to consider carbon efficiency alongside other factors such as performance, scalability, financial cost and security. Like almost everything in software architecture, this should be considered a trade-off; our advice is to think about this as one additional characteristic in a whole set of relevant quality attributes that are driven and prioritized by organizational goals and not left to a small cadre of experts to ponder in a siloed manner.

How do you approach writing good code? How do you judge if you've written good code? As software developers, we're always looking for catchy rules, principles and patterns that we can use to share a language and values with each other when it comes to writing simple, easy-to-change code.

Daniel Terhorst-North has recently made a new attempt at creating such a checklist for good code. He argues that instead of sticking to a set of rules like SOLID, using a set of properties to aim for is more generally applicable. He came up with what he calls the CUPID properties to describe what we should strive for to achieve "joyful" code: Code should be composable, follow the Unix philosophy and be predictable, idiomatic and domain based.

The accidental publication of secrets seems to be a perennial issue with tools such as Talisman popping up to help with the problem. Before now, GitHub Enterprise Cloud users with an Advanced Security License could enable security scanning on their accounts, and any secrets (API keys, access tokens, credentials, etc.) that were accidentally committed and pushed would trigger an alert. GitHub push protection takes this one step further, and brings it one step earlier in the development workflow, by blocking changes from being pushed at all if secrets are detected. This needs to be configured for the organization and applies, of course, only to license holders, but additional protection from publishing secrets is to be welcomed.

In a centralized application, the data on the server is the single source of truth — any modification to the data must go through the server. Local data is subordinate to the server version. This seems like a natural and inevitable choice to enable collaboration among multiple users of the software. Local-first application, or local-first software, is a set of principles that enables both collaboration and local data ownership. It prioritizes the use of local storage and local networks over servers in remote data centers or the cloud. Techniques like conflict-free replicated data types (CRDTs) and peer-to-peer (P2P) networks have the potential to be a foundational technology for realizing local-first software.

Metrics store, sometimes referred to as headless business intelligence (BI), is a layer that decouples metrics definitions from their usage in reports and visualizations. Traditionally, metrics are defined inside the context of BI tools, but this approach leads to duplication and inconsistencies as different teams use them in different contexts. By decoupling the definition in the metrics store, we get clear and consistent reuse across BI reports, visualizations and even embedded analytics. This technique is not new; for example, Airbnb introduced Minerva a year ago. However, we're now seeing considerable traction in the data and analytics ecosystem with more tools supporting metrics stores out of the box.

Server-driven UI continues to be a hot topic of discussion in mobile circles because it offers the potential for developers to take advantage of faster change cycles without falling foul of an app store's policies around revalidation of the mobile app itself. Server-driven UI separates the rendering into a generic container in the mobile app while the structure and data for each view is provided by the server. This means that changes that once required a round trip to an app store can now be accomplished via simple changes to the responses the server sends. While some very large mobile app teams have had great success with this technique, it also requires a substantial investment in building and maintaining a complex proprietary framework. Such an investment requires a compelling business case. Until the case is made, it might be best to proceed with caution; indeed, we've experienced some horrendous, overly configurable messes that didn't actually deliver on the promised benefits. But with the backing of behemoths such as Airbnb and Lyft, we may very well see some useful frameworks emerge that help tame the complexity. Watch this space.

Since Google first popularized service-level indicators (SLIs) and service-level objectives (SLOs) as part of their site reliability engineering (SRE) practice, observability tools like Datadog, Honeycomb and Dynatrace started incorporating SLO monitoring into their toolchains. OpenSLO is an emerging standard that allows defining SLIs and SLOs as code, using a declarative, vendor-neutral specification language based on the YAML format used by Kubernetes. While the standard is still quite new, we're seeing some encouraging momentum, as with Sumo Logic's contribution of the slogen tool to generate monitoring and dashboards. We're excited by the promise of versioning SLI and SLO definitions in code and updating observability tooling as part of the CI/CD pipeline of the service being deployed.

During our discussions for this edition of the Radar, several tools and applications for synthetic data generation came up. As the tools mature, we've found that using synthetic data for testing models is a powerful and broadly useful technique. Although not intended as a substitute for real data in validating the discrimination power of machine-learning models, synthetic data can be used in a variety of situations. For example, it can be used to guard against catastrophic model failure in response to rarely occurring events or to test data pipelines without exposing personally identifiable information. Synthetic data is also useful for exploring edge cases that lack real data or for identifying model bias. Some helpful tools for generating data include Faker or Synth, which generate data that conforms to desired statistical properties, and tools like Synthetic Data Vault that can generate data that mimics the properties of an input data set.

We continue to be excited by the TinyML technique and the ability to create machine learning (ML) models designed to run on low-powered and mobile devices. Until recently, executing an ML model was seen as computationally expensive and, in some cases, required special-purpose hardware. While creating the models still broadly sits within this classification, they can now be created in a way that allows them to be run on small, low-cost and low-power consumption devices. If you've been considering using ML but thought it unrealistic because of compute or network constraints, then this technique is worth assessing.

When we first included it in the Radar two years ago, verifiable credentials (VC) was an intriguing standard with some promising potential applications, but it wasn't widely known or understood outside the community of enthusiasts. This was particularly true when it came to the credential-granting institutions, such as state governments, who would be responsible for implementing the standards. Two years and one pandemic later, the demand for cryptographically secure, privacy-respecting and machine-verifiable electronic credentials has grown and, as a result, governments are starting to wake up to VC's potential. We're now starting to see VC crop up in our work for public-sector clients. The W3C standard puts credential holders at the center, which is similar to our experience when using physical credentials: users can put their verifiable credentials in their own digital wallets and show them to anyone at any time without the permission of the credentials' issuer. This decentralized approach also enables users to better manage and selectively disclose their own information which greatly improves data privacy protection. For example, powered by zero-knowledge proof technology, you can construct a verifiable credential to prove that you're an adult without revealing your birthday. It’s important to note that although many VC-based decentralized identity solutions rely on blockchain technology, blockchain is not a prerequisite for all VC implementations.

The term "remote team setup" does not just describe one setup; it encompasses multiple patterns and flavors. And many teams have been changing patterns recently. They're coming out of the "everybody always remote" mode that was forced on them by a pandemic and moving into a pattern of (often rotating) satellite workers, where part of the team is co-located and part of the team is remote. We see many of them failing to properly consider what this means for their ways of working. Satellite workers without "remote native" ways of working is a slip back into privileging co-located practices. In a setup with satellite workers, it's important to still use "remote native" processes and approaches by default. For example, if the co-located part of the team joins a meeting together, they should still all be on their individual laptops to participate in digital collaboration or meeting chat. Teams need to be aware of the risk of excluding their satellite workers and creating silos and feelings of exclusion. If you know that you'll always have at least one satellite team member, the default ways of working should assume remoteness.

The prevalence of teams choosing a single-page application (SPA) when they need a website continues. We remain concerned that people aren't properly recognizing SPAs as an architectural style to begin with; instead they're immediately jumping into framework selection. SPAs incur complexity that simply doesn't exist with traditional server-based websites: issues such as search engine optimization, browser history management, web analytics and first page load time all need to be addressed. Proper analysis and consideration of the trade-offs is required to determine if that complexity is warranted for business or user experience reasons. Too often teams are skipping that trade-off analysis, blindly accepting the complexity of SPAs by default even when business needs don't justify it. We still see some developers who aren't aware of an alternative approach because they've spent their entire career in a framework like React. We believe that many websites will benefit from the simplicity of server-side logic, and we're encouraged by techniques like Hotwire that help close the gap on user experience.

The term "cloud native" was originally used to describe architectures with characteristics that took maximum advantage of public cloud hosting. Examples include distributed architectures composed of many small, stateless and collaborating processes, and systems with high levels of automation for building, testing and deploying applications. However, we've noticed a growing trend toward superficial cloud native designs that simply use a lot of a cloud vendor's proprietary services and stop there without revisiting the fundamentally monolithic, brittle or toil-intensive nature of the application. It’s important to remember that serverless functions by themselves don't make an application more resilient or easier to maintain and that cloud native is really a matter of design rather than a set of implementation choices.