We've come up with various ways to show blips evolving from one Radar volume to another. Blips can be new to the volume or move between rings.
We discussed several topics in this edition of the Radar (some of which eventually failed to make the final cut) where teams are employing tools to adapt to/from Kafka. Some of these tools allow more traditional interfaces to Kafka (such as ksqlDB, Confluent Kafka REST Proxy, and Nakadi), while others are designed to provide extra services such as GUI frontends and orchestration add-ons. We suspect that part of the underlying reason for this bounty of tools is the underlying sharp-edged complexity of some of Kafka’s parts combined with increased presence in organizations that need to bend it to existing architectures and processes. Some teams end up treating Kafka as a next-generation enterprise service bus — one example of The slippery slope of convenience theme — but other teams use Kafka to provide universal access to business events as they happen. These organizations recognize that it is sometimes easier to have a centralized infrastructure with adaptation at the edges and try to avoid sprawl with careful design and governance. In any case, it shows that Kafka continues toward status as a de facto standard for asynchronous publish/subscribe messaging at volume.
An antipattern as old as the Radar is the tendency for teams to place behavior within their ecosystem at convenient but improper nexus points that lead to long-term technical debt and worse. Examples abound, including using a database as an integration point, using Kafka as a global orchestrator, intermingling business logic with infrastructure code and so on. Modern software development offers many places for developers to stash behavior, and inexperienced or inconsiderate teams often entangle concerns by not carefully considering the long-term consequences of inappropriate coupling. Inappropriate team structures and other deviations from Conway’s Law don't help either. As software systems become more complex, development teams must show diligence to both create and maintain thoughtful architecture and design, not slap-dash decisions for expediency. Often, thinking about the testability of a particular approach leads teams away from some of these potentially problematic decisions. Software tends toward complexity when left to its own devices. Careful design and, perhaps more importantly, ongoing governance works to ensure that schedule pressure or one of the other numerous disruptive forces doesn't cause teams to make convenient but improper decisions.
Many architects cite Conway's Law, the 1960s observation that teams’ communication structures influence design, to justify changes to team organization, and we discovered across several nominated blips in this edition that an organization's team structure remains a key enabler when handled well and a serious impediment when handled poorly. Examples we discuss include the need for product thinking around platform teams rather than treating them as order-takers; Team Topologies and the increasing recognition of team cognitive load in relation to effectiveness; and the new framework developed around programmer productivity called SPACE. Organizations spend enormous funds on tools, yet many find better productivity gains by paying attention to the people who build the software and what makes them effective within a particular organization.
Many in the software world prize clever solutions to complex problems, yet often those clever solutions result from self-inflicted accidental complexity. Numerous examples of this phenomenon exist today, including the unfortunate but common practice of secreting orchestration or coordination code in an inappropriate location. For example, we see clever workflow management tools such as Airflow or Prefect that are overeagerly used to manage complex data pipelines through orchestration. We find a host of tools that work around the issues caused by monorepos, such as Nx and many more. Teams often don't realize they're doubling or tripling down on needless complexity without stepping back to look at the big picture and question whether the current solution is worse than the problem. Rather than jump to more technology to solve a problem, teams should do root cause analysis, address the underlying essential complexity and course correct. Data mesh is an example of an approach that addresses the underlying organizational and technical assumptions that have led to overly complex data pipelines and tooling.
We found a serious drop in the number of platform-related blips in this edition of the Radar, which we attribute to the increased consolidation on some industry standards: most companies have already chosen their cloud vendors, and they've mostly standardized on Kubernetes for container orchestration and Kafka for high-performance messaging. Does this mean that platforms no longer matter? Or are we experiencing the equivalent of a business cycle of alternating periods of expansion and contraction — we've seen similar periods of rapid innovation followed by stasis (what Stephen Jay Gould called "punctuated equilibrium") in database technologies, for example. Perhaps the industry has entered a period of relative calm as organizations assimilate the tectonic shift to the cloud and await the next wave of disruptive innovation.
The Technology Radar is prepared by the Thoughtworks Technology Advisory Board, comprised of:
Rebecca Parsons (CTO) • Martin Fowler (Chief Scientist) • Bharani Subramaniam • Birgitta Böckeler • Brandon Byars • Camilla Falconi Crispim • Cassie Shum • Erik Doernenburg • Fausto de la Torre • Hao Xu • Ian Cartwright • James Lewis • Lakshminarasimhan Sudarshan • Mike Mason • Neal Ford • Perla Villarreal • Scott Shaw • Shangqi Liu • Zhamak Dehghani