Enable javascript in your browser for better experience. Need to know to enable it? Go here.

Tools

Adopt ?

  • Releasing applications for iOS involves a code-signing step. Although supported by Apple's toolchain, the process can be cumbersome, error prone and full of unexpected surprises. We're happy to report that fastlane, already our tool of choice for automating the release process of mobile applications, provides a better solution: match is integrated into fastlane's smooth process, and it implements a new approach to manage code signing for teams. Instead of storing the signing keys in the developer's macOS keychain — the default approach — the new approach revolves around storing the keys and certificates in a Git repository. This not only makes it easier to on-board new team members and set up new development machines; in our experience, it also is the easiest method to integrate signing into continuous delivery pipelines.

Trial ?

  • In recent years we've seen the rise of generic and domain-specific workflow management tools. The drivers behind this rise include the increased usage of data-processing pipelines and the automation of the machine-learning (ML) model development process. Airflow is one of the early open-source task orchestration tools that popularized the definition of directed acyclic graphs (DAGs) as code, an improvement over an XML/YAML pipeline configuration. Although Airflow remains one of the most widely adopted orchestration tools, we encourage you to evaluate other tools based on your unique situation. For example, you may want to choose Prefect, which supports dynamic data-processing tasks as a first-class concern with generic Python functions as tasks; or Argo if you prefer a tight integration with Kubernetes; or Kubeflow or MLflow for ML-specific workflows. Given the rise of new tools, combined with some of the shortfalls of Airflow (such as lack of native support for dynamic workflows and its centralized approach to scheduling pipelines), we no longer recommend Airflow as the default orchestration tool.

    We believe that with the increased usage of streaming in analytics and data pipelines, as well as managing data through a decentralized data mesh, the need for orchestration tools to define and manage complex data-processing pipelines is reduced.

  • Batect continues to gain traction among our developers and is considered by many to be a default approach for configuring local development and test environments. This open-source tool (which happens to be developed by a Thoughtworker) makes it easy to set up and share a build environment based on Docker. Batect then becomes the entry point for your build system, replacing the ubiquitous go script as the basis for a “check out and go” approach. Batect continues to evolve in response to developer feedback and recently added support for Docker's BuildKit and shell tab completion.

  • Berglas is a tool for managing secrets on Google Cloud Platform (GCP). We've recommended secrets as a service as a technique to store and share secrets in modern distributed architectures in the past, and GCP offers Secret Manager for that purpose, and Berglas works well with Secret Manager. This is especially useful for those GCP services that don't have direct integration with Secret Manager yet; the alternative in such cases would be to write custom code or scripts. Berglas ships as a command-line tool and as a library, and both also come in handy in use cases beyond secrets as a service. The author of Berglas, who also happens to be the original author of HashiCorp Vault, now works at Google; however, Berglass is not an official Google product.

  • Contrast Security offers a security platform with multiple components, including static application security testing (SAST), interactive application security testing (IAST), open-source scanning and runtime application self-protection (RASP). It's been around for a few years now, and we've used it in multiple projects. One of the things we quite like about the Contrast platform is its run-time analysis of libraries; it helps identify libraries that are not used, which in turn helps our teams prioritize vulnerabilities and potentially get rid of unused libraries. This is particularly relevant given the increased importance of securing the software supply chain. We also quite like its IAST component; we've found it effective in our continuous delivery (CD) pipeline with reduced false positives, and it manages to catch a good range of vulnerabilities.

  • Dive is a tool for analyzing Docker images; it helps explore each layer in the image and identify what's changed in each layer. Dive estimates image efficiency and wasted space in an image and can be integrated into the continuous integration (CI) pipeline to fail the build based on the efficiency score or amount of wasted space. We've used it in a few projects, and it has proven to be a useful tool — particularly if you're building images with a very low tolerance for additional tools or space consumption.

  • Our teams continue to report good results when using Lens to visualize and manage their Kubernetes clusters. Billed as an "IDE for Kubernetes," Lens makes it possible to interact with the cluster without having to memorize commands or manifest file structures. Kubernetes can be complex, and we understand that a tool for visualizing cluster metrics and deployed workloads can save time and reduce some of the toil involved in maintaining a Kubernetes cluster. Instead of hiding complexity behind a simple point-and-click interface, Lens brings together the tools an administrator would run from the command line. But be cautious about interactively making changes to a running cluster via any mechanism. We generally prefer that infrastructure changes be implemented in code so they are repeatable, testable and less prone to human error. However, Lens does excel as a one-stop tool to interactively navigate through and comprehend your cluster status.

  • Over the years we've debated several times whether to feature monorepos in the Radar. Each time we ended up concluding that the trade-offs introduced by monorepos require a nuanced discussion and the technique is "too complex to blip." Now we're seeing increased interest in monorepos in the JavaScript community, for example, for building applications composed of micro frontends, as discussed in this podcast episode. Whether this is a good idea depends a lot on your situation, and we certainly don't want to give a general recommendation. What we do want to comment on is the tooling. In our teams we see a shift away from Lerna and a strong preference to use Nx for managing JavaScript-based monorepos.

  • Wav2Vec 2.0 is a self-supervised learning framework for speech recognition. With this framework the model is trained in two phases. First, it begins in self-supervised mode using unlabeled data and tries to achieve the best possible speech representation. Then it uses supervised fine-tuning, during which labeled data teaches the model to predict particular words or phonemes. We've used Wav2Vec and find its approach quite powerful for building automatic speech recognition models for regional languages with limited availability of labeled data.

Assess ?

  • cert-manager is a tool to manage your X.509 certificates within your Kubernetes cluster. It models certificates and issuers as first-class resource types and provides certificates as a service securely to developers and applications working within the Kubernetes cluster. With built-in support for Let's Encrypt, HashiCorp Vault and Venafi, cert-manager is an interesting tool to assess for certificate management.

  • Stakeholders increasingly expect businesses to account for the environmental externalities of their decisions, as evidenced by the rise of environmental, social and corporate governance (ESG) investing and employee activism around climate change. Migrating to the cloud offers the potential for more efficient energy usage — the cloud providers have much more scale to justify investment in green energy sources and R&D — but the downside of software abstractions for cloud users is that those abstractions also hide the energy impact as the actual data centers are hidden from view and financed by another company. Cloud Carbon Footprint, a new open-source tool, takes advantage of cloud APIs to provide visualizations of estimated carbon emissions based on usage across AWS, GCP and Azure. It uses heuristics like Etsy's Cloud Jewels to estimate energy usage and public data sources to convert energy usage into emissions based on the carbon intensity of the cloud region's underlying energy grid (GCP publishes this data already). The tool's dashboards act as information radiators, allowing decision makers to modify setups to cut costs and emissions at the same time. The linkage of cloud regions to carbon intensity of the underlying grid provides a nudge to switch dirty workloads to regions with greener energy sources.

  • JetBrains' collaborative coding tool, Code With Me, has been increasing in popularity as many teams use various JetBrains tools in this remote-first world. Along with other remote collaboration tools such as VSCode's Visual Studio Live Share, Code With Me gives development teams an improved experience with remote pairing and collaboration. Code With Me's abilities to invite teammates into the IDE projects and collaborate in real time are worth exploring. However, we've seen some limitations with regard to refactoring seamlessly and some issues in high-latency environments. We'll continue to watch this tool in this space.

  • This edition of the Radar introduces two tools that search and replace code using an abstract syntax tree (AST) representation. They occupy a similar space as jscodeshift but contain parsers for a wide range of programming languages. Although they share some similarities, they also differ in several ways. One of these tools, Comby, is unique in its simple, command-line interface designed in the spirit of Unix tools such as awk and sed. While the Unix commands are based on regular expressions operating matching text, Comby employs a pattern syntax that is specific to programming language constructs and parses the code before searching. This helps developers search large code bases for structural patterns. Like sed, Comby can replace the patterns it matches with new structures. This is useful for automating wholesale changes to large codebases or for making repetitive changes across a suite of microservice repositories. Since these tools are fairly new, we expect to see a range of creative uses that have yet to be discovered.

  • Conftest is a tool for writing tests against structured configuration data. It relies on the Rego language from Open Policy Agent to write tests for Kubernetes configurations, Tekton pipeline definitions or even Terraform plans. Configurations are a critical part of the infrastructure, and we encourage you to assess Conftest to verify assumptions and get quick feedback.

  • Cosign is a container signing and verification tool. Part of Sigstore — a project under the Cloud Native Computing Foundation (CNCF) umbrella aimed at simplifying software signing and transparency — Cosign supports not only Docker and Open Container Initiative (OCI) images but also other artifacts that can be stored in a container registry. We previously talked about Docker Notary, which also operates in this space; Notary v1, however, has some disadvantages: it's not registry native and needs a separate Notary server. Cosign avoids this problem and stores the signatures in the registry next to an image. It currently has integrations with GitHub actions and Kubernetes using a Webhook with further integrations in the pipeline. We've used Cosign in some of our projects and it looks quite promising.

  • Crossplane is another entry in the class of tools implemented by the Kubernetes Operator pattern but with side effects that extend beyond the Kubernetes cluster. In our last Radar we mentioned Kube-managed cloud services as a technique, and Crossplane does just that. The idea is to leverage the Kubernetes control plane to provision cloud services on which your deployment is dependent, even if they aren't deployed on the cluster itself. Examples include managed database instances, load balancers or access control policies. This tool is noteworthy for two reasons. First, it demonstrates the powerful and flexible execution environment of the underlying Kubernetes control plane. There is no real limit to the range of supported custom resources. Second, Crossplane provides an alternative to the usual options of Terraform, CDK or Pulumi. Crossplane comes with a set of predefined providers for the major cloud services that cover the most commonly provisioned services. It isn't trying to be a general-purpose infrastructure-as-code (IaC) tool but rather a companion to workloads being deployed in Kubernetes. Often associated with the practice of GitOps, Crossplane stands on its own and allows you to stay within the Kubernetes ecosystem when it's necessary to manage external cloud resources. However, Crossplane doesn't help with provisioning Kubernetes itself; you'll need at least one other IaC tool to bootstrap the cluster.

  • gopass is a password manager for teams, built on GPG and Git. It's a descendant of pass and adds several features, including interactive search and multiple password stores in a single tree. Since we first mentioned gopass, our teams have used it on several projects, sometimes stretching it beyond its limits. A sorely missed feature was the ability to deprecate secrets. Discoverability was already an issue, but not being able to mark secrets as no longer in use compounded this problem. The biggest issue, though, was scale. When you have teams with 50+ people using the same repository for several years, we found that the repository could grow to multiple gigabytes in size. Re-encrypting the secrets when onboarding new members could take more than half an hour. The underlying issue seems to be that in our teams everything changes all the time: people come and go, secrets are rotated, the architecture evolves, new secrets are added, old ones are no longer needed. gopass seems to work well, even for large numbers of users, when there's less change.

  • Micoo is a new entrant into the crowded space of visual regression tools; it's an open-source solution and is self-contained, providing Docker images to enable an easy and quick environment setup. It also provides different clients for Node.js, Java and Python as well as a Cypress plugin so it can be easily integrated with most of the common frontend UI automation testing frameworks or solutions. Although Micoo doesn't provide all the functionality of some of the SaaS-based or other commercial solutions, our teams have been using it extensively and have had positive experiences. They've especially called out that it works for mobile and desktop apps as well as the web.

  • Sometimes you come across a tool that you didn't realize you needed until you do; mob is just such a tool. Living as we do in a world where remote pair programming has become the norm for many teams, having a tool that allows for seamless handover either between pairs or a wider group as part of a mob programming session is super useful. mob hides all the version control paraphernalia behind a command-line interface that makes participating in mob programming sessions simpler. It also provides specific advice around how to participate remotely, for example, to "steal the screenshare" in Zoom rather than ending a screenshare, ensuring the video layout doesn't change for participants. A useful tool and thoughtful advice, what's not to like?

  • There are many reasons to love Unix, but the one that has profoundly affected our industry is the Unix philosophy of building applications that "do one thing and do it well." Unix commands embody this philosophy. A set of small functions that can be piped together to create more complex solutions. In recent years, programmers have contributed to a growing set of modern Unix commands. These modern versions attempt to be smaller and faster, often written in Rust. They include additional features such as syntax highlighting and utilize features of modern terminals. They aim to support programmers natively by integrating nicely with git and recognizing source code files. For example, bat is a replacement for cat with paging and syntax highlighting; exa is a replacement for ls with extended file information and ripgrep is a faster grep replacement that by default ignores gitignore, binary and hidden files. The Modern Unix repository has a reference to some of these commands. We've been enjoying using these Unix commands. You should try them in improving your command-line experience. However, we caution against using them in scripts as replacements for the standard command-line utilities that are shipped in default OS distributions, because they reduce the scripts' portability running on other machines.

  • Plaintext secrets checked into source control (usually Github) are one of the most pervasive security mistakes developers make. For this reason we thought it useful to feature Mozilla Sops, a tool for encrypting secrets in text files that our developers find useful in situations where it is impossible to remove secrets from legacy code repositories. We've mentioned many tools of this type before (Blackbox, git-crypt), but Sops has several features that set it apart. For example, Sops integrates with cloud-managed keystores such as AWS and GCP Key Management Service (KMS) or Azure Key Vault as sources of encryption keys. It also works cross-platform and supports PGP keys. This enables fine-grained access control to secrets on a file-by-file basis. Sops leaves the identifying key in plain text so that secrets can still be located and diffed by git. We're always supportive of anything that makes it easier for developers to be secure; however, remember that you don't have to keep secrets in source control to begin with. See Decoupling secret management from source code in our November 2017 issue.

  • We continue to see the adoption of Kubernetes in new and novel scenarios. For example, we see Kubernetes is being extended to manage resources running outside of its cluster or across multiple infrastructure providers, or it is used in managing stateful applications beyond Kubernetes's original scope. These extensions are possible using the Kubernetes Operator pattern: building Kubernetes controllers that have the domain-specific knowledge of the custom resource they manage. For example, an operator that manages a stateful application can use the Kubernetes primitives to automate an application's specific tasks beyond its deployment, such as restore, backup and upgrade its database.

    Operator Framework is a set of open-source tools that simplifies building and managing the lifecycle of Kubernetes operators. Although there are multiple frameworks to help you build Kubernetes operators, Operator Framework remains a good choice. It supports rich operator lifecycle management using its Operator Lifecycle Manager module; it supports multiple languages to build the operator code itself using its Operator SDK; and it provides a catalog for publishing and sharing the operators. If you're planning to build Kubernetes operators, we recommend giving the Operator Framework a try to accelerate your development reliably.

  • For organizations with larger and more complex API ecosystems, especially those who are already using Pact, we think it's worth assessing whether Pactflow could be useful. Pactflow manages the workflow and continuous deployment of tests written in Pact, lowering the barrier to consumer-driven contract testing. The complexity of coordination between multiple producers and various disparate consumers can become prohibitive. We've seen some teams invest significant effort in hand-crafting solutions to this problem and think it's worth assessing whether Pactflow can look after this for you.

  • Prefect is a data workflow management tool that makes it easy to add semantics such as retries, dynamic mapping, caching and failure notifications to data pipelines. You can mark Python functions as tasks and chain them together through function calls to build the data flow. The Python API combined with a collection of predefined tasks for common data operations makes Prefect a noteworthy option to assess for your data pipeline needs.

  • It may not be a tool that you need everyday, but when you're in the weeds trying to diagnose a nasty network problem, it's very useful to be able to reach for a feature-rich HTTP debugging proxy. Proxyman is just such a tool. Quite a few of our teams have been using it for a while now as a macOS-specific drop-in replacement for Charles and really like its streamlined interface and cert management.

  • One of the key tenets of infrastructure as code (IaC) is automated testing. If we have a solid test pyramid with good code-level coverage at the bottom, we can produce a better and more secure infrastructure. Unfortunately, tools to assist in this space have been sparse. Conftest is frequently used to test Terraform JSON and HCL code, but it is a general-purpose tool. Regula is an attractive alternative. Similar to Conftest, Regula checks for compliance of infrastructure code by applying rules written in Open Policy Agent's Rego language, but it also provides a set of primitives specifically for validating infrastructure configurations. Because both tools are based on the Rego language, Regula rules can be run by Conftest. However, Regula comes with its own command-line tool for running tests as part of a pipeline with no dependence on Conftest or OPA. Our developers have found that Regula saves time and produces much more readable, maintainable and succinct test code. Still, both tools only validate the infrastructure code. A complete suite should also test the infrastructure to ensure the code is being accurately interpreted.

  • Another abstract syntax tree–based code search tool that received our attention is Sourcegraph. In contrast to Comby, which is open source, Sourcegraph is a commercial tool (with a 10-user free tier). Sourcegraph is particularly suited for searching, navigating or cross-referencing in large codebases. The cloud-hosted version can be accessed through Sourcegraph's website and is designed to search publicly available open-source repositories. Whereas Comby is a lightweight command-line tool for automating repetitive tasks, Sourcegraph's emphasis is on interactive developer tools for understanding and navigating large code bases. Unlike Comby's sed-like interface, Sourcegraph's automated code rewriting capability is driven from a UI, allowing users to review changes before they're made. Because Sourcegraph is a hosted service, it also has the ability to continuously monitor code bases and send alerts when a match occurs.

  • Telepresence is a tool that helps shorten the feedback loop of changes that usually require a deployment for proper testing. Developers can use it to plug a process that is running on their local machines into a remote Kubernetes cluster. This gives the local process access to the remote cluster's services and features, and the local service can also temporarily replace one of the cluster services.

    In situations where the service integration setup has become somewhat unwieldy, Telepresence can boost developer productivity and enable more effective local testing. However, if you get into the habit of using a clever tool like this, you may have bigger problems. For example, if you use Telepresence because it has become impossible to set up all necessary dependencies for local development, you may want to investigate the complexity of your setup and architecture. If it becomes the only way for you to do service integration tests, consider looking into consumer-driven contract testing or other automated ways of integration testing.

  • Fast feedback is crucial for a good developer experience. Nothing breaks the flow of development more than having to wait a minute or two before getting feedback on the last code changes. Unfortunately, with applications growing in size and complexity, the popular build tools for front-end pipelines are often not fast enough anymore. Previously, we featured esbuild, which offers a significant performance improvement, because it's implemented in a compile-to-native language rather than JavaScript. Vite, which is built on top of esbuild, delivers significant improvements over other tools. It consists of two major parts: a dev server that provides rich feature enhancements over native ES modules, such as extremely fast Hot Module Replacement (HMR), and a build command that bundles your code with Rollup. Vite relies on ES modules, and unlike most older tools, it doesn't provide shimming or polyfills, which means it's not compatible with older browsers that don't support ES modules. In cases where older browsers had to be supported, some of our teams used Vite during development and other tools for production builds.

Hold ?

 
New
Moved in/out
No change

Unable to find something you expected to see?

 

Each edition of the Radar features blips reflecting what we came across during the previous six months. We might have covered what you are looking for on a previous Radar already. We sometimes cull things just because there are too many to talk about. A blip might also be missing because the Radar reflects our experience, it is not based on a comprehensive market analysis.

Unable to find something you expected to see?

 

Each edition of the Radar features blips reflecting what we came across during the previous six months. We might have covered what you are looking for on a previous Radar already. We sometimes cull things just because there are too many to talk about. A blip might also be missing because the Radar reflects our experience, it is not based on a comprehensive market analysis.

Radar

Download Technology Radar Volume 25

English | Español | Português | 中文

Radar

Stay informed about technology

 

Subscribe now

Visit our archive to read previous volumes