Enable javascript in your browser for better experience. Need to know to enable it? Go here.

Tools

Adopt ?

  • For our projects using Terraform, tfsec has quickly become a default static analysis tool to detect potential security risks. It's easy to integrate into a CI pipeline and has a growing library of checks against all of the major cloud providers and platforms like Kubernetes. Given its ease of use, we believe tfsec could be a good addition to any Terraform project.

Trial ?

  • AKHQ is a GUI for Apache Kafka that lets you manage topics, topics data, consumer groups and more. Some of our teams have found AKHQ to be an effective tool to watch the real-time status of a Kafka cluster. You can, for example, browse the topics on a cluster. For each topic, you can visualize the name, the number of messages stored, the disk size used, the time of the last record, the number of partitions, the replication factor with the in-sync quantity and the consumer group. With options for Avro and Protobuf deserialization, AKHQ can help you understand the flow of data in your Kafka environment.

  • cert-manager is a tool to manage your X.509 certificates within your Kubernetes cluster. It models certificates and issuers as first-class resource types and provides certificates as a service securely to developers and applications working within the Kubernetes cluster. The obvious choice when using the Kubernetes default ingress controller, it's also recommended for others and much preferred over hand-rolling your own certificate management. Several of our teams have been using cert-manager extensively, and we've also found that its usability has much improved in the past few months.

  • Cloud Carbon Footprint (CCF) is an open-source tool that uses cloud APIs to provide visualizations of estimated carbon emissions based on usage across AWS, GCP and Azure. The Thoughtworks team has successfully used the tool with several organizations, including energy technology companies, retailers, digital service providers and companies that use AI. Cloud platform providers realize that it's important to help their customers understand the carbon impact of using their services, so they've begun to build similar functionality themselves. Because CCF is cloud agnostic, it allows users to view energy usage and carbon emissions for multiple cloud providers in one place, while translating carbon footprints into real-world impact such as flights or trees planted.

    In recent releases, CCF has begun to include Google Cloud and AWS-sourced optimization recommendations alongside potential energy and CO2 savings, as well as to support more cloud instance types such as GPU instances. Given the traction the tool has received and the continued addition of new features, we feel confident moving it to Trial.

  • Conftest is a tool for writing tests against structured configuration data. It relies on the Rego language from Open Policy Agent to write tests for Kubernetes configurations, Tekton pipeline definitions or even Terraform plans. We've had great experiences with Conftest — and its shallow learning curve. With fast feedback from tests, our teams iterate quickly and safely on configuration changes to Kubernetes.

  • kube-score is a tool that does static code analysis of your Kubernetes object definitions. The output is a list of recommendations for what you can improve to make your application more secure and resilient. It has a list of predefined checks which includes best practices such as running containers with non-root privileges and correctly specifying resource limits. It's been around for some time, and we've used it in a few projects as part of a CD pipeline for Kubernetes manifests. A major drawback of kube-score is that you can't add custom policies. We typically supplement it with tools like Conftest in these cases.

  • Lighthouse is a tool written by Google to assess web applications and web pages, collecting performance metrics and insights on good development practices. We've long advocated for performance testing as a first-class citizen, and the additions to Lighthouse that we mentioned five years ago certainly helped with that. Our thinking around architectural fitness functions created strong motivation for tools such as Lighthouse to be run in build pipelines. With the introduction of Lighthouse CI, it has become easier than ever to include Lighthouse in pipelines managed by various tools.

  • Metaflow is a user-friendly Python library and back-end service that helps data scientists and engineers build and manage production-ready data processing, ML training and inference workflows. Metaflow provides Python APIs that structure the code as a directed graph of steps. Each step can be decorated with flexible configurations such as the required compute and storage resources. Code and data artifacts for each step's run (aka task) are stored and can be retrieved either for future runs or the next steps in the flow, enabling you to recover from errors, repeat runs and track versions of models and their dependencies across multiple runs.

    The value proposition of Metaflow is the simplicity of its idiomatic Python library: it fully integrates with the build and run-time infrastructure to enable running data engineering and science tasks in local and scaled production environments. At the time of writing, Metaflow is heavily integrated with AWS services such as S3 for its data store service and step functions for orchestration. Metaflow supports R in addition to Python. Its core features are open sourced.

    If you're building and deploying your production ML and data-processing pipelines on AWS, Metaflow is a lightweight full-stack alternative framework to more complex platforms such as MLflow.

  • Micrometer is a platform-agnostic library for metrics instrumentation on the JVM that supports Graphite, New Relic, CloudWatch and many other integrations. We've found that Micrometer has benefited both library authors and teams: library authors can include metrics instrumentation code in their libraries without needing to support each and every metrics system that their users are using; and teams can support many different metrics on back-end registries which enables organizations to collect metrics in a consistent way.

  • NUKE is a build system for .NET and an alternative to either the traditional MSBuild or Cake and Fake which we've featured previously in the Radar. NUKE represents build instructions as a C# DSL, making it easy to learn and with good IDE support. In our experience, NUKE made it really simple to build automation for .NET projects. We like the accurate static code checks and hints. We also like that we can use any NuGet package seamlessly and that the automation code can be compiled to avoid problems at runtime. NUKE isn't new, but its novel approach — using a C# DSL — and our positive overall experience prompted us to include it here.

  • We've used Pact for contract testing long enough to see some of the complexity that comes with scale. Some of our teams have successfully used Pactflow to reduce that friction. Pactflow runs both as software as a service and as an on-prem deployment with the same features as the SaaS offering, and it adds improved usability, security and auditing on top of the open-source Pact Broker offering. We've been pleased with our use so far and are happy to see continued effort to remove some of the overhead of managing contract testing at scale.

  • As an alternative to Docker, Podman has been validated by many of our teams. Podman introduces a daemonless engine for managing and running containers which is an interesting approach in comparison to what Docker does. Additionally, Podman can be easily run as a normal user without requiring root privileges, which reduces the attack surface. By using either Open Container Initiative (OCI) images built by Buildah or Docker images, Podman can be adapted to most container use cases. Apart from some compatibility issues with macOS, our team has had generally good experiences with Podman on Linux distributions.

  • In our previous Radar, we featured two tools that search and replace code using an abstract syntax tree (AST) representation, Comby and Sourcegraph. Although they share some similarities, they also differ in several ways. Sourcegraph is a commercial tool (with a 10-user free tier). It's particularly suited for searching, navigating or cross-referencing in large codebases, with an emphasis on an interactive developer experience. In contrast, Comby is a lightweight open-source command-line tool for automating repetitive tasks. Because Sourcegraph is a hosted service, it also has the ability to continuously monitor code bases and send alerts when a match occurs. Now that we've gained more experience with Sourcegraph, we decided to move it into the Trial ring to reflect our positive experience — which doesn't mean that Sourcegraph is better than Comby. Each tool focuses on a different niche.

  • One of the key elements of improving "supply chain security" is using a Software Bill of Materials (SBOM), which is why publishing an SBOM along with the software artifact is increasingly important. Syft is a CLI tool and Go library for generating an SBOM from container images and file systems. It can generate the SBOM output in multiple formats, including JSON, CycloneDX and SPDX. The SBOM output of Syft can be used by Grype for vulnerability scanning. One way to publish the generated SBOM along with the image is to add it as an attestation using Cosign. This allows consumers of the image to verify the SBOM and to use it for further analysis.

  • When working on multiple JavaScript codebases at the same time, it's often necessary to use different versions of Node and other JavaScript tools. On developer machines, these tools are usually installed in the user account or the machine itself, which means a solution is needed to switch between multiple installations. For Node itself there's nvm, but we want to highlight Volta as an alternative that we're seeing in use with our teams. Volta has several advantages over using nvm: it can manage other JavaScript tools such as Yarn; it also has the notion of pinning a version of the toolchain on a project basis, which means that developers can simply use the tools in a given code directory without having to worry about manually switching between tool versions — Volta simply uses shims in the path to select the pinned version. Written in Rust, Volta is fast and ships as a single binary without dependencies.

  • Web Test Runner is a package within the Modern Web project, which provides several high-quality tools for modern web development with support for web standards like ES Modules. Web Test Runner is a test runner for web applications. One of its advantages compared to existing test runners is that it runs tests in the browser (which could be headless). It supports multiple browser launchers — including Puppeteer, Playwright, and Selenium — and uses Mocha by default for the test framework. The tests run pretty fast, and we like that we can open a browser window with devtools when debugging. Web Test Runner internally uses Web Dev Server which allows us to leverage its great plugin API for adding customized plugins for our test suite. Modern Web tools look like a very promising developer toolchain, and we're already using it in a few projects.

Assess ?

  • By now many organizations have created sprawling landscapes of services in the cloud. Of course, this is only possible when using infrastructure as code and mature tooling. We still like Terraform, not the least because of its rich and growing ecosystem. However, the lack of abstractions in HCL, Terraform's default configuration language, effectively creates a glass ceiling. Using Terragrunt pushes that up a bit further, but more and more often our teams find themselves longing for the abstractions afforded by modern programming languages. Cloud Development Kit for Terraform (CDKTF), which resulted from a collaboration between AWS's CDK team and Hashicorp, makes it possible for teams to use several programming languages, including TypeScript and Java, to define and provision infrastructure. With this approach it follows the lead of Pulumi while remaining in the Terraform ecosystem. We've had good experiences with CDKTF but have decided to keep it in the Assess ring until it moves out of beta.

  • Chrome Recorder panel is a preview feature in Google Chrome 97 that allows for simple record and playback of user journeys. While this definitely isn't a new idea, the way in which it is integrated into Chrome allows for quick creation, editing and running of scripts. The panel also integrates nicely with the performance panel, which makes getting repeated consistent feedback on page performance easier. While record/playback style testing always needs to be used with care in order to avoid brittle tests, we think this preview feature is worth assessing, especially if you're already using the Chrome Performance panel to measure your pages.

  • Excalidraw is a simple but powerful online drawing tool that our teams enjoy using. Sometimes teams just need a quick picture instead of a formal diagram, for remote teams Excalidraw provides a quick way to create and share diagrams. Our teams also like the "lo-fi" look of the diagrams it can produce, which is reminiscent of the whiteboard diagrams they would have produced when co-located. One caveat: you need to pay attention to the default security — at the time of writing, anyone who has the link can see the diagram. A paid-for version provides further authentication.

  • GitHub Codespaces allows developers to create development environments in the cloud and access them through an IDE as though the environment were local. GitHub isn't the first company to implement this idea; we previously blipped about Gitpod. We like that Codespaces allows environments to be standardized by using dotfiles configuration, making it quicker to onboard new team members, and that they offer VMs with up to 32 cores and 64GB memory. These VMs can be spun up in under ten seconds, potentially offering environments more powerful than a developer laptop.

  • GoReleaser is a tool that automates the process of building and releasing a Go project for different architectures via multiple repositories and channels, a common need for Go projects targeting different platforms. You run the tool either from your local machine or via CI, with the tool available via several CI services thus minimizing set-up and maintenance. GoReleaser takes care of build, packaging, publishing and announcement of each release and supports different combinations of package format, package repository and source control. Although it's been around for a few years, we're surprised that more teams are not using it. If you're regularly releasing a Go codebase, this tool is worth assessing.

  • Securing the software supply chain has become a commonplace concern among delivery teams, a concern that is reflected by the growing number of new tools in this space. Grype is a new lightweight vulnerability scanning tool for Docker and OCI images. It can be installed as a binary, can scan images before they're pushed to a registry, and it doesn't require a Docker daemon to run on your build agents. Grype comes from the same team that is behind Syft, which generates SBOMs in various formats from container images. Grype can consume the SBOM output of Syft to scan for vulnerabilities.

  • One often-cited advantage of moving to the cloud is transparency around infrastructure spend. In our experience, this is often not the case. Teams don't always think about the decisions they make around infrastructure in terms of financial cost which is why we previously blipped about run cost as architecture fitness function. We're intrigued by the release of a new tool called Infracost which aims to make cost trade-offs visible in Terraform pull requests. It's open-source software and available for macOS, Linux, Windows and Docker and supports pricing for AWS, GCP and Microsoft Azure out of the box. It also provides a public API that can be queried for current cost data. Our teams are excited by its potential, especially when it comes to gaining better cost visibility in the IDE.

  • In our previous Radar, we placed modern Unix commands in Assess. One of the commands featured in that collection of tools was jq, effectively a sed for JSON. jc performs a related task: it takes the output of common Unix commands and parses the output into JSON. The two commands together provide a bridge between the Unix CLI world and the raft of libraries and tools that operate on JSON. When writing simple scripts, for example, for software deployment or gathering troubleshooting information, having the myriad of different Unix command output formats mapped into well-defined JSON can save a lot of time and effort. As with jq, you need to make sure the command is available. It can be installed from many of the well-known package repositories.

  • skopeo is a command line utility that performs various operations on container images and image repositories. It doesn't require a user to be root to do most of its operations nor does it require a daemon to be running. It's a useful part of a CI pipeline; we've used it to copy images from one registry to another as we promote the images. It's better than doing a pull and a push as we don't need to store the images locally. It's not a new tool, but it's useful enough and underutilized that we felt it's worth calling it out.

  • While linting is an ancient practice in the software world, it's had slower adoption in the data world. SQLFluff is a cross-dialect SQL linter written in Python that ships with a simple command line interface (CLI), making it easy to incorporate into a CI/CD pipeline. If you're comfortable with the default conventions, then SQLFluff works without any additional configuration after installing it and will enforce a strongly opinionated set of formatting standards; setting your own conventions involves adding a configuration dotfile. The CLI can automatically fix certain classes of violations that involve formatting concerns like whitespace or uppercasing of keywords. SQLFluff is still new, but we're excited to see SQL getting some attention in the linting world.

  • Organizations that have adopted infrastructure as code and self-service infrastructure platforms are looking for ways to give teams a maximum of autonomy while still enforcing good security practices and organizational policies. We've highlighted tfsec before and are moving it into the Adopt category in this Radar. For teams working on GCP, Terraform Validator could be an option when creating a policy library, a set of constraints that are checked against Terraform configurations.

  • Typesense is a fast, typo-tolerant text search engine. For use cases with large volumes of data, Elasticsearch might still be a good option as it provides a horizontally scalable disk-based search solution. However, if you're building a latency-sensitive search application with a search index size that can fit in memory, Typesense is a powerful alternative and another option to evaluate alongside tools such as Meilisearch.

Hold ?

 
  • tools quadrant with radar rings Adopt Trial Assess Hold Adopt Trial Assess Hold
  • New
  • Moved in/out
  • No change

Unable to find something you expected to see?

 

Each edition of the Radar features blips reflecting what we came across during the previous six months. We might have covered what you are looking for on a previous Radar already. We sometimes cull things just because there are too many to talk about. A blip might also be missing because the Radar reflects our experience, it is not based on a comprehensive market analysis.

Unable to find something you expected to see?

 

Each edition of the Radar features blips reflecting what we came across during the previous six months. We might have covered what you are looking for on a previous Radar already. We sometimes cull things just because there are too many to talk about. A blip might also be missing because the Radar reflects our experience, it is not based on a comprehensive market analysis.

Radar

Download Technology Radar Volume 26

English | Español | Português | 中文

Radar

Stay informed about technology

 

Subscribe now

Visit our archive to read previous volumes