Tools

Great Expectations has become a sensible default for our teams in the data quality space, which is why we recommend adopting it — not only for the lack of better alternatives but also because our teams have reported great results in several client projects. Great Expectations is a framework that allows you to craft built-in controls that flag anomalies or quality issues in data pipelines. Just as unit tests run in a build pipeline, Great Expectations makes assertions during the execution of a data pipeline. We like its simplicity and ease of use — the rules stored in JSON can be modified by our data domain experts without necessarily needing data engineering skills.

Since we first mentioned it in the Radar, k6 has become a go-to tool for performance testing. We continue to be fans of how easy it is to write JavaScript code for tests, but k6 also has a low-code test builder to make playing with the tool even easier. The documentation shows how easy it is to add performance testing to a pipeline across multiple CI/CD tools. Our teams find it easy to integrate visualization tools like Grafana and New Relic, which help them tune both infrastructure and applications. The developer friendliness and ecosystem make k6 a compelling option for investigating a system's behavior under heavy load.

Apache Superset is a great business intelligence (BI) tool for data exploration and visualization to work with large data lake and data warehouse setups. It supports several data sources — including AWS Redshift, BigQuery, Azure MS SQL, Snowflake and ClickHouse. Moreover, you don't have to be a data engineer to use it; it's meant to benefit all engineers exploring data in their everyday work. For demanding use cases, we found it easy to scale Superset by deploying it in a Kubernetes cluster. Since we last talked about it in the Radar, Superset has graduated as an Apache product, and we've seen great success in several projects.

When implementing robust, secure and reliable disaster recovery, it’s necessary to ensure that backups can't be deleted or altered before their expiry, either maliciously or accidentally. Previously, with AWS Backup, these policies and guarantees had to be implemented by hand. Recently, AWS has added the Vault Lock feature to ensure backups are immutable and untamperable. AWS Backup Vault Lock enforces retention and deletion policies and prevents even those with administrator privileges from altering or deleting backup files. This has proved to be a valuable addition and fills a previously empty space.

Multi-team account management is a challenge in AWS, especially in setup and governance; AWS Control Tower is an attempt to address this challenge. Our team has reported good results using it to manage accounts and access control for multiple teams in the organization through a single, centralized place.

We've had success with Clumio Protect for backing up AWS data, particularly S3. A commercial SaaS solution, Clumio Protect can also back up a range of other AWS services and stores the data offline where it is not accessible through the internet. Our teams responsible for handling data protection and recovery at massive scale found that Clumio Protect is easy to set up and maintain and far outperforms the native AWS Backup service when S3 buckets are particularly big.

We've been talking about tailored service templates ever since we first identified microservices as a thing. If an organization sets out to create a collection of small services that can be developed, built, deployed and operated independently but consistently, it makes sense to give teams a solid starting point that aligns to the standard. However, one of the enduring problems with that approach is that as the template evolves over time in response to changing technical and business requirements, projects based on older versions of the template fall out of date. Retrofitting template improvements into an established project becomes a major pain. Cruft attempts to address this problem by providing tools to identify and patch differences between a local project and the current head of a master template repository. It combines the Cookiecutter templating engine with git hashes to identify and apply changes to the templates. Think of it as a package manager for a project boilerplate. Keeping templates up-to-date is a notoriously difficult and long-standing problem, so to us the solution Cruft provides sounds almost too good to be true. Based on early feedback from our team, however, Cruft actually works and makes life easier for service builders and maintainers. We're anxious to see how it performs over the long term, but for now it's worth taking a look at this potentially useful tool.

We continue to hear enthusiastic reports about Excalidraw from our teams, but our previous caveat about security remains in place. Excalidraw is a simple yet powerful online drawing tool. Sometimes teams just need a quick picture instead of a formal diagram; for remote teams, Excalidraw provides a quick way to create and share diagrams. Our teams also like the "lo-fi" look of the diagrams it can produce, which is reminiscent of the whiteboard diagrams they would have produced when co-located. Regarding security, at the time of writing, anyone who has the link can see your diagrams; note, though, that the paid version of Excalidraw provides further authentication and options to run a server locally do exist.

We like spreading the word about linting tools that actually help you find issues rather than just shortcut style disputes in the team. Hadolint is one of those tools — it helps find common issues in Dockerfiles. We find it to be fast, accurate and with good documentation. It explains both how to fix an issue and why it's an issue in the first place, thus nudging Dockerfile authors toward good practices. Incidentally, Hadolint is built on top of ShellCheck, which we recommend in its own right for checking your shell scripts.

Most of today's CI/CD pipeline tools and platforms are built on containers as runtimes. Many of our teams are using Kaniko to build container images from within those container-based pipelines. This comes as part of a trend away from Docker as the de facto standard for container runtimes. With Kaniko, you can build your images without using a Docker daemon. This helps avoid the security issue of Docker's "privileged" mode, which would be necessary for any "Docker-in-Docker" activity. Moreover, you don't have to assume that your pipeline has access to a Docker daemon in the first place, which cannot be taken for granted anymore and often requires extra configuration.

As data work becomes more common, we continue to see tools that try to enhance the SQL language; Kusto Query Language (KQL) is one of them. KQL was created by Azure, and it brings modularity, encapsulation, composability, reusability, extensibility and dynamism to relational querying. Our teams quite like its interactivity: you can pipe a query to the render operator and see a chart instantly. You can also combine these charts into dashboards and get insights from logs to execs in minutes. Although the KQL language is currently limited to the Azure Data Explorer, we anticipate the move to enhance SQL to achieve better data operability will not stop.

Spectral is a JSON/YAML linter with an emphasis on OpenAPI and AsyncAPI specifications. It ships with a comprehensive set of out-of-the-box rules for these specs that can save developers headaches when designing and implementing APIs or event-driven collaboration. These rules check for proper API parameter specifications or the existence of a license statement in the spec, among other things. The CLI makes it easy to incorporate Spectral into both local development and CI/CD pipelines, and the JavaScript API supports more advanced use cases. The GitHub site links to publicly available real-world rule sets from companies like Adidas, giving teams a head start on adopting their own linting rules.

Styra Declarative Authorization Service (DAS) is a governance and automation tool for managing Open Policy Agent (OPA) at scale. Built by the creators of OPA, the tool allows us to deploy policies across "systems," including Kubernetes clusters, infrastructure code repositories, namespaces and more. Most importantly, it allows for real-time analysis of decisions made by an OPA agent, along with replayability for debugging and investigating what-if scenarios for policy changes. It also comes with an audit log that can help security teams with historical reporting.

On remote teams, we sorely lack having a dedicated build monitor in the room; unfortunately, newer continuous integration (CI) tools lack support for the old CCTray format. The result is that broken builds aren't always picked up as quickly as we'd like. To solve this problem, many of our teams have started using xbar for build monitoring. With xbar, one can execute a script to poll build status, displaying it on the menu bar. It can be further scripted to track other team metrics such as pending credential expiries or how far the production release lags behind the user acceptance testing (UAT) release. Of course, xbar is more general purpose, but it solves an immediate and emergent problem caused by remote working. Rumps, among other tools, can solve the same problem.

Unfortunately, a big part of the world still runs on spreadsheets and will continue to do so. They're the ultimate tool to let anyone build those small custom tools tailored to their exact needs. However, when you want to enhance them with a level of logic that requires "real" code, the low-code nature of spreadsheets can then become a constraint. If you're with a company that, like Thoughtworks, uses Google's G-Suite, Clasp enables you to apply at least some Continuous Delivery practices to Apps Script code. You can write the code outside of the Apps Script project, which creates options for testing, source control and build pipelines; it even lets you use TypeScript. Clasp has been around for a while, and you shouldn’t expect a programming environment with all of the usual comforts, but it can greatly improve the experience of using Apps Script.

Databricks Overwatch is a Databricks Labs project that enables teams to analyze various operational metrics of Databricks workloads around cost, governance and performance with support to run what-if experiments. It's essentially a set of data pipelines that populate tables in Databricks, which can then be analyzed using tools like notebooks. Overwatch is very much a power tool; however, it's still in its early stages and it may take some effort to set it up — our use of it required Databricks solution architects to help set it up and populate a price reference table for cost calculations — but we expect adoption to get easier over time. The level of analysis made possible by Overwatch is deeper than what is allowed by cloud providers' cost analysis tools. For example, we were able to analyze the cost of job failures — recognizing that failing fast saves money compared to jobs that only fail near the final step — and break down the cost by various groupings (workspace, cluster, job, notebook, team). We also appreciated the improved operational visibility, as we could easily audit access controls around cluster configurations and analyze operational metrics like finding the longest running notebook or largest read/write volume. Overwatch can analyze historical data, but its real-time mode allows for alerting which helps you to add appropriate controls to your Databricks workloads.

Data Vault 2.0 is a data modeling methodology and design pattern intended to improve the flexibility of data warehouses compared to other popular modeling approaches. Data Vault 2.0 can be applied to any data store such as Snowflake or Databricks. When implementing Data Vault warehouses, we've found the dbtvault package for dbt to be a helpful tool. dbtvault provides a set of jinja templates that generate and execute the ETL scripts necessary to populate a Data Vault warehouse. Although dbtvault has some rough edges — it lacks support for enforcing implied uniqueness or performing incremental loads — overall, it fills a niche and requires minimal configuration to get started.

We're always looking for ways to remove small frictions from pair programming, which is why we're excited by git-together, a tool written in Rust that simplifies git commit attribution during pairing. By aliasing git-together as git, the tool allows you to add extensions to git config that capture committer information, aliasing each committer by their initials. Changing pairs (or switching to soloing or mob programming) requires you to run git with, followed by the initials of the pair (for example: git with bb cc), allowing you to resume your regular git workflow afterward. Every time you commit, git-together will rotate through the pair as the official author that git stores, and it will automatically add any other authors to the bottom of the commit message. The configuration can be checked in with the repo, allowing git-together to work automatically after cloning a repo.

Harness Cloud Cost Management is a commercial tool that works across all three of the major cloud providers and their managed Kubernetes clusters to help visualize and manage cloud costs. The product calculates a cost efficiency score by looking at idle resources as well as resources not allocated to any workload and uses historical trends to help optimize resource allocation. The dashboards highlight cost spikes and allow a user to register unexpected anomalies, which are then fed into their reinforcement learning algorithm around anomaly detection. Cloud Cost Management can recommend adjustments to limits for memory and CPU usage, with options to optimize for either cost or performance. "Perspectives" allows you to group costs based on organizationally defined filters (which could correspond to business units, teams or products) and automate report distribution to bring visibility into cloud spend. We believe Cloud Cost Management offers a compelling feature set to help organizations mature their FinOps practices.

We continue to see organizations move to the cloud without properly understanding how they will track ongoing spend. We previously blipped run cost as architecture fitness function, and Infracost is a tool that aims to make these cloud cost trade-offs visible in Terraform pull requests. It's open-source software and available for macOS, Linux, Windows and Docker and supports pricing for AWS, GCP and Microsoft Azure out of the box. It also provides a public API that can be queried for current cost data. We remain excited by its potential, especially when it comes to gaining better cost visibility in the IDE.

One of the fundamental capabilities of Kubernetes is its ability to automatically launch new pods when additional capacity is needed and shut them down when loads decrease. This horizontal autoscaling is a useful feature, but it can only work if the nodes needed to host the pods already exist. While Cluster Autoscaler can do some rudimentary cluster expansion triggered by pod failures, it has limited flexibility; Karpenter, however, is an open-source Kubernetes Operator autoscaler with more smarts built in: it analyzes the current workloads and the pod scheduling constraints to automatically select an appropriate instance type and then start or stop it as needed. Karpenter is an operator in the spirit of tools like Crossplane that can provision cloud resources outside the cluster. Karpenter is an attractive companion to the autoscaling services cloud vendors provide natively with their managed Kubernetes clusters. For example, AWS now supports Karpenter as a first-class alternative in their EKS Cluster Autoscaler service.

Mizu is an API traffic viewer for Kubernetes. Unlike other tools, Mizu does not require instrumentation or code changes. It runs as a DaemonSet to inject a container at the node level in your Kubernetes cluster and performs tcpdump-like operations. We find it useful as a debugging tool, as it can observe all API communications across multiple protocols (REST, gRPC, Kafka, AMQP and Redis) in real time.

Soda Core is an open-source data quality and observability tool. We talked about Great Expectations previously in the Radar, and Soda Core is an alternative with a key difference — you express the data validations in a DSL called SodaCL (previously called Soda SQL) as opposed to Python functions. Once the validations are written, it can be executed as part of a data pipeline or scheduled to run programmatically. As we become increasingly data-driven, it's critical to maintain data quality, and we encourage you to assess Soda Core.

Teller is an open-source universal secret manager for developers that ensures the correct environment variables are set when starting an application. However, it's not a vault itself — it's a CLI tool that connects to a variety of sources, ranging from cloud secrets providers to third-party solutions like HashiCorp Vault to local environment files. Teller has additional functionality to scan for vault-kept secrets in your code, to redact secrets from logs, to detect drift between secrets providers and to sync between them. Given the sensitivity of accessing secrets, we can't emphasize enough the need to secure the supply chain for open-source dependencies, but we appreciate how easy the CLI is to use in local development environments, CI/CD pipelines and deployment automation.

Xcode Cloud is a CI/CD tool that is built into Xcode and used to build, test and deploy Apple apps. It provides an integrated experience with familiar tools for Apple developers like Xcode, App Store Connect and TestFlight. Based on our team's experience, it does a good job of simplifying the pipeline configuration and provisioning profiles and certificates. This tool is quite fresh and most of our mobile development teams are still using the more mature Bitrise. Still, we think it's worth assessing and tracking its progress.

We previously called out production data in test environments and now want to highlight another common practice that needs to be approached with care or even stopped entirely: online services for formatting or parsing code. There are many useful sites for formatting or parsing formats such as JSON and YAML, as well as sites that assess code tutorials or produce online code metrics. Great care is needed when using these. Pasting a block of JavaScript, JSON or similar into an unknown website can easily create security and privacy issues and might unknowingly export personal data into a different jurisdiction. These sites should never be used with production data and should be approached with caution in all other circumstances.