Tools
Adopt
-
DVC continues to be our tool of choice for managing experiments in data science projects. The fact that it's Git-based makes it a known turf for developers to bring engineering practices to the data science ecosystem. DVC's opinionated view of a model checkpoint carefully encapsulates a training data set, a test data set, model hyperparameters and the code. By making reproducibility a first-class concern, it allows the team to time travel across various versions of the model. Our teams have successfully used DVC in production to enable continuous delivery for ML (CD4ML); it can be plugged in with any type of storage (including AWS S3, Google Cloud Storage, MinIO and Google Drive). However, with data sets getting bigger, file system–based snapshotting could become particularly expensive. When the underlying data is changing rapidly, DVC on top of a good versioned storage allows tracking model drifts over a period of time. Our teams have effectively used DVC on top of data storage formats like Delta Lake which optimizes versioning (COW). A majority of our data science teams set up DVC as a day zero task while they bootstrap a project; for this reason we're happy to move it to Adopt.
Trial
-
As more organizations adopt cloud computing, many are starting to integrate multiple cloud providers simultaneously to maximize flexibility and minimize vendor lock-in. However, managing keys and access controls across multiple cloud providers can be a significant challenge, leading to increased complexity and security risks. Akeyless is a centralized, cloud-based platform that provides unified secrets management with a range of advantages for managing secrets and sensitive data. It integrates seamlessly with different providers, simplifying the management of secrets and access controls to monitor and control who has access to sensitive data; with encryption, access controls, multi-factor authentication and other security mechanisms it ensures only authorized users are able to access sensitive data. Additionally, it provides an intuitive interface for administration and monitoring, providing a less complex and more scalable developer and administration experience.
-
Within any organization, API producers and consumers need to stay in sync about the schemas that will be used for communication among them. Especially as the number of APIs and related producers and consumers grow in the organization, what may start with simply passing around schemas among teams will start to hit scaling challenges. Faced with this issue, some of our teams have turned to Apicurio Registry, an open-source, centralized registry for various types of schemas and API artifacts, including OpenAPI specifications and Protobuf and Avro schemas. Apicurio Registry allows users to interact with it through a UI as well as a REST API and a Maven plugin. It also has the option to enforce schema evolution restrictions, such as backward compatibility. Moreover, when it comes to working with Kafka clients, Apicurio Registry is compatible with Confluent Schema Registry. While our teams have found Confluent Schema Registry's documentation more helpful, Apicurio Registry meets their needs for a source of truth for various schemas.
-
Enterprises now often use event streaming as the source of truth and as an information-sharing mechanism in microservices architectures. This creates the need to standardize event types and share those standards across the enterprise. Event schema registries are commonly deployed but the existing offerings tend to be specialized to a single broker such as Apache Kafka or Azure Event Hub. They also fall short of conveying rich documentation about event types that goes beyond simple schema definitions. EventCatalog is an open-source project that provides something we often see businesses building for themselves: a widely accessible repository of documentation for events and schemas. These describe the role the events play in the business, where they belong in a business domain model and which services subscribe and publish them. If you're looking for a way to publish event documentation to your organization, this tool might save you the trouble of building it yourself.
-
FOSSA is an open-source compliance tool that helps developers and teams determine which open-source components their code relies on and which licenses these components are released under. This information is essential for ensuring compliance with various open-source licenses and maintaining the Software Bill of Materials. FOSSA integrates with dependency management tools of various tech stacks to identify which open-source components are used in a project. It also highlights any license issues based on the organization’s policies and generates reports of the same. Some key features of FOSSA include its ability to integrate with development workflows, such as the CI, and to perform real-time compliance monitoring. Many of our clients and teams have found FOSSA to be a valuable and effective tool.
-
Gitleaks is an open-source SAST (static application security testing) command line tool for detecting and preventing hardcoded secrets like passwords, API keys and tokens in Git repositories. It can be used as a Git pre-commit hook or in the CI/CD pipeline. Our teams found Gitleaks to be more sensitive than some of the other secret-scanning tools. Gitleaks utilizes regular expressions and entropy string coding to detect secrets. In our experience, the flexibility to supply custom regex along with entropy coding allowed the teams to better categorize secrets based on their needs. For example, instead of categorizing all API keys as "generic-api-key," it allowed categorization as specific "cloud provider key."
-
Helmfile is an open-source command line tool and a declarative specification for managing and installing a collection of Helm charts. You can use it to help with version control of the Helm values files, the charts used and other overrides. It enables CI/CD workflows with Helm charts and helps create reproducible environments. We've used Helmfile to manage complex deployments with multiple dozens of Helm charts and found it simplifies the deployment workflow.
-
Defects are cheaper to fix when they're caught early. That's why we always try to get the fastest possible feedback to developers in the form of static analysis, unit tests or end-to-end tests run in the local environment. Accessibility is no exception to this and that’s why we've featured tools such as Lighthouse, axe-core and axe Linter in the past. When it comes to automatically testing web pages that are already deployed in production, one of our teams chose instead to go with IBM Equal Access Accessibility Checker in a head-to-head comparison. Although we're still in the process of assessing the results, we can say that it offers an efficient way to test pages once they've been deployed. We emphasize that this should be used to augment, not replace, early automated testing by the developer. The tool is distributed under a Creative Commons license and is free to use under those restrictions.
-
The Kotlin ecosystem continues to evolve, and our teams report positive experiences with Ktlint, a simple and easy-to-configure linter and formatter for Kotlin code. We like opinionated and automated code formatting as it lets developers focus more on what the code does rather than how it looks; this tool enables development teams to maintain consistency and readability in their codebases efficiently, reducing the likelihood of messy merges due to formatting issues. Ktlint can be easily configured to run in pre-commit hooks, targeting only the files with changes and resulting in faster integration processes.
-
Kubeflow is a Kubernetes-native machine learning (ML) platform that simplifies build, train and deploy lifecycles of models to diverse infrastructure. We've extensively used Pipelines to encode ML workflows for several models across experimentation, training and serving use cases. Besides Pipelines, Kubeflow ships with multiple components, among which we find hyperparameter tuning with Katib and multi-tenancy to be quite useful.
-
Mend SCA (software composition analysis), previously Whitesource, helps detect open-source software dependencies by identifying if they are up to date, contain security flaws or have licensing requirements. Our teams have had good experience with integrating Mend SCA in their paths to production. Right from IDE integration, raising an automatic PR based on an identified issue to integrating into the CI/CD pipeline, this tool offers a great developer experience. Other popular SCA tools, such as Snyk, are comparable and also worth exploring for your security needs.
-
Our advice when it comes to secrets management has always been to decouple it from source code. However, teams are often presented with a tradeoff between full automation (in the spirit of infrastructure as code) versus a few manual steps (using tools like vaults) for managing, seeding and rotating seed secrets. For instance, our teams use SOPS to manage seed credentials for bootstrapping infrastructure. In some situations, however, it's impossible to remove secrets from legacy code repositories. For such needs, we found Mozilla SOPS to be a good choice for encrypting secrets in text files. SOPS integrates with cloud-managed keystores such as AWS and GCP Key Management Service (KMS) or Azure Key Vault as sources of encryption keys. It also works cross-platform and supports PGP keys.
-
Ruff is a new linter for Python. For us, the question is not whether to use a linter or not but which linter to use, and there are several choices for Python. Ruff stands out for two reasons: its out-of-box experience and its speed. It has over 500 rules built in and readily replaces Flake8, including many of that linter's plug-ins. The claims by the team behind Ruff about its performance are borne out by our experience. It really is at least an order of magnitude faster than other linters, which is a huge benefit because it helps reduce build times on large codebases.
-
Soda Core is an open-source data quality and observability tool. Our teams have used it to validate data as it arrives in a system, before and after transformations, and set up automated monitoring checks for anomalies. We're happy with SodaCL, the DSL for writing data checks in Soda Core, as it helps team members other than data engineers write quality checks. Overall, our experience using Soda Core to find and resolve data issues at scale has been positive.
-
Steampipe is an open-source tool that lets you instantly query cloud services like AWS, Azure and GCP with SQL. With 100+ plugins and built-in support for creating dashboards, Steampipe makes it trivial to connect live cloud configuration data with internal or external data sets and create security or compliance dashboards. We've enjoyed working with Steampipe and created several such dashboards with AWS cloud configurations.
-
More and more teams are using the Kubernetes Operators pattern to manage their Kubernetes clusters. We used to recommend Crossplane for this, and now we have an alternative tool, Terraform Cloud Operator for Kubernetes. This tool integrates Terraform Cloud and Kubernetes by extending the Kubernetes control plane to enable lifecycle management of cloud and on-premise infrastructures through Kubernetes manifests. Our team uses it to provision resources from Kubernetes namespaces and RoleBindings to cloud database instances and other SaaS resources. We quite like it because it leverages the Terraform module, which is a more familiar abstraction layer for us to operate cloud resources.
-
TruffleHog is an open-source SAST (static application security testing) tool for detecting secrets in various sources. While GitHub and GitLab repositories are the most popular use cases, it can also be used to scan cloud storage buckets like S3 and GCS, local files and directories and CircleCI logs. Developers can set up TruffleHog as a pre-commit hook or scan the history of existing repositories in an entire GitHub organization to detect secrets. The tool supports detecting custom regex patterns, which have been found to be quite useful even in its current alpha stage. TruffleHog also has an enterprise version, but our devs have found the open-source version easy to set up and sufficient for the most common use cases. The tool has a very active community who regularly adds features.
-
Typesense is an open-source, typo-tolerant search engine optimized for low-latency and high-performance search experiences. If you're building a latency-sensitive search application with a search index size that can fit in memory, Typesense is a powerful alternative. Our teams use Typesense in high availability multi-node clusters to distribute workload and ensure critical search infrastructure is resilient. They had a good experience with Typesense in production, which is why we've moved it to Trial.
-
Vite, a front-end build tool, has continued to mature and grow in popularity since we featured it in the Assess ring in the previous Radar. It is rapidly becoming the default choice among our teams when starting a new front-end project. Vite provides a set of defaults for building, bundling and managing dependencies in applications that depend on ES modules in the browser. Because it takes advantage of the native speed of esbuild and the Rollup bundler, Vite significantly improves the front-end developer experience. Moreover, when used with React, Vite offers an attractive alternative to the stalwart but nearly defunct Create React App. Vite relies on ES modules, and unlike most older tools, it doesn't provide shimming or polyfills, which means you need a different strategy for older browsers that don't support ES modules. In cases where older browsers had to be supported, some of our teams import polyfills at the module level so that Vite can be used consistently across environments.
Assess
-
It's becoming increasingly easy for developers to catch accessibility issues early in the development process. While tools like axe-core scan code for accessibility issues in your pipelines, the axe Linter VSCode extension helps find them even before that, while writing code. The vast majority of accessibility issues fall into categories that could be prevented by automated testing and using live feedback linters like this.
-
ChatGPT is an interesting tool that has the potential to be useful for various aspects of the software creation process. As a large language model (LLM) that has "read" billions of web pages, ChatGPT can provide additional perspectives and assist with different tasks, from generating ideas and requirements to creating code and tests. Its ability to work across multiple parts of the software lifecycle makes it a versatile tool that might improve efficiency and reduce errors in the development process. GPT4, the LLM that powers ChatGPT, now also has the ability to integrate with external tools such as a knowledge management repository, sandboxed coding environment or web search. For now, we think that ChatGPT is best used as an input to a process, such as helping with a first draft of a story or the boilerplate for a coding task, rather than a tool that produces "fully baked" results.
There are concerns around intellectual property and data privacy with these AI tools, including some unresolved legal questions, so we recommend organizations seek advice from their legal teams before use. Some of our clients have already begun experimenting with ChatGPT across various stages of the software lifecycle, and we encourage others to explore the tool and assess its potential benefits. We expect that, like GitHub Copilot, a "for business" offering will soon be available which may ease intellectual property concerns.
-
DataFusion is a part of the data community's exploration of Rust's performance, memory safety and concurrency features applied to data processing. It shares similarities with Polars, namely a familiar DataFrame API in Rust (with Python bindings), the use of Apache Arrow under the hood and SQL support. Even though it's primarily designed for single-process execution, distributed processing support is in the works within Ballista. We think the Rust libraries for data processing are an evolving space worth following and exploring, and DataFusion is a part of it.
-
As machine learning finds its way into the mainstream, practices are maturing around automatically testing models, validating training data and observing model performance in production. Increasingly, these automated checks are being incorporated into continuous delivery pipelines or run against production models to detect drift and model performance. A number of tools with similar or overlapping capabilities have emerged to handle various steps in this process (Giskard and Evidently are also covered in this volume). Deepchecks is another of these tools that’s available as an open-source Python library and can be invoked from pipeline code through an extensive set of APIs. One unique feature is its ability to handle either tabular or image data with a module for language data currently in alpha release. At the moment, no single tool can handle the variety of tests and guardrails across the entire ML pipeline. We recommend assessing Deepchecks for your particular application niche.
-
Design tokens are a useful mechanism for defining standard elements in design systems. But, keeping those design elements consistent across media such as mobile apps or web frameworks is an increasingly formidable task. Design token translation tools simplify this problem by organizing and automating transformation from the token description (in YAML or JSON) into the code that actually controls rendering in a given medium such as CSS, React components or HTML. Style Dictionary is an open-source example that is widely used and integrates well into automated build pipelines, but there are also commercial alternatives such as Specify.
-
Devbox provides an approachable interface for creating reproducible, per-project development environments leveraging the Nix package manager. Our teams use it to eliminate version and configuration mismatches in their development environments, and they like it for its ease of use. Devbox supports shell hooks, custom scripts and devcontainer.json generation for integration with VSCode.
-
Evidently is an open-source Python tool designed to help build monitoring for machine learning models to guarantee their quality and stable production operations. It can be used at various stages of a model lifecycle: as a dashboard to review the model in a notebook, as part of a pipeline or as a monitoring service after deployment. With a particular focus on model drift detection, Evidently also offers features such as model quality, data quality inspection and target drift detection. In addition, it has many built-in metrics, associated visualizations and tests which are easily combined into a report, dashboard or a test-driven pipeline.
-
Giskard is an open-source tool designed to help organizations build more robust and ethical AI models by providing quality assurance capabilities with a focus on explainability and fairness. It facilitates cooperation between technical and nontechnical stakeholders, allowing them to evaluate models collaboratively and establish acceptance criteria based on bias avoidance and other essential quality metrics. Giskard ensures model outcomes are better aligned with business objectives and helps to solve quality issues before production deployment.
-
GitHub Copilot is an AI coding assistant, created by a collaboration between Microsoft and OpenAI. It uses machine learning models to generate suggestions based on the context of what a developer is working on. It features strong IDE integration and uses an existing codebase and editor context to create suggestions. Despite being billed as "your AI pair programmer" we would not call what it does "pairing" — we'd probably describe it as a kind of supercharged, context-sensitive Stack Overflow. When it correctly predicts what a developer is trying to do, it can be a powerful tool for getting stuff done. Like all LLM-based AIs, though, it has a tendency to hallucinate the use of plausible but nonexistent APIs and may introduce bugs through slightly faulty algorithms. We've had success generating code at the line, block and method level, as well as creating tests or infrastructure configurations. Interestingly, it works best when you use good naming practices, so it encourages more readable code.
AI tool capabilities are advancing rapidly, and we think it's sensible for organizations to try them. Some sales pitches for Copilot have claimed very high efficiency gains, but we remain skeptical: after all, writing code isn't the only thing that developers spend time on, and it's notoriously difficult to measure developer productivity in the first place. That said, Copilot is a fairly inexpensive tool; if it offers any productivity gain at all, it's probably worth it. Copilot X — in preview as of this writing — offers additional functionality and integration within a software creation workflow. Copilot has a "for business" offering, which provides more clarity around intellectual property issues as well as the ability to manage tool features centrally across an organization. We think these features are critical for enterprise adoption.
-
Creating exactly the minimum viable AWS IAM policies we want, according to the principle of least privilege, can be a long journey of trial and error. iamlive can shorten that journey considerably. It monitors the AWS CLI calls made from a machine and determines the policies needed to execute those calls. The tool generates a policy document with statements, actions, principals and resources that can be used as a good starting point. We've found it particularly useful to create policies needed in CI/CD pipelines that provision infrastructure, reducing the usual back and forth after a Terraform run fails because the IAM role's policy is insufficient.
-
Measuring energy consumption is an important step for teams to reduce the carbon footprint of their software. Cloud Carbon Footprint (CCF) estimates energy based on billing and usage data retrieved from cloud APIs. Kepler — short for Kubernetes-based Efficient Power Level Exporter — goes one step further: it uses software counters via RAPL, ACPI and nvml to measure power consumption by hardware resources and employs an eBPF-based approach to attribute power consumption to processes, containers and Kubernetes pods. Power usage is then converted to energy estimates using a custom ML model and data from the SPEC Power benchmark. Finally, pod-level energy consumption reporting is made available as Prometheus metrics. In cases where Kubernetes is running on virtual machines, for example when not using bare metal instances, Kepler uses cgroups to estimate energy consumption. We've had significant experience with CCF and can attest to its usefulness, but we're intrigued by the Kepler project’s approach.
-
Kubernetes External Secrets Operator allows external secret providers to be integrated with Kubernetes. It reads from the external provider API and injects the result into a Kubernetes Secret. The operator works with a large variety of secret management tools, including some we've featured in previous editions of the Radar. Our teams have found it simplified the use of secrets when working with Kubernetes by allowing the use of a single store across a whole project.
-
Kubeshark is an API traffic viewer for Kubernetes. Until November 2022, it was known as Mizu. Unlike other tools, Kubeshark does not require instrumentation or code changes. It runs as a DaemonSet to inject a container at the node level in your Kubernetes cluster and performs tcpdump-like operations. We find it useful as a debugging tool, as it can observe all API communications across multiple protocols (REST, gRPC, Kafka, AMQP and Redis) in real time.
-
Knowledge management is critical for tech workers, as we need to be constantly learning and staying up to date with the latest technology developments. Recently, tools such as Obsidian and Logseq have emerged in the category of note-taking tools that support linking notes to form a knowledge graph, while storing them in plain markdown files in a local directory, thus letting users own their data. These tools help users organize and link their notes in a flexible, nonlinear way.
Obsidian has a rich repository of community plugins. Some that have caught our attention, in particular, are Canvas, akin to a local version of Miro or Mural, and Dataview, which effectively treats your notes as a database and provides a query language for filtering, sorting and extracting data from your markdown notes.
-
We've already assessed Ory Hydra as a self-hosted OAuth2 solution, and the feedback from the team has been good. This time, we turn to Ory Kratos, an API-first identity and user management system that’s developer friendly and easy to customize. It already provides common functions we want to achieve in an identity management system, including self-service login and registration, multi-factor authentication (MFA/2FA), account verification and account recovery. Like Hydra, Kratos is headless and requires developers to build the UI themselves, which gives the team more flexibility. Developers can also customize identity schema to fit different business contexts. Kratos has no external dependencies other than the database, and it's easy to deploy and scale in different cloud environments. If you need to build a user management system, we recommend you give Kratos a try.
-
While GitHub Actions runners cover a wide range of the most common runtimes, sometimes you need something that is more specific to your particular use case, such as a less common language runtime or a particular hardware configuration. That's when you need a self-hosted runner. Philips's self-hosted GitHub runner is a Terraform module that lets you spin up custom runners on AWS EC2 spot instances. The module also creates a set of Lambdas to make up for the fact that you lose some of GitHub Actions' lifecycle management when you self-host runners. They do the heavy lifting for scaling runners up and down as needed. That helps manage costs and allows you to make runners ephemeral, which helps improve repeatability and security. When you do need to self-host runners, you might miss a lot of things when building custom runners from scratch. Look for tools like this one instead.
Hold
- New
- Moved in/out
- No change
Unable to find something you expected to see?
Each edition of the Radar features blips reflecting what we came across during the previous six months. We might have covered what you are looking for on a previous Radar already. We sometimes cull things just because there are too many to talk about. A blip might also be missing because the Radar reflects our experience, it is not based on a comprehensive market analysis.
Unable to find something you expected to see?
Each edition of the Radar features blips reflecting what we came across during the previous six months. We might have covered what you are looking for on a previous Radar already. We sometimes cull things just because there are too many to talk about. A blip might also be missing because the Radar reflects our experience, it is not based on a comprehensive market analysis.
