We have moved Consul, the service-discovery tool supporting both DNS- and HTTP-based discovery mechanisms, into Adopt. It goes beyond other discovery tools by providing customizable health checks for registered services, ensuring that unhealthy instances are marked accordingly. More tools have emerged to work with Consul to make it even more powerful. Consul Template enables configuration files to be populated with information from Consul, making things like client-side load balancing using mod_proxy much easier. In the world of Docker, registrator can automatically register Docker containers as they appear with Consul with extremely little effort, making it much easier to manage container-based setups. You should still think long and hard about whether you need a tool like this or whether something simpler will do, but if you decide you need service discovery, you won't go far wrong with Consul.
When combining modern techniques and architecture styles, such as microservices, DevOps and QA in production, development teams need increasingly sophisticated monitoring. Simply looking a graphs of disk usage and CPU utilization is not sufficient anymore, and many teams collect application and business-specific metrics using tools such a Graphite and Kibana. Grafana makes it easy to create useful and elegant dashboards for data from a number of sources. A particularly useful feature allows timescales of different graphs to be synchronized, which helps with spotting correlations in the underlying data. The templating system that is being added shows a lot promise and will likely make managing sets of similar services even easier. Based on its strengths, Grafana has become our default choice in this category.
Machine images have become a staple of modern deployment pipelines, and there are a number of tools and techniques to create the images. Because of its comprehensive feature set and the positive experiences we've had with it, we recommend Packer over the alternatives. We also recommend against trying to write custom scripts to do what Packer does out of the box.
Many organizations are now looking closely at new data architectures that capture information as immutable sequences of events at scale. Apache Kafka continues to build momentum as an open source messaging framework that provides a solution for publishing ordered event feeds to large numbers of independent, lightweight consumers. Configuring Kafka is nontrivial, but our teams are reporting positive experiences with the framework.
At the top of the testing pyramid for Android application development, our teams are increasingly using Espresso as the functional-testing tool. Its small-core API hides the messy implementation details and helps in writing concise tests, with faster and reliable test execution. With Espresso, you can run automated UI tests simulating user interactions within a single target app on both emulators and real devices across different Android versions.
fastlane is our go-to tool for automating most of the boring activities involved in getting iOS and Android mobile apps built, tested, documented and provisioned. Simple configuration, a range of tooling and multiple pipelines make this a key ingredient in doing continuous delivery for mobile.
Testing that layout and styling of responsive websites is working as expected across various form factors can be a slow and often manual process. Galen helps ease this problem by providing a simple language, running on top of Selenium, that allows you to specify expectations for the appearance of your website in various screen sizes. Although Galen suffers from the typical brittleness and speed issues of any end-to-end testing approach, we have found benefit in the early feedback on design issues.
Having a way to securely manage secrets is increasingly becoming a huge project issue. The old practice of keeping secrets in a file or in environment variables is becoming hard to manage, especially in environments with multiple applications and large numbers of microservices. HashiCorp Vault addresses the problem by providing mechanisms for securely accessing secrets through a unified interface. It has served us well on a number of projects, and our teams liked how easy it was to integrate Vault with their services. Storing and updating secrets is a bit cumbersome, because it relies on a command-line tool and a fair amount of discipline from the team.
Let's Encrypt first appeared on the Radar last edition, and since December 2015 this project has moved its beta status from private to public, meaning users will no longer be required to have an invitation in order to try it. Let's Encrypt grants access to a simpler mechanism to obtain and manage certificates for a larger set of users who are seeking a way to secure their websites. It also promotes a big step forward in terms of security and privacy. This trend has already begun within ThoughtWorks, and many of our projects now have certificates verified by Let's Encrypt.
Load Impact is a SaaS load-testing tool that can generate highly realistic loads of up to 1.2 million concurrent users. Record and playback web interactions using a Chrome plugin simulate network connections for mobile or desktop users and generate load from up to 10 different locations around the world. While not the only on-demand load-testing tool we've used—we also like BlazeMeter—our teams were very enthusiastic about Load Impact.
In a world full of libraries and tools that simplify the life of many software developers, deficiencies in their security have become visible and have increased the vulnerability surface in the applications that use them. OWASP Dependency-Check automatically identifies potential security problems in the code, checking if there are any known publicly disclosed vulnerabilities, then using methods to constantly update the database of public vulnerabilities. Dependency-Check has some interfaces and plugins to automate this verification in Java and .NET (which we have used successfully) as well as Ruby, Node.js and Python.
Pa11y is an automatic accessibility tester that can run from the command line and be embedded into a build pipeline. Our teams have had success using Pa11y on a highly dynamic site by first creating a static HTML version, then running the accessibility tests against that. For many systems—especially government websites—accessibility testing is a requirement, and Pa11y makes it all a lot easier.
In the past we have included automated Provisioning Testing as a recommended technique, and in this issue we highlight Serverspec as a popular tool for implementing those tests. Although this tool is not new, we are seeing its use become more common as more cross-functional delivery teams take on responsibility for infrastructure provisioning. Serverspec is built on the Ruby library RSpec and comes with a comprehensive set of helpers for asserting that server configuration is correct.
With the maturity of tools such as Vault, there is no longer an excuse for storing secrets in code repositories, particularly since this often ends up being the soft underbelly of important systems. We’ve previously mentioned repository-scanning tools such as Gitrob, but we are now pushing proactive tools such as (the ThoughtWorks-created) Talisman, which is a prepush hook for Git that scans commits for secrets matching predefined patterns.
With Terraform, you can manage cloud infrastructure by writing declarative definitions. The configuration of the servers instantiated by Terraform is usually left to tools like Puppet, Chef or Ansible. We like Terraform because the syntax of its files is quite readable and because it supports a number of cloud providers while making no attempt to provide an artificial abstraction across those providers. Following our first, more cautious, mention of Terraform almost two years ago, it has seen continued development and has evolved into a stable product that has proven its value in our projects. The issue with state file management can now be sidestepped by using what Terraform calls a "remote state backend." We’ve successfully used Consul for that purpose.
Pair programming is an essential technique for us, and—given that we’re seeing more and more teams whose members are distributed across multiple locations—we have experimented with a number of tools to support remote pairing. We certainly liked ScreenHero but are concerned about its future. For teams that don’t rely on a graphical IDE, using tmate for pairing has turned out to be a great solution. tmate is a fork of the popular tmux tool, and compared to tmux for remote pairing, the setup is much easier. Compared to graphical screen-sharing solutions, the bandwidth and resource requirements are modest, and it obviously never suffers from blurry screens. Teams can also set up their own server, thus retaining full control of the privacy and integrity of the solution.
Development on Zipkin has continued apace, and since the middle of 2015 it has moved to the openzipkin/zipkin organization at GitHub. There are now bindings for Python, Go, Java, Ruby, Scala and C#; and there are Docker images available for those wanting to get started quickly. We still like this tool. There is an active and growing community around usage of it, and implementation is getting easier. If you need a way of measuring the end-to-end latency of many logical requests, Zipkin continues to be a strong choice.
Android-x86 is a port of the Android open source project to x86 platforms. The project started by hosting various patches from the community for x86 support but then later created its own codebase to provide support for different x86 platforms. We have seen significant time savings by utilizing Android-x86 in our CI servers instead of emulators for hermetic UI testing. However, for UI-specific tests targeting a particular device resolution—simulating low memory, bandwidth and battery—it is better to stick with emulators.
With the growth of interest in streaming data architectures and the downstream data lakes they feed, we have seen an increased reliance on "change data capture" tooling to connect transactional data stores to stream-processing systems. Bottled Water is a welcome addition to this field, converting changes in PostgreSQL’s write-ahead log into Kafka events. One downside of this approach, however, is that you are tied to low-level database events rather than the higher-level business events we recommend as the foundation for an event-oriented architecture.
One of those perpetual developer debates involves language typing: How much is just right? Clojure, the dynamically typed functional Lisp on the JVM, added a new entry into this discussion that blurs the lines. Clojure.spec is a new facility built into Clojure that allows developers to wrap type and other verification criteria around data structures, such as allowable value ranges. Once they are established, Clojure uses these specifications to provide a slew of benefits: generated tests, validation, destructuring of data structures and others. Clojure.spec is a promising way to have the benefits of types and ranges where developers need them but not everywhere.
Testing the visual portion of iOS applications can be painful, slow and flakey, which is why we’re happy to include FBSnapshotTestcase in our toolkit. It automates taking, storing and diff-ing snapshots of UI components so you can keep your interfaces pixel-perfect. Since it runs as a unit test (in the simulator), it is faster and more reliable than functional-testing approaches.
LambdaCD provides teams with a way to define Continuous Delivery pipelines in Clojure. This brings the benefits of Infrastructure as code to the configuration of CD servers: source-control management, unit testing, refactoring and code reuse. In the "pipelines as code" space, LambdaCD stands out for being lightweight, self-contained and fully programmable, allowing teams to work with their pipelines in the same way that they do with their code.
Teams using the Phoenix Server or Phoenix Environment techniques have found little in the way of support from Application Performance Management (APM) tools. Their licensing models, based on long-running, limited amounts of tin, and their difficulty in dealing with ephemeral hardware, have meant that they are often more trouble than they are worth. However, distributed systems need monitoring, and at some point many teams recognize the need for an APM tool. We think Pinpoint, an open source tool in this space, is worth investigating as an alternative to AppDynamics and Dynatrace. Pinpoint is written in Java, with plugins available for many servers, databases and frameworks. While we think you can go a long way using a combination of other lightweight open source tools—Zipkin, for example—if you are in the market for an APM, Pinpoint is worth considering.
Pitest is a test coverage analysis tool for Java that uses a mutation-testing technique. Traditional test coverage analysis tends to measure the number of lines that are executed by your tests. It is therefore only able to identify code that is definitely not tested. Mutation testing, on the other hand, tries to test the quality of those lines that are executed by your test code and yet might contain general errors. Several problems can be spotted this way, helping the team to measure and grow a healthy test suite. Most of such tools tend to be slow and difficult to use, but Pitest has proven to have better performance, is easy to set up, and is actively supported.
Attacks on web properties using bots are becoming more sophisticated. Identifying these bad actors and their behaviors is the goal of the Repsheet project. It's a plugin for either Apache or NGINX that records user activity, fingerprints actors using predefined and user-defined rules, and then allows action to be taken, including the ability to block offensive actors. It includes a utility that visualizes current actors; this puts the ability to manage bot-based threats in the hands of team members, increasing security awareness for teams. We like this since it's a good example of a simple tool solving a very real but often invisible problem—bot-based attacks.
Scikit-learn is an increasingly popular machine-learning library written in Python. It provides a robust set of machine-learning models such as clustering, classification, regression and dimensionality reduction, and a rich set of functionality for companion tasks like model selection, model evaluation and data preparation. Since it is designed to be simple, reusable in various contexts and well documented, we see this tool accessible even to nonexperts to explore the machine-learning space.
We know we're in perilous territory here, since we build a competing tool, but we feel we have to address a persistent problem. Continuous Integration tools like CruiseControl and Jenkins are valuable for software development, but as your build process gets more complex it requires something beyond just Continuous Integration: It requires a deployment pipeline. We frequently see people trying to use Jenkins as a Deployment Pipeline with the aid of plugins, but our experience is that these quickly become a tangle. Jenkins 2.0 introduces "Pipeline as Code" but continues to model pipelines using plugins and fails to change the core Jenkins product to model pipelines directly. In our experience, tools that are built around a first-class representation of deployment pipelines are much more suitable, and this is what drove us to replace CruiseControl with GoCD. Today we see several products that embrace deployment pipelines, including ConcourseCI, LambdaCD, Spinnaker, Drone and GoCD.