Web application developers have it easy when it comes to simplifying and automating diverse application workflows; they can choose from a variety of solutions to help automate release processes. When developing for mobile, however, we're dealing with two operating systems with two different ways of building, testing, distribution, generating screenshots, signing and distributing applications. To help ease the pain, our teams have adopted fastlane as the go-to tool to automate the release process for iOS and Android applications. Using simple configurations and multiple pipelines, they can achieve continuous delivery for mobile development.
Our teams very much like the hosted CI/CD tool Buildkite for its simplicity and quick setup. With Buildkite, you provide your own machines to execute builds — on premise or in the cloud — and install a lightweight agent application to connect the build agent to the hosted service. In many cases, having this level of control over the configuration of your build agents is a plus when compared to using hosted agents.
CircleCI is a continuous integration engine offered as SaaS and on premise. CircleCI has been the go-to SaaS CI tool for many of our development teams, who needed a low-friction and easy-to-setup build and deployment pipeline. CircleCI version 2.0 supports workflows of build jobs, with fan-in and fan-out flows and manual gates, as well as mobile development. It allows developers to run the pipelines locally and easily integrates with Slack and other notification and alerting systems. We recommend you take a closer look at the security practices of CircleCI, just as you would with any other SaaS product that hosts your company’s assets.
gopass is a password management solution for teams, built on GPG and Git. It's a descendant of pass and adds features such as: support for recipient management and multiple password stores in a single tree; an interactive search functionality; time-based one-time password (TOTP) support; and storage of binary data. Migration of your pass store is fairly straightforward, because gopass is largely compatible with the format pass uses. This also means integration into provisioning workflows can be achieved with a single call to a stored secret.
We've seen both continuing improvements in and an uptick in adoption of Prometheus, the monitoring and time series database tool originally developed by Soundcloud. Prometheus primarily supports a pull-based HTTP model but it also supports alerts, making it an active part of your operational toolset. As of this writing, Prometheus 2.0 is in prerelease, and continues to evolve. Prometheus developers have focused their efforts on core time series databases and the variety of metrics available. Grafana has become the dashboard visualization tool of choice for Prometheus users and support for Grafana ships with the tool. Our teams also find that Prometheus monitoring nicely complements the indexing and search capabilities of an Elastic Stack.
Scikit-learn is not a new tool (it is approaching its tenth birthday); what is new is the rate of adoption of machine-learning tools and techniques outside of academia and major tech companies. Providing a robust set of models and a rich set of functionality, Scikit-learn plays an important role in making machine-learning concepts and capabilities more accessible to a broader (and often non-expert) audience.
Apex is a tool to build, deploy and manage AWS Lambda functions with ease. With Apex, you can write functions in languages that are not yet natively supported in AWS, including Golang, Rust and others. This is made possible by a Node.js shim, which creates a child process and processes events through stdin and stdout. Apex has a lot of nice features that improve the developer experience, and we particularly like the ability to test functions locally and perform a dry run of the changes before they're applied to AWS resources.
An AssertJ library, assertj-swagger enables you to validate an API implementation's compliance with its contract specification. Our teams use assertj-swagger to catch problems when the API endpoint implementation changes without updating its Swagger specification, or fails to publish the updated documentation.
Fixing end-to-end test failures in CI can be a painful experience, especially in headless mode. Cypress is a useful tool that helps developers build end-to-end tests easily and records all test steps as a video in an MP4 file. Instead of reproducing the issue in headless mode, developers can watch the testing video in order to fix it. Cypress is a powerful platform, not only a testing framework. Currently, we've integrated its CLI with headless CI in our projects.
How does a business hand autonomy to delivery teams while still making sure their deployed solutions are safe and compliant? How do you ensure that servers, once deployed, remain secure and compliant over their operational lifetime? These are the problems that InSpec tries to address. InSpec is an infrastructure testing tool inspired by Serverspec, but with modifications that make the tool more useful for security professionals who need to ensure compliance across thousands of servers. Individual tests can be combined into complete security profiles and run remotely from a command line. InSpec is useful for developers but extends to testing deployed production infrastructure continuously, moving toward QA in production.
Over the last couple of years, we've noticed a steady rise in the popularity of analytics notebooks. These are Mathematica-inspired applications that combine text, visualization and code in a living, computational document. In a previous edition, we mentioned GorillaREPL, a Clojure variant of these. But increased interest in machine learning — along with the emergence of Python as the programming language of choice for practitioners in this field — has focused particular attention on Python notebooks, of which Jupyter seems to be gaining the most traction among ThoughtWorks teams.
Kong is an open source API gateway built and sponsored by Mashape, who also provide an enterprise offering integrating Kong with their proprietary API analytics and developer portal tools. They can be deployed in a variety of configurations, as an edge API gateway or an internal API proxy. OpenResty, through its Nginx modules, provides a strong and performant foundation, with Lua plugins for extensions. Kong can either use PostgreSQL for single region deployments or Cassandra for multiregion configurations. Our developers have enjoyed Kong's high performance, its API-first approach (which enables automation of its configuration) and its ease of deployment as a container. Kong API Gateway, unlike overambitious API gateways, has a smaller set of features but it implements the essential set of API gateway capabilities such as traffic control, security, logging, monitoring and authentication. We're excited to assess Kong in a sidecar configuration in the near future.
kops is a command line tool for creating and managing high-availability production Kubernetes clusters. Initially targeting AWS, it now has experimental support for other providers. It can get you up and running fast, and even though a few features (such as rolling upgrades) have yet to be fully developed, we've been impressed by the community.
Lighthouse is a tool written by Google to assess web applications for adherence to Progressive Web App standards. This year's Lighthouse 2.0 release adds performance metrics and accessibility checks to the basic toolset. This added functionality has now been incorporated into the standard Chrome developer tools under the audit tab. Lighthouse 2.0 is yet another beneficiary of Chrome's headless mode. This provides an alternative to Pa11y and similar tools for running accessibility checks in a continuous integration pipeline, since the tool can be run from the command line or standalone as a Node.js application.
Sonobuoy is a diagnostic tool for running end-to-end conformance tests on any Kubernetes cluster in a nondestructive way. The team at Heptio, which was founded by two creators of the Kubernetes projects, built this tool to ensure that the wide array of Kubernetes distributions and configurations conform to the best practices, while following the open source standardization for interoperability of clusters. We're experimenting with Sonobuoy to run as part of our infrastructure as code build pipeline, as well as continuous monitoring of our Kubernetes installations, to validate the behavior and health of the whole cluster.
spaCy is a Natural Language Processing (NLP) library written in Python. It is a high-performance library, intended for use by developers in production, and applies NLP models suited for processing text that often mixes in emoticons and inconsistent punctuation marks. Unlike other NLP frameworks, spaCy is a pluggable library and not a platform; it is aimed at production applications rather than model training for research. It plays well with TensorFlow and the rest of the Python AI ecosystem. We've used spaCy in the enterprise context to build a search engine that takes human language queries and helps users make business decisions.
Netflix has open sourced Spinnaker, its microservices continuous delivery (CD) platform. Compared to other CI/CD platforms, Spinnaker implements cluster management and deployment of baked images to the cloud as first-class features. It supports out-of-the-box deployment and cluster management for multiple cloud providers such as Google Cloud Platform, AWS and Pivotal Cloud Foundry. You can integrate Spinnaker with Jenkins to run a Jenkins job build. We like Spinnaker's opinionated approach for deploying microservices to the cloud—with the exception that Spinnaker's pipelines are created via a user interface (UI) and cannot be configured as code.
If you're implementing Java services using the Spring framework, you may want to consider Spring Cloud Contract for consumer-driven contract testing. The current ecosystem of this tool supports verification of the client calls and the server implementation against the contract. In comparison to Pact, an open source consumer-driven contract testing tool set, it lacks the brokering of the contracts and the support for other programming languages. However, it integrates well with the Spring ecosystem, for instance message routing with Spring Integration.
Yarn is a new package manager that replaces the existing workflow for the npm client while remaining compatible with the npm registry. With the npm client, we may end up with a different tree structure under node_modules based on the order that dependencies are installed. This nondeterministic nature can cause "works on my machine" problems. By breaking the installation steps into resolution, fetching and linking, Yarn avoids these issues using deterministic algorithms and lockfiles and thus guarantees repeatable installations. We've also seen significantly faster builds in our continuous integration (CI) environment because of Yarn caching all the packages it downloads.