We remain excited about Docker as it evolves from a tool to a complex platform of technologies. Development teams love Docker, as the Docker image format makes it easier to achieve parity between development and production, making for reliable deployments. It is a natural fit in a microservices-style application as a packaging mechanism for self-contained services. On the operational front, Docker support in monitoring tools (Sensu, Prometheus, cAdvisor, etc.), orchestration tools (Kubernetes, Marathon, etc.) and deployment-automation tools reflect the growing maturity of the platform and its readiness for production use. A word of caution, though: There is a prevalent view of Docker and Linux containers in general as being "lightweight virtualization," but we would not recommend using Docker as a secure process-isolation mechanism, though we are paying attention to the introduction of user namespaces and seccomp profiles in version 1.10 in this regard.
HTTP Strict Transport Security (HSTS) is a now widely supported policy that allows websites to protect themselves from downgrade attacks. A downgrade attack in the context of HTTPS is one that can cause users of your site to fall back to HTTP rather than HTTPS, allowing for further attacks such as man-in-the-middle attacks. With HSTS, the server sends a header that informs the browser that it should only use HTTPS to access the website. Browser support is now widespread enough that this easy-to-implement feature should be added to any site using HTTPS. Mozilla’s Observatory can help identify this and other useful headers and configuration options that improve security and privacy. When implementing HSTS, it is critical to verify that all resources load properly over HTTPS, because once HSTS is turned on, there is (almost) no turning back until the expiry time. The directive to include subdomains should be added but, again, a thorough verification that all subdomains support secure transport is required.
Application whitelisting has proven to be one of the most effective ways to mitigate cyber intrusion attacks. A convenient way to implement this widely recommended practice is through Linux security modules. With SELinux or AppArmor included by default in most Linux distributions, and with more comprehensive tools such as Grsecurity readily available, we have moved this technology into the Adopt ring in this edition. These tools help teams assess questions about who has access to what resources on shared hosts, including contained services. This conservative approach to access management will help teams build security into their SDLC processes.
We've continued to have positive experiences deploying the Apache Mesos platform to manage cluster resources for highly distributed systems. Mesos abstracts out underlying computing resources such as CPU and storage, aiming to provide efficient utilization while maintaining isolation. Mesos includes Chronos for distributed and fault-tolerant execution of scheduled jobs, and Marathon for orchestrating long-running processes in containers.
We have a growing belief that for most scenarios it is rarely worth rolling your own authentication code. Outsourced identity management speeds up delivery, reduces mistakes and tends to enable a faster response to newly discovered vulnerabilities. Auth0 has particularly impressed us in this field for its ease of integration, range of protocols and connectors supported, and rich management API.
Kubernetes is Google's answer to the problem of deploying containers into a cluster of machines, which is becoming an increasingly common scenario. It is not the solution used by Google internally but an open source project that originated at Google and has seen a fair number of external contributions. Since we mentioned Kubernetes on the previous Radar, our initial positive impressions have been confirmed, and we are seeing successful use of Kubernetes in production at our clients.
The PaaS space has seen a lot of movement since we last mentioned Cloud Foundry in 2012. While there are various distributions of the open source core, we have been impressed by the offering and ecosystem assembled as Pivotal Cloud Foundry. While we expect continued convergence between the unstructured approach (Docker, Mesos, Kubernetes, etc.) and the more structured and opinionated buildpack style offered by Cloud Foundry and others, we see real benefit for organizations that are willing to accept the constraints and rate of evolution to adopt a PaaS. Of particular interest is the speed of development that comes from the simplification and standardization of the interaction between development teams and platform operations.
The emerging Containers as a Service (CaaS) space is seeing a lot of movement and provides a useful option between basic IaaS (Infrastructure as a Service) and more opinionated PaaS (Platform as a Service). While Rancher creates less noise than some other players, we have enjoyed the simplicity that it brings to running Docker containers in production. It can run stand-alone as a full solution or in conjunction with tools like Kubernetes.
Realm is a database designed for use on mobile devices, with its own persistence engine to achieve high performance. Realm is marketed as a replacement for SQLite and Core Data. Note that migrations are not quite as straightforward as the Realm documentation would have you believe. However, more and more teams are choosing Realm as the persistence mechanism in production environments for mobile applications.
After experiencing years of growth as a platform for game development, Unity has recently become the platform of choice for VR and AR application development. Whether you’re creating a fully immersive world for the Oculus or HTC Vive headsets, a holographic layer for your newly spatial enterprise application or an AR feature set for your mobile app, Unity likely provides what you need to both prototype it and get it ready for prime time. Many of us at ThoughtWorks believe that VR and AR represent the next significant shift in the computing platform, and for now, Unity is the single most important tool in the toolbox we use to develop for this change. We’ve used Unity to develop all our VR prototypes, as well as AR functionality for headsets and phone/tablet applications.
.NET Core is an open source modular product for creating applications that can be easily deployed in Windows, macOS and Linux. .NET Core makes it possible to build cross-platform web applications using ASP.NET Core with a set of tools, libraries and frameworks—another choice for microservices architecture. The community around .NET Core and other related projects has been growing. New tools have appeared and evolved quickly, such as Visual Studio Code. There are Docker images based on both Linux and Windows (Nano Server) with .NET Core that simplify applying a microservice architecture. CoreCLR and CoreFX appeared in the Radar in the past. However, a few months ago Microsoft announced the release of .NET Core 1.0, the first stable version. We see good new opportunities, changes and a vibrant community as reasons to keep assessing this product.
Amazon API Gateway is Amazon's offering enabling developers to expose API services to Internet clients. It offers the usual API gateway features like traffic management, monitoring, authentication and authorization. Our teams have been using this service to front other AWS capabilities like AWS Lambda as part of serverless architectures. We continue to monitor for the challenges presented by overambitious API gateways, but at this stage Amazon's offering appears to be lightweight enough to avoid those problems.
Interest continues to build for Apache Flink, a new-generation platform for scalable distributed batch and stream processing. At the core of Apache Flink is a streaming data-flow engine, with support for tabular (SQL-like), graph-processing and machine learning operations. Apache Flink stands out with feature rich capabilities for stream processing: event time, rich streaming window operations, fault tolerance and exactly-once semantics. The project shows significant ongoing activity, with the latest release (1.1) introducing new datasource/sink integrations as well as improved streaming features.
Amazon recently launched the AWS Application Load Balancer (ALB), a direct replacement for Elastic Load Balancers introduced back in 2009. ALB supports Layer 7 traffic inspection and is built to support modern cloud architecture. If you’re building a microservices-based system using ECS, the new load balancers will directly understand container hosting and scaling, with multiple containers and ports per EC2 instance. Content-based routing allows segmentation of requests onto groups of target servers, along with independent scaling of those groups. Health checks performed by the load balancers are much improved, with the ability to capture detailed metrics about application performance. We like everything that we see here, and teams have begun to report successful usage of ALB.
Apache’s Cassandra database is a powerful, scalable Big Data solution for storing and processing large amounts of data, often using hundreds of nodes split over multiple worldwide locations. It’s a great tool and we like it, but too often we see teams run into trouble using it. We recommend using Cassandra carefully. Teams often misunderstand the use case for Cassandra, attempting to use it as a general-purpose data store when in fact it is optimized for fast reads on large data sets based on predefined keys or indexes. Its dependence on the storage schema can also make it difficult to evolve over time. Cassandra also has significant operational complexity and some rough edges, so unless you absolutely need the scaling it provides, a simpler solution is usually better. If you don’t need Cassandra’s specific use-case and scaling characteristics, you might just be choosing it out of Big Data envy. Careful use of Cassandra will include extensive automated testing, and we’re happy to recommend CassandraUnit as part of your testing strategy.
The hype seems to have peaked for blockchain and cryptocurrencies, as evidenced by the previous firehose-scale announcements in this area slowing to a trickle, and we expect some of the more speculative efforts to die out over time. One of the blockchains, Ethereum, is making good progress and is worth watching. Ethereum is a public blockchain with a built-in programming language that allows "smart contracts" to be built into it. These are algorithmic movements of "ether" (the Ethereum cryptocurrency) in response to activity happening on the blockchain. R3Cev, the consortium building blockchain tech for banks, built its first proofs of concept on Ethereum. Ethereum has been used to build a Distributed Autonomous Organization (DAO)—one of the first "algorithmic corporations"—although a recent heist of $150m worth of Ether demonstrates that the blockchain and cryptocurrencies are still the Wild West of the technology world.
In the HoloLens, Microsoft has delivered the first truly usable AR headset. Not only is it a beautiful piece of industrial design and an eminently comfortable device to wear, but it also clearly demonstrates the promise of AR for the enterprise via its gorgeous optics and deep Windows 10 integration. We expect HoloLens to be the first AR platform on which we deliver substantial application functionality to our clients in the near term, and we look forward to its evolution as it gains broader traction.
IndiaStack is a set of Open APIs designed with the goal of transforming India from a data-poor to a data-rich country. The stack emphasizes layered innovation by specifying a minimal set of APIs and encourages the rest of the ecosystem to build custom applications on top of these APIs. Aadhaar serves as one of the foundation layers, providing authentication services for more than a billion Indian citizens. In addition, there are services to provide paperless transactions through digital signatures (eSign), unified online payment (UPI) and an electronic consent layer (e-KYC) to securely provide Aadhaar details to service providers. We believe in the Open API–driven initiative to bring digital innovation, and the design principles behind IndiaStack could be used as a change agent for other regions/countries.
HashiCorp continues to turn out interesting software. The latest to catch our attention is Nomad, which is competing in the ever-more-populated scheduler arena. Major selling points include not just being limited to containerized workloads, and operating in multi–data center / multiregion deployments.
Nuance Mix is a framework for natural language processing from the company that created the speech-to-text technology behind Dragon Speaking and the first roll-out of Siri. This framework supports the creation of grammars that allow for free-form user interaction via voice. The developer defines a domain-specific grammar that the framework can train itself to understand. The outcomes are responses to user input that identify the user's intents and interaction concepts. At first, it is limited to phrases close to the ones used to train it, but over time it can start to identify meaning from more divergent phrasing. Though it is still in beta, the accuracy from early exploration has been compelling, and the eventual product is one to watch for application forms that could benefit from hands-free user interaction—including mobile, IoT, AR, VR and interactive spaces.
OpenVR is the underlying SDK in making many of the VR head-mounted displays (HMDs) work with Unity and will likely keep growing in importance. Much of the VR work at ThoughtWorks was built on top of OpenVR, because it will run on any HMD, unlike the other SDKs. Though it is not open source, it is free via the license. The Oculus SDK is more restrictive in its licensing and only works on Oculus devices. OSVR, while truly open source, doesn't seem to have as much adoption yet. If you're going to develop a VR application and target as many devices as possible—and not use Unity or Unreal to develop them—OpenVR is the most concrete and pragmatic solution right now.
Tarantool is an open source NoSQL solution that combines database and cache into one entity and provides APIs for writing application logic in Lua. Both in-memory and disk-based engines are supported, and users can create multiple indexes (HASH, TREE, RTREE, BITSET) based on their use cases. The data itself is stored in MessagePack format and uses the same protocol to communicate between clients and server. Tarantool supports write-ahead logs, transactions and asynchronous master-master replication. We are happy with the architectural decision of embracing single-writer policy and cooperative multitasking to handle concurrent connections.
We are seeing too many organizations run into trouble as they attempt to use their CMS as a platform for delivering large and complex digital applications. This is often driven by the vendor-fueled hope of bypassing unresponsive IT organizations and enabling the business to drag and drop changes directly to production. While we are very supportive of providing content producers with the right tools and workflows, for applications with complex business logic we tend to recommend treating your CMS as a component of your platform (often in a hybrid or headless mode) cooperating cleanly with other services, rather than attempting to implement all of your functionality in the CMS itself.
One of our regular complaints is about business smarts implemented in middleware, resulting in transport software with ambitions to run critical application logic. Vendors in the highly competitive API gateway market continue to add features that differentiate their products. This results in overambitious API gateway products whose functionality—on top of what is essentially a reverse proxy—encourages designs that are difficult to test and deploy. API gateways can provide utility in dealing with some generic concerns—for example, authentication and rate-limiting—but any domain smarts such as data transformation or rule processing should live in applications or services where they can be controlled by product teams working closely with the domains they support.
We've seen the indisputable productivity gains that come from deployment of applications and services into mature cloud providers. Much of that gain comes from the ability of teams to deploy and operate their own services with a high degree of autonomy and responsibility. We are now regularly coming across Superficial Private Cloud offerings within organizations, where basic virtualization platforms are being given the “cloud” label. Often teams can self-provision a restricted set of fixed service types with limited access and little ability to customize the centrally governed “enterprise blueprints,” leading to kludge solutions. Deployment pace regularly remains constrained by manually provisioned infrastructure such as network, firewall and storage. We encourage organizations to more fully consider the costs of mandating the use of an inadequate private cloud offering.