Our teams report that Apollo has become the library of choice when building a React application that uses GraphQL to access data from a back-end service. Although the Apollo project also provides a server framework and a GraphQL gateway, the Apollo client gets our attention because it simplifies the problem of binding UI components to data served by any GraphQL backend. Put simply, this means less code needs to be written than using REST backends and redux.
MockK is our go-to tool for mocks when writing tests for Kotlin applications. We like to use this library because of its first-class support for Kotlin language features such as coroutines or lambda blocks. As a native library, it helps our teams to write clean and concise code on testing Kotlin applications instead of using the inconvenient wrappers of Mockito or PowerMock.
Apache Beam is an open-source unified programming model for defining and executing both batch and streaming data parallel processing pipelines. The Beam model is based on the Dataflow model which allows us to express logic in an elegant way so that we can easily switch between batch, windowed batch or streaming. The big data-processing ecosystem has been evolving quite a lot which can make it difficult to choose the right data-processing engine. One of the key reasons to choose Beam is that it allows us to switch between different runners — a few months ago Apache Samza was added to the other runners it already supports, which include Apache Spark, Apache Flink and Google Cloud Dataflow. Different runners have different capabilities and providing a portable API is a difficult task. Beam tries to strike a delicate balance by actively pulling innovations from these runners into the Beam model and also working with the community to influence the roadmap of these runners. Beam has SDKs in multiple languages including Java, Python and Golang. We've also had success using Scio which provides a Scala wrapper around Beam.
HiveRunner is an open-source unit test framework for Apache Hadoop Hive queries based on JUnit4. When writing nontrivial analytics or data pipelines in Hive SQL, we found HiveRunner to be a good enabler for writing tests and even TDDing out some moderately complicated SQL. HiveRunner enables you to write Hive SQL as releasable tested artifacts.
Kotlin has demonstrated its value beyond mobile app development. When building microservices and shipping software to production, our teams have had good experiences with Ktor. Ktor is a framework that, unlike other web frameworks that support Kotlin, is written in Kotlin, using language features such as coroutines which allow for an asynchronous nonblocking implementation. The flexibility to incorporate different tools for logging, DI or a template engine — in addition to its lightweight architecture — makes Ktor an interesting option for creating RESTful services.
Much like Cypress and TestCafe, Puppeteer is one of the web UI testing tools garnering praise from our teams. Puppeteer can have fine-grained control over headless browsers, obtain time-trace for performance diagnostics and more. Our teams have found Puppeteer to be stable as well as faster and more flexible than alternatives based on WebDriver.
We've talked about Reactor in the previous Radars. It has continued to gain traction in many of our projects. With the Spring ecosystem embracing Reactor, it has become the dominant implementation of Reactive Streams. Reactive systems come with improved scalability and resilience but with increased cost of debugging and a steeper learning curve. For those projects where this tradeoff is acceptable, Reactor has proven to be a good choice. Some of our projects have observed significant improvements in scalability once they moved to Reactor and the rest of the Reactive stack. With R2DBC we are starting to get reactive support for RDBMS drivers which addresses one of the weaknesses of reactive services.
Resilience4j is a lightweight fault tolerance library inspired by Netflix Hystrix. We like its lightweight and modular structure where we pull in specific modules for specific capabilities such as circuit-breaking, rate-limiting, retry, and bulkhead. While service meshes are taking on some of the fault tolerance capabilities, fault tolerance libraries continue to remain a key component of our systems for more nuanced domain-specific fault tolerance behavior and for non-containerized services. With Hystrix going into maintenance mode, Resilience4j becomes a default choice in the Java ecosystem. It can work with synchronous APIs as well as reactive ones. It also surfaces metrics to dropwizard metrics, Prometheus and others using additional modules.
Room is a persistence library to access SQLite on Android. It makes database access code much simpler, with minimal boilerplate code, and more robust, with compile-time verification of SQL queries. Our developers like its complete integration with observable queries, using LiveData. Room is one of the Android Jetpack components that were created to make application development on Android easier.
Since we last featured it on the Radar in January 2015, we've seen steadily increasing interest in Rust. Some of our clients are now using Rust, mostly in the context of infrastructure tooling but also in high-powered embedded devices. Interest was fuelled by a growing ecosystem as well as improvements to the language itself. The latter included straightforward performance improvements but also changes that make Rust more intuitive, for example the change to non-lexical scoping. Most of the significant changes are included in the Rust 2018 standard released last December.
WebFlux is the Spring Framework implementation of Reactive Streams. We see a rise in reactive programming models across our teams in general and the use of WebFlux in teams who are working in the Spring ecosystem. It's best used in large microservices ecosystems where the high performance of the requests is a major concern. It allows overlapping request processing asynchronously without the complications of using multiple threads. WebFlux uses Reactor as its reactive library but it is interoperable with other reactive libraries via Reactive Streams. It uses Netty as its underlying high-performance communications engine. Although we encourage using Reactive Streams, adopting this programming model requires a significant shift in thinking.
Aeron is an efficient and reliable peer-to-peer message transport. It provides a replicated persistent log of messages via a number of media drivers, including HTTP, UDP and TCP. It also supports persistent storage of message streams for later replay. For many applications, Aeron may be overkill because it operates at a pretty low level (OSI Layer 4 conceptually), but it's peer-to-peer design and low (and predictable) latency are useful in a number of use cases. Indeed, we've found it to be useful in certain machine learning applications as well as playing a part in event-driven architectures. We think it's worth pointing out that alternative messaging protocols exist that don't require additional services such as Apache Kafka to be run.
Arrow is a functional programming library for Kotlin, created by merging two existing popular libraries (kategory and funKTionale). While Kotlin provides building blocks for functional programming, Arrow delivers a package of ready-to-use higher-level abstractions for application developers. It provides data types, type classes, effects, optics and other functional programming patterns as well as integrations with popular libraries. With Arrow, existing libraries are unified, which should go a long way to avoid fractured communities in this space.
The Chaos Toolkit is one of a number of Chaos Engineering tools that made this edition of the Radar. You use the toolkit to describe and then run repeatable experiments on your infrastructure to understand its resilience in the event of failure. Many of our teams have been using homegrown tools to do this, so it's great to see an open-source project dedicated to the practice. The toolkit already has drivers for AWS, Azure Service Fabric and GCE (among others) and plays nicely with build tools which lets you experiment with automation. The usual caveats apply though, Chaos Engineering is a very powerful technique that is best used on resilience-aware systems, that is, systems that have been built to cope with failure. For that reason, we recommend starting using Chaos Toolkit in your nonproduction environments first.
Data scientists and engineers often use libraries such as pandas to perform ad hoc data analysis. Although expressive and powerful, these libraries have one critical limitation: they only work on a single CPU and don't provide horizontal scalability for large data sets. Dask, however, includes a lightweight, high-performance scheduler that can scale from a laptop to a cluster of machines. And because it works with NumPy, pandas and Scikit-learn, Dask looks promising for further assessment.
We've recommended Truffle for decentralized application (dapp) development in the past. Embark too can make your work easier. Embark provides features such as scaffolding, building, testing and debugging and integrates with decentralized storages such as IPFS. Through its declarative configuration, you can manage smart contract configuration, dependencies, artifact and deployment quite easily. Embark's interactive CLI dashboard is also impressive. We keep seeing people use Remix to write smart contracts and manually deploy their apps without automated testing, source control management or artifact management. We'd like to draw people's attention to dapp engineering practice by promoting tools such as Truffle and Embark.
fastai is an open-source Python library that simplifies training fast and accurate neural nets. It is built on top of PyTorch and has become a popular tool for our data scientists. fastai simplifies painful aspects of model training such as preprocessing and loading data down to a few lines of code. It's built on deep learning best practices and has out-of-the-box support for computer vision, natural language processing (NLP) and more. The founders' motivation has been to create an easy-to-use library for deep learning and an improved successor to Keras. GCP, AWS and Azure all have already included fastai in their machine images. The creators of fastai, acknowledging the speed and safety limitations of Python, have announced embracing Swift as an alternative language for deep learning. We'll be closely watching this space.
http4k is an HTTP toolkit written in pure Kotlin for serving and consuming HTTP services. One of the key ideas behind http4k is that HTTP apps are modelled by composing two simple functions — HttpHandler and Filter. They derive inspiration from Twitter's "Your Server as a Function" paper. It's very lightweight with the core module having no dependencies apart from Kotlin StdLib. Apart from its elegance and simplicity, we also like its emphasis on testability — given that the entities in the libraries are immutable and the routes in the app, as well as the app itself, are just functions, they're super easy to test. One of the things to be aware of, though, is that we don't have nonblocking or coroutines support in http4k yet.
Given our experience that tests are the only API specifications that really matter, we're always on the lookout for new tools that might help. Karate is an API testing framework whose unique feature is that tests are written directly in Gherkin without relying on a general-purpose programming language to implement test behavior. Karate is really a domain-specific language for describing HTTP-based API tests. Although this approach is interesting and makes for some very readable specifications for simple tests, the special-purpose language for matching and validating payloads can become quite syntax-heavy and difficult to understand. It remains to be seen if more complex tests written in this style will be readable and maintainable over the long haul.
Micronaut is a new JVM framework for building microservices using Java, Kotlin or Groovy. It distinguishes itself through a small memory footprint and short startup time. It achieves these improvements by avoiding runtime reflection for DI and proxy generation, a common shortcoming of traditional frameworks, and instead uses a DI/AOP container which performs dependency injection at compile time. This makes it attractive not just for standard server-side microservices but also in the context of, for example, the Internet of Things, Android applications and serverless functions. Micronaut uses Netty and has first-class support for reactive programming. It also includes many features that make it cloud-native friendly such as service discovery and circuit breaking. Micronaut is a very promising entrant to the full stack framework for the JVM space and we're keenly watching it.
Taiko is a node.js library with a clear and concise API to assist with chrome or chromium browser automation. You can leverage Taiko's smart selectors and write reliable tests as the structure of the web application evolves. There's no need for ID, CSS or XPath selectors or adding explicit waits (for XHR requests) in test scripts. The interactive REPL recorder comes in handy when you want to develop the tests side by side as you explore the functionality. Although you could use Taiko independently, we've had good success using it with Gauge.
We're strong proponents of polyglot programming but recognize that in some cases it can make sense to focus on a single programming language. If you're heavily invested in Swift, most likely because of iOS development, and you find yourself looking for a technology to write server-side services, have a look at Vapor, a modern web framework for Swift that has gained a fair amount of popularity.