Apache Storm v2.0: Porting from Clojure to pure Java, performance upgrades, & more

Apache Storm underwent quite a journey since its inception. The real-time computation system became open source after Twitter acquired its developer team BackType. In 2013, it underwent incubation and then became an Apache Top-Level Project in 2014.

Now, it crests the horizon with a new major release. Apache Storm 2.0.0 arrived on May 30, 2019 and is available to download. It includes performance improvements, several new features, bug fixes, and more.

New Apache Storm features

What was added in the latest update?

Removal of storm-kafka and storm-kafka-client Subscription interface: replaced with TopicFilter and ManualPartitioner interfaces
KafkaBolt custom callback
Windowing API improvements: including stateful windowing and window checkpointing
Nimbus admin groups: enables group level permissions
Authority through delegation tokens for workers
Optional impersonation
Added Lambda support
Blacklist Scheduler: adds unstable nodes to blacklist
Abstract class ConfigurableTopology
Introduce ClusterMetricsConsumer
“progress” method added to OutputCollector
symlinks alternate
Maven artifacts reorganization: split into client and server-facing parts
Admin commands: new commands for debugging cluster state

View the full in-depth release notes for v2.0.0.

Performance improvements

This update comes with several notable changes and performance enhancements. Firstly, Storm changes its architecture from Clojure over to pure Java. Why make this change? One of the reasons is in order to offer a more user-friendly developer experience. Storm v2.0 requires Java 8.

From the Storm website:

In previous releases a large part of Storm’s core functionality was implemented in Clojure. Storm 2.0.0 has been rearchitected with it’s core functionality implemented in pure Java. The new Java-based implementation has improved performance significantly, and made Storm’s internal APIs more maintainable and extensible. While Storm’s Clojure implementation served it well for many years, it was often cited as a barrier for entry to new contributors. Storm’s codebase is now more accessible to developers who don’t want to learn Clojure in order to contribute.

According to a post from the Yahoo! Developer Network, new benchmarks and testing show that its performance improvements now make Storm 2.0 one of the best-performing platforms. Version 2.0 offers less overhead and processes more data with a new leaner performance core. This new core also supports backwards compatibility.

See the tests they ran on CPU usage and Storm Latency.