Tuesday, July 5, 2016

End of the Storm

We noted a couple weeks ago that Twitter has open-sourced their Storm-replacement Heron.

I questioned if this indicated the death of Storm (after noting that many event-streaming shops) seemed to take to Spark Streaming instead of Storm.  Here is blog posting that has more detail behind that:

At Twitter, Storm has been decommissioned and Heron is now the de-facto streaming system. It has been in production for several months and runs hundreds of development and production topologies in multiple data centers. These topologies process several tens of terabytes of data, generating billions of output tuples…. results from an empirical evaluation of Heron demonstrate large reductions in CPU resources when using Heron, while delivering 6-14X improvements in throughput, and 5-10X reductions in tuple latencies.

One of the big changes to Heron versus Storm is the rebuilt native platform.  Storm spawned threads inside of a JVM process, while Heron's C++ core launches JVM processes.  It still uses the familiar Spout and Bolt API which makes it easy to work for existing projects using Storm, but has a substantially different core.