Stream processing is useful for many things, including real-time suggestions to users about content that may interest them based on their actions. You can feed streams of events to an event processor to have it search text. Designing Data Intensive Applications by Martin Kleppmann has great information about Streaming and Stream Processing in Chapter 11 Stream processing, this chapter covers the subject in great depth. If you don’t own that book you absolutely should if you are a software engineer. This book doesn’t cover Microservices, however it covers all of the software you use to create microservices and explains each in depth. If you are engineering any kind of modern app you either have that book or you are lost as to what you are doing and haphazardly choose software because it is all hypie and cool sounding. This book explains the internals of most Databases, NoSQL, RDBMS etc., it really gives you the info you need to make sound decisions. This book will save you hundreds of hours of raw research.
Stream processor list
There are many stream processors, below I list a few.
- Apache Flink – accepts input from message queue or file system, can save output to a message queue, database file system and more. Easily connects to KAFKA and Cassandra. Processes streams event by event
- Apache Spark – Spark website. processes streams in mini batches Interesting point from Wikipedia article “Spark Streaming has support built-in to consume from Kafka, Flume, Twitter, ZeroMQ, Kinesis, and TCP/IP sockets” Another interesting point from the wikipedia is the Spark MLib a machine learning framework built on top of Spark Core
- Apache Storm – Apache storm website Integrates with KAFKA, Databases and other Messaging Queues.
- Apache Kafka – This excellent piece of software functions as a Stream Processor, Message Broker and more.