Splitting Apache Kafka® eventsVisit the split-stream recipe on GitHub.
Splitting or routing a stream of events
A common use case is where you will need to split a stream of events or route specific events to a specific location.
In this recipe, you are going to consume events from Apache Kafka which have a
Priority value in the payload.
All events with a
Major priority will be routed to different outputs.
This recipe for Apache Flink is a self-contained recipe that you can directly copy and run from your favorite editor. There is no need to download Apache Flink or Apache Kafka.
The JSON input data
The recipe uses Kafka topic
input, containing JSON-encoded records.
Define your side output
You are going to use Flink's Side Output to split the stream of events. Side Outputs can be used to split a stream in n-ways, into streams of different types with excellent performance.
To use Side Outputs, you first define an
OutputTag to identify your side output stream.
Emit to side output
You are going to emit to your defined side output using Flink's
By using the
Context parameter you are emitting data to the previously defined side output.
The full recipe
This recipe is self-contained. You can run the
SplitStreamTest#testProductionJob class to see the full recipe
in action. That test uses an embedded Apache Kafka and Apache Flink setup, so you can run it directly via Maven or in
your favorite editor such as IntelliJ IDEA or Visual Studio Code.
See the comments included in the code for more details.