Tag: Apache Beam

Datascape podcast episode 30 – learn about streaming

In this episode of the Datascape Podcast, we welcome back Danil Zburivsky from Pythian to talk about the new streaming technologies that he and his team are working on. The step up from batch processing to streaming seems to be…

Read More >

Apache Beam pipelines with Scala: part 3 – dynamic processing

In the third part of the series we will develop a pipeline to transform messages from “data” Pub/Sub using messages from the “control” topic as source code for our data processor. The idea is to utilize Scala toolBox. It’s much…

Read More >

Apache beam pipelines With Scala: Part 2 – Side Input

In the second part of this series we will develop a pipeline to transform messages from “data” Pub/Sub topic with the ability to control the process via “control” topic. How to pass effectively non-immutable input into DoFn, is not obvious,…

Read More >

Apache beam pipelines with Scala: part 1 – template

In this 3-part series I’ll show you how to build and run Apache Beam pipelines using Java API in Scala. In the first part we will develop the simplest streaming pipeline that reads jsons from Google Cloud Pub/Sub, convert them…

Read More >