Minimal Twitter to Google Pub/Sub example with Scala

Recently I was looking for a simple Twitter to Pub/Sub streaming pipeline and ended up with own implementation in Scala. I tried to make it as compact as possible. So I chose the dispatch and Google Pub/Sub client libraries for Java.

You should have a Google Cloud Platform service account key and Twitter API consumer key and tokens ready to start.

1. Create publisher:

  val publisher = Publisher.
    newBuilder(TopicName.of("projectId", "topic")).
    setCredentialsProvider(FixedCredentialsProvider.create(
      GoogleCredentials.fromStream(new FileInputStream("key.json")))).
    build()

2. Get stream of statuses/sample messages and publish them!

  Http.default(
    url("https://stream.twitter.com/1.1/statuses/sample.json") <@ (
      new ConsumerKey("consumerKey", "consumerSecret"),
      new RequestToken("accessToken", "accessTokenSecret")) >
    as.stream.Lines { tweet =>
        publisher.publish(PubsubMessage.newBuilder.setData(ByteString.copyFromUtf8(tweet)).build())
      })

As improvements you may want to configure BatchingSettings settings for Publisher and add various exception handlers.
You can find full source code here.