Snowpipe Streaming Deep Dive
Disclaimer
Recent Posts
Exploring Stateful Streaming with Spark Structured Streaming
In a previous post, we explored how to do stateful streaming using Sparks Streaming API with the DStream abstraction. Today, I’d like to sail out on a journe...
When Life Gives You Options, Make Sure Not To Choose Some(null)
It was a casual afternoon at the office, writing some code, working on features. Then, all of the sudden, Spark started getting angry. If you run a productio...
Improving Spark Streaming Checkpointing Performance With AWS EFS
Update 10.03.2017 - There is a “gotcha” when using EFS for checkpointing which can be a deal breaker, pricing wise. I have updated the last part of the p...
Why You Might Be Misusing Sparks Streaming API
Disclaimer: Yes, I know the topic is controversial a bit, and I know most of this information is conveyed in Sparks documentation for it’s Streaming API, ...