Snowpipe Streaming Deep Dive
Disclaimer
Posts
2024
2017
Exploring Stateful Streaming with Spark Structured Streaming
In a previous post, we explored how to do stateful streaming using Sparks Streaming API with the DStream abstraction. Today, I’d like to sail out on a journe...
When Life Gives You Options, Make Sure Not To Choose Some(null)
It was a casual afternoon at the office, writing some code, working on features. Then, all of the sudden, Spark started getting angry. If you run a productio...
Improving Spark Streaming Checkpointing Performance With AWS EFS
Update 10.03.2017 - There is a “gotcha” when using EFS for checkpointing which can be a deal breaker, pricing wise. I have updated the last part of the p...
Why You Might Be Misusing Sparks Streaming API
Disclaimer: Yes, I know the topic is controversial a bit, and I know most of this information is conveyed in Sparks documentation for it’s Streaming API, ...
Diving Into Spark 2.1.0 Blacklisting Feature
Disclaimer: What I am about to talk about is an experimental Spark feature. We are about to dive into an implementation detail, which is subject to change...
2016
MutableList And The Short Path To A StackOverflowError
When working with collections in Scala, or any other high level programming language, one does not always stop to think about the underlying implementation o...
The Case Of The Immutable Map and Object Who Forgot To Override HashCode
Disclaimer: What we’re about to look at is an implementation detail valid for the current point in time, and is valid for Scala 2.10.x and 2.11.x (not sur...
Exploring Stateful Streaming with Apache Spark
Update (01.08.2017): Spark v2.2 has recently come out with a new abstraction for stateful streaming called mapGroupsWithState which I’ve recently blogged abo...
Decompiling Traits with Scala 2.11 and 2.12
Introduction
Supporting Avro in WebAPI via AvroMediaTypeFormatter
When you’re dealing with a large amount of data over the wire, you want to be able to reduce your payload size as much as you can. Using JSON is fine for mos...
Reducing Two Option[T]’s in Scala
Every Scala programmer is familiar with the Option[T] monad. Option[T] is a container which either has a value (Some), or doesn’t have a value (None). It is ...
Spark, Uber Jars and Shading with sbt-assembly
At work we started using Spark Streaming as the underlying framework for a new project. Spark is a fast and general engine for large-scale data processing, w...
2015
Using Async-Await With Mutable Structs
Not long ago an interesting question appeared on StackOverflow. We know that generally, mutable structs are considered evil, primarily for the fact that they...