To me this weekend wasn’t the Panthers vs. Broncos match-up for Super Bowl 50, or when we found out that Bernie Sanders won the New Hampshire primary. Although both of these were hoooge: it WAS when these parallel but significant facts emerged:
- Google makes it’s historical first open source contribution to the Apache Foundation in the form of Apache Beam
- Apache Beam supports three run time engines: Google Cloud, Apache Spark, and Apache Flink
- Independent reproducible academic research shows that Apache Flink handily out-performs Apache Spark on in-memory terasort workload
- Google releases a rigorous point-by-point comparison showing that Apache Beam’s programming model requires less code than Apache Spark to do the same tasks
So for whoever drank the Spark cool-aid let me translate: you write more code to do things more slowly *and* now have the privilege of competing head-to-head with Google.
This is what’s called a bubble, folks.
Please VC funders: end this quickly it’s more painless that way. And don’t put Spark on your resume because then people might notice the cool-aid stains.