Apache Spark 4.1 is officially here and it's a massive release! 🔥
1️⃣ TIME WITH TIME ZONE: finally, proper timezone support! 🕒
2️⃣ WITH RECURSIVE: native support for recursive CTEs in SQL. 🔄
3️⃣ Real-Time Mode: sub-second latency (<300ms) for streaming. ⚡
4️⃣ Declarative Pipelines: define ETL state, let Spark handle the execution. 🏗️
5️⃣ Pythonization: native Python data sources, remapped JVM exceptions etc. 🐍