Back in the days when I worked as a "data engineer", we had two teams. One used a GUI to create pipelines, and the other built a custom Scala app to do the same, but with all the perks like config in git, CI/CD, tests, etc. At first, the GUI team was much more productive, but guess what?

After some time, their system started to become messy and hard to manage. Our team, on the other hand, had been able to deliver features steadily by just adding things to the config. This feature 👇 looks like it combines both approaches. #spark #databricks https://youtu.be/GmWIvAyhkZU