anyone with airflow experience would know how i can schedule a DAG with a CRON and a dataset ? (dataset being the new "data aware scheduling" paradigm airflow introduce in 2.4)
Discussion
Don’t know airflow (yet) but I’ve completely replaced cron with hashicorp Nomad
and how do you find it ?
was using aws cloud watch triggers prior but that wasn’t great.
Don’t understand your question; Nomad is a versatile orchestrator (can replace or augment k8s), and besides regular job types, it can run scheduled jobs too, ie you can schedule a container to run periodically with cron syntax. It has outstanding documentation
i was asking what was you're experience with it. ie do you like it.
Gotcha. I am a HUGE fan. The hashicorp stack in general adheres to the Unix philosophy of doing one thing and doing it very well. Nomad is an excellent orchestrator, far easier to learn and deeply than k8s, has features missing from k8s, cannot say enough good things