Master

< back to main page



Architecting Airflow for High Availability


By Vishal Srivastava

A trend we’re noticing at Thoughtworks is people increasingly looking to utilise Airflow as a scheduling tool of choice. One of the things considered for an enterprise solution is high availability which does not come by default as part of Airflow. In order to make Airflow highly available, individual components (web server, schedule, executor, metadata store and queuing service) have to be designed to sustain failure. Out of the lot, the challenging bit is to make the scheduler highly available. Following is one example of an airflow deployment using Kubernetes which could overcome the mentioned challenges. If you’re interested in discussing how we could help you with these challenges, or want to chat about how you’re tackling them, we’d love to hear from you!

Want help to unlock your data potential?