Metaflow

Metaflow is an open source Python framework for building and managing machine learning, AI and data science projects. It was developed at Netflix to serve data scientists and ML engineers working on demanding production problems, and Netflix open sourced it in 2019. Flows are written in plain Python, developed and debugged locally, and then deployed to production without changing the code. The framework covers the stages between a notebook experiment and a running system. It manages library dependencies locally and in the cloud, tracks and versions the variables stored inside each flow for experiment tracking and debugging, and moves data between steps. Work can be scaled out to cloud compute, including GPUs, multiple cores and large memory instances, and deployed workflows can react to events such as updated data. Metaflow is aimed at ML and AI engineers and data scientists. Users can start on a laptop and later deploy the Metaflow stack on their own cloud account or an on premise Kubernetes cluster. Documented deployment targets include AWS on EKS and S3 or AWS Batch and Step Functions, Azure on AKS and Blob Storage, Google Cloud on GKE and Cloud Storage, and custom Kubernetes. A browser based Metaflow Sandbox is offered to try it. The framework is open source and the site publishes no paid plans or prices.