Luigi is a Python module used to build complex pipelines for batch jobs. Developed by Spotify, it was open-sourced in 2012. Popular companies like Stripe, Foursquare, and Asana use Luigi as a workflow engine. It manages dependency resolutions, workflow management, visualization, and more. Also, there’s built-in Hadoop support.
- Framework: Luigi
- Author: Erik Bernhardsson, Elias Freider
- Released: 2012
- Type: Open source workflow management
- License: Apache License 2.0
- Support: NA
- GitHub: spotify/luigi
- Runs on: Application setup without a framework, Angular, SAPUI5/OpenUI5, VUE.JS, React
- Custom calendar Scheduling
- Dependency resolution
- Workflow management
- Handling failures
- Command-line integration
- Provides a framework to manage and develop data processing pipelines.
- It automatically works on various related tasks for final output.
- Luigi API provides PySparkTask for writing custom spark jobs.
- Visualizer page in the web interface for viewing all your ongoing tasks.