Luigi is a Python module used to build complex pipelines for batch jobs. Developed by Spotify, it was open-sourced in 2012. Popular companies like  Stripe, Foursquare, and Asana use Luigi as a workflow engine. It manages dependency resolutions, workflow management, visualization, and more. Also, there’s built-in Hadoop support. 

Project Background

• Framework: Luigi 
• Author: Erik Bernhardsson, Elias Freider
• Released: 2012
• Type: Open source workflow management
• License: Apache License 2.0
• Support: NA
• Language: Python, JavaScript, HTML
• GitHub: spotify/luigi
• Runs on: Application setup without a framework, Angular, SAPUI5/OpenUI5, VUE.JS, React

Applications

•  Custom calendar Scheduling
• Dependency resolution
• Workflow management
• Visualization
• Handling failures
• Command-line integration 

Summary

• Provides a framework to manage and develop data processing pipelines.
• It automatically works on various related tasks for final output.
• Luigi API provides PySparkTask for writing custom spark jobs.
• Visualizer page in the web interface for viewing all your ongoing tasks.