Logical Volume Manager
File systems work directly with disk drives (SSD, SAS, SATA) by dividing the storage space into equally sized compartments. An index keeps track of where all the files are stored in the different compartments. Windows supports NTFS and Linux several types.
- Windows 10: NTFS
- Linux: NTFS, FAT32, and Ext4. Ext4 is the default and popular choice
Each file system comes with its own set of features. The operating system will dictate the file system used for the machine learning project. Since most ML projects will run on Linux, Ext4 is a natural choice.
NTFS
- Support volumes of 8 petabytes
- File permissions and encryption on individual files
- Compress individual files
- Journaling file system
- Self-repairs during power failure
- Works only on Windows
Ext 4
- Developed for Linux
- Max volume size one exabyte
- Max file size is 16TB
- Journaling file system
- No Windows or Mac support
FAT32
- Supports 2TB drives
- Max file size is 4GB
- Compatible with different operating systems
- Used on USB drive, flash drives, memory card, digital cameras, etc.
- No built-in compression or encryption
Logical Volume Manager
The LVM (logical volume manager) is an abstraction software used on Linux systems that sits between the disk drives and file system. LVM adds a robust set of features to the file system that allows users to create volumes across disks and supports various RAID levels. Volume managers have been around for more than a decade. Some of the popular ones back in the day were the Veritas Volume Manager, HP-UX, and Sun Volume Manager. Now, it’s free and comes with various Linux distros. Some of the features include the following:
- One volume group can run on multiple drives
- Resize, create, and delete volumes on the fly
- Add disk drives, and volume expands dynamically
- Supports snapshots, copy on writes, and read/write snapshots
- Supports software RAID 0, 1, 5, and 6
