Seldon Core is an open-source platform that helps users deploy machine learning models at scale. It has 2M+ installs with companies like EdgeCast (formerly Verizon Media Platform) using it as part of their software stack. The company behind the product is Seldon.
The platform runs on Kubernets and it enables users to deploy thousands of ML workloads out of the box, supporting multiple frameworks/libraries like TensorFlow, PyTorch, and scikit-learn.
In the white paper “Hidden Technical Debt in Machine Learning Systems,” Google engineers explain an “uncomfortable trend” in the industry that’s occurring, companies are rushing to provide fast and cheap deployment of machine learning models but failing to see the technical debt which is accumulating. When it comes to technical debt, it’s different in machine learning because ML systems incur issues not only in traditional code but also ML-specific issues that are at the system level. Thus, in the long term, even if software code is well maintained, ignoring system issues will continue to build until something breaks or worse. The diagram below illustrates the different areas where users must focus on to alleviate technical debt in the ML process.
Seldon Core plays an important part in a few area as illustrated in the graph below. The areas are serving infrastructure, machine resource management, analysis tools, process management tools, and monitoring. However, the product focus is on deployment.
Seldon Core is cloud-native and runs on-prem or in the cloud. It sits on top of Kubernetes and leverages existing tools like Argo (workflow engine), Istio (service mesh), and Abassador (Kubernetes-native API Gateway). For commercial users, they have the option of buying prepackaged inference servers that are loaded with a software stack that includes SKLearn, XGBoost, TensorFlow, MLflow, or custom.
One of the tasks in the deployment process is to containerize (Dockerize) the model so it can be used. This can be done with open source tools like Source-to-Image which take the source code and converts it into an image. All in all, Seldon Core supports a wide spectrum of open source tools in the areas of machine learning, Kubernetes, monitoring, and much more. Here is a sample of 3rd party products supported:
- SkLearn
- TensorFlow
- SageMaker
- MLFlow
- PyTorch
- H2O
- NVIDIA TensorRT Server
- TensorFlow Serving
- Intel nGraph
- Intel OpenVINO