TensorFlow-Serving: Flexible, High-Performance ML Serving
arXiv (Cornell University)2017
Citations Over Time
Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, Jordan Soyke
Abstract
We describe TensorFlow-Serving, a system to serve machine learning models inside Google which is also available in the cloud and via open-source. It is extremely flexible in terms of the types of ML platforms it supports, and ways to integrate with systems that convey new models and updated versions from training to serving. At the same time, the core code paths around model lookup and inference have been carefully optimized to avoid performance pitfalls observed in naive implementations. Google uses it in many production deployments, including a multi-tenant model hosting service called TFS^2.
Related Papers
- → Deep Residual Learning for Image Recognition(2016)216,875 cited
- → TensorFlow: a system for large-scale machine learning(2016)6,329 cited
- → Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective(2018)579 cited
- Clipper: a low-latency online prediction serving system(2017)
- → MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems(2015)1,917 cited