CODEWITHSUNDEEP
deep-learningmodeltensorflow-serving
user602874
user602874

Reputation: 1489

how to serve deep learning model without GPU

Due to the cost-saving, I'm running a deep learning model with a regular CPU. It takes 10 seconds to finish a request and it's written in python.

I'm thinking about to improve the perf by using java, C++, or rust. Is there any existing rust framework to pick a deep learning model.

Upvotes: 1

Views: 540

Answers (2)

user14090549
user14090549

Reputation:

Is there any existing rust framework to pick a deep learning model.

While I am not familiar with rust framework. If you are running you model on intel cpu, I would suggest to export model using ONNX and run it with mxnet with Intel MKLDNN backend. This should give you the most performance as it uses Intel MKLDNN and Intel MKL library. You can use C++/Python.

Install mxnet with MKLDNN

https://mxnet.apache.org/versions/1.6/api/python/docs/tutorials/performance/backend/mkldnn/mkldnn_readme.html

Upvotes: 1

tritsu
tritsu

Reputation: 161

Tensorflow's performance critical parts are written in C++. Using other language won't cause a drastic performance difference. You may Quantize your network or do a Network Pruning to increase performance.

Upvotes: 0

Related Questions