前面系列博客中有用tensorRT、OpenVINO加速模型推理
TensorRT加速方法介绍（python pytorch模型）_竹叶青lvye的博客-CSDN博客_tensorrt加速
?TensorRT加速模型推断时间方法介绍（c++ pytorch模型）_竹叶青lvye的博客-CSDN博客
OpenVINO使用介绍_竹叶青lvye的博客-CSDN博客_openvino resnet在
?
这边再简单提下ONNX Runtime的使用，上面博客中只是将ONNX模型作为一个中间转换模型用，可能不怎么去接触更为原生态的推理框架ONNX Runtime
ONNX Runtime | Home
https://onnxruntime.ai/从官网介绍看，其也是提供训练用API调用的
此时cuda、cuddn、python版本同前面博客时的配置
OpenVINO使用介绍_竹叶青lvye的博客-CSDN博客_openvino resnet
Ubuntu 20.04
python3.6.13 (Anaconda)
cuda version: 11.2
cudnn version: cudnn-11.2-linux-x64-v8.1.1.33
这边还是按照官方配置，去推断一张图片，pip安装下如下库
pip install onnxruntime-gpu?博主这边还是拿tensorflow来示例
pip install tf2onnx代码如下：
import tensorflow as tf
import numpy as np
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications import resnet50
from tensorflow.keras.applications.resnet50 import preprocess_input, decode_predictions
from PIL import Image
import time
import tf2onnx
import onnxruntime as rt
import cv2
import time
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)
#加载预训练模型
model = resnet50.ResNet50(weights='imagenet')
spec = (tf.TensorSpec((None, 224, 224, 3), tf.float32, name="input"),)
output_path = "test.onnx"
model_proto, _ = tf2onnx.convert.from_keras(model, input_signature=spec, opset=13, output_path=output_path)
output_names = [n.name for n in model_proto.graph.output]
providers = ['CPUExecutionProvider']
m = rt.InferenceSession(output_path, providers=providers)
img = cv2.imread('2008_002682.jpg')
img = cv2.resize(img,(224,224))
img_np = np.array(img, dtype=np.float32) / 255.
img_np = np.expand_dims(img_np, axis=0)
print(img_np.shape)
t_model = time.perf_counter()
onnx_pred = m.run(output_names, {"input": img_np})
print(f'do inference cost:{time.perf_counter() - t_model:.8f}s')
print('ONNX Predicted:', decode_predictions(onnx_pred[0], top=3)[0])
?测试图片，还是前面常用的小猫图片
?
部分执行结果如下：
2022-05-01 22:41:51.165567: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1144] Optimization results for grappler item: graph_to_optimize
constant_folding: Graph size after: 556 nodes (-320), 891 edges (-320), time = 253.008ms.
function_optimizer: function_optimizer did nothing. time = 0.565ms.
constant_folding: Graph size after: 556 nodes (0), 891 edges (0), time = 111.51ms.
function_optimizer: function_optimizer did nothing. time = 0.847ms.
(1, 224, 224, 3)
do inference cost:0.01735498s
ONNX Predicted: [('n01930112', 'nematode', 0.13559905), ('n03041632', 'cleaver', 0.04139605), ('n03838899', 'oboe', 0.03445778)]
Process finished with exit code 0
预测结果同OpenVINO下的结果，耗时更少。暂时不再深入了解了，官网上也提供了一些方面资料，后面需要时再用吧。
?
