Benchmarking NLP and Computer Vision Models on Domain-Specific Architectures: Standard vs. TensorRT-Optimized Performance

Abstract

In this work we systemically study performance of deep learning models for Natural Language Processing (NLP) and Computer Vision (CV) tasks using two popular representative architectures, ResNet-50 and BERT across two configurations: a standard ONNX Runtime configuration and an optimized TensorRT configuration. The main goal is to measure and compare inference time, throughput, CPU and GPU utilization as well as memory usage of all models on Domain-Specific Architectures (DSAs), in this case an NVIDIA GeForce RTX 3060 GPU. We experimentally demonstrate these trade-offs of latency-focused and throughput-focused optimizations and implications for at-scale deployment in realistic resource-constrained environments. The main findings show that the TensorRT-Optimized configuration yields a much higher throughput (up to 432 inferences per second with ResNet-50), and that the Standard configuration shows lower inference time, which is more appropriate for latency-sensitive applications. It should be noted that due to its dense transformer structure and large number of parameters, BERT has a much higher resource demand than ResNet-50, highlighting how model choices need to match performance with the constraints imposed by their deployment. Analysis of CPU and GPU utilization further illustrates the efficiency gains and potential bottlenecks associated with each configuration. Along with the benchmarking results, we also describe optimizations for serving the model: dynamic batching, mixed-precision training and memory management techniques to improve throughput as well as inference time. This study gives practitioners rich information to choose between model configurations and optimization strategies for an effective deployment of NLP and CV models on DSAs.