Nvidia Orin Appears in MLPerf

The newest set of MLPerf Inference results showcase the same old vendors; almost all the data-center and edge accelerators came from Nvidia and Qualcomm. Orin was the notable newcomer.

Linley Gwennap

The newest set of MLPerf Inference results showcase the same old vendors; almost all the data-center and edge accelerators came from Nvidia and Qualcomm. The one notable newcomer was Nvidia’s Orin processor, which targets automotive and edge equipment.

Officially in the “preview” category until production shipments in 2H22, Orin achieved 6,139 images per second (IPS) on ResNet-50 v1.5 Offline, less than a low-end Ampere A10 card but using much less power. In fact, Orin scores twice Ampere’s IPS per watt on ResNet, though its lead shrinks on larger benchmarks that stress the edge processor’s diminutive cache.

Qualcomm’s Cloud AI 100 DLA trounces Orin on ResNet at 23,808 IPS. At that speed, however, it uses 75W, and the comparison is unfair since the Nvidia chip lacks host CPUs. To even things up, Qualcomm paired a Snapdragon host processor with a 20W Cloud AI chip for the MLPerf power comparison; this combination nearly triples Orin’s system-level ResNet IPS per watt while also topping Orin’s efficiency on the other three benchmarks Qualcomm submitted.

By packing 18 PCIe cards into a single 2U server, Qualcomm generated greater ResNet-50 performance than a 6U DGX-A100 system with eight Ampere cards. This comparison shows the Cloud AI 100 can deliver greater performance density, at least for image-processing models.