MLCommonstm Releases MLPerftm Tiny Inference Benchmark

Today, MLCommons, an open engineering consortium, launched a new benchmark, MLPerftm Tiny Inference, to measure how quickly a trained neural network can process new data for extremely low-power devices in the smallest form factors and included an optional power measurement. MLPerf Tiny v0.5 is the organization's first inference benchmark suite that targets machine learning use cases on embedded devices.

Embedded machine learning is a burgeoning field where AI-driven sensor data analytics is performed in real-time, close to where the data resides. The new MLPerf Tiny Inference benchmark suite captures a variety of use cases that involve "tiny" neural networks, typically 100 kB and below, that process sensor data such as audio and vision to provide endpoint intelligence.

The first v0.5 round included five submissions from academic, industry organizations, and national labs, producing 17 peer-reviewed results. Submissions this round included software and hardware innovations from Latent AI, Syntiant, PengCheng Labs, Columbia, UCSD, CERN, and Fermilab. To view the results, please visit https://www.mlcommons.org/en/inference-tiny-05.

MLPerf Tiny Inference: A New Measurement to Advance Intelligence in Everyday Devices

As a new benchmark, MLPerf Tiny Inference enables reporting and comparison of embedded ML devices, systems, and software. Developed in partnership with EEMBCtm, the Embedded Microprocessor Benchmark Consortium, the benchmark consists of four machine learning tasks that encompass the use of microphone and camera sensors with embedded devices:

• Keyword Spotting (KWS), which uses a neural network that detects keywords from a spectrogram;
• Visual Wake Words (VWW), a binary image classification task for determining the presence of a person in an image;
• Tiny Image Classification (IC), a small image classification benchmark with 10 classes; and
• Anomaly Detection (AD), which uses a neural network to identify abnormalities in machine operating sounds.

KWS has several use cases in endpoint consumer devices, such as earbuds and virtual assistants. VWW has application use cases, for instance, with in-home security monitoring. IC has myriad use cases for smart video recognition applications. AD has several applications in industrial manufacturing for tasks such as predictive maintenance, asset tracking and monitoring.

"To understand progress and advance innovation, particularly in edge computing, the ML industry needs benchmarks," said Peter Torelli, President of EEMBC. "Creating new metrics and measurement across neural networks and a variety of form factors is challenging, and we're thrilled to partner with MLCommons to make MLPerf Tiny a reality."

"The goal of MLPerf is to measure performance for machine learning across the full spectrum of systems - from microwatts to megawatts," said Professor Vijay Janapa Reddi of Harvard University and MLPerf Tiny Inference working group chair. "This new benchmark will bring intelligence to devices like wearables, thermostats, and cameras, and further MLCommons' mission to accelerate machine learning innovation to benefit everyone."

"Tiny machine learning is a fast-growing field and will help to infuse ?intelligence' in the small everyday items that surround us," said MLPerf Tiny Inference working group chair, Colby Banbury, of Harvard University. "By bringing MLPerf benchmarks to these devices, we can help to measure performance and drive efficiency improvements over time."

MLPerf Tiny v0.5 marks a major milestone in MLCommons' line-up of MLPerf inference benchmark suites. With the addition of MLPerf Tiny, MLCommons covers the full range of machine learning inference benchmarks, ranging from cloud and datacenter benchmarks that consume kiloWatts of power down to tiny IoT devices that consume only a few milliWatts of power, and everything in between. MLPerf Tiny benchmarks will stimulate tinyML innovation in the academic and research communities and push the state-of-the-art forward in embedded machine learning.

The MLPerf Tiny v0.5 inference benchmarks were created thanks to the contributions and leadership of our working members over the last 18 months, including representatives from: Harvard University, EEMBC, CERN, Columbia, Digital Catapult, Fermilab, Google, Infineon, Latent AI, ON Semiconductor, Peng Cheng Laboratories, Qualcomm, Renesas, SambaNova Systems, Silicon Labs, STMicroelectronics, Synopsys, Syntiant, UCSD, and Voicemed.

The MLPerf Tiny working group recently submitted a paper to the NeurIPS benchmarks and datasets track that provides in-depth information about the design and implementation of the benchmark suite (https://openreview.net/pdf?id=8RxxwAut1BI). Additional information about the MLPerf Tiny Inference benchmarks is available at the github repository (https://github.com/mlcommons/tiny).

About MLCommons

MLCommons is an open engineering consortium with a mission to accelerate machine learning innovation to benefit everyone. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding member partners - global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.

For additional information on MLCommons and details on becoming a member of the organization, please visit http://mlcommons.org/ or contact [email protected].