SPEC Machine Learning Committee to Develop Vendor-Agnostic Benchmark to Measure End-to-End Performance for Machine Learning Training and Inference Tasks

By Arthur Kang, SPEC ML Committee Chair

The SPEC Open Systems Group (OSG) reached an important milestone this year with the establishment of the Machine Learning Committee, which is developing practical methodologies for benchmarking artificial intelligence (AI) and machine learning (ML) performance in the context of real-world platforms and environments.

Over the last few years, AI and machine learning (ML) have matured from small applications and pilot programs to increasingly being incorporated into large-scale production systems. In fact, IDC expects enterprises to spend nearly $432.8 billion on AI software, hardware and services in 2022. These enterprises deserve to know exactly what that money will buy, but the complexity of ML algorithms and the wide variety of use cases makes determining the right hardware and software to purchase a tricky proposition. This is why the ML Committee is designing and developing vendor-agnostic benchmarks based on the real-world needs of the industry. These benchmarks will enable vendors to prove their solutions against those of their competitors and help enterprises make more informed buying decisions.

The ML Committee's first benchmark, SPEC ML, will measure end-to-end performance of a system under test (SUT) handling ML training and inference tasks. This benchmark will be designed to better represent industry practices compared to other existing benchmarks by including major parts of the end-to-end ML/DL pipeline, including data prep and training/inference. As a vendor-neutral third-party benchmark, SPEC ML will enable ML system designers to accurately benchmark the performance of their solutions, while allowing ML users, such as enterprises and scientific research institutions, to better understand how solutions will perform in real-world environments.

The creation of the ML Committee is also a great opportunity for anyone interested in the future of AI and ML processing, especially ML/DL end users and manufacturers, to join the Committee and help shape these invaluable benchmarks, which in turn will help shape how AI and ML are deployed and used in enterprises, government agencies and the scientific community. For more information on becoming a member, visit membership, or contact info@spec.org.

[Back to SPEC blog]