Abstract: While there are many benchmark suites for heterogeneous computing containing collections of benchmarks in various accelerator language extensions, there is none for vendor-independent systematic benchmarking and analysis of heterogeneous algorithms. Rosetta-Bench is a benchmarking framework inspired by Google Benchmark, as well as a benchmark suite containing equivalent benchmarks written in the most common parallel languages (OpenMP, CUDA, HIP, SYCL, \dots) which permits the use of the best supported programming model for each vendor, as well as the comparison of the different languages to each other.
As a more general benchmarking framework, Rosetta-Bench makes it easy to run custom benchmarks. Just add the benchmark code to run into a folder and Rosetta-Bench will do the rest: Distributed execution, Time measurement on CPU and accelerator, probing, autotuning, statistical analysis, bias mitigation, hypothesis testing, comparison, console and HTML report, and generation of figures. We hope that the availability of this framework will improve the quality of performance evaluations in paper publications.
*Footnote: Inspired by https://youtu.be/fX2W3nNjJIo?t=4303
Bio: Michael Kruse graduated with a Master’s degree in computer science from the University of Paderborn (Germany) and a PhD from the University Paris-Sud 11. After a Post-Doc at the École Normale Supérieur, he currently works at Argonne National Laboratory.