Abstract
The rising complexity of Artificial Intelligence (AI) applications significantly increases the demand for computing power to execute and train Machine Learning (ML) models, thus boosting the energy consumption of data centers. GPUs, enhanced by developments like tensor cores (2017), have become the preferred architecture. Building more efficient ML computing systems relies on a deep understanding of the limits of both parts of a tightly coupled hardware/software paradigm. However, the high abstraction of ML frameworks and the closed-source, proprietary design of state-of-the-art GPU architectures obscure the execution process and make performance evaluation tedious.The main goal of this thesis is to provide new methodologies to evaluate performance and energy bottlenecks of GPU-accelerated ML workloads. Existing profiling solutions are limited in three ways. First, ML framework profiling tools are designed to assist the development of ML models but do not give insights into the runtime execution of the ML framework. While these profiling tools provide high-level metrics on the GPU device execution, these metrics can be misleading and overestimate the utilization of the GPU resources. Second, lower-level profiling tools provide access to performance counters and insights on how to optimize GPU kernels. However, these tools cannot capture the efficiency of host/device interactions occurring at a higher level. Finally, when evaluating energy bottlenecks, the mentioned profiling tools cannot provide a detailed breakdown of the energy consumed by modern GPUs during ML training. To tackle these shortcomings, this thesis makes three key contributions organized as a top-down analysis of GPU-accelerated ML workloads.First, we analyze ML frameworks' runtime execution on a CPU-GPU tandem. We propose a new profiling methodology that leverages data from an ML framework's profiler. We use this methodology to provide new insights into the runtime execution of inference, for three ML models. Our results show that GPU kernels' execution must be long enough to hide the runtime overhead of the ML framework, increasing GPU utilization. However, this strive for longer kernel execution leads to the use of bigger batches of data, seemingly pushing the need for more GPU memory.Second, we analyze the utilization of GPU resources when performing ML training. We propose a new profiling methodology combining the use of high-level and low-level profilers to provide new insights into the utilization of the GPU's inner components. Our experiments, on two modern GPUs, suggest that bigger GPU memory helps enhance throughput and utilization from a high level. However, our results also suggest that a plateau has been reached, eliminating the push for bigger batches. Furthermore, we observe that the fastest GPU cores (tensor cores) are idle most of the time, and the tested workloads are now limited by kernels that do not use these cores. Thus, our results suggest that the current GPU paradigm is reaching a saturation point.Finally, we analyze the energy consumption of GPUs during ML training. We propose an energy model and calibration methodology that uses microbenchmarks to provide a breakdown of the GPU energy consumption. We implement and validate this approach with a modern NVIDIA GPU. Our results suggest that data movement is responsible for most of the energy consumption (up to 84% of the dynamic energy consumption of the GPU). This further motivates the push for newer architectures, optimizing memory accesses (e.g., processing in/near memory, vectorized architectures).This thesis provides a comprehensive analysis of the performance and energy bottlenecks of GPU-accelerated ML workloads. We believe our contributions uncover some of the limitations of current GPU architectures and motivate the need for more advanced profiling techniques to design more efficient ML accelerators. We hope that our work will inspire future research in this direction.