Programming Limitations and Challenges on Consumer Hardware GPU

Abstract

Traditionally, high-performance computing was dominated by processors specifically designed to execute highly efficient mathematical calculations. But there are major architectural, thermal and software challenges involved in using consumer-grade and edge GPU hardware. We discuss the structural shortcomings in GPU programming, including a lack of fine-grained preemptive multitasking, thermal throttling under tight power limits, and memory vulnerabilities in the absence of enterprise-grade Error Checking and Correction on graphics cards. We also examine new software bottlenecks, specifically in LLM serving, and address challenges in distributed orchestration. In this work, we review existing frameworks such as FLEP, Phoenix and FlowPrefill to exemplify the need for full-stack solutions that are holistic in scope (encompassing hardware-level pre-emption via resource-efficient flow control, spatial sharing among diverse workloads, as well as dynamic scheduling) if we want to navigate around limitations posed by emerging heterogeneous compute architectures.