Beyond job orchestration, AI workloads face challenges related to the underlying
infrastructure: hardware topology affects communication performance,
heterogeneous GPU types require careful placement, expensive GPUs should be
shared when possible, and long-running jobs must tolerate failures. This paper
examines these resource and infrastructure challenges: topology awareness,
resource heterogeneity, GPU utilization and sharing, scalability, I/O bottlenecks,
fault tolerance and elasticity, and budget constraints.