Running AI workloads on Kubernetes introduces job orchestration challenges that
the default scheduler was not designed to handle. Distributed training requires
all workers to start simultaneously (gang scheduling). Multiple teams competing
for GPUs need fair resource allocation. Jobs must be prioritized, queued, and
sometimes preempted to make room for higher-priority work. This paper examines
these orchestration challenges in detail: gang scheduling, resource fairness, queue
management, preemption, priority scheduling, and resource reservation with
backfill.