APEX: Efficient LLM Inference on Low-Memory GPUs

THE CHALLENGE

Deploying large language models efficiently has become a major business obstacle as GPU memory limitations increasingly drive-up infrastructure costs and restrict scalability. During inference, the rapid growth of the key value cache consumes GPU memory quickly, limiting batch sizes and output lengths and reducing overall system throughput. While hybrid CPU GPU approaches attempt to offset this by leveraging abundant CPU memory, they often introduce significant performance penalties due to data transfer overheads, redundant linear layer computations, and imbalanced processor utilization. For organizations, this translates into slower response times, underused hardware, and the need for costly hardware upgrades or model downsizing to maintain service quality. The absence of adaptive solutions that can dynamically balance workloads across heterogeneous hardware further exacerbates these inefficiencies, making it difficult to deploy large, high-quality models in a cost effective and predictable manner on commodity or legacy systems.

OUR SOLUTION

APEX enables organizations to run large language models efficiently on existing, memory constrained GPUs by intelligently coordinating CPU and GPU resources to remove the key value cache bottleneck that limits scale and performance. By allowing the GPU to focus on compute intensive linear layers while asynchronously offloading only memory bound attention tasks to the CPU, APEX keeps expensive GPU hardware fully utilized without repeating work or increasing latency. A profiling informed dynamic scheduler continuously adapts execution decisions based on the available hardware and live workload conditions, ensuring consistent throughput across decode heavy and mixed inference scenarios. From a business perspective, this approach unlocks larger batch sizes, higher throughput, and predictable performance without sacrificing model accuracy or requiring costly hardware upgrades. As a drop in software system compatible with standard LLM serving stacks, APEX helps enterprises, cloud providers, and edge deployments maximize existing infrastructure, reduce operational costs, and scale AI services more reliably and efficiently.

Figure: APEX System Architecture.

Advantages:

• Up to 2× higher inference throughput on memory constrained GPUs
• Larger batch sizes and longer sequences without GPU out of memory errors
• Low and stable latency for decode heavy workloads
• Drop in compatibility with existing LLM serving stacks

Potential Application:

• Cloud and SaaS LLM inference on memory constrained GPUs
• Enterprise on premise and private cloud LLM deployments
• Edge and micro datacenter LLM inference on commodity hardware
• Chatbot and generative assistant platforms